Alexander Gallego

The Effects of CPU Turbo: 768X stddev

Sat, 30 Jun 2018 20:00:00 -0400

I build and maintain smf - the fastest RPC in the west. The main language supported is C++. We use Seastar as the actual asynchronous framework.

The RPC protocol uses flatbuffers to make cross language support possible, in addition to its main goal of providing zero cost deserialization on the receiving side. Currently, Go and Java are in the process of being added to the repo, with partial support already merged.

On Friday, June 28 2018, I got a text from a friend that said: last time I checked flatbuffers was slow vs cap 'n proto. At first, I was suspicious, since I’ve never seen flatbuffers be in any top $> perf profile of any kind, on real production applications. However, I had no numbers or benchmarks to prove it.

I thought I had a quick-and-dirty hack to amortize the cost of object graphs - it never occurred to me to measure large type trees or large buffers.

BLUF - Bottom Line Up Front.

I had introduced a performance optimization that actually turned out to be bad for large buffers.

To be precise:

On small very buffers it was ~6% faster
On large buffers it was ~41% slower

nooooooooooooooooooooooooooo!

The beginning of yak-shaving

I found a project that had benchmarked cap'n proto vs flatbuffers. In particular, it measures buffer construction, since both cap’n proto and flatbuffers deserialization is effectively a pointer cast - Yay for little endian enforced encoding into an aligned byte array.

The cpp-serializers project measures cap’n proto vs flatbuffers encoding and to my surprise, flatbuffers measured at 2.5X slower.

Their test:

auto start = std::chrono::high_resolution_clock::now();
for (size_t i = 0; i < iterations; i++) {
    flatbuffers::FlatBufferBuilder builder;
    strings.clear();

    for (size_t i = 0; i < kStringsCount; i++) {
        strings.push_back(builder.CreateString(kStringValue));
    }

    auto ids_vec = builder.CreateVector(kIntegers);
    auto strings_vec = builder.CreateVector(strings);
    auto r1 = CreateRecord(builder, ids_vec, strings_vec);
    builder.Finish(r1);

    auto p = reinterpret_cast<char*>(builder.GetBufferPointer());
    auto sz = builder.GetSize();
    std::vector<char> buf(p, p + sz);
    auto r2 = GetRecord(buf.data());
    (void)r2->ids()[0];

    builder.ReleaseBufferPointer();
}
auto finish = std::chrono::high_resolution_clock::now();
auto duration = std::chrono::duration_cast<std::chrono::milliseconds>(finish - start).count();

As you can see, the benchmark creates a flatbuffers builder (a kind of std::vector). It stresses the builder by allocating an array of strings and ints… and that’s about it.

Internally, the flatbuffers::FlatbufferBuilder encodes from top-to-bottom (downward growth) and when it reaches the bottom, it reallocs and grows the size of the underlying array to fit the contents.

The actual code for reallocation is:

void reallocate(size_t len) {
    FLATBUFFERS_ASSERT(allocator_);
    auto old_reserved = reserved_;
    auto old_size = size();
    auto old_scratch_size = scratch_size();
    reserved_ += (std::max)(len,
                            old_reserved ? old_reserved / 2 : initial_size_);
    reserved_ = (reserved_ + buffer_minalign_ - 1) & ~(buffer_minalign_ - 1);
    if (buf_) {
      buf_ = allocator_->reallocate_downward(buf_, old_reserved, reserved_,
                                             old_size, old_scratch_size);
    } else {
      buf_ = allocator_->allocate(reserved_);
    }
    cur_ = buf_ + reserved_ - old_size;
    scratch_ = buf_ + old_scratch_size;
}

It grows by powers of 2 (reserved_ + buffer_minalign_ - 1) & ~(buffer_minalign_ - 1); , and usually by 1024 bytes which is the default initial_size_;

Setup

Let’s start with a Flatbuffers IDL of a simple key=value struct.

    table kvpair {
      key:   string;
      value: string;
    }

The C++ API has a nice object api on top of the raw flatbuffers::FlatbufferBuilder, so a quick GoogleBenchmark yields something like this:

static inline kvpairT
gen_kv(uint32_t sz) {
  kvpairT ret;
  ret.key.resize(sz, 'x');
  ret.value.resize(sz, 'y');
  return ret;
}

static void
BM_alloc_simple(benchmark::State &state) {
  for (auto _ : state) {
    state.PauseTiming();
    auto kv = gen_kv(state.range(0));
    state.ResumeTiming();

    // build it!
    flatbuffers::FlatBufferBuilder bdr;
    bdr.Finish(kvpair::Pack(bdr, &kv, nullptr));
    auto mem = bdr.Release();
    auto ptr = reinterpret_cast<char *>(mem.data());
    auto sz = mem.size();
    auto buf = seastar::temporary_buffer<char>(
      ptr, sz, seastar::make_object_deleter(std::move(mem)));
    benchmark::DoNotOptimize(buf);
  }
}
BENCHMARK(BM_alloc_simple)
  ->Args({1 << 1, 1 << 1})
  ->Args({1 << 2, 1 << 2})
  ->Args({1 << 4, 1 << 4})
  ->Args({1 << 8, 1 << 8})
  ->Args({1 << 12, 1 << 12})
  ->Args({1 << 16, 1 << 16})
  ->Args({1 << 18, 1 << 18});

A few things to note:

We use auto mem = bdr.Release(); and wrap that into a seastar::temporary_buffer<char> with effectively zero copy (+- some pointer assignments).
This is important because our entire messaging API is about bridging the code generation from flatbuffers and seastar.
In addition to this, our RPC mechanism codegen’s seastar <–> flatbuffers glue code.

---------------------------------------------------------------------------
Benchmark                                    Time           CPU Iterations
---------------------------------------------------------------------------
...

BM_alloc_simple/2/2/threads:1                        586 ns        588 ns    1184218
BM_alloc_simple/4/4/threads:1                        586 ns        589 ns    1181340
BM_alloc_simple/16/16/threads:1                      598 ns        600 ns    1153577
BM_alloc_simple/256/256/threads:1                    601 ns        602 ns    1147963
BM_alloc_simple/4096/4096/threads:1                  824 ns        828 ns     837036
BM_alloc_simple/65536/65536/threads:1               6825 ns       6845 ns     101321

...

To encode a 2 x 64KB buffers it takes 6.8 micros. Yikes! - But is this bad? (as of flatbuffers checkin 34cb163e389e928db08ed2bd0e16ee0ac53ab1ce). Note this is only 3X the cost of std::malloc + std::memset, which is the fastest thing I can think of as a base comparison.

Before we dive into possible optimizations, let’s fix our desktop for these micro optimizations.

Fixing the environment

function cpu_disable_performance_cpupower_state(){
    sudo cpupower frequency-set --governor powersave
}
function cpu_enable_performance_cpupower_state(){
    sudo cpupower frequency-set --governor performance
}
function cpu_available_frequencies() {
    for i in /sys/devices/system/cpu/cpu[0-9]*; do
        echo "$i:"
        echo "$i/cpufreq/scaling_min_freq: $(cat $i/cpufreq/scaling_min_freq)";
        echo "$i/cpufreq/scaling_max_freq: $(cat $i/cpufreq/scaling_max_freq)";
    done
}

function cpu_set_min_frequencies() {
    local freq=$1;
    if [[ $freq == "" ]]; then exit 1; fi
    for i in /sys/devices/system/cpu/cpu[0-9]*; do
        echo "$i:"
        echo "$i/cpufreq/scaling_min_freq: $(cat $i/cpufreq/scaling_min_freq)";
        echo "$freq" | sudo tee "$i/cpufreq/scaling_min_freq"
        echo "$i/cpufreq/scaling_min_freq: $(cat $i/cpufreq/scaling_min_freq)";
    done
}

function cpu_set_max_frequencies() {
    local freq=$1;
    if [[ $freq == "" ]]; then exit 1; fi
    for i in /sys/devices/system/cpu/cpu[0-9]*; do
        echo "$i:"
        echo "$i/cpufreq/scaling_max_freq: $(cat $i/cpufreq/scaling_max_freq)";
        echo "$freq" | sudo tee "$i/cpufreq/scaling_max_freq"
        echo "$i/cpufreq/scaling_max_freq: $(cat $i/cpufreq/scaling_max_freq)";
    done
}

My CPU is a Intel(R) Xeon(R) CPU E3-1535M v6 @ 3.10GHz with a turbo of 4.20GHz. That’s great for desktop experience where interactivity matters and terrible for performance benchmarking.

When I first tried to show the results to my partner (non CS Major, but puts up w/ me asking her to stare at my screen) I had unpredictable results.

Armed w/ the shell funcs above, my checklist is now as follows:

Settings

Ensure that your BIOS says performance when connected to AC
Check your CPU freqencies via cat /proc/cpuinfo | grep model
Set your CPU governor via cpu_enable_performance_cpupower_state
Set the min frequency to the frequency reported by your CPU/model/vendor via: cpu_set_min_frequencies 3100000 in my case
Set the max frequency to the frequency reported by your CPU/model/vendor via: cpu_set_max_frequencies 3100000 in my case
Verify that you always build in Release mode

Verying the frequencies is now simple via cpu_available_frequencies

 $> cpu_available_frequencies
/sys/devices/system/cpu/cpu0:
/sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq: 3100000
/sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq: 3100000
/sys/devices/system/cpu/cpu1:
/sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq: 3100000
/sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq: 3100000
/sys/devices/system/cpu/cpu2:
/sys/devices/system/cpu/cpu2/cpufreq/scaling_min_freq: 3100000
/sys/devices/system/cpu/cpu2/cpufreq/scaling_max_freq: 3100000
/sys/devices/system/cpu/cpu3:
/sys/devices/system/cpu/cpu3/cpufreq/scaling_min_freq: 3100000
/sys/devices/system/cpu/cpu3/cpufreq/scaling_max_freq: 3100000
/sys/devices/system/cpu/cpu4:
/sys/devices/system/cpu/cpu4/cpufreq/scaling_min_freq: 3100000
/sys/devices/system/cpu/cpu4/cpufreq/scaling_max_freq: 3100000
/sys/devices/system/cpu/cpu5:
/sys/devices/system/cpu/cpu5/cpufreq/scaling_min_freq: 3100000
/sys/devices/system/cpu/cpu5/cpufreq/scaling_max_freq: 3100000
/sys/devices/system/cpu/cpu6:
/sys/devices/system/cpu/cpu6/cpufreq/scaling_min_freq: 3100000
/sys/devices/system/cpu/cpu6/cpufreq/scaling_max_freq: 3100000
/sys/devices/system/cpu/cpu7:
/sys/devices/system/cpu/cpu7/cpufreq/scaling_min_freq: 3100000
/sys/devices/system/cpu/cpu7/cpufreq/scaling_max_freq: 3100000

The take away is you don’t necessarily want speed, you want predictability.

Next - Compiler Explorer - my new fav tool

First, let’s execute the commands above and get the compiler flags that we’ll need for compiler explorer.

If you are using CMake, follow these screencasts and don’t forget to set set(CMAKE_EXPORT_COMPILE_COMMANDS 1) on your main CMakeLists.txt file

Now that we have the compiler flags ready, let’s fire up compiler explorer:

When you navigate to localhost:1024 you are welcomed with the usual and friendly CompilerExplorer UI.

With these tools we can now benchmark & optimize our code

The results

Running /home/agallego/workspace/smf/build/release/src/benchmarks/fbs_alloc/smf_fbsalloc_benchmark_test
Run on (8 X 4200 MHz CPU s)
CPU Caches:
  L1 Data 32K (x4)
  L1 Instruction 32K (x4)
  L2 Unified 256K (x4)
  L3 Unified 8192K (x1)
---------------------------------------------------------------------------
Benchmark                                    Time           CPU Iterations
---------------------------------------------------------------------------
BM_malloc_base/2/2/threads:1                          11 ns         11 ns   62858390
BM_malloc_base/4/4/threads:1                          11 ns         11 ns   61565906
BM_malloc_base/16/16/threads:1                        11 ns         11 ns   64118519
BM_malloc_base/256/256/threads:1                      20 ns         20 ns   35574778
BM_malloc_base/4096/4096/threads:1                    98 ns         98 ns    7123958
BM_malloc_base/65536/65536/threads:1                2339 ns       2336 ns     281254
BM_malloc_base/262144/262144/threads:1             11648 ns      11635 ns      58503
BM_alloc_simple/2/2/threads:1                        586 ns        588 ns    1184218
BM_alloc_simple/4/4/threads:1                        586 ns        589 ns    1181340
BM_alloc_simple/16/16/threads:1                      598 ns        600 ns    1153577
BM_alloc_simple/256/256/threads:1                    601 ns        602 ns    1147963
BM_alloc_simple/4096/4096/threads:1                  824 ns        828 ns     837036
BM_alloc_simple/65536/65536/threads:1               6825 ns       6845 ns     101321
BM_alloc_simple/262144/262144/threads:1            32838 ns      32784 ns      21351
BM_alloc_thread_local/2/2/threads:1                  557 ns        559 ns    1220122
BM_alloc_thread_local/4/4/threads:1                  557 ns        559 ns    1221065
BM_alloc_thread_local/16/16/threads:1                569 ns        571 ns    1198623
BM_alloc_thread_local/256/256/threads:1              578 ns        580 ns    1181208
BM_alloc_thread_local/4096/4096/threads:1            806 ns        819 ns     845698
BM_alloc_thread_local/65536/65536/threads:1        10182 ns      10237 ns      68206
BM_alloc_thread_local/262144/262144/threads:1      43066 ns      42998 ns      16279
BM_alloc_hybrid/2/2/threads:1                        563 ns        565 ns    1212235
BM_alloc_hybrid/4/4/threads:1                        563 ns        566 ns    1209207
BM_alloc_hybrid/16/16/threads:1                      577 ns        579 ns    1185042
BM_alloc_hybrid/256/256/threads:1                    587 ns        589 ns    1166182
BM_alloc_hybrid/4096/4096/threads:1                  843 ns        847 ns     819194
BM_alloc_hybrid/65536/65536/threads:1               6810 ns       6832 ns     101497
BM_alloc_hybrid/262144/262144/threads:1            32838 ns      32774 ns      21368

As expected, for small buffers, there is a large cost vs the base of malloc+memset, and at the higher ends, the allocation & byte traversal start to dominate. The range is ~58x (worst) - ~2.8X (best).

Our benched code

#include <cstring>
#include <memory>
#include <thread>

#include <benchmark/benchmark.h>
#include <core/print.hh>
#include <core/sstring.hh>
#include <core/temporary_buffer.hh>

#include "kv_generated.h"
#include "smf/native_type_utils.h"

static inline kvpairT
gen_kv(uint32_t sz) {
  kvpairT ret;
  ret.key.resize(sz, 'x');
  ret.value.resize(sz, 'y');
  return ret;
}

static void
BM_malloc_base(benchmark::State &state) {
  for (auto _ : state) {
    auto buf = reinterpret_cast<char *>(std::malloc(2 * state.range(0)));
    std::memset(buf, 'x', 2 * state.range(0));
    benchmark::DoNotOptimize(buf);
    std::free(buf);
  }
}
BENCHMARK(BM_malloc_base)
  ->Args({1 << 1, 1 << 1})
  ->Args({1 << 2, 1 << 2})
  ->Args({1 << 4, 1 << 4})
  ->Args({1 << 8, 1 << 8})
  ->Args({1 << 12, 1 << 12})
  ->Args({1 << 16, 1 << 16})
  ->Args({1 << 18, 1 << 18})
  ->Threads(1);


static void
BM_alloc_simple(benchmark::State &state) {
  for (auto _ : state) {
    state.PauseTiming();
    auto kv = gen_kv(state.range(0));
    state.ResumeTiming();

    // build it!
    flatbuffers::FlatBufferBuilder bdr;
    bdr.Finish(kvpair::Pack(bdr, &kv, nullptr));
    auto mem = bdr.Release();
    auto ptr = reinterpret_cast<char *>(mem.data());
    auto sz = mem.size();
    auto buf = seastar::temporary_buffer<char>(
      ptr, sz, seastar::make_object_deleter(std::move(mem)));
    benchmark::DoNotOptimize(buf);
  }
}
BENCHMARK(BM_alloc_simple)
  ->Args({1 << 1, 1 << 1})
  ->Args({1 << 2, 1 << 2})
  ->Args({1 << 4, 1 << 4})
  ->Args({1 << 8, 1 << 8})
  ->Args({1 << 12, 1 << 12})
  ->Args({1 << 16, 1 << 16})
  ->Args({1 << 18, 1 << 18})
  ->Threads(1);

static void
BM_alloc_thread_local(benchmark::State &state) {
  static thread_local flatbuffers::FlatBufferBuilder bdr(1024);
  for (auto _ : state) {
    state.PauseTiming();
    auto kv = gen_kv(state.range(0));
    state.ResumeTiming();

    // key operations
    bdr.Clear();
    bdr.Finish(kvpair::Pack(bdr, &kv, nullptr));
    // ned key operations

    // std::cout << "Size to copy: " << bdr.GetSize() << std::endl;

    void *ret = nullptr;
    auto r = posix_memalign(&ret, 8, bdr.GetSize());
    if (r == ENOMEM) {
      throw std::bad_alloc();
    } else if (r == EINVAL) {
      throw std::runtime_error(seastar::sprint(
        "Invalid alignment of %d; allocating %d bytes", 8, bdr.GetSize()));
    }
    DLOG_THROW_IF(r != 0,
      "ERRNO: {}, Bad aligned allocation of {} with alignment: {}", r,
      bdr.GetSize(), 8);
    std::memcpy(ret, reinterpret_cast<const char *>(bdr.GetBufferPointer()),
      bdr.GetSize());
    benchmark::DoNotOptimize(ret);
    std::free(ret);
  }
}
BENCHMARK(BM_alloc_thread_local)
  ->Args({1 << 1, 1 << 1})
  ->Args({1 << 2, 1 << 2})
  ->Args({1 << 4, 1 << 4})
  ->Args({1 << 8, 1 << 8})
  ->Args({1 << 12, 1 << 12})
  ->Args({1 << 16, 1 << 16})
  ->Args({1 << 18, 1 << 18})
  ->Threads(1);


template <typename RootType>
seastar::temporary_buffer<char>
hybrid(const typename RootType::NativeTableType &t) {
  static thread_local auto builder =
    std::make_unique<flatbuffers::FlatBufferBuilder>();

  auto &bdr = *builder.get();
  bdr.Clear();
  bdr.Finish(RootType::Pack(bdr, &t, nullptr));

  if (SMF_UNLIKELY(bdr.GetSize() > 2048)) {
    auto mem = bdr.Release();
    auto ptr = reinterpret_cast<char *>(mem.data());
    auto sz = mem.size();

    // fix the original builder
    builder = std::make_unique<flatbuffers::FlatBufferBuilder>();

    return seastar::temporary_buffer<char>(
      ptr, sz, seastar::make_object_deleter(std::move(mem)));
  }


  // always allocate to the largest member 8-bytes
  void *ret = nullptr;
  auto r = posix_memalign(&ret, 8, bdr.GetSize());
  if (r == ENOMEM) {
    throw std::bad_alloc();
  } else if (r == EINVAL) {
    throw std::runtime_error(seastar::sprint(
      "Invalid alignment of %d; allocating %d bytes", 8, bdr.GetSize()));
  }
  DLOG_THROW_IF(r != 0,
    "ERRNO: {}, Bad aligned allocation of {} with alignment: {}", r,
    bdr.GetSize(), 8);
  seastar::temporary_buffer<char> retval(reinterpret_cast<char *>(ret),
    bdr.GetSize(), seastar::make_free_deleter(ret));
  std::memcpy(retval.get_write(),
    reinterpret_cast<const char *>(bdr.GetBufferPointer()), retval.size());
  return std::move(retval);
}

static void
BM_alloc_hybrid(benchmark::State &state) {
  for (auto _ : state) {
    state.PauseTiming();
    auto kv = gen_kv(state.range(0));
    state.ResumeTiming();
    auto buf = hybrid<kvpair>(kv);
    benchmark::DoNotOptimize(buf);
  }
}
BENCHMARK(BM_alloc_hybrid)
  ->Args({1 << 1, 1 << 1})
  ->Args({1 << 2, 1 << 2})
  ->Args({1 << 4, 1 << 4})
  ->Args({1 << 8, 1 << 8})
  ->Args({1 << 12, 1 << 12})
  ->Args({1 << 16, 1 << 16})
  ->Args({1 << 18, 1 << 18})
  ->Threads(1);


BENCHMARK_MAIN();

Lessons Learned: Benchmarking with turbo is dishonest.

Not that people writing OSS (or any software really) are out there to get you. It is just easy to forget to tune your machine specifically for benchmarking. There is no such thing as a quick and dirty benchmark especially if the results are not categorically different, i.e.: 1 minute vs 10 mins vs 1hr.

Let’s compare the stability of multiple runs with turbo vs without.

That’s a stddev difference of up to 768X !!!

No-turbo means precisely this:

cpu_available_frequencies

/sys/devices/system/cpu/cpu0:
/sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq: 800000
/sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq: 4200000
/sys/devices/system/cpu/cpu1:
/sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq: 800000
/sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq: 4200000
/sys/devices/system/cpu/cpu2:
/sys/devices/system/cpu/cpu2/cpufreq/scaling_min_freq: 800000
/sys/devices/system/cpu/cpu2/cpufreq/scaling_max_freq: 4200000
/sys/devices/system/cpu/cpu3:
/sys/devices/system/cpu/cpu3/cpufreq/scaling_min_freq: 800000
/sys/devices/system/cpu/cpu3/cpufreq/scaling_max_freq: 4200000
/sys/devices/system/cpu/cpu4:
/sys/devices/system/cpu/cpu4/cpufreq/scaling_min_freq: 800000
/sys/devices/system/cpu/cpu4/cpufreq/scaling_max_freq: 4200000
/sys/devices/system/cpu/cpu5:
/sys/devices/system/cpu/cpu5/cpufreq/scaling_min_freq: 800000
/sys/devices/system/cpu/cpu5/cpufreq/scaling_max_freq: 4200000
/sys/devices/system/cpu/cpu6:
/sys/devices/system/cpu/cpu6/cpufreq/scaling_min_freq: 800000
/sys/devices/system/cpu/cpu6/cpufreq/scaling_max_freq: 4200000
/sys/devices/system/cpu/cpu7:
/sys/devices/system/cpu/cpu7/cpufreq/scaling_min_freq: 800000
/sys/devices/system/cpu/cpu7/cpufreq/scaling_max_freq: 4200000

SMF

smf just got 40% faster for large buffers thanks to these benchmarks, if you give it a shot let me know. Stay tuned for the Java and Go code generators and performance benchmarks coming soon.

Let me know if you found this useful, any missinformation, or additional performance tunning on twitter @emaxerrno.

Special thanks to my partner Sarah Rohrbach as well as Chris Heller, Noah Watkins for reading earlier drafts of this post.

A tale of performance debugging: from 1.3X slower to 48X faster than Apache Kafka

Thu, 08 Feb 2018 19:00:00 -0500

I recently wrote a log file writer that allows for multiple non-sequential writes to an O_DIRECT file handle on XFS. Specifically, it dispatches 4 (configurable) concurrent writes, then does a join of the page-aligned buffer writes. That file writer handles timed-based flushing policy, so that you can have a reasonable idea of when your data actually makes it to disk by either exceeding capacity (say 1MB), or exceeding a timeout (say 1second).

All of this work, was to design a ( very? ) fast Write Ahead Log (the astute reader will be quick to point out that if you don’t flush at every write, is not really a WAL, but just a log writer). After all of this work (~2K LOC), my heart was broken to realize I was actually -1.3X slower than Apache Kafka after the big re-write! sigh… (tested with the non DPDK runtime - for those of you who have been following along).

For the impatient, what follows is the performance debugging process to understand my bottlenecks, and slowly bring performance to a new high of 48X times faster than Apache Kafka, which was my baseline.

1) I used Apache Kafka tail latency as a my largest latency budget.

2) These tests did not use the DPDK runtime. Only epoll + new aio

Max latency budget: Kafka on NVMe + XFS + kernel 4.14.16-300.fc27.x86_64

To have a useful baseline for my largest latency budget, I enabled lz4 compression for Kafka (much slower w/out it), and also increase the number of partitions to 32. This is the same number of partitions that our broker used for the tests.

# config/server.properties
compression.codec=3
num.partitions=32

I was using the latest 1.0 release: kafka_2.11-1.0.0.

# produce 100MM records
#
#
time ./bin/kafka-run-class.sh   \ 
    org.apache.kafka.tools.ProducerPerformance \
        --topic test            \
        --num-records 100000000 \
        --record-size 100       \
        --throughput -1         \
        --producer-props acks=1 bootstrap.servers=localhost:9092

Latency distribution for writing 100MM records with the command above to Apache Kafka 2.11-1.0.

Acknowledgements:

1) I disabled all other apps, browsers, etc.

2) localhost has in-kernel optimizations which will only add latency on a prod deployment which will likely go through a network rack

3) This ignores CPU scaling (power states) among others

perf {stat,record,list} & toplev (PMU_tools)

When in doubt, measure. When 100% sure, still measure. 90% of the improvements actually came from an area I wasn’t expecting. 10% did come from a code path I intuited needed some TLC.

In fact, to be 100% honest, I am still in awe at the results. In part the starting profile and ending profile look superficially similar, as you will soon see. In part because this was the first time I was using hardware counters to actually debug a program and realized that I would not have missed out on some sleep had I learned this earlier in life.

My weight-loss-program is as follows:

perf record -g -p <pid>
perf report
If something non-obvious wasn’t there, then:
- perf stat -d -p <pid>
- perf record -g -p <pid>
- perf report
- CPU Flamegraph
- toplev.py

1st non-obvious performance bottleneck std::sort

diff --git a/src/filesystem/wal_write_behind_cache.cc b/src/filesystem/wal_write_behind_cache.cc
index a35f8ccd..dab6711f 100644
--- a/src/filesystem/wal_write_behind_cache.cc
+++ b/src/filesystem/wal_write_behind_cache.cc
@@ -68,8 +68,6 @@ wal_write_behind_cache::put(uint64_t offset, item_ptr data) {
   stats_.bytes_written += data->on_disk_size();
   puts_.emplace(offset, data);
   keys_.push_back(offset);
-  // keeping order is hugely important for removing data.
-  std::sort(keys_.begin(), keys_.end());
 }

If you decide to repro at home, please comment wal_write_ahead_log.cc:57 and wal_write_ahead_log.cc:23

This is a very slow commit. This is when I found out I needed to do something about it immediately. Look at these numbers! yikes.

….this is 24x improvement!!! - One line of code.

Finding the culprit!

Going back to our weight-loss program:

# assume 24776 is the process id
#
perf record -F 99 -g -p  24776

# run the client test then call:
#
perf report

perf report for the first non-intuitive bugfix

… woah! 74% of the time is spent sorting uint64_t

perf report AFTER the change

… Yay!! no more bottleneck… but wait! Doesn’t this change break correctness? - No, this was a leftover code from a previous imlementation and honestly, pretty harmless for the behavior of the program other than it made it unbearably slow!

2nd improvement: poor data locality - code slaying

Remember our recipe from above? The perf report no longer shows something obvious! … what to do!!??!

perf stat -d -p <pid> to the rescue!

perf stat -d -p

The issue is then… what does this actually mean! LLC stands for last level cache, which in turn it means you are fetching from main memory. … a lot.

This is the first time I’ve used this tool, and instead of doing a perf record -e LLC-cache-misses I decided to fix it. I went ahead and said - oh I know better - it’s basically a bunch of pointer chasing (correct), let me fix it right here…(incorrect)

perf stat -d -p

Changing code from pointers to stack allocated structures did have an improvement on the LLC-cache-misses but, not nearly enough. I ended up ‘fixing’ non broken code because I thought I knew better than my profiler.

3rd improvement: poor data locality - non intuitive fix 2

perf record \ 
   -e LLC-loads,LLC-load-misses,instructions \ 
   -e cycles,branch-load-misses,faults \
   -e bus-cycles,mem-loads,mem-stores -a -p <pid>

perf report with --poll-mode

perf report without --poll-mode

It took a long time to play with the annotated perf report and reading the assembly and the culprits for the LLC-cache-misses, but the biggest contributor was to move fron an indexed datastructure to a simple array. Effectively the map was already pre-ordered as writes happen monotonically. Switching algorithms from a map.find() to std::lower_bound with a std::deque<> was the biggest contributor.

non-intuitive fix 2

The second contributor was to actually to cache the size of the item in wrapper datastructure. That is, I moved from key.ptr->get_size_on_disk() to a strcut that remembered the size_on_disk such that you could simply do key.size_on_disk. That was cached upon insertion into the cache.

This was the the most mind blowing part of this whole thing that in retrospect makes total sense. Jumping around memory (insert refs) is slow and causes CPU stalls if the access patterns are pseudo-random and the CPU prefetcher cannot help you.

latency distribution non-intuitive fix 2

Results

At this stage you can tell:

Kafka.Slowest(1618ms) / SMF.Slowest(34ms) ~= 48 lower latency

# CLIENT: inside smf/build_release
#
#
./src/smfb/client/smfb_low_level_client \ 
              --req-num=12202           \
              --batch-size=8196         \ 
              --cpuset 3,4              \ 
              --key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \ 
              --value=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy    \
              --poll-mode

# SERVER: inside smf/build_release
#
#
../src/smfb/smfb --poll-aio=1             \ 
                 --write-ahead-log-dir=.  \
                 --cpuset 1,2             \
                 --poll-mode

SMF is 48x lower tail latency than Apache Kafka. In fact SMF wrote was at a slight disadvantage in that it wrote more than 100MM records (100'007'592 total records)

End-To-End consistent latency

Server observed latency

Performance, riddle me this!

At this point I was excited that we were back on track for having possibly the fastest open source write ahead log … so I decided to do one more pass through at perf stat -F 99 -d -p <pid> … bad idea.

Original perf stat

After the bugfixes perf stat

…………WHATTTTTT!!!!!! HULK SMASH KEYBOARD!!!!!

After all this work I have more Last-Level-Cache misses?!!? I was honestly going in circles at this point. I asked my friend @duarte_nunes to see what he thought and he recommented for me to take a look at toplev - a program from Andi Kleen to measure and make sense of CPU hardware counters.

Toplev next…

This is an area of ongoing effort, and I hope to write more about toplev and how I got another 10X performance improvement (fingers crossed). What prevented me from digging a bit futher is that my toplev profiles look difficult to understand.

Depending on the queue workload, the profiles change drastically from 0% Front End to 21% Front End bound workload.

On a benchmark to write 100MM with 8196 messages in each batch, it guarantees that my lz4 compression filter will be hit and therefore I'm mostly BackEnd bound.

On a benchmark to write 100MM messages with 100 messages in each batch, I'm 21% Front End bound.

Before i dug deeper into the issues, I created a CPU FlameGraph to see if maybe there was something that I missed recording traces & probes… the results were suprisingly positive:

FlameGraph of the current runtime witout --poll-mode and using the normal linux epoll & the new libaio that Avi re-wrote for the seastar framework.

When I accidentally created this graph, I realized all along that the reason for LLC-cache-misses came from the --poll-mode flag. Without it, my perf stat -F 99 -d -p <pid> looked pretty good!

perf stat after running smfb without the --poll-mode flag on

Low latency queueing might sounds like a silly goal for some, given that most BigData systems are used for offline flows. Dequeue, process, re-enqueue is the usual paradigm for most log brokers, where processing is in the 100s of milliseconds.

I do not think there exists a system today that can handle the load for the next generation of information flows in a cost efficient manner - Terabits/second. Low latency and high throughput will be a requirement to process drone logs, handle new security attacks, etc. I hope by then, either SMF or a system like it, designed to take advantage of every core, every storage device (of varying speeds) with a multitude of SLA’s can step up to the challenge.

Let me know if you found this useful on twitter @emaxerrno or on the comments.

Special thanks to my partner Sarah Rohrbach for reading earlier drafts of this post.

References

Concurrent writes to the same file with O_DIRECT

Fri, 02 Feb 2018 19:00:00 -0500

Imagine you were trying to write a very fast Write Ahead Log

…like - dunno - this: smf…

…well you pack up, use it and go home.

Unless, of course, you are still waiting for your build system to finish and then you start digging deeper.

I wrote a wal_segment which basically allows you to write to a file in append only mode. Prior to it, I couldn’t control the flush rate to the disk. That is you can only - by definition of O_DIRECT - flush page-aligned memory buffers - and the seastar::file_output_stream() does not allow you to flush unfinished pages.

You deal with that by having to zero-out the tail of the unfinished page followed by a write, and then followed by a truncate call which will set the file to the correct size - minus all the zeros you wrote. I know zeroes are not strictly necessary, but wanted to make the reads safer - more on that later.

However, you are in a world where concurrency is the norm (as well as parallelism, but let’s focus on the structure - concurrency - not the simulataneous execution - parallelism) and right before you close the file, you want to flush all the remaining pages as fast as possible.

I couldn’t find any information on the web that specifically answered the question of

Can you write to the same file handle, multiple pages at the same time (dispatched at the same time), and each write need not be sequential.

That is, on pages 1,2,3,4 - write a sequence of 4,3,2,1 (worst case scenario)?

The answer on for my SSD (INTEL SSDSC2BP48), with XFS is yes.

What follows are the tests I wrote to prove it with my favorite systems framework seastar

The strategy below is as follows:

Write 10 pages
Allow for at most 4 concurrent execution of page writes
Always skip the first page - proving the point - explicitly - though implicitly proved by concurrent execution.
Make sure that you can read the file on the file system with less afterwards

This is what it looks like when you `less` the file on your terminal at page boundaries

A follow up of this will include how to safely dispatch half written pages and how to distribute the lock/semaphore contention - Spoiler alert: jump_consistent_hash()

Comparing Lemire's fastrange.h vs Google's jump-consistent-hashing

This idea from a set of unit tests I was writing to prove that my wal_segment behaved - at a high level - like the seastar::make_file_output_stream() handle seastar provides.

The exciting news is that you should expect an update with this concurrency primitive turned on for smf in the next month or so.

Let me know if you found this useful!

Join the smf mailing list.

Appendix

This full integration test.

seastar: the future<> is here

Sat, 16 Dec 2017 19:00:00 -0500

On June 8, 2016, Avi Kivity came to NYC to present ScyllaDB. During his search for a quick open desk to do some work, I volunteered open spaces we had at Concord¹. We talked lock-free algorithms, memory reclamation techniques, threading models, Concord and distributed streaming engines, even C vs C++. Five hours later I was convinced that seastar was the best systems framework I’d ever come across.

I’ve now been using seastar for almost two years and I haven’t changed my mind.

The future<> is all about concurrency.

For the truly impatient, the future<> is here².

In 1978 news³, T. Hoare prophetically said the future was about computers getting more cores and not increasing in clock speed. In 2004 Herb Sutter coined the same trend as The Free Lunch is Over⁴. The seastar::future<> is a tool to take advantage of multi-core, multi-socket machines - a way to structure your software to grow gracefully with your hardware. There are many other tools that fit this new modality, from lock-free algorithms and to co-routines, to channels³, not to mention actor-style message passing, among many other paradigms like full-on distributed programming languages⁵ ⁶ ⁷ ⁸.

instead of driving clock speeds and straight-line instruction throughput ever higher, they are instead turning en masse to hyperthreading and multicore architectures – Herb Sutter

seastar::future<>’s are for concurrent software construction. In addition, their design makes them composable. You can take any 2 futures and chain them together via .then() operator and yield a new future⁹. Although you can combine, mix, map-reduce, filter, chain, fail, complete, generate, fulfill, sleep, expire futures, etc, they are fundamentally about program structure. Such program structure can execute in parallel, but doesn’t have to. When you have concurrent structure, parallelism is a free variable¹⁰. That is to say, you can turn up or down the number of simultaneous execution units/cores/threads without changing your program. In this paradigm, you worry about correct program structure and someone else worries about the execution.

The promise is the producer end of the channel, the future is the consumer end. A note to users. When you choose a future library you are implicitly choosing a threading model - protect your cacheline.

Seastar is an intrusive building block. Once you start composing seastar-driven asynchronous building blocks, you have to go out of your way - really - to build anything synchronous, and that’s powerful. Structurally, seastar has the same effect as actor frameworks like Akka⁵, Orleans⁶, or even languages like Pony⁷ or Erlang⁸ have. Once you have an actor, they spread virally through your system making everything an actor.

Philosophically, actor frameworks and distributed languages differ from seastar. While the former try to give the programmer higher abstractions and a runtime to hide machine details like IO or CPU scheduling, seastar takes the opposite approach. It gives you - the wise programmer - abilities to tune and control almost every part of the future<> runtime. This includes IO shares scheduling, CPU shares scheduling, in addition to batteries included approach when it comes to taking advantage of hardware for dealing with filesystems, networking, DMA, etc.

Both approaches, however, are intrinsically safe. The programmer worries about correctness and construction while the frameworks worry about efficient execution. Counter to general wisdom, it is actually faster and more scalable than the synchronous approach. While the machine does more work, it is executing your code simultaneously. This simultaneity is the key to finishing work sooner.

At its core, from the project site, seastar promises:

Shared-nothing design¹¹
High-performance networking¹²
Futures and promises¹³
Message passing¹⁴

… but it is much more, so let’s get technical and find out how Seastar executes these concurrency building blocks.

Enter Seastar… at your own risk, you might not come back

In a past life, I helped build Concord.io with facebook’s folly::futures¹⁵, and wangle¹⁶ for networking and async execution. While these libraries enabled us to deliver high performance code using similar primitives, their use of asynchronous operations is not as pervasive as that of seastar. They are libraries and not frameworks, which is the first distinction. That is, you can use the parts of the libraries that you need without needing to include or use the rest. You can tick your own clocks, your own IOEventLoops, your own CPU Scheduling, your own syscall() thread pool, etc. Seastar, on the contrary, tells you that you have to operate within their framework. It is not possible to take parts of seastar and use them on your code base without the IO subsystem or the CPU subsystem.

While this decision seems like a disadvantage, it is actually an enforcer of asynchronicity - very much like actors. It is front and center to everything you do. This is a good thing.

No locks, atomics, cache polluting primitives

Seastar takes one extreme approach to data locality. It uses almost no locks, atomics, or in any way implicit memory sharing with other cores. Your view into any application starts with a seastar::distributed<T> type. This means a copy of the T is thread local.

They of course cover all the basics for high performance applications:

Small type optimizations (although seastar::small_set<T> and seastar::small_map<K,V> are missing).
Non thread safe non-polymorphic shared pointer (local to core) via seastar::lw_shared_ptr<T>¹⁷
Non thread safe polymorphic shared pointer (local to core) via seastar::shared_ptr<T>
String with small type optimizations¹⁸ nor atomics like the libc++¹⁹
Move only bag-o’-bytes²⁰
Cicular buffers
Linux DAIO
and many many more! 🎉🎉🎉🎉

A mental model

Figure 1: Seastar Mental Model. Everything in seastar happens in a `thread_local' (per hyper-thread) with the exception of explicit cross core communication. As with all mental models, this is a simplification and omits details.

IRL Impact

I’ve been using seastar for a year and a half on a project called smf²¹ and it has been eye-opening.

➜  smf git:(master) cloc --vcs=git .                                                       
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
C++                              61            712            512           3607
C/C++ Header                     77            650            824           2761

smf is a set of libraries and utilities (like boost:: for C++ or guava for java) designed to be the building blocks of your next distributed systems.

Current benchmarks in microseconds make smf’s RPC (seastar-backed through DPDK) the lowest tail latency system I’ve tested - including gRPC, Thrift, Cap n’Proto, etc. What matters however is not that I’ve managed to build a fast RPC, but the fact that doing it with seastar was no more work than doing the same thing with facebook::folly and facebook::wangle, boost::asio, or libevent.

Figure 3: smf end-to-end latency.

In addition to the RPC, smf has its own Write Ahead Log (WAL).

It is a write ahead log modeled after an Apache Kafka-like interface or Apache Pulsar. It has topics, partitions, etc. It is designed to have a single reader/writer per topic/partition.

Current benchmarks in milliseconds ==> 41X faster than Apache Kafka

Figure 4: smf vs kafka - 3 producers.

These massive gains should be expected of many server side applications.

If you are looking for an even more batteries included system, please stop by and ask us questions on our mailing list.

Join The Mailing List

Special thanks to duarte nunes and sarah rohrbach for reading drafts of this post.

References

concord - my previous startup - http://concord.io ↩
future header file - https://github.com/scylladb/seastar/blob/master/core/future.hh ↩
csp - t hoare - http://weblab.cs.uml.edu/~bill/cs515/CSP_Hoare_78.pdf ↩ ↩²
herb sutter - free lunch is over - https://www.cs.utexas.edu/~lin/cs380p/Free_Lunch.pdf ↩
akka - actor framework for the jvm - http://akka.io/ ↩ ↩²
orleans - actor framework by msft - https://github.com/dotnet/orleans ↩ ↩²
pony - actor language - https://www.ponylang.org/ ↩ ↩²
erlang - distributed programming lang - https://www.erlang.org/ ↩ ↩²
continuations docs - https://github.com/scylladb/seastar/blob/master/doc/tutorial.md#continuations ↩
parallelism is a free variable - rob pike - https://www.youtube.com/watch?v=cN_DpYBzKso ↩
seastar shared nothing - http://www.seastar-project.org/shared-nothing/ ↩
seastar networking - http://www.seastar-project.org/networking/ ↩
seastar promises - http://www.seastar-project.org/futures-promises/ ↩
seastar message passing - http://www.seastar-project.org/message-passing/ ↩
facebook’s folly::futures - https://github.com/facebook/folly ↩
facebook wangle - https://github.com/facebook/wangle ↩
non-thread-safe shared ptr - https://github.com/scylladb/seastar/blob/master/core/shared_ptr.hh ↩
seastar::sstring - string with small type optimization - https://github.com/scylladb/seastar/blob/40a68fa50ebeeb17cd3797af7cddbbcdf07ce61a/core/sstring.hh ↩
std::basic_string - https://gcc.gnu.org/onlinedocs/gcc-6.2.0/libstdc++/api/a01076_source.html ↩
seastar::temporary_buffer - https://github.com/scylladb/seastar/blob/743723fc79d8f40a926908181026a709a8cbe719/core/temporary_buffer.hh ↩
smf - the fastest RPC in the west - https://github.com/senior7515/smf ↩

Emacs: No modeline

Sat, 16 Jan 2016 19:00:00 -0500

tl;dr: distraction-free writing

For the impatient, here is how my emacs looks now:

Having less fluff, helps me concentrate. I don’t really need scroll bars, copy and paste buttons, file open/close, etc.

Over 80% of my time I spend reading code if I’m using emacs. I’m optimizing for that use case.

My emacs opens up fully maximized and I immediately F11 because I’m not even interested in having edges. Really, what’s the point of looking at the scroll bar?

Don’t worry it is still fully functional…

Emacs! you bae

I’ve been tweaking my emacs setup for almost 7 years now. My operating systems instructor used it once and he was a hero to me - I thought, as did every other kid in the class, if I use this editor I will be as good.

During my internships people were always surprised I was using emacs. I was pretty much stuck. I Couldn’t go back to eclipse, netbeans, jetbrains, visual studio, etc. 7 years pass by and I am so very slow with the default setup. I have to go through the manual to figure out how to pipe things from a shell into my buffer.

I suppose people realize they are old(er?) when they start to reminisce about the good ‘ol days - i.e.: remember floppy disks, when you had 2 disk entries, one for your operating system and one for storage… or something like that. I realize I’m older because I have about 400 key mappings that are mode dependent. I promise I also do regular/productive work outside of tweaking my setup.

I empathize with the ‘pursuit of perfection for your editing life’. 3 years ago for example, wrote an RPC client/server between my emacs and a Go (back at rc-59 when it wasn’t cool and hip) server to get pprof data into my buffer directly. I dumped it because I decided to curl http://localhost:6060/debug/pprof/ into a buffer - yay! for (shell-command-on-region) M-|

$ cloc ~/.emacs.d/agallego/

http://cloc.sourceforge.net v 1.60  T=0.19 s (160.6 files/s, 35393.7 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
.....
Lisp                             27            351            442           1056
....
--------------------------------------------------------------------------------

However, I have about 1 thousand lines which are mostly configuration around these modes:

(defvar my-packages '(
                      thrift
                      scala-mode2      ;; scala
                      sbt-mode         ;; ensime depends on this
                      ensime           ;; for scala
                      scss-mode        ;; scss lang
                      haskell-mode     ;; haskell lang
                      protobuf-mode    ;; protobuf lang
                      simplezen        ;; zencoding for html but much simpler
                      yaml-mode        ;; .yaml files
                      js2-mode         ;; js files -- auto lints
                      yasnippet        ;; insert snippets on my code with tab
                      elisp-slime-nav  ;; elisp
                      use-package
                      cpputils-cmake      ;; cmake utils for navigating code
                      cmake-mode
                      markdown-mode+   ;; for markdown
                      multi-web-mode   ;; js, css & tml in the same buffer
                      jsx-mode
                      jsfmt
                      string-inflection   ;; snake_case -> UPPER -> cammelCase
                      zygospore           ;; revert delete other windows w/ c-x 1
                      ;; god-mode            ;; vi-like navigation
                      demangle-mode       ;; demangle c++ symbols using flint++
                      vlf       ;; view very large files
                      ws-butler ;; clean up only modified lines for whitespace
                      clean-aindent-mode ;; better auto indent
                      buffer-move      ;; swap buffer position
                      shift-text          ;; for left,right,up,down blocks shift
                      clang-format
                      google-c-style
                      company
                      goto-last-change
                      web-beautify       ;; for unminifying javascript,etc
                      helm               ;; incremental search
                      helm-ag            ;; helm for aga silver searcher
                      helm-c-yasnippet   ;;
                      helm-helm-commands ;; help with common helm commands
                      helm-ls-git      ;; inc search on git repos
                      helm-ls-hg       ;; for mercurial repos
                      helm-git-grep    ;; grep in git repos through helm -
                      helm-swoop       ;; groups similar word in buffer
                      helm-flx         ;; flx matching on helm results.
                      helm-hayoo       ;; haskell package search
                      anzu             ;; counts the  occurances for isearch
                      nyan-mode        ;; progress bar for buffer scroll
                      magit            ;; git interface for emacs
                      git-timemachine
                      git-messenger
                      mark-multiple    ;; allow for multiple marks on a buffer
                      undo-tree        ;; allow to undo changes on a file
                      elscreen         ;; allow to split screens in tabs
                      yagist           ;; post to github gists
                      expand-region    ;; avbrev on steroids
                      avy              ;; jump to anything you see with one keystroke
                      ace-window       ;; jump like ace but for windows
                      rainbow-delimiters ;; colors parens
                      browse-kill-ring ;; amazing visual kill ring search
                      exec-path-from-shell ;; fix mac not reading the shell issue
                      diminish         ;;  cleans up your powerline
                      smartparens      ;;
                      ;; themes
                      color-theme-sanityinc-solarized
                      spacemacs-theme
                      cyberpunk-theme ;; high contrast
                      gandalf-theme ;; blue'ish background - day theme
                      vimgolf ;; fun! game
                      )
  "A list of packages to ensure are installed at launch.")

It all started with an OpenVMS terminal

During one of my internships I had to ssh into these OpenVMS machines. Apart of the terrifying operating system and weird command /line /switches=1 I really enjoyed having one full terminal for my editor. You F11 your way into a pretty enjoyable and quiet programming environment. At the time we were using emacs 18 or 19 - so broken.

It took me about a month to get syntax highlighting. I remember having conversations with people that used nano (I’m almost certain. Definitely not emacs or vi) because… well… getting emacs to work in this environment was just close impossible.

Honestly, I still had doubts about using emacs then. I first started enjoying emacs is when I realized that the results of searches would appear on your buffer and you could edit them as text, it had me at find-grep-dired!

About 4 years ago, I decided to do all my work over git indexes :boom: You have no idea how productive you will be once you force your directories to have a convention. So much so that projects that don’t have git indexes, I manually add them and all the files in them so I can use git-grep and git ls-files. See my other post about notational-velocity-clone.

Give me the codez - (add-hook ‘prog-mode-hook ‘no-mini-buffer)

(add-hook 'prog-mode-hook
          (lambda ()
              (setq mode-line-format nil)
              (setq-default mode-line-format nil)
              )
          )

This is just a minor improvement on the UI. It makes a significant difference on my laptop. At work, my monitor is 4K resolution and the amount of real state that the mode-line covers is pretty insignificant.

The size of the code deployed matters

Sat, 28 Mar 2015 20:00:00 -0400

Infrastructure as code?

I had an interesting conversation with a friend of mine @dtcb about the size of the code deployed.

In all of the infrastructure efforts I’ve been part of for the last 6 years the size of the code deployed grew, sometimes to the point that the operational complexity was so high that the code had to broken apart. This is in line with the thought process of micro-services as an architectural solution to break the complexity of components from a monolithic architecture to a piece-meal composition that in turn is easier to deploy and debug.

What struck me at that time is that micro services might in fact might be too big for deployment? What if instead you could deploy classes. Literally, a simple interface that did one (or very few) things well. What if your operating system was in a way an API to deploy services (it is!) but the size of the code deployed was so small that it would in turn be hard to make mistakes. What if each of these classes had built in monitoring, tracing and debugging hooks. What if each running operator (see where I’m going?) had an ability to run multiple versions of it. What if you allowed a higher level runtime (higher than programming language runtime) worry about the performance and gluing of components.

In a way this is exactly what a CQRS/Streaming architecture affords the developer. Notice that I mentioned ‘architecture’ and not ‘samza, storm, etc’ because as of today all these services require you to deploy a fat jar,tar.gz file that contains the glue itself. It is not left up to a higher level construct to build up the ‘Computational Topology’ of your code. As a developer you specify: (in pseudo code)

// assuming an aggregation here.
//
val t = new topology
t.group_by ( some field in stream, typically user id)
 .operator( usually a commutative and associative - like Sum())
 .save( to some database )
 .emit( next tuple to chain more operators )
 .....

So where does the ‘infrastructure as code’ comes into play?

Imagine some magical command line tool that allowed you to deployed a single class/interface and run it, regardless of the programming language. What is more important is not just ‘running’ of your code but it managed the connections in between components.

What if you could write a source of data in Python and then have a heavy machine learning algorithm in Java. What if you empowered each team in an organization to deploy code and the code have enough information (via some implicit information) about the size of job and how it would in turn execute in physical space. If you achieve this goal you have in turn achieved ‘infrastructure as code’. You would have the ability to modify your running infrastructure by running more code.

For example, supposed you started code as follows:

  new topology().addStream( hdfs://file ).map( x: String => println (x) )

and then you added a simple filter:

new topology()
    .addStream( hdfs://file )
    .filter( x: String => x == "blue you know you're my boy!" )
    .map( x: String => println (x) )

What if instead of just tearing everything down there was a system that allowed you to modify the runtime of these application to run the filter function either in the same process or in the same machine or in a series of machines without have to take down your running code.

I do know that the erlang vm allows you to do byte code deployment on a running vm. And I know that spark gives you this native (scala) interface for dealing with streams in multiple machines.

I want both - really, I want the ability to redeploy the map function at runtime, pausing all its upstream operators, deploy a new one and keep producing. I also want the ability to add some arbitrary computation graph above the mapping function without affecting anything else. In fact I don’t even want to write them in the same file. I just want to refer to these functions by name.

I suppose I’m trying to solve some of these challenges. I think that as of today we don’t have the tools (at least i don’t know of them) that would allow us to have this simple programming paradigm, but we have some of the tools.

Obviously not complete, here is a sample of the system I’m working on:

class MockTweetComputation:
    def initialize(self, ctx):
        ctx.set_timer(milliseconds_now())

    def process_timer(self, ctx, timer):
        for tweet in gen_tweets():
            ctx.produce_record('tweets', '_', thrift_to_bytes(tweet))
        ctx.set_timer(milliseconds_now() + 1000)

    def metadata(self):
        return rbonut.Metadata(
                name='mock-tweets-generator',
                istreams=[],
                ostreams=['tweets']
                )

You’ve figured it out!

Essentially a globally unique pub-sub system gives you the flexibility I’m talking about here. Essentially a pub sub system would allow you to refer to functions by name (same as a program) and your arguments to your function would be explicity delcared as would your output of your program such that some scheduler/oracle could co-locate/inline/scale work for you.

In brief, the above operator is something like this in C++:

struct Tweet {
  char[140] data;
};

std::list<Tweet> mockTweetGenerator() {
  std::list<Tweet> retval;
  // ... do lots of work
  return retval;
}

Notice that the istreams of mock-tweets-generator is an empty list. However, the ostreams (return arguments) is a list of one ‘type’.

Ok… so what’s that about code size?

Well.. the argument is that by making the code small, you’ll make less mistakes. You’ll be able to iterate faster on your problems. Same deal that people have been arguing for for decades.

What is exciting for me, is that we - at work - built a system that actually empowers you to do just that. For all programming languages and through a simple pub-sub model you can effectively change how you chain complex computational models - with some small added overhead.

We built the glue, and hopefully our application developers can just build the application and deploy code without fear. Deploy code fast.

Note: It is clear to me that this paradigm doesn’t work for all the problems in the world. You would need a unified runtime and a unified programming which is almost what erlang gives you. However, this would defeat one of our goals which is to allow anyone with any programming language to take advantage of this model.

Soon, I’ll talk about our isolation, message guarantees, etc.

Thoughts? - please let me know!

Productionizing storm is difficult

Sun, 22 Feb 2015 18:10:48 -0500

Today I got a call from a friend using storm in production:

‘Yo I restarted the supervisor process, can’t get the topology to pick up the new Kafka node while reading it. The producers are working fine, but there are no consuming nodes yet.’

First, there is a huge mismatch of expectations there. My friend didn’t know that you never really need to restart the supervisor nodes (most of the time… ) Second, when you deploy a storm topology nimbus creates a .ser file which contains all of the environment (.property files for example, log4j.XML etc). Then this file gets deployed to all supervising nodes that are going to execute your code.

The issue with his intuition of just killing the process consuming the properties file and reading from Kafka matches that of local development. It’s easy to simply kill -9 anything on your computer and… bam! restart it… (minus the implication of possibly data loss – read_from_start=true was given)

Obviously running a distributed computation is not trivial.

Here are a few reasons:

Restarting a process that interacted with external state (very typical) may not produce the same results when you need to replay records.

In fact, you can’t rely on the results being the same given the presence of arbitrary external state. period.

There is a huge value in the speedy processing of records and them being processed correctly with enough information for debugging. That’s the main reason for using a stream processor. Timely and correct processing.

Interacting with external state is hard. supa’ hard.
Usually all stream processors provide very different tools for reasoning about distributing work and processing work locally:
- Exceptions are hard to reason about in a distributed system: Do you restart the entire topology, do you restart the node, do you just crash and let the containerizer clean up your resources and reschedule your worker somewhere else in the cluster loosing data locality ? , etc. List is long
- Restarting remote processes have very different semantics than local procs. If the system has no way of detecting failure and replaying records then you just lost data. If the error propagates upstream and the source of data cannot replay data, you are f**ed. You just lost data which usually translates to real money.
- Deploying code in common streaming systems is really awkward. First you have to plan for downtime. The reason is that you usually can’t deploy code piecemeal. You need to deploy code in big chunks (DAGs) which contain the whole computational topology.
  
  The issue with this monolithic deployment is fairly obvious. Topologies are mostly long lived things, their uptime is months on a fast moving startup. Literally half year on bigger companies. This means that deploying a single component might introduce a bug in other components so deployment is a nightmare.
  
  Also, think about this bastard proposition. First you have to stop making money - pause your topology, leave those cpu’s idle. Then you upload new code and cross your fingers another team didn’t introduce a bug and hope that your unit,regression,integration tests got all the bugs. You then wait for about 5 - 10 minutes for the topology to be settled (i.e.: caches are hot, processing is humming along, no stack traces are registered, nagios hasn’t called you, your dashboard numbers are updating, etc..). If this process failed you have to roll back and cross your fingers again. Not to mention that your team must keep all deployed binaries properly versioned and in a repository that is easy to get to - not the case for most startups.
- I’ll stop at this note, but say something went wrong, where do you start? Do you go onto zookeeper and make sure that your topics are being updated? (in the case of the kafka driver for storm). Do you restart the process and increase the log level (assuming no jmx hooks). On a real note, debugging distributed systems is usually all about printf and tcpdump. Attaching a debugger to a running process is scary. Press the up arrow one more time than you intended and you just brought down part of the cluster accidentally. Don’t attach a debugger and you have almost no visibility into the process. jmx metrics help, the linux /proc helps, sending signals to processes help but some things are just logic errors on very specific code paths and this is impossible to debug without the old printf, pen & paper.

Luckily the solution was trivial. Restart the topology from nimbus (storm master node) and et voila’. New configs serialized and deployed.

Stay tunned for updates as we make progress on our attempt at solving some issues mentioned here.

Emacs - Helm + Git Grep == Notational Velocity Clone

Mon, 02 Feb 2015 18:10:48 -0500

If you haven’t used notational velocity on a Mac OS computer you are missing out :)

It is an incredibly powerful tool.

I wanted the same freedom and flexibility while taking notes in meetings but I obviously wanted to use emacs to do it.

emacs deft

First emacs deft is a great plugin, did everything I wanted. Filter, search, edit, and create buffers with ease.

The more I used, it the more I hacked it to do what I wanted. One day I realized that I was already using a phenomenal incremental search and narrowing framework - emacs helm!!

I’ve been using helm-git-grep and helm-ls-git for a while. Put those 2 functions together and you have exactly what I wanted. A git backed, incremental search and narrowing framework that searches both content and filenames. This is key! - In fact this was so useful that I started using it for all my git based repos.

This is my default way of navifgating all repositories. I just think of ideas, don’t really care wether it is the name of a file or a comment in a file. I want it all, in an easy to use interface.

Alexander Gallego

The Effects of CPU Turbo: 768X stddev

BLUF - Bottom Line Up Front.

The beginning of yak-shaving

Setup

Fixing the environment

Settings

Next - Compiler Explorer - my new fav tool

With these tools we can now benchmark & optimize our code

The results

Our benched code

Lessons Learned: Benchmarking with turbo is dishonest.

That’s a stddev difference of up to 768X !!!

SMF

A tale of performance debugging: from 1.3X slower to 48X faster than Apache Kafka

Max latency budget: Kafka on NVMe + XFS + kernel 4.14.16-300.fc27.x86_64

perf {stat,record,list} & toplev (PMU_tools)

1st non-obvious performance bottleneck std::sort

Finding the culprit!

2nd improvement: poor data locality - code slaying

3rd improvement: poor data locality - non intuitive fix 2

Results

Performance, riddle me this!

Toplev next…

References

Concurrent writes to the same file with O_DIRECT

Appendix

seastar: the future<> is here

The future<> is all about concurrency.

Enter Seastar… at your own risk, you might not come back

No locks, atomics, cache polluting primitives

A mental model

IRL Impact

Join The Mailing List

References

Emacs: No modeline

tl;dr: distraction-free writing

Don’t worry it is still fully functional…

Emacs! you bae

It all started with an OpenVMS terminal

Give me the codez - (add-hook ‘prog-mode-hook ‘no-mini-buffer)

The size of the code deployed matters

Infrastructure as code?

So where does the ‘infrastructure as code’ comes into play?

You’ve figured it out!

Ok… so what’s that about code size?

Thoughts? - please let me know!

Productionizing storm is difficult

Obviously running a distributed computation is not trivial.

Emacs - Helm + Git Grep == Notational Velocity Clone

emacs deft

This is the result: