Issue
According to https://docs.confluent.io/platform/current/streams/developer-guide/memory-mgmt.html it is sufficient to do the following to use a non-standard ie. more efficient memory allocator on Linux, in this case primarily for RocksDB:
# example: install jemalloc (on Debian)
apt install -y libjemalloc-dev
# set LD_PRELOAD before you start your Kafka Streams application
export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so"
Since with my workload I measured the same 4.5GiB memory consumption as without jemalloc I tried a run with tcmalloc too, which also showed the same memory usage.
So I tried to analyse the usage of the dynamically linked libraries as follow.
The output of ldd /tmp/librocksdbjni16525886107314830425.so is:
linux-vdso.so.1 (0x00007fffc3ba6000)
/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 (0x00007f5e50dff000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5e51d94000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5e51d8f000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5e51d8a000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5e50bd3000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5e51ca1000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5e51c81000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5e509aa000)
/lib64/ld-linux-x86-64.so.2 (0x00007f5e51d9f000)
libunwind.so.8 => /lib/x86_64-linux-gnu/libunwind.so.8 (0x00007f5e51c66000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f5e51c3b000)
The output of objdump -T /tmp/librocksdbjni16525886107314830425.so | grep malloc is:
0000000000000000 DF *UND* 0000000000000000 (GLIBC_2.2.5) malloc_usable_size
0000000000000000 DF *UND* 0000000000000000 (GLIBC_2.2.5) malloc
0000000000000000 w D *UND* 0000000000000000 mallocx
0000000000000000 w D *UND* 0000000000000000 malloc_stats_print
0000000000558360 w DF .text 00000000000000e2 Base _ZN7rocksdb23JemallocNodumpAllocatorD1Ev
00000000002fedd0 g DF .text 0000000000000093 Base rocksdb_jemalloc_nodump_allocator_create
0000000000558360 w DF .text 00000000000000e2 Base _ZN7rocksdb23JemallocNodumpAllocatorD2Ev
0000000000bf85e0 w DO .data.rel.ro 00000000000000c0 Base _ZTVN7rocksdb23JemallocNodumpAllocatorE
0000000000557fa0 g DF .text 00000000000002f7 Base _ZN7rocksdb26NewJemallocNodumpAllocatorERNS_24JemallocAllocatorOptionsEPSt10shared_ptrINS_15MemoryAllocatorEE
0000000000557bd0 g DF .text 000000000000015b Base _ZN7rocksdb23JemallocNodumpAllocatorC1ERNS_24JemallocAllocatorOptionsE
0000000000557d30 g DF .text 0000000000000096 Base _ZN7rocksdb23JemallocNodumpAllocator11IsSupportedEPSs
00000000005582d0 w DF .text 0000000000000008 Base _ZNK7rocksdb23JemallocNodumpAllocator4NameEv
0000000000557bd0 g DF .text 000000000000015b Base _ZN7rocksdb23JemallocNodumpAllocatorC2ERNS_24JemallocAllocatorOptionsE
00000000007eb510 g DF .text 0000000000000005 Base _Z12toku_xmallocm
0000000000301fb0 g DF .text 000000000000000b Base rocksdb_options_set_dump_malloc_stats
0000000000557dd0 g DF .text 00000000000001c3 Base _ZN7rocksdb23JemallocNodumpAllocator14PrepareOptionsERKNS_13ConfigOptionsE
00000000007ebe30 g DF .text 00000000000000e5 Base _ZN8memarena17malloc_from_arenaEm
0000000000558450 w DF .text 00000000000000bb Base _ZN7rocksdb23JemallocNodumpAllocatorD0Ev
Despite/due to this output I'm still unsure whether RocksDB uses the desired memory allocator.
Solution
According to https://github.com/jemalloc/jemalloc/issues/388 the following two commands do it:
# negative test
MALLOC_CONF=abort_conf:true,invalid_option:foo /layers/paketo-buildpacks_bellsoft-liberica/jre/bin/java -version
MALLOC_CONF=stats_print:true /layers/paketo-buildpacks_bellsoft-liberica/jre/bin/java -version
Additionally with the following command can be verified whether the dynamic link has happened:
pmap 1 | grep malloc
Answered By - Andras Hatvani Answer Checked By - Candace Johnson (WPSolving Volunteer)