twitter / fatcache Goto Github PK

like the following snippet:
n = pwrite(fd, slab, size, off);
if (n < size) {
log_error("pwrite fd %d %zu bytes at offset %"PRId64" failed: %s",
fd, size, off, strerror(errno));
return FC_ERROR;
}

when n<size ,pread or pwrite may interrupt by signal ,errno is equal to EINTR,
it should process on writing,not return an error?

Now we have a question not writting data on a ssd for the fatcache.

Steps are as follows：

1. wget -c http://fatcache.googlecode.com/files/fatcache-0.1.0.tar.gz (Missing fc_client.h and fc_server.h files.)
2. wget -c https://github.com/twitter/fatcache/archive/master.zip
3. installing:

tar zxf fatcache-0.1.0.tar.gz

unzip master.zip

cp fatcache-master/src/fc_server.* fatcache-0.1.0/src/

cp fatcache-master/src/fc_client.* fatcache-0.1.0/src/

cd fatcache-0.1.0

./configure --prefix=/usr/local/fatcache

make && make install

Machine environment：

Installed two SSD.
The first, installed centos 6.4 64bit
The second, data storage.

a)fdisk or not to fdisk the ssd, we have tried, both can not write on the ssd.

start:
/usr/local/fatcache/bin/fatcache -d -o /usr/local/fatcache/log/fatcache.log -D /dev/sdb

fatcache log:
Mon Aug 12 12:39:47 2013] fc.c:683 fatcache-0.1.0 started on pid 9446
[Mon Aug 12 12:39:47 2013] fc.c:688 configured with debug logs disabled, asserts disabled, panic disabled
[Mon Aug 12 12:39:47 2013] fc_slab.c:85 slab size 1048576, slab hdr size 12, item hdr size 52, item chunk size 88
[Mon Aug 12 12:39:47 2013] fc_slab.c:88 index memory 0, slab memory 67108864, disk space 240056795136
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 0: items 11915 size 88 data 36 slack 44
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 1: items 9362 size 112 data 60 slack 20
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 2: items 7281 size 144 data 92 slack 100
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 3: items 5698 size 184 data 132 slack 132
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 4: items 4519 size 232 data 180 slack 156
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 5: items 3542 size 296 data 244 slack 132
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 6: items 2788 size 376 data 324 slack 276
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 7: items 2221 size 472 data 420 slack 252
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 8: items 1771 size 592 data 540 slack 132
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 9: items 1409 size 744 data 692 slack 268
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 10: items 1120 size 936 data 884 slack 244
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 11: items 891 size 1176 data 1124 slack 748
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 12: items 712 size 1472 data 1420 slack 500
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 13: items 569 size 1840 data 1788 slack 1604
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 14: items 455 size 2304 data 2252 slack 244
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 15: items 364 size 2880 data 2828 slack 244
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 16: items 291 size 3600 data 3548 slack 964
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 17: items 232 size 4504 data 4452 slack 3636
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 18: items 186 size 5632 data 5580 slack 1012
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 19: items 148 size 7040 data 6988 slack 6644
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 20: items 119 size 8800 data 8748 slack 1364
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 21: items 95 size 11000 data 10948 slack 3564
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 22: items 76 size 13752 data 13700 slack 3412
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 23: items 60 size 17192 data 17140 slack 17044
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 24: items 48 size 21496 data 21444 slack 16756
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 25: items 39 size 26872 data 26820 slack 556
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 26: items 31 size 33592 data 33540 slack 7212
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 27: items 24 size 41992 data 41940 slack 40756
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 28: items 19 size 52496 data 52444 slack 51140
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 29: items 15 size 65624 data 65572 slack 64204
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 30: items 12 size 82032 data 81980 slack 64180
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 31: items 10 size 102544 data 102492 slack 23124
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 32: items 8 size 128184 data 128132 slack 23092
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 33: items 6 size 160232 data 160180 slack 87172
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 34: items 5 size 200296 data 200244 slack 47084
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 35: items 4 size 250376 data 250324 slack 47060
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 36: items 3 size 312976 data 312924 slack 109636
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 37: items 2 size 391224 data 391172 slack 266116
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 38: items 2 size 489032 data 488980 slack 70500
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 39: items 1 size 611296 data 611244 slack 437268
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 40: items 1 size 764120 data 764068 slack 284444
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 41: items 1 size 955152 data 955100 slack 93412
[Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 42: items 1 size 1048564 data 1048512 slack 0

Hi,

I'm looking for a distributed hierarchical caching solution (two levels: mem, disk). Hence I'm wondering whether you have plans to implement some sort of "first level" in-memory cache for FC? Or am I missing out on the pointlessness of this approach (saving a round trip in case of a first level miss seems worthwhile to me)?

Also, do you have any remarks on using FC with ordinary HDDs?

Cheers,

Björn

hi @manjuraj

The stats command wasn't implemented in fatcache, and i'm going to do that.

the stats command would be similar to memcached:

stats\r\n
stats slabs\r\n
stats settings\r\n

do you have any suggestions?

It would be great if this could work with Rails and I think for that to happen this would have to work with the Dalli gem. Currently when trying to perform any memcached operations an exception occurs:

require 'dalli'
dc = Dalli::Client.new('localhost:11211')
dc.set('abc', 123)

method server_for_key in ring.rb at line 45
method perform in client.rb at line 347
method set in client.rb at line 199

log from fatcache:

[Fri Sep  6 05:56:01 2013] fc_server.c:87 accepted c 7 on s 6
[Fri Sep  6 05:56:01 2013] fc_memcache.c:785 parsed bad req 12 res 1 type 0 state 0
00000000  80 0b 00 00 00 00 00 00  00 00 00 00 00 00 00 00   |................|
00000010  00 00 00 00 00 00 00 00                            |........|
[Fri Sep  6 05:56:01 2013] fc_core.c:81 recv on c 7 failed: Invalid argument
[Fri Sep  6 05:56:01 2013] fc_core.c:113 close c 7 on event 0005 eof 0 done 0 rb 24 sb 0: Invalid argument
[Fri Sep  6 05:56:01 2013] fc_client.c:64 close c 7 discarding pending req 12 len 24 type 0

Hi, is fatcache currently in production use? I'm interested in contributing to optimizing memcache for SSDs, and would like to start with performance analysis to expose optimization possibilities. Should I start with fatcache or is there another deployment you would recommend? Thanks.

We're evaluating the potential benefits of deploying Fatcache and I've run into a problem I haven't been able to resolve. The code compiles cleanly on the platform we're using, Ubuntu 16/Trusty, and the server process start just fine. But if I run performance tests with "memtier_benchmark" the server process crashes with a segmentation fault for any non-trivial test configuration.

I've tried running minimal configurations that use all defaults, I've compiled the v0.1.1 tag and the lastest code from the GitHub repo. And I've compiled with full debugging enabled and optimization (O0 flag) disabled. The process crashes each time no matter which configuration I use.

I'm hoping it's a known issue or something dumb I'm doing to setup/configure Fatcache. But since I haven't found anything by searching public documents related to Fatcache I wanted to post here to look for clues.

Here's the stack trace from "gdb" in case it helps.

Program received signal SIGSEGV, Segmentation fault.
__memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:36
36 ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S: No such file or directory.
(gdb) where
#0 __memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:36
#1 0x000000000040ccfc in mbuf_copy (mbuf=0x5889f60, pos=0x7ffff6ff0cf0 "", size=8144) at fc_mbuf.c:233
#2 0x000000000040ce1c in mbuf_copy_from (mhdr=0x737b68, pos=0x7ffff6ff0cf0 "", size=2347828697) at fc_mbuf.c:267
#3 0x000000000040c46f in rsp_send_value (ctx=0x7fffffffea00, conn=0x634ab0, msg=0x735a30, it=0x7ffff1ef104c, cas=423) at fc_response.c:338
#4 0x000000000040aaae in req_process_get (ctx=0x7fffffffea00, conn=0x634ab0, msg=0x735a30) at fc_request.c:212
#5 0x000000000040b8ae in req_process (ctx=0x7fffffffea00, conn=0x634ab0, msg=0x735a30) at fc_request.c:548
#6 0x000000000040b9f6 in req_recv_done (ctx=0x7fffffffea00, conn=0x634ab0, msg=0x735a30, nmsg=0x0) at fc_request.c:610
#7 0x0000000000409898 in msg_parsed (ctx=0x7fffffffea00, conn=0x634ab0, msg=0x735a30) at fc_message.c:168
#8 0x0000000000409c99 in msg_parse (ctx=0x7fffffffea00, conn=0x634ab0, msg=0x735a30) at fc_message.c:317
#9 0x0000000000409e85 in msg_recv_chain (ctx=0x7fffffffea00, conn=0x634ab0, msg=0x735a30) at fc_message.c:383
#10 0x0000000000409f30 in msg_recv (ctx=0x7fffffffea00, conn=0x634ab0) at fc_message.c:417
#11 0x000000000040415f in core_recv (ctx=0x7fffffffea00, conn=0x634ab0) at fc_core.c:77
#12 0x000000000040454f in core_core (ctx=0x7fffffffea00, conn=0x634ab0, events=1) at fc_core.c:157
#13 0x00000000004046f2 in core_loop (ctx=0x7fffffffea00) at fc_core.c:215
#14 0x000000000041396d in main (argc=13, argv=0x7fffffffeb08) at fc.c:746

hi @manju @thinkingfish

I have some questions:

1. why not put expire time in itemx, as we have to use it frequently, and it may lead to read disk slab.
2. why not add stats command for fatcache, as it may be useful.

Best Regards~

Some minor but nice to have changes have been made since last release.
How about releasing them to the public for testing?

Hi,

I received the following kernel error while running the fatcache:

 kernel: Out of socket memory

Setup:

• 96 GB RAM
• 1 TB SSD
• 8 fatcache instances each with ~21GB of memory each -m 21760 -i 64 (12 core box)
• Running a "memslap" client creating thousands of connections (~36 K connections) to fatcache
• When the free memory left is around 5 GB, I see the above error and memslap stops
• 2.6.32 kernel

Let me know if you need any other information

You say your in-memory objects look like this:

struct itemx {
  STAILQ_ENTRY(itemx) tqe;    /* link in index / free q */
  uint8_t             md[20]; /* sha1 message digest */
  uint32_t            sid;    /* owner slab id */
  uint32_t            offset; /* item offset from owner slab base */
  uint64_t            cas;    /* cas */
} __attribute__ ((__packed__));

I suggest you do not pack these, and that you put cas first. I have seen very bad performance doing atomic ops on fields that cross a cache line boundary, and here you presumably do atomic ops on cas, which is only 4-byte aligned. Depending where itemx is used (and particularly if you ever allocate an array of itemx), you may see strange behavior. See http://www.tokutek.com/2012/12/packing-for-the-holidays/ for some benchmarks.

Unless, of course, you deal with this elsewhere.

Hi,

I am testing fatcache with libmemcached. I have write a simple program, and I found that during the working process, the program consumes a lot of my memory. My computer has 64G memory, and the program can take all these memories.

The setup is like this:
OS: ubuntu 14.04
database: mysql 5.6
C++ library: libmemcached
Memory: 64GB
SSD: Use a usb driver flash instead

By debugging the program, I found that the set operation needs a lot of memory. So, do you have any idea about this problem? Is there any problem with libmemcached? Looking forward for your replay. Thank you very much!

Best!

when I use autoconf to generate config file,there is an error happend. And I don't know how to fix it.
the error is:

configure.ac:17: error: possibly undefined macro: AM_INIT_AUTOMAKE
If this token and others are legitimate, please use m4_pattern_allow.
See the Autoconf documentation.
configure.ac:32: error: possibly undefined macro: AM_PROG_CC_C_O

autoconf version is 2.69
Red Hat Enterprise Linux Server release 6.3

In your summary, you write that you have code in place to check for SHA-1 collisions. I suggest getting rid of that code and its performance overhead.

There are simply no collisions for reasonably short keys. The chance of accidentally producing a hash collision for SHA-1 is astronomically small. Even purposely finding SHA-1 collisions with brute force takes ca. 2^80 instructions. Just do the math on how long your system would need to be running for that to happen.

The only reason for doing the check would be to protect against more elaborate attacks which a user might be able to inject. It is unlikely that SHA-1 will be susceptible to preimage attacks (which would be necessary to, for example, retrieve other chosen data from the cache) any time soon (not even MD5 is). More likely but still a far fetch would be a collision attack on SHA-1 that would allow a complexity attack on your system. If that ever becomes a potential threat, you should migrate to a better hash as checking won't help you.

The release link in the README points to the Google code archive, and there hasn't been any tarball release since 2013!

Maybe you should make a new one, or simply get rid of release tarballs if this does not make sense.

Hi, I noticed that GETs do not result in writes to the disk. This makes sense for the common case.

I'm wondering if there are cases where fatcache would be required to write to disk in the future? e.g. support CAS, maintain state of cache across power cycle.

BTW, Is there a plan to support "persistent" fatcache that keeps state across power cycle? Is a persistent memcache useful in real deployments?

Thanks.