Not sure blockCacheMaxDirty is working about s3backer HOT 9 CLOSED

I think maybe the kernel is just flushing its own buffers. Kernel buffering is 
complicated... e.g. there is a buffer in the kernel for the "file" that appears 
in the s3backer filesystem. It's not clear that sync(8) will wait for dirty 
pages of this file to be written back.

However, there very well could be a bug. If so, you should see it in the block 
cache dirty ratio in the stats file. Also you can apply the attached patch, run 
s3backer in the foreground (-f --debug) and watch the actual dirty block count 
change.

Let me know what you find.

block_cache_current_size     103962 blocks
block_cache_initial_size     1792 blocks
block_cache_dirty_ratio      0.2071

Sure looks like a bug to me ;-)

OK, so I installed your patch, created a new s3backer bucket, mounted it with 
max dirty set to 10 blocks, and then wrote 10 blocks of data into it with dd. 
Here's what I saw:

#dirties=1
#dirties=2
#dirties=3
#dirties=4
2010-10-21 07:49:52 DEBUG: PUT http://s3.amazonaws.com/jik2-backup-dev2/00000000
#dirties=5
2010-10-21 07:49:52 DEBUG: PUT http://s3.amazonaws.com/jik2-backup-dev2/00000001
#dirties=6
2010-10-21 07:49:52 DEBUG: PUT http://s3.amazonaws.com/jik2-backup-dev2/00000002
#dirties=7
2010-10-21 07:49:52 DEBUG: PUT http://s3.amazonaws.com/jik2-backup-dev2/00000003
#dirties=8
2010-10-21 07:49:52 DEBUG: PUT http://s3.amazonaws.com/jik2-backup-dev2/00000004
#dirties=9
2010-10-21 07:49:52 DEBUG: PUT http://s3.amazonaws.com/jik2-backup-dev2/00000005
#dirties=10
#dirties=11
#dirties=12
#dirties=13
#dirties=14
#dirties=15
#dirties=16
#dirties=17
#dirties=18
#dirties=19
#dirties=20
2010-10-21 07:49:53 DEBUG: success: PUT 
http://s3.amazonaws.com/jik2-backup-dev2/00000001
2010-10-21 07:49:53 DEBUG: success: PUT 
http://s3.amazonaws.com/jik2-backup-dev2/00000005
#dirties=19
2010-10-21 07:49:53 DEBUG: PUT http://s3.amazonaws.com/jik2-backup-dev2/00000006
#dirties=18
2010-10-21 07:49:53 DEBUG: PUT http://s3.amazonaws.com/jik2-backup-dev2/00000007
2010-10-21 07:49:53 DEBUG: success: PUT 
http://s3.amazonaws.com/jik2-backup-dev2/00000003
#dirties=17
2010-10-21 07:49:53 DEBUG: PUT http://s3.amazonaws.com/jik2-backup-dev2/00000008
2010-10-21 07:49:53 DEBUG: success: PUT 
http://s3.amazonaws.com/jik2-backup-dev2/00000000
#dirties=16
2010-10-21 07:49:53 DEBUG: PUT http://s3.amazonaws.com/jik2-backup-dev2/00000009
2010-10-21 07:49:53 DEBUG: success: PUT 
http://s3.amazonaws.com/jik2-backup-dev2/00000004
#dirties=15
2010-10-21 07:49:53 DEBUG: PUT http://s3.amazonaws.com/jik2-backup-dev2/0000000a
2010-10-21 07:49:53 DEBUG: success: PUT 
http://s3.amazonaws.com/jik2-backup-dev2/00000002
#dirties=14
2010-10-21 07:49:53 DEBUG: PUT http://s3.amazonaws.com/jik2-backup-dev2/0000000b
2010-10-21 07:49:54 DEBUG: success: PUT 
http://s3.amazonaws.com/jik2-backup-dev2/00000006
2010-10-21 07:49:54 DEBUG: success: PUT 
http://s3.amazonaws.com/jik2-backup-dev2/00000007
#dirties=13
2010-10-21 07:49:54 DEBUG: PUT http://s3.amazonaws.com/jik2-backup-dev2/0000000c
#dirties=12
2010-10-21 07:49:54 DEBUG: PUT http://s3.amazonaws.com/jik2-backup-dev2/0000000d
2010-10-21 07:49:54 DEBUG: success: PUT 
http://s3.amazonaws.com/jik2-backup-dev2/00000008
#dirties=11
2010-10-21 07:49:54 DEBUG: PUT http://s3.amazonaws.com/jik2-backup-dev2/0000000e
2010-10-21 07:49:54 DEBUG: success: PUT 
http://s3.amazonaws.com/jik2-backup-dev2/00000009
#dirties=10
2010-10-21 07:49:54 DEBUG: PUT http://s3.amazonaws.com/jik2-backup-dev2/0000000f
2010-10-21 07:49:54 DEBUG: success: PUT 
http://s3.amazonaws.com/jik2-backup-dev2/0000000a
#dirties=9
2010-10-21 07:49:54 DEBUG: PUT http://s3.amazonaws.com/jik2-backup-dev2/00000010
2010-10-21 07:49:54 DEBUG: success: PUT 
http://s3.amazonaws.com/jik2-backup-dev2/0000000b
#dirties=8
2010-10-21 07:49:54 DEBUG: PUT http://s3.amazonaws.com/jik2-backup-dev2/00000011
2010-10-21 07:49:54 DEBUG: success: PUT 
http://s3.amazonaws.com/jik2-backup-dev2/0000000c
#dirties=7
2010-10-21 07:49:54 DEBUG: PUT http://s3.amazonaws.com/jik2-backup-dev2/00000012
2010-10-21 07:49:54 DEBUG: success: PUT 
http://s3.amazonaws.com/jik2-backup-dev2/0000000d
#dirties=6
2010-10-21 07:49:54 DEBUG: PUT http://s3.amazonaws.com/jik2-backup-dev2/00000013
2010-10-21 07:49:54 DEBUG: success: PUT 
http://s3.amazonaws.com/jik2-backup-dev2/0000000e
#dirties=5
2010-10-21 07:49:54 DEBUG: success: PUT 
http://s3.amazonaws.com/jik2-backup-dev2/0000000f
#dirties=4
2010-10-21 07:49:54 DEBUG: success: PUT 
http://s3.amazonaws.com/jik2-backup-dev2/00000010
#dirties=3
2010-10-21 07:49:55 DEBUG: success: PUT 
http://s3.amazonaws.com/jik2-backup-dev2/00000011
#dirties=2
2010-10-21 07:49:55 DEBUG: success: PUT 
http://s3.amazonaws.com/jik2-backup-dev2/00000013
#dirties=1
2010-10-21 07:49:55 DEBUG: success: PUT 
http://s3.amazonaws.com/jik2-backup-dev2/00000012
#dirties=0

Now, if I understand the desired behavior correctly, what was supposed to 
happen is once #dirties got to 10, s3backer should have blocked until a dirty 
was successfully written, and the count never should have gotten above 10. But 
that doesn't appear to be what happened.

The problem is that max_dirty is enforced when a block is added to the cache 
but not when an existing cache block is dirtied.

I believe the attached patch fixes the issue.

Thanks! That was indeed the problem. Please verify the attached patch fixes 
this problem for you.

Seems to work fine, although I like my patch better ;-)

Thanks, committed in r441.