[RESEARCH] Dokany vs WinFsp about cppcryptfs HOT 20 CLOSED

I've created a dokany_2.0 branch of cppcryptfs . It seems to work, but I wouldn't use it for production yet.

I have not implemented asynchronous i/o yet.

However, I don't think asynch i/o would help with speeding up dealing with lots of small files when the storage is on a local SSD.

from cppcryptfs.

I've looked at WinFSP. In my tests it's at least 2x faster than Dokany.

I might port cppcryptfs to use WinFSP in the future, but so far I haven't had any problems with Dokany other than the slower performance.

from cppcryptfs.

Thanks for the heads up. I'll test Dokany 2.

If it looks like it's going to be a lot of work to move to Dokany 2, and if Dokany 2 is still significantly slower than winfsp, then that will be a good motivation to switch to winfsp.

cppcryptfs does not use a buffer pool, but I think using a buffer pool would be a bigger boost in go than in c++. A buffer pool would be easy to implement, though. I will add it to my list.

But I think the streaming read and write performance is quite good already. It's when cppcryptfs is dealing with lots of small files that it is slow. Maybe Dokany 2 or winfsp will solve that.

from cppcryptfs.

@bailey27 as discussed before, if you do decide to go the winfsp route you can count on my support.

BTW, while I am not familiar with the buffer pool implementation in gocryptfs, I would expect that a file system that uses the NT cache manager would far outperform anything that can be done in user mode.

from cppcryptfs.

@Liryna , I think the big change in Dokany 2.0 was to make it asynchronous, right?

I did some testing with Dokany 2.0 beta (mirror.exe) yesterday, and it looks to me that it isn't any faster than 1.5 when dealing with lots of small files.

I think the reason why Dokany is slow when dealing with lots of small files is the latency. I think making it asynchronous wouldn't help with that.

I also tested WinFSP (passthrough-64) again.

It looks to me that WinFSP is about 1.5 to 2x faster than Dokany when dealing with lots of small files.

But WinFSP (passthrough-64) is 3 to 4 times slower than native NTFS in my tests.

So I think if I changed cppcryptfs to use WinFSP, it will be faster, but it will still not be fast.

I think gocryptfs doesn't have these problems because it runs on Linux and uses Fuse which is supported by the Linux kernel. In other words, I think gocryptfs is fast because your are actually supposed to be able to have filesystems in user space on Linux, but on Windows you aren't.

@billziss-gh, I realize that passthrough-64 is not the reference fileystem implementation for WinFSP (memfs is). But I think passthrough-64 would be my starting point if I were to change to WinFSP because I am not really implementing a whole filesystem. I'm basically doing what passthrough or Dokany mirror does, but with encryption/decryption.

I'm going to add a i/o buffer pool to cppcryptfs. But I am sure it won't help with the lots of small files speed issue. It may help speeding up streaming reads and writes a little bit, but those are already fast.

from cppcryptfs.

But WinFSP (passthrou#64) is 3 to 4 times slower than native NTFS in my tests.

So I think if I changed cppcryptfs to use WinFSP, it will be faster, but it will still not be fast.

NTFS is hard to beat! After all both WinFsp passthrough and Dokany mirror use NTFS to satisfy the I/O.

But NTFS is fast because it uses the "cache manager" extensively. Effectively when an I/O comes into the file system it becomes a simple "memcpy" from the OS's cache buffers into the application buffers. [In memory mapped cases even that "memcpy" is eliminated as the user mode application accesses the cache mapping directly!]

I think gocryptfs doesn't have these problems because it runs on Linux and uses Fuse which is supported by the Linux kernel. In other words, I think gocryptfs is fast because your are actually supposed to be able to have filesystems in user space on Linux, but on Windows you aren't.

I have not tested gocryptfs, so I am not familiar with its performance. I do have my own FUSE (low-level) file systems and I have not found FUSE to be as performant as you are describing it. I often end up implementing my own user mode buffer cache under FUSE, which seems to improve matters. Some times I even implement a dirname cache as I find getattr too chatty.

At the end of the day a FUSE implementation has to do the same thing that WinFsp and Dokany do. Context switch between the process that issues a file system call to the process that implements the file system, then context switch back. The primary optimization opportunity is: try to satisfy the I/O without context switching to the file system. A secondary optimization goal is: if you do context switch to the file system do it really FAST!

But of course FUSE is very mature by now and probably optimized as well as it can be (at least on Linux). I might try to recreate MEMFS for low-level FUSE and run some benchmarks to get a baseline if I ever get the time.

@billziss-gh, I realize that passthrou#64 is not the reference fileystem implementation for WinFSP (memfs is). But I think passthrou#64 would be my starting point if I were to change to WinFSP because I am not really implementing a whole filesystem. I'm basically doing what passthrough or Dokany mirror does, but with encryption/decryption.

It is actually trivial to enable the cache manager for a WinFsp file system and gain a huge performance boost for free. Unfortunately passthrough will not work well under the cache manager without modifications. The primary reason is that passthrough simply maps its own file system handles onto (different) handles in the underlying (NTFS) file system. Unfortunately the cache manager and other Windows components have the tendency to keep files open for longer periods than those requested by user mode applications and consequently passthrough also keeps its own handles open for long times; this often creates conflicts in the underlying (NTFS) file system.

I believe it is possible to modify passthrough to work around those problems. But it is not a trivial task to do correctly. I may do it some time in the future to enable file systems like yours to easily use it as a base for their development.

Some of the reasons that Windows likes to keep files open:

• It will often buffer write requests and "lazy" write them at a later time, often after the file is closed!
• Open, I/O, Close, ... is a very common pattern in applications, so if the file is not "really" closed on the OS side (despite what the application thinks) the cache remains intact and does not need to be "reprimed" at the next Open.

Lastly can you describe the scenario that you use to measure performance in more detail? You mention lots of small files, but you do not mention if you just open and close them or you also do some I/O on them, etc. I would like to add a similar scenario to my own testing and see if there are optimization opportunities that I have missed. Thanks!

from cppcryptfs.

Dokany released version 2 beta, which seems to be focused on speed enhancements. How does it compare with Dokany 1.x in your tests, @bailey27? Is it still slower than @billziss-gh's WinFSP?

from cppcryptfs.

And, by the way, recent changes of GoCryptFS include mention of using memory pools for buffer handling which doubles the streaming read speed on its developer’s machine. Have you tried that, @bailey27?

from cppcryptfs.

@bailey27 Yes Dokan 2 is still a beta and the stability may not be here for now.
The migration of cryptdokan may not be so hard to do since you based your code on the mirror.
Anyway, there is still some work to do but it would be really great to have feedback/help on it !

from cppcryptfs.

@sergeevabc , I also meant to say that I think if you need performance like native NTFS, then you maybe should try something like VeraCrypt or Bitlocker.

from cppcryptfs.

Thanks for your explanations. They are really helpful and well-written.

My tests were somewhat informal, and revolved around linux-3.0.tar.gz
https://www.kernel.org/pub/linux/kernel/v3.0/linux-3.0.tar.gz

That's because that what gocryptfs uses for benchmarks. It has over 36,000 files in it.

I timed untaring it with cygwin tar, ls -lR (piped in to /dev/null), dir /s/b (piped into >nul), rm -rf, rd /s/q, and also I used native Windows (but 32bit) info-zip to zip it up from the virtual drive onto the virtual drive.

I did the tests with Windows defender realtime scanning turned off.

I wasn't particularly thourough, and the benchmarks were run in a 2 cpu 2GB Virtualbox VM with the VDI file residing on an SSD in a USB 3.0 enclosure.

I was mainly interested in determining if Dokany 2.0 was any faster than 1.x in these cases, and also to re-validate what I remembered from testing passthrough before.

from cppcryptfs.

Thanks for the test scenario. It is very useful.

from cppcryptfs.

@bailey27, thanks for mentioning notable VeraCrypt by @idrassi and Bitlocker by Microsoft, but the former is about one large container which might be tricky to backup/sync and the latter is a proprietary.

Also there is a promising undertaking for transparent encryption called Cryptomator, but is dependent on WebDAV with so far unresolved 4 GiB hitch. even thought I suggested switching to FUSE or even smooth and solid Pismo File Mount (used by EncFSMP, for example). Be sure to benchmark PFM, by the way.

Meanwhile, Windows users are fortunate to have a FUSE-based SecureFS by @netheril96 to play with.

from cppcryptfs.

I tried Cryptomator, but I realized that Windows WebDAV was caching the plaintext versions of my files on the local disk, which kind of threw at-rest encryption out the window. I could not find any way of disabling the WebDAV cache either.

from cppcryptfs.

Tried to launch CPPCrypfFS with Dokany 2 with no luck, since the former requires dokan1.dll and the latter is shipped with dokan2.dll. Aren't they compatible enough to implement the support, @bailey27?

from cppcryptfs.

@sergeevabc Dokan is using Semantic Versioning. We had to change the API compatibility and therefore we updated the major version (2.x.x).
It is planned to have a legacy interface later to keep supporting the 1.x.x API but would still need to link to the dokan2 library.

To make your test, you would need to change the include path to dokan 2 folder and update API functions. They are the same functions, but just that all parameters are included in one struct now.

from cppcryptfs.

Some on-the-fly encryption apps seem to be affected by space reservation issue, CPPCryptFS is not.

from cppcryptfs.

If a file is extended without being written to, then cppcryptfs will at most only have to read and re-write what was formerly the last encryption block.

from cppcryptfs.

I'm switching from Veracrypt to cppcryptfs (Win 8.1, USB 3.0 drives, Dokany 1.1.0.2000).

I noticed slow writing performance to cppcryptfs (19 MB/s), while reading is around 40 MB/s.
When reading from my VeraCrypt volume, I get around 80 MB/s. Writing to VeraCrypt is at around 35 MB/s.

Although this might be slower than generally possible, I prefer Dokany & cppcryptfs because every component is opensource & file based. Although VeraCrypt is opensource, it it container based, which makes backup up a lot harder.

I also have EncFSMP installed, and although performance is better, I'm going to replace it with cppcryptfs because EncFSMP uses the closed tool file system tool Pismo File Mount.

With Crypotmator, I noticed that every file is temporarily stored on the system drive.
Now imagine what encrypting several TB of files would mean to your SSD system drive life...

Performance is important. Nevertheless, especially with Win10 upgrading every 6 months now, I think it is more important to have an open source base, that a large community is going to adapt to new OS requirements.

Just my two cents :-)

from cppcryptfs.

[ Related #14 ]

@bailey27
FYI, WinFSP 2018.2 will be coming out in a "month or so" per comments here winfsp/winfsp@f09597a

It implements FUSE3 API (version 3.2) among other improvements like "The FUSE layer now supports multiple file systems within a single process."

I know you said that you're busy, just thought I'd throw this info in there in case thing free up in the future

@heikoh81
I passed on Cryptomator because it uses Java and I've been clean for 2 years now. I use VeraCrypt for my local storage encryption and cppcryptfs for my "portable" apps and cloud storage. Have you looked at https://rclone.org/commands/rclone_mount/ ?

from cppcryptfs.