Comments (8)
sahlberg
commented on August 12, 2024
On Wed, Apr 19, 2017 at 2:05 PM, Julian Hjortshoj ***@***.***> wrote:
One of our users pointed out to us that flock applied on fuse-nfs mounts
doesn't enforce locking between different clients. After confirming that
1. It's true and
2. kernel NFS mounts *do* enforce flock locks between clients
We took a look at the fuse-nfs implementation and noticed that
fuse-nfs just treats lock() and flock() as no-ops.
So, we are wondering if you left the locking implementation out because
you didn't get to it, or if you left it out because NFS locking is flaky
and inconsistent between servers and could cause stuff to hang or fail or
is otherwise a bad idea.
A little bit of both. NFS and file locking have traditionally been
"fragile" so many/most applications used to recommend to not use locking at
all over nfs.
That said, I am not against adding basic locking to the fuse module if you
want to add that. I am happy to review a patch for this.
Unfortunately, due to the way nfs locking works, nlm and nsm, I think there
would have to be a few compromises for locking as some things may be
difficult to implement in libnfs and/or fuse.
Some of the compromises I think you might need to do would be to
1, From fuse, only use the non-blocking version on the nlm lock request and
have a sleep loop to retry in fuse.
I recall (from memory) that blocking NLM locks require that you set up a
local nlm server on the client side to listen to when the server does the
async
grant of the lock by sending a NLM_GRANTED request back to the client.
That would involve setting up and rpc server for nlm inside fuse and
register with the clients portmapper and would not work as it would
interfere with the native lock manager
that linux itself is running/depending on :-(
2, In nfs locking, lock recovery is driven by the nsm service. Similar to
above, when the server reboots and thus require the clients to recover the
locks, the server does a requaer does a rpc back to the client
to the nsm (rpc.statd) notify procedure to tell the client about the need
for lock recovery.
Again, to do this correctly in fuse would require that you run your own nsm
service and register it with portmapper. Again it would interfere with the
systems own nms service adn thus is not viable.
The main consequence of 2 is that I think you will not be able to do lock
recovery across server restarts. :-(
As server restart is reasonably rare,a nd applications using locking are
even more so, maybe you can live with the fact that locking will break
across every server restart.
3, Lock recovery also depends on the client to send nsm notify rpc back to
the server after a client restart/reboot. That will likely also be
difficult to do from fuse as who will be sending these requests
if the user just shuts down fuse and never restarts it?
Again, not the end of the world as this will just result in permanent
orphaned locks on the server. Something that is quite common and happens
normally in most NFS systems anyway.
You can probably live with these compromises. Just be aware that it will
not be "perfect" but then again nothing really is when it comes to nfs and
locking :-(
If it is the former, are you open to a PR to include a locking
implementation? And if so do you have an opinion about where the
synchronous implementation of rpc_nlm4_lock_async() should live?
(presumably either in libnfs or in fuse-nfs)
I am open to a PR.
Maybe it should be a nice wrapper in libnfs.c : nfs_flock_async()
and a sync varient in sync.c ?
—
… You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#20>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAeNkGMXY5LEle4la5uXOuYddaTVnbvGks5rxnb-gaJpZM4NCPbs>
.
from fuse-nfs.
julian-hj
commented on August 12, 2024
That all makes sense. Thanks very much for the quick and thoughtful feedback. We will have a think about how we want to proceed.
from fuse-nfs.
sahlberg
commented on August 12, 2024
Thinking about it even more,
I think you can solve item 3 in my list above without too much effort.
To track the state of the peer, the NSM protocol (rpc.statd) is used to
send notifications to the peer to indicate when the service is
stopped/started.
This is all signalled by a monotonically increasing integer describing the
state.
Sending an even state number means "client is starting up"
and an odd state means "client is shutting down".
In both cases the peer should discard any and all locks that were
registered to this client.
If you go this path, I suggest you add local state on the client to track
the state number. Store it persistently in the filesystem somewhere or any
other persistent store.
Everytime you start the fuse module you bump it to the next higher odd
number and send a nsm notify.
Everytime the fuse module is unloaded normally you do the same but send an
even number for state.
That should make it reasonably robust against leaking orphaned locks on the
server.
===
NLM / NSM docs. Below is pretty much any/all public documentation that
exists for these protocols. It explains why locking is so fragile in nfsv3.
Both NLM and NSM are very poorly documenting. Pretty much only thing that
exist is an interface specification but no real protocol specification
exist :-(
(This makes it so that folks that really do understand how the protocols
are supposed to work will never be unemployed :-) )
From /usr/include/rpcsvc/sm_inter.x :
/*
* state # of status monitor monotonically increases each time
* status of the site changes:
* an even number (>= 0) indicates the site is down and
* an odd number (> 0) indicates the site is up;
*/
struct sm_stat {
int state; /* state # of status monitor */
};
And also the PDU interface description:
http://archive.opengroup.org/publications/archive/CDROM/c702.pdf
…On Wed, Apr 19, 2017 at 4:10 PM, Julian Hjortshoj ***@***.***> wrote:
That all makes sense. Thanks very much for the quick and thoughtful
feedback. We will have a think about how we want to proceed.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#20 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAeNkNGwJDtc2ZyL8WlQzBRTLM2cZ0BGks5rxpR1gaJpZM4NCPbs>
.
from fuse-nfs.
julian-hj
commented on August 12, 2024
Interesting....so, how does the server know what "this client" means? In our use case, we may have numerous fuse-nfs processes starting and stopping on the same host, potentially mounting the same nfs share with different options.
Also, I noticed in the packet trace that libnfs just sends "libnfs" as the client machine name in the credentials block. Could that result in the nfs server getting confused when different fuse processes go up and down?
Thanks!
from fuse-nfs.
sahlberg
commented on August 12, 2024
Client that NSM is using should be the string that you provide in the nlm lock request as
NLM4_LOCKargs.lock.caller_name
The credentials block is part of the ONC-RPC header and should not affect (or even be visible) to the NLM layer, but it may be a good idea to change this from "libnfs" to something more descriptive.
If for no other reason making it easier to analyze and filter on network traces.
As few things really look at or depend on the cred->host field, having "libnfs" as the default might be ok-ish. Possibly this should be changed to something more descriptive such as libnfs- or something.
In your case you probably want to have something more specific to your use case and to easily distinguish between different instances of the fuse module. You can actually override and replace cred->host with whatever you want using the nfs_set_auth and libnfs_authunix_create calls.
Maybe fuse-nfs.c should use these functions and set the credentials explicitely to "fuse-nfs-" or something ?
Even better might be to be able to set the hostname string from the command line via the nfs url.
The NFS url already allow you to set uid and gid. Maybe just add an argument to also set the hostname.
from fuse-nfs.
sahlberg
commented on August 12, 2024
FYI.
NFSv4 support in libnfs is coming along and should be reasonably complete and to the state where
one can use it for for example fuse-nfs. This is likely going to be ready towards the end of the year.
At that stage, when we can run libnfs in a NFSv4 mode, it should be fairly straightforward to do the plumbing to add locking as we no longer need to rely on NLM or NSM.
from fuse-nfs.
julian-hj
commented on August 12, 2024
That's excellent to hear! NFSv4 and file locking have both been on our roadmap for a while. If we can add support for v4, then I think support for locking in v3 might be quite a bit less critical.
from fuse-nfs.
sahlberg
commented on August 12, 2024
Libnfs has support for both lockf() and fcntl() locking now but only for nfs v4.
With this, it should be fairly straightforward to add locking support to fuse-nfs.
from fuse-nfs.
Related Issues (20)
- Multi-threaded support HOT 15
- NFSv4 issue HOT 1
- Additional dependencies not mentioned in README.md HOT 1
- How could i set the mount option like noresvport nolock in fuse-nfs?
- data write issue with v4 mounts in latest fuse-nfs HOT 2
- Use the poll polyfill from win32/win32_compat.h from libnfs for windows builds HOT 1
- SIGSEGV in fuse_nfs_readdir due to permission denied on NFS server
- Use fuse_opt_parse instead of getopt & change usage
- can not read by non-root user, after mount? HOT 2
- failed to open /dev/fuse: Permission denied
- nfs4 kerberos验证支持
- Mount goes OK, but first access results in "Software caused connection abort" error
- Can't connect to remote NFS drive
- Failing to compile HOT 3
- Failed to mount nfs share : mount_cb: RPC error: Mount failed with error MNT3ERR_ACCES(13) Permission denied(13)
- EFS NFS Mount - Input/output error
- Readme Markdown Document has error in option part HOT 4
- Not able to write to mounted system HOT 1
- Mac Support
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fuse-nfs.