Comments (10)
Instead of using lock()
, maybe we should use tryLock()
with a timeout? I'd rather overwrite some data, than have no data written at all.
from secor.
Underneath, locking is implemented on top of twitter's DistributedLock. You can read about the details of that algorithm here: http://twitter.github.io/commons/apidocs/index.html#com.twitter.common.zookeeper.DistributedLockImpl
Since locking is based on ephemeral nodes, no cleanup is required. It is possible that intermediate paths will stay in zk after the thread holding the lock dies but I don't think it's a big deal since those paths will be reused.
from secor.
Indeed, it doesn't seem right. That's why I reported the issue here.
from secor.
What doesn't seem right?
from secor.
The part where the ZK locks were stuck, and the only solution was to manually remove them, is not right.
from secor.
What are the circumstances under which that would happen?
from secor.
In my particular case, it happened upgrading the secor cluster from the previous build (which was c982615 at the time) to the latest build (which was 96969a8 at the time).
from secor.
Did you wait long enough for the zk lease to expire before diagnosing the problem?
According to the Java API linked above, a deadlock would happen if the worker is hanging but the zk thread is alive. I don't see a code path that would lead to this unless the consumer somehow gets stuck on s3 operations.
from secor.
I think I waited as long as 60 minutes, which should have been more than enough. The other possibility is that there was a java process hung somewhere (holding the lock), but I never thought to check. If that was the case, it's no longer running.
from secor.
We need a better understanding of the issue before attempting a fix. The next time it happens, go to zookeeper and check the values under /secor/locks///. If there is a process holding a lock, ephemeralOwner should be set to a non-zero value. You should be also able to see mtimes and ctimes. Those should provide a bit more context.
For the use case at Pinterest, it is instrumental that there is no message duplication and loss so changing the semantics to best-effort locking is not a preferred solution. FWIW, in the last 6 months or so since we started using Secor, I didn't see an issue with locking but of course this doesn't prove anything.
from secor.
Related Issues (20)
- Blocking Issue when running on K8s on GCP HOT 1
- manually assign avro schema to topic HOT 1
- Crashing exception after some time HOT 1
- Cannot parse timestamp with timezone HOT 1
- Racing condition with parquet writer HOT 3
- Cannot make secor work with AWS instance profile HOT 8
- Ignore parsing exceptions HOT 2
- Expose JVM metrics when using Micrometer HOT 3
- Secor uploads (different) files with the same "name" into different "days". HOT 2
- Kuberntes OOM Error. Very High memory usage > 15gb HOT 2
- Is there any alternative to AvroParquetWriter so that we can avoid using IndexedRecord? HOT 1
- ParquetWriter write method is taking too long to write single avro message HOT 1
- A Simple Study about Configuration Java Types
- PartitionFinalizer fails when there is no data produced in the topic eventhough there are other topics with data HOT 3
- Add headers while calling schema registry for decoding messages HOT 1
- Duplicate message when partition revoking or rebalancing HOT 1
- secor-monitor container not getting started. HOT 2
- Tune file size create by Secor
- Secor with zookeerper less Kafka HOT 1
- Secor filling up local disk space with intermittent data
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from secor.