meituan-dianping / leaf Goto Github PK
View Code? Open in Web Editor NEWDistributed ID Generate Service
License: Apache License 2.0
Distributed ID Generate Service
License: Apache License 2.0
compile 'com.sankuai.inf.leaf:leaf-parent:1.0.1'
compile 'com.sankuai.inf.leaf:leaf-server:1.0.1'
compile 'com.sankuai.inf.leaf:leaf-core:1.0.1'
想尝试用核心包来做开发,发现这样引用引不成功,在maven center也没有找到leaf的包。
if (cache.containsKey(key)) {
SegmentBuffer buffer = cache.get(key);
if (!buffer.isInitOk()) {
synchronized (buffer) {
if (!buffer.isInitOk()) {
try {
updateSegmentFromDb(key, buffer.getCurrent());
logger.info("Init buffer. Update leafkey {} {} from db", key, buffer.getCurrent());
buffer.setInitOk(true);
} catch (Exception e) {
logger.warn("Init buffer {} exception", buffer.getCurrent(), e);
}
}
}
}
return getIdFromSegmentBuffer(cache.get(key));
}
if containsKey then get
NA
官方博客介绍snowflake模式下整个启动流程如下:
服务启动时首先检查自己是否写过ZooKeeper leaf_forever节点:
- 若写过,则用自身系统时间与leaf_forever/${self}节点记录时间做比较,若小于leaf_forever/${self}时间则认为机器时间发生了大步长回拨,服务启动失败并报警。
- 若未写过,证明是新服务节点,直接创建持久节点leaf_forever/${self}并写入自身系统时间,接下来综合对比其余Leaf节点的系统时间来判断自身系统时间是否准确,具体做法是取leaf_temporary下的所有临时节点(所有运行中的Leaf-snowflake节点)的服务IP:Port,然后通过RPC请求得到所有节点的系统时间,计算sum(time)/nodeSize。
- 若abs( 系统时间-sum(time)/nodeSize ) < 阈值,认为当前系统时间准确,正常启动服务,同时写临时节点leaf_temporary/${self} 维持租约。
- 否则认为本机系统时间发生大步长偏移,启动失败并报警。
- 每隔一段时间(3s)上报自身系统时间写入leaf_forever/${self}。
请问有关第2步、第3步临时节点leaf_temporary的相关逻辑,源码中好像没有,是否是版本更新摒弃了这一部分逻辑。
`
private void waitAndSleep(SegmentBuffer buffer) {
int roll = 0;
while (buffer.getThreadRunning().get()) { //此处会引发code hoisting造成死循环,建议使用必要的同步实现
roll += 1;
if(roll > 10000) {
try {
Thread.currentThread().sleep(10);
break;
} catch (InterruptedException e) {
logger.warn("Thread {} Interrupted",Thread.currentThread().getName());
break;
}
}
}
}
`
测试样例
`
public static void main(String[] args) throws InterruptedException, NoSuchMethodException {
SegmentIDGenImpl segmentIDGen = new SegmentIDGenImpl();
final SegmentBuffer buffer = new SegmentBuffer();
Method waitAndSleepMethod = segmentIDGen.getClass().getDeclaredMethod("waitAndSleep", SegmentBuffer.class);
new Thread(() -> {
try {
waitAndSleepMethod.setAccessible(true);
waitAndSleepMethod.invoke(segmentIDGen, buffer);
} catch (Exception e) {
e.printStackTrace();
}
}).start();
System.out.println(System.nanoTime() + " main thread start sleep");
Thread.sleep(1000);
System.out.println(System.nanoTime() + " main thread wake up");
buffer.getThreadRunning().compareAndSet(false, true); //set value of threadRunning to true
System.out.println(System.nanoTime() + " main thread - " +
"the current value of threadRunning : " + buffer.getThreadRunning().get());
}
`
我司用的是docker发布, 每次发布后容器的ip都不一样, 如果拿ip注册到zookeeper,是不是会有问题, 本地文件缓存的workerId的意义就不是很大了, 你们有遇到这种情况吗? 据我了解, 美团也是容器发布的, 你们的ip是不变的是吗
SnowflakeIDGenImpl
//如果是新的ms开始
sequence = RANDOM.nextInt(100);
random没什么意义,而且这样导致同一时刻tps下降。原本同一时刻tps可能为1024,但是random后可能减少100了。
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'leafController': Unsatisfied dependency expressed through field 'segmentService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'SegmentService' defined in file [D:\Leaf\leaf-server\target\classes\com\sankuai\inf\leaf\server\SegmentService.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.sankuai.inf.leaf.server.SegmentService]: Constructor threw exception; nested exception is java.lang.NullPointerException
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:588) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1272) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867) ~[spring-context-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543) ~[spring-context-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:124) ~[spring-boot-1.5.18.RELEASE.jar:1.5.18.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693) ~[spring-boot-1.5.18.RELEASE.jar:1.5.18.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360) ~[spring-boot-1.5.18.RELEASE.jar:1.5.18.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:303) ~[spring-boot-1.5.18.RELEASE.jar:1.5.18.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1118) ~[spring-boot-1.5.18.RELEASE.jar:1.5.18.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1107) ~[spring-boot-1.5.18.RELEASE.jar:1.5.18.RELEASE]
at com.sankuai.inf.leaf.server.LeafServerApplication.main(LeafServerApplication.java:10) ~[classes/:na]
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'SegmentService' defined in file [D:\Leaf\leaf-server\target\classes\com\sankuai\inf\leaf\server\SegmentService.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.sankuai.inf.leaf.server.SegmentService]: Constructor threw exception; nested exception is java.lang.NullPointerException
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1163) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1107) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:585) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
... 19 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.sankuai.inf.leaf.server.SegmentService]: Constructor threw exception; nested exception is java.lang.NullPointerException
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:154) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:89) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1155) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
... 30 common frames omitted
Caused by: java.lang.NullPointerException: null
at com.alibaba.druid.util.JdbcUtils.getDriverClassName(JdbcUtils.java:360) ~[druid-1.0.18.jar:1.0.18]
at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java:642) ~[druid-1.0.18.jar:1.0.18]
at com.sankuai.inf.leaf.server.SegmentService.<init>(SegmentService.java:35) ~[classes/:na]
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:na]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[na:na]
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142) ~[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
... 32 common frames omitted
看下了下源码。workId直接取的zk序列号, 如果ZK序列号超过1023, 那岂不是信节点无法加入。是不是需要取摩处理呢, 望解答。
这里我理解有一个问题
segment切换后,原来的号段没有到达maxId,这时转到新的segment,value值重新计算,会废弃掉前一个segment剩余没有用到的id。
rt
我们有个IM场景,需要每个会话都是趋势递增(可以不连续),但现在如果布多个leaf做不到这一点,有没有什么思路,谢谢了。
我看到segment模式下每分钟都更新了cache,但在updateSegmentFromDb方法中duration与SEGMENT_DURATION比较时的SEGMENT_DURATION是十五分钟,这个怎么解释。duration在这里不可能大于1分钟。肯定是小于15分钟的。
Leaf Snowflake是一个很棒的项目,非常感谢Leaf Snowflake的开源!
基于Leaf Snowflake生成一个UID后,我希望将各部分拆解出来,得知通过什么key,什么时间生成的ID,我需要怎么做?
谢谢!
如何引入 leaf-core, 似乎没有发布到 maven 仓库
拜读了一下源码,目前workerI都是根据zk排队序号生成,大多时候为0。
这样的话根据workerId,也就不能反推出申请业务,在SnowflakeIDGenImpl的get方法中,参数key没有使用途径。
我建议workerId,其实可以由key的规则生成,这样workerId不是更有意义吗
如题
考虑极端情况
假设step=10,当有21个请求同时进来
1.编号1-10通过bufferA返回 第二个请求去load bufferB数据
2.编号11-21进入waitAndSleep等待
3.bufferB加载完成 编号11-20通过bufferB返回 编号21将抛异常
com.sankuai.inf.leaf.common.Utils#getIp linux下获取到127.0.0.1
发现一个问题想请教:
我DB设置的step的步长为2000,消费了27个id,然后重启应用发现,
id不是从28开始,而是直接从2001开始, (2001-27)个id被丢弃,
丢弃掉了前一个segment剩余没有用到的id。
Is the code only guaranteed to be concurrent in stand-alone mode?
从数据库取tag, id, step是用SQL: @select("SELECT biz_tag, max_id, step FROM leaf_alloc WHERE biz_tag = #{tag}")
但没有使用锁, 如果保证多个服务同时取而不重复呢
配置文件路径错误:
修正PROP_PATH :
System.getProperty("java.io.tmpdir") => System.getProperty("java.io.tmpdir") + File.separator
也开源个springcloud版本的吧。
[WARNING]
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.springframework.boot.maven.AbstractRunMojo$LaunchRunner.run (AbstractRunMojo.java:528)
at java.lang.Thread.run (Thread.java:745)
Caused by: org.springframework.boot.context.embedded.tomcat.ConnectorStartFailedException: Connector configured to listen on port 8080 failed to start
at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.checkThatConnectorsHaveStarted (TomcatEmbeddedServletContainer.java:237)
at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.start (TomcatEmbeddedServletContainer.java:213)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.startEmbeddedServletContainer (EmbeddedWebApplicationContext.java:308)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh (EmbeddedWebApplicationContext.java:147)
at org.springframework.context.support.AbstractApplicationContext.refresh (AbstractApplicationContext.java:546)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh (EmbeddedWebApplicationContext.java:124)
at org.springframework.boot.SpringApplication.refresh (SpringApplication.java:693)
at org.springframework.boot.SpringApplication.refreshContext (SpringApplication.java:360)
at org.springframework.boot.SpringApplication.run (SpringApplication.java:303)
at org.springframework.boot.SpringApplication.run (SpringApplication.java:1118)
at org.springframework.boot.SpringApplication.run (SpringApplication.java:1107)
at com.sankuai.inf.leaf.server.LeafServerApplication.main (LeafServerApplication.java:10)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.springframework.boot.maven.AbstractRunMojo$LaunchRunner.run (AbstractRunMojo.java:528)
at java.lang.Thread.run (Thread.java:745)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 51.736 s
[INFO] Finished at: 2019-03-08T10:23:55+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:1.5.18.RELEASE:run (default-cli) on project leaf-server: An exception occurred while running. null: InvocationTargetException: Connector configured to listen on port 8080 failed to start -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
<curator.version>4.0.1</curator.version>
无法直接运行所打的jar包
你好,在 SegmentIDGenImpl.class 中, updateSegmentFromDb 方法里, 有一句注释, " // must set value before set max ",请问这里这样做是如何考虑的. 谢谢.
请问关于雪花算法,我看了一下默认生成的是19位的id,请问怎么控制id生成的长度呢?旧有系统有16位id的,也有22位id的,都想用id自动生成
twepoch应该是twwiter开始部署这个项目的时间吧
我想问下目前是否支持高可用部署
某些情况下ID号会作为参数的一部分编在二维码里,所以希望ID号能短一些。
我的想法是使用36进制进行标识。(09 AZ)
是否可以实现一下。
1.snowflake 模式新的毫秒数时random的考虑。
2.snowflake 模式生成的id是不定长的吧?定长的话有什么方案建议么?
getIdFromSegmentBuffer() 在调用waitAndSleep()方法前currentSegment获取正确的value已经失败,在调用waitAndSleep()等待nextSegment完成对接DB的更新操作(如有必要)。
waitAndSleep()方法返回后,又对当前currentSegment尝试获取value。如果失败,则切换segment,下次循环中再次获取value
waitAndSlee()前后都对同一segment进行对获取value操作。这是有什么特殊含义在里面吗?
import com.google.common.base.Preconditions;
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>23.0</version>
</dependency>
你好,看代码是做了map缓存,那么如果多节点部署leaf server, 是否会得到重复的ID
如果加锁失败的话,会无缘故地释放锁,是否考虑把加锁代码放到try ..catch外面,如下代码:
class X {
// ...
public void m() {
lock.lock(); // block until condition holds ①
try {
// ... method body
} finally {
lock.unlock() // ②
}
}
}
support Monotone increasing ID Monotone increasing feature
当步长设置为50。然后运行SegmentIDGenImplTest
中的测试方法,循环5000次,就会报错:
Both two segments in SegmentBuffer{key='leaf-segment-test', segments=[Segment(value:53,max:51,step:50), Segment(value:103,max:101,step:50)], currentPos=1, nextReady=false, initOk=true, threadRunning=true, step=50, minStep=50, updateTimestamp=1556434867190} are not ready!
从错误信息来看,threadRunning=true
表示当前buffer
正在填充数据,但是获取nextID
的逻辑并没有等待数据填充完,就尝试获取导致报错
当需要批量插入记录时,循环调用获取ID的方式效率如何?
如果不支持批量获取ID,针对这种场景有什么建议吗?谢谢~
leaf-server
下 com.sankuai.inf.leaf.server
的class路径比较混淆,如果将controller、service放到对应的包下,是否更合适一点?
Leaf服务内部两个号段缓存区,当前号段使用10%时,去获取下一个号段,想请教一下10%这个值是出于什么来设定的?这个10%应该是有一个计算的逻辑把,很好奇这个值为什么是10%,希望指点一下
com.sankuai.inf.leaf.segment.SegmentIDGenImpl 的waitAndSleep方法,在 Thread.currentThread().sleep(10); 后的那个 break 是否应该改成 continue ?使用break直接跳出循环了
最近一次提交是6个月前,也没看到正式的release版本,是不是已经放弃这个项目了?
//如果是新的ms开始
sequence = RANDOM.nextInt(100);
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.