Git Product home page Git Product logo

jprotobuf-rpc-socket's Introduction

Jprotobuf-rpc-socket

Build status

Maven Central

Protobuf RPC是一种基于TCP协议的二进制高性能RPC通信协议实现。它以Protobuf作为基本的数据交换格式,支持完全基于POJO的发布方式,极大的简化了开发复杂性。
Features:

  • 完全支持POJO方式发布,使用非常简单
  • 内置连接池,具备更高的性能,低延迟 QPS: 5w+
  • 支持自动重连功能
  • Client支持Ha的负载均衡功能
  • 支持附件发送
  • 压缩功能,支持GZip与Snappy
  • 支持多包拆分与合并功能
  • 支持多级超时设定,灵活控制请求超时时间
  • 支持RPC服务元数据动态生成,简单易用
  • 集成内置HTTP管理功能(3.1.1版本+)

关联项目: https://github.com/jhunters/jprotobuf
golang 协议实现: https://github.com/baidu-golang/baidurpc

使用示例

环境要求:JDK 6+

<dependency> 
	<groupId>com.baidu</groupId>
	<artifactId>jprotobuf-rpc-core</artifactId>
	<version>4.2.1</version>
</dependency>

<!-- 提供spring扩展 -->
<dependency>
	<groupId>com.baidu</groupId>
	<artifactId>jprotobuf-rpc-core-spring</artifactId>
	<version>4.2.1</version>
</dependency>

<!-- 提供spring boot扩展 -->
<dependency>
	<groupId>com.baidu</groupId>
	<artifactId>jprotobuf-rpc-spring-starter</artifactId>
	<version>4.2.1</version>
</dependency>

<!-- 基于redis实现服务注册,发现功能 -->
<dependency>
	<groupId>com.baidu</groupId>
	<artifactId>jprotobuf-rpc-registry-redis</artifactId>
	<version>4.2.1</version>
</dependency>

使用Jprotobuf pre compile插件进行预编译,提升启动速度

    <plugin>
        <groupId>com.baidu</groupId>
        <artifactId>jprotobuf-precompile-plugin</artifactId>
        <version>1.2.8</version>
        <configuration>
            <skipErrorNoDescriptorsFound>true</skipErrorNoDescriptorsFound>
            <filterClassPackage>com.baidu</filterClassPackage>
        </configuration>
        <executions>
            <execution>
                <phase>compile</phase>
                <goals>
                    <goal>precompile</goal>
                </goals>
            </execution>
        </executions>
    </plugin>

filterClassPackage 用来指定进行预编译时需要扫描的package,目前只支持配置一个package名称
maven执行命令如下:

mvn jprotobuf:precompile
or
mvn package 

下载发行包

Qucik Start

Jprotobuf-rpc-socket基于JProtobuf基础上开发,可帮助大家开发中省去编写Google Protobuf的IDL描述文件的过程。

客户端开发

1.EchoService功用实现

EchoService 提供一个echo方法 ,参数对象EchoInfo只有一个message属性。 下面是EchoInfo对象定义

public class EchoInfo {
    
    @Protobuf
    public String message;
}

注解方式的定义可以极大简化大家的工作量,上面等同于下面的IDEL配置

package pkg;  

option java_package = "com.baidu.bjf.remoting.protobuf.rpc";

//这里声明输出的java的类名  
option java_outer_classname = "EchoInfo";  

message InterClassName {  
  required string message = 1;
} 


2.定义EchoService接口

public interface EchoService {

    /**
     * To define a RPC client method. <br>
     * serviceName is "echoService"
     * methodName is use default method name "echo"
     * onceTalkTimeout is 200 milliseconds
     * 
     * @param info
     * @return
     */
    @ProtobufRPC(serviceName = "echoService", onceTalkTimeout = 200)
    EchoInfo echo(EchoInfo info);
}

RPC的方法必须要指定@ProtobufRPC注解. serviceName与methodName要与服务端保持一致。 这里未指定methodName,则使用方法的名称 "echo"

3.创建RPC Client进行访问

RpcClient rpcClient = new RpcClient();
// 创建EchoService代理
ProtobufRpcProxy<EchoService> pbrpcProxy = new ProtobufRpcProxy<EchoService>(rpcClient, EchoService.class);
pbrpcProxy.setPort(1031);
// 动态生成代理实例
EchoService echoService = pbrpcProxy.proxy();
EchoInfo request = new EchoInfo();
request.message = "hello";
EchoInfo response = echoService.echo(request);
rpcClient.stop();
服务端开发

1.开发服务实现类

public class EchoServiceImpl {

    @ProtobufRPCService(serviceName = "echoService", methodName = "echo")
    public EchoInfo doEcho(EchoInfo info) {
        EchoInfo ret = new EchoInfo();
        ret.setMessage("hello:" + info.message);
        
        return ret;
    }
}

服务发布的RPC方法必须用@ProtobufPRCService注解进行标识

2.发布RPC服务

	RpcServer rpcServer = new RpcServer();
	
	EchoServiceImpl echoServiceImpl = new EchoServiceImpl();
	rpcServer.registerService(echoServiceImpl);
	rpcServer.start(1031);

上面的代码实现把 EchoServiceImpl 的RPC服务发布出去

更多使用说明

性能测试

机器配置:

  • Linux 64G内存 6核 12线程
  • Intel(R) Xeon(R) CPU E5645 @ 2.40GHz

性能测试结果如下(客户端与服务端部署在同一台机器中): 单线程:平均QPS: 9000+ 多线程:QPS: 最高 40000+

---------------------Performance Result-------------------------
send byte size: 44;receive byte size: 50
|         total count|       time took(ms)|           average(ms)|                 QPS|             threads|
|              100000|               11807|                     0|                8469|                   1|
---------------------Performance Result-------------------------
---------------------Performance Result-------------------------
send byte size: 44;receive byte size: 50
|         total count|       time took(ms)|           average(ms)|                 QPS|             threads|
|              100000|               10407|                     0|                9608|                   1|
---------------------Performance Result-------------------------
---------------------Performance Result-------------------------
send byte size: 1139;receive byte size: 1139
|         total count|       time took(ms)|           average(ms)|                 QPS|             threads|
|              100000|               11513|                     0|                8685|                   1|
---------------------Performance Result-------------------------
---------------------Performance Result-------------------------
send byte size: 44;receive byte size: 50
|         total count|       time took(ms)|           average(ms)|                 QPS|             threads|
|              100000|                5904|                     0|               16937|                   2|
---------------------Performance Result-------------------------
---------------------Performance Result-------------------------
send byte size: 44;receive byte size: 50
|         total count|       time took(ms)|           average(ms)|                 QPS|             threads|
|              100000|                3754|                     0|               26638|                   4|
---------------------Performance Result-------------------------
---------------------Performance Result-------------------------
send byte size: 44;receive byte size: 50
|         total count|       time took(ms)|           average(ms)|                 QPS|             threads|
|              100000|                1736|                     0|               57603|                  20|
---------------------Performance Result-------------------------
---------------------Performance Result-------------------------
send byte size: 1139;receive byte size: 1139
|         total count|       time took(ms)|           average(ms)|                 QPS|             threads|
|              100000|                2381|                     0|               41999|                  20|
---------------------Performance Result-------------------------
---------------------Performance Result-------------------------
send byte size: 1139;receive byte size: 1139
|         total count|       time took(ms)|           average(ms)|                 QPS|             threads|
|              100000|                2012|                     0|               49701|                  40|
---------------------Performance Result-------------------------
License

/*
 * Copyright 2002-2007 the original author or authors.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

沟通群号:QQ: 644867264 ### 进群口令 jprotobuf

jprotobuf-rpc-socket's People

Contributors

baidu-etech avatar jhunters avatar jiangfeng avatar luciferyang avatar ruiyun avatar sduzh avatar xiemalin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

jprotobuf-rpc-socket's Issues

增加WEB查看功能

支持如果部署在web环境下,可以通过页面查看发布的服务情况

不能支持Proxy对象的服务发布

不能支持Proxy对象的服务发布
当我们发布一个代理类的时候,在下面代码中会出现问题。

com.baidu.jprotobuf.pbrpc.server.RpcServiceRegistry

    /**
     * Register service.
     *
     * @param target the target
     */
    public void registerService(final Object target) {
        if (target == null) {
            throw new IllegalArgumentException("Param 'target' is null.");
        }
        Class<? extends Object> cls = target.getClass();
        ReflectionUtils.doWithMethods(cls, new ReflectionUtils.MethodCallback() {
            public void doWith(Method method) throws IllegalArgumentException, IllegalAccessException {
                ProtobufRPCService protobufPRCService = method.getAnnotation(ProtobufRPCService.class);
                if (protobufPRCService != null) {
                    doRegiterService(method, target, protobufPRCService);
                }
            }
        });
    }

在运行的时候,下面的代码,拿到的类为"class com.sun.proxy.$Proxy81"。

Class<? extends Object> cls = target.getClass();

这就导致在下面的语句中,我们拿不到需要注册的方法.

ProtobufRPCService protobufPRCService = method.getAnnotation(ProtobufRPCService.class);

我们的实现代码是这样的。

@Slf4j
@RpcExporter(port = "1033", rpcServerOptionsBeanName = "rpcServerOptions")
@Service(DemoConstants.PMP_SERVICE)
public class PmpServiceImpl implements PmpService {

    @Autowired
    private HelloWorldRunService helloWorldRunService;

    @ProtobufRPCService(serviceName = DemoConstants.PMP_SERVICE, methodName = "helloWorld")
    @Override
    public HelloWorldResponse helloWorld(HelloWorldRequest request) {
        log.info("Called by request:" + request);
        HelloWorldResponse response = helloWorldRunService.helloWorld(request);
        return response;
    }

}

实际上我们可以通过下面的代码,拿到被代理类,不知道开发者不做这件事情是否有其他考虑?

public static Object getJdkDynamicProxyTargetObject(Object proxy) throws Exception {  
    InvocationHandler invocationHandler = Proxy.getInvocationHandler(proxy);  
    Object target = ((AdvisedSupport) ReflectionUtils.getFieldValue(invocationHandler,"advised")).getTargetSource().getTarget();  
    return target;  
}  

没用spring,一定要依赖spring吗?

Exception in thread "main" java.lang.NoClassDefFoundError: org/springframework/beans/factory/InitializingBean
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.baidu.jprotobuf.pbrpc.server.RpcServiceRegistry.(RpcServiceRegistry.java:46)
at com.baidu.jprotobuf.pbrpc.transport.RpcServer.(RpcServer.java:110)
at com.baidu.jprotobuf.pbrpc.transport.RpcServer.(RpcServer.java:118)

部署在linux时,netty会一直抛出一个warn

[ WARN ] [2016-06-29 17:35:04] io.netty.channel.DefaultChannelPipeline [151] - An exception was thrown by a user handler's exceptionCaught() method while handling the following exception:
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)

[ WARN ] [2016-06-30 17:04:46] io.netty.channel.DefaultChannelPipeline [151] - An exception was thrown by a user handler's exceptionCaught() method while handling the following exception:
java.io.IOException: 远程主机强迫关闭了一个现有的连接。
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)

请教Connection的requestQueue设计问题

Connection里的requestQueue队列的设计用意是在网络不通的时候缓存请求,网络通畅的时候再去消费的吧?

生产者代码会在RpcChannel#doTransport调用:

        if (!connection.getFuture().isSuccess()) {
            try {
                connection.produceRequest(state);
            } catch (IllegalStateException e) {
                RpcClientCallState callState = rpcClient.removePendingRequest(correlationId);
                if (callState != null) {
                    callState.handleFailure(e.getMessage());
                    LOG.log(Level.FINE, "id:" + correlationId + " is put in the queue");
                }
            }
        }

消费代码是注册在ChannelPoolObjectFactory#wrap回调里:

    @Override
    public PooledObject<Connection> wrap(Connection connection) {
        InetSocketAddress address;
        if (host == null) {
            address = new InetSocketAddress(port);
        } else {
            address = new InetSocketAddress(host, port);
        }
        ChannelFuture future = this.rpcClient.connect(address);

        // Wait until the connection is made successfully.
        future.awaitUninterruptibly();
        if (!future.isSuccess()) {
            LOGGER.log(Level.SEVERE, "failed to get result from stp", future.cause());
        } else {
            connection.setIsConnected(true);
        }

        future.addListener(new RpcChannelFutureListener(connection));
        connection.setFuture(future);
        
        return new DefaultPooledObject<Connection>(connection);
    }

1.如果是非短连接的情况下,因为缓存池会校验连接是否成功,所以第一步生产者代码都跑不进去,这个理解没问题吧?

public boolean validateObject(PooledObject<Connection> p) {
        Connection c = p.getObject();
        Channel channel = c.getFuture().channel();
        return channel.isOpen() && channel.isActive();

    }

2.那么考虑短连接的情况,断点跟踪,在服务端无法连通的情况下,确实会进入生产者代码,但是,消费者代码真的会跑进回调代码去消费吗?

future.awaitUninterruptibly()会一直等到这个future的结果,也就是拿到了这个连接的结果(连接成功 / 失败),接下去的注册future回调直接就会跑到EventLoop里“同步”执行,如果这个awaitUninterruptibly操作失败了,后续生产者生产缓存请求,但是后续也不会再去激活回调去消费缓存代码了吧?

也就是说:

  • 如果连接是可用的,这个队列不会去缓存请求;
  • 如果这个连接不可用,这个队列是会去缓存请求,但是无法被消费;

所以Connection里的这个任务缓存队列,是否都像是一个多余的设计?或者可能是我没有考虑到其他的情况。

另一方面,如果这个队列设计是work的情况下,用ArrayBlockingQueue,在大量可用连接存在的情况,是否存在浪费内存的嫌疑,能否考虑用LinkedBlockingQueue替代?

以上,谢谢~

3.5.3依赖jprotobuf-jre:1.0.0会导致jar包冲突

+--- com.baidu:jprotobuf-rpc-registry-redis:3.5.3
|    +--- com.baidu:jprotobuf-rpc-core-spring:3.5.3
|    |    +--- com.baidu:jprotobuf-rpc-core:3.5.3
|    |    |    +--- com.baidu:jprotobuf-rpc-core-test:3.5.3
|    |    |    |    +--- org.springframework:spring-web:3.2.11.RELEASE -> 4.2.7.RELEASE (*)
|    |    |    |    +--- junit:junit:4.10 -> 4.12
|    |    |    |    |    \--- org.hamcrest:hamcrest-core:1.3
|    |    |    |    +--- com.baidu:jprotobuf-jre:1.0.0

如果我在项目中引入com.baidu:jprotobuf:2.0.7 有可能会导致jar包冲突,2.0.7中的FieldType比jprotobuf-jre:1.0.0中多了MAP这个枚举

spring boot 打出来的jar包无法执行

使用 Spring Boot Initializr 创建的 1.3.0 Release POM 工程。只选取最简单的 aop 支持。并在 POM 文件中添加:

<dependency>
    <groupId>com.baidu</groupId>
    <artifactId>jprotobuf-rpc-core</artifactId>
    <version>3.2.4</version>
</dependency>
<dependency>
    <groupId>com.baidu</groupId>
    <artifactId>jprotobuf-rpc-core-spring</artifactId>
    <version>3.2.4</version>
</dependency>

在自动创建的DemoApplication.java 中添加如下配置代码:

@Bean
public RpcServiceExporter pbrpcServiceExporter() {
    List<Object> services = new ArrayList<>();

    // 直接import com.baidu.jprotobuf.pbrpc.EchoServiceImpl;
    services.add(new EchoServiceImpl()); 

    RpcServiceExporter exporter = new RpcServiceExporter();
    exporter.setServicePort(1031);
    exporter.setRegisterServices(services);
    exporter.setConnectTimeout(1000);

    return exporter;
}

打包执行:

mvn package
java -jar output/demo-0.0.1-SNAPSHOT.jar

异常摘要如下:

Compilation failed. class: com.baidu.bjf.remoting.protobuf.descriptor.FileDescriptorProtoPOJO$$JProtoBufClass
错误: 程序包com.google.protobuf.Descriptors不存在

升级到3.5.9还是不行,赶紧解决吧!

public class ID {
@protobuf
private String id;
}
升级了3.5.9,编译时生成的代理ID$$JProtoBufClass.class大小是0K
退回到以前的3.4.1,就没这个问题。
但是3.4.1有性能问题,之前已经提过issue了,跑一段时间之后必须重启,否则吃光CPU 。
哎,你们能不能好好测一下再发布?别砸了百度的牌子。

用jdk8报compiler is null maybe you are on JRE enviroment please change to JDK enviroment.

nInInitializerError
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1578)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:772)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:538)
at org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:93)
at com.omniprimeinc.componets.id.Appliaction.main(Appliaction.java:8)
Caused by: java.lang.ExceptionInInitializerError
at com.baidu.bjf.remoting.protobuf.ProtobufProxy.doCreate(ProtobufProxy.java:249)
at com.baidu.bjf.remoting.protobuf.ProtobufProxy.create(ProtobufProxy.java:159)
at com.baidu.bjf.remoting.protobuf.ProtobufProxy.create(ProtobufProxy.java:126)
at com.baidu.bjf.remoting.protobuf.CodedConstant.(CodedConstant.java:64)
at com.baidu.bjf.remoting.protobuf.CodeGenerator.getAccessByField(CodeGenerator.java:621)
at com.baidu.bjf.remoting.protobuf.CodeGenerator.getEncodeMethodCode(CodeGenerator.java:485)
at com.baidu.bjf.remoting.protobuf.CodeGenerator.getCode(CodeGenerator.java:201)
at com.baidu.bjf.remoting.protobuf.ProtobufProxy.doCreate(ProtobufProxy.java:224)
at com.baidu.bjf.remoting.protobuf.ProtobufProxy.create(ProtobufProxy.java:159)
at com.baidu.bjf.remoting.protobuf.ProtobufProxy.create(ProtobufProxy.java:126)
at com.baidu.jprotobuf.pbrpc.server.AnnotationRpcHandler.(AnnotationRpcHandler.java:57)
at com.baidu.jprotobuf.pbrpc.server.RpcServiceRegistry.doCreateRpcHandler(RpcServiceRegistry.java:119)
at com.baidu.jprotobuf.pbrpc.spring.RpcServiceRegistryBean.doCreateRpcHandler(RpcServiceRegistryBean.java:70)
at com.baidu.jprotobuf.pbrpc.server.RpcServiceRegistry.doRegiterService(RpcServiceRegistry.java:131)
at com.baidu.jprotobuf.pbrpc.server.RpcServiceRegistry.access$000(RpcServiceRegistry.java:41)
at com.baidu.jprotobuf.pbrpc.server.RpcServiceRegistry$1.doWith(RpcServiceRegistry.java:107)
at com.baidu.jprotobuf.pbrpc.utils.ReflectionUtils.doWithMethods(ReflectionUtils.java:64)
at com.baidu.jprotobuf.pbrpc.utils.ReflectionUtils.doWithMethods(ReflectionUtils.java:41)
at com.baidu.jprotobuf.pbrpc.server.RpcServiceRegistry.registerService(RpcServiceRegistry.java:101)
at com.baidu.jprotobuf.pbrpc.transport.RpcServer.registerService(RpcServer.java:172)
at com.baidu.jprotobuf.pbrpc.spring.RpcServiceExporter.afterPropertiesSet(RpcServiceExporter.java:195)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1637)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1574)
... 12 more
Caused by: java.lang.RuntimeException: compiler is null maybe you are on JRE enviroment please change to JDK enviroment.
at com.baidu.bjf.remoting.protobuf.utils.compiler.JdkCompiler.(JdkCompiler.java:84)
at com.baidu.bjf.remoting.protobuf.utils.JDKCompilerHelper.(JDKCompilerHelper.java:32)
... 35 more

请问是否支持其他语言调用

用Jprotobuf-rpc-socket做的服务端,用其他语言不知道如何调用。
用ProtobufIDLGenerator.getIDL()想获得服务的定义,好像不行啊。

异步化业务处理线程

我看现在的实现是基于注册表查找,然后同步的去调用handler.doHandle(request)进行的业务处理,如果业务耗时比较长,那么IO Thread就可以可能被长期占用,是否有计划把这里改为异步处理,通过回调的方式让结果回归到IOThread,从而在一部分时间让出IOThread的使用权,如果是Java 6或者7的话,可以利用利用Google Guava的ListenableFuture来实现这个目的

连接在60s之后断开,并未保持

做了一下测试:

在一个函数里面:

STEP 1: 第一次RPC调用,建立链接如下:
tcp6 0 0 :::18801 :::* LISTEN 31958/java
tcp6 0 0 127.0.0.1:18801 127.0.0.1:49116 ESTABLISHED 31958/java
tcp6 0 0 127.0.0.1:49116 127.0.0.1:18801 ESTABLISHED 3053/java

STEP2: 然后程序sleep(80s), 在大约sleep了60s之后观察链接:已经被客户端关闭?
tcp6 0 0 :::18801 :::* LISTEN 31958/java
tcp6 0 0 127.0.0.1:18801 127.0.0.1:49116 TIME_WAIT -

STEP3: 然后程序再次发起RPC请求,查看链接:重新建立了新的链接!
tcp6 0 0 :::18801 :::* LISTEN 31958/java
tcp6 0 0 127.0.0.1:18801 127.0.0.1:49126 ESTABLISHED 31958/java
tcp6 0 0 127.0.0.1:49126 127.0.0.1:18801 ESTABLISHED 3053/java
tcp6 0 0 127.0.0.1:18801 127.0.0.1:49116 TIME_WAIT -

我的理解是链接建立之后一直保持着呀,为什么60s后会断掉(貌似是被客户端关闭的)

基本功能实现

功能包括:

  • 【协议】实现的功能包括对基本协议的支持,但暂不支持压缩算法, 附件,验证和Chunk模式
  • 【客户端】集成Jprotobuf, 实现注释化开发支持, 暂不支持Google protobuf的RPC service生成代码方式
  • 【服务端】集成Jprotobuf, 实现注释化开发支持, 暂不支持Google protobuf的RPC service生成代码方式

不开启httpServer服务导致内存泄露问题

所以现在有问题了,ServerStatus 这个地方启动,当 httpServerPort 参数没有填的时候 即为 -1 的时候,他无法启动 ServerStatus 构造方法中的 消费者线程,导致了
public static final BlockingQueue ASYNC_REQUEST = new LinkeBlockingQueue<ServerStatus.RequestInfo>();
内存溢出!

当前版本 jprotobuf.version 过旧

当前版本 jprotobuf.version(jhunters/jprotobuf) 使用的版本:
<jprotobuf.version>1.10.2</jprotobuf.version>

ProtobufProxy.doCreate 中
long lastModify = ClassHelper.getLastModifyTime(cls);
代码过早(在缓存 CACHED 判断前)的调用赋值,导致几乎每次进入 doCreate 方法都会被调用,而该行代码非常耗cpu。

该问题在 1.10.5 中已经修复,能否升级:
https://github.com/jhunters/jprotobuf/releases/tag/jprotobuf-1.10.5

在特定网络操作环境下,是否有可能会出现服务端所在主机网络连接数暴涨的情况?

目前我有40台左右客户端电脑(每台电脑在周期性(10秒)断网后自动拨号上网操作,公网IP会不断发生变化),如果使用该组件构建客户端与服务端交互操作程序,因客户端程序在与服务端交互操作过程中总是会出现物理网络被强行断开的情况,会不会导致服务端无效连接暴涨,若不进行人工干预将最终撑爆主机连接数上限的情况?程序模块中有提供自动网络心跳和连接失效监测并自动关闭连接清除功能吗?

idle时间使用默认的60s,严格的按照每60s调用一次,偶发性发现服务连不上

idle时间使用默认的60s,严格的按照每60s调用一次,偶发性发现服务连不上。
报错如下:

java.lang.reflect.UndeclaredThrowableException: null
        at com.sun.proxy.$Proxy122.helloWorld(Unknown Source)
        at com.hf.process.web.listener.PmpRpcListener.monitorPmpRpcServer(PmpRpcListener.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
        at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException: null
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at com.baidu.jprotobuf.pbrpc.client.ha.HaProtobufRpcProxy.invoke(HaProtobufRpcProxy.java:375)
        at com.baidu.jprotobuf.pbrpc.spring.HaProtobufRpcProxyBean.invoke(HaProtobufRpcProxyBean.java:86)
        at com.baidu.jprotobuf.pbrpc.spring.HaRpcProxyFactoryBean.invoke(HaRpcProxyFactoryBean.java:179)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        ... 16 common frames omitted
Caused by: java.lang.RuntimeException: Unable to validate object
        at com.baidu.jprotobuf.pbrpc.transport.ChannelPool.getChannel(ChannelPool.java:85)
        at com.baidu.jprotobuf.pbrpc.transport.RpcChannel.getConnection(RpcChannel.java:72)
        at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy.invoke(ProtobufRpcProxy.java:453)
        at com.sun.proxy.$Proxy126.helloWorld(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at com.baidu.jprotobuf.pbrpc.client.ha.lb.LoadBalanceProxyFactoryBean.doInvoke(LoadBalanceProxyFactoryBean.java:496)
        at com.baidu.jprotobuf.pbrpc.client.ha.lb.LoadBalanceProxyFactoryBean.invokeWithMaxTry(LoadBalanceProxyFactoryBean.java:441)
        at com.baidu.jprotobuf.pbrpc.client.ha.lb.LoadBalanceProxyFactoryBean.invoke(LoadBalanceProxyFactoryBean.java:410)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        at com.sun.proxy.$Proxy122.helloWorld(Unknown Source)
        ... 25 common frames omitted
Caused by: java.util.NoSuchElementException: Unable to validate object
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:506)
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
        at com.baidu.jprotobuf.pbrpc.transport.ChannelPool.getChannel(ChannelPool.java:79)
        ... 38 common frames omitted

厉害了我的哥

hi,挺赞的东西,请问能否提供QQ群交流互动哈,更好更及时的反馈和交流,谢谢

io.netty.handler.timeout.ReadTimeoutException

2016-05-13 20:19:23.936 [nioEventLoopGroup-3-2] WARN io.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler's exceptionCaught() method while handling the following exception:
五月 13, 2016 8:19:23 下午 com.baidu.jprotobuf.pbrpc.transport.handler.RpcServerChannelIdleHandler userEventTriggered
io.netty.handler.timeout.ReadTimeoutException
警告: channel:[id: 0xab1d7593, /192.168.100.102:48107 => /192.168.100.200:8112] is time out./192.168.100.102:48107

经常出现这种情况,不知道有没有问题!

参数类型

你好,参数类型只支持自定义的类吗, 且这个类必须带有 @protobuf注解的属性。

两个问题

1.Jprotobuf与Protostuff有多大优势?
2.执行mvn clean install -Dmaven.test.skip=true命令时出现:
GPG Passphrase: *
我回车,就提示“*'gpg.exe' 不是内部或外部命令,也不是可运行的程序”,怎么解决?谢谢!

怎么关掉日志?

十二月 04, 2015 11:04:37 上午 com.baidu.bjf.remoting.protobuf.utils.compiler.JdkCompiler doCompile
信息: Begin to compile source code: class is 'com.baidu.bjf.remoting.protobuf.descriptor.FileDescriptorProtoPOJO$$JProtoBufClass'
十二月 04, 2015 11:04:41 上午 com.baidu.bjf.remoting.protobuf.utils.compiler.JdkCompiler doCompile
信息: compile source code done: class is 'com.baidu.bjf.remoting.protobuf.descriptor.FileDescriptorProtoPOJO$$JProtoBufClass'
十二月 04, 2015 11:04:41 上午 com.baidu.bjf.remoting.protobuf.utils.compiler.JdkCompiler doCompile
信息: loading class 'com.baidu.bjf.remoting.protobuf.descriptor.FileDescriptorProtoPOJO$$JProtoBufClass'
十二月 04, 2015 11:04:41 上午 com.baidu.bjf.remoting.protobuf.utils.compiler.JdkCompiler doCompile
信息: loading class done 'com.baidu.bjf.remoting.protobuf.descriptor.FileDescriptorProtoPOJO$$JProtoBufClass'
十二月 04, 2015 11:04:41 上午 com.baidu.bjf.remoting.protobuf.utils.compiler.JdkCompiler doCompile

怎么这些关掉日志?

ChannelPoolObjectFactory.wrap方法的实现问题

ChannelPoolObjectFactory的wrap方法应该是有问题,这里节选了一部分代码如下:

public PooledObject wrap(Connection obj) {
Connection connection = fetchConnection();

    // 中间代码省略....

    return new DefaultPooledObject<Connection>(connection);
}

wrap输入参数obj应该是已经调用ChannelPoolObjectFactory.create方法,在create方法里面已经调用fetchConnection, 为什么wrap里面还要再次fetchConnection,导致两次创建Connection.

jprotobuf打到spring boot jar中通过java -jar运行报错

[ WARN ] [2016-09-12 19:26:29] org.springframework.beans.factory.support.DefaultListableBeanFactory [1480] - Bean creation exception on FactoryBean type check: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'idGeneratorService' defined in URL [jar:file:/G:/payment-platform/payment-cpcn/payment-cpcn-server/payment-cpcn-server-1.0-SNAPSHOT.jar!/lib/id-generator-client-1.1.0-SNAPSHOT.jar!/id-generator-client.xml]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: Compilation failed. class: com.omniprimeinc.commonservice.idgen.redis.model.ID$$JProtoBufClass, diagnostics: [ID$$JProtoBufClass.java:2: 错误: 程序包com.google.protobuf不存在
import com.google.protobuf.;
^, ID$$JProtoBufClass.java:4: 错误: 程序包com.baidu.bjf.remoting.protobuf.utils不存在
import com.baidu.bjf.remoting.protobuf.utils.
;
^, ID$$JProtoBufClass.java:6: 错误: 程序包com.baidu.bjf.remoting.protobuf不存在
import com.baidu.bjf.remoting.protobuf.*;
^, ID$$JProtoBufClass.java:8: 错误: 找不到符号
import com.omniprimeinc.commonservice.idgen.redis.model.ID;
^
符号: 类 ID
位置: 程序包 com.omniprimeinc.commonservice.idgen.redis.model, ID$$JProtoBufClass.java:9: 错误: 程序包com.baidu.bjf.remoting.protobuf不存在
public class ID$$JProtoBufClass implements com.baidu.bjf.remoting.protobuf.Codec<com.omniprimeinc.commonservice.idgen.redis.model.ID> {
^, ID$$JProtoBufClass.java:9: 错误: 找不到符号
public class ID$$JProtoBufClass implements com.baidu.bjf.remoting.protobuf.Codec<com.omniprimeinc.commonservice.idgen.redis.model.ID> {
^
符号: 类 ID
位置: 程序包 com.omniprimeinc.commonservice.idgen.redis.model, ID$$JProtoBufClass.java:10: 错误: 程序包com.google.protobuf.Descriptors不存在
private com.google.protobuf.Descriptors.Descriptor descriptor;
^, ID$$JProtoBufClass.java:11: 错误: 找不到符号
public byte[] encode(com.omniprimeinc.commonservice.idgen.redis.model.ID t) throws IOException {
^
符号: 类 ID
位置: 程序包 com.omniprimeinc.commonservice.idgen.redis.model, ID$$JProtoBufClass.java:24: 错误: 找不到符号
public com.omniprimeinc.commonservice.idgen.redis.model.ID decode(byte[] bb) throws IOException {
^
符号: 类 ID
位置: 程序包 com.omniprimeinc.commonservice.idgen.redis.model, ID$$JProtoBufClass.java:47: 错误: 找不到符号
public int size(com.omniprimeinc.commonservice.idgen.redis.model.ID t) throws IOException {
^
符号: 类 ID
位置: 程序包 com.omniprimeinc.commonservice.idgen.redis.model, ID$$JProtoBufClass.java:57: 错误: 找不到符号
public void writeTo(com.omniprimeinc.commonservice.idgen.redis.model.ID t, CodedOutputStream output) throws IOException {
^
符号: 类 ID
位置: 程序包 com.omniprimeinc.commonservice.idgen.redis.model, ID$$JProtoBufClass.java:57: 错误: 找不到符号
public void writeTo(com.omniprimeinc.commonservice.idgen.redis.model.ID t, CodedOutputStream output) throws IOException {
^
符号: 类 CodedOutputStream
位置: 类 com.omniprimeinc.commonservice.idgen.redis.model.ID$$JProtoBufClass, ID$$JProtoBufClass.java:64: 错误: 找不到符号
public com.omniprimeinc.commonservice.idgen.redis.model.ID readFrom(CodedInputStream input) throws IOException {
^
符号: 类 CodedInputStream
位置: 类 com.omniprimeinc.commonservice.idgen.redis.model.ID$$JProtoBufClass, ID$$JProtoBufClass.java:64: 错误: 找不到符号
public com.omniprimeinc.commonservice.idgen.redis.model.ID readFrom(CodedInputStream input) throws IOException {
^
符号: 类 ID
位置: 程序包 com.omniprimeinc.commonservice.idgen.redis.model, ID$$JProtoBufClass.java:86: 错误: 程序包com.google.protobuf.Descriptors不存在
public com.google.protobuf.Descriptors.Descriptor getDescriptor() throws IOException {
^]

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.