alibaba / ageiport Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
父工程pom的maven.compiler是8,子工程pom的maven.compiler是17,能统一一下吗
1.测试类中的示例最终好像都是基于反射调用的,这样对于使用 spring 管理的 bean(比如使用 @resource 或者 @Autowired 修饰),反射构建执行时会报出空指针,因此需要对实现类做特殊处理,请问有比较优雅的使用示例嘛,求解
AGEIPort版本:0.0.9
问题:执行非MultiSheetExportProcessorTest以外的测试用例都会失败,原因NPE
错误堆栈:
17:35:41.685 [AGEI-MainWorker-t-2] DEBUG com.alibaba.excel.context.WriteContextImpl - Initialization 'WriteContextImpl' complete 2023-01-11 17:35:41.687 [AGEI-MainWorker-t-2] ERROR c.a.a.p.c.t.exporter.worker.ExportMainTaskWorker - StandaloneExportMainTaskWorker#doReduce failed, main:20230111173540-65bdcde1 java.lang.NullPointerException: Cannot invoke "java.util.Map.get(Object)" because "meta" is null at com.alibaba.ageiport.processor.core.file.excel.ExcelFileWriter.write(ExcelFileWriter.java:85) at com.alibaba.ageiport.processor.core.task.exporter.worker.ExportMainTaskWorker.doReduce(ExportMainTaskWorker.java:194) at com.alibaba.ageiport.processor.core.spi.task.factory.MainTaskWorker.run(MainTaskWorker.java:34) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at com.alibaba.ageiport.common.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:95) at com.alibaba.ageiport.common.concurrent.InterruptibleTask.run(InterruptibleTask.java:50) at com.alibaba.ageiport.common.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) 2023-01-11 17:35:41.688 [AGEI-MainWorker-t-2] INFO c.a.a.test.processor.core.TestMainTaskCallback - ---beforeError:20230111173540-65bdcde1 2023-01-11 17:35:41.690 [AGEI-MainWorker-t-2] INFO c.a.a.p.core.spi.task.monitor.TaskStageEvent - main:20230111173540-65bdcde1, stage:执行失败, cost:-ms 2023-01-11 17:35:41.690 [AGEI-MainWorker-t-2] INFO c.a.a.test.processor.core.TestMainTaskCallback - ---afterError:20230111173540-65bdcde1 2023-01-11 17:35:41.690 [AGEI-eb-local-t-2] INFO c.a.a.p.core.task.monitor.TaskProgressServiceImpl - update progress, main:20230111173540-65bdcde1, stage:执行失败 17:35:41.910 [AGEI-MainWorker-t-2] DEBUG com.alibaba.excel.context.WriteContextImpl - Finished write. 17:35:41.937 [main] INFO com.alibaba.ageiport.test.processor.core.TestHelper - getTaskProgress, percent:1.0, stageName:主任务执行失败
空指针原因:ExcelFileWriter执行write函数时未对meta进行判空或者初始化。只有需要多sheet页导出时,在实现ExportProcessor的group方法时才会设置meta数据,其余场景未设置meta的ExportProcessor在执行时都会由此问题。
修改建议:判空然后初始化meta
开发者,你好,我完全按照你的导出示例来跑,还是会报出:java.lang.NullPointerException: null at com.alibaba.ageiport.processor.core.spi.service.TaskServiceImpl.executeTask(TaskServiceImpl.java:41)
的错误,名称我都核对了,请问为啥呢?
AgeiPortOptions.Debug debug = new AgeiPortOptions.Debug();
这里的Debug是私有的 我本地做demo的时候会引用不到
version: 0.1.6
java.lang.IllegalStateException: No such extension com.alibaba.ageiport.processor.core.spi.task.callback.MainTaskCallback by name TestMainTaskCallback
代码片段
String url = ''http://ageiport-task-server.demo.xxxxx.com.cn";
AgeiPortOptions options = new AgeiPortOptions();
HttpTaskServerClientOptions clientOptions = new HttpTaskServerClientOptions();
clientOptions.setEndpoint(host);
clientOptions.setPort(80);
options.setTaskServerClientOptions(clientOptions);
AgeiPort ageiPort = AgeiPort.ageiPort(options);
调试出调用了
请求url http://ageiport-task-server.demo.xxxxx.com.cn:80/v1/CreateTaskSpecification
请求json:
{
"action": "CreateTaskSpecification",
"domain": "http://ageiport-task-server.demo.xxxxx.com.cn:80",
"responseClass": "com.alibaba.ageiport.task.server.model.CreateTaskSpecificationResponse",
"taskCode": "CSVExportProcessor",
"taskDesc": "",
"taskExecuteType": "STANDALONE",
"taskHandler": "com.alibaba.ageiport.test.processor.core.exporter.CSVExportProcessor",
"taskName": "CSVExportProcessor",
"taskType": "EXPORT",
"timestamp": 0,
"url": "/v1/CreateTaskSpecification",
"version": "v1"
}
task-server控制台报错
2023-07-04 23:57:29,986 ERROR [com.ali.age.tas.ser.htt.TaskSpecificationApiV1] (vert.x-eventloop-thread-1) TaskSpecificationApiV1#create failed, request:CreateTaskSpecificationRequest(taskCode=CSVExportProcessor, taskName=CSVExportProcessor, taskDesc=, taskType=EXPORT, taskExecuteType=STANDALONE, taskHandler=com.alibaba.ageiport.test.processor.core.exporter.CSVExportProcessor): javax.persistence.PersistenceException: org.hibernate.HibernateException: java.util.concurrent.CompletionException: org.hibernate.PropertyValueException: not-null property references a null or transient value : com.alibaba.ageiport.task.server.entity.TaskSpecificationEntity.app
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154)
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:181)
at org.hibernate.reactive.session.impl.ReactiveExceptionConverter.convert(ReactiveExceptionConverter.java:31)
at org.hibernate.reactive.session.impl.ReactiveSessionImpl.lambda$firePersist$18(ReactiveSessionImpl.java:685)
at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930)
at java.base/java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:946)
at java.base/java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2266)
数据库表都是空表,从文档中的建表语句创建的
能否增加配置,可以自定义文件导出位置
能否增加默认导出类型csv
项目起步阶段时,访问量和数据都不是很大,是否可先用内存模式,后续再改为使用http模式,部署task-server。会有什么隐患吗?
我们会有很多图片压缩成zip。服务端需要解压Zip,拿到一张张图片,服务将图片一张张上传到文件中心获得文件ID,根据规则解析图片名称,将文件ID按照要求存库。不知道这个框架能否导入含有图片的ZIP包
如何导出多sheet/多文件
请问这部分的监控是不是还没完善;
clone了master,pom文件报错,很多依赖找不到
比如某主节点有三个主任务在执行中,服务发版了,就会导致这三个任务都一直在运行中,未执行最终态闭环。数据库会留下很多垃圾数据,有全链路闭环的Demo吗?
version 0.0.1
java.lang.NullPointerException: null
at com.alibaba.ageiport.processor.core.spi.service.TaskServiceImpl.executeTask(TaskServiceImpl.java:41)
1、TestHelper中 assertWithFile和assertWithoutFile 方法中 GetTaskProgressParam 这个类有误
解决办法: 将其修改为TaskProgressParam 而不是GetTaskProgressParam
2、重点 扩展类找不到 文档中的描述
可使用如下方式测试扩展有无加载成功
@Slf4j
public class StandaloneExportProcessorTest {
@Test
public void test1() {
Set<Processor> instances = ExtensionLoader.getExtensionLoader(Processor.class).getSupportedExtensionInstances();
log.info(instances);
}
}
也可以查看打印的示例是否是自己的
3、导出测试类中报错
AgeiPortOptions.Debug debug = new AgeiPortOptions.debug();
#不存在这个属性。。
options.setDebug(debug);
解决办法:注释掉即可
3、依赖下载不到 (可能是网络原因?)
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>ageiport-processor-core</artifactId>
<version>0.0.1-SNAPSHOT</version>
<scope>compile</scope>
</dependency>
官方文件推荐如上 依赖无法下载
解决办法:源码构建 在主pom ageiport-parent 中注释掉maven的插件 maven-gpg-plugin 和 maven-javadoc-plugin (可选 不然可能遇到很多坑。。需要使用经验)
源码构建成功 install到本地 可直接使用
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>ageiport-processor-core</artifactId>
<version>0.0.4</version>
<scope>compile</scope>
</dependency>
4、示例无法跑起来 以及没有正确的配置步骤
遇到agei-server-task 未配置的问题
现在支持多sheet导入导出吗
ImportMainTaskWorker.java中doReduce()中对existView赋值逻辑有误,导致View数据, 无法生成文件
验证
前提:只有两个子任务
测试代码基于ClusterImportProcessorTest.java改造
public BizImportResult<View, Data> convertAndCheck(BizUser user, Query query, List<View> views) {
BizImportResultImpl<View, Data> result = new BizImportResultImpl<>();
List<Data> data = new ArrayList<>();
for (View view : views) {
Data datum = new Data();
datum.setId(view.getId());
datum.setName(view.getName());
datum.setGender(view.getGender());
data.add(datum);
}
result.setData(data);
ImportSubTaskContextImpl context = getContext();
String subTaskId = context.getSubTask().getSubTaskId();
String no = subTaskId.substring(subTaskId.lastIndexOf("_") + 1);
// 第一个子任务输出View
if ("1".equals(no)){
result.setView(views);
}else{
// 最后一个子任务没有输出View
result.setView(new ArrayList<>());
}
return result;
}
ageiport-task 有使用说明文档吗
相关代码 TaskProgressMonitorImpl.java
子任务执行失败是否需要执行合并操作
如题,请教集群搭建方式,以及与gitops如何整合?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.