ssssssss-team / spider-flow Goto Github PK

View Code? Open in Web Editor NEW

9.1K 95.0 1.7K 3.3 MB

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

Home Page: https://www.spiderflow.org

License: MIT License

Dockerfile 0.01% Java 99.99%

spider crawler jsoup xpath web-spider webspider webcrawler web-crawler spider-flow

spider-flow's People

Contributors

Stargazers

Watchers

Forkers

u-simon dodolooking code-lmj xinghen0911 xiaodin1 kinglf domineer-long zfs990 mlzboy guyian lgb020 eastones smallblackbeans fofocx zhanglei nekolr oszhugc yuaotian iamprdyz wangmb faronsince2016 wuzhendong123 wangwox xiaolingis runbean lwllovewf2010 caizengjun web-with-ai hubiao7 fuxuzhi y00231236 ittoyxk bottleman khq007110 wxl200854 liuhangbiaoo gaoxs1985 jinshuchu panzhiping simpluson constantine008 teakong smilekes xuijanfeng tikazyq jq-k believems fillholetogether nemozhang1987 1046517444 linyimin0812 sumerzhang zhouzach justcoworker ybelongedtom yagamilight junneyang laashub-sda laashub-sua yephetsskys dkgee primillet melodyjerry smilemilk1992 kingking888 ltlyaoni pengjinfu 739568808 willgeek fuyu1993 fight1988 brock0624 imloama love-moon papalian billdegithub eatcodecat georgejzzz liangshiyuan blank-1 jokerandking cake654326 911steven robben766 ashwhx bestjex liumingning lewis888 erickwang microtrip dick318 zjy090 huyaauto wj2303221 tutu1617 luckyjady yk741715534 realhellosunsun charleskun linzhonghong

spider-flow's Issues

Dependency org.yaml:snakeyaml, leading to CVE problem

Hi, In /，there is a dependency org.yaml:snakeyaml:1.19 that calls the risk method.

CVE-2022-25857

The scope of this CVE affected version is ** [0,1.31)**

After further analysis, in this project, the main Api called is org.yaml.snakeyaml.composer.Composer: composeNode(org.yaml.snakeyaml.nodes.Node)Lorg.yaml.snakeyaml.nodes.Node;

Risk method repair link : GitHub

CVE Bug Invocation Path--

Path Length : 6

CVE Bug Invocation Path : 
org.spiderflow.model.SpiderLog: init(java.lang.String,java.lang.String,java.util.List)V /.m2/repository/org/springframework/boot/spring-boot-starter-quartz/2.0.7.RELEASE/spring-boot-starter-quartz-2.0.7.RELEASE.jar
org.yaml.snakeyaml.Yaml$1: next()Ljava.lang.Object; /.m2/repository/org/springframework/boot/spring-boot-starter-quartz/2.0.7.RELEASE/spring-boot-starter-quartz-2.0.7.RELEASE.jar
org.yaml.snakeyaml.constructor.BaseConstructor: getData()Ljava.lang.Object; /.m2/repository/org/springframework/boot/spring-boot-starter-quartz/2.0.7.RELEASE/spring-boot-starter-quartz-2.0.7.RELEASE.jar
org.yaml.snakeyaml.composer.Composer: getNode()Lorg.yaml.snakeyaml.nodes.Node; /.m2/repository/org/springframework/boot/spring-boot-starter-quartz/2.0.7.RELEASE/spring-boot-starter-quartz-2.0.7.RELEASE.jar
org.yaml.snakeyaml.composer.Composer: composeDocument()Lorg.yaml.snakeyaml.nodes.Node; /.m2/repository/org/springframework/boot/spring-boot-starter-quartz/2.0.7.RELEASE/spring-boot-starter-quartz-2.0.7.RELEASE.jar
org.yaml.snakeyaml.composer.Composer: composeNode(org.yaml.snakeyaml.nodes.Node)Lorg.yaml.snakeyaml.nodes.Node;

Dependency tree--

INFO] org.spiderflow:spider-flow:pom:0.5.0
[INFO] +- org.springframework.boot:spring-boot-starter-web:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-starter-logging:jar:2.0.7.RELEASE:compile
[INFO] |  |  |  +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] |  |  |  |  \- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] |  |  |  +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.10.0:compile
[INFO] |  |  |  |  \- org.apache.logging.log4j:log4j-api:jar:2.10.0:compile
[INFO] |  |  |  \- org.slf4j:jul-to-slf4j:jar:1.7.25:compile
[INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  |  \- org.yaml:snakeyaml:jar:1.19:runtime
[INFO] |  +- org.springframework.boot:spring-boot-starter-json:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.7:compile
[INFO] |  |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
[INFO] |  |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.9.7:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.7:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.9.7:compile
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.9.7:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-tomcat:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-core:jar:8.5.35:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-el:jar:8.5.35:compile
[INFO] |  |  \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:8.5.35:compile
[INFO] |  +- org.hibernate.validator:hibernate-validator:jar:6.0.13.Final:compile
[INFO] |  |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
[INFO] |  |  +- org.jboss.logging:jboss-logging:jar:3.3.2.Final:compile
[INFO] |  |  \- com.fasterxml:classmate:jar:1.3.4:compile
[INFO] |  +- org.springframework:spring-web:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-webmvc:jar:5.0.11.RELEASE:compile
[INFO] |     +- org.springframework:spring-aop:jar:5.0.11.RELEASE:compile
[INFO] |     \- org.springframework:spring-expression:jar:5.0.11.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-quartz:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-context-support:jar:5.0.11.RELEASE:compile
[INFO] |  +- org.springframework:spring-tx:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.quartz-scheduler:quartz:jar:2.3.0:compile
[INFO] |     \- com.mchange:mchange-commons-java:jar:0.2.11:compile
[INFO] +- org.springframework.boot:spring-boot-starter-mail:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-context:jar:5.0.11.RELEASE:compile
[INFO] |  \- com.sun.mail:javax.mail:jar:1.6.2:compile
[INFO] |     \- javax.activation:activation:jar:1.1:compile
[INFO] +- org.springframework:spring-jdbc:jar:5.0.11.RELEASE:compile
[INFO] |  +- org.springframework:spring-beans:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-core:jar:5.0.11.RELEASE:compile
[INFO] |     \- org.springframework:spring-jcl:jar:5.0.11.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-websocket:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-messaging:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-websocket:jar:5.0.11.RELEASE:compile
[INFO] +- com.baomidou:mybatis-plus-boot-starter:jar:3.1.0:compile
[INFO] |  +- com.baomidou:mybatis-plus:jar:3.1.0:compile
[INFO] |  |  \- com.baomidou:mybatis-plus-extension:jar:3.1.0:compile
[INFO] |  |     +- com.baomidou:mybatis-plus-core:jar:3.1.0:compile
[INFO] |  |     |  +- com.baomidou:mybatis-plus-annotation:jar:3.1.0:compile
[INFO] |  |     |  +- org.mybatis:mybatis:jar:3.5.0:compile
[INFO] |  |     |  \- com.github.jsqlparser:jsqlparser:jar:1.4:compile
[INFO] |  |     \- org.mybatis:mybatis-spring:jar:2.0.0:compile
[INFO] |  +- org.springframework.boot:spring-boot-autoconfigure:jar:2.0.7.RELEASE:compile
[INFO] |  \- org.springframework.boot:spring-boot-starter-jdbc:jar:2.0.7.RELEASE:compile
[INFO] |     \- com.zaxxer:HikariCP:jar:2.7.9:compile
[INFO] +- mysql:mysql-connector-java:jar:5.1.47:compile
[INFO] +- com.alibaba:fastjson:jar:1.2.83:compile
[INFO] +- com.alibaba:druid-spring-boot-starter:jar:1.1.16:compile
[INFO] |  +- com.alibaba:druid:jar:1.1.16:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO] +- com.alibaba:transmittable-thread-local:jar:2.11.5:compile
[INFO] +- org.apache.commons:commons-text:jar:1.6:compile
[INFO] |  \- org.apache.commons:commons-lang3:jar:3.7:compile
[INFO] +- org.apache.commons:commons-csv:jar:1.8:compile
[INFO] +- commons-io:commons-io:jar:2.7:compile
[INFO] +- commons-codec:commons-codec:jar:1.11:compile
[INFO] +- com.google.guava:guava:jar:28.2-jre:compile
[INFO] |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  +- org.checkerframework:checker-qual:jar:2.10.0:compile
[INFO] |  +- com.google.errorprone:error_prone_annotations:jar:2.3.4:compile
[INFO] |  \- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] +- org.jsoup:jsoup:jar:1.11.3:compile
[INFO] \- us.codecraft:xsoup:jar:0.3.1:compile
[INFO]    \- org.assertj:assertj-core:jar:3.9.1:compile
[INFO] 
[INFO] -------------------< org.spiderflow:spider-flow-api >-------------------
[INFO] Building spider-flow-api 0.5.0                                     [2/4]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) @ spider-flow-api ---
[INFO] org.spiderflow:spider-flow-api:jar:0.5.0
[INFO] +- org.springframework.boot:spring-boot-starter-web:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-starter-logging:jar:2.0.7.RELEASE:compile
[INFO] |  |  |  +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] |  |  |  |  \- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] |  |  |  +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.10.0:compile
[INFO] |  |  |  |  \- org.apache.logging.log4j:log4j-api:jar:2.10.0:compile
[INFO] |  |  |  \- org.slf4j:jul-to-slf4j:jar:1.7.25:compile
[INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  |  \- org.yaml:snakeyaml:jar:1.19:runtime
[INFO] |  +- org.springframework.boot:spring-boot-starter-json:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.7:compile
[INFO] |  |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
[INFO] |  |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.9.7:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.7:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.9.7:compile
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.9.7:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-tomcat:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-core:jar:8.5.35:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-el:jar:8.5.35:compile
[INFO] |  |  \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:8.5.35:compile
[INFO] |  +- org.hibernate.validator:hibernate-validator:jar:6.0.13.Final:compile
[INFO] |  |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
[INFO] |  |  +- org.jboss.logging:jboss-logging:jar:3.3.2.Final:compile
[INFO] |  |  \- com.fasterxml:classmate:jar:1.3.4:compile
[INFO] |  +- org.springframework:spring-web:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-webmvc:jar:5.0.11.RELEASE:compile
[INFO] |     +- org.springframework:spring-aop:jar:5.0.11.RELEASE:compile
[INFO] |     \- org.springframework:spring-expression:jar:5.0.11.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-quartz:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-context-support:jar:5.0.11.RELEASE:compile
[INFO] |  +- org.springframework:spring-tx:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.quartz-scheduler:quartz:jar:2.3.0:compile
[INFO] |     \- com.mchange:mchange-commons-java:jar:0.2.11:compile
[INFO] +- org.springframework.boot:spring-boot-starter-mail:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-context:jar:5.0.11.RELEASE:compile
[INFO] |  \- com.sun.mail:javax.mail:jar:1.6.2:compile
[INFO] |     \- javax.activation:activation:jar:1.1:compile
[INFO] +- org.springframework:spring-jdbc:jar:5.0.11.RELEASE:compile
[INFO] |  +- org.springframework:spring-beans:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-core:jar:5.0.11.RELEASE:compile
[INFO] |     \- org.springframework:spring-jcl:jar:5.0.11.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-websocket:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-messaging:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-websocket:jar:5.0.11.RELEASE:compile
[INFO] +- com.baomidou:mybatis-plus-boot-starter:jar:3.1.0:compile
[INFO] |  +- com.baomidou:mybatis-plus:jar:3.1.0:compile
[INFO] |  |  \- com.baomidou:mybatis-plus-extension:jar:3.1.0:compile
[INFO] |  |     +- com.baomidou:mybatis-plus-core:jar:3.1.0:compile
[INFO] |  |     |  +- com.baomidou:mybatis-plus-annotation:jar:3.1.0:compile
[INFO] |  |     |  +- org.mybatis:mybatis:jar:3.5.0:compile
[INFO] |  |     |  \- com.github.jsqlparser:jsqlparser:jar:1.4:compile
[INFO] |  |     \- org.mybatis:mybatis-spring:jar:2.0.0:compile
[INFO] |  +- org.springframework.boot:spring-boot-autoconfigure:jar:2.0.7.RELEASE:compile
[INFO] |  \- org.springframework.boot:spring-boot-starter-jdbc:jar:2.0.7.RELEASE:compile
[INFO] |     \- com.zaxxer:HikariCP:jar:2.7.9:compile
[INFO] +- mysql:mysql-connector-java:jar:5.1.47:compile
[INFO] +- com.alibaba:fastjson:jar:1.2.83:compile
[INFO] +- com.alibaba:druid-spring-boot-starter:jar:1.1.16:compile
[INFO] |  +- com.alibaba:druid:jar:1.1.16:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO] +- com.alibaba:transmittable-thread-local:jar:2.11.5:compile
[INFO] +- org.apache.commons:commons-text:jar:1.6:compile
[INFO] |  \- org.apache.commons:commons-lang3:jar:3.7:compile
[INFO] +- org.apache.commons:commons-csv:jar:1.8:compile
[INFO] +- commons-io:commons-io:jar:2.7:compile
[INFO] +- commons-codec:commons-codec:jar:1.11:compile
[INFO] +- com.google.guava:guava:jar:28.2-jre:compile
[INFO] |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  +- org.checkerframework:checker-qual:jar:2.10.0:compile
[INFO] |  +- com.google.errorprone:error_prone_annotations:jar:2.3.4:compile
[INFO] |  \- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] +- org.jsoup:jsoup:jar:1.11.3:compile
[INFO] \- us.codecraft:xsoup:jar:0.3.1:compile
[INFO]    \- org.assertj:assertj-core:jar:3.9.1:compile
[INFO] 
[INFO] ------------------< org.spiderflow:spider-flow-core >-------------------
[INFO] Building spider-flow-core 0.5.0                                    [3/4]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) @ spider-flow-core ---
[INFO] org.spiderflow:spider-flow-core:jar:0.5.0
[INFO] +- org.spiderflow:spider-flow-api:jar:0.5.0:compile
[INFO] +- org.springframework.boot:spring-boot-starter-web:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-starter-logging:jar:2.0.7.RELEASE:compile
[INFO] |  |  |  +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] |  |  |  |  \- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] |  |  |  +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.10.0:compile
[INFO] |  |  |  |  \- org.apache.logging.log4j:log4j-api:jar:2.10.0:compile
[INFO] |  |  |  \- org.slf4j:jul-to-slf4j:jar:1.7.25:compile
[INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  |  \- org.yaml:snakeyaml:jar:1.19:runtime
[INFO] |  +- org.springframework.boot:spring-boot-starter-json:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.7:compile
[INFO] |  |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
[INFO] |  |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.9.7:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.7:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.9.7:compile
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.9.7:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-tomcat:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-core:jar:8.5.35:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-el:jar:8.5.35:compile
[INFO] |  |  \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:8.5.35:compile
[INFO] |  +- org.hibernate.validator:hibernate-validator:jar:6.0.13.Final:compile
[INFO] |  |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
[INFO] |  |  +- org.jboss.logging:jboss-logging:jar:3.3.2.Final:compile
[INFO] |  |  \- com.fasterxml:classmate:jar:1.3.4:compile
[INFO] |  +- org.springframework:spring-web:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-webmvc:jar:5.0.11.RELEASE:compile
[INFO] |     +- org.springframework:spring-aop:jar:5.0.11.RELEASE:compile
[INFO] |     \- org.springframework:spring-expression:jar:5.0.11.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-quartz:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-context-support:jar:5.0.11.RELEASE:compile
[INFO] |  +- org.springframework:spring-tx:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.quartz-scheduler:quartz:jar:2.3.0:compile
[INFO] |     \- com.mchange:mchange-commons-java:jar:0.2.11:compile
[INFO] +- org.springframework.boot:spring-boot-starter-mail:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-context:jar:5.0.11.RELEASE:compile
[INFO] |  \- com.sun.mail:javax.mail:jar:1.6.2:compile
[INFO] |     \- javax.activation:activation:jar:1.1:compile
[INFO] +- org.springframework:spring-jdbc:jar:5.0.11.RELEASE:compile
[INFO] |  +- org.springframework:spring-beans:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-core:jar:5.0.11.RELEASE:compile
[INFO] |     \- org.springframework:spring-jcl:jar:5.0.11.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-websocket:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-messaging:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-websocket:jar:5.0.11.RELEASE:compile
[INFO] +- com.baomidou:mybatis-plus-boot-starter:jar:3.1.0:compile
[INFO] |  +- com.baomidou:mybatis-plus:jar:3.1.0:compile
[INFO] |  |  \- com.baomidou:mybatis-plus-extension:jar:3.1.0:compile
[INFO] |  |     +- com.baomidou:mybatis-plus-core:jar:3.1.0:compile
[INFO] |  |     |  +- com.baomidou:mybatis-plus-annotation:jar:3.1.0:compile
[INFO] |  |     |  +- org.mybatis:mybatis:jar:3.5.0:compile
[INFO] |  |     |  \- com.github.jsqlparser:jsqlparser:jar:1.4:compile
[INFO] |  |     \- org.mybatis:mybatis-spring:jar:2.0.0:compile
[INFO] |  +- org.springframework.boot:spring-boot-autoconfigure:jar:2.0.7.RELEASE:compile
[INFO] |  \- org.springframework.boot:spring-boot-starter-jdbc:jar:2.0.7.RELEASE:compile
[INFO] |     \- com.zaxxer:HikariCP:jar:2.7.9:compile
[INFO] +- mysql:mysql-connector-java:jar:5.1.47:compile
[INFO] +- com.alibaba:fastjson:jar:1.2.83:compile
[INFO] +- com.alibaba:druid-spring-boot-starter:jar:1.1.16:compile
[INFO] |  +- com.alibaba:druid:jar:1.1.16:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO] +- com.alibaba:transmittable-thread-local:jar:2.11.5:compile
[INFO] +- org.apache.commons:commons-text:jar:1.6:compile
[INFO] |  \- org.apache.commons:commons-lang3:jar:3.7:compile
[INFO] +- org.apache.commons:commons-csv:jar:1.8:compile
[INFO] +- commons-io:commons-io:jar:2.7:compile
[INFO] +- commons-codec:commons-codec:jar:1.11:compile
[INFO] +- com.google.guava:guava:jar:28.2-jre:compile
[INFO] |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  +- org.checkerframework:checker-qual:jar:2.10.0:compile
[INFO] |  +- com.google.errorprone:error_prone_annotations:jar:2.3.4:compile
[INFO] |  \- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] +- org.jsoup:jsoup:jar:1.11.3:compile
[INFO] \- us.codecraft:xsoup:jar:0.3.1:compile
[INFO]    \- org.assertj:assertj-core:jar:3.9.1:compile
[INFO] 
[INFO] -------------------< org.spiderflow:spider-flow-web >-------------------
[INFO] Building spider-flow-web 0.5.0                                     [4/4]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) @ spider-flow-web ---
[INFO] org.spiderflow:spider-flow-web:jar:0.5.0
[INFO] +- org.spiderflow:spider-flow-core:jar:0.5.0:compile
[INFO] |  \- org.spiderflow:spider-flow-api:jar:0.5.0:compile
[INFO] +- org.springframework.boot:spring-boot-starter-web:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-starter-logging:jar:2.0.7.RELEASE:compile
[INFO] |  |  |  +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] |  |  |  |  \- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] |  |  |  +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.10.0:compile
[INFO] |  |  |  |  \- org.apache.logging.log4j:log4j-api:jar:2.10.0:compile
[INFO] |  |  |  \- org.slf4j:jul-to-slf4j:jar:1.7.25:compile
[INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  |  \- org.yaml:snakeyaml:jar:1.19:runtime
[INFO] |  +- org.springframework.boot:spring-boot-starter-json:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.7:compile
[INFO] |  |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
[INFO] |  |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.9.7:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.7:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.9.7:compile
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.9.7:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-tomcat:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-core:jar:8.5.35:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-el:jar:8.5.35:compile
[INFO] |  |  \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:8.5.35:compile
[INFO] |  +- org.hibernate.validator:hibernate-validator:jar:6.0.13.Final:compile
[INFO] |  |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
[INFO] |  |  +- org.jboss.logging:jboss-logging:jar:3.3.2.Final:compile
[INFO] |  |  \- com.fasterxml:classmate:jar:1.3.4:compile
[INFO] |  +- org.springframework:spring-web:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-webmvc:jar:5.0.11.RELEASE:compile
[INFO] |     +- org.springframework:spring-aop:jar:5.0.11.RELEASE:compile
[INFO] |     \- org.springframework:spring-expression:jar:5.0.11.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-quartz:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-context-support:jar:5.0.11.RELEASE:compile
[INFO] |  +- org.springframework:spring-tx:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.quartz-scheduler:quartz:jar:2.3.0:compile
[INFO] |     \- com.mchange:mchange-commons-java:jar:0.2.11:compile
[INFO] +- org.springframework.boot:spring-boot-starter-mail:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-context:jar:5.0.11.RELEASE:compile
[INFO] |  \- com.sun.mail:javax.mail:jar:1.6.2:compile
[INFO] |     \- javax.activation:activation:jar:1.1:compile
[INFO] +- org.springframework:spring-jdbc:jar:5.0.11.RELEASE:compile
[INFO] |  +- org.springframework:spring-beans:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-core:jar:5.0.11.RELEASE:compile
[INFO] |     \- org.springframework:spring-jcl:jar:5.0.11.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-websocket:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-messaging:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-websocket:jar:5.0.11.RELEASE:compile
[INFO] +- com.baomidou:mybatis-plus-boot-starter:jar:3.1.0:compile
[INFO] |  +- com.baomidou:mybatis-plus:jar:3.1.0:compile
[INFO] |  |  \- com.baomidou:mybatis-plus-extension:jar:3.1.0:compile
[INFO] |  |     +- com.baomidou:mybatis-plus-core:jar:3.1.0:compile
[INFO] |  |     |  +- com.baomidou:mybatis-plus-annotation:jar:3.1.0:compile
[INFO] |  |     |  +- org.mybatis:mybatis:jar:3.5.0:compile
[INFO] |  |     |  \- com.github.jsqlparser:jsqlparser:jar:1.4:compile
[INFO] |  |     \- org.mybatis:mybatis-spring:jar:2.0.0:compile
[INFO] |  +- org.springframework.boot:spring-boot-autoconfigure:jar:2.0.7.RELEASE:compile
[INFO] |  \- org.springframework.boot:spring-boot-starter-jdbc:jar:2.0.7.RELEASE:compile
[INFO] |     \- com.zaxxer:HikariCP:jar:2.7.9:compile
[INFO] +- mysql:mysql-connector-java:jar:5.1.47:compile
[INFO] +- com.alibaba:fastjson:jar:1.2.83:compile
[INFO] +- com.alibaba:druid-spring-boot-starter:jar:1.1.16:compile
[INFO] |  +- com.alibaba:druid:jar:1.1.16:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO] +- com.alibaba:transmittable-thread-local:jar:2.11.5:compile
[INFO] +- org.apache.commons:commons-text:jar:1.6:compile
[INFO] |  \- org.apache.commons:commons-lang3:jar:3.7:compile
[INFO] +- org.apache.commons:commons-csv:jar:1.8:compile
[INFO] +- commons-io:commons-io:jar:2.7:compile
[INFO] +- commons-codec:commons-codec:jar:1.11:compile
[INFO] +- com.google.guava:guava:jar:28.2-jre:compile
[INFO] |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  +- org.checkerframework:checker-qual:jar:2.10.0:compile
[INFO] |  +- com.google.errorprone:error_prone_annotations:jar:2.3.4:compile
[INFO] |  \- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] +- org.jsoup:jsoup:jar:1.11.3:compile
[INFO] \- us.codecraft:xsoup:jar:0.3.1:compile
[INFO]    \- org.assertj:assertj-core:jar:3.9.1:compile

Suggested solutions:

Update dependency version

Thank you very much.

Linux下安装提示,jdk.nashorn.api.scripting does not exist,但是jdk是1.8的

[ERROR] /spider-flow/spider-flow-core/src/main/java/org/spiderflow/core/script/ScriptManager.java:[3,33] package jdk.nashorn.api.scripting does not exist

循环节点会打乱数据的顺序

顺序的数据

生成乱序的数据

爬虫
sp_flow.txt

这个问题是由多线程引起的，除了使用单线程执行外，还有没有别的办法让数据按顺序插入数据库吗

大佬,这个框架非常好用啊,简单易懂,看了更新日志有0.5.1,还有在维护吗

jdk.nashorn.api.scripting程序包不存在

How can i do some AJAX requests by selenium in spiderflow?

I don't know how to build AJAX requests by dragging and dropping
Bro, if you give me some examples, i will thank for your help really!

能不能跟magic-api一样，可以设置密码 security-config

Language file

Hi ! Would like to know if i can translate Spider Flow in english ? Looks like really cool tool but it's a bit confusing for me ^^'

Thanks !

增加Puppeteer这种方式来抓取动态页面数据,有啥好的方式集成进来没？

也就是：https://github.com/fanyong920/jvppeteer

通过插件的形式么？

在vscode上引入了nashorn包后，启动springboot显示找不到主类。求解决

org.jsoup.validateTLSCertificates函数不存在

maser分支的org.spiderflow.core.io.HttpRequest.java，中jsoup.version:1.15.3版本导致代码不兼容，要改成1.11.3版本，才能编译通过，connection没有validateTLSCertificates函数

@SuppressWarnings("deprecation")
public HttpRequest validateTLSCertificates(boolean value){
	this.connection.validateTLSCertificates(value);
	return this;
}

官网（包含文档）仓库

https://github.com/ssssssss-team/spider-flow-site

demo is invalid

http://demo.spiderflow.org/ gives "An error condition occurred while reading data from the network. Please retry your request."

这个不支持获取解析之后得html.比如天猫的价格信息.这个是获取不到的

文档仓库

https://www.spiderflow.org/ 这个地址挂了，有文档仓库啥的嘛

Hello @javamxd
Is there a way to translate Interface to other languages?
I like how the interface looks, and the project seems very interesting, buy I am unable to read the letters, so it would be nice if you made some translation to English.

啊啪啪啪

保存数据到数据库是否支持去重？

定时执行，抓取数据，然后保存到数据库，插入前需要校验下是否存在该内容，如果存在则忽略，否则会存在多条一样的数据。

已爬取网页的过滤功能

您好，不知道目前平台是否有过滤已爬取网页的功能？或者说这部分功能，需要自己单独写插件来支持？

当选择每隔几个小时执行一次时，会变成0-N小时，每隔一个小时执行一次

DockerFile构建失败

我使用Docker部署项目，编译DockerFile时出现缺少重要文件。

ADD ./spider-flow-web/target/spider-flow.jar ./

spider-flow-web目录下无target文件夹及文件。

麻烦可以上传一下吗？

本地运行没问题,发布出现问题. org.apache.catalina.LifecycleException

失败 - 上下文路径[/flow]下，应用程序无法启动
失败 - 发生异常[org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/flow]]]

奇绩创坛投资Reach Out

我是奇绩创坛投资研究团队的罗泽铭，主要关注软件生态和底层infra的投资机会。
现阶段在寻找优质的开源项目，对spider-flow 图形化爬虫挺感兴趣的。

无论是创业项目投资，还是类似基金会早期支持，都希望和你进一步聊一聊。

手机号/微信：18123892890

期待回信，祝生活愉快，谢谢。
罗泽铭

【背景信息】
奇绩创坛是一家加速器和创投基金，由前百度总裁兼COO陆奇博士创立，希望助力**技术驱动早期创新创业生态，前身是Y Combinator**。

我们过去一年投资了80多支初创团队，其中有不少的优秀开源项目，包括代码质量评估的思码逸 Merico、远程桌面控制工具 Rustdesk、面向产业IoT的开源开发平台 Shifu OS、云端研发工具链和协作平台 TeamCode等相关项目。他们大多都是从一个早期的开源项目开始的。

我们尤其希望支持优秀的开源项目找到商业化应用场景落地。技术团队创业，除了需要过硬的技术背景，在产品、团队、商业和战略等方面，同样存在许多潜在挑战。陆奇博士及核心团队会像联合创始人一样，帮助公司从零到一、稳健地走出第一步

大佬们这个自定义变量怎么用呀

能不能定义成数组或者list的格式

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.