Git Product home page Git Product logo

springbatch-neo4j-adapter's Introduction

Java CI codecov Download

Description

Provides Spring-batch JobRepository and related metadata persistence in Neo4J.

Purpose

Currently, the support of NoSQL databases in Spring-Batch is limited to ItemReader/ItemWriter for Neo4j and Mongo. This means that to persist state of Spring-Batch job you would need to have an RDMBS database even if you don't need it as there is no JobRepository persistence adapter available. See these open tickets:

This project aims to solve this problem for Neo4j. It offers ready to use Spring Batch metadata persistence adapter for the Neo4j database. This adapter allows Spring Batch to store its metadata directly in Neo4j so that no RDBMS/SQL database is needed. It is built on top of Spring-Data for entity and repository abstraction and on top of MapStruct to map Spring Batch data structures to Spring-Data Neo4j entities. This way, the amount of boilerplate code to write was kept at the minimum level.

Importing it to your project

Gradle

repositories {
    jcenter()
}

dependencies {
    implementation 'com.github.valb3r.springbatch.adapters:neo4j-adapter:0.1.4'
}

Maven

Add jcenter repository:

<repositories>
    <repository>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
        <id>jcenter</id>
        <name>bintray</name>
        <url>https://jcenter.bintray.com</url>
    </repository>
</repositories>

Import neo4j-adapter:

 <dependency>
    <groupId>com.github.valb3r.springbatch.adapters</groupId>
    <artifactId>neo4j-adapter</artifactId>
    <version>0.1.4</version>
</dependency>

Artifacts repository:

See Bintray repository: https://bintray.com/valb3r/maven/neo4j-adapter.

Spring-Batch DB structure on Neo4j

DB structure

Examples

All examples can be found inside examples folder.

Configuring

This is the example of how to enable Spring-Batch Neo4j adapter

Example:Enable Neo4j adapter

@EnableBatchProcessing
@SpringBootApplication(
    // Spring-Batch includes this by default, disabling them
    exclude = {
        DataSourceAutoConfiguration.class,
        DataSourceTransactionManagerAutoConfiguration.class
    }
)
@EnableSpringBatchNeo4jAdapter
public class ExampleApplication {

    public static void main(String[] args) {
        SpringApplication.run(ExampleApplication.class);
    }
}

After you have enabled Neo4j adapter with @EnableSpringBatchNeo4jAdapter you can use Spring-Batch as if it was RDBMS database. See this snippet for example:

Example:Execute simple batch job

@Slf4j
@Service
@RequiredArgsConstructor
public class SimpleJobService {

    @Getter
    private final AtomicReference<String> result = new AtomicReference<>();

    private final JobRepository jobRepository;
    private final JobBuilderFactory jobBuilderFactory;
    private final StepBuilderFactory stepBuilderFactory;

    @SneakyThrows
    public void runSimpleJob() {
        val job = jobBuilderFactory.get("FOO")
            .start(stepBuilderFactory.get("ONE").tasklet((a, b) -> {
                log.info("STEP ONE!");
                return null;
            }).build())
            .next(stepBuilderFactory.get("TWO").tasklet((a, b) -> {
                log.info("STEP TWO!");
                result.set("Step TWO DONE");
                return null;
            }).build())
            .build();

        val exec = jobRepository.createJobExecution("Test one", new JobParameters());
        job.execute(exec);
    }
}

Used by

You can see how to configure the adapter and how to use it in 'real' project here:

Connection configuration

The adapter will reuse spring.data.neo4j database connection properties if you provide them in i.e. application.yml. For example:

spring:
  # Run neo4j with docker:
  # docker run --rm -d --publish=7474:7474 --publish=7687:7687 --volume=$HOME/neo4j/data:/data -e NEO4J_AUTH=neo4j/docker neo4j/neo4j-experimental:4.0.0-rc01
  data:
    neo4j:
      uri: bolt://localhost:7687
      username: neo4j
      password: docker
      open-in-view: false
      use-native-types: true

All repositories are provided by:

@EnableNeo4jRepositories(
    basePackages = {
        "com.github.valb3r.springbatch.adapters.neo4j.ogm.repository"
    }
)

This way, all of them are configured using the same Spring-Data configuration as you have for your own Neo4j-repositories.

springbatch-neo4j-adapter's People

Contributors

valb3r avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar

springbatch-neo4j-adapter's Issues

A SessionFactory is required

Good afternoon,

I have this exception:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'writer' defined in file [batch/Writer.class]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: A SessionFactory is required
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1796) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:893) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143) ~[spring-boot-2.3.4.RELEASE.jar:2.3.4.RELEASE]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758) ~[spring-boot-2.3.4.RELEASE.jar:2.3.4.RELEASE]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750) ~[spring-boot-2.3.4.RELEASE.jar:2.3.4.RELEASE]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) ~[spring-boot-2.3.4.RELEASE.jar:2.3.4.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) ~[spring-boot-2.3.4.RELEASE.jar:2.3.4.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237) ~[spring-boot-2.3.4.RELEASE.jar:2.3.4.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) ~[spring-boot-2.3.4.RELEASE.jar:2.3.4.RELEASE]
	at BatchApplication.main(BatchApplication.java:26) ~[classes/:na]

Caused by: java.lang.IllegalStateException: A SessionFactory is required
	at org.springframework.util.Assert.state(Assert.java:76) ~[spring-core-5.2.9.RELEASE.jar:5.2.9.RELEASE]
	at org.springframework.batch.item.data.Neo4jItemWriter.afterPropertiesSet(Neo4jItemWriter.java:81) ~[spring-batch-infrastructure-4.2.4.RELEASE.jar:4.2.4.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1855) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1792) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	... 17 common frames omitted

Here is an extract from my POM.XML:

  <properties>
    <java.version>11</java.version>
    <spring-batch.version>2.3.4.RELEASE</spring-batch.version>
    <testcontainers.version>1.14.3</testcontainers.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-batch</artifactId>
      <version>${spring-batch.version}</version>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
      <version>${spring-batch.version}</version>
    </dependency>
    <dependency>
      <groupId>org.neo4j</groupId>
      <artifactId>neo4j-ogm-bolt-native-types</artifactId>
      <version>3.2.17</version>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-data-neo4j</artifactId>
      <version>${spring-batch.version}</version>
    </dependency>

    <dependency>
      <groupId>com.github.valb3r.springbatch.adapters</groupId>
      <artifactId>neo4j-adapter</artifactId>
      <version>0.1.4</version>
    </dependency>

    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-api</artifactId>
      <version>1.7.30</version>
    </dependency>
    <dependency>
      <groupId>javax.validation</groupId>
      <artifactId>validation-api</artifactId>
      <version>2.0.1.Final</version>
    </dependency>

    <dependency>
      <groupId>org.springframework.batch</groupId>
      <artifactId>spring-batch-test</artifactId>
      <version>4.2.4.RELEASE</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.testcontainers</groupId>
      <artifactId>junit-jupiter</artifactId>
      <version>${testcontainers.version}</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.testcontainers</groupId>
      <artifactId>neo4j</artifactId>
      <version>${testcontainers.version}</version>
      <scope>test</scope>
    </dependency>
  </dependencies>

You'll find my relevant Java code attached.

Can you please shed a light on what is wrong?

Thanks & Regards

Code.zip

Fix behaviour discrepancies due to attached ExectutionContext

There are minor issues with Neo4jJobExecutionDao that prevent findByJobInstanceId/getLastJobExecution/findRunningJobExecutions to get entities when there is no Neo4jStepExecution attached.
The issue seems to be minor as in reality it is hard to observe this situation as when job execution starts step execution starts too.

Add non-null validations and tests

Many spring-batch dao methods implementations have following validations:

Assert.notNull(jobExecution, "jobExecution cannot be null");
Assert.notNull(jobExecution.getJobId(), "JobExecution Job-Id cannot be null.");
Assert.notNull(jobExecution.getStatus(), "JobExecution status cannot be null.");
Assert.notNull(jobExecution.getCreateTime(), "JobExecution create time cannot be null");

Needs, to re-implement these in current project.

Allow to set ids on neo4j entites

Currently, ids are generated by neo4j and ones that are set directly are ignored. It doesn't seem to cause issues at this moment, but we need to investigate the behaviour and improve it.

Exception after batch launch => IllegalStateException: Failed to execute ApplicationRunner

Good evening,

I discovered your project this afternoon, so first: thanks a lot for this neat piece of code.

I have a first version of my code working, but * after * the batch has run, I have this stacktrace:

2020-10-05 17:50:47.476 ERROR 11864 --- [           main] o.s.boot.SpringApplication               : Application run failed

java.lang.IllegalStateException: Failed to execute ApplicationRunner
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:789) ~[spring-boot-2.3.4.RELEASE.jar:2.3.4.RELEASE]
	at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:776) ~[spring-boot-2.3.4.RELEASE.jar:2.3.4.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:322) ~[spring-boot-2.3.4.RELEASE.jar:2.3.4.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237) ~[spring-boot-2.3.4.RELEASE.jar:2.3.4.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) ~[spring-boot-2.3.4.RELEASE.jar:2.3.4.RELEASE]
	at blah.main(BlahApplication.java:26) ~[classes/:na]
Caused by: java.lang.NullPointerException: null
	at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:104) ~[spring-batch-core-4.2.4.RELEASE.jar:4.2.4.RELEASE]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127) ~[spring-batch-core-4.2.4.RELEASE.jar:4.2.4.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at com.sun.proxy.$Proxy78.run(Unknown Source) ~[na:na]
	at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.execute(JobLauncherApplicationRunner.java:199) ~[spring-boot-autoconfigure-2.3.4.RELEASE.jar:2.3.4.RELEASE]
	at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.executeLocalJobs(JobLauncherApplicationRunner.java:173) ~[spring-boot-autoconfigure-2.3.4.RELEASE.jar:2.3.4.RELEASE]
	at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.launchJobFromProperties(JobLauncherApplicationRunner.java:160) ~[spring-boot-autoconfigure-2.3.4.RELEASE.jar:2.3.4.RELEASE]
	at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.run(JobLauncherApplicationRunner.java:155) ~[spring-boot-autoconfigure-2.3.4.RELEASE.jar:2.3.4.RELEASE]
	at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.run(JobLauncherApplicationRunner.java:150) ~[spring-boot-autoconfigure-2.3.4.RELEASE.jar:2.3.4.RELEASE]
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:786) ~[spring-boot-2.3.4.RELEASE.jar:2.3.4.RELEASE]
	... 5 common frames omitted

Do you have any idea why this occurs, please? You'll find attached my main files.

Thanks in advance & Regards

neo4japp.zip

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.