Git Product home page Git Product logo

michael-simons / neo4j-migrations Goto Github PK

View Code? Open in Web Editor NEW
112.0 6.0 22.0 98 MB

Automated script runner aka "Migrations" for Neo4j. Inspired by Flyway.

Home Page: https://michael-simons.github.io/neo4j-migrations/

License: Apache License 2.0

Java 95.12% Groovy 0.15% Smarty 0.07% Shell 0.21% HTML 0.23% Cypher 3.86% JavaScript 0.35%
neo4j database versioning graph database-migrations database-refactoring

neo4j-migrations's Introduction

Neo4j-Migrations

Neo4j-Migrations is a database migration and refactoring tool that allows running Cypher scripts and programmatic refactorings in a controlled and repeatable fashion against one or more Neo4j database.

Neo4j-Migrations is a proud member of the Neo4j-Labs and maintained by authors of the Neo4j-Drivers- and SDN/OGM team.

badge measure?project=eu.michael simons measure?project=eu.michael simons badge

Introduction

Neo4j-Migrations are a set of tools to make your schema migrations as easy as possible. They provide a uniform way for applications, the command line and build tools alike to track, manage and apply changes to your database, in short: to refactor your database. The project is inspired to a large extent by FlywayDB, which is an awesome tool for migration of relational databases. Most things evolve around Cypher scripts, however the Core API of Neo4j-Migrations allows defining Java classes as migrations as well.

Neo4j-Migrations builds directly on top of the official Neo4j Java driver, supports Neo4j 3.5, Neo4j 4.1 to 4.4 and Neo4j 5, including enterprise features such as multidatabase support and impersonation.

The only dependencies are said driver and ClassGraph, the latter being used to find migrations on the classpath.

The history of migrations applied is stored as a subgraph in your database.

The graph will look like this:

chain of migrations

Downloads

Binary downloads for the CLI are available on our release page for each version. Maven artifacts are available on central under the following coordinates:

  • Core API: eu.michael-simons.neo4j:neo4j-migrations:2.11.0

  • Spring-Boot-Starter: eu.michael-simons.neo4j:neo4j-migrations-spring-boot-starter:2.11.0

  • Quarkus extension: eu.michael-simons.neo4j:neo4j-migrations-quarkus:2.11.0

  • Maven-Plugin: eu.michael-simons.neo4j:neo4j-migrations-maven-plugin:2.11.0

The quickest way to use the CLI is JBang. With JBang installed, just enter jbang neo4j-migrations@neo4j --help to get started. Other options include Homebrew and Zip bundles. Please checkout the full list of download options.

Compatibility

Neo4j-Migrations can be used against Neo4j 3.5, all Neo4j 4 versions from 4.1 up to 4.4 and Neo4j 5, including all current Neo4j-Aura versions.

Note
Neo4j 4.0 is only partially supported since version 2.0 due to the fact that Neo4j Java Driver does not support it fully anymore since 5.x. If you need to use this library with Neo4j 4.0 (which you shouldn’t, because 4.0 is out of support anyway), look at the latest 1.x version of Neo4j-Migrations. This does not affect new versions of the 4.x series, such as Neo4j 4.4, which is fully supported by this project!

The Core API and the JVM based version of the CLI module of Neo4j-Migrations requires at least Java 17 or higher since version 2.0. Neo4j-Migrations can safely be used on both the class- and module-path. Native binaries are provided for 64bit versions of macOS, Linux and Windows. The native binaries don’t require a JVM to be installed.

For a version compatible with JDK 8, check the 1.x releases. We still do maintain the latest minor, including support for older versions of Spring Boot (prio to Spring Boot 3). These are also the versions you should be using against Neo4j 4.0.

The older releases of Neo4j-Migrations are compiled with JDK 17 while targeting JDK 8. The Core API is provided as a Multi-Release-Jar in the older releases, providing a module-info.java for JDK 11 and higher, making it a good citizen on the Java module path as well.

While the CLI module actually does not require a JVM installed (it is a native binary, available for a Linux, macOS and Windows alike), some people might prefer a solution native to their ecosystem. The following projects serve the same purpose as Neo4j-Migrations and use the same graph, check-summing and versioning scheme as this project:

If you want to be listed here too, please reach out, and we can collaborate to ensure compatibility.

Manual

The complete manual is available here: michael-simons.github.io/neo4j-migrations. The API documentation for the core module is available here: Neo4j Migrations (Core) 2.11.0 API. The comprehensive set of system diagrams including build, module and class diagrams is automatically generated weekly.

Presentations and features

We try to promote this project as good as we can internally and externally while trying not to be too obtrusive. If you think it’s a good idea to talk about it at your conference, just ask. Happy to answer CfPs. We are grateful about the coverage, presentations and features so far:

CLI in a nutshell

cli demo

neo4j-migrations's People

Contributors

aalmiray avatar alexkarezin avatar allcontributors[bot] avatar atomfrede avatar bsideup avatar dependabot-preview[bot] avatar dependabot[bot] avatar ggrossetie avatar injectives avatar katya-dovgalets avatar meistermeier avatar michael-simons avatar seankilleen avatar shanon84 avatar szabopeter avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

neo4j-migrations's Issues

Add some documentation for the CLI + Cypher script approach

Thanks for creating this helpful tool! :)

My approach is to run the CLI version with only Cypher scripts, that will be fed from a repository and build to a Docker image within the CI/CD process. I was stumbling a bit until I figured out the correct usage and especially naming of the scripts. Before V001__SomeMigration.cypher I initially tried with 01-some-migration.cypher, V01_some_migration.cypher, etc. until I figured out the correct one.

I hope I used it correctly, but with the following naming and command it works well for me:

ls cyphers/*
cyphers/V001__SomeMigration.cypher
cyphers/V002__AnotherMigration.cypher

neo4j-migrations --password ... --location file:///tmp/neo4j/cyphers/ migrate

Would be helpful to add a paragraph or two about allowed file names, and such usage.

Latest 1.5.5 exception on JDK8

This is running in a Docker container with 8u212-jdk-alpine. This problem did not exist in 1.5.4

Performing migration
neo4j-migrations    | [jbang] Resolving dependencies...
neo4j-migrations    | [jbang] eu.michael-simons.neo4j:neo4j-migrations-cli:jar:1.5.5
neo4j-migrations    | Done
neo4j-migrations    | [jbang] Dependencies resolved
neo4j-migrations    | [jbang] Building jar...
neo4j-migrations    | Exception in thread "main" java.lang.UnsupportedClassVersionError: org/graalvm/nativeimage/ImageInfo has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
neo4j-migrations    | 	at java.lang.ClassLoader.defineClass1(Native Method)
neo4j-migrations    | 	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
neo4j-migrations    | 	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
neo4j-migrations    | 	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
neo4j-migrations    | 	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
neo4j-migrations    | 	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
neo4j-migrations    | 	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
neo4j-migrations    | 	at java.security.AccessController.doPrivileged(Native Method)
neo4j-migrations    | 	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
neo4j-migrations    | 	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
neo4j-migrations    | 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
neo4j-migrations    | 	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
neo4j-migrations    | 	at ac.simons.neo4j.migrations.cli.MigrationsCli.getConfig(MigrationsCli.java:222)
neo4j-migrations    | 	at ac.simons.neo4j.migrations.cli.ConnectedCommand.call(ConnectedCommand.java:58)
neo4j-migrations    | 	at ac.simons.neo4j.migrations.cli.ConnectedCommand.call(ConnectedCommand.java:40)
neo4j-migrations    | 	at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
neo4j-migrations    | 	at picocli.CommandLine.access$1300(CommandLine.java:145)
neo4j-migrations    | 	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2358)
neo4j-migrations    | 	at picocli.CommandLine$RunLast.handle(CommandLine.java:2352)
neo4j-migrations    | 	at picocli.CommandLine$RunLast.handle(CommandLine.java:2314)
neo4j-migrations    | 	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
neo4j-migrations    | 	at picocli.CommandLine$RunLast.execute(CommandLine.java:2316)
neo4j-migrations    | 	at picocli.CommandLine.execute(CommandLine.java:2078)
neo4j-migrations    | 	at ac.simons.neo4j.migrations.cli.MigrationsCli.main(MigrationsCli.java:110)
neo4j-migrations    | 	at neo4j_migrations.main(neo4j_migrations.java:8)

neo4j-migrations on Windows fails with invalid character in path

Microsoft Windows 10 Version 21h1 ( OS Build: 19043.1706)

Neo4j 4.4.7. validate connectivity via browser and with username password neo4j/football

unzipped https://github.com/michael-simons/neo4j-migrations/releases/download/1.7.0/neo4j-migrations-1.7.0-windows-x86_64.zip into

c:\customers

.migrations.properties at C:\customers\migrattions\neo4j-migrations-1.7.0-windows-x86_64\neo4j-migrations-1.7.0-windows-x86_64\bin contains

#Thu Jun 09 07:28:52 EDT 2022
autocrlf=false
transaction-mode=PER_MIGRATION
address=bolt\://localhost\:7687
validate-on-migrate=true
username=neo4j
password=football

running

C:\customers\migrattions\neo4j-migrations-1.7.0-windows-x86_64\neo4j-migrations-1.7.0-windows-x86_64\bin>neo4j-migrations.exe -v show-catalog

results in

Will search for Cypher scripts in "file://C:/customers/migrattions/neo4j-migrations-1.7.0-windows-x86_64/neo4j-migrations-1.7.0-windows-x86_64/bin/neo4j/migrations"
Statements will be applied in one transaction per migration
java.nio.file.InvalidPathException: Illegal character [:] in path at index 3: //C:/customers/migrattions/neo4j-migrations-1.7.0-windows-x86_64/neo4j-migrations-1.7.0-windows-x86_64/bin/neo4j/migrations
        at sun.nio.fs.WindowsPathParser.nextSlash(WindowsPathParser.java:212)
        at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:111)
        at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
        at sun.nio.fs.WindowsPath.parse(WindowsPath.java:92)
        at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:232)
        at java.nio.file.Path.of(Path.java:147)
        at java.nio.file.Paths.get(Paths.java:69)
        at ac.simons.neo4j.migrations.core.ResourceDiscoverer.scanFilesystemLocations(ResourceDiscoverer.java:148)
        at ac.simons.neo4j.migrations.core.ResourceDiscoverer.discover(ResourceDiscoverer.java:116)
        at ac.simons.neo4j.migrations.core.DiscoveryService.lambda$findCallbacks$7(DiscoveryService.java:143)
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
        at java.util.Collections$2.tryAdvance(Collections.java:4853)
        at java.util.Collections$2.forEachRemaining(Collections.java:4861)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
        at ac.simons.neo4j.migrations.core.DiscoveryService.findCallbacks(DiscoveryService.java:144)
        at ac.simons.neo4j.migrations.core.Migrations.getCallbacks(Migrations.java:117)
        at ac.simons.neo4j.migrations.core.Migrations.invokeCallbacks(Migrations.java:392)
        at ac.simons.neo4j.migrations.core.Migrations.executeWithinLock(Migrations.java:365)
        at ac.simons.neo4j.migrations.core.Migrations.getDatabaseCatalog(Migrations.java:341)
        at ac.simons.neo4j.migrations.cli.ShowCatalogCommand.withMigrations(ShowCatalogCommand.java:70)
        at ac.simons.neo4j.migrations.cli.ConnectedCommand.call(ConnectedCommand.java:68)
        at ac.simons.neo4j.migrations.cli.ConnectedCommand.call(ConnectedCommand.java:40)
        at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
        at picocli.CommandLine.access$1300(CommandLine.java:145)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2358)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2352)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2314)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
        at picocli.CommandLine$RunLast.execute(CommandLine.java:2316)
        at picocli.CommandLine.execute(CommandLine.java:2078)
        at ac.simons.neo4j.migrations.cli.MigrationsCli.main(MigrationsCli.java:110)

Reported version on Aura defaults to patch version 0

Migrations outputs 4.4.0 instead of 4.4-aura due to the fact that the server version gets parsed by the driver.

Also the same outcome can be seen in the browser.
image

Maybe we should do our own version parsing from the dbms.components() output.

Error when loading from Spring Boot resources

If I use Spring resource loader to determine URL to resource it will give:
url:jar:file:/pathtothe.jar!/BOOT-INF/classes!/neo4j/migrations/V001__firstMigration.cypher

When executing the migration an error is reported:

Could not read script file jar:file:/pathtothe.jar!/BOOT-INF/classes!/BOOT-INF/classes/neo4j/migrations/V001__firstMigration.cypher

Add autocrlf option.

See #232. The issue @AndreasBoehme describes appears only when dealing with files containing multiline statements. We do indeed checksum the line endings (but not other trailing and leading spaces), and thus CRLF will be different than LF separated files.

The problem can arise when Gits prominent feature autoctrlf is set to true for example: During checkout (and maybe build), LF will be converted to CRLF. Or, when the scripts only are managed by git.

The idea is to provide a autoctrlf option in Neo4j migrations itself too. It can be simpler than what git does, it needs to convert things "one way" only, when reading a migration (basically, the checkin part of what git does).

This option must be set to false by default, otherwise older migrations version will break on windows systems.

Migrations don't support Neo4j instances with anonymous access.

User management procedures go away when auth is disabled and thus migrations fail with

There is no procedure with the name `dbms.showCurrentUser` registered for this database instance. Please ensure you've spelled the procedure name correctly and that the procedure is properly deployed.

Mac OS line endings (single `\r`) are not treated correctly.

When executing DB-Migrations in different environments (Windows, Linux), the current systems' line-ending is taken into account of the checksum (i.e. \r\n vs \r).
When the DB-data is copied to another environment, validation fails due to different checksums (of the previously applied script and the same script with different endings in another environment).

Add a BOM project.

Versions should be managed in one place, regardless whether a submodule inherits from the parent or not.

Failed to load cypher migration from Spring Boot jar

Spring Boot 2.4.3, neo4j-migrations 0.1.2
The similar issue as #89.
Invocation of init method failed; nested exception is ac.simons.neo4j.migrations.core.MigrationsException: Could not read script file jar:file:/app/service.jar!/BOOT-INF/classes!/BOOT-INF/classes/neo4j/migrations/schema/V0001__init_constraints.cypher

Workaround - override classgraph version to 4.8.90.
Looks like the reason is the changes in classgraph-4.8.91.

Support for sbt?

I'm using the Neo4j Java lib in a Scala app, and Neo4j-Migrations looks like a really useful way to maintain constraints and such. However, my workflow is entirely in sbt, and I wonder how I might get this lib to work with that. Does anyone have any insight into that? Is anyone aware of a blog post somewhere that describes it?

Cheers!

Add info command.

The info command should print all applied migrations and their status.

Needed:

  • Data structure for information about migrations
  • Data structure for holding a list of above informations and additional database information
  • A service to orchestrate that
  • The command itself

Question: neo4j-migrations + SDN + Harness + Application Tests

Hi Michael,

I've been working with neo4j in Java environment for a few months now and I love the work you and your mates are doing. I already wrote some issues in the SDN project before and you (and your team) answer quickly.

I'm trying to add neo4j-migrations to my project to make changes automatically and don't forget them in the deployments 😅
The problem that I have is to integrate neo4j-migrations with the tests, both at the application and the service level.
I saw in this project that the tests are using TestContainers but I would like some similar with neo4j-harness.

The application.yml file has for the test profile:

spring.config.activate.on-profile: test
spring.neo4j.uri: bolt://localhost:6666

For the service tests I have the next configuration:

@DataNeo4jTest
@ActiveProfiles("test")
public class MyServiceTest {
    private final static String HOSTNAME = "localhost";

    private final static int PORT = 6666;

    protected static Neo4j embeddedDatabaseServer;

    protected static void initializeNeo4j() {
        initializeNeo4j(null);
    }

    protected static void initializeNeo4j(@Nullable String file) {
        try {
            var pluginDir = new File(ApocConfig.class.getProtectionDomain().getCodeSource().getLocation().toURI()).getParentFile().toPath();
            Neo4jBuilder neo4JBuilder = Neo4jBuilders
                    .newInProcessBuilder()
                    .withDisabledServer()
                    .withConfig(BoltConnector.listen_address, new SocketAddress(HOSTNAME, PORT))
                    .withConfig(GraphDatabaseSettings.plugin_dir, pluginDir)
                    .withConfig(GraphDatabaseSettings.procedure_unrestricted, List.of("apoc.*"));

            if (file != null) {
                var pathFixtureFile = Paths.get(AbstractNeo4jTestSupport.class.getClassLoader().getResource(file).getPath());
                neo4JBuilder.withFixture(pathFixtureFile);
            }

            embeddedDatabaseServer = neo4JBuilder.build();
        } catch (URISyntaxException e) {
            log.error("Error configuring Apoc Plugin", e);
        }
    }

    @BeforeAll
    public static void initDatabase() {
        initializeNeo4j();
    }

    @AfterAll
    static void closeNeo4j() {
        embeddedDatabaseServer.close();
    }

The tests work well but the migrations are never applied (there are no log traces).

For the application tests I have the next configuration:

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT, classes = Application.class)
public class MyControllerIT extends AbstractSpringContextAndNeo4jTestSupport {

AbstractSpringContextAndNeo4jTestSupport has the same previous configuration that the service tests but also RestAssured is configured using the random port number and custom Jackson mappings injected in the application context load.
The tests are ok as well but I have the next errors in the log:

16949 --- [           main] a.s.n.m.s.b.a.MigrationsInitializer      : Cannot apply Neo4j migrations, driver instance cannot reach any database.

org.neo4j.driver.exceptions.ServiceUnavailableException: Unable to connect to localhost:6666, ensure the database is running and that there is a working network connection to it.

I think neo4j-migrations is trying to connect to the database before the instance is running.

If I change the annotations in MyControllerIT to

@DataNeo4jTest
public class MyControllerIT extends AbstractSpringContextAndNeo4jTestSupport {

There are some necessary beans to configure RestAssured that are not injected (@Autowired doesn't work) and the tests crash.

Do you have some manual / tutorial where all of those pieces are combined? Thank you.

Upgrade to Java 17

This will happen with version 2.0 and includes:

  • Removing the Java 11 and Java 17 extra directories (compiling only for Java 17)
  • General improvements (text blocks, a couple of switch statements and the usual)
  • Upgrade to Graal 22.2
  • Add explicit tests for the module path (Turns out: I don't have too, stuff runs by default on mp, so 🤷 )
  • Test resources are patched into the main module, class graph finds them twice, tests fails (classgraph/classgraph#704, unsure if this should be seen as maven / buildtool, module path or class graph issue, decided to move the affected resources into a separate (maven) module instead of using <useModulePath>false</useModulePath>)

ZIP download link is broken

The download link to https://repo1.maven.org/maven2/eu/michael-simons/neo4j/neo4j-migrations/0.0.12/neo4j-migrations-0.0.12.zip in the readme is outdated and doesn't work anymore.

Apparently, the Maven repo has been migrated to repo.maven.apache.org, however, the following folder only contains the ZIP file: https://repo.maven.apache.org/maven2/eu/michael-simons/neo4j/neo4j-migrations/0.0.12/

Searching for the ZIP 0.0.12 archive via search.maven.org worked, but there is probably a better way to link it directly.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.