It's not used and it has libraries that are constantly going stale and causing Github dependabot alerts.

The mechanics of ConstrainedPropertyDefTrait are causing problems. It turns out that adding ConstrainedPropertyDefTrait to a PropertyDefTrait using the 'as' keyword is super-neat, but also causes problems. See implementing traits at runtime. Keeping the original PropertyDefTrait object, which will be an Enum normally, is desirable.

Having PropertyDefTrait and ConstrainedPropertyDefTrait seemed cleaner, but since it is causing issues and is not really required, will move the functionality of ConstrainedPropertyDefTrait to PropertyDefTrait and call it a day.

Clean up and finalize what is currently in the main branch as v1. Existing applications should be compatible with v1.

We will be doing a major refactor/reorg for v2.

When interacting with an API that return JSON, it's not desirable to require shoe-horning data into a DataTable. It might possible to have a trait that leverages Java Generics and Jackson to cache JSON serialized objects.

localBuildAndPublish was implemented because the default publishToMavelLocal was not re-publishing if the version was already present in the local repo, which was an issue during a dev cycle. @hjwilli suggested that -SNAPSHOT versions will overwrite.

• test that -SNAPSHOT works as hoped/expected
• remove localBuildAndPublish

The JsonVine model has proven to be a good one. Standardize Reaper methods so they follow the JsonVine pattern.

• similar builder methods
• de-dup reaper runs using reaper args similar to vine method

Probably will store argument hash in the vertex that is created when a reaper runs.

The core functionality has been replaced by vine and graph. Not all of the functionality has been migrated.... Tag the repository before removing the classes, so they can be easily round in the future.

Related to #20.

We can remove secondaryIdFieldMap prior to removing KeyType. Probably best to factor out into a distinct issue.

APOC is not working. Since the upgrade of Groovy, Tinkerpop, and Neo4j libraries, I cannot find an APOC library that will work. Tested the following versions:

• apoc-3.4.0.8-all
• apoc-3.5.0.15-all
• apoc-4.0.0.18-all
• apoc-4.1.0.2-all

APOC has been carved out of the Carnival Dockerfile.

Find an APOC library that works or verify that one does not exist.

Harmonize VertexDefTrait and EdgeDefTrait.

// having to call instance() is potentially non-intuitive

Pmbb.VX.PATIENT.instance().withProperties(Core.PX.NAME, 'Hayden').vertex(graph, g)

// doesn't matter
Pmbb.VX.PATIENT.withProperties(Core.PX.NAME, 'Hayden').create(graph, g)

// existing 0 or 1
Pmbb.VX.PATIENT.withProperties(Core.PX.NAME, 'Hayden').getOrCreate(graph, g)

// existing 1
Pmbb.VX.PATIENT.withProperties(Core.PX.NAME, 'Hayden').get(graph, g)

// maybe add find() that will return all pre-existing
// exists() returns true/false
// just have get() return the GraphTraversal tryNext() thing???
// get returns vertex or null?
// get returns an optional like thing?


Pmbb.EX.PARTICIPATED_IN_ENCOUNTER.relate(pV, encV, g)
Pmbb.EX.PARTICIPATED_IN_ENCOUNTER.instance().withProperties(Core.PX.THING, 'ada').from(pV).to(encV).create(graph, g)

Pmbb.EX.PARTICIPATED_IN_ENCOUNTER.withProperties(Core.PX.THING, 'ada').from(pV).to(encV).create(graph, g)
Pmbb.EX.PARTICIPATED_IN_ENCOUNTER.withProperties(Core.PX.THING, 'ada').from(pV).to(encV).getOrCreate(graph, g)
Pmbb.EX.PARTICIPATED_IN_ENCOUNTER.withProperties(Core.PX.THING, 'ada').from(pV).to(encV).get(graph, g)

Pmbb.EX.PARTICIPATED_IN_ENCOUNTER.withProperties(Core.PX.THING, 'ada').create(pV, encV, graph, g)


Core.PX.NAME.of(pV).isPresent()
Core.PX.NAME.valueOf(pV)

See carnival-core/src/test/groovy/carnival/core/util/MethodsHolderSpec.groovy. Without this change, we cannot extend GraphMethods classes add new methods or functionality, which could be useful.

For example, there could be an abstract GraphMethods class that contains a set of standardized methods to interact with a resource and also has an abstract method getServerUrl() that points to that resource. Concrete sub-classes could get the server URL from a configuration utility without having to re-implement the GraphMethod classes.

It would be nice to be able to inherit from GraphMethods classes.

GPars is no longer being maintained and has some very old dependencies. Java concurrency has come a long way. Our uses of GPars could be migrated to core Java (fairly) easily.

Add extensions to Tinkerpop classes GraphTraversal and Graph to work with VertexDef, EdgeDef, and PropertyDef.

For applications that have their own configuration machinery, it would be nice not to require a separate configuration file for Carnival configuration. Figure out how to provide the config data to Carnival directly so no separate file is required.

Scope carnival configuration within a top level element, which would enable integration of a carnival configuration in an existing application.yaml file that includes other application configuration.

Publishing to Github Packages is useful, but users need to set up proper github authentication even if they are just attempting to pull from a public repo. Also, due to the inability of deleting or updating public packages on github, we cannot easily maintain SNAPSHOT versions.

Investigate alternative OSS package repos that would more easily support these features.

Central Repository:
https://central.sonatype.org/
https://central.sonatype.org/pages/ossrh-guide.html
https://central.sonatype.org/pages/requirements.html

The method:
Map execute(...)

is a pain. The result generally does not mean much and is just boilerplate. Add a setResult() method that can be used inside execute() to set a result. Change signature to void execute(...).

I think it's mostly there. The Groovy code should look in only one place for the home setting. Why not just pick carnival.home. If it's an app, the plugin will set this value automatically. If it's framework development, build.gradle can set carnival.home from the environment.

Migrate v1 Vine/Method to the v2 JsonVine style. The two will be merged. The class JsonVine will be renamed to Vine as the code at that level is generic. At the VineMethod level, there will be JsonVineMethod and TabularVineMethod. This will be a multi-step migration. Should probably be a milestone rather than a single issue, but here we are.

See VertexDef, where one is created dynamically with no propertyDefs or anything. There would have to be a repository of created DynamicVertexDefs so they can be looked up to keep track of any properties they have. Maybe just get rid of the concept altogether.

Every time I compile any module in Windows, I see an error like this printed to STDOUT

Could not open/create prefs root node Software\JavaSoft\Prefs at root 0x80000002. Windows RegCreateKeyEx(...) returned error code 5.

The performance of the application does not seem to be in any way effected by this message.

`

It is currently the default in CachingVine.com. I'm not sure much thought went into it and it is seeming more annoying than useful... Change it?

Remove references to PMBB wherever possible/practical.

• org.pmbb to org.carnival
• references in tests

Migrate Reaper, so it follows the Vine pattern.

Basic Call

myReaper.method('doSomething').arguments(Map).call()

Vertices representing all executed reaper method processes

myReaper.method('doSomething').processes()

Vertices representing all reaper method processes given arguments

myReaper.method('doSomething').arguments(Map).processes()

Vertices representing all executed reaper method processes

myReaper.method('doSomething').processes().vertices()

Outputs to any reaper process regardless of arguments

myReaper.method('doSomething').outputs()

Outputs to reaper processes with given arguments

myReaper.method('doSomething').arguments(Map).outputs()

• basic "hello world" micronaut app
• add graph model
• add basic unit test
• enhancing docker/adding docker-compose (@hjwilli)
• basic readme in example app
• add link/documentation from the main carnival page

See https://github.com/pmbb-ibi/carnival-micronaut

When a reaper method is called, current behavior will create a reaper process vertex in the graph that has a label as per the reaper method, but does not contain any unique reference to the arguments that were used to call the reaper method. What this means is that the ensure() or optionallyRunSingletonProcess() methods are limited. Two calls to a reaper method with different arguments are viewed as the same call. So, the reaper method truly is a "singleton process" which is expected to be run only once.

optionallyRunSingletonProcess() methods can stay this way as they have "singleton" in the name. ensure() methods should be more flexible, taking the reaper method arguments into account so a reaper method can be called multiple times on different inputs.

There is probably deep design discussion to be had here.... Taking into account the reaper method and the arguments is nice, but it does not take into account the state of the graph. There is no guarantee that the same reaper method called with the same arguments at different times will have the same effect, which I suppose is the assumption here. Is that useful? Should we enable programmers to classify reaper methods and have different behavior of ensure() depending on the class? I do not know.

For now, making this change will at least bring us more in-line with vine behavior. There is no guarantee that a vine method, which queries a database, called with the same parameters at different times will have the same result. The database can change in that interval. Carnival vine caching makes the assumption that the programer is choosing to ignore any changes refreshing when they choose to do so by invalidating (deleting) cache files. Implementing a similar scheme at the reaper method level seems to make sense.

Add code coverage to current groovy reporting.
Generate aggregated code coverage report across all subproject
Integrate code coverage with travis and with coveralls.

For reference, see:
https://lkrnac.net/blog/2016/10/aggregate-test-coverage-report/
https://gist.github.com/aalmiray/e6f54aa4b3803be0bcac
csMACnz/coveralls.net#88

Update travis to build dynamic docs and move them to the gh-pages branch.

Add validation to the setter methods to check that the property is valid for the provided Vertex.

Travis is migrating support for open source projects from travis-ci.org to travis-ci.com, and will be closing travis-ci.org at the end of the year.

https://mailchi.mp/3d439eeb1098/travis-ciorg-is-moving-to-travis-cicom

• update dependency versions
• factor build properties into gradle.properties file
• update carnival version number?

The following cleanup can happen regardless of a potential refactor.

Vine

• Vine.writeToCsvFile does not seem to be used anywhere.
• Vine.readFromCsvFile(String) does not seem to be used anywhere.
• Vine.readFromCsvFile(File) does not seem to be used anywhere.

IterativeCsvWriter

IterativeCsvWriter does not seem to be used anywhere.

With GPars removed, it should be possible to upgrade a number of Carnival dependencies. We have fallen behind on many fronts, including Groovy and Tinkerpop.

KeyType was important for Carnival v0 when there were complex data operations that consumed and produced data tables. With the introduction of the graph in Carnival v1, these operations are done in the graph. KeyType is no longer central and should be removed for simplicity's sake.

There is overlap here with secondaryIdFieldMap of GenericDataTable and MappedDataTable. Like KeyType, the secondaryIdFieldMap was important for data table operations. Because secondaryIdFieldMap depends on KeyType, it is not possible to remove KeyType without removing secondaryIdFieldMap.

Currently test suites can take a long time to run, some of that time is due to the recommended way to clear a neo4j database is to stop the server and delete the files.

Research the feasibility and potential speed benefits of using a neo4j ImpermanentDatabase for testing.

See https://neo4j.com/docs/stable/tutorials-java-unit-testing.html

MappedDataTable in the base dataAdd(Map) method formats dates as per the default SqlUtils date format, which is not the same as the default SQLServer date format. Add an option to specify the data format to use so that the date strings in MappedDataTable match the default format of SQLServer enabling easy data upload via standard SQLServer clients.

We've moved past a single institution.

• TrackedProcess*
  ...

See https://help.github.com/en/github/managing-packages-with-github-packages/configuring-gradle-for-use-with-github-packages

There are currently two styles of vines: Json and MappedDataTable. Json vines have a single unified cache file which contains both the data and meta-data. MappedDataTable has two files, a Yaml meta file and a CSV data file. The cache related vine methods for these two styles of vines differ. Json cache is a bare File. MappedDataTable cache is a DataTableFiles object. Unify these two into a single CacheFiles (or similar) object that has standardized functionality.

Add a convenience method to instantiate a GraphMethod object by class. Currently, the method(String name) does so by name only.

find a free Jekyll theme that works.

Standardize the behavior so Reasoner and Reaper take the arguments into account when matching on a previous run.

The Vine, CachingVine, and VineMethod classes/interfaces/traits are muddled.... Clean things up.

• Move the basic call() mechanism from CachingVine to Vine. Currently all Vine has are static methods. The mechanism to call VineMethods (without caching) should be in Vine, so there is some meaning/benefit to extending the Vine class.
• Override the Vine call() method to hook into the caching machinery. Keeping CachingVine as a trait probably makes sense, so it can be added to the man branches of Vine subclasses as needed.
• Evaluate potential changes to VineMethod... Maybe it's ok to stay how it is for now.

Implement GenericDataTableVine* classes. Attempt to leverage inheritance/composition to share common code with MappedDataTableVine*.

Reasoners are very much like Reapers, except they can have inputs from the graph, whereas Reapers are expected not to read from the graph, only write to it.

ReasonerProcess

inputs() - returns the input vertices of the process

This method displays progress information ('current' of 'total' chunks complete). Total = 0 is valid and should not throw an exception.

For negative total value, display a warning instead of throwing an exception.

See https://github.com/pmbb-ibi/carnival/blob/aae4500acaa2f150832812a9d2f6f63397961dc4/app/carnival-util/src/main/groovy/carnival/util/Log.groovy#L39-L42

Currently the Carnival Gradle plugin is about building an Carnival Micronaut application. It would be helpful to have a plugin that just supplies the Carnival dependencies for library building.

• Move plugin carnival-gradle to carnival-micronaut-app
• Create a stripped down carnival-library plugin with just the dependencies

Currently, the methods that case bare strings as field name and identifier field values coerce the case of the bare string. Field names become upper case, identifier field values become lower case. This is probably the correct default behavior as it is safer in terms of detecting data collisions.

However, FeatureReport, which extends MappedDataTable, is restricted by this behavior. FeatureReport is useful as a class to create data exports that are intended to be imported into other systems. Case coercion is not desirable in this case.

Add a "case-sensitive" flag to DataTable with a default of false. When true, field names and identifier field values are not case-coerced. This would make the data set more dangerous of data integrity, but more flexible.

• get process vertices
• ensure

Travis tests are failing.