rjrudin / marklogic-java Goto Github PK

If a project depends on A and B in mlRestApi, A is being installed first because it's before B alphabetically. But if A depends on B - like if A installs a service that depends on a library in B - then installation of A's modules will fail.

We don't have this issue yet anywhere, but we'll need to figure something out for expressing an order for dependencies to be installed in.

The AppConfig instance has all the inputs needed for a default "watch" task:

task watch (type: JavaExec, dependsOn: 'classes') {
main = 'com.marklogic.client.configurer.ml7.ConfigurationFilesWatcher'
classpath = sourceSets.main.runtimeClasspath
args 'src/main/xqy,src/test/xqy', mlHost, mlAppConfig.restPort, mlUsername, mlPassword
}

At some point during mlDeploy, we need to clear the modules database - of course before mlLoadModules is called. Should be able to call mlInstall, then call mlClearModules.

This is a nice-to-have - when you modify the metadata file for a resource, you then have to touch the resource file in order for the metadata to be loaded.

As an ml-gradle user
I want mlConfigureApp to TITLE
So that CVS/SVN directories aren't loaded into my modules database

• Should be able to get away for now with ignoring folders with a name starting with "."

This is a nice-to-have - if the REST API throws an error when loading a module, mlWatch fails and must be restarted.

I've noticed gulp does the same thing while working on an Angular app, so this behavior doesn't seem that awful.

The default directory where the modules get loaded is under src/main/xqy. I have also been storing user interface code (html, css, js, imgs, etc) inside the modules database. But I don't want to store the ui code under src/main/xqy. I would rather put it in a second home directory that I can configure. For example I may want to put all my front end code in src/main/ui.

Allow for database packages to override elements with text in the /db:package-database/db:config/db:links block. I don't think we need to support overwriting the forests yet, haven't had a use case for that.

Should behave like AppConfig - confusing that the two are different right now.

DatabasePackageManager is doing an install, and so is InstallAppTask. Ought to move the code for installing a triggers database to DPM, then have one place where we check to see if the package needs to be installed.

Need to shift to an ExtensionLibraryDescriptor along with a way for a client to specify the permissions

Need to confirm that if no package files are registered with ml-gradle, installing a database and the rest of the app will work fine.

content-database-transform.xsl is dropping these right now

Should be more efficient than using JavaExec directly, since we know all the connection details.

RestApiConfigurer has a determineFormat function that is a big hacky if/else block for determining the format type of a module based on extension.

We should really allow for a properties file to define these mappings, thus allowing a developer to provide a custom mapping of extensions to formats. This would address issues where a developer has an .html file that is valid HTML - e.g. an unclosed link - but not valid XML, and thus the developer wants the .html file to have a format of TEXT.

Will allow for #33 to be easy to integrate into the test framework

This isn't being picked up yet in the merge process for database packages.

e.g. gradle mlClearContentDatabase -Pcollection=foo

http://docs.marklogic.com/REST/PUT/v1/config/transforms/%5Bname%5D allows for metadata to be specified. This would be helpful to document what a particular transform is used for.

Rob - I think a modification you made last Friday to get the zip file to install affected the mlInstallApp task

:mlPrepareRestApiDependencies
Found mlRestApi configuration, will unzip all of its dependencies to build/mlRestApi
Finished unzipping mlRestApi dependencies; will now include modules at [src/main/xqy]
:mlMergeDatabasePackages
No mergePackageFilePaths specified, will not produce a new database package file:mlMergeHttpServerPackages
No mergePackageFilePaths specified, will not produce a new HTTP server package file
:mlInstallApp FAILED

FAILURE: Build failed with an exception.

• What went wrong:
  Execution failed for task ':mlInstallApp'.

  Could not find property 'mlUsername' on root project 'geonames'.

• Try:
  Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 4.036 secs

An idea from Charles Greer - we should be able to speed up installation of lots of modules under /ext by zipping them up, then sending those to a new resource, which would then unzip the file and load them into the modules database.

This would require that that resource be installed first, but that shouldn't be hard to do.

Need a way to sleep after running mlUninstallApp, resuming once ML has finished restarting. Should be able to use /admin/v1/timestamp. Will need a GetTimestampTask that will probably populate a new timestamp property on ManageConfig. The sleep task will then loop on calls to that endpoint until it gets a newer timestamp.

This isn't urgent, but it's more elegant than sleeping for 6 or 7 seconds, which is fallible too.

Would be nice to have a task that generates a new transform, either xquery or xslt. Possible syntax:

gradle mlCreateTransform -PtransformType=xqy

Seems that it's only picking up ".xsl" transforms now. (DefaultConfigurationFilesFinder.java)

AppConfig and ManageConfig usually need to point to the same host, so having an mlHost property that they default to would avoid some duplication.

We have support for removing domains, need it as well for removing pipelines and domain configurations.

We need to market these gradle plugins to the World. We can submit plugins through the following website: http://plugins.gradle.org/submit

In this rollout, I suggest we rename the plugin to use the full MarkLogic name. (Yes, this is a marketing story).

ml-gradle = MarkLogic
ml-java = MarkLogic-Java
ml-junit = MarkLogic-JUnit

When I have mlWatch on, it would be nice if it just logged out an invalid xqy or xslt file as opposed to just failing. I have to restart everytime I save something that is invalid or not well formed.

Should have an mlReloadModules task that does the following:

• clears all modules (which needs to depend on mlDeleteLastConfigured)
• calls mlConfigureApp

This is useful for ensuring you have all the latest modules and no deleted ones without having to do a full deploy.

Support a /namespaces folder, where the filename minus the suffix is the name of the prefix. Should be able to support either format easily, just use the filename extension as the format.

Right now, when using mlMergeDatabase/HttpServer, you have to configure mlManageConfig with the location of the merged package files.

Should be possible for this plumbing to use a default location - e.g. /build/ml-gradle/merged-content-database.xml and merged-http-server.xml. Then a developer doesn't need to configure this on the mlManageConfig.

mlClearContent

Loading hundreds of modules via the REST API is slow. Should have an option for using MLCP to load them instead. Would need to update the timestamp that each module was loaded. But this should be straightforward since the /ext modules are loaded based on their directory structure.

Only issue may be ensuring ml-gradle can pull MLCP, as the Maven pom for that isn't correct, have to do this instead:

mlcp('com.marklogic:mlcp-Hadoop2:1.2-3') {
exclude module:"commons-modeler"
}
mlcp('commons-modeler:commons-modeler:2.0') {
transitive = false
}

An XccTask would simplify creating tasks for running ad hoc XQuery - e.g.

task sample (type: XccTask) {
xquery = "world"
}

This is much faster than the REST API endpoint when there are dozens or hundreds of modules.

Include users, roles, amps, CPF, alerting, etc. Maybe even a custom task.

mlClearModules - handy when some modules need to be deleted, since you can then do mlClearModules followed up by mlConfigureApp

If you want to use AbstractSpringTest, you need to provide some Spring configuration. It should be possible to have a basic @configuration class that does the following:

• sets up a DatabaseClientProvider
• sets an XccTemplate
• Reads properties for those two projects from a "test.properties" file. Ideally, this could pull config from the gradle file, since all the info we need - host, username/password, rest/xdbc ports - is in that config. If those are in gradle.properties in the root project dir, then it's easy. If they're not, I don't know if it's possible to read from the build.gradle file.

We already have standard props of mlUsername, mlPassword, and mlHost. Can add mlRestPort and mlXdbcPort, which AppConfig could look for. This generic config class can then look for those properties in gradle.properties.

This would all be based on convention, but it could be easily overridden or just not used at all.