marklogic / ml-javaclient-util Goto Github PK

Should deprecate XccAssetLoader - performance should be just fine with a DocumentWriteSet for loading assets via the REST API

This should replace using XCC

See #41 . This should have a significant impact, at least when 2 of those 3 take a little bit of time to load.

As a developer
I want to TITLE
So that I can separate out individual property values from the Spring Configuration resulting in cleaner code.

I came across this article which influenced my rationale.
https://www.petrikainulainen.net/programming/spring-framework/spring-from-the-trenches-injecting-property-values-into-configuration-beans/

I'm regularly getting this error now. Can we add logic to ignore .DS_Store? 'gradle test' breaks now in certain situations on my MBP with OS X Yosemite because there are invalid chars in files that reside under .DS_Store.

java.lang.RuntimeException: Unable to insert content at URI: /ext/.DS_Store; cause: Authorization failed for user 'obi-tester'.
    at com.rjrudin.marklogic.modulesloader.impl.XccAssetLoader.loadFile(XccAssetLoader.java:153)
    at com.rjrudin.marklogic.modulesloader.impl.XccAssetLoader.visitFile(XccAssetLoader.java:114)
    at com.rjrudin.marklogic.modulesloader.impl.XccAssetLoader.visitFile(XccAssetLoader.java:35)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:135)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:199)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:69)
    at java.nio.file.Files.walkFileTree(Files.java:2602)
    at java.nio.file.Files.walkFileTree(Files.java:2635)
...

https://github.com/marklogic-community/ml-javaclient-util/blob/master/src/main/java/com/marklogic/client/ext/modulesloader/ssl/SimpleX509TrustManager.java#L41

This method returns null which ends up in the hands of OkHttp and explodes in this code:

for (X509Certificate caCert : caCerts) {

The method should work around OkHttp not doing the null check by returning an empty array.

@rjrudin I'm working on a pull request that piggybacks on top of your latest addition of loading files from a jar. The code fix you supplied is not passing in any metadata about services and I'd like to be able to grab the service metadata from the sidecar XMLs like you do for regular service loading.

I have a fix in progress. I just wanted to start this ticket so you knew iI was working on it and to have a place to discuss anything that I might need guidance on.

Need an interface here instead of having to subclass RBW and override buildRunnable.

Created in support of marklogic/ml-app-deployer#121

And reference them from ml-gradle Wiki

http://docs.marklogic.com/guide/rest-dev/search#id_53303

Should disable this by default. Activate it in ml-app-deployer via e.g. mlVerifySearchOptions=true

Should be able to ditch the XCC stuff and load modules via 8000 and specify the modules database

Including the IntelliJ IDEA gradle plugin allows also to create project and module files easily.

Similar to loading modules, provide a way to load schemas from the filesystem into a schemas database.

Should allow for a command-line argument

In support of marklogic/ml-gradle#160

As a developer
I want to TITLE
So that I can handle the exception appropriately and continue to write the next batch.

I have ran into a Batch error where I have a conflicting update error (the batch of documents have two docs with the same URI). Unfortunately it causes the whole program to crash. I would like to take note and log the error but I would like the program to continue.

I believe it is just adding a throws statement on the BatchWriter.write() method.

As a developer
I need to TITLE
So that the correct mime-type is set when POST documents to MarkLogic REST API.

TempRestBatchWriter

RestBatchWriter

I need to call the DocumentManager.setContentFormat method offered in the ML Java Client 4.0 API. Once you uptick this library to the latest Java Client API, I would like to get the RestBatchWriter to call this method.

As an ml-gradle user
I want to TITLE
So that I have fine-grained control over the workspaces I import/export

• WorkspaceManager has a "String... workspaceNames" parameter that is currently ignored
• On export, the query for workspaces should constrain on any names passed in this array
• On import, the XML in each file should be parsed (DocumentBuilderFactory should suffice here) and the workspace name extracted. If the workspaceNames array has any values, only those workspaces should be imported (if the array is empty, import all workspaces).
• Automated testing will be covered in ml-app-deployer (I'll manually test it as part of merging this work into the dev branch)

When you specify a minimumFileTimestampToLoad for PropertiesModuleManager the comparison is:

if (minimumFileTimestampToLoad > 0 && file.lastModified() < minimumFileTimestampToLoad) {
  if (logger.isDebugEnabled()) {
    logger.debug(String.format("lastModified for file '%s' is %d, which is before the minimumFileTimestampToLoad of %d",
				    file.getAbsolutePath(), file.lastModified(), minimumFileTimestampToLoad));
  }
  return false;
}

This needs to be:

if (minimumFileTimestampToLoad > 0 && file.lastModified() <= minimumFileTimestampToLoad) {
  if (logger.isDebugEnabled()) {
    logger.debug(String.format("lastModified for file '%s' is %d, which is before the minimumFileTimestampToLoad of %d",
				    file.getAbsolutePath(), file.lastModified(), minimumFileTimestampToLoad));
  }
  return false;
}

Note the < becomes <=

This is to support a Roxy project, where most of the modules are stored under "src".

Need support for http://docs.marklogic.com/REST/PUT/v1/config/properties . Can use samplestack convention, which is a rest-properties.json file (or XML) in the root of the modules directory.

In support of marklogic/ml-gradle#158

See marklogic/ml-gradle#224

This bombed when trying to load the samplestack transforms

Will need to make it easy for ml-app-deployer to feed its set of properties and customTokens in

Need a default impl of LoadModulesFailureListener in DefaultModulesLoader

DefaultModuleTokenReplacer has

private String propertyPrefix = "@ml.";

which is causing the token replace to always append @ml. to tokens. That seems bad for the intended default behavior.

I think it should be more like:

private String propertyPrefix;

then in RoxyModuleTokenReplacer.java

    public RoxyModuleTokenReplacer() {
        super();
        setPropertyPrefix("@ml.");
        addPropertiesSource(new FilePropertiesSource(new File("deploy/default.properties")));
        addPropertiesSource(new FilePropertiesSource(new File("deploy/build.properties")));
        addPropertiesSource(new FilePropertiesSource(new File("deploy/local.properties")));
    }

If this is indeed the case then I can submit a PR.

Example: xcc://admin:admin###@localhost:8000/Documents

This will use TemporalDocumentManager instead of DocumentManager

A subclass may specify a different path.

I was debugging a long execution time of mlLoadModules and came across to this library. I didn't dive deep into the code, but it seems to me (and please correct me, if I'm wrong), by looking into DefaultModulesLoader.java, that certain things are sequential instead of parallel. Was there a good reason to do this?

As documented at https://github.com/marklogic/roxy/wiki/Working-with-the-REST-API , a typical Roxy app will have asset modules under "src" or perhaps "modules", while REST resources and query options are under rest-api/ext and rest-api/config. This is a bit different from the samplestack approach, where everything is under a single directory. So it'd be nice if the code for loading modules made it easy to specify where each kind of module can be found.

A DocumentWriteSet can be written with an optional Transaction. Need a way for a client to be able to inject a Transaction - perhaps a TransactionProvider interface.

DefaultModulesLoader doesn't have a way to listen for errors when loading REST API modules.

In addition to looking for .tdex and .tdej suffixes, load any schemas document with a URI starting with "/tde" as a TDE ruleset

This is to support a use case where modules are stored in a JAR file that's on the classpath. Can use Spring's PathMatchingResourcePathResolver to find resources and then load them via Spring's Resource interface.

Main use case is for ml-migration-starter and marklogic-spring-batch

Currently the module finder only supports files ending with ".xml", since MarkLogic server allows also to specify search options in JSON, it would be great to support it also.

This will be a Consumer that can be plugged into DMSDK's ExportListener.