cinchapi / concourse Goto Github PK

While executing following case :
callA(record); find_or_add('name', 'concourse'); callB(record"); add('name', 'jeff', 2);

handleShortSyntax() convert find_or_add() to find_or_concourse.add()

ER : concourse.find_or_add()
AR : find_or_concourse.add()

Broken Links are as follows

https://github.com/cinchapi/concourse/blob/develop/concourse/README.md
http://concoursedb.com/guide/installation
http://concoursedb.com/guide/tutorial

We depend on the .maven file to tell Gradle the correct repo for archive uploading. Why don't we just add logic that checks the correct conditions in code:

maven {
          credentials {
            username "admin"
            password "password"
          }
          if(project.version.endsWith('-SNAPSHOT')) {
            url "http://xxx"
          } else {
            url "http://xxx"
          }
        }

Similar to what was done in f369df1, but it needs to happen for Map and Timestamp.

The Apache License from the root should be copied into each of the license dists. For java, this means putting it in the META-INF folder of the jar.

The NOTICE files for the drivers will probably be different than that which is included for the server, so each of the project directories should have a separate notice file, if necessary.

Though, it might not be necessary to include the NOTICE or the jars since dependencies are not being bundled.

For the get/select records (and maybe others) we should specify the size of the maps we're using where possible. This means replacing stuff like

Map<Long, Map<String, Set<TObject>>> result = Maps.newLinkedHashMap();

with

Map<Long, Map<String, Set<TObject>>> result = TMaps.newLinkedHashMapWithCapacity(records.size());

Same thing for Sets, if possible.

diff "1452270300424000", time("last week")

seems to cause an infinite loop

and use G1GC..its time 👍

For example, concourse-driver-php has an init-repo.sh script. This should be called from init.sh.

[default/cash]$ add "baz", 10, 1
Returned 'true' in 0.011 sec

[default/cash]$ add "baz", 9, 1
Returned 'true' in 0.012 sec

[default/cash]$ add "baz", 9, 2
Returned 'true' in 0.01 sec

[default/cash]$ add "baz", 10, 2
Returned 'true' in 0.01 sec

[default/cash]$ select([1L, 2L])
Returned '
+------------------+
| Record | baz     |
+------------------+
| 1      | [9, 10] |
| 2      | [9, 10] |
+------------------+
' in 0.027 sec

I thought the order of the values was supposed to be insertion order? but it seems to place them in sort order? Not sure if this is incorrect or not, but we should figure out.

Interesting enough, it always seems to display links in descending order:

+---------------------------------------------------------------------------------------------------------------+
| Record           | bar | baz         | foo        | company    | direct_reports | name          | title | pi  | 
+---------------------------------------------------------------------------------------------------------------+
| 1447234653927000 | [1] | [true, @50] | [bar, baz] | null       | null           | null          | null  | null | 
| 1447234653927006 | null | null        | null       | [Cinchapi] | [@2, @1]       | [Jeff Nelson] | [CEO] | null | 
| 1447234653928002 | null | null        | null       | null       | null           | null          | null  | [3] | 
+---------------------------------------------------------------------------------------------------------------+

https://github.com/cinchapi/concourse/blob/develop/update-fork.sh

Only perform git stash pop if git stash actually stores something. We'll know if something was stashed if the output from the command isn't equal to No local changes to save

def data = [
    ['name':'John Doe',
    'department': 'Engineering',
    'title': 'Senior Software Engineer',
    'role': 'Software Engineer - Backend',
    'manager': Link.toWhere('title = Director of Engineering'),
    'salary': 10.00,
    'location': 'Atlanta',
    'exempt': true
    ]
]

def concourse = Concourse.connect();
println(concourse.select(concourse.insert(data)))

This yields

Caught: java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to com.google.common.collect.Multimap
java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to com.google.common.collect.Multimap
    at com.cinchapi.concourse.util.Convert.mapsToJson(Convert.java:207)
    at com.cinchapi.concourse.Concourse.insert(Concourse.java:1314)
    at com.cinchapi.concourse.Concourse$insert$0.call(Unknown Source)
    at quickstart.run(quickstart.groovy:20)

which happens because the Concourse API methods expect Multimap.

This will make debugging A LOT easier. Right now those logs get deleted when the temporary server is wiped away

Similar to CON-237

Need to profile what slow downs may (or may not) be occurring with G1GC (in Java 8...not Java 7) and optimize where necessary.

Add an option for the user to specify the header on the command line. This will help in cases when the header is not provided in the file but the user doesn't want to write custom code to provide the header.

In the implementation, ImportCli should set the header in each importer. Each importer should also check the first line of input against the header that was provided. If the first line of input is identical to the header, that line should be skipped during import. This is meant to help in cases where multiple files are being imported, but only one of them has the header. In that case, the user can simply specify the header on the command line.

In the doJsonify method, I didn't know if I should've used StringBuilder or StringBuffer.

[default/cash]$ select(["name", "salary", "title"], "role = Director and salary > 10")
Returned '
+--------------------------------------------------------------------+
| Record           | title                     | name       | salary |
+--------------------------------------------------------------------+
| 1448732216678007 | [Director of Engineering] | [Jane Doe] | [20.0] |
+--------------------------------------------------------------------+
' in 0.011 sec

The order of the columns should match the order of the input list. If that isn't possible, then it should be alphabetical.

http://blog.sokolenko.me/2014/11/javavm-options-production.html

These tests should mimic the related ones in AtomicOperationWorkflowTest

The Import framework allows people to specify a resolve key to find existing records where data should be placed. Now that the findOrInsert method exists, extend the Import framework to take advantage by supporting resolve criteria.

            catch (Exception e) {
                if(e.getCause() instanceof TTransportException) {
                    throw new ProgramCrash(e.getMessage());
                }
                else if(e.getCause() instanceof SecurityException) {
                    throw new ProgramCrash(
                            "A security change has occurred and your "
                                    + "session cannot continue");
                }
                else if(e instanceof CompilationFailedException) {
                    throw new MultiLineRequest(e.getMessage());
                }
                else if(e instanceof MissingMethodException
                        && script != null
                        && e.getMessage().contains(
                                "No signature of method: ConcourseShell.")) {
                    String method = e.getMessage().split("ConcourseShell.")[1]
                            .split("\\(")[0];
                    input = input.replaceAll(method, "ext." + method);
                    return evaluate(input);
                }
                else {
                    String message = e.getCause() instanceof ParseException ? e
                            .getCause().getMessage() : e.getMessage();
                    throw new EvaluationException("ERROR: " + message);
                }
            }

All the places where we do e instance of ... should just be caught directly and handled in a separate block.

1. Create a new user named "foo" with a password that is not "admin"
2. Add a concourse_client.prefs file to the user's home directory

username = foo

1. Try to connect to concourse shell

$ concourse shell
ERROR: Invalid username/password combination.

The issue here is that the logic for reading the prefs file always returns the default if its not present so it is returning "admin" as the password.

1. Exceptions are expensive to create. For whatever reason, we use Exceptions to control program flow in CaSH (though it kind of makes sense to do this in a REPL, so forgive the departure from convention). We don't need to create a new Exception object every time for things that are merely there to control flow. We should create a singleton, similar to what we do for BufferCapacityException and throw that instead.

I think we can do this for

• NewLineRequest
• MultiLineRequest - not sure this exception needs to take in a message?
• ExitRequest

But honestly, even for ones that do have state (i.e. a useful message and other attributes), we could use a singleton and just change the state whenever the exception needs to be thrown since CaSH is single threade

We had added timestamp with each TransactionToken precisely because we wanted to differentiate among different transactions that belonged to the same client session. However, it turns out that isn't really necessary because a client session can only have one transaction open at a time.

In actuality, we don't even need TransactionTokens anyway. We can just associate the AccessToken with the Transaction object and that should be sufficient. And, instead of sending over a possibly null TransactionToken for each RPC call to ConcourseServer, we could just send over a boolean that indicates whether the call should be routed to a transaction or not.

Gson does some special conversion for the built in java map that causes it to serialize Link as a JSON object.

Repro:

   @Test
    public void testMapWithLinkObjectToJson(){
        Map<String, Object> data = Maps.newHashMap();
        data.put("foo", Link.to(1));
        String json = Convert.mapToJson(data);
        Assert.assertEquals("{\"foo\":\"@1\"}", json);

    }

error:

org.junit.ComparisonFailure: expected:<{"foo":["@1"]}> but was:<{"foo":[{"record":1}]}>
    at org.junit.Assert.assertEquals(Assert.java:115)
    at org.junit.Assert.assertEquals(Assert.java:144)
    at com.cinchapi.concourse.util.ConvertTest.testMapWithLinkObjectToJson(ConvertTest.java:637)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

• allow transformer on command line of import cli by providing the path to a groovy script
• allow transformer on command line of import cli by providing the fully qualified class name of a java file and ensuring that the file is on the classpath
• allow specifying multiple transformers?

Right now the script is place in /usr/local/bin, which is cool, but a symlink should also be added to /etc/init.d. If we do add this symlink then we must be sure to remove it when the uninstall script is invoked.

Allow CCL statements to contain || for OR and &/&& for AND

Right now the initialization script for the upgrade framework runs as part of the server installation process. The purpose of this script is to place the schema version files in the data directories so it should not run before the user has had a chance to modify concourse.prefs to set the appropriate data directories.

The correct way to do this is for ConcourseServer to check the data directories for the schema files. If they don't exist, then ConcourseServer should spawn a separate JVM to run initialization.

Additionally, instead of putting this logic in ConcourseServer.java, we can add it to the concourse ctrl script to check for the presence of the schema files and run the initializer if necessary. I don't like this approach as much though because it requires the implementation of more one off pref parsing. Nevertheless it does have the benefit of making the JVM spawning/displaying console logging easier and more natural for the user.

Use vagrant to setup an env with PHP 5.6 to checkout the code and run the unit tests. This is necessary to ensure that the logic for handling 64-bit pack/unpack works across versions.

http://refspecs.linuxfoundation.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html

The force-reload action should be an alias for restart. In the future, it may actually reload the configuration.

The POM file that is generated for Java artifacts using the fully qualified version number ([X.Y.Z.B[-SNAPSHOT]) instead of the correct X.Y.Z[-SNAPSHOT] version number that conforms to what is used for the pom.version

The ete test framework should use the skip-integration flag when installing the managed concourse server. Right now, the framework doesn't use the flag and employs a hack where it kills the installer process after 1 second in order to avoid hanging on waiting for the admin password.

Its possible the logic will need to check if the version >= 0.5.0 before trying to use the flag.

[default/cash]$ find_or_add "name", "jeff"
ERROR: No signature of method: org.cinchapi.concourse.Concourse$Client.find_or_add() is applicable for argument types: (java.lang.String, java.lang.String) values: [name, jeff]
Possible solutions: findOrAdd(java.lang.String, java.lang.Object), findOrAdd(java.lang.String, java.lang.Object)

[default/cash]$ findOrAdd "name", "jeff"

In CaSH, if compilation fails because of "No signature of method" and the method name has underscores, then convert the method name to camel case and try again

ConnectionPool registers its own shutdown hook in order to ensure that the connections are released when the JVM dies. This behaviour isn't well documented and some well to do user might register their own shutdown hook to do the same.

Shutdown hooks in Java are run concurrently, so there may be a case of two threads trying to shutdown the pool at the same time.

The easiest fix for this should to change ConnectionPool#353 to

if(open.compareAndSet(true, false)){

in order to ensure that the check is atomic and two threads won't miss updates from each other.

1. Better documentation
2. Move general logic in LineBasedImporter to Importer base class (e.g. transformValue(), validateFileFormat(), etc).
   ...For transformValue it'll have to be able to return something more generic than a JsonElement

The add, set and remove methods have a signature with order key, value, record
The link and unlink methods have a signature with order key, source, destination

it would probably be more intuitive to have the link signature be key, destination, source

Use https://github.com/diffplug/spotless

Has a few errors right now:

Users/curtis/git/concourse/concourse-server/src/main/java/jsr166e/ConcurrentHashMapV8.java:4663: error: spliterator() in KeySetView cannot implement spliterator() in Set
        public ConcurrentHashMapSpliterator<K> spliterator() {
                                               ^
  return type ConcurrentHashMapSpliterator<K> is not compatible with Spliterator<K>
  where K,E are type-variables:
    K extends Object declared in class KeySetView
    E extends Object declared in interface Set
/Users/curtis/git/concourse/concourse-server/src/main/java/jsr166e/ConcurrentHashMapV8.java:4729: error: spliterator() in ValuesView cannot implement spliterator() in Collection
        public ConcurrentHashMapSpliterator<V> spliterator() {
                                               ^
  return type ConcurrentHashMapSpliterator<V> is not compatible with Spliterator<V>
  where V,E are type-variables:
    V extends Object declared in class ValuesView
    E extends Object declared in interface Collection
/Users/curtis/git/concourse/concourse-server/src/main/java/jsr166e/ConcurrentHashMapV8.java:4824: error: spliterator() in EntrySetView cannot implement spliterator() in Set
        public ConcurrentHashMapSpliterator<Map.Entry<K, V>> spliterator() {
                                                             ^
  return type ConcurrentHashMapSpliterator<Entry<K,V>> is not compatible with Spliterator<Entry<K,V>>
  where K,V,E are type-variables:
    K extends Object declared in class EntrySetView
    V extends Object declared in class EntrySetView
    E extends Object declared in interface Set

Similar to the way the concourse-cli framework supports it

concourse.add("foo", 1L, 1);
concourse.find(Criteria.where().key("foo").operator(Operator.EQUALS).value(1)); // empty but expect [1]
concourse.find(Criteria.where().key("foo").operator(Operator.EQUALS).value(1L)); // empty but expect [1]

The data is being force added as a Long, but the query interprets the value as an int, regardless of whether we use 1 or 1L

Add logic to validate each line of the import file for conformance with the spec as it is being processed.

When writing code that uses transactions, applications need to keep track of whether the client they are using is in a transaction so that they don't accidentally start a new one before committing.

public void methodA() {
    concourse.stage();
    try {
        // do some work
        concourse.commit();
    }
    catch(TransactionException e) {
        concourse.abort();
    }
}

public void methodB() {
    concourse.stage();
    try {
        // do some work
        methodA();
        concourse.commit();
    }
    catch(TransactionException e) {
        concourse.abort();
    }
}

In the example above, the work that methodA() is done when called from methodB is an extension of the work methodB is doing, so the fact that methodB is already in a transaction should already be considered by the client when methodA is called

I can't think of a reason why the behaviour of calling stage when already in a transaction is to create a new transaction. I think its best to simply make it a no-op and continue with the current transaction, if it exists. This should be a small change.

Each client should just to see if the transaction token is not null, and if that is the case just return immediately.

We could implement this on the server if we edited the thrift API to take a transaction token in the stage method.

Instead of

                 Map<Long, Set<T>> pretty = PrettyLinkedHashMap
                            .newPrettyLinkedHashMap("Record", key);
                    for (Entry<Long, Set<TObject>> entry : raw.entrySet()) {
                        pretty.put(entry.getKey(), Transformers.transformSet(
                                entry.getValue(),
                                Conversions.<T> thriftToJavaCasted()));
                    }

should be doing something like

Map<Long, Set<T>> transformed = Maps.transformValues(raw,
                            TransformFunctions.set(Conversions
                                    .<T> thriftToJavaCasted()));

assuming TransformFunctions.set looks like

public static <V1, V2> Function<Set<V1>, Set<V2>> set(
            final Function<V1, V2> function) {
        return new Function<Set<V1>, Set<V2>>() {

            @Override
            public Set<V2> apply(Set<V1> input) {
                return Transformers.transformSet(input, function);
            }

        };
    }

Only challenge here is how do we transformed to be pretty lazily?

It'd be nice to be able to query on the record id

where foo = bar and $id$ != 123

This should be allowable since there is a reserved identifier for referring to the id of a record.