arcadedata / arcadedb Goto Github PK

ArcadeDB Multi-Model Database, one DBMS that supports SQL, Cypher, Gremlin, HTTP/JSON, MongoDB and Redis. ArcadeDB is a conceptual fork of OrientDB, the first Multi-Model DBMS. ArcadeDB supports Vector Embeddings.

Home Page: https://arcadedb.com

License: Apache License 2.0

Java 48.59% Dockerfile 0.01% Shell 0.05% Batchfile 0.04% Lua 0.01% HTML 0.27% CSS 1.75% JavaScript 49.24% SCSS 0.03% Groovy 0.01% Python 0.01%

database dbms multi-model graph document time-series key-value embedded docker kubernetes

arcadedb's Introduction

Multi Model DBMS Built for Extreme Performance

ArcadeDB is a Multi-Model DBMS (created originally as a fork from OrientDB Open Source project after the acquisition by SAP) with a brand-new engine made of Alien Technology, able to crunch millions of records per second on common hardware with the usage of minimal resources. ArcadeDB is written in LLJ: Low Level Java. It's still Java11+ but only using low level API to use advanced mechanical sympathy techniques and a reduced pressure of the Garbage Collector. It's highly optimized for extreme performance. Runs from a Raspberry Pi to multiple servers on the cloud.

ArcadeDB is fully transactional DBMS with support for ACID transactions, structured and unstructured data, native graph engine (no joins but links between records), full-text indexing, geospatial querying, and advanced security.

ArcadeDB supports the following models:

Graph Database (compatible with Neo4j Cypher, Apache Tinkerpop Gremlin and OrientDB SQL)
Document Database (compatible with the MongoDB driver + MongoDB queries and OrientDB SQL)
Key/Value (compatible with the Redis driver)
Search Engine
Time Series
Vector Embedding

ArcadeDB understands multiple languages:

SQL (from OrientDB SQL)
Neo4j Cypher (Open Cypher)
Apache Gremlin (Apache Tinkerpop v3.7.x)
GraphQL Language
MongoDB Query Language

ArcadeDB can be used as:

Embedded from any language on top of the Java Virtual Machine
Remotely by using HTTP/JSON
Remotely by using a Postgres driver (ArcadeDB implements Postgres Wire protocol)
Remotely by using a Redis driver (only a subset of the operations are implemented)
Remotely by using a MongoDB driver (only a subset of the operations are implemented)

For more information, see the documentation.

Getting started in 5 minutes

Start ArcadeDB Server with Docker:

docker run --rm -p 2480:2480 -p 2424:2424 \
           -e JAVA_OPTS="-Darcadedb.server.rootPassword=playwithdata -Darcadedb.server.defaultDatabases=Imported[root]{import:https://github.com/ArcadeData/arcadedb-datasets/raw/main/orientdb/OpenBeer.gz}" \
           arcadedata/arcadedb:latest

Now open your browser on http://localhost:2480 and play with ArcadeDB Studio and the imported OpenBeer database to find your favorite beer.

ArcadeDB is cloud-ready with Docker and Kubernetes support.

You can also download the latest release, unpack it on your local hard drive and start the server with bin/server.sh or bin/server.bat for Windows.

Community

Join our growing community around the world, for ideas, discussions and help regarding ArcadeDB.

Chat live with us on Discord
Follow us on Twitter
Connect with us on LinkedIn
or on Facebook
Questions tagged #arcadedb on Stack Overflow
View our official Blog

Security

For security issues kindly email us at [email protected] instead of posting a public issue on GitHub.

License

ArcadeDB is Free for any usage and licensed under the liberal Open Source Apache 2 license. If you need commercial support, or you need to have an issue fixed ASAP, check our GitHub Sponsor page on both Recurrent and One-Time tiers. All the sponsorship received will be distributed to the active contributors of this project.

Contributing

We would love for you to get involved with ArcadeDB project. If you wish to help, you can learn more about how you can contribute to this project in the contribution guide. The nightly builds of the repository head can be found here.

Have fun with data!

The ArcadeDB Team

TL;DR: ArcadeDB is an open-source multi-model NoSQL database systems.

arcadedb's People

Contributors

Stargazers

Watchers

arcadedb's Issues

build fails ( Java 14.0.1 )

ArcadeDB Version: trunk

JDK Version: JDK 14.0.1

OS: Mac OSX

Expected behavior

build should complete successfully

Actual behavior

failure building engine

Steps to reproduce

clone to local project folder
cd arcadedb
mvnw install

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project arcadedb-engine: Compilation failure: Compilation failure: 
[ERROR] [User Path]/projects/arcadedb/engine/src/main/java/com/arcadedb/database/async/DatabaseAsyncExecutorImpl.java:[376,34] reference to Record is ambiguous
[ERROR]   both interface com.arcadedb.database.Record in com.arcadedb.database and class java.lang.Record in java.lang match
[ERROR] [User Path]/projects/arcadedb/engine/src/main/java/com/arcadedb/serializer/BinarySerializer.java:[51,58] reference to Record is ambiguous
[ERROR]   both interface com.arcadedb.database.Record in com.arcadedb.database and class java.lang.Record in java.lang match
[ERROR] [User Path]/projects/arcadedb/engine/src/main/java/com/arcadedb/engine/Bucket.java:[84,33] reference to Record is ambiguous
[ERROR]   both interface com.arcadedb.database.Record in com.arcadedb.database and class java.lang.Record in java.lang match
[ERROR] [User Path]/projects/arcadedb/engine/src/main/java/com/arcadedb/engine/Bucket.java:[88,34] reference to Record is ambiguous
[ERROR]   both interface com.arcadedb.database.Record in com.arcadedb.database and class java.lang.Record in java.lang match
[ERROR] [User Path]/projects/arcadedb/engine/src/main/java/com/arcadedb/engine/Bucket.java:[192,19] reference to Record is ambiguous
[ERROR]   both interface com.arcadedb.database.Record in com.arcadedb.database and class java.lang.Record in java.lang match
[ERROR] [User Path]/projects/arcadedb/engine/src/main/java/com/arcadedb/engine/Bucket.java:[374,42] reference to Record is ambiguous
[ERROR]   both interface com.arcadedb.database.Record in com.arcadedb.database and class java.lang.Record in java.lang match
[ERROR] [User Path]/projects/arcadedb/engine/src/main/java/com/arcadedb/engine/Bucket.java:[452,46] reference to Record is ambiguous
[ERROR]   both interface com.arcadedb.database.Record in com.arcadedb.database and class java.lang.Record in java.lang match
[ERROR] [User Path]/projects/arcadedb/engine/src/main/java/com/arcadedb/database/async/DatabaseAsyncExecutor.java:[75,21] reference to Record is ambiguous
[ERROR]   both interface com.arcadedb.database.Record in com.arcadedb.database and class java.lang.Record in java.lang match
[ERROR] [User Path]/projects/arcadedb/engine/src/main/java/com/arcadedb/engine/BucketIterator.java:[32,49] reference to Record is ambiguous
[ERROR]   both interface com.arcadedb.database.Record in com.arcadedb.database and class java.lang.Record in java.lang match
[ERROR] [User Path]/projects/arcadedb/engine/src/main/java/com/arcadedb/engine/BucketIterator.java:[40,3] reference to Record is ambiguous
[ERROR]   both interface com.arcadedb.database.Record in com.arcadedb.database and class java.lang.Record in java.lang match
[ERROR] [User Path]/projects/arcadedb/engine/src/main/java/com/arcadedb/engine/BucketIterator.java:[126,10] reference to Record is ambiguous
[ERROR]   both interface com.arcadedb.database.Record in com.arcadedb.database and class java.lang.Record in java.lang match
[ERROR] [User Path]/projects/arcadedb/engine/src/main/java/com/arcadedb/schema/EmbeddedSchema.java:[335,23] reference to Record is ambiguous
[ERROR]   both interface com.arcadedb.database.Record in com.arcadedb.database and class java.lang.Record in java.lang match
[ERROR] [User Path]/projects/arcadedb/engine/src/main/java/com/arcadedb/graph/ImmutableVertex.java:[61,11] reference to Record is ambiguous
[ERROR]   both interface com.arcadedb.database.Record in com.arcadedb.database and class java.lang.Record in java.lang match
[ERROR] [User Path]/projects/arcadedb/engine/src/main/java/com/arcadedb/query/sql/function/misc/SQLFunctionEncode.java:[57,13] reference to Record is ambiguous
[ERROR]   both interface com.arcadedb.database.Record in com.arcadedb.database and class java.lang.Record in java.lang match
[ERROR] [User Path]/projects/arcadedb/engine/src/main/java/com/arcadedb/graph/ImmutableEdge.java:[61,11] reference to Record is ambiguous
[ERROR]   both interface com.arcadedb.database.Record in com.arcadedb.database and class java.lang.Record in java.lang match
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :arcadedb-engine

Engine: NPEs accessing rid when it is null

ArcadeDB Version: 21.9.1-SNAPSHOT

JDK Version: N/A

OS: N/A

I found at least two points where there is an explicit use of the rid variable when it is for sure null, because there is an explicit test on the null condition.

arcadedb/engine/src/main/java/com/arcadedb/query/sql/parser/Rid.java

Line 167 in bb85560

public PInteger getBucket() {

    public PInteger getBucket() {
        if (expression != null) {
            RID rid = toRecordId((Result) null, new BasicCommandContext());
            if (rid == null) {
                PInteger result = new PInteger(-1);
                result.setValue(rid.getBucketId());
                return result;
            }
        }
        return bucket;
    }

    public PInteger getPosition() {
        if (expression != null) {
            RID rid = toRecordId((Result) null, new BasicCommandContext());
            if (rid == null) {
                PInteger result = new PInteger(-1);
                result.setValue(rid.getPosition());
                return result;
            }
        }
        return position;
    }

Authentication failed on postgres driver

ArcadeDB Version: 21.10

JDK Version: Oracle 8

OS: Windows 10 (happens on docker as well)

Expected behavior

Expected postgres driver to authenticate correctly.

Actual behavior

Authentication failed. "FATAL: password authentication failed for user "root"

Steps to reproduce

Set root password to testing123
Run the following code (Kotlin but can use Java equivalent)

val props = Properties()
props.setProperty("user", "root")
props.setProperty("password", "testing123")
props.setProperty("ssl", "false")
DriverManager.getConnection("jdbc:postgresql://localhost/Database", props)

JetBrains "DataGrip/Database plugin for IDEs" requires configuration tweaks but mostly works.

ArcadeDB Version: v21.10.2-SNAPSHOT (main `a7e3074`)

JDK Version: azul-11

OS: MacOS Big Sur

(Reproed on docker 21.10.2-SNAPSHOT (build 8c0fb3d99b4153a1de33fa0b455a3d1cbf7efbfc/1634191813975/main) as well)

I just wanted to add some notes regarding my investigation around getting JetBrains' IDE tool's database support connecting properly.

By default, it will authenticate and correctly select the database. Any attempts to query result in ArcadeDB closing the connection immediately. Log output with debug = true in PostgresNetworkExecutor looks like this:

2021-10-14 13:18:15.534 SEVER [PostgresNetworkExecutor] <ArcadeDB_0> Postgres wrapper: Error on reading request: com.arcadedb.exception.CommandExecutionException: Unknown function name :version
2021-10-14 13:18:15.534 INFO  [PostgresNetworkExecutor] <ArcadeDB_0> PSQL: execute (portal=) (limit=0)-> select version() (thread=31)
2021-10-14 13:18:15.534 INFO  [PostgresNetworkExecutor] <ArcadeDB_0> PSQL:-> ready for query (Z - 5b) (thread=31)
2021-10-14 13:18:15.547 INFO  [PostgresNetworkExecutor] <ArcadeDB_0> PSQL: parse (portal=) -> SET application_name = 'IntelliJ IDEA 2021.2.2' (params=0) (thread=31)
2021-10-14 13:18:15.547 INFO  [PostgresNetworkExecutor] <ArcadeDB_0> PSQL:-> parse complete (1 - 4b) (thread=31)
2021-10-14 13:18:15.548 INFO  [PostgresNetworkExecutor] <ArcadeDB_0> PSQL: bind (portal=) ->  (thread=31)
2021-10-14 13:18:15.548 INFO  [PostgresNetworkExecutor] <ArcadeDB_0> PSQL:-> bind complete (2 - 4b) (thread=31)
2021-10-14 13:18:15.548 INFO  [PostgresNetworkExecutor] <ArcadeDB_0> PSQL: execute (portal=) (limit=1)-> SET application_name = 'IntelliJ IDEA 2021.2.2' (thread=31)
2021-10-14 13:18:15.548 INFO  [PostgresNetworkExecutor] <ArcadeDB_0> PSQL:-> empty query response (I - 4b) (thread=31)
2021-10-14 13:18:15.548 INFO  [PostgresNetworkExecutor] <ArcadeDB_0> PSQL:-> ready for query (Z - 5b) (thread=31)
2021-10-14 13:18:15.573 INFO  [PostgresNetworkExecutor] <ArcadeDB_0> PSQL: parse (portal=) ->  (params=0) (thread=31)
2021-10-14 13:18:15.574 INFO  [PostgresNetworkExecutor] <ArcadeDB_0> PSQL:-> bind complete (2 - 4b) (thread=31)
2021-10-14 13:18:15.578 SEVER [PostgresNetworkExecutor] <ArcadeDB_0> Unexpected message type '�' for message any

After poking around for a while, I found a setting buried in the connection properties advanced tab "perferQueryMode":

By default, this is set to extended. If you change it to simple, we get a bit further:

Now any queries will fail with the following error: ERROR: Syntax error on executing query: Encountered " <SET> "SET "" at line 1, column 1. because the driver JetBrains uses by default prepends queries with the following:

SET extra_float_digits = 3

ArcadeDb doesn't support this, and it is meaningless and might be due to the driver having trouble figuring out the pgsql version or something. Anyway, a Google led me to the next setting to tweak:

You must set assumeMinServerVersion exactly as shown: 9.0, and this will prevent the driver sending this extra SET command.

This did it, and I could run queries!

The introspection features fail, but harmlessly.

Anyway, not sure what actions there are to be taken. There might be a way for ArcadeDb to ask for "simple" queries and it could just ignore the extra_float_digits set, I guess? Maybe this should end up summarized in the documentation?

Hope this helps someone. :)

Embedded mode on Ubuntu doesn't start up

ArcadeDB Version: 21.10

JDK Version: OpenJDK 11

OS: Ubuntu Server 20.04

Expected behavior

Server starts up when application is run

Actual behavior

Caused by: com.arcadedb.exception.DatabaseOperationException: Error on creating new database instance
	at com.arcadedb.database.EmbeddedDatabase.openInternal(EmbeddedDatabase.java:233) ~[arcadedb-engine-21.9.1.jar!/:na]
	at com.arcadedb.database.EmbeddedDatabase.create(EmbeddedDatabase.java:183) ~[arcadedb-engine-21.9.1.jar!/:na]
	at com.arcadedb.database.DatabaseFactory.create(DatabaseFactory.java:79) ~[arcadedb-engine-21.9.1.jar!/:na]
	at com.example.CustomClass.<init>(CustomClass.kt:18) ~[classes!/:0.0.1-SNAPSHOT]
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:na]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[na:na]
	at kotlin.reflect.jvm.internal.calls.CallerImpl$Constructor.call(CallerImpl.kt:41) ~[kotlin-reflect-1.5.31.jar!/:1.5.31-release-548(1.5.31)]
	at kotlin.reflect.jvm.internal.KCallableImpl.call(KCallableImpl.kt:108) ~[kotlin-reflect-1.5.31.jar!/:1.5.31-release-548(1.5.31)]
	at kotlin.reflect.jvm.internal.KCallableImpl.callDefaultMethod$kotlin_reflection(KCallableImpl.kt:159) ~[kotlin-reflect-1.5.31.jar!/:1.5.31-release-548(1.5.31)]
	at kotlin.reflect.jvm.internal.KCallableImpl.callBy(KCallableImpl.kt:112) ~[kotlin-reflect-1.5.31.jar!/:1.5.31-release-548(1.5.31)]
	at org.springframework.beans.BeanUtils$KotlinDelegate.instantiateClass(BeanUtils.java:854) ~[spring-beans-5.3.10.jar!/:5.3.10]
	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:196) ~[spring-beans-5.3.10.jar!/:5.3.10]
	... 29 common frames omitted
Caused by: com.arcadedb.exception.DatabaseMetadataException: Error on loading dictionary (error=java.io.FileNotFoundException: /<PATH_TO_DATABASE>/dictionary.0.327680.dict (No such file or directory))
	at com.arcadedb.schema.EmbeddedSchema.create(EmbeddedSchema.java:120) ~[arcadedb-engine-21.9.1.jar!/:na]
	at com.arcadedb.database.EmbeddedDatabase.openInternal(EmbeddedDatabase.java:209) ~[arcadedb-engine-21.9.1.jar!/:na]
	... 42 common frames omitted
Caused by: java.io.FileNotFoundException: /<PATH_TO_DATABASE>/dictionary.0.327680.dict (No such file or directory)
	at java.base/java.io.RandomAccessFile.open0(Native Method) ~[na:na]
	at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345) ~[na:na]
	at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259) ~[na:na]
	at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214) ~[na:na]
	at com.arcadedb.engine.PaginatedFile.open(PaginatedFile.java:247) ~[arcadedb-engine-21.9.1.jar!/:na]
	at com.arcadedb.engine.PaginatedFile.<init>(PaginatedFile.java:57) ~[arcadedb-engine-21.9.1.jar!/:na]
	at com.arcadedb.engine.FileManager.getOrCreateFile(FileManager.java:170) ~[arcadedb-engine-21.9.1.jar!/:na]
	at com.arcadedb.engine.PaginatedComponent.<init>(PaginatedComponent.java:54) ~[arcadedb-engine-21.9.1.jar!/:na]
	at com.arcadedb.engine.PaginatedComponent.<init>(PaginatedComponent.java:65) ~[arcadedb-engine-21.9.1.jar!/:na]
	at com.arcadedb.engine.PaginatedComponent.<init>(PaginatedComponent.java:44) ~[arcadedb-engine-21.9.1.jar!/:na]
	at com.arcadedb.engine.Dictionary.<init>(Dictionary.java:66) ~[arcadedb-engine-21.9.1.jar!/:na]
	at com.arcadedb.schema.EmbeddedSchema.create(EmbeddedSchema.java:112) ~[arcadedb-engine-21.9.1.jar!/:na]
	... 43 common frames omitted

Steps to reproduce

Run the following code

private val factory: DatabaseFactory = DatabaseFactory(<PATH_TO_DB>)
val database: Database = if (factory.exists()) {
    factory.open()
} else {
    factory.create()
}

database.transaction {
    if (!it.schema.existsType(<TYPE_NAME>)) {
        val doc = it.schema.createDocumentType(<TYPE_NAME>)
        doc.createProperty(<PROPERTY_NAME>, Type.STRING)
        doc.createProperty(<PROPERTY_NAME>, Type.STRING)
        doc.createProperty(<PROPERTY_NAME>, Type.STRING)
        doc.createProperty(<PROPERTY_NAME>, Type.INTEGER)
    }
}

Add unit tests to SQL

Expected behavior

Test most of SQL operators, functions, etc.

Actual behavior

Some classes, even if very simple, are not tested yet

Server side transactions

Discussed in #117

^{Originally posted by lvca October 8, 2021}
Right now ArcadeDB HTTP/JSON protocol is totally stateless. This allows to use the transactions only with SQL scripts, but it's not the most user-friendly API to use.

I think the easiest thing to do is to bind a client session to a server-side session. On the server side, we could create a session id as a UUID and send it to the client at authentication as a token. If the client passes this token in the HTTP request header, then the server looks for the session id. In this way, we could save a transaction object and expose the begin + commit commands in the remote database API.

Add code coverage on Codacy

Expected behavior

Add code coverage:

configure Jacoco
configure Codacy action to push coverage
badge is present on the README

Actual behavior

Code coverage is not computed nor displayed on README

HTTP create document doesn't insert properties

ArcadeDB Version: 21.9.1-SNAPSHOT

JDK Version: 11

OS: MacOSX

This issue was originally reported on the ArcadeDB Discord channel.

Expected behavior

The document is correctly inserted and all the properties are stored

Actual behavior

Steps to reproduce

Call POST /document/db with the following payload, no properties are saved.

{"alert_severity": 5, "alert_group": "monitor:mail-antispam", "agent": "E", "agent_location": "Mail-antispam", "alert_id": 1950007447, "alert_count": 2, "alert_custom_info": {"colo": "west1", "alert_workflow": "EMAIL", "company": {"colo": "", "monitor": {"component": "abacainbound", "chefenvironment": "abacainbound-prod-west1", "_threshold_name": "host_life", "namespace": "Mail-antispam", "hostgroup": "abaca_inbound", "id": "host_life: component:abacainbound chefenvironment:abacainbound-prod-west1 _threshold_name:host_life namespace:Mail-antispam hostgroup:abaca_inbound env:prod", "env": "prod", "version": "2"}, "agent": "monitor", "recover": "1", "flags": 0, "alertKey": "host_life", "type": "live", "runbook_id": "KB0022158", "context": "live", "property": "Mail-ASD/GLB", "farm": "monitor2-west", "alertGroup": "monitor:Mail-antispam", "mysig": "cee04eaa6c1500493db765f0c50270ed", "display_count": "1", "event_time": 1601489288, "LamInstanceName": "monitor", "dashboard_id": "https://olympus.monitor.ourcompany.com/v1/graph/new?start=1601481600&end=1601488800&_t.component=abacainbound&_t.chefenvironment=abacainbound-prod-west1&_t.host=eval101.asd.mail.west1.company.com&_t.namespace=Mail-antispam&_t.hostgroup=abaca_inbound&_t.env=prod&_s.s0._m.m0.name=Mail-antispam.host_life.status_ok&_s.s0._m.m1.name=Mail-antispam.host_life.status_bad&_s.s0._t._threshold_name=host_life", "customer": "connections"}, "cookbook": "EMAIL", "s_host": "eval", "mysig": "cee04eaa6c1500493db765f0c50270ed", "dashboard": "https://olympus.monitor.ourcompany.com/v1/graph/new?start=1601481600&end=1601488800&_t.component=abacainbound&_t.chefenvironment=abacainbound-prod-west1&_t.host=eval101.asd.mail.west1.company.com&_t.namespace=Mail-antispam&_t.hostgroup=abaca_inbound&_t.env=prod&_s.s0._m.m0.name=Mail-antispam.host_life.status_ok&_s.s0._m.m1.name=Mail-antispam.host_life.status_bad&_s.s0._t._threshold_name=host_life"}, "alert_description": "host_life -> Missing Data ", "alert_entropy": 1.0, "alert_external_id": "Mail-antispam", "alert_first_event_time": 1601485329, "alert_int_last_event_time": 1601489290, "alert_last_event_time": 1601489289, "alert_last_state_change": 1601490932, "alert_manager": "alertcons-2.yms.west1", "alert_signature": "cee04eaa6c1500493db765f0c50270ed:1950007447", "alert_source": "eval101.asd.mail.west1.company.com", "alert_source_id": "Mail-ASD/GLB", "sig_id": 13200498, "sig_category": "Closed", "sig_created_at": 1601485329, "sig_custom_info": {"mooghandling": {"situationTemplate": "$UNIQUE(custom_info.company.alertKey) Alerts.  Location: $UNIQUE(custom_info.colo).  From $UNIQUE(external_id), $UNIQUE(source_id) Property(s) and Agent $UNIQUE(custom_info.company.LamInstanceName). $MAP[ $UNIQ(source, hosts) $UNIQ(custom_info.company.alertKey, impacted_procs) ]"}, "impact_start": 1601485329, "hosts": ["eval101.asd.mail.west1.company.com"], "situationClass": "EMAIL.EMAIL SafetyNet", "impacted_procs": ["host_life"]}, "sig_description": "host_life Alerts.  Location: west1.  From Mail-antispam, Mail-ASD/GLB Property(s) and Agent E. ", "sig_first_event_time": 1601485329, "sig_last_event_time": 1601485329, "sig_last_state_change": 1601490932, "sig_story_id": 13200498, "sig_queue": "Investigated", "sig_teamname": "Cloud DevOps", "sig_owner": "user", "sig_owner_fullname": "users name"}

Response [200]

{"result": "#2:2933"}

CypherQueryEngine & PostCommandHandler

ArcadeDB Version: master (`444da9f`) / 21.10.1

JDK Version: 11.0.13

Actual behavior

In CypherQueryEngine collection values in map structure(vertices/edges) will not be sanitized according to the name schema of Arcade DB.
e.g. id -> @rid , label -> @type
In PostCommandHandler the collection values are not handled anymore, because collection values of kind
Map are not handled in com.arcadedb.serializer.JsonSerializer.serializeResult(Result).
Due that fact a RID Object of a map result is not handled and will be serialized not as a String.
The serialization will break due circular references in RID.

Expected behavior

Sanitize all values in CypherQueryEngine and maybe wrap nested Maps as a Result in Lists to be handled again in
PostCommandHandler. But it depends on the interface definition of QueryEngine.command(final String query, final Map<String, Object> parameters) and the returned ReultSet and the values that should returned: plain Map, Result, Element,...

Steps to reproduce

Simple parent with multiple child relationship with the cypher collect function:
eg. match(p:parent)--(c:child) return p, collect(c)
I can provide a PR with test & solution proposal

This is test setup to get the invalid results from query. The 'childs' projection will not be transformed.

@Test
  public void verifyProjectionWithCollectFunction() throws ExecutionException, InterruptedException {
    final ArcadeGraph graph = ArcadeGraph.open(DB_PATH);
    try (Database database = graph.getDatabase()) {
      database.transaction(() -> {
        Schema schema = database.getSchema();
        schema.getOrCreateVertexType("V");
        schema.getOrCreateEdgeType("E");

        MutableVertex v1 = database.newVertex("V").save();
        MutableVertex v2 = database.newVertex("V").save();
        MutableVertex v3 = database.newVertex("V").save();

        v1.newEdge("E", v2, true);
        v1.newEdge("E", v3, true);
        try (ResultSet query = database.query("cypher",
          "match(parent:V)-[e:E]-(child:V) where id(parent) = $p return parent as parent, collect(child) as childs", "p",
          v1.getIdentity())) {

        }

      });
    } finally {
      graph.drop();
    }
  }

Embedded mode is not persistent on windows

ArcadeDB Version: 21.10

JDK Version: Coretto 11

OS: Windows 10

Expected behavior

Stopping JVM should recover data correctly as indicated by docs.

Actual behavior

Following errors are outputted and data is lost:
Cannot find bucket type_0 for type 'Type', removing it from type configuration
Cannot find bucket type_1 for type 'Type', removing it from type configuration
Cannot find bucket type_2 for type 'Type', removing it from type configuration
Cannot find bucket type_3 for type 'Type', removing it from type configuration
Cannot find bucket type_4 for type 'Type', removing it from type configuration
Cannot find bucket type_5 for type 'Type', removing it from type configuration
Cannot find bucket type_6 for type 'Type', removing it from type configuration
Cannot find bucket type_7 for type 'Type', removing it from type configuration

<More Errors>

Error on deleting file '/<PATH_TO_DB>/txlog_0.wal'
Error on deleting file '/<PATH_TO_DB>/txlog_1.wal'
Error on deleting file '/<PATH_TO_DB>/txlog_2.wal'
Error on deleting file '/<PATH_TO_DB>/txlog_3.wal'
Error on deleting file '/<PATH_TO_DB>/txlog_4.wal'
Error on deleting file '/<PATH_TO_DB>/txlog_5.wal'

Steps to reproduce

Run the following and stop JVM

private val factory: DatabaseFactory = DatabaseFactory(<PATH_TO_DB>)
val database: Database = if (factory.exists()) {
    factory.open()
} else {
    factory.create()
}

database.transaction {
    if (!it.schema.existsType(<TYPE_NAME>)) {
        val doc = it.schema.createDocumentType(<TYPE_NAME>)
        doc.createProperty(<PROPERTY_NAME>, Type.STRING)
        doc.createProperty(<PROPERTY_NAME>, Type.STRING)
        doc.createProperty(<PROPERTY_NAME>, Type.STRING)
        doc.createProperty(<PROPERTY_NAME>, Type.INTEGER)
    }
}

Postgres: NPE executing command when portal is null

ArcadeDB Version: 21.9.1-SNAPSHOT

JDK Version: N/A

OS: N/A

Expected behavior

if the portal is null, the "no data" message should be written to the channel

Actual behavior

the "no data" is sent to the channel, but also a NPE is raised because also a writeCommandComplete() is executed.

arcadedb/postgresw/src/main/java/com/arcadedb/postgres/PostgresNetworkExecutor.java

Line 279 in bb85560

if (portal == null) {

      if (portal == null) {
        writeNoData();
        writeCommandComplete(portal.query, portal.cachedResultset == null ? 0 : portal.cachedResultset.size());

I think that we could just drop the writeCommandComplete() statement.

Cypher: Querying by @rid might be impossible

ArcadeDB Version: 21.10.2-SNAPSHOT (build `ae608bf`/1633588344745/main)

OS: docker

Expected behavior

One of these queries should return results:

MATCH (m:Movie{@rid:'#1:0'})<-[a:ACTED_IN]-(p:Person) RETURN *
MATCH (m:#1:0})<-[a:ACTED_IN]-(p:Person) RETURN * (not sure this will/should ever work, adapted from the SQL example)
MATCH (m:Movie)<-[a:ACTED_IN]-(p:Person) WHERE m.@rid = '#1:0' RETURN *

Actual behavior

org.opencypher.v9_0.util.SyntaxException: Invalid input '@' expected whitespace, a property key name, '}', an identifier or UnsignedDecimalInteger (line 1, column 16 (offset: 15))
org.opencypher.v9_0.util.SyntaxException: Invalid input '#': expected whitespace or a label name
Invalid input '@'

I tried several other less obvious variations (leaving off the @, using #1:0 without quotes, etc.) without success.
It's possible I just missed the correct way to do this in the docs or something. 😄

Index null values in LSM index

Right now null values are not indexed. Based on the index definition, if a property is null the index throws an error (ERROR null strategy) or just ignores the record from the index (SKIP null strategy) which means a query retrieving all the values that are null will not use the index.

The api/v1/query endpoint commits server-side transactions when called.

ArcadeDB Version: 21.10.2-SNAPSHOT (build `88f7d00`/1634331428839/main)

OS: docker

Expected behavior

Given the following:

POST to begin/_test
POST to query/_test (with the arcadedb-session-id)
POST to rollback/_test (with the arcadedb-session-id)

The rollback should succeed.

Actual behavior

The rollback fails with Transaction not begun.

Steps to reproduce

It seems that any call to query with a session-id will commit that transaction immediately. So if you do an insert or something, then call query, the rollback will fail and the inserted record will show in the database.

This seems only to impact query. The command endpoint, if used in place of query, works as expected.

You can see this in action via this gist that will run in the Visual Studio Code plugin Rest Client. (should be easy enough to follow even if you don't run it, though!)

https://gist.githubusercontent.com/tetious/e253574e8f1073273f2493313331cdf4/raw/aedeaf8e3bc4b82866e7e44a24b87722a59f4c50/transactions-test.rest

No-Lock database backup and restore

Discussed in #62

^{Originally posted by lvca September 18, 2021}
ArcadeDB stored the database on the file system using one directory per database. If it's running as a server, the database directories are under the databases directory of the ArcadeDB package.

Before the more sophisticated Delta Backup (a new discussion about this soon), we need a full backup and restore that avoid blocking the execution of transactions.

Requirements:

ArcadeDB uses "txlog" files for transactions. No need to save in the backup
ArcadeDB writes the changes to "txlog" first and then updates the actual database pages in an asynchronous way. During the backup, those pending pages in RAM must be all flushed to disk
A new lock must be acquired to the database preventing concurrent changes to the schema/indexes. Right now the EmbeddedDatabase has an RW lock, but we need something that supports different operations:
write lock: no concurrent access to the instance. This is typically used when the schema is changed
read lock: this is acquired by pretty much every operation, even a transaction commit
backup lock: this must block acquiring a write lock, and at the same time should still allow acquiring read locks
During the backup, the flushing of the pages to disk must be suspended. This doesn't impact the database reliability, because all the changes are still in the txlog
Transactions can still run in parallel

The actual backup operation will be creating a zip file with the content of the files on the filesystems (all but txlogs that can be modified during the backup).

Code cleanup: remove "O" prefix

Expected behavior

no more "O" prefix

Actual behavior

there are still some

Steps to reproduce

find . -name "O*.java"

Extend Server Security

Discussed in #56

^{Originally posted by lvca September 17, 2021}

SUMMARY

Status: Design
Progress: Design 100% -> Development & Unit Tests 0% -> Integration Tests 0%
Currently Assigned: @lvca
ETA: 5 days
From: Ideas [#44]
To: Issues []

ArcadeDB supports super-simple security for users in a server. Example of security.json file:

{
  "users": {
    "root": {
      "databases": [],
      "password": "PBKDF2WithHmacSHA256$65536$esd1hjs=...",
       "name": "root",
       "databaseBlackList": true
    },
    "elon": {
      "databases": ["Universe"],
      "password": "PBKDF2WithHmacSHA256$65536$lYhGpkO4Tgu53hBc3D...=",
      "name": "elon",
      "databaseBlackList": false
    }
  }
}

Below you can find the proposal to improve the Server Side security. This is going to be the new server-groups.json file:

{
  "databases": {
    "*": {
      "groups": {
        "admin": {
          "access": [
            "updateSchema"
          ],
          "types": {
            "*": {
              "access": [
                "create",
                "read",
                "update",
                "delete"
              ]
            }
          }
        },
        "*": {
          "access": "",
          "types": {
            "*": {
              "access": ""
            }
          }
        }
      }
    }
  },
  "version": 1
}

Where all the user/password pairs are moved under the new file server-users.jsonl in JSONL format (one JSON per line). This allows protecting the file with passwords from the actual security configuration. Also, the security.json file is designed to be a JSON file easily modifiable, while the server-users.jsonl file is managed by the ArcadeDB server and the passwords are not meant to be edited manually. Example of server-users.jsonl:

{"user":"root","password": "PBKDF2WithHmacSHA256$65536$BHV+MGHvpXH89e0a6ZedqABDCnyeC8kSdBx+ik6APOs=$W8H+AXFC+YJHdM9X+PPoL59VBY1kY6uMUDTIQ515GwU="}

That means the admin group has access to every database installed in the server, and full CRUD operations on all the types.

Support for group

In order to provide a greater level of security, we need to introduce fine-level permissions. Instead of creating such profiling for each user, it's more flexible configuring groups and assign users to one or more groups. Example.

Given the "elon" users in server-users.jsonl:

{"elon": {"groups": ["pragmatist", "genius"]}}

You can define the rules for each groups in a database

{
  "databases": {
    "Heroes": {
      "groups": {
        "pragmatist" : {
        ...
        }
      }
    }
  },
}

Type access rules are strings where each character means a type of permission (case sensitive).

For types the typical CRUD access:

'create' allows creating new records for the type
'read' allows reading records for the type
'update' allows updating records for the type
'delete' allows deleting records for the type

For the databases:

'updateSchema' allows modifying the schema

For the users:

'updateSecurity' allows modifying the security

In case a user belongs to multiple groups, the permissions are simply added to each other and the resulting is the sum of all the permissions.

The wildcard * will be used to assign the defaults. In the default configuration above, the "admin" user applies to all groups, databases and types. Creating a database or a group with the name * is forbidden.

Profile access to types for CRUD operations and more

Example of profiling at type level:

"Reviewers": {
  "databases": {
    "*": {
      "types": {
        "Post": {
          "access": "r",
          "limit": 10
        },
        "Comment": {
          "access": "r",
          "resultSetLimit": 100,
          "readTimeout": 5000
        },
        "Note": {
          "access": "crud"
        }
      }
    }
  },
}

The example above does not specify a database, but it's rather using the wildcard "*". This is a typical setting for multi-tenant, where a server hosts multiple databases, one per tenant, with the same structure.

The property resultSetLimit, if present, tells to cut off resultsets larger than the value in this field. This is useful to avoid some users can stress the database by returning huge resultsets.

The property readTimeout, if present, limits the query execution to the timeout expresses in milliseconds.

Add version for further compatibility

Adding a version allows the server to support the evolution of the security.json file. Example:

{
  "users": {},
  "groups": {},
  "version": 1
}

API

if ArcadeDB runs as embedded without a server, the default implementation returns UnsupportedOperationException. When the server is running, it installs the server security. Users can implement their own security and register the component at startup. No SQL will be provided at first, if there is demand, we could implement server commands.

File auto-reload and format

The ServerSecurity module should watch changes to the file and reload it automatically (with a message in console). At every reload of the file, the consistency must be checked before replacing the current configuration in RAM. In case of error, the current configuration must be serialized in JSON into the file "config/security-prev.json". If the configuration in RAM is empty, then the previous file "config/security-prev.json" (if any) must be loaded and used.

The Security JSON file will be written with an indent = 2 to make easier editing offline with an editor.

FAQ

Wouldn't be better to have a separate database to save users, groups, etc?

There are pros and cons, but the flexibility to edit a JSON file to modify the security is the first reason, then all the security is loaded in RAM, so there would be no access to the database.

What if I have hundreds or thousands of users?

The JSON file can be large. All the users are loaded in RAM, so check the amount of RAM you're reserving for the ArcadeDB Server process.

add test report to actions

Expected behavior

get a test report for each build on both prs and main branch

Actual behavior

open the build log to understand what went wrong

cannot restore database

ArcadeDB Version: 21.10.2

JDK Version: running in docker image

OS: macOS 11.6 (20G165)

Expected behavior

database restored

Actual behavior

null pointer exception

Steps to reproduce

From inside image. Tried without database created, after trying to create from console, and after creating from Studio.

/home/arcadedb # pwd
/home/arcadedb
/home/arcadedb # ls
LICENSE                                  bin                                      databases                                questions-backup-20211011-015429344.zip
README.md                                config                                   lib
/home/arcadedb # bin/restore.sh -f /home/arcadedb/questions-backup-20211011-015429344.zip  -database databases/questions
Exception in thread "main" com.arcadedb.integration.restore.RestoreException: Error during restore of database from file '/home/arcadedb/questions-backup-20211011-015429344.zip'
	at com.arcadedb.integration.restore.Restore.restoreDatabase(Restore.java:55)
	at com.arcadedb.integration.restore.Restore.main(Restore.java:42)
Caused by: java.lang.NullPointerException
	at java.base/java.io.File.<init>(File.java:278)
	at com.arcadedb.integration.restore.format.FullRestoreFormat.restoreDatabase(FullRestoreFormat.java:50)
	at com.arcadedb.integration.restore.Restore.restoreDatabase(Restore.java:52)
	... 1 more
/home/arcadedb # ls databases/
/home/arcadedb # ls bin/
arcadedb.pid      backups           console.bat       console.sh        restore.sh        server.bat        server.sh         stresstester.bat  stresstester.sh
/home/arcadedb # bin/console.sh 
ArcadeDB Console v.21.10.2-SNAPSHOT - Copyrights (c) 2021 Arcade Data Ltd (https://arcadedb.com)

> create database questions


{questions}> ?

HELP
begin                               -> begins a new transaction
check database                      -> check database integrity
close                               -> closes the database
create database <path>|remote:<url> -> creates a new database
commit                              -> commits current transaction
connect <path>|remote:<url>         -> connects to a database stored on <path>
help|?                              -> ask for this help
info types                          -> print available types
info transaction                    -> print current transaction
rollback                            -> rollbacks current transaction
quit or exit                        -> exits from the console


{questions}> exit
/home/arcadedb # ls databases/
/home/arcadedb # bin/restore.sh -f /home/arcadedb/questions-backup-20211011-015429344.zip  -database databases/questions
Exception in thread "main" com.arcadedb.integration.restore.RestoreException: Error during restore of database from file '/home/arcadedb/questions-backup-20211011-015429344.zip'
	at com.arcadedb.integration.restore.Restore.restoreDatabase(Restore.java:55)
	at com.arcadedb.integration.restore.Restore.main(Restore.java:42)
Caused by: java.lang.NullPointerException
	at java.base/java.io.File.<init>(File.java:278)
	at com.arcadedb.integration.restore.format.FullRestoreFormat.restoreDatabase(FullRestoreFormat.java:50)
	at com.arcadedb.integration.restore.Restore.restoreDatabase(Restore.java:52)
	... 1 more
/home/arcadedb #

Allow changing the schema from a replica server

With v21.10.1 changes to the schema are only allowed through the leader server. Since transactions can be executed on any servers, we'd need to implement the same forward messages in case of schema update.

Load wrappers on separate class Loader

Expected behavior

Load wrappers (mongo, redis, postgres, gremlin) on dedicated class loader

Actual behavior

wrappers are loaded on the main class loader

depends on #156

Console: invalid database directory location

Discussed in #161

^{Originally posted by oldstager October 23, 2021}
I've managed to run server using:

$ bin/server.sh -Dserver.databaseDirectory=/some/path/data/arcadedb/

then I run console.sh and from there I create a db and import OpenBeer:

create database beer
{beer}> IMPORT DATABASE https://github.com/ArcadeData/arcadedb-datasets/raw/main/orientdb/OpenBeer.gz.

I suspect, import ran successfully since I got this:

However, Studio doesn not show beer database and /some/path/data/arcadedb/ is empty. I still can use console.sh later and connect to beer database.

I am still confuse, where are all those files located?

importer is no longer available?

ArcadeDB Version: /arcadedb-21.10.1-SNAPSHOT ( pulled from main and built )

JDK Version: 14.0.1

OS: OSX Big Sur

Expected behavior

bin/importer is no longer there

Actual behavior

AFAICS, it is no longer possible to import data

Steps to reproduce

pull, build run bin/server.sh

Engine: NPE instead of DatabaseMetadataException if the bucket is null during index creation

ArcadeDB Version: 21.9.1-SNAPSHOT

JDK Version: N/A

OS: N/A

Expected behavior

DatabaseMetadataException should be raised

Actual behavior

NPE is raised, hiding the real exception and related error message

Steps to reproduce

I have no use case yet, but it's clear what happens looking at the code

arcadedb/engine/src/main/java/com/arcadedb/schema/EmbeddedSchema.java

Line 569 in bb85560

if (bucket == null)

    if (bucket == null)
      throw new DatabaseMetadataException(
          "Cannot create index on type '" + typeName + "' because the specified bucket '" + bucket.getName() + "' is not part of the type");

@lvca what the real message should be? Because if bucket is null something here is going very weird.

WDYT?

Postgres plugin not activating on windows start file

ArcadeDB Version: 21.10

JDK Version: Oracle 8

OS: Windows 10

Expected behavior

Postgres Protocol plugin started displayed.

Actual behavior

Starting ArcadeDB Server with plugins []... displayed

Steps to reproduce

server.bat -Darcadedb.server.plugins=Postgres

Cypher queries that return nodes fail with an internal error

ArcadeDB Version: 21.9.1

OS: The official Docker image, on Linux.

Expected behavior

When running a Cypher query that returns nodes/vertexes via the web interface (Studio?):

CREATE (p:Person {name: 'Keanu Reeves'})
RETURN p

The matching node(s) show up, without error.

Actual behavior

An internal error results:

{
    "error": "Internal error",
    "detail": "java.lang.ClassCastException: class org.apache.tinkerpop.gremlin.structure.T$2 cannot be cast to class java.lang.String (org.apache.tinkerpop.gremlin.structure.T$2 is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')",
    "exception": "java.lang.ClassCastException"
}

Note that this will occur with a MATCH too. If I specify a prop in the return, i.e. RETURN p.name, it returns a tabular result without error. Querying via SQL seems to work fine. SELECT from Person works as expected.

This might be a misunderstanding of the compatibility of the Cypher support, as I'm pretty new to this, and was following a tutorial in the neo4j docs. My apologies if I missed something. :)

multi line sql statements can not be parsed by web ui

ArcadeDB Version: ArcadeDB Server v21.9.2-SNAPSHOT (build `1963901`/1632261478984/main)

JDK Version: openjdk 11.0.12 2021-07-20 OpenJDK Runtime Environment Temurin-11.0.12+7 (build 11.0.12+7) OpenJDK 64-Bit Server VM Temurin-11.0.12+7 (build 11.0.12+7, mixed mode)

OS: from docker image Linux 9d77ff622b47 3.10.0-1160.31.1.el7.x86_64 #1 SMP Wed May 26 20:18:08 UTC 2021 x86_64 Linux

Expected behavior

sql statements that have multiple lines in the browser console should execute

Actual behavior

SQL statements containing a new line will error out from the browser with a parsing exception
Error { "error" : "Internal error", "detail":"com.arcadedb.exception.CommandSQLParsingException: create Edge type a; create Edge type b;", "exception": "com.arcadedb.exception.CommandSQLParsingException"}

Steps to reproduce

create Edge type a;
create Edge type b;

Support for custom key/value in types and properties

Discussed in #168

^{Originally posted by PhantomYdn October 25, 2021}
It will be nice to have:

DocumentType.getCustom(String key), DocumentType.setCustom(String key, String value), Property.getCustom(String key), Property.setCustom(String key, String value) to be able to save additional metainformation in the schema.

docker instance has no webpage

ArcadeDB Version: v21.9.1-SNAPSHOT ( via docker latest )

JDK Version: 11.0.1

OS: ubuntu 19.x

Expected behavior

http://localhost:2480 should show something

Actual behavior

This localhost page can’t be foundNo webpage was found for the web address: http://localhost:2480/

Steps to reproduce

docker run --rm -p 2480:2480 -p 2424:2424 --env arcadedb.server.rootPassword=zzzzzzzz arcadedata/arcadedb:latest

Speedup Cypher/Gremlin execution

There are mainly 3 things to be implemented to dramatically speedup Cypher and Gremlin query execution:

(1) Use of Gremlin custom steps

Both Gremlin and Cypher implementations can be largely improved by defining custom steps that use the indexes if defined.

This query executes a scan of all the vertices even if an index on Movie.title is defined.

match (m:Movie {title: 'The Da Vinci Code'})<-[r:REVIEWED]-(p)
where r.rating > 4
return r,p

(2) Avoid using mutable graph elements when not necessary

Support both immutable and mutable graph elements and modify under the hood only if necessary.

(3) Cache GremlinExecutor instance to speedup gremlin parsing

Instead of creating a new GremlinExecutor instance every time, it should be created only once and reused.

Gremlin doesn't work with Java 16+

ArcadeDB Version: 21.9.1-SNAPSHOT

JDK Version: 16

OS: Mac

Expected behavior

Error on using Gremlin and Cypher.

Actual behavior

At runtime this exception is thrown from the Gremlin stack: Unsupported class file major version 60

Steps to reproduce

mvn clean install

'ALTER TYPE' not working

ArcadeDB Version: 21.10.1

JDK Version: OpenJDK 11

OS: Win7

Steps to reproduce (I tested on Studio)

Create vertex type Person without extends any type
Then create vertex type V.
Run command 'ALTER TYPE Person SUPERTYPE V', nothing changed, the type Person has not become subtype of V as it should.

Console cannot connect to remote database

ArcadeDB Version: v21.9.1-SNAPSHOT

JDK Version: openjdk version "11.0.2" 2019-01-15

OS: macOS Mojave v.10.14.6

Expected behavior

Console must connect to a remote database

Actual behavior

Console returns the following error:

 WARNI [RemoteDatabase] Error on executing command, retrying... (payload=Not Found, error=org.json.JSONException: A JSONObject text must begin with '{' at 1 [character 2 line 1])
Error: A JSONObject text must begin with '{' at 1 [character 2 line 1]

Steps to reproduce

Start the server (server.sh)
Create a database (via Studio)
Start the console (console.sh)
Try to connect to the remote database:

> connect remote:127.0.0.1/ciaone root ciaone1234
connect remote:127.0.0.1/ciaone root ciaone1234

2021-09-03 07:52:03.952 WARNI [RemoteDatabase] Error on executing command, retrying... (payload=Not Found, error=org.json.JSONException: A JSONObject text must begin with '{' at 1 [character 2 line 1])
Error: A JSONObject text must begin with '{' at 1 [character 2 line 1]

>

The problem is that Console doesn't use the API version prefix. It tries to connect to the URL http://127.0.0.1:2480/server instead of http://127.0.0.1:2480/api/v1/server.

See:

arcadedb/network/src/main/java/com/arcadedb/remote/RemoteDatabase.java

Line 218 in 13276ff

 String url = protocol + "://" + connectToServer.getFirst() + ":" + connectToServer.getSecond() + "/" + operation; 

and:

arcadedb/server/src/main/java/com/arcadedb/server/http/HttpServer.java

Line 81 in 13276ff

routes.addPrefixPath("/api/v1",//

At the moment, the fix is trivial, but we should find a way to sync the API version among the projects to avoid a similar problem in the future.

Create fatjars for wrappers

Expected behavior

Each wrapper (mongo, redis, gremlin) should be packaged as a single fatjar

Actual behavior

Each wrapper brings its own libraries to the classpath

No enum constant com.arcadedb.schema.Type.LINKLIST on DB import

ArcadeDB Version: 2.2.10

JDK Version: JDK 14.0.1

OS: OS/X

Expected behavior

database should be able to import ( OrientDB 3.2.0 )

Actual behavior

an exception is thrown on Type.LINKLIST

Steps to reproduce

import a DATABASE with some RIDs in a LINKLIST

stack trace of exception:

java.lang.IllegalArgumentException: No enum constant com.arcadedb.schema.Type.LINKLIST
	at java.base/java.lang.Enum.valueOf(Enum.java:273)
	at com.arcadedb.schema.Type.valueOf(Type.java:43)
	at com.arcadedb.importer.OrientDBImporter.createType(OrientDBImporter.java:822)
	at com.arcadedb.importer.OrientDBImporter.createType(OrientDBImporter.java:780)
	at com.arcadedb.importer.OrientDBImporter.parseSchema(OrientDBImporter.java:720)
	at com.arcadedb.importer.OrientDBImporter.parseInputFile(OrientDBImporter.java:272)
	at com.arcadedb.importer.OrientDBImporter.run(OrientDBImporter.java:184)
	at com.arcadedb.importer.format.OrientDBFormatImporter.load(OrientDBFormatImporter.java:44)
	at com.arcadedb.importer.Importer.loadFromSource(Importer.java:92)
	at com.arcadedb.importer.Importer.load(Importer.java:57)
	at com.arcadedb.server.http.handler.ImportDatabaseHandler.execute(ImportDatabaseHandler.java:59)
	at com.arcadedb.server.http.handler.AbstractHandler.handleRequest(AbstractHandler.java:110)
	at io.undertow.server.RoutingHandler.handleRequest(RoutingHandler.java:93)
	at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:104)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:387)
	at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:256)
	at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
	at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:162)
	at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:100)
	at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:57)
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
	at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:291)
	at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286)
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
	at org.xnio.nio.QueuedNioTcpServer2.acceptTask(QueuedNioTcpServer2.java:178)
	at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612)
	at org.xnio.nio.WorkerThread.run(WorkerThread.java:479)

Any plans for a remote java API

Are there any plans to do a remote java API like the one in Orient?

internal error on create property

ArcadeDB Version: 21.10.2

JDK Version: in official image

OS: latest macOS

Expected behavior

add property without error

Actual behavior

property added with error

Steps to reproduce

loom recording

Begin to document areas where we can take advantage of Java's Vector API

Required Reading: https://www.youtube.com/watch?v=VYo3p4R66N8
Video: https://www.youtube.com/watch?v=VYo3p4R66N8

One of the goals of ArcadeDB is to maximize data parallelism using various techniques in low level Java, and design paradigms.
Data parallelism can likely be further enhanced for performance with hardware CPU's such as Intel and ARM and possibly others later (Apple Mx?) by using the Java Vector API (currently now in Second Incubator).

Additionally, Java JVM also already provides and uses many intrinsic functions. In the future some of these will be re-rewritten to take first-hand advantage of the new Vector API as discussed in the video.

It would be good to use this issue to begin to document areas of the code base that likely would be starting points for exploration where data parallelism can likely take extra advantage of hardware.

Support for Append Only buckets

Having append only structures allows some interesting use cases like ledgers, block chains,

@rid is possibly not parameterized properly

ArcadeDB Version: 21.10.2-SNAPSHOT (build `44e2d86`/1633547118877/main)

OS: Docker

Expected behavior

Executing a POST against the /api/v1/command/Movies endpoint with this json payload:

{"language": "sql", "command": "SELECT FROM :rid", "params": { "rid": "#1:0" }, "serializer": "record"}

should return the record with rid #1:0.

Actual behavior

It appears as if parameterizing the rid puts quotes around it, which breaks things.

{
  "exception": "com.arcadedb.exception.SchemaException",
  "detail": "Type with name '#1:0' was not found",
  "error": "Internal error"
}

Of course, I could be doing this totally wrongly. 😄

SELECT FROM #1:0 works as expected. (no quotes)

Postgres listener hardcoded to bind on localhost only?

ArcadeDB Version: 21.10.2-SNAPSHOT (build `ae608bf`/1633588344745/main)

OS: Docker

Expected behavior

The Postgres listener starts on 0.0.0.0, or is at least configurable as such.

Actual behavior

From looking at the code, It seems like it is hardcoded to bind to localhost, which is not ideal for using outside the container itself.

The log message seems to confirm this.

[PostgresNetworkListener] <ArcadeDB_0> Listening for replication connections on 127.0.0.1:5432 (protocol v.-1)

The example in the docs implies it will work, but it doesn't expose 5432, so I'm not sure what's going on there. I suspect a typo. 😄

docker run --rm -p 2480:2480 -p 2424:2424 --env arcadedb.server.rootPassword=playwithdata --env arcadedb.server.plugins="Postgres:com.arcadedb.postgres.PostgresProtocolPlugin" arcadedata/arcadedb:latest

Cypher queries that return multiple objects fail

ArcadeDB Version: 21.10.1-SNAPSHOT (build `f68623e`/1633172100065/main)

OS: docker

This is very similar to #86, except it occurs when returning multiple objects.

When executing the following against the movieratings example dataset:

match (m:Movies {id: 1})<-[r:rated]-(p)
where r.rating > 4
return r,p

I get the following internal error:

detail: class org.apache.tinkerpop.gremlin.structure.T$2 cannot be cast to class java.lang.String (org.apache.tinkerpop.gremlin.structure.T$2 is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')

If I use return * the same error occurs.

Remove duplication of BaseGraphServerTest

Expected behavior

Avoid code duplication of BaseGraphServerTest

Actual behavior

this class:

public abstract class BaseGraphServerTest

is duplicated in every module, at least

Index doesn't work as expected with partial NULL keys

ArcadeDB Version: 10.2021

JDK Version: 11

OS: win

The following testcase is failing if executed first time and pretty OK if executed second and etc.

@Test
	public void testDocumentAfterCreation() {
		DocumentType typeRoot = database.getSchema().getOrCreateDocumentType("TestRoot");
		typeRoot.getOrCreateProperty("name", String.class);
		typeRoot.getOrCreateTypeIndex(INDEX_TYPE.FULL_TEXT, true, "name");
		database.command("sql", "delete from TestRoot");
		
		DocumentType typeChild = database.getSchema().getOrCreateDocumentType("TestChild");
		typeChild.setParentTypes(Arrays.asList(typeRoot));
		MutableDocument doc =  database.newDocument("TestChild");
		doc.set("name", "Document Name");
		assertEquals("Document Name", doc.get("name"));
		doc.save();
		assertEquals("Document Name", doc.get("name"));
		try(ResultSet rs = database.query("sql", "select from TestChild where name = :name", CommonUtils.toMap("arg0", "Test2", "name", "Document Name"))) {
			assertTrue(rs.hasNext());
			Document docRetrieved = rs.next().getElement().orElse(null);
			assertEquals("Document Name", docRetrieved.get("name"));
			assertFalse(rs.hasNext());
		}
	}

Type of the index and uniqueness doesn't matter here.

Moreover, slight modification of indexes leads to the testcase which always fail: first, second and etc.

	@Test
	public void testDocumentAfterCreation2() {
		DocumentType typeRoot = database.getSchema().getOrCreateDocumentType("TestRoot2");
		typeRoot.getOrCreateProperty("name", String.class);
		typeRoot.getOrCreateProperty("parent", Type.LINK);
		typeRoot.getOrCreateTypeIndex(INDEX_TYPE.LSM_TREE, true, "name", "parent");
		database.command("sql", "delete from TestRoot2");
		
		DocumentType typeChild = database.getSchema().getOrCreateDocumentType("TestChild2");
		typeChild.setParentTypes(Arrays.asList(typeRoot));
		MutableDocument doc =  database.newDocument("TestChild2");
		doc.set("name", "Document Name");
		assertEquals("Document Name", doc.get("name"));
		doc.save();
		assertEquals("Document Name", doc.get("name"));
		try(ResultSet rs = database.query("sql", "select from TestChild2 where name = :name", CommonUtils.toMap("arg0", "Test2", "name", "Document Name"))) {
			assertTrue(rs.hasNext());
			Document docRetrieved = rs.next().getElement().orElse(null);
			assertEquals("Document Name", docRetrieved.get("name"));
			assertFalse(rs.hasNext());
		}
	}

Both test cases become successful if delete line with the creation of an index.

Run CI gh actions on Windows too

Expected behavior

Tests are running smootly on Windows

Actual behavior

Tests are failing smootly on Windows

linked issues: #100 #97

Realtimey features, aka change streams

Discussed in #149

^{Originally posted by tetious October 18, 2021}
I'm wondering if there are plans around or interest in implementing something like Mongo's Change Streams?

It seems like it might not be too bad to add, given it is essentially a special-case of replication. This might be an oversimplification based on optimistic naivety, though. 😄

Being able to subscribe to changes at the "Type" level, as well as database level, would be very interesting I think. It would also allow for fleshing out some of the more flashy bits of the Redis (pub/sub) and Mongo feature-sets.

Thoughts?

Need some way preventing duplicate edges on the same pair of vertex.

ArcadeDB Version: <version here, you can find it on the first line of server output>

JDK Version: OpenJDK 11

OS: Win7 64bit

Ex: When run this command "CREATE EDGE Own FROM aPerson TO aPet" multiple times, I don't want the DB create multiple 'Own' edges to the same pet. How can I do? I read the document but nowhere mentions about this.

Evaluate forcing the user to update the schema in transactions

Schema updates don't require a transaction, because when a transactional operation is required, a new (sub) transaction is created. Also, with the last changes, in order to replicate the creation of indexes, we needed to both listen for file changes and page changes. The solution was to embed a database change structure request in the TxRequest object.

Evaluate if makes sense to force the user to open a transaction for any schema changes. In this way, we could simplify the current workflow and store database changes ONLY in the TxRequest object.

Quotes are stripped out of arrays, possibly only in GraphML imports.

ArcadeDB Version: 21.10.2-SNAPSHOT (build `ae608bf`/1633588344745/main)

OS: docker

For this query:
MATCH (m:Movie{title:'The Polar Express'})<-[e:ACTED_IN]-(v:Person) RETURN e, v

Expected behavior

The edge property 'roles' should be a properly formed array, as is shown here from Neo4J's view of the import:

Actual behavior

The quotes are missing, making the array unparsable:
"roles": "[Hero Boy, Father, Conductor, Hobo, Scrooge, Santa Claus]"

I've confirmed arrays can be created and returned successfully via parameterized inserts, so I suspect this is a bug in the import code. Could be wrong, though!

arcadedata / arcadedb Goto Github PK

arcadedb's Introduction

Multi Model DBMS Built for Extreme Performance

Getting started in 5 minutes

Community

Security

License

Contributing

TL;DR: ArcadeDB is an open-source multi-model NoSQL database systems.

arcadedb's People

Contributors

Stargazers

Watchers

Forkers

arcadedb's Issues

ArcadeDB Version: trunk

JDK Version: JDK 14.0.1

OS: Mac OSX

Expected behavior

Actual behavior

Steps to reproduce

ArcadeDB Version: 21.9.1-SNAPSHOT

JDK Version: N/A

OS: N/A

ArcadeDB Version: 21.10

JDK Version: Oracle 8

OS: Windows 10 (happens on docker as well)

Expected behavior

Actual behavior

Steps to reproduce

ArcadeDB Version: v21.10.2-SNAPSHOT (main a7e3074)

JDK Version: azul-11

OS: MacOS Big Sur

ArcadeDB Version: 21.10

JDK Version: OpenJDK 11

OS: Ubuntu Server 20.04

Expected behavior

Actual behavior

Steps to reproduce

Expected behavior

Actual behavior

Discussed in #117

Expected behavior

Actual behavior

ArcadeDB Version: 21.9.1-SNAPSHOT

JDK Version: 11

OS: MacOSX

Expected behavior

Actual behavior

Steps to reproduce

ArcadeDB Version: master (444da9f) / 21.10.1

JDK Version: 11.0.13

Actual behavior

Expected behavior

Steps to reproduce

ArcadeDB Version: 21.10

JDK Version: Coretto 11

OS: Windows 10

Expected behavior

Actual behavior

Steps to reproduce

ArcadeDB Version: 21.9.1-SNAPSHOT

JDK Version: N/A

OS: N/A

Expected behavior

Actual behavior

ArcadeDB Version: 21.10.2-SNAPSHOT (build ae608bf/1633588344745/main)

OS: docker

Expected behavior

Actual behavior

ArcadeDB Version: 21.10.2-SNAPSHOT (build 88f7d00/1634331428839/main)

OS: docker

Expected behavior

Actual behavior

Steps to reproduce

Discussed in #62

Expected behavior

Actual behavior

Steps to reproduce

Discussed in #56

ArcadeDB Version: v21.10.2-SNAPSHOT (main `a7e3074`)

ArcadeDB Version: master (`444da9f`) / 21.10.1

ArcadeDB Version: 21.10.2-SNAPSHOT (build `ae608bf`/1633588344745/main)

ArcadeDB Version: 21.10.2-SNAPSHOT (build `88f7d00`/1634331428839/main)