xtdb / xtdb-tutorial Goto Github PK

The xtdb "Space Adventure" tutorial

License: MIT License

xtdb-tutorial's Introduction

XTDB is an open-source immutable database with comprehensive time-travel. XTDB has been built to simplify application development and address complex data compliance requirements. XTDB can be used via SQL and XTQL.

XTDB 2.x is currently in early access; if you are looking for a stable release of an immutable document database with bitemporal query capabilities, we are continuing to develop and support XTDB 1.x at https://github.com/xtdb/xtdb/tree/1.x.

Major features:

Immutable - while it’s optimised for current-time queries, you can audit the full history of your database at any point, with the need for snapshots.
'Bitemporal' - all data is accurately versioned as updates are made ('system' time), but it also allows you to separately record and query when that data is, was, or will become valid in your business domain ('valid' time).
Dynamic - you don’t need to specify schema up-front before documents (rows with arbitrarily nested data) can be inserted.
Speaks both SQL and XTQL.

XTQL is a data-oriented, composable query language - designed from the ground up to be amenable to both hand-written and generated queries. It is heavily inspired by the theoretical bases of both Datalog and the relational algebra.

It also supports SQL, for compatibility with existing experience and tooling. Particularly, it supports the bitemporal functionality as specified in the SQL:2011 standard.
Cloud native - the ACID, columnar engine is built on Apache Arrow and designed for object storage
It is written and supported by JUXT.

Quick links

2.x Documentation
1.x Documentation (see the 1.x branch)
Maven releases
Release notes
Support: discuss.xtdb.com | [email protected]
Developing XTDB 2.x
Bibliography

Inside-out Architecture

XTDB embraces the transaction log as the central point of coordination when running as a distributed system.

What do we have to gain from turning the database inside out?

Simpler code, better scalability, better robustness, lower latency, and more flexibility for doing interesting things with data.

— Martin Kleppmann

Pre-Release Snapshot Builds

Maven snapshot versions are periodically published under 2.0.0-SNAPSHOT and are used to facilitate support and debugging activities during the development cycle. To access snapshots versions, the Sonatype snapshot repository must be added to your project definition:

<repository>
  <id>sonatype.snapshots</id>
  <name>Sonatype Snapshot Repository</name>
  <url>https://s01.oss.sonatype.org/content/repositories/snapshots</url>
  <releases>
    <enabled>false</enabled>
  </releases>
  <snapshots>
    <enabled>true</enabled>
  </snapshots>
</repository>

;; project.clj
:repositories [["sonatype-snapshots" {:url "https://s01.oss.sonatype.org/content/repositories/snapshots"}]]

;; deps.edn
:mvn/repos {"sonatype-snapshots" {:url "https://s01.oss.sonatype.org/content/repositories/snapshots"}}

In contrast to regular releases which are immutable, a 2.0.0-SNAPSHOT release can be "updated" - this mutability can often be useful but may also cause unexpected surprises when depending on 2.0.0-SNAPSHOT for longer than necessary. Snapshot versions, including full 2.0.0-<timestamp> coordinates (which are useful to avoid being caught out by mutation), can be found here.

XTDB 1.x

XTDB 1.x is a mature product offering that is used in production by many organizations, and its ongoing development is focused on hardening and performance. XTDB 1.x is an embeddable database that emphasizes in-process JVM usage to enable advanced functionality like user-defined transaction functions, speculative transactions, programmatic Datalog rules, and more.

XTDB 2.x’s initial objective is to take the key principles embodied in XTDB 1.x — immutability, schemaless records, and temporal querying — to a mainstream audience.

	XTDB 1.x	XTDB 2.x (early access)
Status	Stable	Experimental (pre-alpha)
Initial Stable Release	2019	TBD
Query languages	EDN Datalog	XTQL + SQL:2011
Bitemporal Querying	Timeslice only (point-in-time)	Fully bitemporal - SQL:2011 and beyond
Query focus	OLTP	OLAP + OLTP ('HTAP')
Storage & Compute	Coupled (nodes are full replicas)	Separated (cost-effective scale out)
Primary Storage Format	Custom Key-Value encodings	Columnar Apache Arrow
-	-	-
Immutable Semantics	Yes	Yes
Online Transactions (ACID, strong consistency)	Yes	Yes
Always-On Bitemporality	Yes	Yes
Dynamism (ad-hoc graph joins, union typing, schemaless, etc.)	Yes	Yes

Repo Layout

2.x is split across multiple projects which are maintained within this repository.

api contains the user API to XTDB 2.x.
core contains the main functional components of XTDB along with interfaces for the pluggable storage components (Kafka, S3 DB etc.). Implementations of these storage options are located in their own projects.
http-server and http-client-jvm contain the HTTP server implementation, and a remote client for JVM users.
Storage and other modules are under modules. Modules are published to Maven independently so that you can maintain granular dependencies on precisely the individual components needed for your application.

Questions, Thoughts & Feedback

We would love to hear from you: [email protected]

License and Copyright

XTDB is licensed under the Mozilla Public License, version 2 or (at your option) any later version.

xtdb-tutorial's People

Contributors

Stargazers

Watchers

Forkers

bombaywalla zampino johantonelli abhinavomprakash jgomo3 walterl ovistoica jf

xtdb-tutorial's Issues

entity-history does not retrieve recent transactions

Hi, I was trying out the xtdb tutorial at nextjournal, and I've noticed there are no data returned in the first entity-history call. The author was indeed expecting all transaction data to be returned instead.

https://nextjournal.com/xtdb-tutorial/delete

How to reproduce.

create a test clojure file to create a node, add the transactions in the example and query about the history

(ns tutorial.test
  (:require [xtdb.api :as xt]
            ))


(def node (xt/start-node {}))

(xt/submit-tx node
              [[::xt/put {:xt/id :kaarlang/clients
                          :clients [:encompass-trade]}
                #inst "2110-01-01T09"
                #inst "2111-01-01T09"]

               [::xt/put {:xt/id :kaarlang/clients
                          :clients [:encompass-trade :blue-energy]}
                #inst "2111-01-01T09"
                #inst "2113-01-01T09"]

               [::xt/put {:xt/id :kaarlang/clients
                          :clients [:blue-energy]}
                #inst "2113-01-01T09"
                #inst "2114-01-01T09"]

               [::xt/put {:xt/id :kaarlang/clients
                          :clients [:blue-energy :gold-harmony :tombaugh-resources]}
                #inst "2114-01-01T09"
                #inst "2115-01-01T09"]])

(def res
  (xt/entity-history
   (xt/db node #inst "2116-01-01T09")
   :kaarlang/clients
   :desc
   {:with-docs true}))

res

Fire up a REPL and eval the file. The key here is to eval the file in one go.
res is [].

Is this a timing issue with db synching? if I (xt/sync node) just before the query, it works.

Thanks

"2: Put" link to "3: Datalog" is broken

Should link to /datalog not /datalog-queries. See https://juxt-oss.zulipchat.com/#narrow/stream/194466-xtdb-users/topic/broken.20link.20in.20the.20tutorials/near/253900264

Update "SNAPSHOT" to `1.19.0` in `deps.edn`

Fix evict-meteor image

It looks like we somehow have a badly cropped meteor image

We should replace it with the old one https://github.com/crux-labs/crux-tutorial/blob/main/images/6b-evict-meteor.jpg

Tutorial 7: Evict - Image not visible

At the ending of the "Departure" section of the Evict tutorial, it seems like there must be an image but all that is displayed for me is:
![ice.jpg][nextjournal#file#f0026e73-9244-45f5-ac13-bab33663707d]

Evict has has a run error in NextJournal

This doesn't seem related to the code, it hits an error before getting there.

`(require '[xtdb.api :as xt])` vector should not be quoted

CIDER is giving me this:
Found lib name 'xtdb.api' containing period with prefix 'quote'. lib names inside prefix lists must not contain periods

No issue with simply doing (require [xtdb.api :as xt])

Automated deployments?

See conversation from #5 ... may involve begging for APIs from the @nextjournal folks, though. ;)

Typos

Lesson 5 ("match"): s/sees the 1000 credits move from/sees the 100 credits move from/
Lesson 7 ("evict"): There's a reference to history-descending, which I assume was changed to entity-history as used in the example code.
via https://clojurians.slack.com/archives/CG3AM2F7V/p1660480043092479 (thanks for reporting!)

Clarify `match`

From Sean Corfield in Clojurians:

the discussion of match in the tutorial was very confusing for us and seemed... a bit pointless... since it seems it has to exactly match the most recent existing document in all fields? Specifically for the trading example, it seems like the precondition for the trade TX is that both parties have "enough" (enough credits and enough product) so an exact match there just seems weird... and it doesn't explain that the only way to tell if a match-based TX "worked" is to (await it and then?) ask if it was actually committed... at which point, wouldn't you need to re-fetch the original docs, re-compute the changes, and then re-run the TX?

Extracted from #4

Remove references to `:args`

See: https://clojurians.slack.com/archives/CG3AM2F7V/p1630609495040600

Keep sample data in repo

commodities.txt, sample-data.txt, etc.

Bug in filter-appearance in tutorial 3

There is an insidious typo in the example code for filter-appearance.

Currently:

(defn filter-appearance
  [description]
  (xt/q (xt/db node)
        '{:find [name IUPAC]
          :where [[e :common-name name]
                  [e :IUPAC-name IUPAC]
                  [e :appearance appearance]]
          :in [apperance]}
        description))

the :in parameter is misspelled.

The usage example (filter-appearance "white solid") always returns the correct result since there is only one item with an :appearance attribute.

Clarify `delete`

Users should not confuse it for evict.

Issue extracted from #4

Process Clojure Los Angeles Feedback

Since these messages will disappear from Slack in a matter of days, copying them here:

seancorfield 3:46 AM

@steven Deobald We did run into a couple of things -- they'll be obvious from the recording but I figure I can give you a head start: the link in the planets tutorial to the datalog section is a 404 (it has /datalog-queries instead of just /datalog); the discussion of match in the tutorial was very confusing for us and seemed... a bit pointless... since it seems it has to exactly match the most recent existing document in all fields? Specifically for the trading example, it seems like the precondition for the trade TX is that both parties have "enough" (enough credits and enough product) so an exact match there just seems weird... and it doesn't explain that the only way to tell if a match-based TX "worked" is to (await it and then?) ask if it was actually committed... at which point, wouldn't you need to re-fetch the original docs, re-compute the changes, and then re-run the TX?

I think we also struggled with how exactly a delete works, since it was pitched as helping support data privacy laws, but then there's evict which seems more suitable for things like GDPR "right to forget"?

dorab 12:49 PM

Thanks to everyone who attended and participated. Thanks to @nate for driving. Definitely a lot of subtleties around how delete works and implications on the history of each entity.

@steven Deobald In addition to what @seancorfield mentioned about the tutorial, the files (e.g., the commodities) associated with some of the tutorials seemed to have the crux ids and not the xtdb ids. Also, the manifest update at the end of the Saturn tutorial has the "BITEMP" badge missing.

dorab 12:59 PM

In any new tutorials, it would be great to have a few exercises after each (sub-)section that can be used by the viewer to better understand the topic and perhaps explore some of the edge cases.

Steven Deobald

@seancorfield @dorab Thank you! That's really helpful feedback. The Space Adventure tutorial is much older than the Learn Datalog tutorial and also grew as XT (well, Crux) grew. I think the evolutionary nature of the tutorial probably explains the confusion between delete and evict, and that's definitely something we need to correct. For the record, evict is the operation you want for non-functional things like GDPR compliance. delete is for the business domain and leaves the entity history intact (as I'm sure you saw).

Everyone is welcome to create GitHub issues (https://github.com/xtdb/xtdb-tutorial/issues) if they like, but I'm also happy to transcribe your feedback when I get a minute.