moby / datakit Goto Github PK
View Code? Open in Web Editor NEWConnect processes into powerful data pipelines with a simple git-like filesystem interface
License: Apache License 2.0
Connect processes into powerful data pipelines with a simple git-like filesystem interface
License: Apache License 2.0
The current structure is a bit odd:
sdk
Dockerfile builds the i9p
library, which is a combination of a 9p pseudo-filesystem support library and most of irmin9p
. It does not appear to contain an SDK.db
Dockerfile builds the rest of irmin9p
(the command-line interface and transports) on top of sdk
.I suggest:
Any objection to this @samoht?
Is there a maximum supported size for files? I get the following error when trying to ls
a file with size of 1.5gb:
fs9p [DEBUG] S ((tag (1))
(payload
(Err
((ename "Out of memory") (errno ())))))
With a slightly smaller file (700mb) I get the following error:
fs9p [DEBUG] S ((tag (1))
(payload
(Err
((ename
"Unix.Unix_error(Unix.EACCES, \"open\", \"/data/.git/objects/pack/pack-5b5629393a4b074d789894c51cc237f0cc1232f2.idx\")")
(errno ())))))
Smaller files I can successfully ls
and cat
This would probably involves some local caching to be efficient
Looks like the inode number changes:
root@ucp:/srv/moby/branch/test/transactions/a/rw# strace -v -y cp foo bar
[...]
stat("bar", 0x7ffcd0cb1740) = -1 ENOENT (No such file or directory)
stat("foo", {st_dev=makedev(0, 153), st_ino=18456, st_mode=S_IFREG|0664, st_nlink=1, st_uid=4294967294, st_gid=4294967294, st_blksize=8192, st_blocks=0, st_size=0, st_atime=0, st_mtime=0, st_ctime=0}) = 0
stat("bar", 0x7ffcd0cb14d0) = -1 ENOENT (No such file or directory)
open("foo", O_RDONLY) = 3</srv/moby/branch/test/transactions/a/rw/foo>
fstat(3</srv/moby/branch/test/transactions/a/rw/foo>, {st_dev=makedev(0, 153), st_ino=18457, st_mode=S_IFREG|0664, st_nlink=1, st_uid=4294967294, st_gid=4294967294, st_blksize=8192, st_blocks=0, st_size=0, st_atime=0, st_mtime=0, st_ctime=0}) = 0
Currently you can get notified of new PRs and see the current head ID, but you can't easily fetch them. Datakit should probably download the commit before returning the ID to the user.
Seems that writing the url
file removes the description, when updating a GitHub PR status.
See https://github.com/docker/pinata/issues/77
Need a way to sync two DB instances, possibly over 9p. e.g. open /export, write the heads you have and read the slice, then pipe that into /import at the other end.
This looks like a project with lots of good ideas and well-documented protocols:
The Linux 9p client writes large files (e.g. the Moby iso) 8KB at a time. Datakit handles each write by allocating a new buffer, copying the old version into it, and adding the new 8KB. This gets slow quickly ;-)
Change the type for files in Ivfs_tree
from
| Blob of Cstruct.t
to
| Blob of Blob.t
and make a suitable Blob
module for efficient non-destructive updates. e.g. copy-on-write, or just use a list of blocks in reverse order (and concat them all together at the end).
The com.docker.db
process memory usage is growing up regularly, even when there is no activity -- this can grow very large over time.
Need to be fixed ASAP
Reported by @aluzzardi and @stevvooe on https://docker.slack.com/archives/pinata-dev/p1459565205001093
On:
root@ucp:~# uname -a
Linux ucp 4.3.0-0.bpo.1-amd64 #1 SMP Debian 4.3.5-1~bpo8+1 (2016-02-23) x86_64 GNU/Linux
Moby build under Datakit failed, with dmesg reporting:
[259075.913336] ------------[ cut here ]------------
[259075.913347] WARNING: CPU: 2 PID: 6525 at /home/zumbi/linux-4.3.5/fs/inode.c:273 drop_nlink+0x39/0x40()
[259075.913353] Modules linked in: 9p 9pnet fscache xt_nat xt_tcpudp veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables x_tables nf_nat nf_conntrack br_netfilter bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c loop dm_mod intel_rapl iosf_mbi x86_pkg_temp_thermal coretemp crct10dif_pclmul crc32_pclmul jitterentropy_rng sha256_ssse3 sha256_generic hmac drbg ansi_cprng aesni_intel evdev aes_x86_64 lrw pcspkr gf128mul glue_helper ablk_helper cryptd autofs4 ext4 crc16 mbcache jbd2 xen_netfront xen_blkfront crc32c_intel
[259075.913404] CPU: 2 PID: 6525 Comm: git Tainted: G W 4.3.0-0.bpo.1-amd64 #1 Debian 4.3.5-1~bpo8+1
[259075.913409] 0000000000000000 0000000073b84b68 ffffffff812e1ac9 0000000000000000
[259075.913415] ffffffff81074451 ffff88001064ba80 ffff88006cc23060 0000000000000200
[259075.913421] ffff8800102a6c18 ffff8800e975b000 ffffffff811ea029 ffff88001064ba80
[259075.913426] Call Trace:
[259075.913431] [<ffffffff812e1ac9>] ? dump_stack+0x40/0x57
[259075.913435] [<ffffffff81074451>] ? warn_slowpath_common+0x81/0xb0
[259075.913438] [<ffffffff811ea029>] ? drop_nlink+0x39/0x40
[259075.913443] [<ffffffffa02c6791>] ? v9fs_remove+0x71/0xb0 [9p]
[259075.913447] [<ffffffff811dc6dc>] ? vfs_rmdir+0xac/0x120
[259075.913451] [<ffffffff811dfbd0>] ? do_rmdir+0x1e0/0x200
[259075.913455] [<ffffffff8158a9f6>] ? system_call_fast_compare_end+0xc/0x6b
[259075.913458] ---[ end trace ce481105300b8298 ]---
Current status:
Cloning into 'docker.git'...
error: Unable to create /database/branch/initrd-of-5da84d3c468a6ee03d550e74b58bba9439e362a8/transactions/log/rw/alpine/packages/docker/docker.git/.git/HEAD
The immediate issue is that we don't support rename in transactions. However, running Git inside a transaction is probably a bad idea anyway (will need cross-directory rename sooner or later, which 9p doesn't support).
Ideally, I think we want to handle cloning of repositories inside Datakit itself (which would be more efficient than downloading the whole thing on every build). However, a short-term fix would be to clone the repository into /tmp or similar.
Following instructions in README:
tal@Thomass-MBP ~/d/db> make
docker build -t datakit-db .
Sending build context to Docker daemon 233.5 kB
Step 1 : FROM datakit-sdk
---> 9ddeae81a768
Step 2 : COPY src /home/opam/src/db
---> Using cache
---> 497f2b16c136
Step 3 : RUN opam pin add db.dev /home/opam/src/db
---> Using cache
---> c0a41ad7457a
Step 4 : RUN sudo chown -R opam.nogroup /home/opam/src
---> Using cache
---> 0735a3e9d444
Step 5 : WORKDIR /home/opam/src/db
---> Using cache
---> 5c6e5fbdb27b
Step 6 : RUN opam config exec make
---> Running in c169cb14edcc
make: *** No rule to make target '../../VERSION', needed by 'version.ml'. Stop.
The command '/bin/sh -c opam config exec make' returned a non-zero code: 2
make: *** [sdk] Error 1
Linux reported:
Error checking context: 'readdirent: errno 526'.
The Datakit logs show:
+22823959us fs9p [DEBUG] C ((tag (1))
(payload
(Walk ((fid 6) (newfid 7) (wnames ())))))
+22824016us fs9p [DEBUG] S ((tag (1)) (payload (Walk ((wqids ())))))
+22824176us fs9p [DEBUG] C ((tag (1))
(payload
(Open
((fid 7)
(mode
((io Read) (truncate false)
(rclose false) (append false)))))))
+22824287us fs9p [DEBUG] S ((tag (1))
(payload
(Open
((qid
((flags (Directory)) (version 0)
(id 92)))
(iounit 0)))))
+22824465us fs9p [DEBUG] C ((tag (1))
(payload
(Read ((fid 7) (offset 0) (count 8168)))))
+22824611us fs9p [DEBUG] S tag 1 Read(len(data): 8129)
+22824847us fs9p [DEBUG] C ((tag (1))
(payload
(Read ((fid 7) (offset 8129) (count 39)))))
+22824866us fs9p [DEBUG] S ((tag (1))
(payload
(Err
((ename "Buffer too small") (errno ())))))
So, it tried to read into a buffer, got a short read and retried with the remaining space in the buffer, which was too small for even a single entry.
The spec says:
For directories, read returns an integral number of directory entries
With auto-push mode, DataKit takes a long time to start. It prints lots of messages like this (one per branch I think), with a pause after each one:
*** Please tell me who you are.
Run
git config --global user.email "[email protected]"
git config --global user.name "Your Name"
to set your account's default identity.
Omit --global to set the identity only in this repository.
Can we speed this up?
Currently, everything is stored in one local repository and there is no way to pull/push from other repos. We would like to add a way to allow that, for instance:
mkdir /remotes/origin
echo https://github.com/docker/datakit > /remotes/origin/url
echo master > /remotes/origin/pull
89bfdb2c4f920bc043c044c1945bb2e2d836238d
Seems that terminal likes to seek on stdout of files put into background (not sure why, need to investigate):
+321731243us fs9p [DEBUG] S tag 1 Read(len(data): 8)
+321731281us fs9p [DEBUG] S tag 2 Read(len(data): 8)
+321762447us fs9p [DEBUG] C ((tag (1))
(payload
(Read
((fid 9) (offset 16) (count 8168)))))
+321762511us vfs [DEBUG] offset:16 current-offset:16
+321762530us fs9p [DEBUG] S tag 1 Read(len(data): 0)
+321762808us fs9p [DEBUG] C ((tag (2))
(payload
(Read
((fid 10) (offset 8) (count 8168)))))
+321762977us vfs [DEBUG] offset:8 count:16
+321763229us fs9p [DEBUG] S ((tag (2))
(payload
(Err
((ename "Attempt to seek in stream")
(errno ())))))
+321763377us fs9p [DEBUG] C ((tag (1))
(payload
(Read
((fid 9) (offset 16) (count 8168)))))
+321763559us vfs [DEBUG] offset:16 current-offset:16
+321763578us vgithub [DEBUG] wait
+321803168us fs9p [DEBUG] C ((tag (2))
(payload
(Read
((fid 10) (offset 8) (count 8168)))))
+321803252us vfs [DEBUG] offset:8 current-offset:16
+321803278us fs9p [DEBUG] S ((tag (2))
(payload
(Err
((ename "Attempt to seek in stream")
(errno ())))))
+321803514us fs9p [DEBUG] C ((tag (2))
(payload
(Read
((fid 10) (offset 8) (count 8168)))))
+321803670us vfs [DEBUG] offset:8 current-offset:16
+321803702us fs9p [DEBUG] S ((tag (2))
(payload
(Err
((ename "Attempt to seek in stream")
(errno ())))))
+321803967us fs9p [DEBUG] C ((tag (2))
(payload
(Read
((fid 10) (offset 8) (count 8168)))))
+321804020us vfs [DEBUG] offset:8 current-offset:16
+321804039us fs9p [DEBUG] S ((tag (2))
(payload
(Err
((ename "Attempt to seek in stream")
(errno ())))))
+321804771us fs9p [DEBUG] C ((tag (2)) (payload (Clunk ((fid 10)))))
+321804959us fs9p [DEBUG] S ((tag (2)) (payload (Clunk ())))
To repro:
$ cat /db/github.com/samoht/test/pr/8/updates
< PRESS CTL-G >
$ bg
$ echo pending db/github.com/samoht/test/pr/8/state
pending
pending
cat: read error: No error information
The Go hooks often start to fail after a few hours. Logs show:
INFO[58782] Received event: status 30f87880-3917-11e6-99eb-2033dc3be24e host=... name=... user=docker
2016/06/23 07:50:45 Dialling tcp 127.0.0.1:5640
DEBU[58785] user=docker, repo=...
2016/06/23 07:51:02 I don't know how to flush: leaking 1
2016/06/23 07:51:02 fatal error reading msg: read tcp 127.0.0.1:5640: use of closed network connection
At the DataKit end, the client always seems to die at around fid 127. Maybe this leaking
message is related?
See https://github.com/docker/pinata/issues/52
It'd be great if DB users can registers their own merge functions to be used when a transaction is committed, (possibly) written in different languages (Lua, Go, OCaml, etc.) or using interesting combinators that we will provide later (such as CRDTs).
Currently the merge functions are statically decided and the one being used is the simplest one: if a file modified in the transaction is modified in the main branch, the then transaction returns a conflict.
It is not clear that we want to store that information in the database itself, as we clearly need some session (secret) data as private keys.
Ideally we would re-use and integrate existing auth* mechanism, need to check what we can do here.
Looks like lwt doesn't detect libev
on Alpine 3.3 and falls back to select mode. Perhaps select mode is buggy (I guess not many people use it)?
See https://github.com/docker/pinata/issues/45
Currently Irmin conflates the Git metadata which are not directories.
The full list of possible Git metadata is here:
type perm = [ `Normal | `Exec | `Link | `Dir | `Commit ]
which means that symlink and executable are considered as normal file. (and not sure what to do with the Commit bit, though...). A solution would be to (i) write an Irmin backend which can read/write Git metadata; (ii) when importing a Git repository into the DB, import both the contents and the metadata. When serialising the DB into the container filesystem, use both the imported contents and metadata.
Note: we might need to have a more general system based on continuity at one point.
See https://github.com/docker/pinata/issues/44
This functionality can be provided by stevvooe/contintuity.
Expected Result
When attempting to create a directory in the database root, Datakit should respond to without error.
Actual Result
Datakit responds with Rerror
, Directory is read-only
Information
Discovered this while writing C# bindings for Datakit.
When you attempt to create a directory in the database root, branch
in this example, Datakit will respond with an Rerror
message claiming that the Directory is read-only
+4968us Datakit Starting com.docker.db...
+5041us Datakit [DEBUG] Using Git-format store "/data"
+5516us irmin-io [DEBUG] mkdir /data/.git
+5727us irmin-io [DEBUG] mkdir /data/.git/tmp
+6009us irmin-io [INFO] Writing /data/.git/HEAD (/data/.git/tmp/HEADd54231write)
+6405us vgithub [DEBUG] Datakit does not use the Github bindings
+6580us Datakit [DEBUG] Waiting for connections on socket "tcp://0.0.0.0:5640"
+101477671us Datakit [DEBUG] New unix client
+101477709us ivfs-remote [DEBUG] create
+101485531us fs9p [DEBUG] C ((tag (0))
(payload
(Version
((msize 16384) (version TwoThousand)))))
+101485591us fs9p [DEBUG] S ((tag (0))
(payload
(Version
((msize 16384) (version TwoThousand)))))
+101485685us fs9p [INFO] Using protocol TwoThousand msize 16384
+101490338us fs9p [DEBUG] C ((tag (1))
(payload
(Attach
((fid 1) (afid 0) (uname Dave)
(aname /) (n_uname ())))))
+101490396us fs9p [DEBUG] S ((tag (1))
(payload
(Attach
((qid
((flags (Directory)) (version 0)
(id 5)))))))
+101494532us fs9p [DEBUG] C ((tag (2))
(payload
(Walk ((fid 1) (newfid 2) (wnames ())))))
+101494581us fs9p [DEBUG] S ((tag (2)) (payload (Walk ((wqids ())))))
+101495497us fs9p [DEBUG] C ((tag (3))
(payload
(Walk ((fid 2) (newfid 3) (wnames ())))))
+101495527us fs9p [DEBUG] S ((tag (3)) (payload (Walk ((wqids ())))))
+101497487us fs9p [DEBUG] C ((tag (4))
(payload
(Create
((fid 3) (name branch)
(perm
((owner (Read Write Execute))
(group (Read Execute))
(other (Read)) (is_directory true)
(append_only false)
(exclusive false) (is_mount false)
(is_auth false) (temporary false)
(is_device false) (is_symlink false)
(is_hardlink false)
(is_namedpipe false)
(is_socket false) (is_setuid false)
(is_setgid false) (is_any false)))
(mode
((io Read) (truncate false)
(rclose false) (append false)))
(extension ())))))
+101497586us fs9p [DEBUG] S ((tag (4))
(payload
(Err
((ename "Directory is read-only")
(errno ())))))
As you can see, a subsequent Walk
and Create
operations within the branch
directory DO succeed...
+101515630us fs9p [DEBUG] C ((tag (5)) (payload (Clunk ((fid 3)))))
+101515671us fs9p [DEBUG] S ((tag (5)) (payload (Clunk ())))
+101516094us fs9p [DEBUG] C ((tag (6))
(payload
(Walk
((fid 2) (newfid 2) (wnames (branch))))))
+101516133us fs9p [DEBUG] S ((tag (6))
(payload
(Walk
((wqids
(((flags (Directory)) (version 0)
(id 4))))))))
+101516351us fs9p [DEBUG] C ((tag (7))
(payload
(Walk ((fid 2) (newfid 4) (wnames ())))))
+101516386us fs9p [DEBUG] S ((tag (7)) (payload (Walk ((wqids ())))))
+101516688us fs9p [DEBUG] C ((tag (8))
(payload
(Create
((fid 4) (name master)
(perm
((owner (Read Write Execute))
(group (Read Execute))
(other (Read)) (is_directory true)
(append_only false)
(exclusive false) (is_mount false)
(is_auth false) (temporary false)
(is_device false) (is_symlink false)
(is_hardlink false)
(is_namedpipe false)
(is_socket false) (is_setuid false)
(is_setgid false) (is_any false)))
(mode
((io Read) (truncate false)
(rclose false) (append false)))
(extension ())))))
+101516792us fs9p [DEBUG] S ((tag (8))
(payload
(Create
((qid
((flags (Directory)) (version 0)
(id 13)))
(iounit 0)))))
I've verified the same behaviour with @djs55's Go bindings so I think this is likely a problem with Datakit...
This is the same whether running over a named pipe (on Windows) or over TCP to Datakit running in a Linux container.
Looking inside the /github.com
directory causes Datakit to quit.
GitHub support should probably be optional, and the server should continue running if the token is missing or doesn't work.
Can allow to download and upload big blobs to S3 directly using the filesystem interface.
I assume this should work:
github.com/docker/pinata/pr/3661/status/ci> mkdir pinataci
Directory is read-only
Especially for how to use the Dockerfile. Also it seems like https://github.com/docker/datakit/blob/master/ci/Dockerfile needs to add map.sh since it is sourced from ci.sh
We need a way to describe pipeline syntax. Initially, shell scripts are fine, but we might want something nicer. @talex5 can you describe what you have in mind and what is missing?
In running through the tutorial, I try the following command to watch a branch.
cat /db/branch/4.03/watch/tree.live
After an initial irmin.watch [DEBUG] watch [1: 0k/0g]: id=0
,
the server starts start to echo
irmin-io [DEBUG] polling /data/.git/refs/heads: no changes!
every second. When I Ctrl-C
the client cat
command the irmin-io
logs continue, with no irmin.watch
command signifying the watch has ended.
Docker version 1.11.2, build b9f10c9
I am running the datakit server as
docker run -it --net datakit-net --name datakit -v $(realpath .):/data docker/datakit
and the client as
docker run -it --privileged --net datakit-net docker/datakit:client
Raw-9p is quite a low-level interface to the DB. We should design a higher-level Go interface. An interface is already there in pinata and variious discussion have been taken place about that in the past:
This would allow to watch for Github events directly in the 9p filesystem, and interact with PR (can toggle the green tick on successful builds for instance)
Similar to #8 but for OCaml instead of Go. We already have some basic bindings:
https://github.com/docker/vpnkit/tree/master/src/ofs
It would make sense to put all the language bindings in the datakit repo instead (for instance under a bindings/
directory)
There is no GC in Irmin (we just use git gc
for now). This needs to be fixed if we want to version control everything. Moreover, we need to understand which hooks need to be exposed by the storage backend so we can register the hooks that we need to have a high-performance GC.
See mirage/irmin#71 and http://lists.xenproject.org/archives/html/mirageos-devel/2015-10/msg00040.html
In the pinata database, everytime a key is changed the commit seems to have a duplicate parent:
$ git cat-file d52cdb50bd829f13c7c6aaacfa0fe8ecc3092660 -p
tree 145d2e6a4ca8f9c665947085dd2b147bd09ca019
parent 910c3dc127eb6e35a88d8316eb1d96232acb39b2
parent 910c3dc127eb6e35a88d8316eb1d96232acb39b2
author irmin9p <[email protected]> 1456743397 +0000
committer irmin9p <[email protected]> 1456743397 +0000
(no commit message)
$ git show d52cdb50bd829f13c7c6aaacfa0fe8ecc3092660
commit d52cdb50bd829f13c7c6aaacfa0fe8ecc3092660
Merge: 910c3dc 910c3dc
Author: irmin9p <[email protected]>
Date: Mon Feb 29 10:56:37 2016 +0000
(no commit message)
diff --cc com.docker.driver.amd64-linux/on-sleep
index 316fcb7,316fcb7..280e4ae
--- a/com.docker.driver.amd64-linux/on-sleep
+++ b/com.docker.driver.amd64-linux/on-sleep
@@@ -1,1 -1,1 +1,1 @@@
--shutdown
++freeze
diff --cc com.docker.driver.amd64-linux/schema-version
index d8263ee,d8263ee..e440e5c
--- a/com.docker.driver.amd64-linux/schema-version
+++ b/com.docker.driver.amd64-linux/schema-version
@@@ -1,1 -1,1 +1,1 @@@
--2
++3
Is this an issue with the client library or with i9p?
git init --bare .git
)git push
to master branch.master
. git branch
does show master
.foo
branch and make a commit, using irmin9p.foo
. git branch
shows only foo
(master
has gone!).git push
master again. Now everything works.Building (with docker build .
) currently fails at RUN opam install alcotest
with:
[WARNING] External solver failed with inconsistent return value. Request saved
to "/home/opam/.opam/log/solver-error-9-1.cudf"
Something wrong with the ocaml/opam:alpine
image @avsm?
Currently, the Git history in pinata looks like this:
commit 7be99481c37abb6a4d92a3686d72491c793af810
Merge: 390f6e1 390f6e1
Author: irmin9p <[email protected]>
Date: Mon Feb 29 10:56:34 2016 +0000
(no commit message)
commit 390f6e1b55ab3e7f0b5aaaff9b1d33c779d5d933
Merge: 4209536 4209536
Author: irmin9p <[email protected]>
Date: Mon Feb 29 10:56:31 2016 +0000
(no commit message)
Would be great to make this a bit more useful...
Moby (Linux 4.1.18) requires an EOF to push data to the application, with two EOFs in a row meaning a real end-of-file. Perhaps older versions treat a single EOF as end-of-file, because I see:
user@nuc1:~$ uname -a
Linux nuc1 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 GNU/Linux
user@nuc1:~$ cat /mnt/datakit/branch/master/head.live
89bfdb2c4f920bc043c044c1945bb2e2d836238d
user@nuc1:~$
I need to make a datakit server which
What do you think is the best approach?
After echo master > fetch
in a remote dir on a client I get the following trace on the datakit server
+158506053us git.sync [DEBUG] RECEIVED: b5a3d612f75867c7dfd00c48ecf13f325f1ea21c HEAD\x00multi_ack thin-pack side-band side-band-64k ofs-de[..] (197)
+158506147us git.sync [DEBUG] PacketLine.input
+158506174us git.sync [DEBUG] PacketLine.input_raw
+158507520us git.sync [DEBUG] RECEIVED: b5a3d612f75867c7dfd00c48ecf13f325f1ea21c refs/heads/master\x0A (59)
+158507560us git.sync [DEBUG] PacketLine.input
+158507588us git.sync [DEBUG] PacketLine.input_raw
+158507619us git.sync [DEBUG] RECEIVED: a95c7d33869a7e485a135b0b5d6887e0ec75ebcb refs/remotes/origin/master\x0A (68)
+158507656us git.sync [DEBUG] PacketLine.input
+158507683us git.sync [DEBUG] PacketLine.input_raw
+158509422us git.sync [DEBUG] RECEIVED: FLUSH
+158509460us git.sync [DEBUG] listing:
CAPABILITIES:
multi_ack, thin-pack, side-band, side-band-64k, ofs-delta, shallow, no-progress, include-tag, multi_ack_detailed, symref=HEAD:refs/heads/master, agent=git/2.8.3
REFERENCES:
a95c7d33869a7e485a135b0b5d6887e0ec75ebcb refs/remotes/origin/master
b5a3d612f75867c7dfd00c48ecf13f325f1ea21c refs/heads/master
b5a3d612f75867c7dfd00c48ecf13f325f1ea21c HEAD
+158509535us git.sync [DEBUG] Sync.fetch_commits b5a3d612f75867c7dfd00c48ecf13f325f1ea21c
+158509577us git.fs [DEBUG] mem: cache miss!
+158509624us git.fs [DEBUG] mem: cache miss!
+158509652us git.fs-packed [DEBUG] list /data/.git
+158509897us git.fs-packed [DEBUG] mem_in_pack 5b5629393a4b074d789894c51cc237f0cc1232f2:b5a3d612f75867c7dfd00c48ecf13f325f1ea21c
+158509926us git.fs-packed [DEBUG] read_pack_index 5b5629393a4b074d789894c51cc237f0cc1232f2
+158509968us irmin-io [INFO] Reading /data/.git/objects/pack/pack-5b5629393a4b074d789894c51cc237f0cc1232f2.idx
+158510013us git.sync [DEBUG] PHASE1
+158510036us git.sync [DEBUG] Upload.phase1
+158510064us git.sync [DEBUG] Upload.output
+158510099us git.sync [INFO] SENDING: "0069want b5a3d612f75867c7dfd00c48ecf13f325f1ea21c agent=git/ogit.1.8.0 side-band-64k ofs-delta thin-pack\n"
+158510195us git.sync [INFO] SENDING: "0000"
+158510233us git.sync [DEBUG] PHASE2
+158510380us git.pack-index [DEBUG] create: entering fanout table (ofs=8)
+158510513us git.pack-index [DEBUG] create: n_hashs:3
+158510561us git.pack-index [DEBUG] create: entering hash listing (ofs=1032)
+158510596us git.pack-index [DEBUG] create: entering crc checksums (ofs=1092)
+158510628us git.pack-index [DEBUG] create: entering packfile offsets (ofs=1104)
+158510658us git.pack-index [DEBUG] create: entering large packfile offsets (ofs=1116)
+158510711us git.pack-index [DEBUG] mem: b5a3d612f75867c7dfd00c48ecf13f325f1ea21c
+158510741us git.pack-index [DEBUG] find_offset: b5a3d612f75867c7dfd00c48ecf13f325f1ea21c
+158510762us git.pack-index [DEBUG] get_hash_idx: b5a3d612f75867c7dfd00c48ecf13f325f1ea21c
+158510790us git.pack-index [DEBUG] find_offset: found:12
+158510809us git.pack-index [DEBUG] mem: true
+158510830us git.graph [DEBUG] closure
+158510871us git.fs [DEBUG] mem: cache miss!
+158510901us git.fs [DEBUG] mem: cache miss!
+158510923us git.fs-packed [DEBUG] list /data/.git
+158510982us git.fs-packed [DEBUG] mem_in_pack 5b5629393a4b074d789894c51cc237f0cc1232f2:b5a3d612f75867c7dfd00c48ecf13f325f1ea21c
+158511003us git.fs-packed [DEBUG] read_pack_index 5b5629393a4b074d789894c51cc237f0cc1232f2
+158511028us git.fs-packed [DEBUG] read_pack_index cache hit!
+158511048us git.pack-index [DEBUG] mem: b5a3d612f75867c7dfd00c48ecf13f325f1ea21c
+158511070us git.pack-index [DEBUG] find_offset: b5a3d612f75867c7dfd00c48ecf13f325f1ea21c
+158511090us git.pack-index [DEBUG] find_offset: cache hit!
+158511113us git.pack-index [DEBUG] mem: true
+158511182us git.sync [DEBUG] Upload.output
+158511216us git.sync [INFO] SENDING: "0032have b5a3d612f75867c7dfd00c48ecf13f325f1ea21c\n"
+158511259us git.sync [INFO] SENDING: "0032have a95c7d33869a7e485a135b0b5d6887e0ec75ebcb\n"
+158511303us git.sync [INFO] SENDING: "0009done\n"
+158511588us git.sync [DEBUG] Ack.input
+158511612us git.sync [DEBUG] PacketLine.input
+158511631us git.sync [DEBUG] PacketLine.input_raw
+158511655us git.sync [DEBUG] RECEIVED: ACK b5a3d612f75867c7dfd00c48ecf13f325f1ea21c\x0A (45)
+158511679us git.sync [DEBUG] PHASE3
+158511697us git.sync [DEBUG] Side_band.input
+158511718us git.sync [DEBUG] PacketLine.input_raw
+158511831us git.sync [DEBUG] RECEIVED: ACK a95c7d33869a7e485a135b0b5d6887e0ec75ebcb\x0A (45)
+158511870us git.sync [ERROR] Side_band: 65 is not a valid message type
+158511976us fs9p [DEBUG] S ((tag (1))
(payload
(Err
((ename "Sync.Make(IO)(Store).Error")
(errno ())))))
The url is [email protected]:/data
.
I am successfully able to ssh into the same url and git clone
from the same url.
On a fresh install of pinata, I got lots of dangling commits:
~/Library/Containers/com.docker.docker/Data/database [master L|โ] $ git fsck
Checking object directories: 100% (256/256), done.
dangling commit fe6900771b5713a058474bc5d1ef01382f387180
dangling commit fea2689b2773f602cfc769b5f7178269996debc4
dangling commit 0e922051e4e107552bae0297c398e9de88c7f6a9
dangling commit 11e6239f98cef8b560b0e73e7544bcc88631751c
dangling commit 2598545173b2d5189ef930b3699f14169b24f334
dangling commit 36990a0af1f1abf26cd33151bb0a96c4e6aa2395
dangling commit 3afdc07295054a9f43306f3e5a51d9af8e92f6ee
dangling commit 4708d2968b9859bd714f209997715ad70d220d7c
dangling commit 58fbf2460f7c8efc51ca5d512ac677cab437f9ca
dangling commit 674be8a6e54bbcdeca960261192804ddc4ecb566
dangling commit 6795e43af803f90e2d2013acc0b8db39bf5b3c0c
dangling commit 81c25fa23c220548b27a2e17dc0f95c7bbb3ba4a
dangling commit 89d69dd1aa43042abb7ed4278b304655ebb8e5b6
dangling commit 8e807b65b736cb63b351f79b50d0b1c2d800a5ac
dangling commit b82e074b08d9a2c3dd6d16466909d5ca80fe97fa
dangling commit f1256ee9e5cd87f3b78ab57bf4dfe1e09db2e03d
dangling commit f564753f8d22aac948591057e280170621feb6ce
dangling commit fb21948c079c0160fcbf0e747d4cf7f9d0bdb947
Again I'm not sure if the issue is with the client code or with i9p. Note: this doesn't seem to affect the repo sanity.
Currently with host the API docs on our local server, and everything is very manual. Would be nice if we can integrate this in the rest of the CI flow (e.g. re-publish automatically on every PR, in a location available to all).
Note: usually Github pages are great for that, but they don't work well for private repos (ie they are public).
Because Git doesn't store empty directories, mkdir
in a transaction's rw
directory doesn't change the tree. However, because we do want the new directory to appear in ls
listings we add it to extra_dirs
.
However, when we remove a directory we don't delete it from its parent's extra_dirs
and it still appears in the ls
output.
I'm on Centos 7 using datakit 51bcded and ran in one terminal:
$ ./scripts/start-datakit.sh
After building finished, in another term I ran:
datakit-mount
mount: mounting 172.17.0.2 on /db failed: No such device
Is this expected behavior?
The datakit-gh
hook should only listen for Commits made.
Adding a comment to a PR shouldn't re-trigger a check.
This is the tracking issue for the Pinata CI requirements. /cc @MagnusS
The short-term new features that are needed to deploy a local Datakit instance for CI purposes are:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.