It's a name-collision-cleanliness thing.

I don't care which. Travis?

Suppose the laptop turns off. We come back and can't write keys because our lease has expired in etcd. We need to safely recover.

As I couldn't get the kubernetes integration to work I tried to run latest from the container. It didn't work either but it should be a start.

#!/bin/bash
# Wrapper for launching kubelet via rkt-fly stage1.

set -e

AGRO_ACI="${AGRO_ACI:-quay.io/coreos/agro}"

exec /usr/bin/rkt run \
  --volume run,kind=host,source=/run \
  --mount volume=run,target=/run \
  --volume volume-data,kind=host,source=/tmp/foo \
  --volume volume-plugin,kind=host,source=/tmp/foo  \
  --insecure-options=image \
  $RKT_OPTS \
  --stage1-path=/usr/share/rkt/stage1-fly.aci \
  docker://${AGRO_ACI}:latest --exec=/go/bin/agromount -- $@ 2> /dev/null

core@localhost ~ $ cat /etc/rkt/auth.d/docker.json
{
    "rktKind": "dockerAuth",
    "rktVersion": "v1",
    "registries": ["quay.io"],
    "credentials": {
        "user": "coreos+agro",
        "password": "stuff"
    }
}

Let's simplify:

Right now, there's a separate INode store. I did that early on so that I could think about other abstractions, but it's time to centralize the storage.

My proposal is this: INodes consist of blocks, with special BlockRefs.

What's special about them? Well, BlockRefs would have a type now. We can steal a few bits (I'm thinking 16 from volume, meaning 64k potential blocktypes and 2^48 or 2.8e14 different volumes -- which is still plenty). The INode abstraction translates *model.INodes into blockns and back again.

This simplifies all the steps greatly (no second store) and has another advantage: The ring becomes even more the source of truth. If you want to modify how INodes are replicated, that's a change to the ring.

This has an added advantage: new and different blocktypes can now be known to the ring. For instance, it's probably unwise to keep Reed-Solomon blocks on all the same hosts as their original blocks (or probabalistically so). Or maybe you want to keep them on the same rack or whatever.

So too with alternate replication strategies.

And this simplifies RPCs; it's just blocks.

This also obviates #63.

Looks like this discussion is still happening: https://github.com/grpc/grpc-common/issues/284

(Tick marks to avoid linking the issue)

https://github.com/soheilhy/cmux is a reasonable workaround that exists today. Let's do it -- one port for HTTP API and internal gRPC. And when proper single-port support comes through grpc, then everyone wins.

We need to get ready to do the first release of agro. So, we need to document and automate the release process. rkt has a pretty good example of the things that should be covered: https://github.com/coreos/rkt/blob/master/Documentation/devel/release.md

NBD is the 'network block device': https://github.com/yoe/nbd/blob/master/doc/proto.md

nbd can run over tcp/ip and is supposed by the linux kernel.

i believe @barakmich is working on this.

If the connection dies, if a follower errors out and we need to cancel, having a semantic reason is important.

Start /client and an associated /cmd/agroctl -- the design of which is a little bit clever:

Start /client as another concrete implementation of the agro.Server interface. The difference being that the inode storage, the file abstraction, and so on, are fairly simple stubs into a new set of things: RemoteINodeStorage, RemoteBlockStorage, and RemoteMetadata. These basically make HTTP calls for all their implementations, calling the internal/http endpoint as a proxy and off we go.

The cool part is then agroctl (and the meta-client calls; I would guess things like ls) can self-host. The long-future goal being that it could talk directly to etcd for metadata if it needed, or could do local caching, or can do a lot of things -- which will make the FUSE client quite easy to implement well, without needing to proxy everything. In effect, a long-running FUSE daemon could itself be another node that doesn't register as storage, but speaks the native node-to-node interface.

Meanwhile, for other scenarios, the proxy works great.

As of PR #65, rebalancing is done by iterating over the store and copying over desired blocks/inodes. This is optimized for the case where you're deleting more than you're keeping. I would venture that most rebalances are only going to be deleting a small amount of values and would thus benefit from a strategy where you iterate over the blocks/inodes and mark what you're going to delete rather than copying what you'll keep.

Peers need to register with the MDS (extend the MDS interface to do this), but that's all. It's up to the client library to create (and commit) a (future) ring from a list of potential peers.

Registration should include a bunch of metadata about the host. How many blocks can it hold? What IP/port is it listening on? Does it have an optional name? What's its permanent UUID?

Along with this is a heartbeat. Heartbeat(UUID, status, timeout) gets saved to the MDS every so often.

(This works with #24 in that a client won't register or heartbeat!)

I really hate the name "types" for packages. It pretty much implies absolutely nothing and it isn't clear unless you look at the package source that it's for protos. We should call this package proto so that it's clear when you're using the types that they're protobufs.

We should provide a POSIX-compliant interface. The standard way to do that is with FUSE, and we can use Bazil's Go FUSE library to build it.

We may or may not want to have it as a separate command-line tool.

Add specific removal types to agroctl -- things which will block until the node is removed (safe or hard, in either case)

Adding an mode for really ramping up the rebalance traffic if you're draining also falls in this bug. Right now it's rate limited, but I think the better option will be a fast-divestment-of-data.

You cannot currently share host mounts via docker (until moby/moby#17034 lands). The author of this PR has a blogpost expanding on this PR here. rhdan has some food for thought on the subject here as well.

We have MFiles, let's use them to store blocks. This requires keeping a block map (easy enough).

We also have need of an INode store. Longer-term, this could be the same block store too -- if we split INodes (serialized protos) into blocks when written. But that's a deliberate abstraction. For now, a simple, separate boltdb will do the trick. It does mean we'll be using more space than we advertise, but running conservatively (ie, --size=something-less-than-half-a-disk) will be fine through a few phases.

To persist the temporary metadata? A flat file for that would be fine.

• Have instructions for doing the k8s pull secret
• Deploy daemon set
• Talk to etcd and agro machine to mkfs
• Use a flexvolume: https://github.com/kubernetes/kubernetes/tree/master/examples/flexvolume
  • Use a solution like CoreOS Kubernetes does for CNI plugins, using a mount flag
  • https://github.com/coreos/agro/blob/d3872a14f62579ae74b50a40ce64b6b671965e71/cmd/agrom

ideally agro would have a mount(8) helper, that can be invoked by e.g. mount -t agro -o etcd:32378,volume=blah ....

the use case here is systemd.mount units which operate solely through execution of mount(8).

As currently implemented, there's no need for the VolumeID in the INodeRef; it simply takes up space because right now there's only a single source of INodes, with no duplicates.

Now, there's two options:

1. Lose the VolumeID, have a single source of INodes:

• Pros:
  • Smaller Refs
  • Smaller MDS storage (no need for an inode printer for every volume)
• Cons:
  • Smaller representation
    • With 8k blocks, we're still representing up to 10^18 Yottabytes.
  • Potential etcd contention with a lot of write traffic.
    • Could be almost nullified by adding an "INC" operation in etcd, which atomically adds one to the value and returns that count

2. Keep the VolumeID, have an inode chain per volume

• Pros:
  • etcd contention only per-volume -- if we're mounting a lot of these, with an average ratio of almost 1:1 per machine that has it mounted, that means very little contention.
  • Friggin huge representation
    • 10^18 Yottabytes -- PER VOLUME. For an insane 10^37 Yottabytes of possible representation space. That's 10 tera-yottabytes.
• Cons:
  • More MDS storage (O(volumes * 4 bytes), so tiny)
  • Bigger refs (could be more smartly encoded)

We could waste some time building a scaffolding one (either via HTTP API or something) but why not go direct to the real plan.

• Write an "etcd-mds" bash script that fetches the latest etcd binary from github releases (if not exist) to a local gitignored dir and runs it in single-node v3 mode on a known port for ease of testing.
• Follow eg https://github.com/coreos/etcd/blob/master/etcdctlv3/command/put_command.go to connect the MDS interface to etcd. Single node concerns only, for the moment.

The good news is the v3API is hella cleaner and more powerful than it used to be. Build proto. Send proto on grpc. Done.

We should use Reed-Solomon or something similar to reduce replication storage overhead and increase resiliency to data loss.

Also, this may be useful.

We should expose Prometheus metrics for monitoring.

Go through and vet the logging/errors. Add more logs using a favorite logging package (it need not be capnslog, but I'm game/ Just needs to be BSD/Apache2). Add errors.go files on appropriate layers. Use them.

With the current structure of entrypoint.sh only a single etcd host can be added. Porentially this could be overridden by supplying a string of hosts as ETCD_HOST but in this case it will also require setting ETCD_PORT="" to avoid the automatic assignment of that variable.

We have a prometheus metrics API but it isn't documented. Do that including:

1. The metrics endpoint
2. An example query or two for prometheus
3. Explanations of the various metrics exported

https://github.com/coreos-inc/company/tree/master/eng/github#using-labels

We should have a command-line tool (agro), which has subcommands allowing a user to add a new file and get an existing file.

I should be able to stream content to a file in multiple chunks, without each chunk needing to be flushed to disk. The chunks should not be visible to other readers until I choose to explicitly commit.

tl;dr: Writes should be independent from commits in the low-level API.

Related to #147 I think we need to document how people can build applications on top of torus easily. I know that @barakmich has been working on the libraries and factoring things away so that building the next app on top of torus is easy to do but I think we should do this pre-launch. At least paint the rough picture.

Some of us don't have our go get enabled to use ssh. Instead of Go get maybe:

git clone [email protected]:coreos/agro.git $GOPATH/src/github.com/coreos/agro
cd $GOPATH/src/github.com/coreos/agro
go get ./...

Data which has been replaced should be periodically removed to free up space.

Add prometheus in a lot of places. Count block reads, count files opened, count files that are write-dirty, count metadata interactions, etc, etc....

a la https://github.com/google/cayley/blob/master/integration/integration_test.go

Sets up a single node (or connects as a client a la #24) and generates a couple files of different sizes to tmp. Writes them with fun rs/rep/crc/etc options. Measure how long it takes to write (randomly generated) 1K/100k/1M/100M files. Then measure how long it takes to read them back again.

We should support rolling upgrades.

We're happily passing contexts now, and timeouts can happen. In the case of timeout, a drop-and-reconnect is warranted. However, we currently have just one connection. We need to fix this.

Agro is the current name which is the name of the dog from Shadow of the Colossus.

Following the Colossus idea, but staying relevant to computer science, Bletchley Park was the where Colossus the computer was created by codebreakers (including Alan Turing) after the war. My proposal is to call the project "bletchley".

There is a parameter called INodeReplication, which defines how many replica each INode is expected to have.

What is the plan for handling Block replication? For N copy, we can simply reuse the INodeReplication parameter. But when doing RS encoding for blocks, the replication factor does not make a lot of sense.

Another question is that, are we trying to replicate the block on the same node as the INode?

error creating volume agroblock2: rpc error: code = 3 desc = "etcdserver: request is too large"

when trying to create a 512M block volume.

the marshalled inode size in createBlockVol ends up being ~1.75M, and etcd recently set a limit of 1.5M etcd-io/etcd@3556722

Hash out the lingo on the wiki

The very first question we are going to get about is how this thing fails in various scenarios:

• What happens in single machine failure?
• What happens during network partition between peers?
• What happens during network partition of etcd?

And in each of these scenarios we should provide an information about:

• Increased latency
• Read/write availability
• Estimated time to recovery

I think part of the experiment with this project is to see what people build outside of block devices. We should create a simple tool that people can cat in a stream and cat out a stream from an agro volume.

We should have good structured logging from the start. Looking for suggestions.

@barakmich and i talked briefly about having 9p interface for agro. it would allow agro to be mounted over the network, and there are a number of libraries implementing 9p, including a linux kernel driver.

https://www.kernel.org/doc/Documentation/filesystems/9p.txt

additionally if this is implemented it might be interesting to use agro w/ diod - https://github.com/chaos/diod

if i have the time i'll look at adding a 9p server to agro.

AoE is ATA over Ethernet: http://brantleycoilecompany.com/AoEr11.pdf

it runs only over ethernet, but this is not a problem as long an an agro node is on the local network segment.

i'm slowly poking at this.

We're probably going to need this pretty soon.