I might keep the primary one on its own ELB still, but it's cool that the only URL you'd ever need to know is stream.place

Pretty easy. I just wanted something nice to be way down here at the #1 issue.

https://travis-ci.org/streamplace/streamplace/builds/206068330 and a few others have failed because of random hiccups in npm and Docker Hub and such. (No failures from Quay so far, so kudos to CoreOS I guess.) We should retry at least once, get fewer failing builds.

should just take a slight tweak in package-docker-build.sh.

This will be an addition to run/bump-version.sh. Currently that file just takes care of creating the git tag and bumping all of the versions in all of the package.json files.

It should also post to the Streamplace Twitter! Bug me for those credentials if you want to do this ticket.

It can be based off of my Drumstick container, which is what I use for development anyway. https://github.com/iameli/drumstick

This is important for Lerna executing helm package commands in the right order.

Yeah, this is probably good to have.

It'll definitely be Docker-based, but I'm not sure if it's going to be docker-compose or Vagrant or just some shell scripts around Docker commands.

minimally build and start, i think

Currently, you have to take your stream down to change the inputs coming into your stream. This is because we can only have one CompositeVertex active currently, boo.

What's needed is an OutputProxy vertex that sits between the Composite vertex and the outputs. It is able to take multiple Composite vertices as inputs, and when it gets a new one it will wait until the PTS lines up then switch. This will enable so much good stuff.

I strongly suspect this is the root cause of streamplace/wheelhouse#80

I already moved almost everything, but this would be nice to make everything consistent. Off the top of my head, shoko needs to be moved.

get-color.js is the script for printing pretty colored names of each module.

I think this is literally the only thing keeping us from supporting Node 4. I mean, there might be another reason to only support 6+, but it's silly to have it be this one utility file.

Things that would be nice to generate:

• New back-end microservices (Like channel-manager or somesuch)
• New schema plugins (Like sp-plugin-auth)
• New create-react-app based frontend services

It's totally scattered right now. The node apps use an ENV file or ad-hoc checks to process.env. The client-side apps have a window.SK_PARAMS blob.

I'm aiming for something that accommodates our use cases but is as lightweight as possible.

• A new node module: sk-config. Everyone gets at their settings by doing import config from "sk-config".
• There's one global set of runtime variables.
• Configuration parameters are UPPER_CASE_WITH_UNDERSCORES.
• Runtime configuration can come from a few places. Things higher up on this list trump things lower on this list.
  • Any values set explicitly with config.set({KEY: "VALUE", KEY2: "VALUE2"})
  • Values in window.SK_CONFIG, if present.
  • Values in process.env, if present. Environment variables get prefixed with SK_, i.e. SK_API_SERVER_URL="http://localhost:8100".
  • Values present in the YAML file at SK_CONFIG_FILE, if present.
  • Default values, as defined in config.default.yml in the sk-config module itself.

I'm not doing command-line params; I don't actually think people will be manually running these apps from the command line very often. (And if you are, hey, look at that, environment variables.) If that ends up being a valuable use case, we'll revisit.

package.json and Chart.yaml should have the same name, version, and description.

Those two and the directory should be the same name.

These things not being true should be a linting error.

I did literally the bare minimum amount of effort to get auto-update working in sp-app. The logic is here -- when it sees an update, it downloads it in the background and then unceremoniously quits the app and restarts no matter what you were doing.

No good. Chrome provides a template for the best auto-updating, I think: download the update in the background, install it next time they restart the app.

Logging in SK is interesting. We definitely don't want to always log, say, every line of FFMpeg output from every Vertex -- there's just too much there. Or every time a TCP connection happens or anything. It's just too much. Our metadata shouldn't exceed the size of our data.

But I do think it'll be really useful once we're starting to scale up to have the ability flip on verbose logging on an ad-hoc basis. If some particular vertex is behaving strangely, I want SK to tell me every bit of data about that vertex.

Additionally, I don't want this ad-hoc setting to occur over the usual API channels. If the API server is malfunctioning, I want to be able to get very verbose output from it.

Prepare's the one that imitates the same "local install/prepublish" behavior, right? Double check that too.

This will entail telling sp-app to use a UA string that isn't Electron.

The UA string used should identify both name and version, if pertinent.

I forget what it's called

Done!

We publish auto-updating apps for Windows and Mac but for the Linux folks out there, they'll be getting the app with npm install -g streamplace.

So, sp-app will need to get a linux-y electron-builder thing.

Right now, vertices of type FileOutput log all inputs and compositors to files. This is good. One of the nice features of SK will be having it be an archive for all of your files.

But right now they're just unceremoniously dumped onto thirty-second clips on S3. I have to go dig through and squint at the file names to figure out what's what, and I have to manually recombine everything. This is bad.

Some of the work has happened already -- check out the S3ConcatStream class.

So instead, the FileOutput plugin will have File objects. These will keep track of some metadata, and where to look/concat on S3 to download stuff.

If an app crashes, its button doesn't currently turn back to green, and you're unable to restart it.

$ wheelhouse dev
fake-meteor bash: meteor: command not found
fake-meteor
fake-meteor
fake-meteor process exited with code 127

sk-schema exists, and it has some nice JSON-Schema documents within its Swagger spec, but those aren't accurate or enforced anywhere. We should make their content up-to-date with the reality of the system, and we should enforce them on both the client and server.

Not sure why quite yet, but it stops new broadcasts from working. No good.

This is good practice. It makes all runtime configuration easily greppable.

Right now the client is double-subscribing to lots of things, and it doesn't need to be. The new binding interface should be smart enough.

• Refactor sp-frontend/src/sp-binding.js into a new sp-binding package.
• Make sp-binding reuse watchers and keep refcounts of which are currently bound
• Write some unit tests for sp-binding, now that it does what we want

Bonus: I love sp-binding. How can we use it for server-side microservices?

I'm pretty happy with this monorepo, but the interrelated dependencies are currently managed by setting NODE_PATH.

Lerna is the f'real solution to this problem, utilized by Babel and others. Let's use it instead.

Ideally this will also make sure all appropriate apps reload when one of their dependencies change in development.

This will still be a shared helper library that we'll use to manage configuration variables in Javascript code, but it should originate from the Helm chart underlying the cluster

magic TLS cert retrieval for everyone

Right now it just an empty page. Let's at least get a message on there that the connection no work good.

Lerna takes care of managing the dependency tree here, which is sweet. Problems that need solving:

1. How do we know which repo/tag to build/push to? (Probably blocked by #24, then)
2. How do we keep references consistent between Helm charts/Dockerfile building?

I want people to pick a license for their stream's content before they go live. This is so, going forward, we can really easily implement folks re-broadcasting others streams in a legally coherent way.

It'll basically be this thing from the Creative Commons site, just made more specific for Streamplace content.

It should result in these licenses: https://en.wikipedia.org/wiki/Creative_Commons_license#Seven_regularly_used_licenses

Plus an "all rights reserved" license if you don't want to put any kind of CC license on your work. But it should be at the bottom.

I think this is good.

https://github.com/prettier/prettier

Right now all vertices exist within a single Pipeland. This is severely limiting and contrary to how the software needs to function.

Instead:

• In development, and in the docker-compose distribution, each Vertex should create its own Docker container. In development, they should all share the same volume containing the code. If you make some changes, just bring up new vertices and they'll use the new code.
• In the Kubernetes distribution, the scheduler should create a Job object for each Vertex. That way Kubernetes will take care of restarting them if the crash, but they can safely exit on their own (indicating the vertex was deleted.)

Vertices should have their license as a required in SPDX Format, similar to how node has licenses specified in package.json. This way, once Stream Kitchen is a huge, distributed system, servers can know programmatically what sorts of streams they're allowed to mix with.

This is really interesting! Lots of Creative Commons-ish licenses are share-alike, so that means any downstream streams have to be licensed under the terms of the upstream... pretty cool.

To wit: kubectl, helm, minikube

Should set up automated build GitHub repos for these and validate the downloaded binaries using the same methods that the official repos use

When a Vertex gets deleted or disabled, Pipeland should gracefully shut down FFmpeg, stop listening on all of its ports, shut down the ZMQ connection... the list goes on. Right now that doesn't happen, and the whole system only works because of what is perhaps the ugliest hack currently in the entire codebase: upon a broadcast being deleted, Pipeland crashes itself. Yuck!

Probably my top priority as soon as I have bandwidth to start writing code again. This essentially blocks anyone from using the software themselves.

All the containers are based on Node 5 right now, because I tried to switch to 6 on the day it came out and got hit with nodemon bugs that spammed the development environment full of stack traces. I'll bet it's fixed now; probably just need to update the sk-node container.

So each package that has a Chart.yml file will be built by lerna on that step...

Probably we just need a root chart that includes all the other charts necessary for building things. Then we build/publish that, and that's what people can use in prod environments.

Dev environment will take some more thinking.

The API server doesn't go get the new cert for signing its dove-jwt tokens. dove-jwt refuses to sign with an expired cert, which is nice.

Quick fix: make sure you deploy to the API server at least once a month.

They're getting the job done, but I am not an expert in Helm or gotpl, and I'll bet we're doing silly disorganized optimizable things all over the place.

Helm templates can be found in the chart directory of each package in the packages directory.

Candidate for worst offender: packages/sp-api-server/templates/deployment.yaml

It can happen through the same S3 bucket that we have for Files.