depscloud / deps.cloud Goto Github PK

The basic idea would be to have a single repository with configuration use to scaffold out a cleanroom environment for testing. Ideally, we'd be able to test:

• each integration (GitHub, GitLab, BitBucket, Nexus, jFrog)
• each database driver (sqlite, mysql, postgres)
• HTTP and SSH clone support

I imagine this being a git-ops style repository containing kubernetes manifests. These manifests can be applied to a minikube cluster during github-action runs. Looks like there is a plugin for being able to do something like that.

https://github.com/marketplace/actions/setup-minikube-kubernetes-cluster

KinD might even be an option: https://github.com/marketplace/actions/kind-kubernetes-in-docker-action

Now that we have Homebrew and APT packages available, we need to update the documentation on how to install the CLI.

For apt on Ubuntu:

$ echo "deb [trusted=yes] https://apt.fury.io/depscloud/ /" | sudo tee /etc/apt/sources.list.d/depscloud.list
$ sudo apt-get update
$ sudo apt-get install depscloud-cli
$ deps --help

For brew on OSX:

$ brew tap depscloud/tap
$ brew install depscloud-cli
$ deps --help

The binary can still be downloaded through the releases section, but this will be the preferred way.

From gh-53

The page for this content has already been stubbed out. We should add some content on how to connect the tracker to a sqlite database.

From gh-53

There are two child sections when it comes to configuration. We should add some content to the top-level navigation to help folks accordingly.

Now that CLAs are available, let's add a callout to them in our contributing guide section.

It would be nice to have a webhook to be able to pro-actively update state when changes are made to projects.

The site has largely been a kitchen sink up to this point. It's a good chance to really evaluate the content, pair it down to relevant sections, and focus on the main points.

From gh-53

The page for this content has already been stubbed out. We should add some content on how to connect the tracker to a mysql database.

No project integrates monitoring right now. This means that we need to leverage application logs to get a sense of what is going on in a system. By having something like statsd or prometheus monitoring, we would be able to better monitor the systems over time.

My proposal would be to leverage statsd as the main stat emission protocol, but then leverage prometheus sidecar containers to advertise the metrics. This should fit in rather nicely to many existing stat collection tools, like prometheus and datadog, without being too opinionated about which ones companies are using.

Currently, health checks do not hook into any active / passive signals. As a result, services occasionally report healthy when the underlying connections are halted. This will require some pre-work from me to list dependencies.

Right now, a lot of the code is scattered across a few repositories. While this has been nice from a separation of concerns perspective, it's made getting started a bit harder and some additional process and project management. I'd like to start a thread around moving to two monorepos.

The depscloud repository would merge part of api, dockerfiles, gateway, extractor, indexer, part of tracker, and potentially deploy. This would move all business logic under a single location and enable project private "libraries" for common logic. This would require some careful project structuring as it would container code for TypeScript (api, extractor, ui), Python (api), and Golang (api, gateway, indexer, tracker).

The graphstore repository would pull the graph database-specific elements out of the api and tracker projects. It would work to provide a generic graph database solution that runs on top of existing SQL solutions. This has largely been done through packaging, but this would be a distinct segmentation of code. This would likely require "independent" project management, treating depscloud as just one use case.

Sounds like this project is being sunset. Let's remove it from docs.

Part of the goals I laid out for this year was to start a blog to be able to do regular release updates as well as start to include some other content that we've learned as well. This should be pretty easy to do since we're already using Hugo.

Using this issue to start a thread of possible post ideas and to start in on some blog frameworking.

From gh-53

The page for this is currently stubbed out. We should add some user guides on how to get started with the python sdk.

There's a lot of jargon used throughout the project. Let's add a section to help clarify some terms. Some examples of this include:

• dependency graph
• manifest files
• organization
• module
• source
• dependency
• dependent

This should be added to the /docs/concept/ page once gh-53 is merged

From gh-53

The page for this content has already been stubbed out. We should add some content on how to connect the tracker to a postgresql database.

Big questions to answer in the post:

• Where can I find them?
• How can I use them?

As a participant this year, I want to provide a quick block post to help increase visibility. We can probably look into sending an email to the community at some point as well.

Relevant links:

• https://hacktoberfest.digitalocean.com/
• https://engineering.indeedblog.com/indeed-hacktoberfest-2020/

As far as I can tell the architecture documentation is no longer visible on the public site. Was it's removal deliberate? Is there a replacement planed?

From gh-53

The page for this is currently stubbed out. We should add some user guides on how to get started with the nodejs sdk.

Doesn't seem like the google powered search engine is working. I need to look into this a bit more.

https://deps.cloud/search/?q=arch

Some notes I scribbled on a napkin:

# simple alias to binary
$ alias dcc=depcloud-cli

# list sources
$ dcc get sources

# list sources for module
$ dcc get sources --organization <> --module <>

# list modules for source
$ dcc get modules --source <>

# list dependents for source
$ dcc get dependents --organization <> --module <>

# list dependencies for source
$ dcc get dependencies --organization <> --module <>

The extractor and tracker processes can easily be scaled using an HPA. This requires work for monitoring to be done and associated HPA definitions to be added to the charts.

gRPC services are often thought of as private services and only accessible by other projects. With some advancements from cloud providers and tools like Kubernetes, leveraging gRPC for a public service can be really useful. Shouldn't be hard to throw together a quick blog post.

This might turn into a part one of sorts.

honestly, I need to sit down and give all the docs a good once over...

From gh-53

The page for this content has already been stubbed out. We should add some educational information about what a dependency graph is, why it's valuable, and the types of problems it's well suited to solve.

Right now, the build and publication system is kinda a mess.

• Travis CI is responsible for handling the building and releasing of resources.
• When a new version is released, I need to await CI completion to build and publish the docker images
• Once CI is complete, I use buildx to create multi-arch images and push them to docker

It's a rather toilsome process and there have been some significant shifts recently that I think can help clean up this workflow.

• Most of what I'm doing in Travis CI can be done easily in GitHub Actions without requiring the provisioning of an API key.
• GitHub Actions also support buildx now making it easy to build and publish multi-arch images

During the process, we should preserve all existing artifacts.

I've long use tools like gox and simple asset uploads to deploy binaries. There are some nice things that come along with goreleaser such as a programmatic changelog, checksums, and more.

Current: deps-cloud
Proposal: depscloud

Why?

• Consistency
• Some systems don't allow special characters in their group name backing distributions difficult to remember.

Places to change:

• GitHub (public repos)
• Repositories
  • api
  • cli
  • deploy
  • deps.cloud
  • dockerfiles
  • extractor
  • gateway
  • indexer
  • template
  • tracker
• Rebuilds
  • api
  • cli (v0.0.12)
  • deploy (https://github.com/depscloud/deploy/actions/runs/149829727)
  • deps.cloud (https://github.com/depscloud/deps.cloud/actions/runs/149831841)
  • extractor (v0.2.17)
  • gateway (v0.2.12)
  • indexer (v0.2.11)
  • tracker (v0.2.13)
• Docker
• Twitter
• Slack
• Gemfury
• GItLab (private repos)
• npm.js

Changes to deps-cloud/api

• proto needs to be update to include source url
• .d.ts file needs to be updated to include source url field

Changes to deps-cloud/extractor

• version bump api
• Update extractors to attach a sourceUrl where found

Changes to deps-cloud/tracker

• version bump api
• if a source is provided as part of the dependency management file, include as part of the update to the graph.

Changes to deps-cloud/indexer

• version bump api

Right now, most of the documentation is manually laid out. If we leveraged gitbooks, we can get some more structure to the documentation.

From gh-53

Right now, this page just lists all child pages. We should add some top level navigation to help direct people if needed.

Right now, the helm charts live off in their own space, but have no reference on our docs page. Add a section under User Guides, similar to docker and kubernetes, on how to get started using helm.

At one point I had these on the project but removed them. This time, we'll add an individual and corporate agreement with some directions for both.

The CLI comes with a built-in completion utility. We should document how to set up code completion as part of the CLI guide.

See #9

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• fix(deps): update dependency postcss to v8.4.31 [security]
• chore(deps): update actions/checkout action to v4
• chore(deps): update dependency postcss-cli to v11
• chore(deps): update peaceiris/actions-gh-pages action to v4
• Click on this checkbox to rebase all open PRs at once

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• chore(deps): update dependency autoprefixer to v10

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

From gh-53

The page for this is currently stubbed out. We should add some user guides on how to get started with the golang sdk.

When Gitter added support for message threads, their mobile app has gotten a lot less usable. Let's just move to Slack.

Alright, this has now bitten me a couple of times in migrations where I've forgotten to add some basic bits of information. It should be pretty easy to build a template project that contains the minimal amount of information.

• LICENSE
• DCO
• README.md
• renovate.json

This will allow us to track unique dependencies per repository and help solve the conflict errors when indexing multiple repositories that produce the "same" module. e.g. forks