Git Product home page Git Product logo

fedora-coreos-cincinnati's Introduction

Fedora CoreOS updates backend

Checks status minimum rust 1.49

This repository contains the logic for Fedora CoreOS auto-updates backend.

This service provides an implementation of the Cincinnati protocol, which is consumed by on-host update agents (like Zincati).

This workspace can be built with cargo build (see quickstart instructions) and contains the following binaries:

  • fcos-graph-builder: a service which builds and caches the raw update graph
  • fcos-policy-engine: a web service which handles requests from agents

The instance of this service used by default on Fedora CoreOS is hosted in the Fedora infrastructure. More details can be found in the Fedora infra docs.

fedora-coreos-cincinnati's People

Contributors

aaradhak avatar bgilbert avatar c4rt0 avatar cgwalters avatar coreosbot avatar dustymabe avatar jlebon avatar jmarrero avatar lucab avatar travier avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fedora-coreos-cincinnati's Issues

cargo: update dependencies

There are a few stale dependencies that we should take care of updating:

$ cargo outdated -R
commons
================
Name        Project  Compat  Latest  Kind    Platform
----        -------  ------  ------  ----    --------
actix-cors  0.2.0    ---     0.6.1   Normal  ---
actix-web   2.0.0    ---     4.1.0   Normal  ---
prometheus  0.11.0   ---     0.13.1  Normal  ---

fcos-graph-builder
================
Name        Project  Compat  Latest   Kind    Platform
----        -------  ------  ------   ----    --------
actix       0.9.0    ---     0.13.0   Normal  ---
actix-web   2.0.0    ---     4.1.0    Normal  ---
env_logger  0.8.4    ---     0.9.0    Normal  ---
prometheus  0.11.0   ---     0.13.1   Normal  ---
reqwest     0.10.10  ---     0.11.11  Normal  ---

fcos-policy-engine
================
Name        Project  Compat  Latest   Kind    Platform
----        -------  ------  ------   ----    --------
actix       0.9.0    ---     0.13.0   Normal  ---
actix-web   2.0.0    ---     4.1.0    Normal  ---
env_logger  0.8.4    ---     0.9.0    Normal  ---
prometheus  0.11.0   ---     0.13.1   Normal  ---
reqwest     0.10.10  ---     0.11.11  Normal  ---
serde_qs    0.6.1    ---     0.9.2    Normal  ---

Leaving aside for now all the actix ones (which require a full porting effort), all the other ones should be reasonably simple to upgrade to their latest versions.
After that, we should also enable dependabot on this repository.

Hook up to Quay.io or OpenShift CI

I'd like to be able to write upgrade tests that require talking to a local dumnati server serving test-specific metadata. I'd much prefer just pulling down a container instead of rebuilding it locally as part of the test. :)

Easiest would be to hook it up to Quay.io or https://api.ci.openshift.org/. An alternative for users is pulling directly the image built in the Fedora OpenShift project, though offhand it doesn't look like the registry is exposed.

scraper: support multiple graphs (streams)

Currently we scrape and assemble a single graph for a specific stream.

In order to support multiple streams (e.g. testing and stable) we'll need to support fetching and caching multiple graphs, each one derived from its stream-specific updates metadata.

We also need to provide the stream-specific graph to each client, based on the stream parameter in the request.

Make JSON endpoints configurable

I'd like to be able to write upgrade tests that require talking to a local dumnati server serving test-specific metadata. Right now, the releases.json and ${stream}.json is hardcoded into the app. Can we make these be configurable somehow? I see it doesn't use config files yet, and maybe we don't want to go all the way there until we sort out the merger with Cincinnati -- maybe just env vars for now would work?

Make use of new fedmsgs... or not

The FCOS pipeline now broadcasts a fedmsg whenever stream metadata changes:
https://github.com/coreos/coreos-assembler/blob/93b7a510457ffe8f0cd9233db9eafd92ae26c8fe/src/cosalib/fedora_messaging_request.py#L84

Example msg:

https://apps.fedoraproject.org/datagrepper/id?id=2020-5c429b71-ae65-4b50-ad7d-432ce1126244&is_raw=true&size=extra-large

{
  "topic": "org.fedoraproject.prod.coreos.stream.metadata.update", 
  "headers": {}, 
  "msg": {
    "stream": "stable"
  }
}

Cincinnati could use this to optimize its refreshes and lower the polling frequency so it's just a fallback mechanism.

dist/Dockerfile: update to F36 base image

We already attempted the F36 update, but we had to revert due to outages, see #65 (comment).

We have now upgrade the base image to F35 and also the required openssl crates, so the only thing left in the breaking changes is the F36 update.

We should try to tackle this again and see where it breaks.

cargo: port from structopt to clap-v3

For CLI handling we are currently using structopt.

However nowadays we are hitting the ceiling of unmaintained transitive dependencies:
clap-rs/clap#4172

As suggested in the discussion post, at this point we should move to clap v3.

graph-builder: allow queries with aarch64 basearch

Current logic in graph-builder only allows queries with a x86_64 basearch, through an hard-coded value.
Now that FCOS is publishing aarch64 streams too, we should verify that the graphs for that get correctly processed too.
At that point, the service should be updated to start serving requests for aarch64.

graph-builder: split and cache graphs for multi-arch

Right now a single update-graph is cached in the graph-builder, and the service assumes it only contains x86_64 details.
Not what FCOS is publishing aarch64 streams too, we should split and cache the graphs separately.
Caches should be effectively partitioned according to the (architecture, stream) tuple, and for the current scenario we are going to end up with 6 caches: [x86_64, aarch64] x [stable, testing, next].

fcos-cincinnati: port to actix-web 4.1

We are still using an older version of actix-web (v2) and we should move to the latest release (v4.1).

This will involve a bit of porting effort to adapt to the new APIs, but it will allow us to align all the other intertwined dependencies (most notably: actix, reqwest, tokio).

graph-builder: return more structured errors to client

Right now the graph-builder mostly returns just error codes to client, logging the full cause on the backend.
It should be better to return more details to the clients too, via structured errors.
As long as we don't reflect back arbitrary input, it should be safe to do.

Unsupported basearch leads to 500 internal error

Not sure if that's the expected behavior, but sending a request to cincinnati with a non-supported basearch leads to a 500 internal error.

For example:

$ curl  --header 'Accept:application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?stream=stable&basearch=mock-amd64'
HTTP status server error (500 Internal Server Error) for url (http://127.0.0.1:8080/v1/graph?basearch=mock-amd64&stream=stable)

The issue seems to be here: https://github.com/coreos/fedora-coreos-cincinnati/blob/master/fcos-graph-builder/src/main.rs#L147

    let arch_graph = policy::pick_basearch(cached_graph, basearch)?;
    let final_graph = policy::filter_deadends(arch_graph);

Instead of returning an HttpResponse, we're just erroring out.

I haven't used rust in ages, but I can attempt a fix. Should we send a 400 bad request?

config: add settings for status service

This applies to both graph-builder and policy-engine.

The status service needs to configurable via TOML config file. Initial proposal:

#[derive(Deserialize)]
struct FileConfig {
    /// Status service config.
    status: Option<StatusConfig>,
}

#[derive(Deserialize)]
struct StatusConfig {
    /// Address on which the status service will listen,
    pub address: Option<IpAddr>,

    /// Port to which the status service will bind.
    pub port: Option<u16>,
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.