cyberbit / telem Goto Github PK

View Code? Open in Web Editor NEW

18.0 1.0 4.0 8.77 MB

Trivial ETL Engine for Minecraft

Home Page: http://telem.cyberbit.dev

License: Other

Lua 99.18% Shell 0.82%

computercraft computercraft-scripts computercraft-tweaked computercraft-utilities

telem's Introduction

Telem - Trivial ETL Engine for Minecraft

Tired of creating complex logic to monitor inventory or production? Want something more modern, modular, and scalable? Want something that can empower a dashboard like the screenshot below? You have come to the right place.

Requirements

CC: Tweaked recommended, CC: Restitched also supported
http access for installer and certain adapters

Install

wget run https://pinestore.cc/d/14

Usage

Please visit telem.cyberbit.dev for full documentation.

local telem = require 'telem'

local backplane = telem.backplane()                  -- setup the fluent interface
  :addInput('hello_in', telem.input.helloWorld(123)) -- add a named input
  :addOutput('hello_out', telem.output.helloWorld()) -- add a named output

-- call a function that reads all inputs and writes all outputs, then waits 3 seconds, repeating indefinitely
backplane:cycleEvery(3)()

-- alternative threadable option
parallel.waitForAny(
  backplane:cycleEvery(3),
  
  function ()
    while true do
      -- listen for events, control your reactor, etc.
      
      -- make sure to yield somewhere in your loop or the backplane will not cycle correctly
      sleep()
    end
  end)

Input Adapters

Item and Fluid Storage
Modded Storage (AE2, Refined Storage)
Modded Machines (Mekanism)
Custom Inputs
Secure Modem
More to come!

Output Adapters

Basalt labels and graphs
Plotter charts
Custom Outputs
Secure Modem
Grafana (advanced)
More to come!

telem's People

Contributors

Stargazers

Watchers

Forkers

bananasov erb3-forked bakaneko152 scmcgowen

telem's Issues

Basalt Label adapter not working

It doesn't work. Fix it.

Mekanism panel download on Grafana adapter broken

Links to https://telem.cyberbit.dev/assets/Telem_Mekanism_Panel.json, returns a 404

Advanced Peripherals documentation

Migrate generic input adapters to query model

TBD

Add secure modem input and output adapters

There needs to be some kind of in-game communication possible within Telem for coordinating systems across dimensions. Each computer attached to a peripheral can directly output its metrics to whatever local display it wants to, but having a home base that collects everything in one place is going to be important.

~~The input adapter would be a very specialized design as it needs to listen to events in order to collect them. The output would work like a normal output adapter, just screaming metrics into the void.~~

The input adapter sends requests to output adapters. The output adapters listen for requests.

Design notes:

Should modem inputs overwrite adapter? Strictly speaking it should, but I think it makes more sense if it preserves the remote adapter and source names. Maybe prefix adapter? If remote input adapter is chest_1 and modem input adapter is storage_terminal, the final adapter is storage_terminal:chest_1 This has been implemented at the Backplane layer. For every input adapter that passes through cycle(), every metric will have <adapter>: prefixed to its adapter property.
Secure Modem has been implemented with ECNet 1.x for the time being. I am planning on releasing a beta version of this adapter while I work up updating to ECNet 2.

Add Energy Detector input

See https://docs.advanced-peripherals.de/peripherals/energy_detector/.

Offers transfer rate and transfer rate limit.

OpenComputers support

Need abstraction layer for:

I did some brief testing, it looks like it is possible in theory, need a lot of abstraction but managed to get a janky version of inventory reading working.

Using Ocelot Desktop for emulation.

Powah adapters

"POWAH"
someone, probably

I am yet unsure what versions have CC compatibility, in my testing on 1.18 I could not detect any peripherals.

Complete Mekanism integration

Continuing the work by @bananasov.

Need to evaluate input adapter capabilities and implement where possible. These will be utilizing the new Fluent query system established in the base Mekanism adapter. This is expected to significantly increase the size of the build if it is all included as part of the base install. If the need arises, I will start planning an adapter plugin system that can manage mod-specific features.

Creating adapters against non-existent peripherals is not recoverable

Need to make adapters resilient against peripherals not existing when booting up. A good fix may be to execute peripheral.wrap(id) before read() or write() calls if the stored peripheral is determined to be nil.

Powah documentation

Bigger Reactors documentation

Publish to unicornpkg

I'd love to get this on unicornpkg but the installer seems broken at this time :(

Mekanism documentation

Tracking documentation for new adapters.

Add fusion reactor adapter

Complete Advanced Peripherals integration

AP Blocks:

Environment Detector
Player Detector
Inventory Manager
Geo Scanner
Redstone Integrator
ME Bridge (Fluentize)
ME Bridge (add non-storage metrics)
RS Bridge (Fluentize)
RS Bridge (add non-storage metrics)
~~Colony Integrator~~ This deserves its own issue due to complexity

Create adapters

Create these peripherals. These are supported directly by recent versions of Create: https://github.com/Creators-of-Create/Create/wiki/ComputerCraft-Integration

Prometheus Pushgateway output adapter

This is technically for further supporting Grafana, but should support anything that can ingest Prometheus.

Publish to RequireDB

Cool repository, should add: Ale32bit/RequireDB

Opt-in cache for adapters with multi-cycle state

Basalt Graph and Plotter Line Chart record state over many cycles. It would be nice for that state to be somewhat recoverable after a program/computer crashes, or is unloaded/reloaded.

First-party HTTP output target

Public ID + Secret Key authentication
Limited rate per ID
Limited storage size per ID
Automatic bookmarkable live charts for provided ID
History? (would eat storage probably)

Implement middleware

EDIT: Now that Fluent exists, a direct implementation of middleware may be possible.

Things middleware should be able to do:

Renaming (prefix/suffix metric names and labels)
Filtering (isolate metrics by property matching)
Ordering (reorder metrics)

Two kinds of middleware:

MetricMiddleware. These operate on Metric objects. Renaming and Filtering may be written at this scope. This design enables intelligent chaining to execute in O(n) time.
CollectionMiddleware. These operate on MetricCollection objects. Ordering may be written at this scope.

Returning a new collection in custom middleware not working

This doesn't work:

:middleware(mw.custom(function(collection) 
    return telem.metricCollection({
        sanity = -2
    })
end))

MCCM Discord context

Original Mekanism adapters are missing metrics

In particular, the steam amount is missing.

Refresh the original four to ensure all metrics captured:

Fusion Reactor
Fission Reactor
Induction Matrix
Industrial Turbine

Add CloudSolutions output adapter

See CloudSolutions.

KV output would be fast and secure storage, isolated per-gamer
StatsD bridge would be a Grafana-alike with better efficiency

Basalt Graph not scaling chart area correctly

local graph1 = telem.output.basalt.graph(monitorFrame, "rand", colors.black, colors.red, 2 * 60 * 2.5)

results in this:

Plotter output adapter

Now that Plotter is at least mostly stable, I want Telem to use it. Like this!

Bigger Reactors adapters

Targeting Bigger Reactors mod. Suggested by @scmcgowen.

Reactor (Passive Cooling)
Reactor (Active Cooling)
Turbine

Improve build script with auto-discovery

Adding individual files to the build script is bad. Auto-discovering files is good!

Some ideas:

Search source directories for Lua files that are well-defined (maybe with a top-of-file header?)

Actually that's my only idea.

ChartLine: Resizing window does not update chart area

Reported by juh9870 in Discord.

When a window (in this case a monitor) is resized, the plot area does not automatically pick up the change and becomes cut off.

Original monitor configuration:

Removing one column of monitors from the left:

A possible fix would involve checking the size of the assigned window at the start of every write call and updating internal structures. In addition to this, a function should be provided for triggering a layout refresh on the adapter manually.

Metadata middleware

Gather metrics about metrics!

Concepts:

count of input adapters
count of output adapters
last count of output metrics
count of cycles
count of faults
last input read time
last output write time

nice

Precise storage input adapter

gather high level of detail on each slot of an attached storage
preserve slot IDs
no deduplication

Implement update frequency tiers for cycles

Some metric providers may interact with components that have a variety of metric types. Some of these metrics may be static, others may update once every minute, even more may be expected to update every tick. The current design treats all metrics the same, updating every metric during every cycle.

If an input adapter tightly integrates with its components (such as the Mekanism input adapters), it is fair to say there would be established knowledge on which metrics update "fast" and "slow".

The Backplane could be configured to support several cycle speeds, prioritizing certain inputs to be read more often than others in the same period of time. Input adapters themselves could define their APIs in such a way to divide the metrics into groups, like "fast", "slow", and "static". These groups would have default cycle times, but could be overridden by a user.

However, this introduces a challenge with how the Output Adapters should behave. Some output adapters may not benefit from accelerated update times, or cause confusing/corrupted results. Other outputs would greatly benefit, such as charts and other visual indicators.

This issue will serve as a place to record ideas during the planning of this feature.

Delta middleware overwrites itself when different adapters produce same metric name

In the example below, windgen0 and windgen1 are both producing identical metrics, so the delta middleware calculates the values for both, but only keeps the last calculated value. Middleware should be adjusted to produce metrics for each name-adapter-source combination as necessary.

Add computer ID as a label

Speaker output adapter

Ideas:

Simple heartbeat: Speaker adapter at end of list indicates output has completed
Simple indicator: Filter for metric, measuring against a minimum and maximum. Minimum = low tone, maximum = high tone. Trigger on output
Panic indicator: Filter for metric, if value outside a given range (or missing), rapidly trigger a note sequence until recovered
probably other annoying things :)

Filesystem output adapter

Dump point-in-time metrics to filesystem with configurable filename and format (CSV, JSON). For multiplayer worlds, this could be used as a way to enable out-of-game resource leaderboards with a script that scrapes all the computer filesystems on a schedule.

Gist output adapter

The API to make this work is a little annoying, but could be a fun way to export for systems that don't support webhooks but can read from a URL.

Basalt output adapters

Basalt added a Graph, now I want to use it.

Label output adapter
Graph output adapter

these should all require filtering at load time

Sticky middleware

Forces metrics to be "sticky", remaining in the backplane after their source disappears. If and when this occurs, the metric value is set to zero.

This would be intended for general storage inventories whose stored types may fluctuate often.

Add support for custom adapters

Concept:

backplane:addInput('custom_in_1', function ()
  -- do stuff

  -- can also return MetricCollection, this is sugar
  return {
    custom_metric_1 = 123,
    custom_metric_2 = 4.999127
  }
end)

backplane:addOutput('custom_out_1', function (metrics)
  -- metrics = {
  --   {
  --     name = 'metric',
  --     value = 123,
  --     unit = 'unit',
  --     adapter = 'adapter',
  --     source = 'source'
  --   },
  --   ...
  -- }

  -- do stuff
end)

SecureModem: serialized message is too large

When a secure modem output adapter tries to send a large (nearly 1,000 metrics) collection, the adapter crashes the backplane:

The solution may be to send multipart responses, where the requesting input adapter assembles the full collection when all parts are received. If the input adapter times out waiting for all parts, it can either stop listening to parts and assemble what it has received, or throw an error.

Alternatively, I can create a custom serde that is more efficient with keys and utilizes compression.