This is a placeholder to discuss work that will need to be done mostly in various upstream repos.

Relevant repos:

• https://github.com/helium/ecc608-linux-rs
• https://github.com/helium/helium-crypto-rs
• https://github.com/helium/gateway-mfr-rs

Slack convo: https://nebraltd.slack.com/archives/C024BNQ1Y6T/p1640117393174900

Bump to latest https://github.com/helium/gateway-mfr-rs/releases/tag/v0.1.5

Also - is there a reason we are building this every time we build hm-pyhelper? Would it make more sense to build this only once each time there is a gateway-mfr-rs release and then hm-pyhelper builds can just pull in the pre-built release?

would speed up the hm-pyhelper builds considerably. and we could just run a periodic (weekly perhaps?) check as an action that checks gateway-mfr-rs repo, builds it and creates a release with the file attached?

We will need to perform various ECC operations inside the miner in various places. To simplify this, let's include gateway-mfr-rs in this package.

There are build instructions here.

The acceptance criteria is that you should be able to just run gateway-mfr after having installed this python package, so we need to include the ARM64 binary inside the package.

This should all be done in GitHub Actions.

need to ensure the button and gpio pins for rak / sensecap / og are correct

The release process for this repo is not obvious unless you've done it before. Let's add documentation in the README so any developer can easily contribute. Most recently, the process I attempted failed to push a release to PyPi. These were the steps I took.

1. Open PR and tag commit: git tag v0.0.x -a
2. Rebase and merge
3. Create a new release

• Add description of recent changes
• Add link to PyPi release

I then tried deleting the tag, pulling master locally, and retagging that. After pushing the new v0.8.9 tag, PyPi still didn't release.

For the 5g software, we need to add a parameter to the variants file that says USB or SPI and relates to the connection method of the concentrator

This will then be picked up in hm-pktfwd to determine what to put in the config.json

User Story
As a user, I should not hit an exception if running outside Balena.

Notes
We currently detect hardware by inspecting the BALENA_DEVICE_TYPE environment variable, created by Balena. In test cases, this will raise an exception.

    def is_raspberry_pi():
        """
        Pulled from
        https://www.balena.io/docs/reference/base-images/devicetypes/
        """
        device_type = os.getenv('BALENA_DEVICE_TYPE')
        device_type_match = [
            'raspberry-pi2',
            'raspberrypi3',
            'raspberrypi3-64',
            'raspberrypi4-64',
            'nebra-hnt',
            'raspberrypicm4-ioboard'
        ]
    
        for device in device_type_match:
>           if device in device_type:
E           TypeError: argument of type 'NoneType' is not iterable

Acceptance Criteria

• is_rockpi and is_raspberry_pi should not raise exceptions if invoked outside Balena. Return False in both cases.

Not sure why, but finestra shows blank under variant....even though it has got a FRIENDLY name...

It appears as this script is not working properly:
https://github.com/NebraLtd/hm-pyhelper/blob/master/.github/workflows/publish-to-pypi.yml

Currently, only the first build after a version bump will pass .github/workflows/publis-to-pypi-test.yml. Example: https://github.com/NebraLtd/hm-pyhelper/pull/14/checks?check_run_id=3694402894. One solution the docs mention is to use setuptools_scm but the implications of the switch need to be better understood.

If no MAC address is found, we currently return null:

> $ curl -s 192.168.202.90/initFile.txt | base64 -d
{"VA": "NEBHNT-OUT1", "FR": "868", "E0": "00:BD:27:EF:6D:E7", "W0": null, "RPI": "00000000589f8f22", "OK": "112mPWkGW55kcbQTgbtJvgAKMSTrEhHgavrdF1Cbu8FU85tTL4Nc", "PK": "112mPWkGW55kcbQTgbtJvgAKMSTrEhHgavrdF1Cbu8FU85tTL4Nc", "PF": false, "ID": "204f46e27d5a44bc65fac817ebc77e5b"}%                                                                             

This is invalid JSON:

> $ curl -s 192.168.202.90/initFile.txt | base64 -d | jq
parse error: Invalid numeric literal at line 1, column 2

We need to add TPM option to variants file for 5g units

This will be consumed by the miner and gwmfr containers to initialise the TPM

Notes

• All pi4, sx1302. We do not currently believe different variants are needed.
• Need to know i2c and SPI buses
• https://support.getmntd.com/hc/en-us/articles/4418137820951-I-need-to-replace-the-microSD-card-on-my-hotspot

Acceptance criteria

• Determine correct SPI and i2c buses
• Determine correct LED and button pins

MNTD miners by rak have no LED

Not sure about other versions of RAK

Pi 4 based

Move all imports that currently happen in Dockerfiles and/or manually copied into projects into hm-pyhelper. Import using PyPi instead.

Current location: https://github.com/NebraLtd/helium-hardware-definitions

Internals

Issue

Current linter max line length of 80 characters causes unnecessary code fragmentation. In order to please the linter sometimes the resulting code is even harder to understand which is contrary to why we lint.

Example:

Dicts with descriptive key names and medium sized values:

request_payload = {
    ADD_GATEWAY_TXN_NAME_KEY: payload[ADD_GATEWAY_TXN_PAYLOAD_KEY]['destination_name'],
    'serial_number': payload[SERIAL_NUMBER_KEY]
}

Line 2 of the above is > 80 chars so for the linter it would need to be re-written as:

request_payload = {
    ADD_GATEWAY_TXN_NAME_KEY:
    payload[ADD_GATEWAY_TXN_PAYLOAD_KEY]['destination_name'],
    'serial_number': payload[SERIAL_NUMBER_KEY]
}

But this one becomes confusing to read as one key value pair is on 2 lines while the next one is on a single line, so we format it further like:

request_payload = {
    ADD_GATEWAY_TXN_NAME_KEY:
    payload[ADD_GATEWAY_TXN_PAYLOAD_KEY]['destination_name'],
    
    'serial_number':
    payload[SERIAL_NUMBER_KEY]
}

This results in both readable and liner-acceptable code. However this is far less readable (specially for dicts where key and start of value are expected to be on the same line) than the initial version.

To avoid such scenarios, devs are tempted to use shorter variable/constant/key names which again is bad for readability.

Solution

Most modern IDEs now have at least 120 characters of code area visible even with side bars etc. Older editors like VIM don't even suffer from this as they have more screen real-state available. Increasing the max-line-length to 120 is a good balance between readability and fewer code formatting woes.

button - gpio27 / physical pin 13
led - gpio22 / physical pin 15

Right now, we are calling on get_gateway_mfr_test_result() to determine if the device has been provisioned or not. While this works, it is marginally slower than simply calling on get_public_keys_rust(), which would yield roughly the same result.

Issue

We usually define constants in code for key names and then use those constants in the code repository. This saves us from having to change multiple locations when a key name needs to be renamed or a different name/key needs to be used. However some of these key names are used across repositories with each repositories defining their own constants or hard-coding these values. This leads to requiring changes at multiple repositories and going through the whole PR->Review->Approve-Merge->(optional) Release process.

What this results in is unnecessary wastage of time when only a single field name is changed.

Because of this issue there has been resistance to have proper and consistent key naming across repositories. Some examples:

• Usage of lower case and upper case key names within the same dict
• Friendly key names and legacy key names in things like /initFile.txt payload in hm-diag.
• The same things being called as different names in different repositories like serial number in initFile payload to HPT to registration request to hm-dashboard.

Solution

Use a names package within hm-pyhelper. We should create a packkage named names (or something else like constants) where we store the names that are used across repositories. An example would be ADD_GATEWAY_TXN_PAYLOAD_KEY which currently is being called transaction in hm-dashboard and backend db while hm-diag, hm-pyhelper and hpt refer to it as destination_add_gateway_txn.

If such names are defined in one central place then they can be used everywhere and changing them in the future becomes trivial except for when database migrations are involved.

Please note that these are only for names/keys that are used across repos. Things residing inside a single repo can still be handled as the developer sees fit.

@NebraLtd/developers looking for your opinion on this.

Future considerations

For tasks that require changes in multiple repos and involve multiple developers working on parts of the task/epic; having a meeting of involved developers and agreeing on common names for data fields etc before implementation is advised. This just helps avoid unnecessary renaming later.

We currently upload the packages as build artifacts so even if a testpypi build fails the artifacts are still uploaded. For examples this run:

https://github.com/NebraLtd/hm-pyhelper/actions/runs/1369814396

Has the python outputs here:

https://github.com/NebraLtd/hm-pyhelper/suites/4125934376/artifacts/105569066

Similar to #34 we should document this as this can be useful for testing releases without needing to bump version numbers

Related to #20

Add support for Tri colour LEDs as for the Finestra in #100 and the syncrobit

Support feature added in GA 2021.11.04.2.

Projects that use retry_get_region should be updated to handle exceptions thrown. Including hm-pktfwd.

Notes
Objective is to abstract away reusable code. Identify shared components between hm-config/diag/etc

Acceptance criteria

• setup module with README explainig how to import
• publish to pypi
• waiting for miner keys to exist. read from API instead of waiting for file to load.
• miner keys parsing
• diagnostics related things: MAC address
• migrate https://github.com/NebraLtd/helium-hardware-definitions moved to hm-pythelper #3

• Add functionality to read firmware version from Miner API
• Check what the latest GA version is
• Create new issues in other repos that need to be updated to reflect this new logic.

Once done, adopt this in the relevant repos (replaces NebraLtd/hm-diag#89).

We don't have proper exception handing when communicating to the miner over RPC.

Case in point here. We need to be able to wrap this around a try/except with proper handing.

Related PR:
NebraLtd/hm-diag#186

See here for working setup on pisces p100...

Miner - radicale/hm-miner@90bb1c3
Diag - https://github.com/radicale/hm-diag
Config - https://github.com/radicale/hm-config
Helium miner software - https://github.com/radicale/helium-miner-software

Definition (from here):

# Pisces P100 Hotspot
    'COMP-PISCESP100': {
        'FRIENDLY': 'Pisces P100',
        'SPIBUS': 'spidev0.0',
        'ECCBUS': 'i2c-0',
        'KEY_STORAGE_BUS': '/dev/i2c-0',
        'RESET': 23,
        'MAC': 'eth0',
        'STATUS': 17,
        'BUTTON': 22,
        'ECCOB': True,
        'TYPE': 'Full',
        'CELLULAR': False,
        'FCC_IDS': [],
        'CONTAINS_FCC_IDS': [],
        'IC_IDS': [],
        'CONTAINS_IC_IDS': []
        },

The button should enable diagnostics and probably mapped incorrectly right now.

Getting an exec format error on armv6 device when trying to run gateway mfr

We need to build for lowest common denominator

It would make sense to just merge in helium-hardware-definitions into this repo.

this is currently outputting gateway_mfr test --path /dev/i2c-7 but it needs to be gateway_mfr --path /dev/i2c-7 test

sensecap m1 has a fan connected to GPIO13 / physical pin 33 - we should add handling for this

I just deployed the latest hm-diag, which includes hm-pyhelper. However, it isn't working at this point.

>>> from hm_pyhelper.miner_param import get_public_keys_rust
>>> get_public_keys_rust()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/site-packages/hm_pyhelper/miner_param.py", line 15, in get_public_keys_rust
    run_gateway_mfr_keys = subprocess.run(
  File "/usr/lib/python3.8/subprocess.py", line 493, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib/python3.8/site-packages/hm_pyhelper/gateway_mfr'

Observations so far.

• Using a Raspberry Pi 4b 4gb
• Seeed wm1302 concentrator
• Appears to be using OpenBalena
• Does not have an exposed Ethernet port (but later versions apparently do)

The default timeout was set to two seconds. Maybe 5-10 seconds makes more sense. It should be possible to set this value through an environment variable so code changes aren't needed to make adjustments.

Add architecture to hardware definitions

Related to NebraLtd/helium-miner-software#336

We need to refactor the provision_key() function to use the run_gateway_mfr() function rather than trying to invoke gateway_mfr by itself to make things DRY and also work on the RockPi.

Right now, because we're running gateway_ mfr directly, we don't call on the is_rockpi() function, and are therefore not taking the different I2C interface into account.

LED info https://www.linxdot.com/understanding-the-led-statuses/

Perhaps we should change SPIBUS to LORA_BUS as with 5g we will be using USB concentrators.

We could also use the full address like /dev/spidev1.0 or /dev/ttyACM0 for example

Related to #28

Sentry Issue: HM-CONFIG-M

FileNotFoundError: [Errno 2] No such file or directory: '/sys/class/net/eth0/address'
  File "config_program.py", line 991, in <module>
    adv = ConfigAdvertisement(0)
  File "config_program.py", line 82, in __init__
    macAddr = open("/sys/class/net/eth0/address").readline()\

This is related to NebraLtd/hm-config#80. The relevant code has also changed.

Once we have fully functional software for a third party miner, we should do some through testing, perhaps similar to the deployment checks that @NebraLtd/tech-support do on the main repo.

In any case we should check various functionality and we should document what needs to be tested as well before it goes live

As per #35 there is no need to build gateway-mfr-rs with every release here.

May as well build it on a schedule and pull in, to speed up builds.

Also, looks like Helium may produce builds directly at some point helium/gateway-mfr-rs#4

Uses CM3+ with 1gb ram...

Notes

• hm-config logic: https://github.com/NebraLtd/hm-config/blob/900aeed353fb9729b49bca97d7da8a9abf0a2029/gatewayconfig/bluetooth/characteristics/add_gateway_characteristic.py#L48
• RPC code: https://github.com/helium/miner/blob/b9d2cd108cdcc864b641ccf4209f790b1461926d/src/jsonrpc/miner_jsonrpc_txn.erl#L31

Acceptance criteria

• Method should accept a specific wallet to generate the transaction for
• Throw an exception to indicate error
• Reusable logic added to hm-pyhelper to generate AddGateway transaction
• Confirm fee and amount are static values, and what their exact values should be
• Returned format should match

{
  "address": "11TL62V8NYvSTXmV5CZCjaucskvNR1Fdar1Pg4Hzmzk5tk2JBac",
  "fee": 65000,
  "mode": "full",
  "owner": "14GWyFj9FjLHzoN3aX7Tq7PL6fEg4dfWPY8CrK8b9S5ZrcKDz6S",
  "payer": "138LbePH4r7hWPuTnK6HXVJ8ATM2QU71iVHzLTup1UbnPDvbxmr",
  "staking fee": 4000000,
  "txn": "CrkBCiEBrlImpYLbJ0z0hw5b4g9isRyPrgbXs9X+RrJ4pJJc9MkSIQA7yIy7F+9oPYCTmDz+v782GMJ4AC+jM+VfjvUgAHflWSJGMEQCIGfugfLkXv23vJcfwPYjLlMyzYhKp+Rg8B2YKwnsDHaUAiASkdxUO4fdS33D7vyid8Tulizo9SLEL1lduyvda9YVRCohAa5SJqWC2ydM9IcOW+IPYrEcj64G17PV/kayeKSSXPTJOMCEPUDo+wM="
}

Need to rework docs surrounding variants to make sure it matches the variants file