Basically cache alot of things and generate statistics for nodes.json every time data is transmitted. Then we use a .map() to generate the nodes.json (basically add flags), which woult be a hell lot faster than a forEach.
The graph will be a bit more complicated, but it can be a minute outdated, so we can simply regenerate the graph every minute. Also we should have one cache for all output formats.
A bit OT: Is there a way to do gzip? My servers do this via an nginx rev-proxy, but built-in support would be nice.

Merge graph and nodes jsons and include bidirectional links (much like the ffmap-d3 graph json actually!)

since gluon 2016.2.2/.3 there is no nodeinfo-box available on nodes using l2tp (ffrl-tunneldigger-package)

https://map.ffdus.de/#!n:e894f6519496

e.g. "Margarete Franke".

nodes.json shows a mixed set of data "before update" and "after update to 2016.2"

"mac":"e8:94:f6:51:94:96"},"owner":{"contact":"[email protected]"},"system":{"site_code":"ffdus"},"node_id":"e894f6519496","hostname":"Margarete-Franke","hardware":{"model":"TP-Link TL-WR841N/ND v9","nproc":1}},"flags":{"online":true},"statistics":{"uptime":5384.92,"gateway":"ca:b1:e3:ca:b1:e7","gateway_nexthop":"cab1e9cab1e0","memory_usage":0.7969126186092621,"rootfs_usage":0.6771,"clients":3,"loadavg":0.04},"lastseen":"2017-02-14T13:42:07.742Z","firstseen":"2016-02-23T20:36:03.232Z"},{"nodeinfo":{"software":{"autoupdater":{"wifi-fallback":true,"branch":"stable","enabled":true},"batman-adv":{"version":"2016.0","compat":15},"fastd":{"version":"v18","enabled":true},"firmware":{"base":"gluon-v2016.1-128-g40d087d","release":"2016051602-sta"},"status-page":{"api":1}},"network":{"addresses":["fe80::62e3:27ff:fe60:1584","fda0:747e:ab29:9375:62e3:27ff:fe60:1584"],"mesh":{"bat0":{"interfaces":{"wireless":["fe:31:b5:a4:3f:74"],"tunnel":["fe:31:b5:a4:3f:70"]}}}

Something to run on supernodes..
Problem: adding to a multicast group doesn't seem to work.

It would be awesome if there was a way to trigger the hopglass-server to load a nodeinfo update "out-of-order". This could be used by gateway server once a node connected, allowing "live update" of the map.

Although the delay usually is less than a minute, some people are eagerly refreshing the hopglass after connecting the node. As hopglass-server creates a new json for each request anyway, creating them "out-of-order" is not a deal (but was back in times of ffmap-backend).

This can also be used if nodeinfo is updated when the node is "old", so that it won't take 5 minutes (default settings) to update.

Hi,
we have an sitecode filter in hopglass-server, can i implement that in the frontend?

i have tested the follow:

add variable "sitecode" to the config.json

in app.js
for(var h in a.dataPath)c.push(a.dataPath[h]+"nodes.json"),c.push(a.dataPath[h]+"graph.json");
i have modified to that:
for(var h in a.dataPath)c.push(a.dataPath[h]+"nodes.json?filter=site&value="+a.sitecode),c.push(a.dataPath[h]+"graph.json?filter=site&value="+a.sitecode);

but then the statistics page gateway used by nodes not shown right.

can you help me fix this problem?

sorry for my bad english
Thanks

I just used the bootstrap.sh on Debian 10 (buster), and go t some problems:

1. pidof systemd does not return anything, so it sais, systemd is not installed
2. in the process, there is the command apt-get remove nodejs nodejs-legacy npm which results in also removing important still needed packages:

Die folgenden Pakete werden ENTFERNT:                               
  jekyll node-* nodejs npm ruby-coffee* ruby-execjs ruby-jekyll-coffeescript ruby-turbolinks ruby-uglifier

This wouln't be good, if you continue with Yes, since those packages are important on my system.

How can we prevent this? maybe add --no-recommends ? I asked here and here

Bind9 won't load the zone if the serial is longer then 10 digits.

I solved this issue by using seconds instead of milliseconds.

var seconds = Math.round(new Date().getTime()/1000)

…

stream.write(' ' + seconds + ' ; serial number\n')

As I never did anything in node js until this evening, there is probably a better way of solving this issue. :)

I would like to have a table of data available for the hopglas viewer, aggregated like the one coming out of
https://github.com/freifunkMUC/nodelist

Hallo,

ich bin gerade dabei, mir die Provider ein wenig anzupassen.
Ich möchte nun folgendes erreichen: Unsere nodes.json werden viel zu groß und wir möchten diese in einzelne Dateien ablegen. Für jeden Node soll eine .json Datei auf dem Webserver des Hopglass bereitgestellt werden.

Das Format der Provider ist mir bekannt, jedoch hapert es bei der Ausgabe der Datei. Welche Ansätze gibt es hier?

Is there a thogough install guide for a Freifunk-newbie?

I got hopglass-server running on our gateway 18 with the install guide from the README. Now I try to get data into my hopglass-server, but there is no data collected:

http://18.gw.freifunknord.de/hopglass/nodes.json

We use gluon 2016.2.5: https://github.com/freifunk-nord/nord-site

Up to now, we still use alfred, but we want to get rid of alfred.

I started an apache proxy and opened port 1001 UDP and Port 80 in our firewall.

What am I missing? How should the data get to the hopglass-server?

Can you add more info in the readme, so a new comer can successfully run hopglass with hopglass-server?

i would love to see netjson-support in hopglass-server :)

When piping compressed and uncompressed data into a file, the compressed data seems to be larger.

I will turn compression off by default for the next release

Wäre es nicht besser die Config-Datei besser zu strukturieren, um dann den jeweiligen Modulen nur den für sie wichtigen Teil zu übergeben?

Ich habe dazu schonmal eine Struktur auf Basis deiner Module konzipiert:

{
  "receiver": {
    "announced": {
      "target": {
        "ip": "ff02::1",
        "port": 1001
      },
      "port": 45123,
      "interval": {
        "statistics": 60,
        "nodeinfo": 500
      }
    },
    "ifaces": [
      "bat0",
      "enp4s0"
    ]
  },
  "provider": {
    "hopglass": {
      "offlineTime": 1800
    }
  },
  "webserver": {
    "ip": "::",
    "port": 4000
  }
}

( https://gist.github.com/anoymouserver/5794ef8cfd688afe2745b89826c18f28 )

We installed the hs and it seems to work everything, but on our map all nodes are double with slightly different TQ values: https://map.freifunk.in-kiel.de/#!v:m;n:687251221506

You need to add this line to your /etc/hosts to see the site:

138.201.16.163  map.freifunk.in-kiel.de hopglass.freifunk.in-kiel.de

it will be nice if there is any dummy documenation for config hopglass-server on a debian system.

is there any argument for webserver queries?

If there is a device reporting a totally new traffic value to hopglass-server (some values not reported or a new device) and we try to calculate a rate from it, there can be peaks, which are obviously wrong.

The error is logically explainable:
There is just a new or unusual high summand which can't be recognized by prometheus.
A workaround is to sum the rates of single nodes, but this gives a heavy load to grafana clients.

I don't have a practical solution yet, but maybe someone has a good idea?

On the connection view you see only the mac addresses of our vpn server.
But not the name, we have also defined them in the aliases.json but its doesn't work really.

It will be great if the respond collector can collect the firmware version of a node.

We are hosting multiple communities on the same infrastructure. E.g. multiple netmons, multiple fastds etc.
Therefore I'm trying to deploy hopglass-server in the same way and given the systemd layout with @ services I assumed this would work already.
Apparently the collector port 45123 is bound globally though instead of just per configured iface which prevents multiple hopglass-server instances from starting.

I am not sure what's the current development branch as 0.1rc1 is not in master yet. There are huge differences in config file handling between master and 0.1rc1 so I was not sure, on which branch I should base a pull request. Do you consider master branch to be stable or development branch?

curl 'http://192.168.1.74:4000/graph.json'
{"version":1,"batadv":{"multigraph":false,"directed":false,"nodes":[],"links":[],"graph":null},"timestamp":"2016-02-05T18:06:05.531Z"}

curl 'http://192.168.1.74:4000/metrics' | grep meshnodes
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 451k 0 451k 0 0 619k 0 --:--:-- --:--:-- --:--:-- 618k
meshnodes_total 768
meshnodes_online_total 724

IP:4000/site-code/nodes.json or graph.json should return a filtered version
Later a universal filter

I would like to replace all occurrences of @ with -at- in the nodes.json file, before it is served.

WHere would I have to search for the code part, that is serving the nodes.json?

The number of online nodes is reported inconsistently (973 vs. 994):
http://map.freifunk-stuttgart.de/

1102 Knoten (973 online, 576 neu, 134 verschwunden), 2709 Clients, 9 Gateways

Status
online 994
offline 113

Any ideas? Where and how are the two values computed?

Would be awesome if we could add a possibility to provide a Freifunk API file through hopglass-server. A request to /api/xxx.json could return a provided api file from ./api/xxx.json (custom path possible of course) with nodes count for site xxx and lastchange populated. This way the API file is automatically kept up-to-date - and when being there you can even add json schema validation for the API file, to ensure people don't mess things up :)

don't write it to data.json
merge each nodes' content?

This should e.g. prevent wrong client-Counts in HopGlass-Map

I tried to configure the nodes.zone output but I cannot figure out how:

My config.json snippet looks like this:

  "provider": {
    "offlineTime": 900
    "providers": [
      { "module": "named",
        "config": {
          "origin": "nodes.freifunk-neuendettelsau.de.",
          "ns": "ns1.freifunk-neuendettelsau.de.",
          "postmaster": "[email protected].",
          "nameservers": [
            "ns1.freifunk-neuendettelsau.de.",
            "ns2.freifunk-neuendettelsau.de."
          ],
        }
      }
    ]
  },

The problem is that data from L2TP nodes isn't reliable. The tunneldigger package hasn't implemented respondd stuff yet so it doesn't send any data when used in combination with gluon v2015.something or later. This breaks uplink detection and - in some cases - vpn link detection.
If one side reports L2TP that must be the preferred over the other. The other side must also receive uplink status.

ping @mar-v-in
I would appreciate a new PR to fix this. I won't write any code this week.

Because development of hopglass and hopglass-server is relatively tightly coupled, it would be useful to have an easy way to set up a testing instance of a current hopglass-server without actually starting any data collection on a Freifunk node farm.

For that reason it would be nice if hopglass-server had a way to serve a (small, static) set of mock data.

Alternatively: Provide a web service with any data that always runs the newest hopglass-server version.

Hi,
the Wifi Aliases list not work by us.
here an example:
Node primary Mac: e8:94:f6:90:bf:8c
Wifi Mac : 6e:87:70:c5:b4:20
Hopglass Alias entry: ea:96:f7:90:bf:8c|xxxx (e8:94:f6:90:bf:8c)

the wifi mac ist builded with:
nodeinfo.network.mesh.bat0.interfaces.wireless
-1 at last segment is 2,4ghz
-2 at last segment ist 5 ghz

i have try to change in hopglass server but it stop working

we use gluon 2016.2.1 with tunneldigger

here the raw json:
"network": { "addresses": [ "2a03:2260:300c:200:ea94:f6ff:fe90:bf8c", "fe80::ea94:f6ff:fe90:bf8c" ], "mesh": { "bat0": { "interfaces": { "wireless": [ "6e:87:70:c5:b4:21" ], "other": [ "4a:7d:0c:55:f9:a9", "6e:87:70:c5:b4:23" ] } } }, "mac": "e8:94:f6:90:bf:8c"

We already have another nodeJS instance on our server, that needs v0.10.29 to run

I think the only solution to install different apps that use different nodejs versions on the same server would be docker.

Could you provide a docker install instruction, to insstall hopglass-server with the needed dependencies in a doker container?

When there is a downtime in hopglass-server, it will expose old statistics values to prometheus, resulting in wrong datasets being written to the prometheus database. You should not report statistics data if it's old, e.g. Traffic data will be completely misleading: http://i.imgur.com/lPuZhZW.png (obviously there was no jump in traffic usage, just the old value being used on the first report)

Wenn die NodeID eines Knoten ungerade ist, dann werden für diesen Node keine graphen gezeigt.
Es tauchen auch keine Links in der Nodeinfo-Box auf.

Dieses ist relevant, weil Vmware/ESXI per default auch (oder gar mit Vorliebe) ungerade MACs vergibt für die VM-interfaces.

Hotfix: Sobald man im /lib/gluon/core/sysconfig/primary_mac am betreffenen Node in der letzten Stelle einen geraden Wert einträgt sind die Links für den Hoplgass sichtbar. (Mit anderen worten: An /etc/config/network o.ä. muss man nicht dran, da können die ggf. vorhandenen ungeraden MACs bleiben.)
(transfer from hopglass/hopglass#2 )

There are some values in config.json.defaults that are not in the config.json.example:

• receivers->config target and port
• receivers->overlay
• storage interval
• purge interval

Is it ok? Or what happens if those values are missing?
If you cann ommit the values, it should be explained in the README or as comments in the config file.

If I install this like described in the README there is no nodes.json reacheable from outside the server.

I can call it on the console with:

curl 'http://127.0.0.1:4000/nodes.json' 

So please add a simple minimal install instructions how to get the nodes.json out of the server.

Our ffapi.json File is empty.
Can you tell us, how we become an nodelist.json or we can include hopglass data to the FFNet Api File?

bootstrap.sh uses an external source for nodejs, which seems to be problematic because I couldn't install npm afterwards. The reason seems to be that the external nodejs is newer than the one provided by the distro so the dependencies for npm can't be installed.

Is there a good reason to use the nodejs-package from the project site rather than the one provided by the distro? If yes, how can we fix this problem then?

Edit: nervermind, I was misleaded by the fact that npm is available as separate apt-package. I didn't know that it is already available after installing nodejs.

@corny developed the gluon-respondd-airtime package, which was documented here: freifunk-gluon/gluon#826

Hopglass-server should add support, as seen in the respond-collector, to calculate the utilization between the update cycles.

https://github.com/FreifunkBremen/respond-collector/blob/ab01eb0d9ef3f33923982b0e7440c4461e6b9cc3/data/airtime.go#L49

nodes in aliases.json are removed from raw array at startup. I think this should not be the case, because it renders hopglass unusable, if you don't provide certain attributes, like hostname through aliases.json. We used aliases.json to fix location of some nodes before, so we were not required to touch hostname.

With 0.1.1 release (I guess this did not happen on 0.1), we noticed the problem that nodes that are present in aliases.json are not preserved across restarts of hopglass. I was able to trace down this bug:

• here you are removing the lastseen attribute (reseting it). As overlays are applied directly, this happens right at the start of hopglass-server
• here you purge entries without the lastseen attribute set.

This way, not only firstseen is reset (nodes are shown as new after restart), but also nodeinfo is removed. If the overlay contains some attributes of nodeinfo, the incomplete nodeinfo object will be reported to hopglass which is likely to cause bugs (seems like hopglass does not work with nodes that don't have a hostname attribute).

1. Why is it necessary to remove lastseen/lastupdate after restart if there is an aliases entry?
2. I think hopglass should be made more safe, so that broken server results does not cause the entire site to be unusable, but only the single entry being ignored. I guess easiest way is to filter the incoming nodes to have certain fields set - hostname and node_id seem to be important ;)

for monitoring the number of nodes that replied compared to the number of online nodes for example
or the last time we received a message from a node
would be very interesting to try out various timings

I can't get the new repondd multicast address ff05::2:1001 to work, but I don't know if it's nodejs or the network configuration.
The other mcast address ff02::2:1001 works without a problem.

{ Error: getaddrinfo ENOTFOUND ff05::2:1001%br-client
     at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:56:26)
  errno: 'ENOTFOUND',
  code: 'ENOTFOUND',
  syscall: 'getaddrinfo',
  hostname: 'ff05::2:1001%br-client' }

Excerpt of my (not working) config:

{
  "receiver": {
    "receivers": [
      {
        "module": "announced",
        "config": {
          "target": {
            "ip": "ff05::2:1001"
          }
        }
      }
    ],
    "ifaces": [
      "br-client"
    ]
  }
}

for simple tasts such as nodelist.json, nodes.json, maybe a different 'module' for metrics and hosts

read structure from text file
automatically check if data is available
error handling: json parsing and contents

With all the contributors mentioned..
I don't want this to be copyrighted only by me. More people = better.

Currently raw.json is being saved to /etc/hopglass/instance/raw.json. This default does not follow the Filesystem Hierachy Standard¹, as its a state file, not a configuration file.

Please save it to /var/lib/hopglass/instance/raw.json instead.

[1] https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard