schoenw / lmapd Goto Github PK

(whishlist)

Some areas of the code still handle "suppress-by-default" (regardless of it being fully implemented or not in the code as a whole).

This is functionality that was present in drafts, but was dropped from the final RFCs, so it should be removed from the reference implementation.

@schoenw go to https://travis-ci.org/, sign in with GitHub and enable the build for lmapd from the repo list (https://travis-ci.org/profile/schoenw I think will be the repo list).

Currently lmapd can render its output in either XML or JSON, but all of the input files (config, schedule...) must be XML files.

This means libxml2 is required, and libxml2 is quite large as far as FLASH usage for OpenWRT-based embedded devices go. If support for JSON input is added to lmapd, XML support can be made a compile-time option, and the dependency on libxml2 can be dropped when necessary (at the cost of XML support, of course).

We will start working on JSON input support for lmapd at the simetnicbr/simet-lmapd fork, and will submit a PR when ready.

lmapd does not support (or implement) all of the functionality present in the RFCs. The SIMET team from NIC.br will need some of the missing functionality, so we plan to implement them at least partially in the simetnicbr/simet-lmapd fork.

• ietf-lmap-report::result::table::function
• ietf-lmap-report::result::table::column
• ietf-lmap-report::result::conflict (low priority for us)
• ietf-lmap-control::schedule::action::parameters and ietf-lmap-report::result::parameters (only if we require it for scheduler extensions)

(we will edit this issue as we identify more functionality that we need to implement).

The runner depends on a child exiting to do the schedule cleanup (remove schedule from RUNNING state if every action is done). A comment in lmapd_cleanup() says otherwise, but nothing in the event loop is calling lmapd_cleanup() -- instead, it is being called from a signal handler context for SIGCHLD.

This means a schedule that has no actions will never leave the RUNNING state once it runs. And that also means it will cause lmapd to issue warnings if it tries to run again, e.g., because it was driven by a recurring event.

It would make a lot more sense for schedules with no actions to end immediately (with a successful state!), unless the LMAP model defines the opposite very explicitly. Whether such a schedule would be useful at all, is up for discussion though.

If the schedule has actions and all of them are suppressed (or for any other reason they do not cause a fork() to happen), this might well cause lmapd to misbehave.

Currently, the source code lacks copyright information headers on the files, as well as a COPYING or LICENSE file with the GPL version 3 text. This is a problem for Linux distros (e.g. Debian would not accept it for inclusion), at the very least.

The name of the license the code is released under is only present in README.md (GPLv3), and it does not mention the main authors or copyright date. It is also not extremely clear whether the code is released under the GPLv3 only (atypical), or "GPL version 3 or any later version of the GPL" (typical).

The license text itself is not present anywhere.

We'd be willing to help on this if you don't want to waste time with the busywork of annotating every source file: just tell me the copyright line you'd like to have added to every source file, e.g. Copyright (c) #### (your real name, or the name of the institution), and which version of the GPLv3 (just v3, or v3 and later) you want, and I will submit a PR.

For the record, unless requested otherwise, any changes contributed by NIC.br are/will be:
Copyright (c) 2019 NIC.br
And distributed under the same license(s) as the source file they apply to.

Context:
According to RFC 8193 section 4.6.2, and also RFC 8194 section 4.3, the ma-report-result-event-time (RFC 8193) or "leaf event" (RFC 8194) must be the time when the event that triggered the SCHEDULE (that caused that action to "run") happened, prior to any randomization/spread.

This behavior is needed in order to be able to correlate back the reports of distinct actions to a specific invocation of a specific schedule that ran those actions (i.e. to know that the results of those actions "belong together").

Description of the issue:
When one has a schedule that triggers (sequentially) three actions that take a few seconds to run each, the "event" times in the report are not equal as they ought to be. The second action reports an "event" time that is a few seconds later than the first one.

The use of -Wall -Werror is known to cause problems when compilers add new warnings, and at least one such issue is happening right now with gcc 6.3 (Debian stable):

 /tmp/x/lmapd/src/csv.c:24:19: warning: ‘delimiter’ defined but not used [-Wunused-const-variable=]
 static const char delimiter = ';';

This can be worked around by removing -Werror in the toplevel CMakeLists.txt.

The YANG models should be updated to their latest "RFC-blessed" versions, and any relevant changes (if any) should be addressed in the lmapd code.

This is somewhat relevant because lmpad is a reference implementation...

The current data flow from an action (in schedule A) to destination(s) in schedule B... is:

• If an action returns non-zero, its results are destroyed
• If an action returns zero, and has destinations, its output is moved to the destination schedule(s) "base directory", regardless of whether that schedule is already running or not.
• There is no data life-cycle management implemented to deal with consumed data by an action

This really gets in the way of implementing resilient reporting of results to a collector. The desired report flow is: render the report, optionally compress it, and if this fails (e.g. storage full), don't remove the source data. After the report is rendered successfully, remove only the source data that is present in that report, and queue the report for transmission. The report is only removed when the transmission succeeds. The render+transmit steps may be implemented atomically, and then it becomes "the source data present in the report must only be removed when the transmission succeeds".

Proposed action -> schedule data-flow:

1. Action output is sent to an "incoming" queue on each schedule (it can continue to be the base directory of the schedule.
2. Data can arrive at the "incoming" queue of a schedule at any time, including while that scheduling is already running. Either advisory locking or "write then rename" strategies must be used to ensure no "live" pair of (data, meta) files exist. This is already true as implemented.
3. When the runner will start processing a schedule, it moves every pair of (meta, data) files to a "processing" queue of that schedule.
4. actions must consume data only from the "processing" queue of their schedule.
5. If, and only if, every action of a schedule returns a zero status, the "processing" queue is emptied by the runner when the schedule finishes running. If any of the actions return a non-zero status or no action exists, the "processing" queue is left alone (i.e. accumulates data).

The SIMET team will implement the proposed action above, and it will (as usual) be made available for merge upstream in our fork. If we find any problems with the proposed solution, we will edit this issue report accordingly.