gregutas / robust-services-core Goto Github PK

Define object pools for IpBuffer bytestreams to replace the use of Memory::Alloc. Each pool would support a maximum payload size (e.g. 20, 80, 240, 1000, and 4000 bytes).

Support the additional overload control strategies described in "Handling Overload". Factory subclasses decide which strategy to use.

Implement Message.InspectMsg and suitable overrides. This function should be invoked when a message is sent, but only when a debug flag is enabled in a lab load. Its purpose is to verify that the message

• belongs to a valid protocol
• matches its PSM's protocol (if sent by a PSM)
• has a valid signal for its protocol
• contains all parameters that are mandatory for its signal
• contains no parameters that are invalid for its signal

Implement the journal file capability described in "Escalating Restarts".

Implement a LeakyBucketCounter for the scheduling timeout in RootThread.

Re-inject messages that are still queued on a Context when it idles. The Factory or Protocol associated with a message must be consulted to decide whether to re-inject or discard it.

Implement a testcase database and code coverage tool as described in "General Testing".

Implement the capabilities described in "Session Testing".

It invokes MsgPortPool.FindPeerPort, which searches through all ports. A hash table that maps source addresses to destination addresses is one way to improve this. An entry would be created when a destination address was assigned to receive a source address's initial message. The entry would be deleted when the source address sent a message directly to the destination address. However, this would require a large hash table when a protocol often had a significant delay between an initial message and a subsequent message sent in the same direction.

A thread is currently killed via Thread::FunctionInvoked, which is called by Debug::ft. A thread must therefore invoke a traced function to get killed. A more robust way to kill a thread is to change its instruction pointer so that it resumes execution in a function that throws an exception. However, implementing this is platform-specific. On Windows, for example, it probably means using CONTEXT, GetThreadContext, and SetThreadContext.

Provide local and remote versions of OAM&P threads for nodes that do not have a disk or console. See "Networked Threads".

Implement the Dependencies function for each Module and enhance ModuleRegistry to calculate its module initialization order based on these dependencies (a partial ordering).

As the POTS application undergoes distribution, the integrated Operations/Control Node portion of the executable should propagate data changes to the subset of nodes that require it, as described in "Data Distribution". See #37, #38, #39, and #40.

Enhance the log framework as described in "Logs and Alarms".

Implement a Concatenate function for Q1Way and Q2Way. It would empty one queue by moving its elements into another queue.

Create a separate Service Node (sn directory) that runs the POTS basic call and supplementary services software. Afterwards, an integrated Operations/Control Node will remain. See "Networking".

This should be the final phase in distributing the POTS application, after #38.

If an exception is constructed on a heap, what happens if the heap was out of memory? This is a particular risk with an AllocationException. If an exception is constructed on the stack, what happens if it causes a stack overflow?

Inspect socket functions to determine if there are errors from which they could try to recover. Many of these are tagged with //s comments.

Implement the binary database capability described in "Escalating Restarts".

Support write-protected memory and reload restarts. Modify classes derived from Protected accordingly. See "Escalating Restarts".

The first step could be to implement Shutdown and Startup functions for classes derived from Protected. This would handle deletion and reallocation of the protected heap. Actual memory protection would be introduced as the second step, when these classes would also have to use MemUnprotect and MemProtect outside of restarts.

Implement a web-based tool for generating most of the tedious "boilerplate" code for

• a service's subclasses of
  • Service, State, Event, and Trigger (including many function bodies)
  • Initiator, ServiceSM, and EventHandler (function shells to be filled in with service logic)
• a protocol's subclasses of
  • Protocol, Signal, and Parameter (including many function bodies)
  • InputHandler, Message, and ProtocolSM (function shells to be filled in with protocol logic)
• the Factory subclass (a mix of function bodies and shells)

Verify Object.MorphTo by implementing Class for two classes that share a common base class.

Once #34 has been implemented, add an alarm for system overload. See "Alarms".

Implement the capability described in "Babbling Idiot Controls".

Create a separate Access Node (an directory) that runs the POTS shelf. See "Networking".

This should be the second phase in distributing the POTS application, after #37.

Support IPv6 as well as the existing IPv4.

Verify Class.Create by implementing Class for a class and using Class.SetTemplate to register a template that is used to block-initialize objects of that class. Also verify Object.PostInitialize by implementing it for objects in the class.

After a trap, delay the cleanup of a Context to focus on new work and avoid the risk of retrapping. This probably involves introducing a new Context state (recovering) to block incoming messages and postpone the recovery of subtended objects.

Support an else clause for the CLI's >if command.

InitThread and Thread now schedule all threads as the result of #126 and #127. The remaining work items are

• provide a default timewheel
• add a command for creating a timewheel (based on the percentage of time allotted to each faction)
• add a command for switching to another timewheel
• Thread.Select chooses next thread based on timewheel
• InitThread acts as clock interrupt to advance timewheel
• track the time spent in each faction and display it in >sched show
• can apply this design recursively, to an element manager that supervises multiple executables, giving each one a guaranteed percentage of the CPU

Each thread now waits on a per-thread Windows event to be signalled to run; this would probably be implemented using a condition variable on other systems.

• conditions variable supported in C++11; see documentation
• "spurious wakeup" problem means that, when awoken, thread must check that the underlying condition is actually true
• implement SetEvent and WaitEvent using a condition variable: see article

CliThread usually runs unpreemptably, like other threads. Thus, when the RTC timeout is disabled during debugging (>cfgparms set breakenabled t), a runaway unpreemptable thread would prevent CliThread from running. To ensure that CliThread will run, to allow such a thread to be killed, it should run preemptably, at a high priority, when it sleeps and the RTC timeout is disabled. This might be controlled from BreakEnabledCfg.SetCurr.

The system currently generates a continuous console file and a continuous log file. These should periodically be closed, and new files opened, probably at the same time that each statistics report is generated.

As part of log enhancements, a new log file is now created during each restart, with the previous file being closed. There is still a single console file, with no plan to change this.

If heartbeating should be implemented for threads, then

• when a thread is created, optionally register a function that is invoked if its heartbeat is not received
• InitThread, CinThread, CoutThread, FileThread, LogThread, and TimerThread are recreated when a client indirectly invokes their Singleton.Instance (and so they probably do not require heartbeating)
• a replacement InvokerThread could be created by InvokerPoolRegistry.UnbindThread
• a replacement IoThread could be created by IpPort
• also need to consider CliThread, StatisticsThread, and ObjectPoolAudit

Implement a CLI command for disabling/enabling a modifier. This could be used, for example, to disable a modifier that was frequently trapping.

The size of an object derived from Pooled is limited to the maximum size defined by its ObjectPool. If an object is larger, it must currently allocate unaudited memory. It would be preferable if it allocated an auxiliary data block whose ownership was registered in a new Q1Way added to Pooled. Blocks of various sizes would be defined for this purpose, each size being managed by its own ObjectPool.

Create a separate Routing Node (rn directory) that is queried to determine the Service Node to which a CIP IAM should be sent. See "Networking".

This should be the first phase in distributing the POTS application. It could be implemented in the following stages:

1. Send an internal query (within the same executable) from the POTS call to a new Factory.
2. Send the query to a separate executable (the Routing Node), using the looparound address and a pair of well-known ports.
3. Distribute directory numbers to the Routing Node (the first use case for #41).
4. Use the Routing Node's IP address instead of the looparound address (the first use case for #40).
5. Run two Routing Nodes in a load-sharing configuration (another use case for #40 and #41).

Support rebooting the executable in response to the >restart reboot command. See "Escalating Restarts".

See the notes in Symbol.cpp.

An unmodified function trace captures the chain of constructor calls from base class to leaf class, after which FunctionTrace.InvertCtors fixes this so that the chain runs from leaf class to base class. It also adjusts the time when each constructor was invoked, but does a poor job of it. Revisit the adjustments to see if they can be improved.

As the POTS application undergoes distribution, the Control Node portion of the executable should support the following configurations, as described in "Networking":

• two Routing Nodes (load sharing): see #37
• multiple Access Nodes (cold standby): see #38
• multiple pairs of Service Nodes (warm standby): see #39

Convert to/from big endian when sending/receiving over IP if the local node is little endian.

It turns out this is protocol-specific. A 1-byte field requires no conversion, but 2-byte, 4-byte, and 8-byte fields must be converted in different ways, and only a protocol knows how its messages are internally laid out. Two virtual functions were therefore added to InputHandler, one for converting incoming messages, and one for converting outgoing messages.

Implement the Linux equivalents of Windows-specific files (*.win.cpp).

Implement a LeakyBucketCounter that causes a restart when the system experiences too many traps, which are tracked in ThreadAdmin.stats_.traps_.

Track how much time each Context is using to detect CPU hogs (e.g. a messaging loop between two contexts). Kill a context that used more than x% of the CPU during the last n seconds (configurable).

Implement a protocol stack (e.g. CIP over a simplified SIP) to test the PSM protocol stack capability.

Once Service Nodes are running in a paired cold standby configuration, the framework described in "Checkpointing and Failover" should be implemented to support a warm standby configuration. See #39 and #40.

When TraceBuffer.AddRecord allocates memory for a trace record, it should write a fence pattern after the record. This would allow it to detect trampling on its next invocation. For performance reasons, this should only be done in lab loads.

Fork a thread to perform work (e.g. running test scripts) without tying up CliThread.

• introduce a >run command (similar to >read) to fork a non-console version of CliThread
• CliThread might no longer be a singleton
• update testcase pass/fail counts on the main CliThread

Once the checkpointing framework exists and Service Nodes are running in a warm standby configuration (#42), enhance the Service Nodes and the Control Node to support failover. See "Checkpointing and Failover".

This issue used to be an open-ended question, but each Thread now tracks the mutex on which it is blocked, and each SysMutex tracks the thread that is holding it. There is also a MutexRegistry where all mutexes are registered. Consequently, it is now possible to analyze if a deadlock exists and perhaps even take action to resolve it.