ssproessig / ipc-adapter Goto Github PK

View Code? Open in Web Editor NEW

0.0 0.0 0.0 487 KB

Study for an inter-process adapter in C++/Qt 5

License: MIT License

CMake 10.38% C++ 89.40% C 0.22%

ipc-adapter's People

Contributors

Watchers

ipc-adapter's Issues

AmqpComponents: describe and handle error handling

it is possible that Exchange or Queue already exist when we connect AmqpExchangeSink or AmqpQueueSource
hence an error handling is needed here
find out, how qamqp-lib handles errors for that

Sink: Provide AmqpExchangeSink

Sources: Provide UdpSource

Core: Implement self-registering type factory pattern

Converter: provide CustomConverter that converts using JavaScript

Idea

Use Qt's QJsEngine (https://doc.qt.io/qt-5/qjsengine.html) to allow users to define custom conversion functions.

<?xml version="1.0" encoding="UTF-8"?>
<configuration version="1" xmlns="urn:ipca-configuration-v1"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:ipca-configuration-v1 ipca-configuration-v1.xsd">

    <configurations>
        <js:configuration id="cfgHelloWorld" xmlns:js="urn:ipca-configuration-for-custom-converter-v1"><![CDATA[ 
     function(data, metaData) { return "Hello, World: " + data; } 
         ]]>
        </js:configuration>
    </configurations>

    <components>
        <component id="helloWorld" type="CustomConverter">
            <configuration ref="cfgHelloWorld"/>
        </component>

Way Forward

XML handler must support data callback as well
we need #33 for that

Document: Design Decisions

Core: update CMakeLists to carry unit tests CTest compatible

refer to https://cmake.org/cmake/help/latest/command/add_test.html?highlight=s

Components: restructure components by usage, not component type

Current Situation

currently components are structured by their component-type (Converters, Sinks, Sources)
this may lead to duplicate code, if Sinks/Sources share code (shared configuration parsers)

Expected Situation

group components in logical packages
- UdpSink and UdpSource go to NetworkComponents
- AmqpExchangeSink goes to AmqpComponents

Dependencies

This is a prerequisite to add further AMQP components.

Core: Refactor IComponent w/ mandatory getConfigurable() to interface based abstraction

Todo

don't make all IComponent have to implement getConfigurable()
instead provide a new Core interface IProvides<T> with method T* get() const and use it as IProvides<IConfigurable> in those components that really are configurable

Implications

some tests need to be update
Runtime ConfigurationHandler

Details

Status	Code	File	Line	Column	Project
Confirmed reference	Core::IConfigurable* getConfigurable() override;	W:\20_Sources\IpcAdapter\Components\Sources\src\UdpSource.h	39	23	IpcAdapterSources
	class IpcAdapter::Core::IConfigurable	W:\20_Sources\IpcAdapter\Core\api\IConfigurable.h	21	9	
Confirmed reference	IpcAdapter::Core::IConfigurable* ReverseBytesConverter::getConfigurable()	W:\20_Sources\IpcAdapter\Components\Converters\src\ReverseBytesConverter.cpp	18	19	IpcAdapterConverters
Confirmed reference	: Core::IConfigurable	W:\20_Sources\IpcAdapter\Components\Converters\src\XmlToMetaDataConverter.cpp	68	13	IpcAdapterConverters
Confirmed reference	IpcAdapter::Core::IConfigurable* XmlToMetaDataConverter::getConfigurable()	W:\20_Sources\IpcAdapter\Components\Converters\src\XmlToMetaDataConverter.cpp	109	19	IpcAdapterConverters
	class IpcAdapter::Core::IConfigurable	W:\20_Sources\IpcAdapter\Core\api\IConfigurable.h	21	9	
Confirmed reference	Core::IConfigurable* getConfigurable() override;	W:\20_Sources\IpcAdapter\Components\Converters\src\ReverseBytesConverter.h	35	23	IpcAdapterConvertersTest
Confirmed reference	Core::IConfigurable* getConfigurable() override;	W:\20_Sources\IpcAdapter\Components\Converters\src\XmlToMetaDataConverter.h	39	23	IpcAdapterConvertersTest
Confirmed reference	std::function<void(IpcAdapter::Core::IConfigurable&)> configurationCallback,	W:\20_Sources\IpcAdapter\Components\Converters\tst\XmlToMetaDataConverterTest.cpp	19	46	IpcAdapterConvertersTest
	class IpcAdapter::Core::IConfigurable	W:\20_Sources\IpcAdapter\Core\api\IConfigurable.h	21	9	
Confirmed reference	using IpcAdapter::Core::IConfigurable;	W:\20_Sources\IpcAdapter\Core\src\ConfigurationHandler.cpp	22	25	IpcAdapterCore
Confirmed reference	IConfigurable* currentConfigurable = nullptr;	W:\20_Sources\IpcAdapter\Core\src\ConfigurationHandler.cpp	53	9	IpcAdapterCore
Confirmed reference	IpcAdapter::Core::IConfigurable* getConfigurable() override	W:\20_Sources\IpcAdapter\Core\src\RuntimeConfiguration.cpp	29	27	IpcAdapterCore
	class IpcAdapter::Core::IConfigurable	W:\20_Sources\IpcAdapter\Core\api\IConfigurable.h	21	9	
Confirmed reference	IpcAdapter::Core::IConfigurable* getConfigurable() override	W:\20_Sources\IpcAdapter\Core\tst\ComponentRegistryTest.cpp	29	27	IpcAdapterCoreTest
Confirmed reference	using IpcAdapter::Core::IConfigurable;	W:\20_Sources\IpcAdapter\Core\tst\PipelineTest.cpp	16	25	IpcAdapterCoreTest
Confirmed reference	IConfigurable* getConfigurable() override	W:\20_Sources\IpcAdapter\Core\tst\PipelineTest.cpp	32	9	IpcAdapterCoreTest
Confirmed reference	IConfigurable* getConfigurable() override	W:\20_Sources\IpcAdapter\Core\tst\PipelineTest.cpp	58	9	IpcAdapterCoreTest
Confirmed reference	using IpcAdapter::Core::IConfigurable;	W:\20_Sources\IpcAdapter\Core\tst\RuntimeTest.cpp	18	25	IpcAdapterCoreTest
Confirmed reference	, IpcAdapter::Core::IConfigurable	W:\20_Sources\IpcAdapter\Core\tst\RuntimeTest.cpp	28	29	IpcAdapterCoreTest
Confirmed reference	IConfigurable* getConfigurable() override	W:\20_Sources\IpcAdapter\Core\tst\RuntimeTest.cpp	30	9	IpcAdapterCoreTest
Confirmed reference	IConfigurable* getConfigurable() override	W:\20_Sources\IpcAdapter\Core\tst\RuntimeTest.cpp	61	9	IpcAdapterCoreTest
Confirmed reference	IConfigurable* getConfigurable() override	W:\20_Sources\IpcAdapter\Core\tst\RuntimeTest.cpp	77	9	IpcAdapterCoreTest
Confirmed reference	IConfigurable* getConfigurable() override	W:\20_Sources\IpcAdapter\Core\tst\RuntimeTest.cpp	94	9	IpcAdapterCoreTest
	class IpcAdapter::Core::IConfigurable	W:\20_Sources\IpcAdapter\Core\api\IConfigurable.h	21	9	
Confirmed reference	struct AmqpConfigurable: IpcAdapter::Core::IConfigurable	W:\20_Sources\IpcAdapter\Components\Sinks\src\AmqpExchangeSink.cpp	91	48	IpcAdapterSinks
Confirmed reference	IpcAdapter::Core::IConfigurable* AmqpExchangeSink::getConfigurable()	W:\20_Sources\IpcAdapter\Components\Sinks\src\AmqpExchangeSink.cpp	210	19	IpcAdapterSinks
Confirmed reference	: IpcAdapter::Core::IConfigurable	W:\20_Sources\IpcAdapter\Components\Sinks\src\UdpSink.cpp	29	25	IpcAdapterSinks
Confirmed reference	IpcAdapter::Core::IConfigurable* UdpSink::getConfigurable()	W:\20_Sources\IpcAdapter\Components\Sinks\src\UdpSink.cpp	96	19	IpcAdapterSinks
	class IpcAdapter::Core::IConfigurable	W:\20_Sources\IpcAdapter\Core\api\IConfigurable.h	21	9	
Confirmed reference	Core::IConfigurable* getConfigurable() override;	W:\20_Sources\IpcAdapter\Components\Sinks\src\AmqpExchangeSink.h	40	23	IpcAdapterSinksTest
Confirmed reference	using IpcAdapter::Core::IConfigurable;	W:\20_Sources\IpcAdapter\Components\Sinks\tst\AmqpExchangeSinkTest.cpp	15	25	IpcAdapterSinksTest
Confirmed reference	std::function<void(IConfigurable&)> configurationCallback,	W:\20_Sources\IpcAdapter\Components\Sinks\tst\AmqpExchangeSinkTest.cpp	61	28	IpcAdapterSinksTest
Confirmed reference	Core::IConfigurable* getConfigurable() override;	W:\20_Sources\IpcAdapter\Components\Sinks\src\UdpSink.h	39	23	IpcAdapterSinksTest
Confirmed reference	std::function<void(IpcAdapter::Core::IConfigurable&)> configurationCallback,	W:\20_Sources\IpcAdapter\Components\Sinks\tst\UdpSinkTest.cpp	23	46	IpcAdapterSinksTest
	class IpcAdapter::Core::IConfigurable	W:\20_Sources\IpcAdapter\Core\api\IConfigurable.h	21	9	
Confirmed reference	FORWARD_DECLARE(IConfigurable);	W:\20_Sources\IpcAdapter\Core\api\IComponent.h	13	25	IpcAdapterSources
Confirmed reference	virtual IConfigurable* getConfigurable() = 0;	W:\20_Sources\IpcAdapter\Core\api\IComponent.h	34	21	IpcAdapterSources
Confirmed reference	class IConfigurable	W:\20_Sources\IpcAdapter\Core\api\IConfigurable.h	21	15	IpcAdapterSources
Confirmed reference	INTERFACE_DESTRUCTOR(IConfigurable);	W:\20_Sources\IpcAdapter\Core\api\IConfigurable.h	24	34	IpcAdapterSources
Confirmed reference	struct UdpSource::Data: Core::IConfigurable	W:\20_Sources\IpcAdapter\Components\Sources\src\UdpSource.cpp	37	31	IpcAdapterSources
Confirmed reference	IpcAdapter::Core::IConfigurable* UdpSource::getConfigurable()	W:\20_Sources\IpcAdapter\Components\Sources\src\UdpSource.cpp	146	19	IpcAdapterSources
	class IpcAdapter::Core::IConfigurable	W:\20_Sources\IpcAdapter\Core\api\IConfigurable.h	21	9	
Confirmed reference	std::function<void(IpcAdapter::Core::IConfigurable&)> configurationCallback,	W:\20_Sources\IpcAdapter\Components\Sources\tst\UdpSourceTest.cpp	33	46	IpcAdapterSourcesTest

Document: Coding Rules

Init: Set up Repository

Sources: Provide AmqpQueueSource

Todo

provide a new source that receives from AMQP queues
additional needed parameters compared with AmqpExchangeSink:
- queue name and binding key

Way Forward

refer to https://github.com/ssproessig/amqp-training/blob/master/qt/src/consumer.cpp#L36

Refactoring

~~extract AMQP configuration parser from AmqpExchangeSink to a common shared class~~ ✅
make the relevant parts virtual to unit test invocation

Source: Provide a TIBCO EMS Source

find out what needs be configurable to receive from TIBCO EMS with C API

C API: see https://docs.tibco.com/pub/ems-zlinux/8.5.0/doc/pdf/TIB_ems_8.5_c_and_cobol_ref.pdf?id=2

Core: add meta-data to IPipelineFrame

Core: provide core abstractions

Sinks: Provide UdpSink

Converters: Provide XmlToMetaDataConverter

3rd Party: integrate qamqp-lib

Document: Naming Conventions

Core: provide a Watchdog abstraction

Todo

components SHALL be cyclically asked if they are still OK to work
overall Runtime shall trigger an IWatchdogTrigger implementation to report "aliveness" to an external supervisor

Documentation: fix requirements tracing from system requirements / add missing runtime requirements

Bug: building with ninja fails sometimes

Bug

Sometimes building with make -j7 fails with

[ 90%] Linking CXX executable IpcAdapterConvertersTest
make[2]: *** No rule to make target '20_Sources/IpcAdapter/Components/Sinks/CMakeFiles/IpcAdapterSinks.dir/src/UdpSink.cpp.o', needed by '20_Sources/IpcAdapter/Components/Sinks/IpcAdapterSinksTest'.  Stop.
make[2]: *** Waiting for unfinished jobs....
[ 91%] Building CXX object 20_Sources/IpcAdapter/Components/Sinks/CMakeFiles/IpcAdapterSinksTest.dir/tst/SinkTestMain.cpp.o
[ 91%] Built target IpcAdapterConvertersTest
Scanning dependencies of target IpcAdapterApplication
[ 92%] Building CXX object 20_Sources/IpcAdapter/Application/CMakeFiles/IpcAdapterApplication.dir/IpcAdapterApplication_autogen/mocs_compilation.cpp.o
make[2]: *** No rule to make target '20_Sources/IpcAdapter/Components/Sinks/CMakeFiles/IpcAdapterSinks.dir/src/UdpSink.cpp.o', needed by '20_Sources/IpcAdapter/Application/IpcAdapterApplication'.  Stop.
make[2]: *** Waiting for unfinished jobs....
[ 94%] Building CXX object 20_Sources/IpcAdapter/Application/CMakeFiles/IpcAdapterApplication.dir/src/IpcAdapterApplication.cpp.o
[ 95%] Building CXX object 20_Sources/IpcAdapter/Components/Sinks/CMakeFiles/IpcAdapterSinks.dir/src/AmqpExchangeSink.cpp.o
[ 97%] Linking CXX executable IpcAdapterSourcesTest
[ 97%] Built target IpcAdapterSourcesTest

Idea

maybe it has to do with add_library( ... OBJECT... ) ?

qamqp-lib: fix clang-10 compiler warnings, MSVC Code Analysis, Coverity Scan result

mostly C++>11 related warnings (nullptr vs. 0, dead code semicolon)
some seem more relevant
- missing virtual dtor for classes with virtual methods
- narrowing for key things like channel numbers
MSVC analysis and Coverity Scan found uninitialized members

Core: provide means for complex configurations for components

Problem

Some components (as the ByteStreamReader and ByteStreamWriter) need a more complex structured configuration element in our XML, as basically key-value pairs is not sufficient for them.

Idea

Make components provide their own XML namespace and configuration handler for that.

<?xml version="1.0" encoding="UTF-8"?>
<configuration version="1" xmlns="urn:ipca-configuration-v1"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:ipca-configuration-v1 ipca-configuration-v1.xsd">

    <configurations>
        <s:configuration id="cfgSample" xmlns:s="urn:ipca-configuration-for-sample-v1">
            <s:root version="1">
               <s:any>
                   <s:other>
                       <s:structure is="possible"/>
                   </s:other>
                </s:any>
            </s:root>
        </sample:configuration>
    </configurations>

    <components>
        <component id="otherComponent" type="SampleComponent">
            <configuration ref="cfgSample"/>
        </component>
....

Way Forward

update schema to allow sub-schemas
add IXmlDefaultHandler* abstraction
- as we only handle startElement(...) and endElement(...) from QXmlDefaultHandler we can't use it directly
- add a method bool configure(IComponent& aComponent) so that the handler instance can configure a component with the information it parsed
update ConfigurationHandler implementation of HandlerStack to carry IXmlDefaultHandler*

Design

XmlHandlerFactorys are stored in a GlobalXmlHandlerFactoryRegistry (same mechanism as used for component factories)
when the configuration handler encounters a configuration element , it queries the GlobalXmlHandlerFactoryRegistry for a factory for the XML namespace encountered
- fatal error, if no handler was found,
- otherwise create new instance and put it on top of the handler stack
hence HandlerFactory must be registered by the XML namespace they can parse
as configuration happens at start-up time and not at run-time its okay to throw exceptions in the handlers

Dependencies

#31 as we then just realize IProvides<IConfigurationHandler>
#32 as both will pretty much need the same configuration

Core: Provide configuration XSD and parser

App: Provide IpcAdapterApplication

Core: provide minimal logging abstraction

Document: Source Code Documentation

Converter: Provide "declarative" ByteStreamReaderConverter and ByteStreamWriterConverter

Todo

provide a converter component that can be configured to read raw bytes received into meta-data KVPs
make the parsing rules configurable
for now support
- q(u)int8|16|32|64
- PascalString (with encoding)
- C-Style String (with encoding)
- FixedLength String (with encoding)

Converters: Provide ReverseBytesConverter

CI/CD: integrate appveyor / coverage

Document: System Requirements

Converter: Provide "declarative ByteStreamWriter"

Todo

other way around from ByteStreamReader: we have meta-data key-value pairs and want to encode them as raw data

Core: Provide Runtime

Core: make <param/> list reusable between <component/>'s

Idea

reuse the same parameters between different components to avoid duplication, especially handy for large configuration lists (as AMQP configurations)
introduce parameter references for that

How it shall look in XML

  
<?xml version="1.0" encoding="UTF-8"?>
<configuration version="1" xmlns="urn:ipca-configuration-v1"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:ipca-configuration-v1 ipca-configuration-v1.xsd">

    <param-lists>
        <param-list id="paramsHostConfiguration">
            <param key="host" value="localhost"/>
        </param-list>
    </param-lists>

    <components>
        <component id="udpSource" type="UdpSource">
            <param-list ref="paramsHostConfiguration"/>
            <param key="port" value="9999"/>
        </component>

        <component id="udpSink" type="UdpSink">
            <param-list ref="paramsHostConfiguration"/>
            <param key="port" value="9998"/>
        </component>
...

Design Idea

~~create Core::IConfigurator that provides bool doConfigure(IConfigurable&) method; returns false if any parameter was rejected~~ ✅
~~provide private realization ParamListConfigurator that is IConfigurable and only stores the key and value passed to it~~ ✅
~~store ParamListConfigurator instances in RuntimeConfiguration with addParamList(QString const& anId, shared_ptr<ParamList> const&)~~ ✅
~~make it queryable via IRuntimeConfiguration::getParamList(QString const&)~~ ✅
~~extend the <component/> parser to resolve it from <param-list ref=""/> and apply it~~ ✅

CI/CD: integrate CPack

to create a RPM package for later Ansible integration

ssproessig / ipc-adapter Goto Github PK

ipc-adapter's People

Contributors

Watchers

ipc-adapter's Issues

Idea

Way Forward

Current Situation

Expected Situation

Dependencies

Todo

Implications

Details

Todo

Way Forward

Refactoring

Todo

Bug

Idea

Problem

Idea

Way Forward

Design

Dependencies

Todo

Todo

Idea

How it shall look in XML

Design Idea

Recommend Projects

Recommend Topics

Recommend Org