d3scomp / jdeeco Goto Github PK

While testing on the meeting, the leader moved, while the follower did not. Might be there is a bug due to implementation changes.
We could either fix it or remove it altogether. However, I find it a good first-glance example and I vote for fixing it.

Change the API to more conform to the ensemble/component definition. Do not name both component ensembles and mapping function a 'process'.
Doing so might confuse the users since they are used to distinguish components and ensembles at the conceptual level, not 'processes' and 'knowledges'.
Do not treat knowledge separately from process (at the top level level of API) .
Keeping this distinction will prevent us from introducing a compact in-memory representation of components/ensembles.

This is a high-level task, which enables the use of JPF to verify a DEECo system. Itself it is partitioned into the following sub-tasks:

• Replacement of the Apache River tuple space with a hashmap (and copying data upon retrieval from the hashmap so as the behavior is the same as in the distributed case). (Represented by issue #15.)
• Integration with JPF and a way of launching this using JPF
• A demo showing the integration of JDEECo and JPF (leader/follower demo?)

Problems are typically:

• reflection methods for generic types
• UUID get random string (can be worked around by manually setting the component's ids)

Solution is to separate the pre-processing based on reflection into a standalone phase (issue #26).

Add new function allowing to package classes pointed by a user into the OSGi bundles.
This way component and ensemble packaging process will be automated.

When the demo is run on Linux - via "start-ar" and "demo" targets. It makes some non-trivial pauses in execution (i.e. print outs of the progress). I'm not sure how it behaves on Windows and if it wasn't caused by the new mechanism of executing Apache River. But most likely it seems like a timeout in transaction rollbacks. (??)

Would be good to test it on Windows (along with the new way of executing Apache River via the Ant task). If it also lags, it would be good to find out why.

The compilation/packaging/running demo/... with the ant file should work equivalently on Windows and Linux.

Package Components, Ensembles and runtime into OSGI bundles.
Runtime should listen for Component and Ensemble registration, instantiate them in case of registration event and serve their instances within the runtime.

At the moment the functionality there is simplified and does not consider different situations.

It seems that we may need to manipulate just a single item in a hashmap. An example is that an ensemble update only a position of a particular robot, however the robot positions are stored in a hashmap indexed by robot ids. Similarly a process in a component may want to "communicate" with a specific party from a number of parties which dynamically change.

Below is a proposal how to realize this. The proposal assumes only a simple case when we want to modify one item. For more complex modifications of the hashmaps, it is probably better to use programmatic approach, when the process manipulates the whole hashmap while being in a transaction for the whole time. Note however, that this requires implementation of the more strict transactions (I'll add it as a separate issue).

We would extend the path specification in the following way for process specification: xxx.otherRobots[yyy.robotId]. xxx.otherRobots is a path to the hashmap. yyy.robotId is a path to a string. The semantics is such that it corresponds to a field in the hashmap indexed by the content of the yyy.robotId string.

For ensembles, we would introduce two namespaces member and coord. All the path expressions would be prefixed by either member or coord. Then we could write an expression like coord.allRobots[member.id], which would mean a field in a hashmap on a coordinator that is indexed by the member's id.

With such and extension to ensembles, we can drop the distinction in annotations. So we can have same annotations for processes as well as for ensembles. I.e. @DEECoIn, @DEECoOut, @DEECoInOut, instead of @DEECoProcessIn, @DEECoMemberIn, etc.

Check, that we properly detect violation of inherent constraints on the components/interfaces/ensembles/...
It's enough to report it as an exception.

Once issue #5 is solved, we should integrate JDEECo with SDE. I would imagine it as a separate plugin, which does the integration. It should register commands to SDE console to install/uninstall OSGi bundles with components/ensembles. Further, it should provide a command to list running components/ensembles and possibly for the introspection on their knowledge.

The ideas are still quite fuzzy, so the described functionality is quite tentative.

The membership function should return an integer with the semantics like: -1 = no ensemble, 0 .. INT_MAX = ensemble possible where the particular value is the cost (0 is the best).

The runtime includes only a portion (expressed e.g. by limit) of the suitable members with the "big-enough" membership value.

Limit is a new field of the ensemble. It is a function of the coordinators knowledge. It returns how many instances of the particular ensembles may be established with the same component as the coordinator. (0 means unlimited).

This can be done by having limit attribute in the @DEECoEnsemble annotation.

Important thing to consider is to have some way of deciding when coordinator knowledge should be updated.
With current implementation (mapping takes one coordinator and one member), framework is able relying on the limit property to collect all the members which fulfils the membership predicate regarding a particular coordinator (up to the limit) and perform the mapping function for each pair (coordinator, member). This however, does not give us/developer a way to decide whether data in the coordinator should be replaced or not as we don't have this information when we execute single mapping.

Example:
Imagine we have coordinator which has collection of other nodes, which should be updated according to the ensemble definition. Considering current situation (one coordinator, one member) during mapping execution, we are able only to add a node to the coordinator's collection of nodes. We have no way for removing the node from that collection (unless we define a separate ensemble with opposite membership predicate - which might be tricky with fuzzy membership function).

The idea is to have mapping function accepting one coordinator and collection of members.
This way we are able to replace whole data at once, ensuring data freshness.

One possible downside is that we need to use collection of predefined homogeneous interfaces, but perhaps its not a big issue.

The source folder paths set in the project still point to the original location jdeeco-src, demo-src, ... They should point to the new locations src, demo, ...

Introduce @DEECoTrigger annotation (instead of DEECoTriggeredScheduling.

Test Driven Development should be applied

This task implements the knowledge provider using a hashmap and thus allows for simplified local execution. This should also involve deep-clone of the data retrieved from the hashmap, so as the semantics is the same as with the Apache River-based knowledge provider.

This can be possibly tested on the leader/follower demo.

Problems are typically:

• reflection methods for generic types
• UUID get random string (can be worked around by manually setting the component's ids)

The runtime should get instances instead of classes.
The parametric constructor may omit the id field - then it gets assigned automatically (as in the case of non-parametric constructor).

Implement a bulk mapping function alternative for the ensemble class.
It should take either new annotation type or extend existing one with an attribute saying that parameter is to be considered to be taken from all possible members.

Change SDE Tool functionality according to the outcome of the meeting.

At the moment, we atomically read all data before a process is executed and then atomically write everything back. However, there is not transaction when the process is executing, which means that the written data may not be consistent with the current state of the knowledge - i.e. when someone has updated the input data during the process computation. This seems OK for a big class of periodic processes.

However for triggered processes and a special class the periodic processes (e.g. for those which do extensive manipulations of hashmaps) it is necessary to keep the transaction for the whole duration of the process.

The type of the transaction should be distinguished by an annotation. We can also have some defaults -- such as weak semantics as a default for periodic processes and strict semantics as a default for triggered processes.

It should be possible to run JPF verification from the level of the SDE tool.

Split execution process into two steps:
*gather information regarding knowledge structure and
all necessary information in order to execute without Java Reflection usage.
*proper JDEECo runtime execution (without Java Reflection).

Introduce two separate launchers to Convoy demo - one using the local knowledge repository, the other using the Apache River-based repository. This requires also the update of the tutorial and the Ant build.xml file.

This allows executing a process when data get changed.

This also needs being able to register to change of an item in a hashmap - and being called for just that particular item.

Also, there should be a way of specifying the threading policy (at least single threaded execution, potentially multi-threaded execution).

This should create a new case study (at least in some basic shape) from the cloud environment. The goal is to try out different setting (more suitable for RELATE). Also, this will allow us to try out the fuzzy membership and limit features, as most likely we will have scenarios involving communication with "nearest" nodes or similar.

Rename the @...Mapper annotation to @EnsembleKnowledgeExchange or just @knowledgeexchange
Remove @deeco prefix
Remove @DEECoInitiali....

Other potential refactoring options - NEED DISCUSSION:

• in ensemble definition, @DEECoPeriodicScheduling should be put to the knowledge exchange method, instead to the whole class. This would make it more consistent with component definition. Also, the fact that membership is evaluated just before knowledge exchange is our technical trick, conceptually, knowledge exchange is the only part of the ensemble to be scheduled; membership "just happens".
• In the case of triggered scheduling, no annotation is put to the method, only to arguments that trigger scheduling. I would suggest adding a @TriggeredScheduling annotation to the method (to be consistent with periodic). Potentially, we could rename @DEECoTrigger anotation of arguments to @TriggerOnChange (so that we can add further types of triggers in the future), but this is not that important.

In the ensemble definition class, return value of the membership function should be replaced by float.
This will allow us to treat it also as a membership cost evaluator.

This task implements a scheduling of processes such that there is only one thread that sequentially schedules all processes. This may be helpful for JPF mode support (issue #14), but also for debugging as it removes multi-threading.

This enhancement should allow for easy integration with SDE (svn.pst.ifi.lmu.de/trac/sde), which is Eclipse RCP (and thus based on OSGi). It comprises:

• The runtime of JDEECo is packaged as an OSGi bundle.
• Components and ensembles are OSGi bundles.
• A component or ensemble is registered as an OSGi service.
• The activator of the JDEECo runtime registers a listener to component/ensemble service to keep of currently available components/ensembles.

As the result, the execution of JDEECo application will consist of starting OSGi framework with the JDEECo runtime bundle and with the bundles containing components/ensembles that should form the JDEECo application.

JDEECo and components/ensembles should be still usable without OSGi. This should be possible in the following way:

• JDEECo runtime is an OSGi bundle, which is just a jar with more complex manifest. That means that the jar may be used without OSGi - the extra fields in the manifest are then ignored and the Activator class is not used.
• A bundle containing a component/ensemble uses preferably a declarative specification of services (good example is the OSGi Declarative Services Example from the New Plug-in Wizard in Eclipse). Optionally the service may be registered in the Activator. In such a case, there is an extra class, which is not used outside OSGi.

When the convoy application is started, there are multiple calls on ISession.begin(). I found this out when I wanted to create another implementation of the KnowledgeRepository. Below is a log-output, which shows begins and ends of a session. The message "About to lock..." signifies the session begining. The message "Unlocked..." signifies the session end.

New local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
Unavailable entry: f0cb3102-e67c-49c8-b226-cbff5768621a.battery
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
Unavailable entry: f0cb3102-e67c-49c8-b226-cbff5768621a.path.currentPosition
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
Unavailable entry: f0cb3102-e67c-49c8-b226-cbff5768621a.path.remainingPath.3
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
Unavailable entry: f0cb3102-e67c-49c8-b226-cbff5768621a.path.remainingPath.2
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
Unavailable entry: f0cb3102-e67c-49c8-b226-cbff5768621a.path.remainingPath.1
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
Unavailable entry: f0cb3102-e67c-49c8-b226-cbff5768621a.path.remainingPath.0
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
Unavailable entry: f0cb3102-e67c-49c8-b226-cbff5768621a.path.remainingPath.7
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
Unavailable entry: f0cb3102-e67c-49c8-b226-cbff5768621a.path.remainingPath.6
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
Unavailable entry: f0cb3102-e67c-49c8-b226-cbff5768621a.path.remainingPath.5
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
Unavailable entry: f0cb3102-e67c-49c8-b226-cbff5768621a.path.remainingPath.4
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
Unavailable entry: f0cb3102-e67c-49c8-b226-cbff5768621a.path.remainingPath.8
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
Unavailable entry: f0cb3102-e67c-49c8-b226-cbff5768621a.path.remainingPath.keys
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
Unavailable entry: f0cb3102-e67c-49c8-b226-cbff5768621a.path.remainingPath.class
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
Unavailable entry: f0cb3102-e67c-49c8-b226-cbff5768621a.path.remainingPath.elementClass
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
Unavailable entry: f0cb3102-e67c-49c8-b226-cbff5768621a.convoyRobot
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
Unavailable entry: f0cb3102-e67c-49c8-b226-cbff5768621a.crossingRobots.keys
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
Unavailable entry: f0cb3102-e67c-49c8-b226-cbff5768621a.crossingRobots.class
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
Unavailable entry: f0cb3102-e67c-49c8-b226-cbff5768621a.crossingRobots.elementClass
About to lock local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a
Unavailable entry: f0cb3102-e67c-49c8-b226-cbff5768621a.id
Unlocked local session cz.cuni.mff.d3s.deeco.knowledge.local.LocalSession@3af4aa7a

The hashmaps and lists should be decomposed in a way similar to the example below:

class A {
  HashMap <String, String> h;
  List<String> l;
}

....h.put("apple","v1");
....h.put("orange","v2");
...l.add("xyz");
...l.add("wyz");

Keys in the tuple space are:

..."h"."apple" = "v1"
..."h"."orange" = "v2"
..."l"."0" = "xyz"
..."l"."1" = "wyz"

For the time being, it is OK to support only hashmaps which have the key of type String. Later on, if we would like to allow for any serializable object to be the key, we would have to change the definition of class Tuple to use Object for key (instead of String).

The mapping has the feature that a list in the tuple space may be seen also as a hashmap. Similarly an object in the tuple space with all fields of the same type may be seen as a hashmap. The hashmap can be also seen as an interface with fields which are a subset of the keys in the hashmap.

File JDEECoParser.jj (/jdeeco-core/src/cz/cuni/mff/d3s/deeco/path/grammar/JDEECoParser.jj) defines a path expressions. As such it should be named like PathParser or similar. (The "JDEECo" part of the name has little information value.)

Does Knowledge.id have any special semantics? If not, then it does not have to be part of knowledge.

The code should contain a short demo. We could use the wiki for describing the particular parts of the demo with explanation of the annotations and the general concepts.

Once this is done, we should make an internal review and then create a tag corresponding to this "release".

The code below is concerned. It looks very suspicious. Especially the last line, which is part of the loop, but actually does not depend on the loop variable.

        if (process.scheduling instanceof ProcessTriggeredSchedule)
            for (SchedulableProcessTrigger tsp : triggeredProcesses)
                if (tsp.sp == process) {
                    tsp.unregisterListener();
                    triggeredProcesses
                            .remove(new SchedulableProcessTrigger(process));
                } else
                    periodicProcesses.remove(process);

This task should implement a simple configuration mechanism for JDEECo runtime. At the moment it seems that we may have the following configuration dimensions (majority of the is not implemented yet):

• Knowledge provider (Apache River / Local hashmap) + addition configuration based on selected provider
• Threading model (Multi-threaded / All processes sequentially scheduled in a single thread)

(to be a folder sibling of jdeeco-core etc.).
It is a little bit confusing when the top-level Maven project is also in a parent directory of its child processes. The reason is that when opened via Eclipse without Maven, Eclipse sort of fails to open the childern projects.