met4fof / agentmet4fof Goto Github PK
View Code? Open in Web Editor NEWMetrological Agent-based system (MET4FOF project)
License: Other
Metrological Agent-based system (MET4FOF project)
License: Other
We tried that in #50 and found it very promising. The next extractions might be:
ml_uncertainty
demo
ML
ML_EXPERIMENTS
SENSORS
ZEMA_EMC
After that we can clean up the docs and requirements.txt and win
Incorporate the methods for accounting uncertainty developed by Ehlimana into the agents.
We might implement a class for or at least deeply integrate the D-SI into the message passing of the agents. This should be considered closely related to #19 .
Further materials about D-SI will be added here shortly.
A full-text search for the filename of assets/stylesheet.css in the project gave no results. After deleting it, all tests and examples still succeeded. Can that file maybe be deleted?
At the moment the two links targets on the docs startpage
agentMET4FOF agents
agentMET4FOF streams
are empty. We have to check the according rst-files and repair this. The autodocs were their earlier.
A separate tutorial on this will be helpful to illustrate how the data memory is used.
It would be nice, if the function create_monitor_data
(used here and defined here) can be extended to allow for user-provided x values.
This would help with plotting infinite streams of time-series.
The new function could look something like this: (old behaviour is obtainable by explicitly setting xname
or yname
to None
)
def create_monitor_data(data, xname="time", yname="data", label="Monitor Data"):
if xname and yname:
x = data[xname]
y = data[yname]
else:
x = np.arange(len(data))
y = data
trace = go.Scatter(x=x, y=y,mode="lines", name=label)
return trace
When I execute python
run_dashboard.py
in the console I get the following output periodically:
Unable to connect to existing NameServer...
* Serving Flask app "dashboard.Dashboard" (lazy loading)
* Environment: production
WARNING: Do not use the development server in a production environment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://127.0.0.1:8050/ (Press CTRL+C to quit)
127.0.0.1 - - [30/Jul/2019 17:24:32] "GET /_favicon.ico HTTP/1.1" 200 -
127.0.0.1 - - [30/Jul/2019 17:24:33] "GET /_dash-layout HTTP/1.1" 200 -
127.0.0.1 - - [30/Jul/2019 17:24:33] "GET /_dash-dependencies HTTP/1.1" 200 -
127.0.0.1 - - [30/Jul/2019 17:24:33] "POST /_dash-update-component HTTP/1.1" 200 -
127.0.0.1 - - [30/Jul/2019 17:24:33] "POST /_dash-update-component HTTP/1.1" 204 -
127.0.0.1 - - [30/Jul/2019 17:24:33] "POST /_dash-update-component HTTP/1.1" 204 -
127.0.0.1 - - [30/Jul/2019 17:24:33] "POST /_dash-update-component HTTP/1.1" 204 -
[2019-07-30 17:24:33,106] ERROR in app: Exception on /_dash-update-component [POST]
Traceback (most recent call last):
File "PATH\TO\MY\ENV\lib\site-packages\flask\app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "PATH\TO\MY\ENV\lib\site-packages\flask\app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "PATH\TO\MY\ENV\lib\site-packages\flask\app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "PATH\TO\MY\ENV\lib\site-packages\flask\_compat.py", line 35, in reraise
raise value
File "PATH\TO\MY\ENV\lib\site-packages\flask\app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "PATH\TO\MY\ENV\lib\site-packages\flask\app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "PATH\TO\MY\ENV\lib\site-packages\dash\dash.py", line 1147, in dispatch
response.set_data(self.callback_map[output]['callback'](*args))
File "PATH\TO\MY\ENV\lib\site-packages\dash\dash.py", line 1033, in add_context
output_value = func(*args, **kwargs)
File "PATH\TO\MY\REPOS\Met4FoF_Code\agentMet4FoF\dashboard\Dashboard.py", line 423, in plot_monitor_graphs
if agentNetwork._get_mode() != "Running" and n_interval > 0:
File "PATH\TO\MY\REPOS\Met4FoF_Code\agentMet4FoF\agents\AgentMET4FOF.py", line 653, in _get_mode
return self._get_controller().get_attr('current_state')
File "PATH\TO\MY\REPOS\Met4FoF_Code\agentMet4FoF\agents\AgentMET4FOF.py", line 789, in _get_controller
self._controller = self.ns.proxy('AgentController')
AttributeError: 'int' object has no attribute 'proxy'
127.0.0.1 - - [30/Jul/2019 17:24:33] "POST /_dash-update-component HTTP/1.1" 500 -
Do we still need this script? If so we should update it.
@mgrub mentioned, that it took him some time to see the differences but thinks, the structure is useful and should be kept.
We should gather with @mgrub, @BeSeeTek, @bangxiangyong and @BjoernLudwigPTB for a web conference with whiteboards in the background to make up our minds about how to structure #23 .
At hte moment we have to write restructured text to work on the docs. Since we are so much more used to markdown, it would be nice to introduce recommonmark into our pipeline.
This will automatically resolve #32 by the proposed overwriting mechanism.
ZeMA (@TanjaDorst) should use agentMET4FOF together with their machine learning methods with uncertainties and we should support their efforts (e.g. refactor interfaces if this could encourage others to work with agentMET4FOF or integrate required functions)
We should provide a PyPI.org package and upload a new release whenever something worth mentioning changes in agentMET4FOF. At the same time we should consider Python Semantic Release for automatic semantic versioning.
The documentation of components for ML experiments is not complete, and the structure is quite scattered (agents.py , develop.* , etc).
The clarity can be improved by having more explanations and should probably be self-contained as a separate component?
The following classes need class docstrings:
SemVer appears to be very beneficial and time saving, if it is realized with python-semantic-release. It definitely enhances our release process by providing changelogs.
After setting up a clean python-venv and then pip install agentMET4FOF
, execution of import agentMET4FOF.agents
yields:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "%userprofile%\python_venv\agentmet4fof\lib\site-packages\agentMET4FOF\agents.py", line 24, in <module>
from agentMET4FOF.dashboard.Dashboard import AgentDashboard as AgentDashboard
File "%userprofile%\python_venv\agentmet4fof\lib\site-packages\agentMET4FOF\dashboard\Dashboard.py", line 6, in <module>
from agentMET4FOF.dashboard.Dashboard_ml_exp import get_ml_exp_layout, prepare_ml_exp_callbacks, get_experiments_list
File "%userprofile%\python_venv\agentmet4fof\lib\site-packages\agentMET4FOF\dashboard\Dashboard_ml_exp.py", line 15, in <module>
from agentMET4FOF.develop.ML_Experiment import load_experiment
ModuleNotFoundError: No module named 'agentMET4FOF.develop'
Details about the system:
OS: Windows 10 Pro 64bit
Python: 3.7.7 and 3.8.1
agentMET4FOF: 0.1.4
Current Workaround:
Use agentMET4FOF from a clone of the github-repo:
import sys;sys.path.append("<directory_of_your_choice>")
The send_plot funciton converts matplotlib figures into plotly or image base64 string, there is an option to directly send plotly figures as well, but we should test and implement it further.
Will need to investigate whether which option is better and feasible:
There is some stuff missing and some things we don't need anymore.
This leads to a more detailed description of #22.
We want to extract the more specific project related examples into separate repositories which are related to agentMET4FOF by the submodule mechanism.
This would allow for the extracted examples to work without any further tweaking of the execution environment.
We rather want to use the methods Haris already prepared in his subfolder methods by transferring our methods in zema_feature_extract.py and then reusing them by introducing his code as a submodule in ZEMA_EMC.
from agentMET4FOF.agents import AgentMET4FOF, AgentNetwork, MonitorAgent
from agentMET4FOF.streams import SineGenerator
import torch.nn as nn
import torch.nn.functional as F
class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.conv1 = nn.Conv2d(3, 6, 5)
self.pool = nn.MaxPool2d(2, 2)
self.conv2 = nn.Conv2d(6, 16, 5)
self.fc1 = nn.Linear(16 * 5 * 5, 120)
self.fc2 = nn.Linear(120, 84)
self.fc3 = nn.Linear(84, 10)
def forward(self, x):
x = self.pool(F.relu(self.conv1(x)))
x = self.pool(F.relu(self.conv2(x)))
x = x.view(-1, 16 * 5 * 5)
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
x = self.fc3(x)
return x
class TrainerAgent(AgentMET4FOF):
def init_parameters(self,minus_param=0.5,plus_param=0.5):
self.net = Net()
self.first_time = True
def on_received_message(self, message):
if self.first_time:
self.send_output(self.net)
self.first_time = False
class SineGeneratorAgent(AgentMET4FOF):
def init_parameters(self):
self.stream = SineGenerator()
def agent_loop(self):
if self.current_state == "Running":
sine_data = self.stream.next_sample() #dictionary
self.send_output(sine_data['x'])
def main():
# start agent network server
agentNetwork = AgentNetwork()
# init agents
gen_agent = agentNetwork.add_agent(agentType=SineGeneratorAgent)
train_agent = agentNetwork.add_agent(agentType=TrainerAgent)
predict_agent = agentNetwork.add_agent(agentType=TrainerAgent)
monitor_agent = agentNetwork.add_agent(agentType=MonitorAgent)
# connect agents : We can connect multiple agents to any particular agent
agentNetwork.bind_agents(gen_agent, train_agent)
agentNetwork.bind_agents(gen_agent, predict_agent)
# connect
agentNetwork.bind_agents(gen_agent, monitor_agent)
agentNetwork.bind_agents(train_agent, predict_agent)
agentNetwork.bind_agents(predict_agent, monitor_agent)
# set all agents states to "Running"
agentNetwork.set_running_state()
# allow for shutting down the network after execution
return agentNetwork
if __name__ == '__main__':
main()
Error message as below:
Exception in thread oneway-call:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/Pyro4/core.py", line 1891, in run
super(_OnewayCallThread, self).run()
File "/usr/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/osbrain/agent.py", line 1680, in run
self._loop()
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/osbrain/agent.py", line 1208, in _loop
if self._iterate():
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/osbrain/agent.py", line 1245, in _iterate
self._process_events(events)
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/osbrain/agent.py", line 1282, in _process_events
self._process_single_event(socket)
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/osbrain/agent.py", line 1296, in _process_single_event
self._process_sub_event(socket, address, data)
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/osbrain/agent.py", line 1452, in _process_sub_event
handler(self, message)
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/agentMET4FOF-develop/agentMET4FOF/agents.py", line 334, in handle_process_data
proc_msg = self.on_received_message(message)
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/agentMET4FOF-develop/agentMET4FOF_tutorials/tutorial_pytorch.py", line 35, in on_received_message
self.send_output(self.net)
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/agentMET4FOF-develop/agentMET4FOF/agents.py", line 263, in send_output
self.send(self.PubAddr, packed_data, topic='data')
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/osbrain/agent.py", line 1493, in send
topic=topic)
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/osbrain/agent.py", line 1501, in _send_address
serializer=address.serializer)
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/osbrain/agent.py", line 85, in serialize_message
return _serialize_callsserializer
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/osbrain/agent.py", line 50, in
'pickle': lambda message: pickle.dumps(message, -1),
_pickle.PicklingError: Can't pickle <class 'main.Net'>: it's not the same object as main.Net
If it is possible to easily extract the #32 related changes then we open a self-contained Pull Request to make it easier for Max to review, if his needs are met.
Benedikt left me with a sensor board, to be connected to the agent framework.
In #28 we wrote down some links and now have to find out, how to make use of D-SI.
i have buffered data and send to agent, but i want to after each new sending, perevious data romves , i mean remove last buffer and replace with new data.
To make working with the agents easier, we should be able to clarify on how input data should be structured to best fit the agents design.
Regarding streaming data, we should as well keep in mind to adapt the generators of PyDynamic to this design and view this as very closely connected.
That instantly leads to an extended development of the signals themselves probably, which benefits PyDynamic as well, maybe in extending the previously started class Signal. Maybe we can even base the development on that class.
Originally posted by @BjoernLudwigPTB in #18 (comment)
... which includes adapting the tutorial about Haris' work as well. This is the start of #27.
The tutorials are already well documented, but we could add more comments in the code. This could help us to clarify the process and make it even easier to get started.
Considerations:
This partly resolves #36 .
Since we want to extract the examples anyway (see #50 for a start) we should remove the individual requirements from the global requirements.txt since there are errors arising on some platforms resulting from different or missing C++ compiler versions.
As described in #21
We should check if by now we could drop the constraint on matplotlib==3.1.0
not to loose compatibility with newer Python versions.
@bangxiangyong could you provide a test, which checks for the reason of the constraint so we can test if an upgrade would harm the agents?
Had a meeting with Majid, and realised that clearing the garbage is a better way than simply setting the memory to an empty dictionary.
reference:
https://github.com/bangxiangyong/agentMET4FOF/blob/develop/examples/SENSORS/buffer_sensor.py
There seems to be a problem with running the agents on Windows with Python 3.8.1 and multiprocess==0.70.9
. A colleague gets the following error on execution of tutorial 1:
Starting NameServer...
Broadcast server running on 0.0.0.0:9091
NS running on 127.0.0.1:3333 (127.0.0.1)
URI = PYRO:[email protected]:3333
INFO [2020-04-15 10:11:08.404372] (AgentController): INITIALIZED
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'multiprocess'
INFO [2020-04-15 10:11:30.828139] (SineGeneratorAgent_1): INITIALIZED
INFO [2020-04-15 10:11:39.270462] (MonitorAgent_1): INITIALIZED
[2020-04-15 10:11:39.358227] (SineGeneratorAgent_1): Connected output module: MonitorAgent_1
SET STATE: Running
[2020-04-15 10:11:39.868856] (SineGeneratorAgent_1): Pack time: 0.0
[2020-04-15 10:11:39.869853] (SineGeneratorAgent_1): Sending: [0.]
[2020-04-15 10:11:39.869853] (MonitorAgent_1): Received: {'from': 'SineGeneratorAgent_1', 'data': array([0.]), 'senderType': 'SineGeneratorAgent', 'channel': 'default'}
[2020-04-15 10:11:39.869853] (MonitorAgent_1): Tproc: 0.0
On the same machine it is working with Python 3.7.7 and multiprocess==0.70.7
. We haven't checked the multiprocess issues yet and he will continue with the working 3.7.7 configuration, so for now there is no further investigation.
Been having this in mind of viewing data analytics/ML as a workflow (which current implementation for ZEMA EMC with Haris Lulic is kind-of adopting already), that is a sequence of functions applied one after the other.
Example:
group_of_agents = make_pipeline(FFT(), PCA(pca_parameters), LDA(lda_parameters))
fit
and transform
similar to how sklearn structures its classesgroup_of_agents_1 = make_pipeline(FFT(), PCA(pca_parameters), LDA())
group_of_agents_2 = make_pipeline(FFT(), PCA(pca_parameters), ANN())
group_of_agents_1 = make_pipeline(AddNoise(), FFT(), LDA())
group_of_agents_2 = make_pipeline(FFT(), LDA())
Because of the subprocesses we are spawning, in general debugging does not work as usual. We should think about, if it is possible to make that available. It would strip a lot of print statements from the codes we are seeing being written using the agents and make debugging a lot easier.
class Trainer(AgentMET4FOF):
def init_parameters(self):
r = random.randrange(10, 20)
a = {'x': [1, 2, 3], 'y': r}
# print(a['x'])
a = pd.DataFrame(a['x'])
#print(a.shape)
a = np.array(a)
#print(a.shape)
X_train = a.reshape(a.shape[0], 1, a.shape[1])
print(np.shape(X_train))
inputs = Input(shape=(X_train.shape[1], X_train.shape[2]))
L1 = LSTM(16, activation='relu', return_sequences=True,
kernel_regularizer=regularizers.l2(0.00))(inputs)
output = TimeDistributed(Dense(X_train.shape[2]))(L1)
model = Model(inputs=inputs, outputs=output)
model.compile(optimizer='adam', loss='mse')
model.fit(X_train, X_train, epochs=5, batch_size=32)
model.predict(X_train)
#56 AttributeError: '_thread._local' object has no attribute 'value'
Max has got some implementation ready using a Python's deque und we got some in the examples. We should merge these into one residing in the master.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.