Smart City Data Platform Graz
fiware-austria / scdp Goto Github PK
View Code? Open in Web Editor NEWSmart City Data Platform Graz
Smart City Data Platform Graz
Currently Wirecloud is accessed using http, but it should be https.
Comet apparently has a strict limit on the number of values, on the other side browsers can only handle a certain maximum number of values.
We need an operator that reduces the number of entries in a list by replacing chunks of data with their average.
Idea: Config setting with maximum number of values. If the data collection exceeds this value, its reduces to about this given size
Update/Remark: Best implemented as additional feature within Comet. This would limit the data volume exchanged between Comet and Wirecloud
For every incoming message there needs to be an entity and an IDAS device
Automatically import the daily Feedback-Now report into the SDPG
If there are no inbound data package of specific period of time an alert should be raised
Install Apache Spark and connect it to the installed version of Hadoop
Setup and enable FIWARE Security as part of the IoT Platform
Select appropriate tooling (eg. Jmeter)
Write scripts for automatic onboarding and removal of sensors
Define testplans
Perform a series of performance tests on different container layouts in order to find the optimal layout for FIWARE components
Install Jupyter with Spark Integration in order to analyse the data stored in the SDPG
Within FIWARE, Comet is used to access historical data. However, Comet only works with MongoDB. Thus, develop a component exposing the same query interface as Comet but using data from Hadoop
Create new docker image
Test new version
Migrate data
Provide a mean to automatically convert the daily Excel report from FeedbackNow into a CSV file that allows for automatic import into the SCDP
Internal, critical services like Orion, Comet, Cygnus, IDAS, ...) should only be accessible via PEP. Thus, make sure that these services are only reachable within nodes on Kubernetes (Cluster-IP)
In order to deploy the FIWARE container network (docker-compose), containers and networks need to re-configered, in order to run a kubernetes infrastructure
yaml files
Define a set of rules for all the data stored in the SDPG (FeedbackNow, Traffic, Air Quality) such that data can stored in an efficient way over a long period of time
The SOAP Interface provided by the traffic department needs to be queried in regular intervals and the the resulting data forwarded to the SCDP.
This could possibly also be done using NodeRED
Cugnus is currently using the Hadoop Namenode's REST Service (WebHDFS), which is apparently way to slow for high loads. This should be refactored, using a "native" Hadoop interface.
Check whether NodeRED is a suitable platform for full-custom code needed throughout this project
Research and evaluate Kubernetes for use as a basic infrastructure to run and manage our docker containers for hadoop and FIWARE
Create a dashboard to visualize feedback now data in the most appropriate way
The NameNode is a Single Point of Failure. In order to provide high availability a standby namenode is setup.
Steps that have to be performed:
Since the backup-system only supports Mongo, we need a decent way to get this data:
Visualize all Air Quality Data produced by the real-life sensor network
Different deployment layouts shall be developed and performance tested in order to find one with an optimal throughput
In order to prevent accential deletion of entities, devices, ..., bring up a confirm dialog.
Data collected by the proprietary IoT System connected to the Air-Quality sensor network is pushed in real-time to our FIWARE IoT-Platform. This REST-Interface will receive this data, translate its payload into Ultralight and send it to IDAS.
Sensor data , Orion and Cygnus should use a consistent and correct time zone
Affected components: POI Beautifier
Description of the initial Hadoop setup
Do some thorough research to identify tools that could possibly be used for enforcing data policies
Create a suitable visualisation of traffic data
IDAS requires API-Key and device ID to be passed as query parameters. This disables XACML rules. Thus, Node-RED shall be used to rewrite URL-Parameters to Query-Parameters.
PEP will then speak to Node-RED rather than IDAS
Find a way to define dependencies between Kubernetes service, so that they start-up in a defined sequence.
For automatically testing the availability of our NodeRED end-points there should be special request that does not produce any data but returns 200 if everything is OK. This request should be sent by systems like Nagios
Describe possible, performant ways how to access (read/write) HDFS from outside the cluster
A major part of the SDPG will be Hadoop and its relevant Plugins. All required nodes should be rolled out on the local cloud using kubernetes
Setup some sample notebooks demonstrating the use of machine learning on SDPG data
This will create the necessary collections in order to support aggregate queries.
For entity attributes in ORION a datatype can/needs to be specified ("Int", "Float", ...).
However, ORION queries always return strings. So, what are these types good for?
Also check COMET aggregate queries
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.