pravahio / on-it Goto Github PK
View Code? Open in Web Editor NEWThis repo contains the todo's of the team. Feel free to add yours.
on-it's People
Contributors
Watchers
on-it's Issues
[modelling] Store air quality data and train a time series model
Task:
- Subscribe to air quality data in geospace
/in/delhi - Store the data in the disk.
- Train a time series model (maybe ARIMA) to predict value of PM 2.5 and PM 10.
This should be an inspiration to create a script to store data coming out of Pravah #6
[flight-data] A new standard to share flight data
Domain (eg transit, environment, water etc):
Aviation
Description:
Commercial airlines emit a lot a of valuable real-time data like ETD, any delays etc. This standard will capture such information.
Sample Data:
Initial Idea of the standard (in protobuf):
message FeedMessage {
FeedHeader header = 1;
repeated FeedAirport airports = 2;
}
message FeedAirport {
string name = 1;
repeated Terminal terminals = 2;
}
message Terminal {
string name = 1;
repeated Flight flights = 2;
}
message Flight {
AirportNode from = 1;
repeated AirportNode via = 2;
AirportNode to = 3;
}
message AirportNode {
string id = 1;
string name = 2;
string terminal = 3;
string gate = 4;
uint arrivalTime = 5;
uint leaveTime = 6;
Status status = 7;
}
message Status {
enum FlightStatus {
ARRIVED = 0;
DEPARTED = 1;
ON_TIME = 2;
DELAYED = 3;
CANCELLED = 4;
GATE_OPEN = 5;
GATE_CLOSED = 6;
FINAL_CALL = 7;
}
FlightStatus flightStatus = 1;
string message = 2;
}Who will be using this standard?:
Companies/startups into travel and logistics.
[air-quality-static] Create a static data protocol for Air Quality
Domain (eg transit, environment, water etc): Air Quality
Description: Will contain things that won't change for sometime like name, location (lat, lon) etc.
Initial Idea of the standard (in protobuf):
message Message {
string name = 1;
Location location = 2;
}
message Location {
double lat = 1;
double lon = 2;
}Who will be using this standard?: Will be used with Real-time air quality data.
[pune-transit-data] Onboard Pune Smart city live bus data
Description of the Data:
Pune Smart City publishes ITMS realtime and static data on IUDX platform. It has route information along with realtime location of all the buses.
List of Data Source:
Data is available via IUDX platform.
IUDX: Catalogue
Sample: iPython
Data Standard (link):
GTFS-Realtime
Sample Data:
[air-quality-data] Onboard CPCB data to pravah
CPCB (Central Pollution Control Board) has hundreds of Air Weather Stations deployed all across India.
Task: Pull data, check for redundancies and publish it to Pravah.
Data Access: CPCB has open API which can be used to query the data.
Data Standard: Air Quality
AQI all stations:
{
"stations": [
{
"cityName": "Vapi",
"cityID": "Vapi",
"stateID": "Gujarat",
"stationsInCity": [
{
"id": "site_5071",
"name": "Phase-1 GIDC, Vapi - GPCB",
"longitude": "72.918013",
"latitude": "20.362421",
"live": true,
"avg": 258,
"cityID": "Vapi",
"stateID": "Gujarat"
}
]
},
{
"cityName": "Vatva",
"cityID": "Vatva",
"stateID": "Gujarat",
"stationsInCity": [
{
"id": "site_5067",
"name": "Phase-4 GIDC, Vatva - GPCB",
"longitude": "72.643500",
"latitude": "22.969611",
"live": true,
"avg": 141,
"cityID": "Vatva",
"stateID": "Gujarat"
}
]
},
{
"cityName": "Ahmedabad",
"cityID": "Ahmedabad",
"stateID": "Gujarat",
"stationsInCity": [
{
"id": "site_308",
"name": "Maninagar, Ahmedabad - GPCB",
"longitude": "72.591912",
"latitude": "23.002657",
"live": true,
"avg": 85,
"cityID": "Ahmedabad",
"stateID": "Gujarat"
}
]
},
{
"cityName": "Ankleshwar",
"cityID": "Ankleshwar",
"stateID": "Gujarat",
"stationsInCity": [
{
"id": "site_5065",
"name": "GIDC, Ankleshwar - GPCB",
"longitude": "73.010555",
"latitude": "21.613267",
"live": true,
"avg": 227,
"cityID": "Ankleshwar",
"stateID": "Gujarat"
}
]
},
{
"cityName": "GandhiNagar",
"cityID": "GandhiNagar",
"stateID": "Gujarat",
"stationsInCity": [
{
"id": "site_5066",
"name": "Sector-10, Gandhinagar - GPCB",
"longitude": "72.654328",
"latitude": "23.221714",
"live": true,
"avg": 63,
"cityID": "GandhiNagar",
"stateID": "Gujarat"
}
]
},
{
"cityName": "Delhi",
"cityID": "Delhi",
"stateID": "Delhi",
"stationsInCity": [
{
"id": "site_5024",
"name": "Alipur, Delhi - DPCC",
"longitude": "77.153010",
"latitude": "28.815329",
"live": false,
"avg": "",
"cityID": "Delhi",
"stateID": "Delhi"
},
{
"id": "site_301",
"name": "Anand Vihar, Delhi - DPCC",
"longitude": "77.316032",
"latitude": "28.646835",
"live": true,
"avg": 86,
"cityID": "Delhi",
"stateID": "Delhi"
},
{
"id": "site_1420",
"name": "Ashok Vihar, Delhi - DPCC",
"longitude": "77.181665",
"latitude": "28.695381",
"live": true,
"avg": 67,
"cityID": "Delhi",
"stateID": "Delhi"
},
{
"id": "site_108",
"name": "Aya Nagar, Delhi - IMD",
"longitude": "77.1099364",
"latitude": "28.4706914",
"live": true,
"avg": 75,
"cityID": "Delhi",
"stateID": "Delhi"
},
{
"id": "site_1560",
"name": "Bawana, Delhi - DPCC",
"longitude": "77.051074",
"latitude": "28.776200",
"live": true,
"avg": 65,
"cityID": "Delhi",
"stateID": "Delhi"
}
{
"cityName": "Bengaluru",
"cityID": "Bengaluru",
"stateID": "Karnataka",
"stationsInCity": [
{
"id": "site_162",
"name": "BTM Layout, Bengaluru - CPCB",
"longitude": "77.5950804",
"latitude": "12.9135218",
"live": true,
"avg": 18,
"cityID": "Bengaluru",
"stateID": "Karnataka"
},
{
"id": "site_164",
"name": "BWSSB Kadabesanahalli, Bengaluru - CPCB",
"longitude": "77.6814488",
"latitude": "12.9352049",
"live": true,
"avg": 56,
"cityID": "Bengaluru",
"stateID": "Karnataka"
},
{
"id": "site_1553",
"name": "Bapuji Nagar, Bengaluru - KSPCB",
"longitude": "77.539784",
"latitude": "12.951913",
"live": true,
"avg": 78,
"cityID": "Bengaluru",
"stateID": "Karnataka"
}
]
},
{ ... }
]
}All parameters of a station (which could be fetched using station id):
{
"title": "Anand Vihar, Delhi - DPCC",
"nOfCom": 100,
"down": "false",
"downmessage": "",
"date": "Saturday, 24 Aug 2019 05:00 PM",
"temp": "",
"aqi": {
"param": "PM10",
"value": 235,
"remark": "Poor",
"color": "#ff9900"
},
"metrics": [
{
"name": "PM2.5",
"avg": 187,
"avgDesc": "Over the last 24 hours",
"min": 68,
"max": 338,
"pollutantName": "PM2.5"
},
{
"name": "PM10",
"avg": 235,
"avgDesc": "Over the last 24 hours",
"min": 58,
"max": 451,
"pollutantName": "PM10"
},
{
"name": "NO2",
"avg": 93,
"avgDesc": "Over the last 24 hours",
"min": 42,
"max": 289,
"pollutantName": "NO2"
},
{
"name": "NH3",
"avg": 12,
"avgDesc": "Over the last 24 hours",
"min": 2,
"max": 34,
"pollutantName": "NH3"
},
{
"name": "SO2",
"avg": 10,
"avgDesc": "Over the last 24 hours",
"min": 1,
"max": 17,
"pollutantName": "SO2"
},
{
"name": "CO",
"avg": 56,
"avgDesc": "Over the last 8 hours",
"min": 22,
"max": 134,
"pollutantName": "CO"
},
{
"name": "OZONE",
"avg": 15,
"avgDesc": "Over the last 8 hours",
"min": 8,
"max": 21,
"pollutantName": "OZONE"
}
]
}[railways-data-protocol] Create a new protocol to share train data
Domain (eg transit, environment, water etc): Railways
Description: Create a data sharing standard for Railways
Sample Data: https://enquiry.indianrail.gov.in/ntes/index.html
Initial Idea of the standard (in protobuf): Study if GTFS is good for this task.
Who will be using this standard?:
[solar-production-data] Onboarding solar production data from Growatt
Description of the Data:
Growatt is a data visualisation platform for solar power plants. Please refer to pravahio/mesh-solar-production#1 to learn more about the data.
List of Data Source:
https://server.growatt.com
They have an API: https://github.com/DiedB/SolarPanels/files/3115869/Growatt.Server.Api_EN.pdf
We could also use this client but getting a subset of the data from this library.
Data Standard (link):
Solar Power Standard
Sample Data:
N/A
[modelling] Estimating solar power output
Aim is to predict the production of the solar power given the data available on Pravah.
- Collect the data by subscribing to Pravah (example here)
- Create a python script to store this data. (maybe in files with timestamp)
- Think of a simple time series model to predict the power output.
[solar-production-data] Onboard data from dataglen portal
Dataglen is a data visualisation platform for solar power plants. Please refer to pravahio/mesh-solar-production#1 to learn more about the data.
Task: Fetch, clean, convert it into our standard and start publishing it to Pravah.
Get access to the API: Ping @upperwal on slack.
Edit:
API Endpoints:
- Station List: https://dataglen.com/api/v1/solar/plants/
- Station Summary: https://dataglen.com/api/v1/solar/plants/iiitd/summary (Replace "iiitd" with any station slug available in the stations list API)
- Data points for every inverter: https://dataglen.com/api/v1/solar/plants/dtu/live
- Details of each inverter: https://dataglen.com/api/v1/solar/plants/iiitd/live/?device_type=inverter&inverter=IIITD_INV_1_ACADEMIC_BLOCK
Data from the API looks like this. It is a summary of one of the power plants [there are close to 6-7 power plants on this portal].
{
"plant_name": "IPGCL IIIT Delhi",
"plant_slug": "iiitd",
"plant_logo": "http://dataglen.com/static/solarrms/img/transparent_logo/hfe.png",
"plant_location": "A-29, Pocket D, Okhla Phase II, Okhla Industrial Area",
"plant_capacity": "180.0 kWp",
"latitude": 28.67,
"longitude": 77.18,
"performance_ratio": 0.6,
"specific_yield": 2.29,
"cuf": 0.05,
"grid_unavailability": "0 %",
"equipment_unavailability": "0 %",
"unacknowledged_tickets": 1,
"open_tickets": 0,
"closed_tickets": 488,
"plant_generation_today": "219.54 kWh",
"plant_total_energy": "326.13 MWh",
"current_power": 35.525999999999996,
"irradiation": 0.43,
"network_up": "Yes",
"module_temperature": 23.9784946234,
"ambient_temperature": 225.904203324,
"windspeed": 0,
"dc_loss": "0.00 kWh",
"conversion_loss": "0.00 kWh",
"ac_loss": "0.00 kWh",
"status": "disconnected",
"updated_at": "2019-11-28T08:03:39.436000Z",
"inverter_details": {
"numbers": 6,
"make": "DELTA",
"model": "RPIM30A_121",
"capacity": "50.0 kW X 1,20.0 kW X 1,23.04 kW X 1,30.0 kW X 3"
},
"panel_details": {
"numbers": "564",
"make": "VIKRAM SOLAR",
"model": "Polycrystalline_Silicon",
"capacity": "320.0"
},
"max_power": 63.36999999999999,
"gateways_disconnected": 1,
"gateways_powered_off": 0,
"gateways_disconnected_list": [
"IIITD_SPORTS_BLOCK_GATEWAY_SOURCE"
],
"gateways_powered_off_list": [],
"gateways_disconnected_ticket": "https://dataglen.com/solar/plant/iiitd/ticket_view/983847/"
}Live Data from the same power station:
{
"inverters": [
{
"capacity": 23.04,
"name": "IIITD_INV_1_20_KW_HOSTEL_BLOCK",
"power": 4.987,
"last_timestamp": "2019-11-28T08:15:00Z",
"generation": 28.029999999998836,
"inside_temperature": "0.0 C",
"disconnected_ajbs": 0,
"total_yield": "36565.17 kWh",
"last_three_errors": [],
"orientation": "SOUTH",
"connected": "connected",
"key": "0SJCqlwYzY78h43",
"last_inverter_status": [
{
"status": "Grid Tie - Operation",
"timestamp": "2019-11-28T08:15:00Z"
}
],
"solar_group": "IIITD_HOSTEL_BLOCK",
"total_ajbs": 0
},
{
"capacity": 30,
"name": "IIITD_INV_1_ACADEMIC_BLOCK",
"power": 7.435,
"last_timestamp": "2019-11-28T08:15:01Z",
"generation": 43.890000000006694,
"inside_temperature": "0.0 C",
"disconnected_ajbs": 0,
"total_yield": "58950.23 kWh",
"last_three_errors": [],
"orientation": "SOUTH",
"connected": "connected",
"key": "xjVvoG3Qwtw5Eqc",
"last_inverter_status": [
{
"status": "Grid Tie - Operation",
"timestamp": "2019-11-28T08:15:01Z"
}
],
"solar_group": "IIITD_ACADEMIC_BLOCK",
"total_ajbs": 0
},
{
"capacity": 50,
"name": "IIITD_INV_1_LECTURER_BLOCK",
"power": 11.422,
"last_timestamp": "2019-11-28T08:15:00Z",
"generation": 63.33999999999651,
"inside_temperature": "0.0 C",
"disconnected_ajbs": 0,
"total_yield": "83094.79 kWh",
"last_three_errors": [],
"orientation": "SOUTH",
"connected": "connected",
"key": "f5qUJWwBvsJLVfU",
"last_inverter_status": [
{
"status": "Grid Tie - Operation",
"timestamp": "2019-11-28T08:15:00Z"
}
],
"solar_group": "IITD_LECTURER_BLOCK",
"total_ajbs": 0
},
{
"capacity": 30,
"name": "IIITD_INV_2_ACADEMIC_BLOCK",
"power": 7.312,
"last_timestamp": "2019-11-28T08:15:02Z",
"generation": 42.93000000000029,
"inside_temperature": "0.0 C",
"disconnected_ajbs": 0,
"total_yield": "59953.29 kWh",
"last_three_errors": [],
"orientation": "SOUTH",
"connected": "connected",
"key": "G5cPIzqNyAFMMBc",
"last_inverter_status": [
{
"status": "Grid Tie - Operation",
"timestamp": "2019-11-28T08:15:02Z"
}
],
"solar_group": "IIITD_ACADEMIC_BLOCK",
"total_ajbs": 0
},
{
"capacity": 30,
"name": "IIITD_INV__3__ACADEMIC_BLOCK",
"power": 7.63,
"last_timestamp": "2019-11-28T08:15:02Z",
"generation": 45.61000000000058,
"inside_temperature": "0.0 C",
"disconnected_ajbs": 0,
"total_yield": "62711.32 kWh",
"last_three_errors": [],
"orientation": "SOUTH",
"connected": "connected",
"key": "BXU1R5mEAMZ7kax",
"last_inverter_status": [
{
"status": "Grid Tie - Operation",
"timestamp": "2019-11-28T08:15:02Z"
}
],
"solar_group": "IIITD_ACADEMIC_BLOCK",
"total_ajbs": 0
},
{
"capacity": 19.2,
"name": "IIITD_Inverter_1",
"power": 0,
"last_timestamp": "2019-11-28T06:30:00Z",
"generation": 17.13000000000102,
"inside_temperature": "0.0 C",
"disconnected_ajbs": 0,
"total_yield": "24874.99 kWh",
"last_three_errors": [],
"orientation": "SOUTH",
"connected": "disconnected",
"key": "vACzRGjb97QVWxQ",
"last_inverter_status": [],
"solar_group": "Sports_Block",
"total_ajbs": 0
}
],
"solar_groups": [
"IIITD_ACADEMIC_BLOCK",
"IITD_LECTURER_BLOCK",
"IIITD_HOSTEL_BLOCK",
"Sports_Block"
],
"total_group_number": 4
}Data associated with each inverter (group of solar panels)
{
"dc_sld": [
{
"name": "AC Power",
"value": "7.43 kW"
},
{
"name": "AC Power B-Phase",
"value": "2.46 kW"
},
{
"name": "AC Power R-Phase",
"value": "2.47 kW"
},
{
"name": "AC Power Y-Phase",
"value": "2.5 kW"
},
{
"name": "AC Frequency B-Phase",
"value": "49.92 Hz"
},
{
"name": "AC Frequency R-Phase",
"value": "49.92 Hz"
},
{
"name": "AC Frequency Y-Phase",
"value": "49.92 Hz"
},
{
"name": "AC Voltage B-Phase",
"value": "244.0 V"
},
{
"name": "AC Voltage R-Phase",
"value": "245.3 V"
},
{
"name": "AC Voltage Y-Phase",
"value": "247.3 V"
},
{
"name": "Apparent Power B-Phase",
"value": "NA"
},
{
"name": "Apparent Power R-Phase",
"value": "NA"
},
{
"name": "Apparent Power Y-Phase",
"value": "NA"
},
{
"name": "AC Current B-Phase",
"value": "10.16 A"
},
{
"name": "AC Current R-Phase",
"value": "10.12 A"
},
{
"name": "AC Current Y-Phase",
"value": "10.26 A"
},
{
"name": "Todays Generation",
"value": "43.9 kWh"
},
{
"name": "DC Power",
"value": "7.52 kW"
},
{
"name": "Heat Sink Temperature",
"value": "37.0 C"
},
{
"name": "PV1 DC Current",
"value": "7.0 A"
},
{
"name": "PV1 DC Power",
"value": "4.54 kW"
},
{
"name": "PV1 DC Voltage",
"value": "647.9 V"
},
{
"name": "PV2 DC Current",
"value": "4.61 A"
},
{
"name": "PV2 DC Power",
"value": "2.98 kW"
},
{
"name": "PV2 DC Voltage",
"value": "646.5 V"
},
{
"name": "Inverter Operational Status",
"value": "2.0 NA"
},
{
"name": "Total Operational Runime",
"value": "23335.0 H"
},
{
"name": "Total Generation",
"value": "58950 kWh"
}
],
"last_timestamp": "2019-11-28T08:15:01Z"
}[visualisation] Dashboard for Delhi's solar power data
Data Standard: mesh-solar
Visualisations we can try: (plant level and inverter level)
- Time-series plot of power generation (kWh) vs time (t)
- Time-series of solar radiation (kW/m^3) vs time (t)
[data-collector] Storing data coming out of pravah
As of now Pravah is a stateless system meaning no data is stored within it. We need a way to store the data which can be used as historical data later to train ML models or visualisations.
How can it be done?
- [Simple way]: Most of the data is in encoded proto format. We can directly store the data in files in S3. This would mean storing compressed data but we will need to decode it later and store it in a database.
- [Not so simple]: Store it in Cassandra or Couchbase after decoding.
Which database?
AIM:
- Due to high velocity and volume of data coming in, there will be huge number of write operations.
- Read operations will be way less than write and most of the read will be on aggregated data (1 month data etc). A single read will return high volume of data (in GBs).
Couchbase gives pretty good write performance. Cassandra is another good big data database. Experimentation is needed for our use case.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.