envirodiy / modularsensors Goto Github PK

Let's renew our push to test and finalize our base library for YosemitechModbus sensors (https://github.com/EnviroDIY/YosemitechModbus), so that we can develop Yosemitech sensor sub-class files within this ModularSensors library. I just created a new Yosemitech branch for this work: https://github.com/EnviroDIY/ModularSensors/tree/yosemitech

@fisherba is ordering a pile of Yosemitech sensors to hopefully test in the field in October.

@SRGDamia1 said that by early July, she had everything working for her in YosemitechModbus.
Unfortunately, when I tested the code on my newer hardware versions of the sensors with brushes, I had issues. See EnviroDIY/YosemitechModbus#3

@SRGDamia1, are you inspired by http://odm2.github.io/ODM2PythonAPI?
It would be awesome to have this, to get others to start contributing.

I bet @lsetiawan would be happy to provide guidance and help.

No idea why. If I turn on the debug.. it works. I'm stumped.

@SRGDamia1, I just tested the Master branch, and its working great!
I think it's worth creating an official tagged release for this. Also, I'm thinking that we remove the "beta" tag. We're ready to make this public as a stable release.

Right now you can average readings from an SDI-12 sensor, but none of the others.

Hi @SRGDamia1,

We have been running the multisensor_print sketch to test 5TM sensors. They are reading permittivity and temperature, but the calculation for volumetric water content is reporting as -9999.

Example results.....
in air:

Updated all sensors at 2000-05-27T11:57:43-06:00
EnviroDIY Mayfly at EnviroDIY Mayfly reports batteryVoltage is 4.852 Volt
EnviroDIY Mayfly at EnviroDIY Mayfly reports Free SRAM is 14401 Bit
MaximDS3231 at I2C_0x68 reports temperatureRTC is 22.75 degreeCelsius
Decagon5TM at SDI12-2_Pin7 reports permittivity is 1.2900 faradPerMeter
Decagon5TM at SDI12-2_Pin7 reports temperature is 20.7 degreeCelsius
Decagon5TM at SDI12-2_Pin7 reports volumetricWaterContent is -9999.00 percent
In CSV Format:  4.852,14401,22.75,1.2900,20.7,-9999.00

(My calculated VWC with this permittivity is -0.016.)

in water:

Updated all sensors at 2000-05-27T12:16:30-06:00
EnviroDIY Mayfly at EnviroDIY Mayfly reports batteryVoltage is 4.852 Volt
EnviroDIY Mayfly at EnviroDIY Mayfly reports Free SRAM is 14401 Bit
MaximDS3231 at I2C_0x68 reports temperatureRTC is 22.75 degreeCelsius
Decagon5TM at SDI12-2_Pin7 reports permittivity is 53.1000 faradPerMeter
Decagon5TM at SDI12-2_Pin7 reports temperature is 21.3 degreeCelsius
Decagon5TM at SDI12-2_Pin7 reports volumetricWaterContent is -9999.00 percent
In CSV Format:  4.852,14401,22.75,53.1000,21.3,-9999.00

(My calculated VWC with this permittivity is 0.59.)

When I calculate VWC, the in-water one should return a non-error result. I'm wondering if the calculation turns VWC from a float? I tried a version commenting out the percent conversion in Decagon5TM.h and had the same result.

I keep getting the error below when I try to compile "simple_logging.ino":

Arduino: 1.8.1 (Mac OS X), Board: "EnviroDIY Mayfly 1284p"

/~/Arduino/ModularSensors/examples/simple_logging/simple_logging.ino:46:28: fatal error: ProcessorStats.h: No such file or directory
#include <ProcessorStats.h>
^
compilation terminated.
exit status 1
Error compiling for board EnviroDIY Mayfly 1284p.

This report would have more information with
"Show verbose output during compilation"
option enabled in File -> Preferences.

The processor stats file is located in:
/~/Arduino/ModularSensors/src
...which is where cloning the repo placed it. Should it be located in a different directory or structure?

Trigger - power vs filtering

How to use >1

If turb < low range, use low, else use high.

Would enable support for Arduino Zero, Adafruit Feather, Sodaq One, Sodaq Autonomo, etc.

Need to change timing, sleeping, and RTC functions to support.

Would use RTCZero library for clock and sleep: https://github.com/arduino-libraries/RTCZero
SAMD timer library for timing: https://github.com/avandalen/avdweb_SAMDtimer

Will not work at this time for SDI-12 devices b/c the SDI-12 library is avr only.

To better reflect the direction of the measurement (e.g. down to water) and that the values gets larger as water level goes down, which is the inverse of Depth.

So we can get to dreamhost and save bandwidth

When connected to WiFi or cell networks, sync clock.

Right now, in order to call more than one variable from the same update function, the variables are set as static. Unfortunately, this also means that you cannot use more than one sensor of each type and get proper results.

I'm setting up a timer, but not using it at all. I'm only using the clock interrupts and a mod function for timing. So far I've left it in because I may want to use it later to sync the clock or start the modem at the same time as updating the sensors. I'm dubious if its full utility within the library, though, because of the issue referenced in this thread: http://forum.arduino.cc/index.php?topic=196446.0. Essentially, only static functions can be called by the timer.

It crashes the program when in debug mode.

My use case is when YosemiTech Turbidity exceeds 50 NTU.

My code is set to loggingInterval = 5, but my timestamps are 2-4 minutes apart (they are not even always the same). What is going on there? @SRGDamia1, any ideas?

The sketch is here: https://github.com/LimnoTech/EnviroDIY_deployments/tree/master/deploy-RPBCWD/RPB-3

Here's the output (with dummy tokens).

Now running logging_to_EnviroDIY.ino on Logger Mayfly_170010
Logger timezone is UTC-5
RTC timezone is UTC
Initializing logger Mayfly_170010 to record at 5.00 minute intervals ...    ... Success!
Current RTC time is: 2000-01-27T20:12:34-05:00
Attempting to synchronize RTC with NIST
Clock synced!
Successfully connected to SD Card with card/slave select on pin 12
Data will be saved as Mayfly_170010_2018-02-07.csv...   ... File created!
Logger setup finished!
------------------------------------------

Push button NOW to enter debug mode.
------------------------------------------

End of debug mode.
------------------------------------------

 \/---- Post Request to EnviroDIY ----\/ 
POST /api/data-stream/ HTTP/1.1
Host: data.envirodiy.org
TOKEN: 12345678-abcd-1234-efgh-1234567890ab
Content-Length: 475
Content-Type: application/json

{"sampling_feature": "12345678-abcd-1234-efgh-1234567890ab", "timestamp": "2018-02-07T15:02:00-05:00", "9e9f2c7b-0ab3-4dfe-b933-ef0a1920bb63": 4.609, "33218554-9cf6-48f3-b8d0-4d032dcb8578": 10712, "c34b0278-3267-4969-929e-62c45a1d8ff1": 23.75, "47a3117d-9c39-43c9-87c2-a05aa573e1b0": 7389, "a58c3985-45cd-407c-92d6-18b89dde8c4d": 0.0, "908ad4f8-b6db-4b0c-85c6-b4f14c3229bf": 21.22, "f05e563f-456e-459c-8dcb-aa87d06489c5": 0.00, "89baa175-b592-42b5-8322-de2df59e4bfd": 20.88 }

 -- Response Code -- 
201

 \/---- Line Saved to SD Card ----\/ 
2018-02-07 15:02:00,4.609,10712,23.75,7389,0.0,21.22,0.00,20.88
------------------------------------------

------------------------------------------

 \/---- Post Request to EnviroDIY ----\/ 
POST /api/data-stream/ HTTP/1.1
Host: data.envirodiy.org
TOKEN: 12345678-abcd-1234-efgh-1234567890ab
Content-Length: 474
Content-Type: application/json

{"sampling_feature": "12345678-abcd-1234-efgh-1234567890ab", "timestamp": "2018-02-07T15:04:00-05:00", "9e9f2c7b-0ab3-4dfe-b933-ef0a1920bb63": 4.624, "33218554-9cf6-48f3-b8d0-4d032dcb8578": 10712, "c34b0278-3267-4969-929e-62c45a1d8ff1": 25.00, "47a3117d-9c39-43c9-87c2-a05aa573e1b0": 792, "a58c3985-45cd-407c-92d6-18b89dde8c4d": 0.0, "908ad4f8-b6db-4b0c-85c6-b4f14c3229bf": 21.25, "f05e563f-456e-459c-8dcb-aa87d06489c5": 0.00, "89baa175-b592-42b5-8322-de2df59e4bfd": 21.25 }

 -- Response Code -- 
2019

 \/---- Line Saved to SD Card ----\/ 
2018-02-07 15:04:00,4.624,10712,25.00,792,0.0,21.25,0.00,21.25
------------------------------------------

------------------------------------------

 \/---- Post Request to EnviroDIY ----\/ 
POST /api/data-stream/ HTTP/1.1
Host: data.envirodiy.org
TOKEN: 12345678-abcd-1234-efgh-1234567890ab
Content-Length: 475
Content-Type: application/json

{"sampling_feature": "12345678-abcd-1234-efgh-1234567890ab", "timestamp": "2018-02-07T15:05:00-05:00", "9e9f2c7b-0ab3-4dfe-b933-ef0a1920bb63": 4.624, "33218554-9cf6-48f3-b8d0-4d032dcb8578": 10712, "c34b0278-3267-4969-929e-62c45a1d8ff1": 25.25, "47a3117d-9c39-43c9-87c2-a05aa573e1b0": 792, "a58c3985-45cd-407c-92d6-18b89dde8c4d": 0.0, "908ad4f8-b6db-4b0c-85c6-b4f14c3229bf": 21.26, "f05e563f-456e-459c-8dcb-aa87d06489c5": 50.77, "89baa175-b592-42b5-8322-de2df59e4bfd": 21.42 }

 -- Response Code -- 
2019

 \/---- Line Saved to SD Card ----\/ 
2018-02-07 15:05:00,4.624,10712,25.25,792,0.0,21.26,50.77,21.42
------------------------------------------

------------------------------------------

 \/---- Post Request to EnviroDIY ----\/ 
POST /api/data-stream/ HTTP/1.1
Host: data.envirodiy.org
TOKEN: 12345678-abcd-1234-efgh-1234567890ab
Content-Length: 475
Content-Type: application/json

{"sampling_feature": "12345678-abcd-1234-efgh-1234567890ab", "timestamp": "2018-02-07T15:06:00-05:00", "9e9f2c7b-0ab3-4dfe-b933-ef0a1920bb63": 4.639, "33218554-9cf6-48f3-b8d0-4d032dcb8578": 10712, "c34b0278-3267-4969-929e-62c45a1d8ff1": 25.75, "47a3117d-9c39-43c9-87c2-a05aa573e1b0": 792, "a58c3985-45cd-407c-92d6-18b89dde8c4d": 0.0, "908ad4f8-b6db-4b0c-85c6-b4f14c3229bf": 21.26, "f05e563f-456e-459c-8dcb-aa87d06489c5": 38.19, "89baa175-b592-42b5-8322-de2df59e4bfd": 21.58 }

 -- Response Code -- 
201

 \/---- Line Saved to SD Card ----\/ 
2018-02-07 15:06:00,4.639,10712,25.75,792,0.0,21.26,38.19,21.58
------------------------------------------

------------------------------------------

 \/---- Post Request to EnviroDIY ----\/ 
POST /api/data-stream/ HTTP/1.1
Host: data.envirodiy.org
TOKEN: 12345678-abcd-1234-efgh-1234567890ab
Content-Length: 475
Content-Type: application/json

{"sampling_feature": "12345678-abcd-1234-efgh-1234567890ab", "timestamp": "2018-02-07T15:08:00-05:00", "9e9f2c7b-0ab3-4dfe-b933-ef0a1920bb63": 4.639, "33218554-9cf6-48f3-b8d0-4d032dcb8578": 10712, "c34b0278-3267-4969-929e-62c45a1d8ff1": 26.25, "47a3117d-9c39-43c9-87c2-a05aa573e1b0": 792, "a58c3985-45cd-407c-92d6-18b89dde8c4d": 0.0, "908ad4f8-b6db-4b0c-85c6-b4f14c3229bf": 21.30, "f05e563f-456e-459c-8dcb-aa87d06489c5": 30.63, "89baa175-b592-42b5-8322-de2df59e4bfd": 21.75 }

 -- Response Code -- 
201

 \/---- Line Saved to SD Card ----\/ 
2018-02-07 15:08:00,4.639,10712,26.25,792,0.0,21.30,30.63,21.75
------------------------------------------

------------------------------------------

 \/---- Post Request to EnviroDIY ----\/ 
POST /api/data-stream/ HTTP/1.1
Host: data.envirodiy.org
TOKEN: 12345678-abcd-1234-efgh-1234567890ab
Content-Length: 475
Content-Type: application/json

{"sampling_feature": "12345678-abcd-1234-efgh-1234567890ab", "timestamp": "2018-02-07T15:10:00-05:00", "9e9f2c7b-0ab3-4dfe-b933-ef0a1920bb63": 4.639, "33218554-9cf6-48f3-b8d0-4d032dcb8578": 10712, "c34b0278-3267-4969-929e-62c45a1d8ff1": 26.50, "47a3117d-9c39-43c9-87c2-a05aa573e1b0": 792, "a58c3985-45cd-407c-92d6-18b89dde8c4d": 0.0, "908ad4f8-b6db-4b0c-85c6-b4f14c3229bf": 21.31, "f05e563f-456e-459c-8dcb-aa87d06489c5": 25.64, "89baa175-b592-42b5-8322-de2df59e4bfd": 21.88 }

 -- Response Code -- 
201

 \/---- Line Saved to SD Card ----\/ 
2018-02-07 15:10:00,4.639,10712,26.50,792,0.0,21.31,25.64,21.88
------------------------------------------

------------------------------------------

 \/---- Post Request to EnviroDIY ----\/ 
POST /api/data-stream/ HTTP/1.1
Host: data.envirodiy.org
TOKEN: 
Content-Length: 481
Content-Type: application/json

{"sampling_feature": "12345678-abcd-1234-efgh-1234567890ab", "timestamp": "2018-02-07T15:12:00-05:00", "9e9f2c7b-0ab3-4dfe-b933-ef0a1920bb63": 4.655, "33218554-9cf6-48f3-b8d0-4d032dcb8578": 10712, "c34b0278-3267-4969-929e-62c45a1d8ff1": 26.50, "47a3117d-9c39-43c9-87c2-a05aa573e1b0": 792, "a58c3985-45cd-407c-92d6-18b89dde8c4d": -1999.8, "908ad4f8-b6db-4b0c-85c6-b4f14c3229bf": -1982.74, "f05e563f-456e-459c-8dcb-aa87d06489c5": 1.59, "89baa175-b592-42b5-8322-de2df59e4bfd": 22.00 }

 -- Response Code -- 
201

 \/---- Line Saved to SD Card ----\/ 
2018-02-07 15:12:00,4.655,10712,26.50,792,-1999.8,-1982.74,1.59,22.00
------------------------------------------

------------------------------------------

 \/---- Post Request to EnviroDIY ----\/ 
POST /api/data-stream/ HTTP/1.1
Host: data.envirodiy.org
TOKEN: 12345678-abcd-1234-efgh-1234567890ab
Content-Length: 474
Content-Type: application/json

{"sampling_feature": "12345678-abcd-1234-efgh-1234567890ab", "timestamp": "2018-02-07T15:14:00-05:00", "9e9f2c7b-0ab3-4dfe-b933-ef0a1920bb63": 4.670, "33218554-9cf6-48f3-b8d0-4d032dcb8578": 10712, "c34b0278-3267-4969-929e-62c45a1d8ff1": 26.75, "47a3117d-9c39-43c9-87c2-a05aa573e1b0": 793, "a58c3985-45cd-407c-92d6-18b89dde8c4d": 0.0, "908ad4f8-b6db-4b0c-85c6-b4f14c3229bf": 21.36, "f05e563f-456e-459c-8dcb-aa87d06489c5": 0.94, "89baa175-b592-42b5-8322-de2df59e4bfd": 22.13 }

 -- Response Code -- 
201

 \/---- Line Saved to SD Card ----\/ 
2018-02-07 15:14:00,4.670,10712,26.75,793,0.0,21.36,0.94,22.13
------------------------------------------

------------------------------------------

To show both how to sleep without using the "log" function and how to use more than one logging interval.

Right now, sensors output only floats. This means that the proper number of significant figures cannot be set for the sensors. To correct this, it would be better to be able to get data from the sensors as a string.

Need to figure out how to make this happen. I do not think it will be possible to use the timer library because of the requirement that, when used within another library, all of the "every" functions must be static.

Instead of going in order through sensors and waiting for each one to finish, schedule events on a timer.

May want to allow to use AtMega1284P's (AVR) onboard 10bit ADC or SAMD21's onboard 12bit ADC.

@s-hicks2 suggested that the DS18B20 Waterproof Digital temperature sensor is an excellent choice for stand-alone water temperature. I've been thinking that they would make for an idea pairing with a Maxbotix sonar water level sensor, for a low cost system.

The LimnoTech team will try to get a prototype sketch going, using instructions from https://create.arduino.cc/projecthub/TheGadgetBoy/ds18b20-digital-temperature-sensor-and-arduino-9cc806 and https://github.com/milesburton/Arduino-Temperature-Control-Library.

Explain what each example is and why it's useful.

Currently allowing a warm up time of 500ms no matter what because some sensors need it. (Actually OBS3 would do better w/ 1sec or greater.) If each sensor "knew" how long it had been on and how long it needed to warm up, this time could be minimized.

Simple reed switch

This seems much safer.

SWRC has deployed several of these sensors, and LimnoTech is about to deploy one in the next weeks.
SQ-212: Amplified 0-2.5 Volt Sun Calibration Quantum Sensor
https://www.apogeeinstruments.com/sq-212-amplified-0-2-5-volt-sun-calibration-quantum-sensor/

@SRGDamia1 , can you work with @s-hicks2 to add a class for this sensor?

This would also be a relative simple example to describe how to create new sensor sub-classes.

These sensors are used in Workshop Sketch 6.

• https://github.com/EnviroDIY/EnviroDIY_Mayfly_Logger/tree/master/workshop/mayfly_06_humidity-temp

and in this potential workshop sketch:

• https://github.com/EnviroDIY/EnviroDIY_Mayfly_Logger/blob/Mayfly_11_DT-LED/workshop/mayfly_10b_DHT_LED/mayfly_10b_DHT_LED.ino

Info on these sensors can be found at:

• http://wiki.seeed.cc/Grove-TemperatureAndHumidity_Sensor/
• http://wiki.seeed.cc/Grove-Temperature_and_Humidity_Sensor_Pro/

We've used the Adafruit DHT-sensor-library:

• https://github.com/adafruit/DHT-sensor-library/tree/c97897771807613d456b318236e18a04b013410b

Allow more than one sensor to take a measurement at a time, similar to SDI-12's concurrent measurements.

This could be very tricky (or impossible) with averaging multiple measurements from the same sensor.

the modem serial port isn't actually sending anything to the modem.

@SRGDamia1, do you have a conductivity sensor (Y520) to test the sketches? We are pretty sure the one we have is faulty, but it would be nice to have another testing the sketch.

with @fisherba.

Use docs and code examples in https://github.com/EnviroDIY/Paleoterra_Redox

The signal quality returned by the LM request isn't populated until after the first data is received. So running the update before that happens always results in a signal of 0%.

@SRGDamia1, I just noticed that my data usage for my 160065 CROSSLANDSPOND - 10 VARIABLES station (http://data.envirodiy.org/sites/6/) shot up from 0.74 MB per day to 5.3 MB per day from July 26 to Aug. 3, exactly when the Data.EnviroDIY.org system went down. See:

I'm thinking that this jump might be due to repeated attempts by the logger to send the radio data. Can we limit the number of attempts? This 7x increase in data usage for 6 days added a good bit to my monthly cost (almost $30).

Also, I'm thinking that this may have also significantly chewed on power. @s-hicks2 mentioned that she's seen power issues. Are they related? Also, our six stations on the Kinni, deployed by @CTaylorLTI, all permanently lost connectivity around the 26th, and we're wondering if that might be due to the power getting drained. He'll go out soon to find out, and I'll try to download the voltage data for our data gap period to see if we had a similar problem, but I think we did (voltage on our first day back online was lower than usual).

The default output units for Yosemitech Y520 sensor is mS/cm (milli Siemens per centimeter), but nearly everyone looking at freshwater uses µS/cm (microS/cm, sometimes written as uS/cm). This includes the EnviroDIY Data Portal.

It seems we should provide the unit conversion in this library, rather than in https://github.com/EnviroDIY/YosemitechModbus.

@SRGDamia1, Where/how do we do that?

This library seems smaller and simpler and can send data using a stream print, more like how we're already using the WiFiBee in transparent mode. It does not, however, handle powering the Bee on and off.

https://github.com/vshymanskyy/TinyGSM

In case someone just wants to use one sensor without a SensorArray. It's really a waste to use this library if that's all you want to do, but someone may still want to.

Somewhere in the adaptations made for SAMD boards (#68), I messed up something with the DS3231 clock so it is reading years 30 years in the future. The date/hours/mins/seconds all seem perfect, it's just the years that are off.

If you create more than 1 logger instance logging to the same file (ie, at different intervals) the csv output will be pretty confusing because the order of the columns will alternate with which logger it is.

I think things are mature enough that it might be worth creating our first release in the next week or so, after a tiny bit of more testing.

How about v 0.1.0? Do we need to add "alpha" or "beta"? We could, but we're already using this code to run many Mayfly stations, so I'm thinking that we just give it a number. It works. The next bug fixes can be v 0.1.1. The next time we add sensors or any other important feature, it becomes v 0.2.0, etc.

using Modbus RTU.

Probably need to modify the 3GBee library to support headers.

@SRGDamia1, this is awesome. I have it running with Mayfly v0.3 and a MaxBotix sonar.

Activate a mode where data is sent every 10 seconds for faster diagnosis of problems.