earthscope / ispaq Goto Github PK
View Code? Open in Web Editor NEWPython command line script that uses R packages to calculate seismology data quality metrics.
License: GNU Lesser General Public License v3.0
Python command line script that uses R packages to calculate seismology data quality metrics.
License: GNU Lesser General Public License v3.0
A request came in to be able to plot the mean or median values on the PDF plot, instead of just the mode values as is hardcoded right now.
bug report: transfer_function giving inaccurate results for gain_ratio and phase_difference.
I want to use ISPAQ with my own local FDSN server, is it possible to do this without modifying the code?
./run_ispaq.py -M sample_mean -S DZ.ABZH.*.HHZ --starttime 2023-08-14 --endtime 2023-08-15 --dataselect_url http://localhost:8081
2023-08-22 20:27:14 - INFO - Running ISPAQ version 3.1.1 on Tue Aug 22 20:27:14 2023
2023-08-22 20:27:16 - WARNING - Station_url should be the same as dataselect_url when retreiving data from FDSN webservices. Station_url 'IRIS' does not match dataselect_url 'http://localhost:8081'
2023-08-22 20:27:17 - INFO - No stations found for DZ.ABZH.*.HHZ
2023-08-22 20:27:17 - INFO - No available waveforms for 2023-08-14 matching ['DZ.ABZH.*.HHZ']
2023-08-22 20:27:17 - WARNING - "simple" metric calculation generated zero metrics
2023-08-22 20:27:17 - INFO - No simple metrics were calculated
2023-08-22 20:27:17 - INFO - ALL FINISHED!
If the value for csv_dir is left blank in the preference file and it is not specified on the command line, then ISPAQ will produce an error like:
2023-08-02 11:42:20 - CRITICAL - Failed to create UserRequest object
And with DEBUG on, just before that CRITICAL message is
2023-08-02 11:42:20 - DEBUG - expected str, bytes or os.PathLike object, not NoneType
The offending block of code in user_request.py's UserRequest class:
if self.csv_dir is None:
if 'csv_dir' in preferences:
self.csv_dir = os.path.abspath(os.path.expanduser(preferences['csv_dir']))
else:
self.csv_dir = os.path.abspath('.')
else:
self.csv_dir = os.path.abspath(os.path.expanduser(self.csv_dir))
The problem here is that self.csv_dir is None (not specified on the command line), so it hits the first branch of the if-tree. But since the key csv_dir is in preference (csv_dir: None) it then tries to do os.path.expanduser(None), which errors. The code here assumes that if the key csv_dir exists in the perference file, so does the associated value. So it will catch cases where that key isn’t present, but not when the value is null.
Suggesting that we change that block of code to capture all possible issues and default to '.' if it can't read in csv_dir properly:
```
if self.csv_dir is None:
try:
self.csv_dir = os.path.abspath(os.path.expanduser(preferences['csv_dir']))
except:
self.csv_dir = os.path.abspath('.')
else:
self.csv_dir = os.path.absp
This catches the cases where csv_dir is removed from the preference file completely, as well as cases where the field is left blank. Also any other possible error that could exist when trying to create that path.
Can I analyze the acceleration data for the ambient noise using the ispaq?
If possible, can you let me know how to do?
Thanks,
I attempted to install and use ispaq on the macOS M1 chip by setting up the environment in Anaconda. After installation, I activated the ispaq environment (conda activate ispaq) and tried running run_ispaq.py. However, I encountered the following error:
"ERROR: please activate your ispaq environment before running: No module named 'rpy2'"
To verify the presence of the 'rpy2' module, I executed 'import rpy2' in IPython, and the module was indeed available.
It's worth noting that './run_ispaq.py -h' executed successfully.
I would appreciate guidance on resolving this issue.
Feature request:
"The current Probability Density Functions (PDF) plots, that we are receiving as PNG image files are in the units of Power x Period, would it be possible to add a Unit parameter, to be able to produce the PDF in units of Power x Frequency instead?"
Several urls fail when creating the conda env.
Like so:
CondaError: CondaHTTPError: HTTP 404 NOT FOUND for url <https://conda.anaconda.org/r/osx-64/ncurses-5.9-8.tar.bz2>
Attached spec works. I tried to push a branch, but did not have permissions.
ispaq-spec2.txt
We found that when the miniSEED segments are out of order, the availability values (e.g. num_gaps, num_overlaps, percent_availability) are inaccurate.
This was observed using local files for QW.BCV11 which has many segments in a single day file:
msi -T -Q QW.BCV11..HNZ.2022.152
....
Total: 1 trace(s) with 157 segment(s)
Of note, the data within the day file are out of order. For example:
msi QW.BCV11..HNZ.2022.152
...
QW_BCV11__HNZ, 000436, D, 256, 172 samples, 100 Hz, 2022,152,20:27:12.070000
QW_BCV11__HNZ, 000437, D, 256, 172 samples, 100 Hz, 2022,152,21:15:19.150000
QW_BCV11__HNZ, 000438, D, 256, 172 samples, 100 Hz, 2022,152,20:27:13.790000
QW_BCV11__HNZ, 000439, D, 256, 172 samples, 100 Hz, 2022,152,20:27:15.510000
QW_BCV11__HNZ, 000440, D, 256, 172 samples, 100 Hz, 2022,152,21:15:20.870000
...
ISPAQ was calculating unexpectedly high values for the gaps and overlaps metrics, and reporting 0% availability:
QW.BCV11..HNZ.D,2022-06-01T00:00:00,2022-06-02T00:00:00,num_gaps,1210
QW.BCV11..HNZ.D,2022-06-01T00:00:00,2022-06-02T00:00:00,max_gap,8135.86
QW.BCV11..HNZ.D,2022-06-01T00:00:00,2022-06-02T00:00:00,num_overlaps,1008
QW.BCV11..HNZ.D,2022-06-01T00:00:00,2022-06-02T00:00:00,max_overlap,5518.28
QW.BCV11..HNZ.D,2022-06-01T00:00:00,2022-06-02T00:00:00,percent_availability,0
ISPAQ uses ObsPy to read in the miniSEED files, which was also reading the data into the stream object out of chronological order. When this happens, the ObsPy object indicates that there are 2217 traces - which is the num_gaps plus num_overlaps reported (plus 1). This indicated a probable source of the erroneous metric values.
>>> len(obspy.read('QW.BCV11..HNZ.2022.152'))
2217
To test whether this ordering was causing the problem, I tested out using the .sort() method on the ObsPy stream object after reading in the data. After adding the sorting, the availability metrics seem more reasonable:
QW.BCV11..HNZ.D,2022-06-01T00:00:00,2022-06-02T00:00:00,num_gaps,158
QW.BCV11..HNZ.D,2022-06-01T00:00:00,2022-06-02T00:00:00,max_gap,8135.86
QW.BCV11..HNZ.D,2022-06-01T00:00:00,2022-06-02T00:00:00,num_overlaps,0
QW.BCV11..HNZ.D,2022-06-01T00:00:00,2022-06-02T00:00:00,max_overlap,0
QW.BCV11..HNZ.D,2022-06-01T00:00:00,2022-06-02T00:00:00,percent_availability,33.9016
Dear Sir,
I am trying to install the ispaq package. But I have faced problem as below.
Could you please let me know how to solve this problem?
Thank you in advance.
(ispaq) jaeyeol-ui-MacBook-Pro:ispaq jaeyeolyang$ export MACOSX_DEPLOYMENT_TARGET=10.9
(ispaq) jaeyeol-ui-MacBook-Pro:ispaq jaeyeolyang$ ./run_ispaq.py -I
2022-02-12 21:09:16 - INFO - Running ISPAQ version 3.0.0 on Sat Feb 12 21:09:16 2022
Traceback (most recent call last):
File "./run_ispaq.py", line 15, in
main()
File "/Users/jaeyeolyang/ispaq/ispaq/ispaq.py", line 149, in main
import obspy
File "/Users/jaeyeolyang/anaconda3/envs/ispaq/lib/python3.8/site-packages/obspy/init.py", line 39, in
from obspy.core.utcdatetime import UTCDateTime # NOQA
File "/Users/jaeyeolyang/anaconda3/envs/ispaq/lib/python3.8/site-packages/obspy/core/init.py", line 124, in
from obspy.core.utcdatetime import UTCDateTime # NOQA
File "/Users/jaeyeolyang/anaconda3/envs/ispaq/lib/python3.8/site-packages/obspy/core/utcdatetime.py", line 27, in
from obspy.core.util.deprecation_helpers import ObsPyDeprecationWarning
File "/Users/jaeyeolyang/anaconda3/envs/ispaq/lib/python3.8/site-packages/obspy/core/util/init.py", line 27, in
from obspy.core.util.base import (ALL_MODULES, DEFAULT_MODULES,
File "/Users/jaeyeolyang/anaconda3/envs/ispaq/lib/python3.8/site-packages/obspy/core/util/base.py", line 36, in
from obspy.core.util.misc import to_int_or_zero, buffered_load_entry_point
File "/Users/jaeyeolyang/anaconda3/envs/ispaq/lib/python3.8/site-packages/obspy/core/util/misc.py", line 214, in
loadtxt(np.array([0]), ndmin=1)
File "/Users/jaeyeolyang/anaconda3/envs/ispaq/lib/python3.8/site-packages/numpy/lib/npyio.py", line 1086, in loadtxt
ncols = len(usecols or split_line(first_line))
File "/Users/jaeyeolyang/anaconda3/envs/ispaq/lib/python3.8/site-packages/numpy/lib/npyio.py", line 977, in split_line
line = line.split(comment, 1)[0]
AttributeError: 'numpy.int64' object has no attribute 'split'
Thanks a lot for designing, implementing and making ISPAQ available here on github - it's proven very useful for computing and managing data quality metrics for a database that I'm working on!
I would like to report one point regarding ispaq's resource usage: As a lot of the operations in ispaq run on a single core, I am right now starting a few runs of ispaq in parallel from the terminal. But while computing some of the basic metrics, probably within some python libraries (numpy?), ispaq uses all available cores at once and hence slows down the system for all users (and then I get an understandable complaint by the system admin...).
I still don't fully understand which function calls use parallelization as I didn't see any explicit parallelization in ispaq, but for now I was able to limit each run to one thread by setting the following environment variables at the very start of ispaq.py
:
import os
os.environ["OMP_NUM_THREADS"] = "1"
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["MKL_NUM_THREADS"] = "1"
os.environ["NUMEXPR_NUM_THREADS"] = "1"
os.environ["VECLIB_MAXIMUM_THREADS"] = "1"
I hope this can help others who run ispaq on a system that they share with other users. Maybe there could be a command line option to limit the number of cores used by ispaq?
when I am installing ISPAQ, I am getting an error due to applying the .tolist()
on rpy2.robjects.vectors.StrVector
object. This is breaking the code every time I run ./run_ispaq.py.
Traceback (most recent call last):
File "./run_ispaq.py", line 15, in
main()
File "/home/sjohn/AON_PROJECT/ispaq/ispaq/ispaq.py", line 160, in main
installed_names = ro.conversion.rpy2py(r_installed.rownames).tolist()
AttributeError: 'StrVector' object has no attribute 'tolist'
could you please tell me how to resolve this?
Thank you
On a Debian linux I have the following error
conda create -n ispaq --file ispaq-explicit-spec-file.txt
r-r6-2.1.1-r3. 100% |################################| Time: 0:00:00 181.41 kB/s
numpy-1.11.1-p 100% |################################| Time: 0:00:00 3.40 MB/s
r-lattice-0.20 100% |################################| Time: 0:00:01 602.97 kB/s
singledispatch 100% |################################| Time: 0:00:00 22.14 MB/s
jpeg-8d-2.tar. 100% |################################| Time: 0:00:00 629.80 kB/s
r-cluster-2.0. 100% |################################| Time: 0:00:00 1.01 MB/s
libxslt-1.1.28 100% |################################| Time: 0:00:00 1.08 MB/s
r-dplyr-0.4.3- 100% |################################| Time: 0:00:01 1.40 MB/s
curl-7.49.0-1. 100% |################################| Time: 0:00:00 44.56 MB/s
r-dbi-0.3.1-r3 100% |################################| Time: 0:00:00 2.14 MB/s
scipy-0.18.0-n 100% |################################| Time: 0:00:01 9.66 MB/s
r-spatial-7.3_ 100% |################################| Time: 0:00:00 313.62 kB/s
icu-56.1-4.tar 100% |################################| Time: 0:00:02 8.83 MB/s
mkl-11.3.3-0.t 100% |################################| Time: 0:00:02 51.39 MB/s
jbig-2.1-0.tar 100% |################################| Time: 0:00:00 42.59 MB/s
sqlalchemy-1.0 100% |################################| Time: 0:00:00 2.34 MB/s
r-rcurl-1.95_4 100% |################################| Time: 0:00:00 1.01 MB/s
openssl-1.0.2h 100% |################################| Time: 0:00:01 3.03 MB/s
r-boot-1.3_17- 100% |################################| Time: 0:00:00 832.36 kB/s
r-evaluate-0.8 100% |################################| Time: 0:00:00 33.20 MB/s
r-rcpp-0.12.2- 100% |################################| Time: 0:00:00 4.82 MB/s
gsl-1.16-0.tar 100% |################################| Time: 0:00:00 13.19 MB/s
r-stringi-1.0_ 100% |################################| Time: 0:00:00 16.38 MB/s
xz-5.2.2-0.tar 100% |################################| Time: 0:00:00 48.07 MB/s
libxml2-2.9.4- 100% |################################| Time: 0:00:00 8.41 MB/s
python-2.7.12- 100% |################################| Time: 0:00:01 7.72 MB/s
r-git2r-0.11.0 100% |################################| Time: 0:00:00 1.36 MB/s
r-nnet-7.3_11- 100% |################################| Time: 0:00:00 1.27 MB/s
libpng-1.6.24- 100% |################################| Time: 0:00:00 1.45 MB/s
functools32-3. 100% |################################| Time: 0:00:00 26.33 MB/s
r-mime-0.4-r3. 100% |################################| Time: 0:00:00 27.96 MB/s
r-3.2.2-0.tar. 100% |################################| Time: 0:00:00 1.40 MB/s
matplotlib-1.5 100% |################################| Time: 0:00:01 3.82 MB/s
r-quadprog-1.5 100% |################################| Time: 0:00:00 59.06 kB/s
libiconv-1.14- 100% |################################| Time: 0:00:00 1.59 MB/s
python-dateuti 100% |################################| Time: 0:00:00 1.02 MB/s
libgcc-4.8.5-1 100% |################################| Time: 0:00:00 1.70 MB/s
r-stringr-1.0. 100% |################################| Time: 0:00:00 1.02 MB/s
r-rstudioapi-0 100% |################################| Time: 0:00:00 30.35 MB/s
r-devtools-1.9 100% |################################| Time: 0:00:00 870.02 kB/s
obspy-1.0.3-py 100% |################################| Time: 0:00:02 9.60 MB/s
r-magrittr-1.5 100% |################################| Time: 0:00:00 1.06 MB/s
requests-2.10. 100% |################################| Time: 0:00:00 7.17 MB/s
r-mgcv-1.8_9-r 100% |################################| Time: 0:00:00 5.91 MB/s
freetype-2.6.3 100% |################################| Time: 0:00:00 4.89 MB/s
r-foreign-0.8_ 100% |################################| Time: 0:00:00 2.94 MB/s
r-curl-0.9.4-r 100% |################################| Time: 0:00:00 3.28 MB/s
r-class-7.3_14 100% |################################| Time: 0:00:00 48.81 MB/s
r-base-3.2.2-0 100% |################################| Time: 0:00:02 8.70 MB/s
decorator-4.0. 100% |################################| Time: 0:00:00 20.61 MB/s
libtiff-4.0.6- 100% |################################| Time: 0:00:00 53.38 MB/s
r-codetools-0. 100% |################################| Time: 0:00:00 32.21 MB/s
r-xml2-0.1.2-r 100% |################################| Time: 0:00:00 53.54 MB/s
r-matrix-1.2_2 100% |################################| Time: 0:00:00 10.40 MB/s
r-memoise-0.2. 100% |################################| Time: 0:00:00 26.06 MB/s
cycler-0.10.0- 100% |################################| Time: 0:00:00 23.31 MB/s
pip-8.1.2-py27 100% |################################| Time: 0:00:00 2.06 MB/s
pandas-0.18.1- 100% |################################| Time: 0:00:00 66.44 MB/s
r-kernsmooth-2 100% |################################| Time: 0:00:00 260.91 kB/s
r-bitops-1.0_6 100% |################################| Time: 0:00:00 275.45 kB/s
r-bh-1.58.0_1- 100% |################################| Time: 0:00:01 4.51 MB/s
future-0.15.2- 100% |################################| Time: 0:00:00 7.63 MB/s
r-recommended- 100% |################################| Time: 0:00:00 1.47 MB/s
r-rversions-1. 100% |################################| Time: 0:00:00 196.33 kB/s
lxml-3.6.1-py2 100% |################################| Time: 0:00:00 1.19 MB/s
r-jsonlite-0.9 100% |################################| Time: 0:00:00 1.21 MB/s
r-httr-1.0.0-r 100% |################################| Time: 0:00:00 3.89 MB/s
r-pracma-1.8.8 100% |################################| Time: 0:00:00 6.24 MB/s
pcre-8.39-0.ta 100% |################################| Time: 0:00:00 289.48 kB/s
r-signal-0.7_6 100% |################################| Time: 0:00:00 617.51 kB/s
r-roxygen2-5.0 100% |################################| Time: 0:00:00 784.10 kB/s
pyparsing-2.1. 100% |################################| Time: 0:00:00 1.14 MB/s
r-nlme-3.1_122 100% |################################| Time: 0:00:00 2.55 MB/s
r-xml-3.98_1.3 100% |################################| Time: 0:00:00 4.35 MB/s
setuptools-25. 100% |################################| Time: 0:00:00 869.46 kB/s
r-assertthat-0 100% |################################| Time: 0:00:00 372.91 kB/s
ncurses-5.9-8. 100% |################################| Time: 0:00:00 1.86 MB/s
r-survival-2.3 100% |################################| Time: 0:00:00 6.40 MB/s
pytz-2016.6.1- 100% |################################| Time: 0:00:00 719.78 kB/s
r-digest-0.6.8 100% |################################| Time: 0:00:00 621.07 kB/s
r-brew-1.0_6-r 100% |################################| Time: 0:00:00 914.81 kB/s
r-rpart-4.1_10 100% |################################| Time: 0:00:00 1.56 MB/s
r-lazyeval-0.1 100% |################################| Time: 0:00:00 40.14 MB/s
r-mass-7.3_45- 100% |################################| Time: 0:00:00 4.29 MB/s
rpy2-2.7.8-py2 100% |################################| Time: 0:00:00 3.08 MB/s
r-whisker-0.3_ 100% |################################| Time: 0:00:00 41.36 MB/s
ERROR conda.core.link:_execute_actions(330): An error occurred while installing package 'defaults::openssl-1.0.2h-2'.
LinkError: post-link failed for: defaults::openssl-1.0.2h-2
Attempting to roll back.
LinkError: post-link failed for: defaults::openssl-1.0.2h-2
A request came in for the ability to produce PSDs for data that does not have response information available.
Hello,
When using FDSN dataselect as data source, it seems the code is expecting data trimmed to the nearest sample. I noted that trying to perform orientationCheck and crossTalk metrics, where the calculation was not possible because of different stream length returned for each channel.
However, the FDSN 1.1 webservice specification states that the dataselect shoud return samples starting on or after the starttime, and ending on or before the endtime.
http://www.fdsn.org/webservices/FDSN-WS-Specifications-1.1.pdf
Using Obspy get_waveform method instead of the R getDataselect, I was able to download the data, and trim them to the nearest common data section.
However, I lost the data quality flags in the process.
Otherwise, very interesting and great tool !!
Jean-Marie.
HI,
I am trying to retrieve and run data quality metrics on data downloaded by INGV, but I get several error messages. To reproduce:
./run_ispaq.py -M sample_mean -S IV.VAGA.*.HH? --starttime 2018-08-14 --endtime 2018-08-15 --dataselect_url INGV --station_url INGV --event_url INGV --output csv --csv_dir ./test_out --log-level DEBUG
Log file:
2022-12-05 14:09:56 - INFO - Running ISPAQ version 3.0.1 on Mon Dec 5 14:09:56 2022
2022-12-05 14:09:58 - DEBUG - Creating UserRequest ...
2022-12-05 14:09:58 - DEBUG - User request initialization
2022-12-05 14:09:58 - DEBUG - preferences_file ./preference_files/default.txt
2022-12-05 14:09:58 - DEBUG - Explicit metric detected 'sample_mean'
2022-12-05 14:09:58 - DEBUG - Explicit SNCL detected 'IV.VAGA.*.HH?'
2022-12-05 14:09:58 - DEBUG - Creating Concierge ...
2022-12-05 14:09:58 - DEBUG - starttime 2018-08-14T00:00:00, endtime 2018-08-15T00:00:00
2022-12-05 14:09:58 - DEBUG - metric_names ['sample_mean']
2022-12-05 14:09:58 - DEBUG - sncl_patterns ['IV.VAGA.*.HH?']
2022-12-05 14:09:58 - DEBUG - dataselect_url http://webservices.ingv.it
2022-12-05 14:09:58 - DEBUG - dataselect_type fdsnws
2022-12-05 14:09:58 - DEBUG - station_url http://webservices.ingv.it
2022-12-05 14:09:58 - DEBUG - event_url http://webservices.ingv.it
2022-12-05 14:09:58 - DEBUG - resp_dir None
2022-12-05 14:09:58 - DEBUG - output csv
2022-12-05 14:09:58 - DEBUG - db_name ispaq.db
2022-12-05 14:09:58 - DEBUG - csv_dir [...]/ISPAQ/ispaq/test_out
2022-12-05 14:09:58 - DEBUG - pdf_dir [...]/ISPAQ/ispaq/PDFs
2022-12-05 14:09:58 - DEBUG - psd_dir [...]/ISPAQ/ispaq/PSDs
2022-12-05 14:09:58 - DEBUG - pdf_type ['text', 'plot']
2022-12-05 14:09:58 - DEBUG - pdf_interval ['daily', 'aggregated']
2022-12-05 14:09:58 - DEBUG - plot_include ['legend', 'colorbar']
2022-12-05 14:09:58 - DEBUG - sigfigs 6
2022-12-05 14:09:58 - DEBUG - sncl_format N.S.L.C
2022-12-05 14:09:58 - DEBUG - Inside simple business logic ...
2022-12-05 14:09:58 - DEBUG - channelFilter .*
2022-12-05 14:09:58 - DEBUG - read FDSN station web services http://webservices.ingv.it for IV,VAGA,*,HH?,2018.226,2018.227
2022-12-05 14:09:58 - DEBUG - The parameter 'matchtimeseries' is not supported by the service.
2022-12-05 14:09:58 - INFO - No stations found for IV.VAGA.*.HH?
2022-12-05 14:09:58 - INFO - No available waveforms for 2018-08-14 matching ['IV.VAGA.*.HH?']
2022-12-05 14:09:58 - DEBUG - skipping 2018-08-14 with no available data
2022-12-05 14:09:58 - WARNING - "simple" metric calculation generated zero metrics
2022-12-05 14:09:58 - INFO - No simple metrics were calculated
2022-12-05 14:09:58 - INFO - ALL FINISHED!
Thank you in advance for your help!
Best,
Angela
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.