earthscope / ispaq Goto Github PK

View Code? Open in Web Editor NEW

20.0 20.0 11.0 71.39 MB

Python command line script that uses R packages to calculate seismology data quality metrics.

License: GNU Lesser General Public License v3.0

Python 90.87% Jupyter Notebook 9.13%

ispaq's People

Contributors

Stargazers

Watchers

Forkers

ardasysmic jeasonxiu lpk-py rainzhop flixha hasanissa25 tp6u3su6 chandpes linaricoar catmou tugbayk

ispaq's Issues

Some notes/issues on mac installation

PDF plot mean/median/mode options

A request came in to be able to plot the mean or median values on the PDF plot, instead of just the mode values as is hardcoded right now.

bug report: transfer_function giving inaccurate results for gain_ratio and phase_difference

bug report: transfer_function giving inaccurate results for gain_ratio and phase_difference.

Local FDSN with ISPAQ

I want to use ISPAQ with my own local FDSN server, is it possible to do this without modifying the code?

./run_ispaq.py -M sample_mean -S DZ.ABZH.*.HHZ --starttime 2023-08-14 --endtime 2023-08-15 --dataselect_url http://localhost:8081

2023-08-22 20:27:14 - INFO - Running ISPAQ version 3.1.1 on Tue Aug 22 20:27:14 2023
2023-08-22 20:27:16 - WARNING - Station_url should be the same as dataselect_url when retreiving data from FDSN webservices. Station_url 'IRIS' does not match dataselect_url 'http://localhost:8081'
2023-08-22 20:27:17 - INFO - No stations found for DZ.ABZH.*.HHZ
2023-08-22 20:27:17 - INFO - No available waveforms for 2023-08-14 matching ['DZ.ABZH.*.HHZ']
2023-08-22 20:27:17 - WARNING - "simple" metric calculation generated zero metrics
2023-08-22 20:27:17 - INFO - No simple metrics were calculated
2023-08-22 20:27:17 - INFO - ALL FINISHED!

UserRequest fails when csv_dir is blank

If the value for csv_dir is left blank in the preference file and it is not specified on the command line, then ISPAQ will produce an error like:
2023-08-02 11:42:20 - CRITICAL - Failed to create UserRequest object

And with DEBUG on, just before that CRITICAL message is
2023-08-02 11:42:20 - DEBUG - expected str, bytes or os.PathLike object, not NoneType

The offending block of code in user_request.py's UserRequest class:

 if self.csv_dir is None:
                if 'csv_dir' in preferences:
                    self.csv_dir = os.path.abspath(os.path.expanduser(preferences['csv_dir']))
                else:
                    self.csv_dir = os.path.abspath('.')
            else:
                self.csv_dir = os.path.abspath(os.path.expanduser(self.csv_dir))

The problem here is that self.csv_dir is None (not specified on the command line), so it hits the first branch of the if-tree. But since the key csv_dir is in preference (csv_dir: None) it then tries to do os.path.expanduser(None), which errors. The code here assumes that if the key csv_dir exists in the perference file, so does the associated value. So it will catch cases where that key isn’t present, but not when the value is null.

Suggesting that we change that block of code to capture all possible issues and default to '.' if it can't read in csv_dir properly:

```

if self.csv_dir is None:
try:
self.csv_dir = os.path.abspath(os.path.expanduser(preferences['csv_dir']))
except:
self.csv_dir = os.path.abspath('.')
else:
self.csv_dir = os.path.absp


This catches the cases where csv_dir is removed from the preference file completely, as well as cases where the field is left blank. Also any other possible error that could exist when trying to create that path.

About analysis for the acceleration data.

Can I analyze the acceleration data for the ambient noise using the ispaq?
If possible, can you let me know how to do?

Thanks,

Problem with installation and usage on macOS M1 chip

I attempted to install and use ispaq on the macOS M1 chip by setting up the environment in Anaconda. After installation, I activated the ispaq environment (conda activate ispaq) and tried running run_ispaq.py. However, I encountered the following error:

"ERROR: please activate your ispaq environment before running: No module named 'rpy2'"

To verify the presence of the 'rpy2' module, I executed 'import rpy2' in IPython, and the module was indeed available.

It's worth noting that './run_ispaq.py -h' executed successfully.

I would appreciate guidance on resolving this issue.

Feature request: add frequency to the PDF plot axes

Feature request:

"The current Probability Density Functions (PDF) plots, that we are receiving as PNG image files are in the units of Power x Period, would it be possible to add a Unit parameter, to be able to produce the PDF in units of Power x Frequency instead?"

Urls don't resolve in spec file

Several urls fail when creating the conda env.

Like so:
CondaError: CondaHTTPError: HTTP 404 NOT FOUND for url <https://conda.anaconda.org/r/osx-64/ncurses-5.9-8.tar.bz2>

Attached spec works. I tried to push a branch, but did not have permissions.
ispaq-spec2.txt

Incorrect availability values when traces out of order

We found that when the miniSEED segments are out of order, the availability values (e.g. num_gaps, num_overlaps, percent_availability) are inaccurate.

This was observed using local files for QW.BCV11 which has many segments in a single day file:

msi -T -Q QW.BCV11..HNZ.2022.152
....
Total: 1 trace(s) with 157 segment(s)

Of note, the data within the day file are out of order. For example:

msi QW.BCV11..HNZ.2022.152
...
QW_BCV11__HNZ, 000436, D, 256, 172 samples, 100 Hz, 2022,152,20:27:12.070000
QW_BCV11__HNZ, 000437, D, 256, 172 samples, 100 Hz, 2022,152,21:15:19.150000
QW_BCV11__HNZ, 000438, D, 256, 172 samples, 100 Hz, 2022,152,20:27:13.790000
QW_BCV11__HNZ, 000439, D, 256, 172 samples, 100 Hz, 2022,152,20:27:15.510000
QW_BCV11__HNZ, 000440, D, 256, 172 samples, 100 Hz, 2022,152,21:15:20.870000
...

ISPAQ was calculating unexpectedly high values for the gaps and overlaps metrics, and reporting 0% availability:

QW.BCV11..HNZ.D,2022-06-01T00:00:00,2022-06-02T00:00:00,num_gaps,1210
QW.BCV11..HNZ.D,2022-06-01T00:00:00,2022-06-02T00:00:00,max_gap,8135.86
QW.BCV11..HNZ.D,2022-06-01T00:00:00,2022-06-02T00:00:00,num_overlaps,1008
QW.BCV11..HNZ.D,2022-06-01T00:00:00,2022-06-02T00:00:00,max_overlap,5518.28
QW.BCV11..HNZ.D,2022-06-01T00:00:00,2022-06-02T00:00:00,percent_availability,0

ISPAQ uses ObsPy to read in the miniSEED files, which was also reading the data into the stream object out of chronological order. When this happens, the ObsPy object indicates that there are 2217 traces - which is the num_gaps plus num_overlaps reported (plus 1). This indicated a probable source of the erroneous metric values.

>>> len(obspy.read('QW.BCV11..HNZ.2022.152'))
2217

To test whether this ordering was causing the problem, I tested out using the .sort() method on the ObsPy stream object after reading in the data. After adding the sorting, the availability metrics seem more reasonable:

QW.BCV11..HNZ.D,2022-06-01T00:00:00,2022-06-02T00:00:00,num_gaps,158
QW.BCV11..HNZ.D,2022-06-01T00:00:00,2022-06-02T00:00:00,max_gap,8135.86
QW.BCV11..HNZ.D,2022-06-01T00:00:00,2022-06-02T00:00:00,num_overlaps,0
QW.BCV11..HNZ.D,2022-06-01T00:00:00,2022-06-02T00:00:00,max_overlap,0
QW.BCV11..HNZ.D,2022-06-01T00:00:00,2022-06-02T00:00:00,percent_availability,33.9016

About a problem for installation of the ISPAQ on the MACOSX

Dear Sir,

I am trying to install the ispaq package. But I have faced problem as below.

Could you please let me know how to solve this problem?

Thank you in advance.

(ispaq) jaeyeol-ui-MacBook-Pro:ispaq jaeyeolyang$ export MACOSX_DEPLOYMENT_TARGET=10.9
(ispaq) jaeyeol-ui-MacBook-Pro:ispaq jaeyeolyang$ ./run_ispaq.py -I
2022-02-12 21:09:16 - INFO - Running ISPAQ version 3.0.0 on Sat Feb 12 21:09:16 2022
Traceback (most recent call last):
File "./run_ispaq.py", line 15, in
main()
File "/Users/jaeyeolyang/ispaq/ispaq/ispaq.py", line 149, in main
import obspy
File "/Users/jaeyeolyang/anaconda3/envs/ispaq/lib/python3.8/site-packages/obspy/init.py", line 39, in
from obspy.core.utcdatetime import UTCDateTime # NOQA
File "/Users/jaeyeolyang/anaconda3/envs/ispaq/lib/python3.8/site-packages/obspy/core/init.py", line 124, in
from obspy.core.utcdatetime import UTCDateTime # NOQA
File "/Users/jaeyeolyang/anaconda3/envs/ispaq/lib/python3.8/site-packages/obspy/core/utcdatetime.py", line 27, in
from obspy.core.util.deprecation_helpers import ObsPyDeprecationWarning
File "/Users/jaeyeolyang/anaconda3/envs/ispaq/lib/python3.8/site-packages/obspy/core/util/init.py", line 27, in
from obspy.core.util.base import (ALL_MODULES, DEFAULT_MODULES,
File "/Users/jaeyeolyang/anaconda3/envs/ispaq/lib/python3.8/site-packages/obspy/core/util/base.py", line 36, in
from obspy.core.util.misc import to_int_or_zero, buffered_load_entry_point
File "/Users/jaeyeolyang/anaconda3/envs/ispaq/lib/python3.8/site-packages/obspy/core/util/misc.py", line 214, in
loadtxt(np.array([0]), ndmin=1)
File "/Users/jaeyeolyang/anaconda3/envs/ispaq/lib/python3.8/site-packages/numpy/lib/npyio.py", line 1086, in loadtxt
ncols = len(usecols or split_line(first_line))
File "/Users/jaeyeolyang/anaconda3/envs/ispaq/lib/python3.8/site-packages/numpy/lib/npyio.py", line 977, in split_line
line = line.split(comment, 1)[0]
AttributeError: 'numpy.int64' object has no attribute 'split'

Limit / specify number of threads that ispaq uses

Thanks a lot for designing, implementing and making ISPAQ available here on github - it's proven very useful for computing and managing data quality metrics for a database that I'm working on!

I would like to report one point regarding ispaq's resource usage: As a lot of the operations in ispaq run on a single core, I am right now starting a few runs of ispaq in parallel from the terminal. But while computing some of the basic metrics, probably within some python libraries (numpy?), ispaq uses all available cores at once and hence slows down the system for all users (and then I get an understandable complaint by the system admin...).

I still don't fully understand which function calls use parallelization as I didn't see any explicit parallelization in ispaq, but for now I was able to limit each run to one thread by setting the following environment variables at the very start of ispaq.py:

import os
os.environ["OMP_NUM_THREADS"] = "1"
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["MKL_NUM_THREADS"] = "1"
os.environ["NUMEXPR_NUM_THREADS"] = "1"
os.environ["VECLIB_MAXIMUM_THREADS"] = "1"

I hope this can help others who run ispaq on a system that they share with other users. Maybe there could be a command line option to limit the number of cores used by ispaq?

install failing in line 160 of ispaq.py

when I am installing ISPAQ, I am getting an error due to applying the .tolist() on rpy2.robjects.vectors.StrVector object. This is breaking the code every time I run ./run_ispaq.py.

Traceback (most recent call last):
File "./run_ispaq.py", line 15, in
main()
File "/home/sjohn/AON_PROJECT/ispaq/ispaq/ispaq.py", line 160, in main
installed_names = ro.conversion.rpy2py(r_installed.rownames).tolist()
AttributeError: 'StrVector' object has no attribute 'tolist'

could you please tell me how to resolve this?

Thank you

Miniconda Installation error

On a Debian linux I have the following error

conda create -n ispaq --file ispaq-explicit-spec-file.txt

r-r6-2.1.1-r3. 100% |################################| Time: 0:00:00 181.41 kB/s
numpy-1.11.1-p 100% |################################| Time: 0:00:00   3.40 MB/s
r-lattice-0.20 100% |################################| Time: 0:00:01 602.97 kB/s
singledispatch 100% |################################| Time: 0:00:00  22.14 MB/s
jpeg-8d-2.tar. 100% |################################| Time: 0:00:00 629.80 kB/s
r-cluster-2.0. 100% |################################| Time: 0:00:00   1.01 MB/s
libxslt-1.1.28 100% |################################| Time: 0:00:00   1.08 MB/s
r-dplyr-0.4.3- 100% |################################| Time: 0:00:01   1.40 MB/s
curl-7.49.0-1. 100% |################################| Time: 0:00:00  44.56 MB/s
r-dbi-0.3.1-r3 100% |################################| Time: 0:00:00   2.14 MB/s
scipy-0.18.0-n 100% |################################| Time: 0:00:01   9.66 MB/s
r-spatial-7.3_ 100% |################################| Time: 0:00:00 313.62 kB/s
icu-56.1-4.tar 100% |################################| Time: 0:00:02   8.83 MB/s
mkl-11.3.3-0.t 100% |################################| Time: 0:00:02  51.39 MB/s
jbig-2.1-0.tar 100% |################################| Time: 0:00:00  42.59 MB/s
sqlalchemy-1.0 100% |################################| Time: 0:00:00   2.34 MB/s
r-rcurl-1.95_4 100% |################################| Time: 0:00:00   1.01 MB/s
openssl-1.0.2h 100% |################################| Time: 0:00:01   3.03 MB/s
r-boot-1.3_17- 100% |################################| Time: 0:00:00 832.36 kB/s
r-evaluate-0.8 100% |################################| Time: 0:00:00  33.20 MB/s
r-rcpp-0.12.2- 100% |################################| Time: 0:00:00   4.82 MB/s
gsl-1.16-0.tar 100% |################################| Time: 0:00:00  13.19 MB/s
r-stringi-1.0_ 100% |################################| Time: 0:00:00  16.38 MB/s
xz-5.2.2-0.tar 100% |################################| Time: 0:00:00  48.07 MB/s
libxml2-2.9.4- 100% |################################| Time: 0:00:00   8.41 MB/s
python-2.7.12- 100% |################################| Time: 0:00:01   7.72 MB/s
r-git2r-0.11.0 100% |################################| Time: 0:00:00   1.36 MB/s
r-nnet-7.3_11- 100% |################################| Time: 0:00:00   1.27 MB/s
libpng-1.6.24- 100% |################################| Time: 0:00:00   1.45 MB/s
functools32-3. 100% |################################| Time: 0:00:00  26.33 MB/s
r-mime-0.4-r3. 100% |################################| Time: 0:00:00  27.96 MB/s
r-3.2.2-0.tar. 100% |################################| Time: 0:00:00   1.40 MB/s
matplotlib-1.5 100% |################################| Time: 0:00:01   3.82 MB/s
r-quadprog-1.5 100% |################################| Time: 0:00:00  59.06 kB/s
libiconv-1.14- 100% |################################| Time: 0:00:00   1.59 MB/s
python-dateuti 100% |################################| Time: 0:00:00   1.02 MB/s
libgcc-4.8.5-1 100% |################################| Time: 0:00:00   1.70 MB/s
r-stringr-1.0. 100% |################################| Time: 0:00:00   1.02 MB/s
r-rstudioapi-0 100% |################################| Time: 0:00:00  30.35 MB/s
r-devtools-1.9 100% |################################| Time: 0:00:00 870.02 kB/s
obspy-1.0.3-py 100% |################################| Time: 0:00:02   9.60 MB/s
r-magrittr-1.5 100% |################################| Time: 0:00:00   1.06 MB/s
requests-2.10. 100% |################################| Time: 0:00:00   7.17 MB/s
r-mgcv-1.8_9-r 100% |################################| Time: 0:00:00   5.91 MB/s
freetype-2.6.3 100% |################################| Time: 0:00:00   4.89 MB/s
r-foreign-0.8_ 100% |################################| Time: 0:00:00   2.94 MB/s
r-curl-0.9.4-r 100% |################################| Time: 0:00:00   3.28 MB/s
r-class-7.3_14 100% |################################| Time: 0:00:00  48.81 MB/s
r-base-3.2.2-0 100% |################################| Time: 0:00:02   8.70 MB/s
decorator-4.0. 100% |################################| Time: 0:00:00  20.61 MB/s
libtiff-4.0.6- 100% |################################| Time: 0:00:00  53.38 MB/s
r-codetools-0. 100% |################################| Time: 0:00:00  32.21 MB/s
r-xml2-0.1.2-r 100% |################################| Time: 0:00:00  53.54 MB/s
r-matrix-1.2_2 100% |################################| Time: 0:00:00  10.40 MB/s
r-memoise-0.2. 100% |################################| Time: 0:00:00  26.06 MB/s
cycler-0.10.0- 100% |################################| Time: 0:00:00  23.31 MB/s
pip-8.1.2-py27 100% |################################| Time: 0:00:00   2.06 MB/s
pandas-0.18.1- 100% |################################| Time: 0:00:00  66.44 MB/s
r-kernsmooth-2 100% |################################| Time: 0:00:00 260.91 kB/s
r-bitops-1.0_6 100% |################################| Time: 0:00:00 275.45 kB/s
r-bh-1.58.0_1- 100% |################################| Time: 0:00:01   4.51 MB/s
future-0.15.2- 100% |################################| Time: 0:00:00   7.63 MB/s
r-recommended- 100% |################################| Time: 0:00:00   1.47 MB/s
r-rversions-1. 100% |################################| Time: 0:00:00 196.33 kB/s
lxml-3.6.1-py2 100% |################################| Time: 0:00:00   1.19 MB/s
r-jsonlite-0.9 100% |################################| Time: 0:00:00   1.21 MB/s
r-httr-1.0.0-r 100% |################################| Time: 0:00:00   3.89 MB/s
r-pracma-1.8.8 100% |################################| Time: 0:00:00   6.24 MB/s
pcre-8.39-0.ta 100% |################################| Time: 0:00:00 289.48 kB/s
r-signal-0.7_6 100% |################################| Time: 0:00:00 617.51 kB/s
r-roxygen2-5.0 100% |################################| Time: 0:00:00 784.10 kB/s
pyparsing-2.1. 100% |################################| Time: 0:00:00   1.14 MB/s
r-nlme-3.1_122 100% |################################| Time: 0:00:00   2.55 MB/s
r-xml-3.98_1.3 100% |################################| Time: 0:00:00   4.35 MB/s
setuptools-25. 100% |################################| Time: 0:00:00 869.46 kB/s
r-assertthat-0 100% |################################| Time: 0:00:00 372.91 kB/s
ncurses-5.9-8. 100% |################################| Time: 0:00:00   1.86 MB/s
r-survival-2.3 100% |################################| Time: 0:00:00   6.40 MB/s
pytz-2016.6.1- 100% |################################| Time: 0:00:00 719.78 kB/s
r-digest-0.6.8 100% |################################| Time: 0:00:00 621.07 kB/s
r-brew-1.0_6-r 100% |################################| Time: 0:00:00 914.81 kB/s
r-rpart-4.1_10 100% |################################| Time: 0:00:00   1.56 MB/s
r-lazyeval-0.1 100% |################################| Time: 0:00:00  40.14 MB/s
r-mass-7.3_45- 100% |################################| Time: 0:00:00   4.29 MB/s
rpy2-2.7.8-py2 100% |################################| Time: 0:00:00   3.08 MB/s
r-whisker-0.3_ 100% |################################| Time: 0:00:00  41.36 MB/s
ERROR conda.core.link:_execute_actions(330): An error occurred while installing package 'defaults::openssl-1.0.2h-2'.
LinkError: post-link failed for: defaults::openssl-1.0.2h-2

Attempting to roll back.



LinkError: post-link failed for: defaults::openssl-1.0.2h-2

Uncorrected PSDs

A request came in for the ability to produce PSDs for data that does not have response information available.

FDSN ws assumes trimmed data output

Hello,

When using FDSN dataselect as data source, it seems the code is expecting data trimmed to the nearest sample. I noted that trying to perform orientationCheck and crossTalk metrics, where the calculation was not possible because of different stream length returned for each channel.

However, the FDSN 1.1 webservice specification states that the dataselect shoud return samples starting on or after the starttime, and ending on or before the endtime.
http://www.fdsn.org/webservices/FDSN-WS-Specifications-1.1.pdf

Using Obspy get_waveform method instead of the R getDataselect, I was able to download the data, and trim them to the nearest common data section.
However, I lost the data quality flags in the process.

Otherwise, very interesting and great tool !!

Jean-Marie.

Cannot retrieve data from FDSN web service other than IRIS

HI,

I am trying to retrieve and run data quality metrics on data downloaded by INGV, but I get several error messages. To reproduce:

./run_ispaq.py -M sample_mean -S IV.VAGA.*.HH?  --starttime 2018-08-14 --endtime 2018-08-15 --dataselect_url INGV --station_url INGV --event_url INGV --output csv --csv_dir ./test_out --log-level DEBUG

Log file:

2022-12-05 14:09:56 - INFO - Running ISPAQ version 3.0.1 on Mon Dec  5 14:09:56 2022
2022-12-05 14:09:58 - DEBUG - Creating UserRequest ...
2022-12-05 14:09:58 - DEBUG - User request initialization
2022-12-05 14:09:58 - DEBUG - preferences_file ./preference_files/default.txt
2022-12-05 14:09:58 - DEBUG - Explicit metric detected 'sample_mean'
2022-12-05 14:09:58 - DEBUG - Explicit SNCL detected 'IV.VAGA.*.HH?'
2022-12-05 14:09:58 - DEBUG - Creating Concierge ...
2022-12-05 14:09:58 - DEBUG - starttime 2018-08-14T00:00:00, endtime 2018-08-15T00:00:00
2022-12-05 14:09:58 - DEBUG - metric_names ['sample_mean']
2022-12-05 14:09:58 - DEBUG - sncl_patterns ['IV.VAGA.*.HH?']
2022-12-05 14:09:58 - DEBUG - dataselect_url http://webservices.ingv.it
2022-12-05 14:09:58 - DEBUG - dataselect_type fdsnws
2022-12-05 14:09:58 - DEBUG - station_url http://webservices.ingv.it
2022-12-05 14:09:58 - DEBUG - event_url http://webservices.ingv.it
2022-12-05 14:09:58 - DEBUG - resp_dir None
2022-12-05 14:09:58 - DEBUG - output csv
2022-12-05 14:09:58 - DEBUG - db_name ispaq.db
2022-12-05 14:09:58 - DEBUG - csv_dir [...]/ISPAQ/ispaq/test_out
2022-12-05 14:09:58 - DEBUG - pdf_dir [...]/ISPAQ/ispaq/PDFs
2022-12-05 14:09:58 - DEBUG - psd_dir [...]/ISPAQ/ispaq/PSDs
2022-12-05 14:09:58 - DEBUG - pdf_type ['text', 'plot']
2022-12-05 14:09:58 - DEBUG - pdf_interval ['daily', 'aggregated']
2022-12-05 14:09:58 - DEBUG - plot_include ['legend', 'colorbar']
2022-12-05 14:09:58 - DEBUG - sigfigs 6
2022-12-05 14:09:58 - DEBUG - sncl_format N.S.L.C
2022-12-05 14:09:58 - DEBUG - Inside simple business logic ...
2022-12-05 14:09:58 - DEBUG - channelFilter .*
2022-12-05 14:09:58 - DEBUG - read FDSN station web services http://webservices.ingv.it for IV,VAGA,*,HH?,2018.226,2018.227
2022-12-05 14:09:58 - DEBUG - The parameter 'matchtimeseries' is not supported by the service.
2022-12-05 14:09:58 - INFO - No stations found for IV.VAGA.*.HH?
2022-12-05 14:09:58 - INFO - No available waveforms for 2018-08-14 matching ['IV.VAGA.*.HH?']
2022-12-05 14:09:58 - DEBUG - skipping 2018-08-14 with no available data
2022-12-05 14:09:58 - WARNING - "simple" metric calculation generated zero metrics
2022-12-05 14:09:58 - INFO - No simple metrics were calculated
2022-12-05 14:09:58 - INFO - ALL FINISHED!

Thank you in advance for your help!

Best,
Angela