nasa / rtretrievalframework Goto Github PK

Just a quick minor issue.

http://oco.jpl.nasa.gov/science/publications/

Responds with Unknown Host in browser:

$ dig oco.jpl.nasa.gov | grep status
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 28402

$ nslookup oco.jpl.nasa.gov
** server can't find oco.jpl.nasa.gov: NXDOMAIN

Hi,
The documentation says that the model needs a priori CO2 profiles, I want to ask where I can download the CO2 file, and how to add the file into the model.

Thanks,
Jin Zhe

Will this software be updated again?
What's the relationship between RtRetrievalFramework and ReFRACtor?
Is ReFRACtor a better choice?

Can I get the forward model results instead of the ultimate retrieval results?

Hi,

I am still working this version because meet some environment errors during compiling ReFRACtor on our server...

For this version, I have passed all test run, now I am trying to work on the python wrapper.

I following the instruction here http://nasa.github.io/RtRetrievalFrameworkDoc/python_wrappers.html.

from full_physics import *

But got "NameError: name 'L2FpConfigurationLua' is not defined",

I check the full_physics directory, (for me is /home/thl/04.OCO2/RtRetrievalFramework/build_fp/install/lib/python2.7/site-packages/full_physics
). There are no function called 'L2FpConfigurationLua'.

However, I can find this function under "full_physics_swig" directory,

Then I try to import full_physics_swig in python, then got following error:

import full_physics_swig as fps
Traceback (most recent call last):
File "", line 1, in
File "/home/thl/04.OCO2/RtRetrievalFramework/build_fp/install/lib/python2.7/site-packages/full_physics_swig/init.py", line 9, in
exec('from .%s import *' % mname)
File "", line 1, in
File "/home/thl/04.OCO2/RtRetrievalFramework/build_fp/install/lib/python2.7/site-packages/full_physics_swig/init.py", line 9, in
exec('from .%s import *' % mname)
File "", line 1, in
File "/home/thl/04.OCO2/RtRetrievalFramework/build_fp/install/lib/python2.7/site-packages/full_physics_swig/oco_noise_model.py", line 32, in
_oco_noise_model = swig_import_helper()
File "/home/thl/04.OCO2/RtRetrievalFramework/build_fp/install/lib/python2.7/site-packages/full_physics_swig/oco_noise_model.py", line 24, in swig_import_helper
import _oco_noise_model
ImportError: No module named _oco_noise_model

Any suggestions?

Thanks,
Tianhao

Hi,
I attempt to change the input data of CO2 apriori and use my own data. I find in the class PressureLevelInput that the input data must be defined on 20 fixed pressure levels (1hPa, 70hPa ,..., 1000hPa, 1050hPa).
My first question is that sometimes the surface pressure is lower than 1050hPa, for example, 900hPa, is it still necessary to give values of levels higher than 900hPa, and will the values of these levels have a effect on the final results?
My second question is that, in the class Pressure, it says the pressure levels will potentially vary as we do a retrieval, how do they vary?
My third question is that what's the function of class PressureSigma, does the sigma coordinate be used in the retrieval process? And in the pressure section of the result, there are two matrices, named a and b respectively, what do the two matrices represent?
a:
20
[ 0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 ]
b:
20
[ 0.0001 0.0526316 0.105263 0.157895 0.210526 0.263158 0.315789
0.368421 0.421053 0.473684 0.526316 0.578947 0.631579 0.684211
0.736842 0.789474 0.842105 0.894737 0.947368 1 ]

Hello, according to the document, I used the new OCO-2 data and Met data to complete the inversion of CO2 data based on RtRetrievalFramwork, but I found that the generated file（l2_***.h5） does not contain longitude and latitude attributes. I don't know how to solve this problem? I look forward to your reply and wish you good luck

The installation has been completed, and the inversion results of a single sounding have been run. How can we fully run all sounding inversion CO2 in the future? there is my email [email protected]

Hi,

I tried to make unit test and it said "it could not find the file /merra/MERRA_Composite_Selection_M02_02A.hdf" in the attachment. Do you know where can I get the merra_composite data? Or Could you send me an example data for unit test?

Best wishes,

Lili

according to the user guide, when doing the step python wrapper, there is an error:

how can i deal with it?

Hi,

I got following error message when I run fast_check

Running 417 test cases...

0% 10 20 30 40 50 60 70 80 90 100%
|----|----|----|----|----|----|----|----|----|----|
**********unknown location(0): fatal error: in "forward_model_spectral_grid/low_resolution_grid": FullPhysics::Exception: Lua error: std::exception: 'While trying to open file '/groups/algorithm/l2_fp/absco/v3.3.0/lowres/co2_v3.3.0-lowres.hdf' a HDF 5 Exception thrown:
H5Fopen failed'

lib/Interface/forward_model_spectral_grid_test.cc(68): last checkpoint: "low_resolution_grid" fixture entry.
unknown location(0): fatal error: in "forward_model_spectral_grid/high_resolution_grid": FullPhysics::Exception: Lua error: std::exception: 'While trying to open file '/groups/algorithm/l2_fp/absco/v3.3.0/lowres/co2_v3.3.0-lowres.hdf' a HDF 5 Exception thrown:
H5Fopen failed'

What may cause this error?
Thanks,
Tianhao

Hi,

I have successfully installed the B8.1.00 version model on the server. The problem is I can only run the forward model, because the memory needed to run the whole retrieval model is too large. The memory of my server is 48GB. When I try to run the whole retrieval model, memory overflow will happen. Even if I only run the forward model, the whole memory will be occupied.

I want to ask how much memory is needed for the model? If the memory requirement is not too large, what may be the cause of my problem? Is there a problem during the compilation process? My compiling command is /path/to/source/configure THIRDPARTY=build --with-absco=/path/ABSCO --with-merra=/path/MERRA. And I run the model by ./l2_fp_job.sh single_one_sounding_id.

Thanks,
JinZhe

Looks like this is for all the thirdparty libs, by this:

%configure
--with-lidort-3.8=build
--with-2stream=build

$rpm -qa libtool
libtool-2.4.7-7.fc35.x86_64

/bin/sh ./libtool --tag=FC --mode=compile -DPACKAGE_NAME="full_physics" -DPACKAGE_TARNAME="full_physics" -DPACKAGE_VERSION="11.0.05" -DPACKAGE_STRING="full_physics\ 11.0.05" -DPACKAGE_BUGREPORT="" -DPACKAGE_URL="" -DPACKAGE="full_physics" -DVERSION="11.0.05" -DHAVE_STDIO_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_STRINGS_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_UNISTD_H=1 -DSTDC_HEADERS=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=".libs/" -DHAVE_CXX11=1 -DHAVE_BOOST=/**/ -DHAVE_BZLIB=/**/ -DHAVE_GSL=/**/ -DHAVE_NCURSES_H=1 -DHAVE_LUA=/**/ -DHAVE_PTHREAD=1 -DHAVE_FEENABLEEXCEPT=1 -DHAVE_FEENABLEEXCEPT=1 -I. -I./lib/Support -I/usr/include -I/usr/include -DBOOST_TEST_DYN_LINK -I/usr/include -I/usr/include -I./lib/Support -I./lib/Interface -I./lib/Implementation -I./lib/FixedLevel -I./lib/Deprecated -I./lib/RegisterOutput -I./lib/Lua -I/usr/include -I./lib -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -I/usr/lib64/gfortran/modules -O -c -o thirdparty/lidort-3.8/lidort_def/liblidort_3_8_3_la-lidort_inputs_def.lo test -f 'thirdparty/lidort-3.8/lidort_def/lidort_inputs_def.F90' || echo './'thirdparty/lidort-3.8/lidort_def/lidort_inputs_def.F90
Usage: /home/ss/rpmbuild/BUILD/RtRetrievalFramework-B11.0.05/libtool [OPTION]... [MODE-ARG]...
Try 'libtool --help' for more information.
libtool: error: unrecognised option: '-DPACKAGE_NAME="full_physics"'
make: *** [Makefile:8144: thirdparty/lidort-3.8/lidort_def/liblidort_3_8_3_la-lidort_inputs_def.lo] Error 1
make: *** Waiting for unfinished jobs....
/bin/sh ./libtool --tag=FC --mode=compile -DPACKAGE_NAME="full_physics" -DPACKAGE_TARNAME="full_physics" -DPACKAGE_VERSION="11.0.05" -DPACKAGE_STRING="full_physics\ 11.0.05" -DPACKAGE_BUGREPORT="" -DPACKAGE_URL="" -DPACKAGE="full_physics" -DVERSION="11.0.05" -DHAVE_STDIO_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_STRINGS_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_UNISTD_H=1 -DSTDC_HEADERS=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=".libs/" -DHAVE_CXX11=1 -DHAVE_BOOST=/**/ -DHAVE_BZLIB=/**/ -DHAVE_GSL=/**/ -DHAVE_NCURSES_H=1 -DHAVE_LUA=/**/ -DHAVE_PTHREAD=1 -DHAVE_FEENABLEEXCEPT=1 -DHAVE_FEENABLEEXCEPT=1 -I. -I./lib/Support -I/usr/include -I/usr/include -DBOOST_TEST_DYN_LINK -I/usr/include -I/usr/include -I./lib/Support -I./lib/Interface -I./lib/Implementation -I./lib/FixedLevel -I./lib/Deprecated -I./lib/RegisterOutput -I./lib/Lua -I/usr/include -I./lib -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -I/usr/lib64/gfortran/modules -O -c -o thirdparty/lidort-3.8/lidort_def/liblidort_3_8_3_la-lidort_lin_sup_brdf_def.lo test -f 'thirdparty/lidort-3.8/lidort_def/lidort_lin_sup_brdf_def.F90' || echo './'thirdparty/lidort-3.8/lidort_def/lidort_lin_sup_brdf_def.F90
Usage: /home/ss/rpmbuild/BUILD/RtRetrievalFramework-B11.0.05/libtool [OPTION]... [MODE-ARG]...
Try 'libtool --help' for more information.
libtool: error: unrecognised option: '-DPACKAGE_NAME="full_physics"'

Ideas here?

Hello, I have encountered a problem. When executing, I reported an error as shown in the figure below. I don't know how to solve it. If you have time, please solve it. In addition, what are the requirements for the format of oco2 data and ec data? I hope to get your answer

Hi ：
According to the user manual, I have compiled successfully and generated the L2 FP file. I want to use the new oco-2 data to run the program. Do you have any version and other requirements for downloading oco-2 L1B data? Where is the ECMWF data obtained and interpolation processing is required after downloading? Where does absco data need to be placed in the program? Does GOSAT data need noise removal? Is there a specific tutorial dealing with GOSAT and oco-2 to generate L2 products?
Looking forward to your reply
thank you

I see 2 or arguably 3 thirdparty libraries which can only be built here to support the end package so you must run configure using:
--with-lidort-3.8=build
--with-2stream=build

Build own LIDORT 3.8 library: yes
Build own 2stream library: yes

2stream library ( aka libtwostream )
lidort-3.8 library
radiant library

Now in each of the other library cases i.e. gsl can be found elsewhere and installed separately meaning no need to build.
When looking these up, the only references I find are for lidort:

http://www.rtslidort.com/mainprod_lidort.html
https://web.gps.caltech.edu/~vijay/Papers/RT%20Models/lidort_3p6_userguide_v7_28sep2012.pdf?fbclid=IwAR0PTturfB0JJLSe-QP126q7OA2mI1Uzwdsq5Zit9Rad312rs0Jlm7dLAos

Q is, where are these maintained outside of this repo elsewhere as independent libraries like that of gsl? If anyone knows, so I can work to provide an RPM along with those other libraries.

After finding this software I was attempting to build an RPM from source to enable easy installation.
Standard autotools in use with gfortan compiler.
Read all the supporting compile docs, (using Fedora 38) I encounter these issues, note all RPM dependencies are met:

$ rpm -qa *gfortran
libgfortran-13.2.1-4.fc40.x86_64
gcc-gfortran-13.2.1-4.fc40.x86_64

$ rpm -qa ruby* | sort
ruby-3.2.2-183.fc40.x86_64
ruby-default-gems-3.2.2-183.fc40.noarch
ruby-devel-3.2.2-183.fc40.x86_64
ruby-libs-3.2.2-183.fc40.x86_64
rubygem-io-console-0.6.0-183.fc40.x86_64
rubygem-json-2.6.3-204.fc39.x86_64
rubygem-narray-0.6.1.1-28.fc39.x86_64
rubygem-psych-5.0.1-183.fc40.x86_64
rubygem-rdoc-6.5.0-183.fc40.noarch
rubygems-3.4.10-183.fc40.noarch
rubygems-devel-3.4.10-183.fc40.noarch
rubypick-1.1.1-19.fc39.noarch

./config/depf90:38:in block in <main>': undefined method exists?' for File:Class (NoMethodError)

fname = srcdir + "/" + fname unless(File.exists?(fname))
                                        ^^^^^^^^

Did you mean? exist?
from ./config/depf90:28:in each' from ./config/depf90:28:in

fname = srcdir + "/" + fname unless(File.exists?(fname))
                                        ^^^^^^^^

Did you mean? exist?
from ./config/depf90:28:in each' from ./config/depf90:28:in

Hi,
RtRetrievalFramework

or fname in flist: 14 with open(fname) as f: 15 t = cPickle.load(f) 16 xco2_a = t["XCO2_a"] … 23 with open("my_cov.pkl") as f: 24 dist_cov = cPickle.load(f) 25

I would like to report a remote code execution potential vulnerability in Singledop. Pickle module enables binary serialization and loading of Python datatypes and any user supplied sample file can lead to remote code execution on any researches machine processing a serialized file.

Attack binary a valid dop file:

mona@mona-virtual-machine:~/Downloads/SingleDop$ cat t_file cos popen (S'uname -a' tRp100 0c__builtin__ getattr (c__builtin__ file S'read' tRp101 0c__builtin__ apply (g101 (g100 I1000 ltRp102 0c__builtin__ getattr (c__builtin__ file S'close' tRp103 0c__builtin__ apply (g103 (g100 ltRp104 0g102 .
The pickle module is not intended to be secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.

Please let me know if you have any questions. You can also reach back to me at [email protected]
Thanks

Email from Peter:

Hi Mike

My new PhD student, George Cann (in cc:) started a project 6 weeks ago
sponsored by the UK Space Agency on developing retrieval code for 13CH4
isotopologue retrieval in the Martian atmosphere using data from
instruments onboard the ESA Trace Gas Orbiter.

I previously collaborated with Hartmut Boesch at Leicester using his
code on CEMS for 13CH4 from GOSAT data. Hartmut wrote part of this code
when he was a NRC fellow many years ago at JPL.

For George's PhD project, we want to combine your OCO-2 retrieval code
with the Oxford Reference Forward Model (ORFM) to develop a standalone
13CH4 retrieval system for solar occultation data.

George downloaded the software from the github and has been working on
the hints and tips provided at

http://nasa.github.io/RtRetrievalFrameworkDoc/developer_information.html

However, he has hit a brick-wall after several weeks of trying to
compile and link this software on MacOS. Unfortunately, none of the
developer's notes say what the original target system was so it may be
that this code was developed on RHEL or PC Windows and was never
intended to run on MacOS?

George has access to RHEL blade systems so running on linux might be
much easier. However, I wanted to check first and also to ask if you
would be prepared to answer his key questions so he can start to make
some progress?