Comments (10)
srcarter3
commented on June 2, 2024
Hi there,
Our AWIPS does have the ability to decode Himawari data.
What I would suggest is to first get a file or two for manual testing and use our manual endpoint: /awips2/edex/data/manual/
If the data gets decoded you should see something new in the /awips2/edex/data/hdf5/ directory.
You should also be able to "watch" the ingest log file when you copy the file into the manual endpoint:
tail -f /awips2/edex/logs/edex-ingex-[today's date].log
And that should give you some indication if it was decoded.
I have a suspicion it might not decode right away, because it looks like the file name/header may or may not be the same as what we expect.
If it does not decode, you'll want to go edit this file:
/awips2/edex/data/utility/common_static/base/distribution/goesr.xml
At the bottom of that file you'll see a section for Himawari Imagery that is matching with regular expressions for:
<regex>^OR_HFD</regex>
<regex>^H8FD.B</regex>
You can try editing those or adding another <regex> </regex> line to match on your data.
If the data gets ingested, you should be able to see it in CAVE in the Product Browser.
If that all works, then I'd suggest writing a script that retrieves data in your data source and pushes it to your qpid queue for EDEX to decode automatically. You can see a brief example of this in our script /awips2/ldm/dev/checkFileTime.sh at line 43 -- this is where it puts the new file into the qpid queue for decoding.
After you have all that working, you may want to modify your CAVE menus for easier display access. Some information for that can be found in our documentation website.
Let me know if that all makes sense?
from awips2.
GarryLai
commented on June 2, 2024
Hi there,
Our AWIPS does have the ability to decode Himawari data.
What I would suggest is to first get a file or two for manual testing and use our manual endpoint:
/awips2/edex/data/manual/If the data gets decoded you should see something new in the
/awips2/edex/data/hdf5/directory.You should also be able to "watch" the ingest log file when you copy the file into the manual endpoint:
tail -f /awips2/edex/logs/edex-ingex-[today's date].logAnd that should give you some indication if it was decoded.
I have a suspicion it might not decode right away, because it looks like the file name/header may or may not be the same as what we expect. If it does not decode, you'll want to go edit this file:
/awips2/edex/data/utility/common_static/base/distribution/goesr.xmlAt the bottom of that file you'll see a section for Himawari Imagery that is matching with regular expressions for:<regex>^OR_HFD</regex> <regex>^H8FD.B</regex>You can try editing those or adding another
<regex> </regex>line to match on your data.If the data gets ingested, you should be able to see it in CAVE in the Product Browser.
If that all works, then I'd suggest writing a script that retrieves data in your data source and pushes it to your qpid queue for EDEX to decode automatically. You can see a brief example of this in our script
/awips2/ldm/dev/checkFileTime.shat line 43 -- this is where it puts the new file into the qpid queue for decoding.After you have all that working, you may want to modify your CAVE menus for easier display access. Some information for that can be found in our documentation website.
Let me know if that all makes sense?
It seems that EDEX only support NETCDF format. Where I can download NETCDF format of Himawari data?
ERROR 2024-02-07 12:08:44,635 3990 [Ingest.GOESR-1] GoesrNetcdfDecoder: Unable to open the file HS_H09_20240207_0820_B01_FLDK_R10_S0110.DAT for splitting.
java.io.IOException: java.io.IOException: Cant read /tmp/sbn/manual/goesr/20240207/12/HS_H09_20240207_0820_B01_FLDK_R10_S0110.DAT: not a valid CDM file.
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:427) ~[cdm-4.6.10.jar:4.6.10]
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:394) ~[cdm-4.6.10.jar:4.6.10]
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:381) ~[cdm-4.6.10.jar:4.6.10]
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:369) ~[cdm-4.6.10.jar:4.6.10]
at com.raytheon.uf.edex.netcdf.decoder.AbstractNetcdfDecoder.split(AbstractNetcdfDecoder.java:151) ~[com.raytheon.uf.edex.netcdf.jar:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.apache.camel.support.ObjectHelper.invokeMethodSafe(ObjectHelper.java:372) ~[camel-support-3.7.0.jar:3.7.0]
at org.apache.camel.component.bean.MethodInfo.invoke(MethodInfo.java:489) ~[camel-bean-3.7.0.jar:3.7.0]
at org.apache.camel.component.bean.MethodInfo$1.doProceed(MethodInfo.java:311) ~[camel-bean-3.7.0.jar:3.7.0]
at org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:281) ~[camel-bean-3.7.0.jar:3.7.0]
at org.apache.camel.component.bean.AbstractBeanProcessor.process(AbstractBeanProcessor.java:145) ~[camel-bean-3.7.0.jar:3.7.0]
at org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager.process(DefaultAsyncProcessorAwaitManager.java:83) ~[camel-base-engine-3.7.0.jar:3.7.0]
at org.apache.camel.support.AsyncProcessorSupport.process(AsyncProcessorSupport.java:41) ~[camel-support-3.7.0.jar:3.7.0]
at org.apache.camel.language.bean.BeanExpression.invokeBean(BeanExpression.java:347) ~[camel-bean-3.7.0.jar:3.7.0]
at org.apache.camel.language.bean.BeanExpression.evaluate(BeanExpression.java:202) ~[camel-bean-3.7.0.jar:3.7.0]
at org.apache.camel.language.bean.BeanExpression.evaluate(BeanExpression.java:214) ~[camel-bean-3.7.0.jar:3.7.0]
at org.apache.camel.processor.Splitter.createProcessorExchangePairs(Splitter.java:150) ~[camel-core-processor-3.7.0.jar:3.7.0]
at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:276) ~[camel-core-processor-3.7.0.jar:3.7.0]
at org.apache.camel.processor.Splitter.process(Splitter.java:145) ~[camel-core-processor-3.7.0.jar:3.7.0]
at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:312) ~[camel-base-engine-3.7.0.jar:3.7.0]
at org.apache.camel.processor.Pipeline$PipelineTask.run(Pipeline.java:90) ~[camel-core-processor-3.7.0.jar:3.7.0]
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148) ~[camel-base-engine-3.7.0.jar:3.7.0]
at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:60) ~[camel-base-engine-3.7.0.jar:3.7.0]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:147) ~[camel-core-processor-3.7.0.jar:3.7.0]
at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:312) ~[camel-base-engine-3.7.0.jar:3.7.0]
at org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager.process(DefaultAsyncProcessorAwaitManager.java:83) ~[camel-base-engine-3.7.0.jar:3.7.0]
at org.apache.camel.support.AsyncProcessorSupport.process(AsyncProcessorSupport.java:41) ~[camel-support-3.7.0.jar:3.7.0]
at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:130) ~[camel-jms-3.7.0.jar:3.7.0]
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:736) ~[spring-jms-5.3.20.jar:5.3.20]
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:696) ~[spring-jms-5.3.20.jar:5.3.20]
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:674) ~[spring-jms-5.3.20.jar:5.3.20]
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:331) ~[spring-jms-5.3.20.jar:5.3.20]
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:270) ~[spring-jms-5.3.20.jar:5.3.20]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1237) ~[spring-jms-5.3.20.jar:5.3.20]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1227) ~[spring-jms-5.3.20.jar:5.3.20]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1120) ~[spring-jms-5.3.20.jar:5.3.20]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
Caused by: java.io.IOException: Cant read /tmp/sbn/manual/goesr/20240207/12/HS_H09_20240207_0820_B01_FLDK_R10_S0110.DAT: not a valid CDM file.
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:825) ~[cdm-4.6.10.jar:4.6.10]
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:424) ~[cdm-4.6.10.jar:4.6.10]
... 41 common frames omitted
from awips2.
srcarter3
commented on June 2, 2024
Hi there,
I believe they are available from AWS:
https://noaa-himawari9.s3.amazonaws.com/index.html#AHI-L2-FLDK-Clouds/2024/02/07/0800/
from awips2.
GarryLai
commented on June 2, 2024
Hi there,
I believe they are available from AWS:
https://noaa-himawari9.s3.amazonaws.com/index.html#AHI-L2-FLDK-Clouds/2024/02/07/0800/
Same not working
WARN 2024-02-07 16:10:10,457 7974 [Ingest.GOESR-1] GoesrNetcdfDecoder: No valid records were found in file: AHI-CHGT_v1r1_h09_s202402070800203_e202402070809397_c202402070818204.nc
WARN 2024-02-07 16:15:55,130 7976 [Ingest.GOESR-1] GoesrNetcdfDecoder: No valid records were found in file: AHI-CPHS_v1r1_h09_s202402070800203_e202402070809397_c202402070818204.nc
WARN 2024-02-07 16:15:58,132 7978 [Ingest.GOESR-2] GoesrNetcdfDecoder: No valid records were found in file: AHI-CMSK_v1r1_h09_s202402070800203_e202402070809397_c202402070818204.nc
from awips2.
srcarter3
commented on June 2, 2024
Just as a test, can you ingest any current goes data successfully?
from awips2.
GarryLai
commented on June 2, 2024
Just as a test, can you ingest any current goes data successfully?
No, nothing found in /awips2/edex/data/hdf5/
from awips2.
GarryLai
commented on June 2, 2024
Where is the data source used in edex-cloud.unidata.ucar.edu?
from awips2.
srcarter3
commented on June 2, 2024
You have to pull the data or drop in a manual file from somewhere to see if your EDEX will decode GOES data.
I believe it is also available through AWS: https://registry.opendata.aws/noaa-goes/
Our data sources are from LDM feeds here in the United States.
Are you an academic institution? Or a private company? Or a weather enthusiast?
from awips2.
GarryLai
commented on June 2, 2024
I found that the correct file is at AHI-L2-FLDK-ISatSS, with the prefix OR_HFD.
Thanks for your helping.
BTW, I am a college student major in earth science from Taiwan.
from awips2.
srcarter3
commented on June 2, 2024
Okay great, I'm glad you were able to get Himawari data to work!
from awips2.
Related Issues (20)
- Failed to launch cave HOT 1
- Fresh install of EDEX server not ingesting any data HOT 9
- registry.xml in ~/ldm/etc directory has outdated NTP servers listed
- Maps Only Occupy Lower Left Quadrant of Pane HOT 1
- Error instantiating workbench: null HOT 1
- CentOS 8 compatibility HOT 13
- Fuzzy Text HOT 4
- Incorporate LDM version 6.14.3 into the next official release HOT 2
- Windows - Some models report "Unexpected array size" for GridResources/GridData HOT 1
- Issue with exporting .cmap HOT 3
- 2-Panel Layout positioning now horizontally split? HOT 4
- Windows 11 Launch Error. Cannot find 'C\Users\ ' HOT 1
- MRMS Rotation Tracks in Version 20 Beta & Legacy Satellite Menu HOT 2
- "No active IFPServer for site XXX" when using GFE on Windows HOT 1
- ingestGrib not working? HOT 2
- How to add surface / upper air data into EDEX? HOT 2
- ECMWF total precipitation not loaded HOT 8
- An overflow bug during prcessing MSL pressure HOT 4
- Compiling on non RHEL related distro HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from awips2.