mavensdc / pytplot Goto Github PK
View Code? Open in Web Editor NEWA python version of the IDL tplot libraries
License: MIT License
A python version of the IDL tplot libraries
License: MIT License
Inside spd_ui_loaded_data__dproc there is an insane amount of worth that goes into smoothing, involving multiple functions. Figure those out and implement those.
This should probably be an option in tplot_options. Need to make sure it works with both bokeh and pyqtgraph. With bokeh, we would just need to add something like " self.fig.xaxis.axis_label="insert x axis here" "
This is a testing issue
High pass filter I believe just calculates the running average at each point in time, and subtracts that from the value. So its kinda like subtracting the average, except the average is different at each point in time.
Need to take a look inside "thigh_pass_filter.pro" in order to more deeply understand.
This should utilize the HoverTime class in the previous ticket.
Basically, make a new script that, when run, will pop up a completely new window with a simple 1D plot. That plot will change depending on what time the user is hovering over. Specifically, for specplots, it will take the time you're hovered over plot the spec_bins vs the data values at that specific time.
Throw any comments back my way. Thanks!
Add a procedure for smoothing, or figure out which python routine already does it:
http://www.harrisgeospatial.com/docs/SMOOTH.html
(Implement pydl.smooth in tplot_math.derive)
If you have more than a day's worth of data, pyqtgraph only shows the day of the month on the time axis. It should be changed to show hh:mm:ss for much longer, perhaps only switching over after ~15-20 days.
Additionally, it would be great if we could ensure that new dates would always show up as YYYY-MM-DD. I suspect that might involve quite a bit of digging into the pg.AxisItem we are inheriting.
The code to be modified resides in pytplot/QtPlotter/CustomAxis/DateAxis.py
What the title says
We have the ability to make a new x axes, but I think it only works if it is linear. We can't do one where the axis looks like:
1 10 15 100
for example. Is this possible to fix?
Apparently a function that print out tplot into an ascii file is important enough to be demoed in the crib sheet. I guess we should add that too.
I've seen several scientists use a binned map. It looks kind of like a spectrogram plot, but the axes are lat and lon rather than bins and time. Can we get pytplot to do that?
https://agupubs.onlinelibrary.wiley.com/doi/full/10.1002/2015GL065366
Figure 4, chart F
Can our tool do that? Is power spectral density the same thing as the "power spectrum" seen in the other ticket?
If you call a tplot function and a pyqtgraph window pops up, it would be neat if we could specify
pytplot.options([the tvar name], "ylog", 1)
And have the displayed plot change itself automatically
This is the same thing as the "tsub_average" procedure in tplot
Can we specify a better way to handle overplot rather than saying store_date("hello", data=["TVar1", "TVar2"])?
I think a good way to go about it would be to link them together as "oplot"
By supplying "dy" to a store_data call, it will show error bars on the 1D plots
Investigate how the border is selected, and what we can do to change it
Okay I've actually worked through some of the details on this one. Here is how it is done in IDL: dpwrspec.pro
You have a 1D line TVar. You allow the user to specify 2 numbers, nboxpoints and nshiftpoints. They default to 256 and 128 respectively, so that's what I use the rest of this algorithm. But remember you can allow the user to select these.
So you take the first 256 data points, both x (time) and y (the data).
Recalculate y by subtracting the best linear fit of those 256 points, i.e. :
x = x - x[0]
c = np.polyfit(x, y, 1)
y_trend = c[0]*x + c[1]
y = y - y_trend
Then, calculate the "window" (I don't really know what this means) with:
w = scipy.signal.get_window("hanning", 256)
Then calculate the power spectrum of those 256 points:
f,pxx = scipy.signal.periodogram(y[0:256], window=w, detrend=lambda x : x)
Now, shift 128 points over, and do this again for the next 256 points, i.e. points number 128 to 384. And then shift 128 points again, and again, etc, until you are out of points to look at.
Then this data should be read into pytplot, where f is the spec bins, all of the different "pxx" are the data that fill out each column, and the times are at x[128], x[256], x[384], etc. So this will generate a multi-column dataframe from a dataframe that is just one column.
There is also some stuff with binning at the end of the function in IDL. Basically, it looks like a user can set a "bin_size" option, and average each column by bin_size at a time.
For example, if bin_size is 3, and there are 15 columns, then it averages together columns [1,2,3], [4,5,6], [7,8,9], [10,11,12], [13,14,15]. This makes a total of only 5 columns.
As an option:
-Allow users to select whether or not to subtract out the average fit line
-Average out each row of data to compress all the columns into one
-Not use the hanning window (just set window=None in the periodogram function)
Take a look at each of the main functions in pytplot and make sure the comments are accurate.
Feel free to update the material inside the "docs" folders of both pydivide and pytplot as well. They are probably horrifically out of date. I'll try to find a .doc version of that pydivide PDF i made forever ago.
The small tutorials are done in html. You can open those up in a text editor and modify them that way. It might look daunting at first if you haven't seen HTML code before, but its really just a way to format the text fancily kinda like LaTeX.
wav_data.pro exists to compute the wavelet transform of tplot variables. This only works on 1D data, so it will compute the wavelet transform for for each column. But 2D data is output. I have literally no idea what a wavelet transform is.
EDIT:
This looks incredibly complicated, I almost wonder if this is even worth it. We might want to wait for like money or something, because this could easily take 100 hours to deeply understand and port over into python.
There is a package to do the actual computing (https://pywavelets.readthedocs.io/en/latest/), I think we could leverage that one if needed. But who knows. This definitely at least requires more looking into.
When I run pydivide test scripts, I notice that the qt window will occasionally show up empty, and I have no idea why.
get_y_limits
zlog in options.py
ylog in options.py
del_data
cdf_to_tplot
_get_figure in generate.py (both Bokeh and Qt)
Can we simplify these?!
In regular IDL tplot, I think we can make text labels appear at an arbitrary x-y coordinate. Is that possible in bokeh/pyqtgraph?
If so, feel free to add a dictionary object to TVar that contains a list of labels that will be plotted inside the plot. Then, you can add a new function to all those TVarFigures called _add_text() that looks through that dictionary and writes things in the plots.
Like cdf_to_tplot
tplot_math.fn_interp vs. tplot_math.resample? Not sure what the difference is and why resample works.
This is one of those things that sounds good in my head, but i have no idea if it can be implemented.
Clicking near a point on a map plot could create a timebar as a marker on another plot, just to kind of show where that time point is located on other types of visualizations
Here is the IDL code. Function dimen1 figures out the size of dimension 1 for data, i.e. the size of the time dimension. dimen2 figures out the size of dimension 2, i.e. the number of columns in the dataframe.
;+
; Procedure: CLEAN_SPIKES, 'name'
;
; Purpose: Simple routine to remove spikes from tplot data.
;
; Author: unknown, Probably Frank Marcoline
;
; Keywords:
; display_object = Object reference to be passed to dprint for output.
;
; jmm, 4-jul-2007, made to work for negative data, by adding absolute values
; jmm, 30-jul-2007, Added more error checking
; jmm, 15-sep-2009, documentation test
;$LastChangedBy: $
;$LastChangedDate: $
;$LastChangedRevision: $
;$URL: $
;-
pro clean_spikes,name,new_name=new_name,nsmooth=ns,thresh=ft, display_object=display_object
get_data, name, data = d, dlim = dlim, lim = lim
If(size(d, /type) Eq 8) Then Begin
ds = d
if not keyword_set(ns) then ns = 3
ns = (fix(ns)/2)2 + 1
nd1 = dimen1(d.y)
If(nd1 Le 2ns) Then Begin
msg = name+': Not enough data for smoothing, to despike, returning undespiked data'
dprint, msg , display_object=display_object
if not keyword_set(new_name) then new_name = name+'_cln'
store_data, new_name, data = d, dlim = dlim, lim = lim
Endif Else Begin
if not keyword_set(ft) then ft = 10.
ft = float(ft)
nd2 = dimen2(d.y)
for i = 0, nd2-1 do ds.y[, i] = smooth(d.y[, i], ns, /nan)
bad = abs(d.y) gt (ftnsabs(ds.y)/(ns-1+ft) )
if nd2 gt 1 then bad = total(bad, 2)
wbad = where(bad gt .5)
If(wbad[0] Ne -1) Then d.y[wbad, *] = !values.f_nan
if not keyword_set(new_name) then new_name = name+'_cln'
store_data, new_name, data = d, dlim = dlim, lim = lim
Endelse
Endif
end
This could use comments....LOTS of comments
It does like 3 different things
charsize would change the text size of the characters in the labels and stuff in the individual plots
@bryan-harter Idk what this is referencing but it's from 1.5 years ago
In init.py, there is a class called "HoverTime". It is a tiny class right now, but I think it will be important to future development of the tool. The idea behind this class is to allow other scripts to respond to changes to the mouse hovers point.
To that end, there are 2 variables in the class, the actual hover_time, i.e. the actual point the user is currently hovering over, and the "functions_to_call" list, which is a list of functions that will be called whenever the user's mouse is moved.
There are also 2 functions in the class: the "register_listener" function just appends a function to the "function_to_call" list, so it is pretty simple. The "change_hover_time" function is (or should be) called any time the mouse is moved. All that does is change the variable "hover_time" to the new time, and then calls all of the functions located in "functions_to_call".
I've written an example of how it could be used, and attached it here. Try and get this up and running on your computer so you can see how it works. You'll need to:
pip install pyopengl
This script is meant to be run at the same time you've plotted something from 2017-06-19. This can be done in ipython, because running pytplot in ipython doesn't freeze everything. You can plot stuff from 2017-06-19, and then "run -i 3dtesting.py". Then you can see the position of the spacecraft in 3d as you hover over different points!
Or maybe create a way to generate them automatically, or at the very least just rename them to x,y,z
The SPEDAS paper said one of the biggest things that should be converted to python is the plasma moment computation. I've tried looking into it, and I can barely understand what is going on without looking into it more.
Check out the appendix here for moment computation:
https://lasp.colorado.edu/maven/sdc/public/pages/datasets/SISs/maven_sis_SWIA-r1_3.pdf
Allow function call pytplot.tplot([[1,2],[3.4]]) - TVars 1/2 will show up in column 1, 3/4 will show up in column 2. Eventually should be applied to Bokeh as well.
for example, tplot_names("fgm_b") would return all tplot names with "fgm_b" somewhere in the middle
Maybe pull off the documentation for examples and inputs for the code headers, general code commenting would be good as well.
Rather than link an altitude tvar to every tvar individually, can we just maybe set a global tvar? Or say something like "all Tvars that start with "mvn_*" will use this as the altitude?
When you make a new TVar from previous TVars, you need to specify all their line options again. Would it be easy to just default to the line options set in the orginal TVars?
I just took a slightly closer look at the code, is flatten in a spot where it is working yet?
In full flatten and several other functions, you do something like:
df = pytplot.data_quants[tvars[0]].data
But the problem with that is that modifying "df" now affects the original Tvars data too. This is because dataframes don't create a copy by default, you'd need to write
df = pytplot.data_quants[tvars[0]].data.copy()
In order to get a brand new one.
You could provide a crib sheet that describes the most important functions.
One option is a python file that the users can run in Spyder, another option is a Jupyter Notebook (or perhaps both
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.