mavensdc / pytplot Goto Github PK

charsize would change the text size of the characters in the labels and stuff in the individual plots

Inside spd_ui_loaded_data__dproc there is an insane amount of worth that goes into smoothing, involving multiple functions. Figure those out and implement those.

Here is the IDL code. Function dimen1 figures out the size of dimension 1 for data, i.e. the size of the time dimension. dimen2 figures out the size of dimension 2, i.e. the number of columns in the dataframe.

;+
; Procedure: CLEAN_SPIKES, 'name'
;
; Purpose: Simple routine to remove spikes from tplot data.
;
; Author: unknown, Probably Frank Marcoline
;
; Keywords:
; display_object = Object reference to be passed to dprint for output.
;
; jmm, 4-jul-2007, made to work for negative data, by adding absolute values
; jmm, 30-jul-2007, Added more error checking
; jmm, 15-sep-2009, documentation test
;$LastChangedBy: $
;$LastChangedDate: $
;$LastChangedRevision: $
;$URL: $
;-
pro clean_spikes,name,new_name=new_name,nsmooth=ns,thresh=ft, display_object=display_object

get_data, name, data = d, dlim = dlim, lim = lim

If(size(d, /type) Eq 8) Then Begin
ds = d
if not keyword_set(ns) then ns = 3
ns = (fix(ns)/2)2 + 1
nd1 = dimen1(d.y)
If(nd1 Le 2ns) Then Begin
msg = name+': Not enough data for smoothing, to despike, returning undespiked data'
dprint, msg , display_object=display_object
if not keyword_set(new_name) then new_name = name+'_cln'
store_data, new_name, data = d, dlim = dlim, lim = lim
Endif Else Begin
if not keyword_set(ft) then ft = 10.
ft = float(ft)
nd2 = dimen2(d.y)
for i = 0, nd2-1 do ds.y[, i] = smooth(d.y[, i], ns, /nan)
bad = abs(d.y) gt (ftnsabs(ds.y)/(ns-1+ft) )
if nd2 gt 1 then bad = total(bad, 2)
wbad = where(bad gt .5)
If(wbad[0] Ne -1) Then d.y[wbad, *] = !values.f_nan
if not keyword_set(new_name) then new_name = name+'_cln'
store_data, new_name, data = d, dlim = dlim, lim = lim
Endelse
Endif
end

This is the same thing as the "tsub_average" procedure in tplot

This is a testing issue

If you have more than a day's worth of data, pyqtgraph only shows the day of the month on the time axis. It should be changed to show hh:mm:ss for much longer, perhaps only switching over after ~15-20 days.

Additionally, it would be great if we could ensure that new dates would always show up as YYYY-MM-DD. I suspect that might involve quite a bit of digging into the pg.AxisItem we are inheriting.

The code to be modified resides in pytplot/QtPlotter/CustomAxis/DateAxis.py

Throw any comments back my way. Thanks!

Investigate how the border is selected, and what we can do to change it

Can we simplify these?!

If you call a tplot function and a pyqtgraph window pops up, it would be neat if we could specify

pytplot.options([the tvar name], "ylog", 1)

And have the displayed plot change itself automatically

for example, tplot_names("fgm_b") would return all tplot names with "fgm_b" somewhere in the middle

tplot_math.fn_interp vs. tplot_math.resample? Not sure what the difference is and why resample works.

• I just took a slightly closer look at the code, is flatten in a spot where it is working yet?

• In full flatten and several other functions, you do something like:

df = pytplot.data_quants[tvars[0]].data

But the problem with that is that modifying "df" now affects the original Tvars data too. This is because dataframes don't create a copy by default, you'd need to write

df = pytplot.data_quants[tvars[0]].data.copy()

In order to get a brand new one.

• Also, throughout all functions, spec_bins seems neglected, but spec_bins can actually be time varying too. It should be read back in to the new tplot variable with the store_data command.

@bryan-harter Idk what this is referencing but it's from 1.5 years ago

Or maybe create a way to generate them automatically, or at the very least just rename them to x,y,z

Apparently a function that print out tplot into an ascii file is important enough to be demoed in the crib sheet. I guess we should add that too.

I've seen several scientists use a binned map. It looks kind of like a spectrogram plot, but the axes are lat and lon rather than bins and time. Can we get pytplot to do that?

Rather than link an altitude tvar to every tvar individually, can we just maybe set a global tvar? Or say something like "all Tvars that start with "mvn_*" will use this as the altitude?

Take a look at each of the main functions in pytplot and make sure the comments are accurate.

Feel free to update the material inside the "docs" folders of both pydivide and pytplot as well. They are probably horrifically out of date. I'll try to find a .doc version of that pydivide PDF i made forever ago.

The small tutorials are done in html. You can open those up in a text editor and modify them that way. It might look daunting at first if you haven't seen HTML code before, but its really just a way to format the text fancily kinda like LaTeX.

What the title says

The SPEDAS paper said one of the biggest things that should be converted to python is the plasma moment computation. I've tried looking into it, and I can barely understand what is going on without looking into it more.

Check out the appendix here for moment computation:

https://lasp.colorado.edu/maven/sdc/public/pages/datasets/SISs/maven_sis_SWIA-r1_3.pdf

Okay I've actually worked through some of the details on this one. Here is how it is done in IDL: dpwrspec.pro

You have a 1D line TVar. You allow the user to specify 2 numbers, nboxpoints and nshiftpoints. They default to 256 and 128 respectively, so that's what I use the rest of this algorithm. But remember you can allow the user to select these.

So you take the first 256 data points, both x (time) and y (the data).

Recalculate y by subtracting the best linear fit of those 256 points, i.e. :

x = x - x[0]
c = np.polyfit(x, y, 1)
y_trend = c[0]*x + c[1]
y = y - y_trend

Then, calculate the "window" (I don't really know what this means) with:

w = scipy.signal.get_window("hanning", 256)

Then calculate the power spectrum of those 256 points:

f,pxx = scipy.signal.periodogram(y[0:256], window=w, detrend=lambda x : x)

Now, shift 128 points over, and do this again for the next 256 points, i.e. points number 128 to 384. And then shift 128 points again, and again, etc, until you are out of points to look at.

Then this data should be read into pytplot, where f is the spec bins, all of the different "pxx" are the data that fill out each column, and the times are at x[128], x[256], x[384], etc. So this will generate a multi-column dataframe from a dataframe that is just one column.

There is also some stuff with binning at the end of the function in IDL. Basically, it looks like a user can set a "bin_size" option, and average each column by bin_size at a time.

For example, if bin_size is 3, and there are 15 columns, then it averages together columns [1,2,3], [4,5,6], [7,8,9], [10,11,12], [13,14,15]. This makes a total of only 5 columns.
As an option:

-Allow users to select whether or not to subtract out the average fit line
-Average out each row of data to compress all the columns into one
-Not use the hanning window (just set window=None in the periodogram function)

In init.py, there is a class called "HoverTime". It is a tiny class right now, but I think it will be important to future development of the tool. The idea behind this class is to allow other scripts to respond to changes to the mouse hovers point.

To that end, there are 2 variables in the class, the actual hover_time, i.e. the actual point the user is currently hovering over, and the "functions_to_call" list, which is a list of functions that will be called whenever the user's mouse is moved.

There are also 2 functions in the class: the "register_listener" function just appends a function to the "function_to_call" list, so it is pretty simple. The "change_hover_time" function is (or should be) called any time the mouse is moved. All that does is change the variable "hover_time" to the new time, and then calls all of the functions located in "functions_to_call".

I've written an example of how it could be used, and attached it here. Try and get this up and running on your computer so you can see how it works. You'll need to:

pip install pyopengl

This script is meant to be run at the same time you've plotted something from 2017-06-19. This can be done in ipython, because running pytplot in ipython doesn't freeze everything. You can plot stuff from 2017-06-19, and then "run -i 3dtesting.py". Then you can see the position of the spacecraft in 3d as you hover over different points!

By supplying "dy" to a store_data call, it will show error bars on the 1D plots

Add a procedure for smoothing, or figure out which python routine already does it:
http://www.harrisgeospatial.com/docs/SMOOTH.html

(Implement pydl.smooth in tplot_math.derive)

In regular IDL tplot, I think we can make text labels appear at an arbitrary x-y coordinate. Is that possible in bokeh/pyqtgraph?

If so, feel free to add a dictionary object to TVar that contains a list of labels that will be plotted inside the plot. Then, you can add a new function to all those TVarFigures called _add_text() that looks through that dictionary and writes things in the plots.

This could use comments....LOTS of comments

This should probably be an option in tplot_options. Need to make sure it works with both bokeh and pyqtgraph. With bokeh, we would just need to add something like " self.fig.xaxis.axis_label="insert x axis here" "

You could provide a crib sheet that describes the most important functions.
One option is a python file that the users can run in Spyder, another option is a Jupyter Notebook (or perhaps both

It does like 3 different things

Maybe pull off the documentation for examples and inputs for the code headers, general code commenting would be good as well.

get_y_limits
zlog in options.py
ylog in options.py
del_data
cdf_to_tplot
_get_figure in generate.py (both Bokeh and Qt)

Like cdf_to_tplot

Can we specify a better way to handle overplot rather than saying store_date("hello", data=["TVar1", "TVar2"])?

I think a good way to go about it would be to link them together as "oplot"

Allow function call pytplot.tplot([[1,2],[3.4]]) - TVars 1/2 will show up in column 1, 3/4 will show up in column 2. Eventually should be applied to Bokeh as well.

https://agupubs.onlinelibrary.wiley.com/doi/full/10.1002/2015GL065366

Figure 4, chart F

Can our tool do that? Is power spectral density the same thing as the "power spectrum" seen in the other ticket?

When I run pydivide test scripts, I notice that the qt window will occasionally show up empty, and I have no idea why.

We have the ability to make a new x axes, but I think it only works if it is linear. We can't do one where the axis looks like:

1 10 15 100

for example. Is this possible to fix?

When you make a new TVar from previous TVars, you need to specify all their line options again. Would it be easy to just default to the line options set in the orginal TVars?

This is one of those things that sounds good in my head, but i have no idea if it can be implemented.

Clicking near a point on a map plot could create a timebar as a marker on another plot, just to kind of show where that time point is located on other types of visualizations

wav_data.pro exists to compute the wavelet transform of tplot variables. This only works on 1D data, so it will compute the wavelet transform for for each column. But 2D data is output. I have literally no idea what a wavelet transform is.

EDIT:
This looks incredibly complicated, I almost wonder if this is even worth it. We might want to wait for like money or something, because this could easily take 100 hours to deeply understand and port over into python.

There is a package to do the actual computing (https://pywavelets.readthedocs.io/en/latest/), I think we could leverage that one if needed. But who knows. This definitely at least requires more looking into.

This should utilize the HoverTime class in the previous ticket.

Basically, make a new script that, when run, will pop up a completely new window with a simple 1D plot. That plot will change depending on what time the user is hovering over. Specifically, for specplots, it will take the time you're hovered over plot the spec_bins vs the data values at that specific time.

High pass filter I believe just calculates the running average at each point in time, and subtracts that from the value. So its kinda like subtracting the average, except the average is different at each point in time.

Need to take a look inside "thigh_pass_filter.pro" in order to more deeply understand.