Noise floors about internal_waves_barkley_canyon HOT 17 CLOSED

@jklymak any thoughts? No urgency, if you want to wait until our meeting!

from internal_waves_barkley_canyon.

The integral under the curve should be the variance (std^2). So when you determine a white noise level, you need to, somewhere, have a Nyquist frequncey in there.

from internal_waves_barkley_canyon.

PS you want the precision, which they say you need to software to figure out.

from internal_waves_barkley_canyon.

@jklymak I merged (and deleted) my previous comments to reflect updates

For the RDI instruments, since the noise is a flat line, I used the integral I = delta_xdelta_y = 2std^2 (variance for one-sided PSD), with a bandwidth of {0 to the Nyquist frequency} as delta x. To find total noise power, I solved for delta_y = (2*std^2)/delta_x. Then, due to using Welch's method, divided total noise power by the number of FFT segments to get the noise power for each frequency bin.

std = 0.076 # single ping accuracy from instrument datasheet = standard deviation [m/s]
var = 2std**2 # from https://www.osti.gov/servlets/purl/5688766/ pp.25-26, 2 for one-sided [(m/s)**2]
dx = np.max(umup_f/3600) # Nyquist frequency to define bandwidth range, 5.55e-4 Hz
dy = var/dx # total noise power
noise_floor = dy/len(umup_f) # noise power for each Welch FFT segment, which are equal

I got the Nortek software, and for the calibration settings that ONC uses for this ADCP, a horizontal velocity precision of 1.56 cm/s is given.

How does velocity precision relate to standard deviation? I believe they're synonymous. If so, for the Nortek they use 6 pings per ensemble, so I found the standard error of the mean for a single ensemble, and then used the same process as for the RDI instrument to find the noise floor.

The noise floor is very low, compared to the 75 kHz instrument. However, this could be due to the greater number of pings per ensemble (6 vs 1), and the larger depth bins (20 m vs 8 m). It may also be the case that the Nortek software gives a better estimate (based on more configuration parameters) than the list that RDI provides. Whatever is happening, the noise floor is well below the 'whitening' in the Axis 55 kHz spectra, so it could be something physical.

• If the methods are good, add noise floor to all spectra and update analysis document.

from internal_waves_barkley_canyon.

Then, due to using Welch's method, divided total noise power by the number of FFT segments to get the noise power for each frequency bin.

@kurtisanstey The integral of a spectrum is its variance if you have plotted it correctly. So whether you used Welchs' method doesn't matter, and you don't multiply by two. \int_0^{F_N} G_xx (f) df = var(x) by Parseval's theorem. https://en.wikipedia.org/wiki/Parseval%27s_theorem

from internal_waves_barkley_canyon.

@jklymak

OK, it's good that I asked. Then, if I don't multiply variance by two, and if I don't take the Welch frequency bins into account, here is the simplified process:

std = 0.076 # single ping accuracy from instrument datasheet = standard deviation [m/s] (for RDI)
var = std**2 # variance [(m/s)**2]
dx = np.max(umup_f/3600) # Nyquist frequency to define bandwidth range, from 0 to 5.55e-4 Hz
dy = var/dx # noise power, which will be the same for all frequencies (flat line, white noise), integral of rectangle is just dx*dy

And here are the plots:
https://github.com/kurtisanstey/project/blob/master/documents/slope_noise.png
https://github.com/kurtisanstey/project/blob/master/documents/axis55_noise.png

It seems that now the noise floor for the RDI instrument (Slope) is much too high, but the Nortek (Axis55) is closer to where I'd expect it to be (before it was very low). Either I'm doing something wrong, or ONC's parameters for number of pings per ensemble, etc. for the RDI instrument are incorrect.

from internal_waves_barkley_canyon.

You 15-minute data is certainly not just one ping, though, right?

from internal_waves_barkley_canyon.

@jklymak, that's right. I think I found the issue.

I rechecked ONC and I may have been mistaken with the information I was using for ping and ensemble time, before. The website is very confusing with so many regular changes to instruments throughout the deployment period, in some cases. I had to go through the history to make sure I was using the correct settings.

Here are the calibration settings I went with:
Axis 75 (2013-2014) RDI is listed as 1 ping per ensemble, at 2 s ensemble time.
Axis 55 (2017-2018) Nortek is listed as having 6 pings per ensemble, at 108 s 'averaging interval'. I'm not sure how they get 2s data from these settings?
Upper Slope 75 (2013-2014, 2017-2018) RDI is listed as 1 ping per ensemble, at 2 s ensemble time.

That's the 2 s data, and when I download the NetCDF I have the data averaged to 15 minutes. I adjust the std for number of pings over the 15 minute averaging, using standard error of the mean, and then calculate the noise floor. Things look reasonable with the updated numbers, each near to 10^-2, with the Nortek noise floor being a bit lower due to larger depth and ensemble bins, but not excessively so (like before):

https://github.com/kurtisanstey/project/blob/master/documents/axis55_noise.png
https://github.com/kurtisanstey/project/blob/master/documents/axis75_noise.png
https://github.com/kurtisanstey/project/blob/master/documents/slope75_noise.png

from internal_waves_barkley_canyon.

So, I don't know what you plotted, but.. If I take (.076/450)**2 / 5.55e-4, I get a level of 5x10^-5 m^2/s^2 Hz^-1. On your plots you have the level at 0.015 or so.

You for some reason have divided dx by 3600. Why? What are the units of umup? What are the units of your spectra? It seems maybe you are really working in cycles per hour? Units are confusing with spectra, so it pays to be super explicit.

from internal_waves_barkley_canyon.

@jklymak, the code was a bit messy, so maybe there was some confusion. It's cleaned up, below.

For the 75 kHz ADCP, I'm plotting (( 0.076 / sqrt(450) )**2) / 5.5e-4 = 0.023 (m/s)^2/Hz. The sqrt(450) is part of the standard error of the mean for the number of pings and time averaging.

dx is divided by 3600 because the PSD frequency umup_f is generated in cph, and I want Hz. I do the same for plotting, and multiply the power by 3600 to compensate, too. I'm diligent with my spectral units, as they can be so confusing!

from internal_waves_barkley_canyon.

Oops sorry you are right. It's sqrt(450). Sorry. Still a little surprising that the levels are so low because it looks like the data hit a noise floor sooner.

from internal_waves_barkley_canyon.

@jklymak

For the Axis 55 in particular the data whitens much sooner, for 2017, 2018, and 2019. Now that I'm (fairly) confident it's not the noise floor, I'm waiting for the adjusted Axis 75 data from ONC to see if the same whitening is present in 2013 and 2014 (this could indicate whether the Nortek noise floor is accurate).

from internal_waves_barkley_canyon.

Adjusting the noise floor for Axis 55 kHz instrument to reflect that the software outputs a std based on 6 pings per ensemble, it still looks too low for what I believe it should be (the high-frequency whitening). The issue could be in my noise floor calculation (above) or in the way ONC processes the data:
https://github.com/kurtisanstey/project/blob/master/plots/psd_plots/psd_Axis55_2017.pdf

from internal_waves_barkley_canyon.

@jklymak

• I adjusted the std for the 55 kHz instrument to be 6.62 cm/s for an independent ping of 6 s, and a raw data resolution of 18 s. The noise floors for each instrument appear to line up, appropriately, in both the raw and averaged data. I applied a bit of averaging and show only the higher frequencies for visual clarity (the 55 kHz data is more 'zoomed' in):

Raw Axis 55 kHz (Feb 1 - 3, 2017).

15-minute Axis 55 kHz (Feb 1 - 3, 2017).

Raw Axis 75 kHz (Jan 1 - 3, 2014).

15-minute Axis 75 kHz (Jan 1 - 3, 2014).

Raw Slope 75 kHz (Jan 1 - 3, 2017).

15-minute Slope 75 kHz (Jan 1 - 3, 2017).

Additionally, I confirmed that PSD variance is equal to the time series variance, and is slightly off when Welch averaging is applied, as expected. I also added a 50% overlap explicitly (it was there before, as a default), changed to a Hanning window (recommended for long time series, in literature), and adjust the sampling frequency units so that the sig.welch outputs in Hz. I double-checked that these are all working correctly, so I'm confident that the PSD amplitudes are correct.

Here is an example of variance testing with 'no' Welch averaging (a boxcar window, nperseg = len(time_series), and no overlap):

• Check 9 s and 6 s in the Nortek software (probably three 6 s pings per 18 s 'ensemble').
• For raw comparison just look at the high frequencies, and apply 'some' averaging to smooth out the noise a bit.
• Add a 'noise floor' Notebook document to archives, with the calculations, for reference.

from internal_waves_barkley_canyon.

PSD comparison and variance checks out between raw and averaged data. Noise floors are appropriate and consistent between raw and averaged data.

from internal_waves_barkley_canyon.

Need to recheck Axis 55 kHz noise floor. Looks like I used 60 second intervals instead of 6, but in either case it's wrong. The other instrument looks spot on, so I must not be understanding the way ONC has the ping averaging set up. Contact them if necessary (with examples).

from internal_waves_barkley_canyon.

Making separate issue for Axis 55 instrument.

from internal_waves_barkley_canyon.