agottesman710 / magnetometer_correlations Goto Github PK

Date: May 10, 2022

Dataset Title: Data and Methods for Magnetometer Station Correlation Calculations

Dataset Creators: Gottesman, Ari

Dataset Contact: [email protected]

Funding: National Science Foundation (NSF)

Abstract: The occurrence of small-scale and intense ionospheric currents that can contribute to geomagnetically induced currents have recently been discovered. A difficulty in their characterization is that their signatures are often only observed at single widely spaced (typically 300 to 500 km) ground geomagnetic stations. These small-scale structures motivate the examination of the maximum station separation required to fully characterize these small-scale signatures. We analyze distributions of correlation coefficients between closely spaced mid-latitude and auroral zone ground magnetometer stations spanning day to month long intervals to assess the separation distance at which geomagnetic signatures appear in only one station. Distributions were analyzed using periods that included low and high geomagnetic activity. We used data from pairs of magnetometer stations across North America within 200 km of each other, all of which were separated primarily latitudinally. Results show that while measurements remain largely similar up to separations of 200 km, large and frequent differences appear starting at around 130 km separation. Larger differences and lower correlations are observed during high geomagnetic activity, while low geomagnetic activity leads to frequent high correlation even past 200 km separation. Small but identifiable differences can appear in magnetometer data from stations as close as 35 km during high geomagnetic activity. Correlations are consistently higher in the north-south component when compared to the east-west component, giving insight into current signatures driving the differences in observations during storms. With one-second magnetometer stations now being standard practice, we recommend future magnetometer array deployment in the auroral and sub-auroral zone to have separations of 100-150 km. This would enable the monitoring of large scale effects of geomagnetic storms, better temporal and spatial resolution of substorms, and observations of small scale current signatures.

Summary of data and processing: Raw data was taken from 13 magnetometer stations. All of this was processed into a singular data structure created to do correlation calculations. Data of magnetometer stations from SuperMAG, THEMIS, CARISMA, the University of Alaska, Fairbanks, INTERMAGNET, and AUTUMNX can all be easily read in using the Station data type in station_oss.py. Folders containing data from multiple magnetometer stations can also be read in using the StationSet data type from station_set_oss.py, which creates individual Station objects for each magnetometer station's data.

After data is put into Station objects, the correlation_histogram() function can be used to calculate and save data in the same format as Correlation_Distributions.csv. The description for running correlation_histogram() is contained within the docstring of the function.

Data for this specific study was taken from two geomagnetic storms, one taking place on 7-8 September, 2017, and one taking place on 23-24 March, 2023.

ALL DATA MUST BE DOWNLOADED SEPARATELY: THEMIS: http://themis.ssl.berkeley.edu/data/themis/thg/l2/mag/ CARISMA: https://carisma.ca/carisma-data-repository UAF: https://www.gi.alaska.edu/monitors/magnetometer/archive INTERMAGNET: https://imag-data.bgs.ac.uk/GIN_V1/GINForms2 AUTUMN: https://autumn.athabascau.ca/TBS_index.php

Methodology and Files Contained:

station_oss.py: This file contains a data structure used to format magnetic field data from 6 databases used in this project. These sources are: SuperMAG, THEMIS, UAF, INTERMAGNET, AUTUMN, and CARISMA (these are case-sensitive). Initializing a Station object with the filename of the data, along with the source entered as specified in the docstring should handle all the necessary steps to format the data correctly into a Station object. Certain data sources require additional information: THEMIS requires station name, and both THEMIS and INTERMAGNET require latitude and longitude for full functionality. Station objects can also be easily concatenated using + or += operators, though this does not compare times, so ensure that the left side of the operator is the earlier dataset. Station objects also have the ability to be compared with > and <, which compares the names of the stations based on order in the english alphabet. str(Station) will return the IAGA code for the magnetometer station.

Once data is stored in Station objects, data from every magnetometer array should bne fully cross-compatible, though for good results, data should be taken from the same time period. All data that used for the following functions uses the data stored in station.db_horizontal, station.db_polar, or station.deltab depending on the coordinate system specified.

The following functions are methods of the Station data structure, meaning they should be called as Station.Function(). Currently available functionality includes: Finding the distance between two stations, called as Station1.find_distance(Station2)

Finding the difference in magnetic or geodetic latitude or longitude, called as Station2.xxxx_difference(Station2), where xxxx = mlat, mlon, glat, or glon. Note that not all data files come with both magnetic and geodetic coordinates, so certain comparisons may not work correctly.

Calculating the correlation between two stations over an interval in Cartesian, Polar or DeltaB Coordinates. It is also possible to calculate the correlation only for data above a certain value threshold of either the magnetometer data itself or Sym-H, a product of NASA OMNI that measures geomagnetic activity. This function is called as Station1.correlation_over_interval(Station2, start_of_interval, end_of_interval). More information about other possible parameters and customizations can be found in the function's docstring in station_oss.py.

Calculating the time lag between stations that maximizes the correlation. This can be used to find the propagation of an event between stations. This function works by checking every time lag value from a range of offsets given by input parameters. This means that using a larger window of time lags can drastically increase the run time required to calculate the maximum correlation. Negative offsets will offset the first station instead of the second, and the smallest (or most negative) number should be entered as the offset_start. This function is called as Station1.max_offset(Station2, start_of_interval, length_of_interval, offset_start=minimum_time_lag_to_be_checked, offset_end=maximum_time_lag_to_be_checked, offset_interval=interval_between_offsets_checked). Other parameters and customizations can be found in the docstring of the function.

Plotting the data of two stations over top of each other, along with the calculated correlation between the two stations over that interval. This function can also take Sym-H data, and plot it along with the magnetometer data. This function is called as Station1.plot_against_interval(Station2, start_of_interval, end_of_interval, time_lag), where time_lag is an optional shift of the second station's data set by {time_lag} number of points/seconds. If no time_lag is required, input 0.

Plotting multiple copies of the above description, and saving them all to the same folder. These copies will all be within the start and stop times, and will be {shift} seconds apart from each other (if the first plot starts at 0:00 UTC and the shift is 600 seconds, the second plot will start at 0:10 UTC). This function can also optionally be given the same time lag information as max_offset() described above, otherwise the default will be 0 (not using any offset). This function is called as Station1.max_correlation_series(Station2, start_of_total_time_period, end_of_total_time_period, shift_in_seconds, interval_of_individual_plots_in_seconds). Other parameters and customizations can be found in the docstring of the function.

Creating a distribution of correlations and plotting a weighted histogram of the created distributions. This works similarly to the function described directly above (max_correlation_series), but stores the correlation values instead of plotting them with data. This function is called nearly identically to max_correlation_series, called as Station1.correlation_histogram(Station2, start_of_total_time_period, end_of_total_time_period, shift_in_seconds, interval_of_individual_plots_in_seconds). Other parameters and customizations can be found in the docstring of the function. This function saves 4 files, a png of the histogram, a pickle file of the histogram data, and a csv of each dimension of the distribution data.

station_set_oss.py: This file contains a data structure used to hold multiple Station objects and perform certain calculations using data stored in Station objects. Currently, this only supports data from SuperMAG, THEMIS, and CARISMA. Data can be read in by entering a filename for SuperMAG, or a foldername of a folder that contains multiple files for THEMIS and CARISMA, along with the source the data is from: "SuperMAG", "THEMIS", or "CARISMA" (this is case-sensitive).

The main functionality currently available is creating a scatter plot of correlations calculated between all stations within a StationSet object, and plotting them as a function of the separation in latitude between the stations. This function behaves similarly to Station.correlation_histogram(), though it averages every correlation value, instead of keeping them as a distribution. This does mean that error bars could optionally be added to the scatter plot if a large distribution is used. This function is called as StationSet.lat_scatter(start_of_total_time_period, end_of_total_time_period, shift_in_seconds, interval_of_individual_periods_in_seconds). Other parameters and customizations can be found in the docstring of the function. Scatters based on longitude and Distance are currently deprecated, though they could easily be fixed by using the lat_scatter code and replacing latitude with the other measures of separation.

StationSet objects can also be used to print the lats or lons of every station in the StationSet by calling StationSet.print('lat') or StationSet.print('lon').

Correlation_Distributions.csv: This file contains the distribution data of every histogram used in the associated paper, all of which were taken from the Station.correlation_histogram() outputs.