0xcoto / virgo Goto Github PK

View Code? Open in Web Editor NEW

137.0 13.0 24.0 65.04 MB

📡 Virgo: A Versatile Spectrometer for Radio Astronomy

Home Page: https://virgo.readthedocs.io/en/latest/

License: GNU General Public License v3.0

Python 86.24% TeX 13.76%

sdr virgo radio-telescope radio telescope astronomy spectrometer rf rfi-mitigation astrophysics

virgo's Introduction

Virgo: A Versatile Spectrometer for Radio Astronomy

About Virgo

Virgo is an easy-to-use open-source spectrometer and radiometer based on Python and GNU Radio (GR) that is conveniently applicable to any radio telescope working with a GR-supported software-defined radio (SDR). In addition to data acquisition, Virgo also carries out automated analysis of the recorded samples, producing an averaged spectrum, a calibrated spectrum, a dynamic spectrum (waterfall), a time series (power vs time) and a total power distribution plot.

Lastly, an important set of utilities is provided to observers, making the package for a great tool for planning (radio) observations, estimating the system sensitivity of an instrument, and many more.

Statement of Need

Designed to be used by students, educators and amateurs in the field of radio astronomy, Virgo has already been adopted by a number of small and large-aperture radio telescopes, permitting both spectral and continuum observations with great success. These instruments include the ISEC TLM-18 (18m), the ACRO RT-320 (3.2m), the JRT (1.9m), and the PICTOR Telescope (1.5m), among others.

Although the hardware aspect of a radio telescope is generally handled by newcomers with relative ease, the skill set needed to integrate a complete software pipeline to support observations is not something most users are equipped with. Virgo tackles this problem by providing non-experts with a tool to collect and interpret data from radio telescopes, without requiring expertise in digital signal processing and software engineering. An example use case is classroom experiments in which students build a small-aperture antenna connected to a low-noise amplifier followed by an SDR, and with the help of Virgo, obtain data to map out the galactic distribution of neutral hydrogen and/or derive the rotation curve of the Milky Way.

Key Features

4-tap weighted overlap-add (WOLA) Fourier transform spectrometer
- Reduced FFT sidelobes
- Plain FT filterbank pipeline also supported for observatories with limited computational resources
Adjustable SDR parameters
- Device arguments
- RF/IF/BB Gain
Header file
- Observation parameters automatically passed to corresponding .header file
- Includes logged MJD (at observation t₀)
Spectral line support
- Spectrum calibration
- - y axis is automatically rescaled to S:N units with line masking
- - Optional automatic slope correction (based on linear regression) for poorly-calibrated spectra
- Supports median operation for RFI mitigation on the frequency-domain (adjustable n-factor)
- RFI channel masking
- Adjustable f_rest for observation of any spectral line (not just HI)
- Secondary axes for relative velocity automatically adjusted accordingly
- Prevention against strong narrowband RFI rescaling subplot
- The average spectra, calibration spectra and calibrated spectra are optionally saved as a csv file for further analysis
Continuum support
- Supports median operation for time-varying RFI mitigation (adjustable n-factor)
- Total power distribution (histogram) displayed, both for raw and clean data
- - Best Gaussian fits computed automatically
- Prevention against strong short-duration RFI rescaling subplot
- Time series optionally saved as a csv file for further analysis
Pulsars
- Incoherent dedispersion support for giant pulse search (and FRB follow-up, assuming DM is known)
Dynamic spectrum (waterfall)
- Optionally saved as a FITS file for further advanced/custom analysis
Decibel support
- Power units optionally displayed in dB
Observation planning toolkit
- Predict source altitude & azimuth vs time
- Quickly convert galactic to equatorial and Alt/Az to RA/Dec
- Plot telescope position on the 21 cm all-sky survey
- Simulate 21 cm profiles based on the LAB HI survey
Basic calculation toolkit for system sensitivity & performance. Computes:
- Antenna gain (in dBi, linear or K/Jy)
- Effective aperture
- Half-power beamwidth
- Noise figure to noise temperature and vice versa
- Antenna gain-to-noise-temperature (G/T)
- System equivalent flux density (SEFD)
- Radiometer equation (S:N estimation)
Built-in tool for conducting rapid RFI surveys
Argument-parsing support
Works directly from the command line (virgo -h), or as a Python module

Telescopes based on the Virgo Spectrometer

ISEC TLM-18 Telescope (18m)
ACRO RT-320 (3.2m)
SALSA Vale Telescope (2.3m) [potentially soon, but already tested]
SALSA Brage Telescope (2.3m) [potentially soon, but already tested]
JRT (1.9m)
PICTOR Telescope (1.5m)
NanoRT Telescope (15cm)
and more!

Example Observation

Observation of galactic clouds of neutral hydrogen toward the constellation of Cygnus (α = 20h, δ = 40° , l = 77° , b = 3°), observed by the TLM-18 Telescope in New Jersey, U.S. with Virgo. The average spectrum (top left), the calibrated spectrum (top center), the dynamic spectrum (top right) and the time series along with the total power distribution (bottom) are all plotted by the software automatically.

Example Source Location Prediction

Example HI Profile Retrieval

Example HI Map

The red dot indicates the position of the telescope's beam in the sky.

Data Acquisition Flowgraph

Virgo is a four-tap WOLA Fourier transform spectrometer. The raw I/Q samples are processed in real time using GNU Radio, with the amount of data stored to file being drastically reduced for further analysis. The following flowgraph handles the acquisition and early-stage processing of the data:

Example radio map acquired and processed with the help of Virgo (PICTOR Northern HI Survey)

Installation

To use Virgo, make sure Python and GNU Radio (with gr-osmosdr) are installed on your machine.

For Debian/Ubuntu and derivates, the installation is straightforward:

sudo apt install gnuradio gr-osmosdr

Note: The GNU Radio and gr-osmosdr dependencies are only required for acquiring data with the necessary hardware (software-defined radio). They are not required for planning observations, analyzing data, running calculations or any other functionalities provided by the package. For more information, please refer to the Dependencies section.

Once Python and GNU Radio are installed on your system, run

pip install astro-virgo

Documentation

To learn how to use Virgo, please read through the documentation here.

If you believe something is not clarified in the documentation page, you are encouraged to create an issue (or send me an e-mail and I'll be happy to help.

Contributing

If you wish to contribute to the package (either with code, ideas or further documentation), please read through the Contributor Guidelines.

Credits

Virgo was created by Apostolos Spanakis-Misirlis.

Contact: [email protected]

Special thanks to Dr. Cameron Van Eck, Paul Boven and Dr. Cees Bassa for their valuable contributions.

virgo's People

Contributors

Stargazers

Watchers

virgo's Issues

Determining Gain

The obs_parameters ask for 'rf_gain', if_gain, and bb_gain. How are these three determined? Is the RF gain the gain attributed to the LNA? What determines the other two numbers?

module compiled against API version 0xe

Just installed Virgo on Ubuntu20.04 & RPi4 Raspbian/Bullseye. Ran the example snippet: no errors on Ubuntu, but on RPi4 received this error:

RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
Traceback (most recent call last):

Any ideas? Do I have the wrong python3-numpy?

BR/J.Clark

Upgrade to Python 3

I beg you pardon. Are there any plans to upgrade the existing code from Python 2 to Python 3? @0xCoto

Calibration Question

I have a question about calibration. I couldn't find anything in the documentation about calibrating, although the docs did talk about a calibrated graph as part of the output. Is it possible to do hot and cold calibration? If so, what is the format for that file? Can these files be created by CygnusRFI?

Configuring Virgo for multiple SDR using the KrakenSDR SDR module

Trying to find documentation on configuring GNU Radio with Virgo for configuring multiple RTL-SDR, built integration using the new KrakenSDR. How do you configure Virgo for an Interferometer using KrakenSDR.

Any help would be much appreciated. By the way I'm using osmosdr, configured with two RTL-SDR. The KrakenSDr has all five RTL-SDR configured with an internal time source and synced. I'm not sure but believe also a fast-convolution to extract the offsets and then apply them to on-going processing, involving conjugate multiply. For the KrakenSDR, only one conjugate multiply would be required, I believe.

Trying to also figure out how to create the calibration.dat file. When I make the Virgo.Import call, the error that the CAL_FILE is missing?

Again, any help would be very much appreciate or point me in a solid direction, thank you!!

https://github.com/krakenrf/krakensdr_docs/wiki

Useless spectra and wola.grc does not open

Hi 0xCoto,
probably not real issues but wrong handling by myself:

I am building a radio telescope. Until my RTL-SDR.com V3 arrives and my telescope is ready, I play around with an old DVB-T stick.
Hardware: Simple telescopic dipole antenna, RTL2832U + FC0013, Raspberry Pi 4
Software: Latest Raspbian and GNU Radio Compnion 3.7.13.4; Atro-Virgo installed using pip.
The hardware setup works as I can see nice waterfall spectra with GNU radio (RTL-SDR source plus QT waterfall sink) and RF Gain == 1.

I just wanted to aquire some basic data in the range of FM radio to have something to play with.
Here is my code:
`import virgo

counter = 8752e4
while counter < 11000e4:

# Define observation parameters
observation = {
    'dev_args': '',
    'rf_gain': 1,
    'if_gain': 20,
    'bb_gain': 20,
    'frequency': counter,
    'bandwidth': 256e4,
    'channels': 64,
    't_sample': 5,
    'duration': 15
}

# Data acquisition
virgo.observe(obs_parameters=observation, obs_file=str(counter), start_in=0)


import numpy

numpy.savetxt(str(counter)+".csv", numpy.fromfile(str(counter), dtype='float32'), delimiter=",")
counter += 256e4

`
It generates 9 files with 3 spectra each. Here the plotted results:
(https://ibb.co/87hwqKL)
I can just see artifacts. No idea why the first four and the latter fife are different.
As FM radio is so strong, I would have expected to clearly see the fm stations. Do you have an idea what I am doing wrong?

I tried to open wola.grc to analyze the problem but gnu companion just gave me some errors. What is your version of GNU Companion?

RFI Removal

The documentation states that you can remove RFI channels from the ploy by providing a list that includes the high and low frequencies of the channels with RFI. Is it possible to block multiple instances of RFI this way? I've tried passing a list with multiple highs and lows and a list of lists, but neither worked.

Any help would be appreciated.

JOSS review: Documentation general comment

Dear @0xCoto.

Here is a global comment about the documentation. I will create other issues with specific comments.

For such a software I believe that having the complete documentation inside the readme
is probably not appropriate and restrain a bit its content.
For python development, I would advise to use Sphinx to create a documentation that can
be then 'attached' directly to Github. It gives a great flexibility on the content
and its organization. That would greatly help the readibility. A nice example can be seen
here https://fpavogt.github.io/fcmaker/.

If you want to keep the readme format, you will have to at least add the statement of need (it is in the paper) and the community guidelines

Virgo working with Python Virtual Environments

Hi Apostolos,

I always use virtual environments for my python code. (Python venv. I seem to get a crash when I use Virgo with venv. Like this:

  File "/home/image/pictor/venv/lib/python3.8/site-packages/virgo/virgo.py", line 525, in observe
    from run_wola import run_observation
ModuleNotFoundError: No module named 'run_wola'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 25, in <module>
    virgo.observe(obs_parameters=obs, obs_file='observation.dat')
  File "/home/image/pictor/venv/lib/python3.8/site-packages/virgo/virgo.py", line 527, in observe
    from .run_wola import run_observation
  File "/home/image/pictor/venv/lib/python3.8/site-packages/virgo/run_wola.py", line 9, in <module>
    from gnuradio import blocks
ModuleNotFoundError: No module named 'gnuradio'

I can hack round it, but really it needs a proper fix. If you want me to look into it some more, I can.

Kind regards

Steve

Adding Meta parameter to virgo.plot throws error

Hi Apostolos

I have noticed that if you include the meta=True parameter to the virgo.plot command it throws an error. Removing the parameter works correctly.

meta (bool) – Display header with date, time, and target

print ('Generate Plot')
virgo.plot(
       obs_parameters=obs,
       n=20,
       m=35,
       f_rest=freq,
       obs_file=prefix+'_observation.dat',
       cal_file='/home/pi/astro/off.dat',
       power_csv=prefix+'_time_series.csv',
       dB=True,
       meta=True,
       spectra_csv=prefix+'_spectrum.csv',
       plot_file=prefix+'_plot.png')

Generate Plot
Traceback (most recent call last):
File "/home/pi/.local/lib/python3.9/site-packages/astropy/coordinates/angle_formats.py", line 302, in parse
found_angle, found_unit = self._thread_local._parser.parse(
File "/home/pi/.local/lib/python3.9/site-packages/astropy/utils/parsing.py", line 115, in parse
return self.parser.parse(*args, **kwargs)
File "/home/pi/.local/lib/python3.9/site-packages/astropy/extern/ply/yacc.py", line 333, in parse
return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
File "/home/pi/.local/lib/python3.9/site-packages/astropy/extern/ply/yacc.py", line 1201, in parseopt_notrack
tok = call_errorfunc(self.errorfunc, errtoken, self)
File "/home/pi/.local/lib/python3.9/site-packages/astropy/extern/ply/yacc.py", line 192, in call_errorfunc
r = errorfunc(token)
File "/home/pi/.local/lib/python3.9/site-packages/astropy/coordinates/angle_formats.py", line 294, in p_error
raise ValueError
ValueError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/pi/astro/karo2.py", line 119, in
virgo.plot(
File "/home/pi/.local/lib/python3.9/site-packages/virgo/virgo.py", line 710, in plot
obs_location = EarthLocation.from_geodetic(loc[0], loc[1], loc[2])
File "/home/pi/.local/lib/python3.9/site-packages/astropy/coordinates/earth.py", line 297, in from_geodetic
lon = Angle(lon, u.degree, copy=False).wrap_at(180 * u.degree)
File "/home/pi/.local/lib/python3.9/site-packages/astropy/coordinates/angles.py", line 122, in new
angle, angle_unit = form.parse_angle(angle, unit)
File "/home/pi/.local/lib/python3.9/site-packages/astropy/coordinates/angle_formats.py", line 395, in parse_angle
return _AngleParser().parse(angle, unit, debug=debug)
File "/home/pi/.local/lib/python3.9/site-packages/astropy/coordinates/angle_formats.py", line 308, in parse
raise ValueError(
ValueError: Syntax error parsing angle '-'

virgo.simulate does not return the LAB data

When running the virgo.simulate function it returns a PNG plot but it contains no LAB line.

The All Sky map is being returned correctly.

gal_l, gal_b = virgo.galactic(RA,DEC)
print ('Galactic l = '+str(gal_l))
print ('Galactic b = '+str(gal_b))
virgo.simulate(gal_l, gal_b, beamwidth, -400, 400, prefix+'_simulate.png')

Issues with creating an RFI Survey

Hi
I have been having issues with trying to get an RFI plot using Virgo. I tried using .plot_rfi but get the error:

RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
Traceback (most recent call last):
File "Testrfi2.py", line 18, in
virgo.plot_rfi(rfi_parameters=rfi, data='rfi_data', dB=True, plot_file='plot.png')
File "/home/pi/.local/lib/python3.7/site-packages/virgo/virgo.py", line 981, in plot_rfi
import matplotlib
File "/home/pi/.local/lib/python3.7/site-packages/matplotlib/init.py", line 107, in
from . import _api, cbook, docstring, rcsetup
File "/home/pi/.local/lib/python3.7/site-packages/matplotlib/rcsetup.py", line 26, in
from matplotlib.colors import Colormap, is_color_like
File "/home/pi/.local/lib/python3.7/site-packages/matplotlib/colors.py", line 82, in
from matplotlib import _api, cbook, scale
File "/home/pi/.local/lib/python3.7/site-packages/matplotlib/scale.py", line 18, in
from matplotlib.ticker import (
File "/home/pi/.local/lib/python3.7/site-packages/matplotlib/ticker.py", line 179, in
from matplotlib import transforms as mtransforms
File "/home/pi/.local/lib/python3.7/site-packages/matplotlib/transforms.py", line 46, in
from matplotlib._path import (
ImportError: numpy.core.multiarray failed to import

I tried using virgo.observe and virgo.plot, and then but get the error:

Traceback (most recent call last):
File "Test.py", line 17, in
virgo.observe(obs_parameters=obs, obs_file='observation.dat')
File "/home/pi/.local/lib/python3.7/site-packages/virgo/virgo.py", line 551, in observe
duration=duration, t_sample=t_sample, obs_file=obs_file)
File "/home/pi/.local/lib/python3.7/site-packages/virgo/run_wola.py", line 51, in init
self.osmosdr_source_0 = osmosdr.source( args="numchan=" + str(1) + " " + dev_args )
File "/usr/local/lib/python3/dist-packages/osmosdr/osmosdr_swig.py", line 1162, in make
return _osmosdr_swig.source_make(*args, **kwargs)
RuntimeError: No supported devices found (check the connection and/or udev rules).

The same happens with rfi_monitor.
So part of the issue seems to be that the device is unsupported with the source block, but the osmocom Source in the wola.grc says it supports the SDR I have.

I am on a raspberry pi 4, using the PiSDR image, with Virgo version 3.6.6, and using a SDRplay RSPdx, with an Omnilog 90200 antenna.

The code is below:

import virgo

# Define observation parameters
obs = {
    'dev_args': '',
    'rf_gain': 30,
    'if_gain': 25,
    'bb_gain': 18,
    'frequency': 1420e6,
    'bandwidth': 2.4e6,
    'channels': 2048,
    't_sample': 1,
    'duration': 60
}

# Define rfi survey parameters
rfi = {
    'dev_args': '',
    'rf_gain': 30,
    'if_gain': 25,
    'bb_gain': 18,
    'frequency': 1420e6,
    'bandwidth': 2.4e6,
    'channels': 2048,
    't_sample': 1,
    'duration': 60,
    'f_lo': 1400e6
}

virgo.observe(obs_parameters=obs, obs_file='observation.dat')

# Analyze data, mitigate RFI and export the data as a CSV file
virgo.plot(obs_parameters=obs, n=20, m=35, f_rest=1420.4057517667e6,
           obs_file='observation.dat', rfi=[1419.2e6, 1419.3e6],
           dB=True, spectra_csv='spectrum.csv', plot_file='plot.png')

# Begin data acquisition
virgo.plot_rfi(rfi_parameters=rfi, data='rfi_data', dB=True, plot_file='plot.png')

Thanks for your help

Airpsy mini GNU Radio crash

It seems that Airspy Mini does crash GNU radio on Windows 10 while AIRPSY says that it should work with GNU Radio. RTLSDR no problem, Airspy Mini Problem.

Unable to use VIRGO, need guidance

I installed GNU Companion, osmocom, and other dependencies. Installed Virgo also, as per the direction given in the manual. I'm using Linux Mint as OS. But could not understand how to import modules and where to import them. how to use it with GNU Companion? Please give me a step-by-step guide. Python 2.7.17 is there in my system. #python -c "import virgo" shows no output on the command line. virgo is present under module list.
`$ python
Python 2.7.17 (default, Jul 1 2022, 15:56:32)
[GCC 7.5.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.

help('modules')

Please wait a moment while I gather a list of all available modules...

WARNING: AstropyDeprecationWarning: The astropy.vo.samp module has now been moved to astropy.samp [astropy.vo.samp]
/usr/local/lib/python2.7/dist-packages/skimage/novice/init.py:103: UserWarning: The skimage.novice module was deprecated in version 0.14. It will be removed in 0.16.
warnings.warn("The skimage.novice module was deprecated in version 0.14. "
No handlers could be found for logger "OpenGL.Tk"
linux; GNU C++ version 7.3.0; Boost_106501; UHD_003.010.003.000-0-unknown

/usr/lib/python2.7/dist-packages/matplotlib/cbook/deprecation.py:106: MatplotlibDeprecationWarning: The mpl_toolkits.axes_grid module was deprecated in version 2.1. Use mpl_toolkits.axes_grid1 and mpl_toolkits.axisartist provies the same functionality instead.
warnings.warn(message, mplDeprecation, stacklevel=1)
BaseHTTPServer astropy imaplib rlcompleter
Bastion asynchat imghdr robotparser
CDROM asyncore imp runpy
CGIHTTPServer atexit importlib samba
Canvas atk imputil scanext
Cheetah atomicwrites imutils sched
ConfigParser attr iniparse scipy
Cookie attrs inspect select
Crypto audiodev io serial
DLFCN audioop itertools setproctitle
Dialog backports json sets
DocXMLRPCServer base64 keyword setuptools
FileDialog bdb ldb sgmllib
FixTk binascii lib2to3 sha
HTMLParser binhex libxml2 shelve
Hamlib bisect libxml2mod shlex
IN bsddb libxslt shutil
MimeWriter bz2 libxsltmod signal
OpenGL cPickle linecache sip
PIL cProfile linuxaudiodev sipconfig
PyPDF2 cStringIO locale sipconfig_nd
PyQt4 cairo logging site
PyQt5 calendar lsb_release sitecustomize
PySide cgi lxml six
Queue cgitb macpath skimage
ScrolledText chirp macurl2path smtpd
SimpleDialog chunk mailbox smtplib
SimpleHTTPServer cloudpickle mailcap sndhdr
SimpleXMLRPCServer cmath markupbase socket
SoapySDR cmd marshal spwd
SocketServer code math sqlite3
StringIO codecs matplotlib sre
TYPES codeop md5 sre_compile
Tix collada mhlib sre_constants
Tkconstants collections mimetools sre_parse
Tkdnd colorsys mimetypes ssl
Tkinter commands mimify stat
UserDict compileall mmap statvfs
UserList compiler modulefinder string
UserString configobj more_itertools stringold
Xlib contextlib mpl_toolkits stringprep
_Hamlib cookielib multifile strop
_LWPCookieJar copy multiprocessing struct
_MozillaCookieJar copy_reg mutex subprocess
_SoapySDR crypt netrc subprocess32
builtin csv networkx sunau
future ctypes new sunaudio
_abcoll cupsext nis symbol
_ast curses nntplib symtable
_bisect cv2 ntpath sys
_bsddb cycler nturl2path sysconfig
_codecs datetime numbers syslog
_codecs_cn dateutil numpy tabnanny
_codecs_hk dbhash oauth2 talloc
_codecs_iso2022 dbm opcode tarfile
_codecs_jp dbus operator tdb
_codecs_kr decimal optparse telnetlib
_codecs_tw decorator os tempfile
_collections difflib os2emxpath termios
_csv dircache osmosdr tesserocr
_ctypes dis ossaudiodev test
_ctypes_test distutils pango tests
_curses doctest pangocairo textwrap
_curses_panel drv_libxml2 parser this
_dbus_bindings dsextras pathtools thread
_dbus_glib_bindings dumbdbm pcardext threading
_elementtree dummy_thread pdb thrift
_functools dummy_threading pickle time
_hashlib easy_install pickletools timeit
_heapq email pip tkColorChooser
_hotshot encodings pipes tkCommonDialog
_io ensurepip pivy tkFileDialog
_json ephem pkg_resources tkFont
_ldb_text errno pkgutil tkMessageBox
_locale evernote platform tkSimpleDialog
_lsprof exceptions plistlib toaiff
_md5 fcntl pluggy token
_multibytecodec filecmp ply tokenize
_multiprocessing fileinput pmt trace
_osx_support fnmatch popen2 traceback
_posixsubprocess formatter poplib ttk
_pyio fpformat posix tty
_pytest fractions posixfile turtle
_random ftplib posixpath types
_sha funcsigs pprint unicodedata
_sha256 functools profile unittest
_sha512 future_builtins pstats urllib
_socket gc pty urllib2
_sqlite3 genericpath pwd urlparse
_sre getopt py user
_ssl getpass py_compile uu
_strptime gettext pyclbr uuid
_struct gi pydoc validate
_symtable gio pydoc_data virgo
_sysconfigdata glib pyexpat warnings
_sysconfigdata_nd glob pygtk watchdog
_tdb_text gnuradio pygtkcompat wave
_testcapi gobject pylab weakref
_threading_local gps pynotify webbrowser
_version grc_gnuradio pyparsing whichdb
_warnings grp pypdfocr wsgiref
_weakref gtk pysideuic wx
_weakrefset gtkunixprint pytest wxversion
abc gzip pytz xdg
aifc hashlib pywt xdrlib
antigravity heapq quopri xml
anydbm hmac random xmllib
apt hotshot re xmlrpclib
apt_inst hpmudext readline xxsubtype
apt_pkg htmlentitydefs reportlab zbar
aptsources htmllib repr zipfile
argparse httplib resource zipimport
array httplib2 rexec zlib
ast ihooks rfc822 zmq
`

Graph Scales

It there a way to set the graph scales for the average spectrum and calibrated spectrum graphs produced by the plot method. I'm doing multiple scans along the galactic plane, and it would be helpful if all the graphs could be plotted at the same scale.

Matplotlib memory leak on overnight runs

Hi Apostolos

Hello from New Zealand.

I now have my 2.3m dish operational, running a Raspberry Pi4 4Gb RAM, 34Gb storage, RTL-SDR v3 , GPS for NTP timing, 1420MHz filter, LNA, & line amp. Located on our 35 acre farm, 50km north of Auckland, New Zealand.

I am using Virgo and it runs a charm, my S/N is a bit low but I think that is related to my feedhorn alignment so tuning is a work in progress.

What I have found is the while Virgo runs fine when you do one off runs if you have it set for continual running it hits a memory overflow after about 18 hours due to a memory leak.

The attached script will run 5 minute integrations for 24 hours but memory overflows the 4Gb available after about 18 hours.
Karo.py.txt

I have found the following Stack overflow topic that may explain the issue.
https://stackoverflow.com/questions/2364945/matplotlib-runs-out-of-memory-when-plotting-in-a-loop/8862575

Regards
Dave

GR 3.8/3.9 compatibility

I just discovered what has been causing issues for GR 3.8/3.9. Seems like the dev_args and obs_file parameter blocks are set to Type: None, but setting them to String resolves problems.

To-do: Test if Type: String is compatible with GR 3.7 as well, or if only Type: None is the exclusive workaround for the old version, and integrate the fix.

virgo

hello my name is gianni my city is ravenna on the adriatic coast, sorry for the inconvenience i downloaded virgo 3.6.3 i have this problem that i did not have with the previous version. i am using usrp b210, 3d corner reflector antenna. ubuntu 18.04. from the terminal I get this error in bandwidth
All-Series:~/Virgo/virgo$ python virgo.py -f 1420e6 -b 6e6 -c 1024 -t 1 -d 3600 -rf 55
gr-osmosdr 0.1.5 (0.1.5) gnuradio 3.7.14.0
built-in source types: file fcd rtl_tcp uhd rfspace redpitaya
[INFO] [UHD] linux; GNU C++ version 7.5.0; Boost_106501; UHD_3.14.0.HEAD-0-g6875d061
[INFO] [B200] Detected Device: B210
[INFO] [B200] Operating over USB 3.
[INFO] [B200] Initialize CODEC control...
[INFO] [B200] Initialize Radio control...
[INFO] [B200] Performing register loopback test...
[INFO] [B200] Register loopback test passed
[INFO] [B200] Performing register loopback test...
[INFO] [B200] Register loopback test passed
[INFO] [B200] Setting master clock rate selection to 'automatic'.
[INFO] [B200] Asking for clock rate 16.000000 MHz...
[INFO] [B200] Actually got clock rate 16.000000 MHz.
-- Using subdev spec 'A:A A:B'.
[INFO] [B200] Asking for clock rate 48.000000 MHz...
[INFO] [B200] Actually got clock rate 48.000000 MHz.
[WARNING] [MULTI_USRP] Setting DC offset is not possible on this device.
[WARNING] [MULTI_USRP] Setting IQ balance is not possible on this device.
[WARNING] [AD936X] The requested bandwidth 0.000000 MHz is out of range (0.200000 - 56.000000 MHz).
The bandwidth has been forced to 0.200000 MHz.
in this situation I connect the sdr b210 to the rf generator, 0dbm and a frequency of 1420.00500mhz, I see the carrier in plot_png
and that's fine.
any dev_args I enter have the same error.
i ran a test with usrp n210 and it works fine.

JOSS review: functionality documentation & testing

Hi @0xCoto

Functionality documentation:

The in-code documentation is an important problem.
There is no documentation of the functions that are coded in the package.
For example, let's take the function given in the example usage:

>>>import virgo
>>>help(virgo.predict) 
Help on function predict in module virgo.virgo: 
predict(lat, lon, height=0, source='', date='', plot_sun=True, plot_file='')`

>>>help(virgo.observe)
Help on function observe in module virgo.virgo:
observe(obs_parameters, spectrometer='wola', obs_file='observation.dat', start_in=0)

--> There is no documentation of what it does, the input format and the output that is given. Please document every function of the package.

Testing:
In addition, I did not see any automated test mention in the readme. You must be able to test your code. It might be an automatic test suite (with unittest or pytest package for example) or at least a manual procedure that tell us how to test your code.

unexpected keyword argument 'catch_exceptions'

Installed Virgo as per instructions and got the following error when I run virgo.observe:

>> virgo.observe(obs_parameters=obs, obs_file='observation.dat')

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mo/.local/lib/python3.8/site-packages/virgo/virgo.py", line 557, in observe
    observation = run_observation(dev_args=dev_args, frequency=frequency, bandwidth=bandwidth, rf_gain=rf_gain,
  File "/home/mo/.local/lib/python3.8/site-packages/virgo/run_wola.py", line 31, in __init__
    gr.top_block.__init__(self, "Run Observation", catch_exceptions=True)
TypeError: __init__() got an unexpected keyword argument 'catch_exceptions'

Followed the example in the instructions.

Potential gnuradio 3.7/3.8 issue.

Hi Apostolos,

Then if I deactivate the vitual environment and install astro-virgo outside the virtual environment using pip3. I get

built-in source types: file osmosdr fcd rtl rtl_tcp uhd miri hackrf bladerf rfspace airspy airspyhf soapy redpitaya freesrp 
[INFO] [UHD] linux; GNU C++ version 9.2.1 20200304; Boost_107100; UHD_3.15.0.0-2build5
Found Rafael Micro R820T tuner
Using device #0 Realtek RTL2838UHIDIR SN: 00000001
Found Rafael Micro R820T tuner
[R82XX] PLL not locked!
[R82XX] PLL not locked!
Traceback (most recent call last):
  File "test.py", line 25, in <module>
    virgo.observe(obs_parameters=obs, obs_file='observation.dat')
  File "/home/image/.local/lib/python3.8/site-packages/virgo/virgo.py", line 555, in observe
    observation = run_observation(dev_args=dev_args, frequency=frequency, bandwidth=bandwidth, rf_gain=rf_gain,
  File "/home/image/.local/lib/python3.8/site-packages/virgo/run_wola.py", line 107, in __init__
    self.connect((self.osmosdr_source_0, 0), (self.blocks_head_0, 0))
  File "/usr/lib/python3/dist-packages/gnuradio/gr/hier_block2.py", line 37, in wrapped
    func(self, src, src_port, dst, dst_port)
  File "/usr/lib/python3/dist-packages/gnuradio/gr/hier_block2.py", line 105, in connect
    self.primitive_connect(*args)
TypeError: primitive_connect(): incompatible function arguments. The following argument types are supported:
    1. (self: gnuradio.gr.gr_python.hier_block2_pb, block: gnuradio.gr.gr_python.basic_block) -> None
    2. (self: gnuradio.gr.gr_python.hier_block2_pb, src: gnuradio.gr.gr_python.basic_block, src_port: int, dst: gnuradio.gr.gr_python.basic_block, dst_port: int) -> None

Invoked with: <gnuradio.gr.gr_python.top_block_pb object at 0x7f6f56290f30>, <Swig Object of type 'gr::basic_block_sptr *' at 0x7f6f51f5dea0>, 0, <gnuradio.gr.gr_python.basic_block object at 0x7f6f51f5e230>, 0
swig/python detected a memory leak of type 'gr::basic_block_sptr *', no destructor found.

So this is really nice. It detects the RTL.SDR and what looks like a syntax error.

Is this a gnuradio 3.7/3.8 problem? What should I do?

I have a beautiful waveguide just begging to be tested :)

Kind regards

Steve.

VLSR

I'm working on the VLSR transform. I'll provide details when I can. But I wanted to ask about the array of arrays that is loaded from the observation.dat file for plot(). What does each of the individual arrays represent? Is it storing all of the values that are taken during the iteration observation? If so, what is the structure of the array? Does each array represent all the values for one frequency, or does each array represent one reading of all the frequencies, with the next array representing the next reading and so on?

The SpectralCoord class in astropy takes a range of readings and adjusts them so that both the target and the observer are in the same frame of reference (usually ICSR). The results can then be transformed into the LSR.

Also, what is the unit of the stored value? Is it dB, S/N, or something else?

Incorrect waterfall indices in dedispersion operation

There appears to be somewhat of an inaccurate application of the incoherent dedispersion formula, yielding this error:

Traceback (most recent call last):
  File "quick_obs.py", line 34, in <module>
    obs_file='observation.dat', plot_file='plot.png')
  File "/home/apostolos/.local/lib/python3.5/site-packages/virgo/virgo.py", line 565, in plot
    shift(t_bin, n)
  File "/home/apostolos/.local/lib/python3.5/site-packages/virgo/virgo.py", line 450, in shift
    waterfall[:, phase_num] = np.roll(waterfall[:, phase_num], -n_rows)
IndexError: index 256 is out of bounds for axis 1 with size 256

I've only noticed this error just now with a long observation. I suspect there is a mistakenly flipped order (i.e. mixing waterfall indices, ultimately trying to shift channels instead of time bins).

Lines 449-450 and 557-565 are suspicious, but I couldn't figure out a solution from a quick look. Furhter investigation is necessary.

JOSS review: Installation and Usage

Dear @0xCoto

Here are some comments about the instruction:

There is something that is not clear from the installation part of the README. It is stated:
'To use Virgo, make sure Python and GNU Radio (with gr-osmosdr) are installed on your machine.'

Is 'gr-osmosdr' part of GNU Radio? When I go to the gr-osmosdr website it is said
'Marcus D. Leech has kindly integrated the forementioned build steps into his gnuradio installation script at "This is the most user-friendly option so far.'

So it is not clear what I have to install. I could install gnu-radio without problem but I will not install other things if they are not necessary. Please make this clearer.

Usage:
Also, you mention, in 'Usage' to use 'python virgo.py [arguments]'. I believe that if I install the package via pip I do not
have access directly to virgo.py unless I go to the installation directory and use it from here (which is not the point of installing a package)

Example Usage:
The example usage of the readme is relatively short and could be expanded and better explained. What are you trying to do, and what do you expect as a result?

Install Instructions

Hi,
may you give some install instructions for all dependencies?
I'm trying to get it running for hours, but I get stuck in the installation of osmosdr. In addition, the top_block.py can't find gnuradio since it is still running with Python 2.7, while gnuradio is using python3.

I really would like to use your software but I can't get it up and running.