xoolive / traffic Goto Github PK

t = opensky.history(
     '2017-06-14 18:00',
     '2017-06-14 21:00',
     callsign=['AEA1518', 'AFL2604']
)

t['AFL2604'] returns a Flight but t['AEA1518'] returns None in spite of being present in the dataframe.

Line 1010 of opensky_impala.py :

callsigns += "and callsign='{:<8s}' ".format(callsign)

triggers UnboundLocalError: local variable 'callsigns' referenced before assignment

upon call like :

df = opensky.rawdata(start=start_date, stop=end_date, callsign=callsign, cached=False)

HOW TO FIX : change Line 1010 of opensky_impala.py to :

callsigns = "and callsign='{:<8s}' ".format(callsign)

Recently, more tables have been added to the Impala database, sometimes redundant with the REST API interface.

This may call for new methods in opensky_impala.py inspired by the following link for this fork for flights_data4, allcall_replies_data4,

[hadoop-1:21000] > show tables;
+--------------------------+
| name                     |
+--------------------------+
| acas_data4               |
| allcall_replies_data4    |
| flarm_raw                |
| flights                  |
| flights_data4            |  <= ?
| identification_data4     |  <= ?
| operational_status_data4 |  <= ?
| position_data4           |
| rollcall_replies_data4   |
| sensor_visibility        |
| sensor_visibility_data3  |
| state_vectors            |
| state_vectors_data3      |
| state_vectors_data4      |
| velocity_data4           |
+--------------------------+

traffic/data/adsb/opensky.py has been left out the change to UTC aware datetimes.

altair seems a good way to display data, although not great with geographical representations. It is easy to reimplement Flight.plot_time to altair with decent default options. This could be added to the default API.

Another scenario that would be nice to work on would be to the possibility to project trajectories to a 2D space (anything, basic projections, t-SNE, PCA, etc.) and be able by an altair-like selector to come back to a Traffic subset.

We need to rethink the whole workflow to interpret new incoming data as timezone aware UTC datetimes, yet respect backward compatibility with timezone naive datetimes.

Taking summer time into account if applicable in the current timezone would be great!

Hi,
If I try to follow the example here: https://traffic-viz.github.io/airports.html?highlight=airport

from traffic.data import airports
from traffic.data.samples import belevingsvlucht

Then I get the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'belevingsvlucht'

Any ideas what is wrong? This happens with the latest traffic from github.

Hello, I recently needed to access the data on the opensky network. After installing the traffic module according to the requirements, running the sample program in the document found that the traffic module was unavailable and many errors occurred. Just enter the code of the imported module and one line of code will give an error.
import traffic
The errors that appear are as follows, what is causing this, thank you very much! Other examples on the official documentation will not work.

C:\Users\Administrator\Anaconda3\envs\evn-traffic\python.exe C:/Users/Administrator/Desktop/traffic-test/traffic-ce001.py
Traceback (most recent call last):
  File "C:/Users/Administrator/Desktop/traffic-test/traffic-ce001.py", line 4, in <module>
    import traffic
  File "C:\Users\Administrator\Anaconda3\envs\evn-traffic\lib\site-packages\traffic\__init__.py", line 8, in <module>
    import pkg_resources
  File "C:\Users\Administrator\Anaconda3\envs\evn-traffic\lib\site-packages\pkg_resources\__init__.py", line 80, in <module>
    from pkg_resources.extern import appdirs
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "C:\Users\Administrator\Anaconda3\envs\evn-traffic\lib\site-packages\pkg_resources\extern\__init__.py", line 43, in load_module
    __import__(extant)
  File "C:\Users\Administrator\Anaconda3\envs\evn-traffic\lib\site-packages\pkg_resources\_vendor\appdirs.py", line 560, in <module>
    import win32com.shell
  File "C:\Users\Administrator\Anaconda3\envs\evn-traffic\lib\site-packages\win32com\__init__.py", line 6, in <module>
    import pythoncom
  File "C:\Users\Administrator\Anaconda3\envs\evn-traffic\lib\site-packages\pythoncom.py", line 2, in <module>
    import pywintypes
  File "C:\Users\Administrator\Anaconda3\envs\evn-traffic\lib\site-packages\win32\lib\pywintypes.py", line 137, in <module>
    __import_pywin32_system_module__("pywintypes", globals())
  File "C:\Users\Administrator\Anaconda3\envs\evn-traffic\lib\site-packages\win32\lib\pywintypes.py", line 40, in __import_pywin32_system_module__
    for suffix_item in imp.get_suffixes():
AttributeError: module 'importlib' has no attribute 'get_suffixes'

I get the following error message when importing SO6.

ModuleNotFoundError Traceback (most recent call last)
in
----> 1 from traffic.data.so6 import so6
2 #so6 = SO6.from_file('20190708_m1.so6')
3 #so6.to_pickle('./data/sample.pkl')

ModuleNotFoundError: No module named 'traffic.data.so6'

Have you also had this issue ?
Best,
Yves

Implement a parser for files provided according to this standard.

Example file: FAA CIFP
Sample decoder (in Go): sv18/arinc424

Hi again,
I am looking at some historical data and see a different number of unique icao24 registrations between traffic and impala.
Here's what I do in traffic:

from traffic.data import opensky
from datetime import datetime
bounds   =  (-74.7,40.3,-72.9,41.2)
start_dt =  datetime(2019,2,28,17,0)
end_dt   =  datetime(2019,2,28,18,0)
flights = opensky.history(start=start_dt,stop=end_dt, bounds=bounds)
print('Number of unique icao24:  ',flights.data['icao24'].nunique())

Which gives: Number of unique icao24: 133

But if I run this query in impala:
SELECT COUNT(DISTINCT icao24) FROM state_vectors_data4 WHERE time>=1551373200 AND time<=1551376800 AND hour>=1551373200 AND hour<=1551376800 AND lat>=40.3 AND lat<=41.2 AND lon>=-74.7 AND lon<=-72.9;
I get:

+------------------------+
| count(distinct icao24) |
+------------------------+
| 319                    |
+------------------------+

So impala is returning around 2.4 times more unique icao24 addresses than traffic.

Is traffic doing some pre-filtering to this data to remove bad position reports, or have I misunderstood how something works? :)

I have been playing around with the airport and runway guesses. For the landing runway guess, an error is generated for the selection of the last 10 minutes:

Error from pandas:

File "C:\Users\olsc\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\computation\ops.py", line 426, in convert_values
if v.tz is not None:

AttributeError: 'NaTType' object has no attribute 'tz'

Triggered by:

File "C:/EmpaDaten/Idealisierte_Spuren_aus_Radar/opensky_adsb/main_opensky_zrh18.py", line 79, in
o=flight.guess_landing_runway()

File "C:\Users\olsc\AppData\Local\Continuum\anaconda3\lib\site-packages\traffic\core\flight.py", line 1054, in guess_landing_runway
.last(minutes=10)

File "C:\Users\olsc\AppData\Local\Continuum\anaconda3\lib\site-packages\traffic\core\flight.py", line 558, in last
return self.class(self.data.query("timestamp > @bound"))

Deal with the following warnings and check the changelog.

DeprecationWarning: alt40fms() has been renamed to selalt40fms(). It will be removed in the future.
DeprecationWarning: alt40mcp() has been renamed to selalt40mcp(). It will be removed in the future.

Relevant commits seem to be junzis/pyModeS@6f6b507 and junzis/pyModeS@8ded350

ping @junzis

Hi Xavier,
I'm trying to download some icao24-specific flights from opensky, but get Nonetype as a return.
My code, with the specific icao24 code removed here:

ic24 = 'icao_removed'
start_dt = datetime(2020, 2, 23, 18, 30, tzinfo = pytz.utc)
end_dt = datetime(2020, 2, 23, 18, 50, tzinfo = pytz.utc)
flights = opensky.history(start=start_dt, stop=end_dt, icao24=ic24)
print(type(flights))

My debug:

INFO:root:Sending request between time 2020-02-23 18:30:00+00:00 and 2020-02-23 18:50:00+00:00 and hour 2020-02-23 18:00:00+00:00 and 2020-02-23 19:00:00+00:00
INFO:root:Sending request: select time, icao24, lat, lon, velocity, heading, vertrate, callsign, onground, alert, spi, squawk, baroaltitude, geoaltitude, lastposupdate, lastcontact, hour from state_vectors_data4  where hour>=1582480800.0 and hour<1582484400.0 and time>=1582482600.0 and time<1582483800.0 and icao24='icao_removed' 

The print output: <class 'NoneType'>
If I log in to the opensky impala shell via ssh and run the select request shown in the debug prompt above then I do receive the data. It just doesn't seem to get parsed properly in traffic. In case it is important, a cache file is produced with size 141kb

Any idea what the problem might be?

Consider checking the following database instead:
https://opensky-network.org/datasets/metadata/aircraftDatabase.csv

Example attached with one flight intersecting LFBBBDX as a point.

The following could be included in a patch to the clip method:

(
    f.shape is not None and
    not isinstance(
        f.shape.intersection(nm_airspaces["LFBBBDX"].shape),
        shapely.geometry.point.Point
    )
)

example data

When following the Quick Start steps I found that if the claim is to start from the cloned repo
I have to correct the path to the sample data from

from traffic.core import Traffic
t = Traffic.from_file('../data/sample_opensky.pkl')

to (one dot, i.e. data directory under current working directory)

from traffic.core import Traffic
t = Traffic.from_file('./data/sample_opensky.pkl')

Hello,
I would like to clip so6 trajectories to specific FIRs.
For instance, I have collected all the trajectories crossing a specific FIR:

bdx_so6_LIMM = so6_m1.inside_bbox(eurofirs['LIMM'])

and now I would like to clip them.
Unfortunately the clip command does not work with the eurofirs objects (I don't have AIRAC files):

bdx_so6_LIMM['BCS361'].clip(eurofirs['LIMM'].flatten())

I get the following error:
NotImplementedError: Multi-part geometries do not provide a coordinate sequence

I'm probably missing something. Is there a better way to do what I'm trying to do?

Thank you,
Rob

Error downloading a large chunk of data:

File "pandas/_libs/parsers.pyx", line 2172, in pandas._libs.parsers.raise_parser_error

ParserError: Error tokenizing data. C error: Expected 17 fields in line 23725, saw 31

thrown by

File "C:\Users\olsc\AppData\Local\Continuum\anaconda3\lib\site-packages\traffic\data\adsb\opensky_impala.py", line 115, in _read_cache
df = pd.read_csv(s, dtype={"icao24": str, "callsign": str})

example (error occurs after a few minutes):

data_raw = opensky.history(
    "2018-01-01 00:01",
    "2018-02-01 00:00",
    bounds=(
        7.99182, 47.09367, #sw
        9.24188, 47.83655 #ne
    ),
)  

Adding error_bad_lines=False or engine='python' to read_csv did not help.

Any way to skip these erroneous lines?
(Interestingly enough, putting the start time at hour 00:00 yields a different error.)

should be : .isoformat()[:-6] + 'Z'

Feed traffic.data.samples with:

• calibration flights;
• aerial surveys;
• firefighting;
• featured;
• gliders

Hello,
When I try to resample flight data using the built-in method I get a ValueError.
To reproduce:

from traffic.data import opensky
flight = opensky.history("2017-02-05", callsign="EZY158T")
flight2=flight.resample()

Gives this error:

  File "<stdin>", line 1, in <module>
  File "/network/aopp/apres/users/proud/conda/lib/python3.6/site-packages/traffic/core/flight.py", line 628, in resample
    .resample(rule)
  File "/network/aopp/apres/users/proud/conda/lib/python3.6/site-packages/pandas/core/generic.py", line 6835, in interpolate
    **kwargs)
  File "/network/aopp/apres/users/proud/conda/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 519, in interpolate
    return self.apply('interpolate', **kwargs)
  File "/network/aopp/apres/users/proud/conda/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 395, in apply
    applied = getattr(b, f)(**kwargs)
  File "/network/aopp/apres/users/proud/conda/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 1845, in interpolate
    limit=limit),
  File "/network/aopp/apres/users/proud/conda/lib/python3.6/site-packages/pandas/core/arrays/datetimelike.py", line 769, in fillna
    value, method = validate_fillna_kwargs(value, method)
  File "/network/aopp/apres/users/proud/conda/lib/python3.6/site-packages/pandas/util/_validators.py", line 348, in validate_fillna_kwargs
    method = clean_fill_method(method)
  File "/network/aopp/apres/users/proud/conda/lib/python3.6/site-packages/pandas/core/missing.py", line 91, in clean_fill_method
    raise ValueError(msg)
ValueError: Invalid fill method. Expecting pad (ffill) or backfill (bfill). Got linear

This is using the latest build of traffic (obtained with: pip install git+https://github.com/xoolive/traffic), Pandas version 0.24.1 and python version 3.6.7.

Any ideas what's happening? I tried to diagnose it but have not had any success.

Some operations on Flight and Traffic structures require the underlying DataFrame to match a given number of properties (data sorted according to timestamp, data matching a given sampling rate, etc.). One way of ensuring the data matches the specification is to apply the treatment once more, but the overhead may be problematic. In some situations, the choice has been made to delegate the responsibility to the end user.

It could be beneficial to add a set of properties to these structures (Sorted, Sampled['1s'], etc.) and propagate them through all methods

A first use case to consider could be:

Flight[Any].resample('1s') returns a Flight[Sampled['1s']], or Flight[Sorted].sort_values('timestamp') returns self

The implementation of such mechanism as a dictionary seems reasonable (beware of __slots__ though...). A further improvement would be to leverage these behaviours at the typing level. (Check documentation here for a good reference)

It seems that memory is leaking.
When several big data files are handled in chain, memory seems to never be completely freed. I suspect @lru_cache decorators on methods to be responsible for that as they seem to store results and instances at the class level.

This could be a starting point to investigate:
https://stackoverflow.com/a/33672499/1595335

See also: https://github.com/youknowone/methodtools/ where you just need to update:

from methodtools import lru_cache

Typing mistake not detected because buffer_to_iter is not typed either. But @obbe79 found it!

Should probably be replaced (but not before proper unit testing) by:

yield self.__class__(...)

Following discussion with @JarryGabriel it would be helpful to get a cumulative distance wrt the end of the trajectory, esp. in the context of approach analyses.

• Basic use case with QFU change and go-around
• Algorithms: particle model for wind field here
• Basic introduction on Enhanced Mode S (with link to https://mode-s.org/)
• Command line tool

• Quickstart with a word about Iteration and copy on Google colab
• Installation instructions for Windows
• Details in architecture section

Future works:

• Basic use case with QFU change
• Basic introduction on Extended Mode S (with link to https://mode-s.org/)
• Command line tool

When an Impala request returns None, there is currently no way to know whether no data has been fetched or whether a Syntax error caused an error. The output should be raised as a Python Exception.

I think the message being passed about Python, Pandas and Shapely is that these are used by traffic, so the paragraph

Python is a popular language among academics and data scientists for its active
 community and large catalogue of libraries fitting the needs of most in their 
respective areas of research. 
Pandas is a natural solution to represent time series,
 Shapely [@shapely_2007] a package of choice to address geometrical operations
(intersections, inclusion, cascaded joins among others).

is coming IMHO too early...
Maybe something like the following will make that para less standalone:

The traffic library uses the Python language to reach to its large base of academics and data scientists, to serve and benefit its active community and to build on top of a
 large catalogue of libraries.
So trajectories are modeled on top of Pandas dataframe, a natural solution to
 represent time series,  while airspaces delegate to Shapely [@shapely_2007] geometrical operations (intersections, inclusion, cascaded joins among others).

Trying to load data via the opensky/hadoop method causes traffic to hang.
I run this command: flight = opensky.history("2017-02-05", callsign="EZY158T")

And I see:
0%| | 0/24 [00:00<?, ?it/s]

This display stays on the screen forever (or, at least for the 2 hours I left it!) and the data never downloads.
Any ideas?

Data from OpenSky has position and velocity frames merged. Sometimes, position is lost for a while and interpolation does not detect the data has been repeated. Fields from the position frames should be set to None before interpolation to avoid future errors.

Check with distances between the flight in the following Traffic data:

opensky.history(
     '2017-06-14 18:00',
     '2017-06-14 21:00',
     callsign=['AEA1518', 'AFL2604']
)

PermissionError is raised on Windows when trying to remove file still open, at:

PermissionError: [WinError 32] Der Prozess kann nicht auf die Datei zugreifen,
 da sie von einem anderen Prozess verwendet wird

Using commit ccc46c4, upon calling:

ext_flight = flight.query_ehs()

Got one of the following exceptions:

File "[...]\traffic\core\flight.py", line 376, in query_ehs
        flight = t[self.callsign]
File "[...]\traffic\core\traffic.py", line 123, in __getitem__
        subset = list(index)  # noqa: F841

TypeError: 'int' object is not iterable
TypeError: 'float' object is not iterable

It would probably make sense to annotate each column of the pandas dataframe with physical units. A basic improvement could be a basic annotation for features; a more advanced improvement would check consistency and convert units if need be.

Reference:

• Pint https://pint.readthedocs.io/en/latest/

I think in joss/paper.md:

... indexed by time, which derivate as ground speed, track ...

should be reworded to something like

... indexed by time, with derived ones such as ground speed, track ...

The following settings in the configuration file should have precedence over environment variables.

[proxy]
http: xxx
https: xxx

See https://mode-s.org/decode/adsb/uncertainty.html

The part of the QS that says

If you have access to AIRAC files (published by Eurocontrol), you may get information about more complex zones, here the Terminal Maneuvering Area (TMA) for Toulouse.

from traffic.data import nm_airspaces
nm_airspaces['LFBOTMA']

fails for me...(and subsequent ones where nm_airspaces is used)
Does it need some additional data sources?
Probably a link to Data Sources would help...

Hi!
First of all, thanks for the library, it is really awesome!

I were getting errors when tried to make a query to Opensky historical database
with airport, arrival_airport or departure_airport parameters.
The error is related to Impala shell query:

opensky.history(
    "2019-01-19", "2019-01-20",
    airport="LOWS")

ImpalaError: time	icao24	lat	lon	velocity	heading	vertrate	callsign	onground	alert	spi	squawk	baroaltitude	geoaltitude	lastposupdate	lastcontact	hour
select time, icao24, lat, lon, velocity, heading, vertrate, c
callsign, onground, alert, spi, squawk, baroaltitude, geoaltitude, lastposupdate,
, lastcontact, hour from state_vectors_data4 as sv join (select icao24, firstseen
n, estdepartureairport, lastseen, estarrivalairport, callsign, day from flights_d
data4 where estarrivalairport ='LOWS' and (1553904000 <= day and day <= 155407680
00)) as est on sv.icao24 = est.icao24 and sv.callsign = est.callsign and est.firs
stseen <= sv.time and sv.time <= est.lastseen where hour>=1553904000.0 and hour<1
1553907600.0 and time>=1553904000.0 and time<1553990400.0 ;

ERROR: AnalysisException: Column/field reference is ambiguous: 'icao24'

So I've assigned labels for 'icao24' and 'callsign' in flights_data4 table in opensky_impala.py in parts of the query related to airports (rows 654 - 685) and put same labels in join part (rows 625 - 629).
It works now for me.
All the best.

Include the AIRAC decoder by @obbe79

Try to keep the inclusion as smooth as possible, with:

• consistent interfaces and;
• automatic detection between .BASELINE and .are modes.

Hi Xavier,
I have this code:

from traffic.data import opensky, airports
from datetime import timedelta, datetime
lat_ave,lon_ave = airports['VABB'].latlon
bounds = [lon_ave - 0.45, lat_ave - 0.45,
          lon_ave + 0.45, lat_ave + 0.45]
st_time = datetime(2019,8,19,11,8,0)
flights = opensky.history(start=st_time,
                          stop=st_time+timedelta(minutes=1),
                          bounds=bounds,
                          other_params=" and time-lastcontact<=15 ")

Which I think should give me flights at 11:08 UTC on 19th Aug 2019. But it actually gives me flights for 10:08 UTC.
I guess this is some kind of timezone thing (although I'm on UTC). Any ideas?

Hello,

I'm trying to make use of traffic to download data from OpenSky. I have already configured my credentials in the configuration file, but now I get an error regarding the AIRAC directory:

RuntimeError: Edit file /Users/jad/Library/Application Support/traffic/traffic.conf with AIRAC directory

I really do not know how to proceed with this, as I am not very familiar with Air Traffic information systems. How should I proceed in order to be able to use traffic as an easy access point to OpenSky data?

I appreciate you help with this matter.

Thanks,

Jorge.

Under discussion with @ramondalmau

Interpolation on angles should probably be based on the unwrapped signals.

Hi,

I've been following the examples here:
https://traffic-viz.github.io/opensky_usage.html

import random
sample_callsigns = random.sample(sv.callsigns, 6)
sample_callsigns
['RCH319', 'KZR921', 'SKW5996', 'N21714', 'JSA538', 'DAL2225']

flight = sv[sample_callsigns[5]]
flight

returns the following error:

HTTPError: 404 Client Error:  for url: https://opensky-network.org/api/tracks/?icao24=acbe7a

Any hints on how to overcome this?

Many thanks

Current syntax obliges the user to manually cascade filters:

flight.filter().filter(altitude=53)

It would be beneficial, esp. with default parameters to cascade filters in one command:

flight.filter(altitude=(17, 53))

Several use cases result in weird pandas DataFrame due to the naive implementation of the (precious) Flight.agg_time method.

Example with a multi-index

from traffic.data.samples import belevingsvlucht
belevingsvlucht.agg_time('5T', altitude=('max', 'mean')).data.columns[-3:]

produces:

Index([                           'rounded',
        ('altitude_('max', 'mean')', 'max'),
       ('altitude_('max', 'mean')', 'mean')],
      dtype='object')

expected: ['rounded', 'altitude_max', 'altitude_mean']

Example with a lambda function

belevingsvlucht.agg_time('5T', altitude=lambda x: x.sum()).data.columns[-2:]

produces:

Index(['rounded', 'altitude_<function <lambda> at 0x15a24b670>'], dtype='object')

expected: ['rounded', 'altitude_<lambda>'].

Example with a named function

def shh(x): x.sum()
belevingsvlucht.agg_time('5T', altitude=shh).data.columns[-2:]

produces:

Index(['rounded', 'altitude_<function shh at 0x15a4dcee0>'], dtype='object')

expected (use of function.__name__):

Index(['rounded', 'altitude_shh'], dtype='object')

The method predict of the class Clustering inside traffic/algorithms/clustering.py, should use transform instead of fit_transform