I'm trying to get away from Windows. TDH is written in .NET, so that doesn't help.

Hence this effort.

Coders and Graphics Designer requested for assistance, feedback, etc.

https://github.com/eyeonus/Trade-Dangerous/tree/gui-beta

Here is my command line and the result, see Gain/Hop line.

python3 trade.py run --cr 10m --cap 788 --empty 21.29 --ly 12.07 --pad l --age 2 -vvv --prog --ls-max 2000 --avoid alioth,minmar/acropolis,Aganippe/Julian --insurance 3804342 --gpt 1500 --no-planet
* Hop   1: .....2,371 origins
* Hop   2: .......988 origins .. 1,525-3,750,880cr gain, 1,500-4,760cr/ton
HIP 6796/Ibold Port -> HIP 6978/Sterling Station (score: 5277727.942143)
Start CR:  6,195,658
Hops    :          2
Jumps   :          3
Gain CR :  4,992,768
Gain/Hop: 2.49638e+06
Final CR: 11,188,426

Not a very big issue but that has annoyed me since the beginning.

I wanted to update TD after a long absence, so I pulled from the github repository and ran -O clean but got an error that Category.csv was missing. Just in case, I deleted the entire TD directory, cloned it afresh, and still got the following error. Any suggestions?

E:\Elite\TD>trade import -P eddblink
NOTE: Rebuilding cache file: this may take a few moments.
NOTE: Missing "E:\Elite\TD\data\TradeDangerous.prices" file - no price data.
NOTE: Downloading file 'modules.json'.
NOTE: Requesting http://elite.ripz.org/files/modules.json
NOTE: Downloaded 297.8KB of gziped data   2.5MB/s
NOTE: Processing Upgrades: Start time = 2018-12-30 20:45:29.579455
NOTE: Finished processing Upgrades. End time = 2018-12-30 20:45:29.588431
NOTE: Downloading file 'index.json'.
NOTE: Requesting https://raw.githubusercontent.com/EDCD/coriolis-data/master/dist/index.json
NOTE: Downloaded   0.7MB of gziped data  19.5MB/s
NOTE: Processing Ships: Start time = 2018-12-30 20:45:29.875662
NOTE: Finished processing Ships. End time = 2018-12-30 20:45:29.887630
NOTE: Downloading file 'systems_populated.jsonl'.
NOTE: Requesting http://elite.ripz.org/files/systems_populated.jsonl
NOTE: Downloaded  20.5MB of gziped data   6.3MB/s
NOTE: Processing Systems: Start time = 2018-12-30 20:45:33.659592
NOTE: Finished processing Systems. End time = 2018-12-30 20:45:35.524649
NOTE: Downloading file 'stations.jsonl'.
NOTE: Requesting http://elite.ripz.org/files/stations.jsonl
NOTE: Downloaded 115.0MB of gziped data  15.7MB/s
NOTE: Processing Stations, this may take a bit: Start time = 2018-12-30 20:45:43.387194
NOTE: Simultaneously processing ShipVendors.
NOTE: Simultaneously processing UpgradeVendors, this will take quite a while.
NOTE: Finished processing Stations. End time = 2018-12-30 20:52:48.068146
NOTE: Downloading file 'commodities.json'.
NOTE: Requesting http://elite.ripz.org/files/commodities.json
NOTE: Downloaded 100.6KB of gziped data  98.5MB/s
NOTE: Processing Categories and Items: Start time = 2018-12-30 20:52:48.563327
NOTE: Checking for missing items....
Traceback (most recent call last):
  File "E:\Elite\TD\trade.py", line 107, in <module>
    main(sys.argv)
  File "E:\Elite\TD\trade.py", line 80, in main
    results = cmdenv.run(tdb)
  File "E:\Elite\TD\commands\commandenv.py", line 81, in run
    return self._cmd.run(results, self, tdb)
  File "E:\Elite\TD\commands\import_cmd.py", line 124, in run
    if not plugin.run():
  File "E:\Elite\TD\plugins\eddblink_plug.py", line 921, in run
    self.importCommodities()
  File "E:\Elite\TD\plugins\eddblink_plug.py", line 518, in importCommodities
    with open(str(tdb.dataPath / Path("Category.csv")), "r") as fh:
FileNotFoundError: [Errno 2] No such file or directory: 'E:\\Elite\\TD\\data\\Category.csv'

Currently running the olddata command lists stations within one jump’s range of a given location sorted by the age of the stations commodity prices. It doesn’t seem to take any account of the —pad filter so lists all stations. I don’t know if this is a bug or intended behaviour. Could the facility be added, please?

I know we previously discussed this, but given the change to a pip model, I think we should switch up from this hardcoded

    self.dataPath = tdb.dataPath / Path("eddb")

to having a default (as above) but overridable with a $ENV. It will still only be relevant for server operation, I would think, but with automated upgrades, I think it's the right way to deal with it from now on.

Maybe look into making TD able to handle that.

The query trade.py market --sell -vv "Cemplangpa/Gorgani Orbital" shows the selling commodities but not the price data:

    Item                   Selling     Avg  Supply Age/Days
-----------------------------------------------------------
+CHEMICALS
    Explosives                                             
    Hydrogen Fuel                                          
    Surface Stabilisers                                    
+INDUSTRIAL MATERIALS
    Insulating Membrane           

[snip]

The query trade.py market -vv "Cemplangpa/Gorgani Orbital" show all the available commodities but no selling price data is displayed:

    Item                      Buying     Avg     Demand Selling     Avg  Supply Age/Days
----------------------------------------------------------------------------------------
+CHEMICALS
    Explosives                                                                          
    Hydrogen Fuel                                                                       
    Hydrogen Peroxide            659   1,105     8,441H                             0.52
    Liquid Oxygen                598     441         0?                             0.52
    Mineral Oil                  418     472    28,960H                             0.52
    Surface Stabilisers                                                                 
    Water                        601     396       904H                             0.52

[snip]

The query trade.py market --buy -vv "Cemplangpa/Gorgani Orbital" does not seem to be any different from the previous query except that the "Selling", "Ave" and "Supply" headers are not there as would be expected.

    Item                      Buying     Avg     Demand Age/Days
----------------------------------------------------------------
+CHEMICALS
    Explosives                                                  
    Hydrogen Fuel                                               
    Hydrogen Peroxide            659   1,105     8,441H     0.52
    Liquid Oxygen                598     441         0?     0.52
    Mineral Oil                  418     472    28,960H     0.52
    Surface Stabilisers                                         
    Water                        601     396       904H     0.52

[snip]

As with the market sell query I would expect that the market buy query would only show those commodities that are bought and not all the commodities.

StationItem has a foreign key to Item with the definition FOREIGN KEY (item_id) REFERENCES Item(item_id) but Item(item_id) is not unique within Item.
Item has UNIQUE (item_id, fdev_id) and not even a separate unique index on item_id.

It seems to be like that in reality it doesn't matter, until you try to export --all-tables or --table=StationItem and it's generally a bad practice to have a foreign key that is not unique by primary key or unique index.

When the exportcsv.exportTableToFile generates the sqlStmt it ends up with...

 SELECT Station.station_id,Item.item_id,Item.fdev_id,StationItem.demand_price,StationItem.demand_units,StationItem.demand_level,StationItem.supply_price,StationItem.supply_units,StationItem.supply_level,StationItem.modified,StationItem.from_live FROM StationItem INNER JOIN Station USING(station_id) INNER JOIN Item USING(item_id) INNER JOIN Item USING(item_id) ORDER BY Station.station_id,Item.item_id,Item.fdev_id

Please note that there are 2 identical INNER JOIN Item USING(item_id) which definitely would generate ambigous column names.

Why did this happen? Why add restrictive copyleft provisions? Also, technically, you have to get permission from EVERY contributor to change the license, unless you can show their contributions are no longer in the current state of the system. I don't this happened, so I'm not sure at this point that the TD system is properly licensed with any restrictions.

I only ask because my first "run" started with selling at a 4k loss with the return trip purchasing a commodity not sold at that station I tried updating the station info but the route was unchanged

input:
trade run --cap 96 --cr 13m --from "diaguandri/ray gateway" --loop --ly-per 20 -v

output:
Diaguandri/Ray Gateway -> Diaguandri/Ray Gateway (score: 509872.828430)
Load from Diaguandri/Ray Gateway: 95 x Chemicals/Agronomic Treatment (@4669cr), 1 x Textiles/Military Grade Fabrics (@2241cr),
Dock at Wolf 1297/Ivins Depot
Load from Wolf 1297/Ivins Depot: 96 x Metals/Gold (@8986cr),
Dock at Diaguandri/Ray Gateway
Finish Diaguandri/Ray Gateway + 1,018,583cr (5,305cr/ton)=> 14,018,583cr

actual:

#! trade.py import -
# Created by E:D Market Connector 4.0.3 on Windows for Cmdr facadekitsune.
#
#    <item name>             <sellCR> <buyCR>   <demand>   <stock>  <timestamp>

@ Wolf 1297/Ivins Depot
   + Chemicals
      Tritium                   42314   42776          ?       75L  2020-07-25 16:42:21
   + Machinery
      Power Generators           1781       0     38300H         -  2020-07-25 16:42:21
      Water Purifiers            2043       0     13668H         -  2020-07-25 16:42:21
   + Technology
      H.E. Suits                 1987       0    301200H         -  2020-07-25 16:42:21
   + Waste
      Chemical Waste               74       0       603H         -  2020-07-25 16:42:21
      Scrap                        78       0      1059H         -  2020-07-25 16:42:21

Platform: TD 10.6.1 via pip, Python 3.8.0, Windows 10 18362.535

When doing an initial import, the download of the file http://elite.tromador.com/files/listings.csv never proceeds. That is, the last output on the console is: NOTE: Requesting https://elite.tromador.com/files/listings.csv, the download progress bar never appears, not even after hours of waiting.

I tested downloading the file locally, and the file is available and downloads normally, so this doesn't appear to be an issue on the Tromador site.

I was able to work around downloading this by forcing EDDB to be used for the initial download (set TD_SERVER=https://eddb.io/archive/v6/), but this results in a 404 when the -live version of the listings.csv is requested. However, once the first listings.csv is downloaded,, the variable can be unset, allowing the -live file to be sourced from Tromador, and finally the import is complete.

Not really a TD bug, but it's missing from index.json so we need a temp workaround.

Would you mind doing it? I'm up to my eyeballs converting apache 2.2 config to 2.4 so we can migrate to the new server.

Does Python support SNI? (I believe yes per https://en.wikipedia.org/wiki/Server_Name_Indication but want to run it by you first).

Assuming it does -

Current best practice is to connect via SSL for all web traffic. Can TD be configured to connect to my server via https, rather than http?

I would suggest leaving the fallback unchanged as we don't have knowledge of the config of that server. (This is currently https anyway, so non-issue).

Even after fixing the exponentiation, I get the following error in the test case:

F:\Elite\TD>trade.py run -vvv --progress --summary --avoid imperial,slaves --insurance 1.51m --pad L --ly 15.06 --cap 264 --no-planet --credits 24m --from "tollan/gord" --to "LHS 2094/Patt" --hops 3
* Hop   1: .........1 origins
NOTE: Pruned 23 origins too far from any end stations
* Hop   2: ........63 origins .. 33,528-382,536cr gain, 127-1,449cr/ton
[=====================    ] Traceback (most recent call last):
  File "F:\Elite\TD\trade.py", line 104, in <module>
    main(sys.argv)
  File "F:\Elite\TD\trade.py", line 77, in main
    results = cmdenv.run(tdb)
  File "F:\Elite\TD\commands\commandenv.py", line 81, in run
    return self._cmd.run(results, self, tdb)
  File "F:\Elite\TD\commands\run_cmd.py", line 1220, in run
    newRoutes = calc.getBestHops(routes, restrictTo=restrictTo)
  File "F:\Elite\TD\tradecalc.py", line 979, in getBestHops
    penalty = (-1 + 1 / (cruiseKls + 1) ** ((cruiseKls + 1) / 4)) / 2
OverflowError: (34, 'Result too large')

Possibly solved by going to logs. Will look into it if not fixed by this evening.

When running the command trade import --plug=eddblink -O clean I get to inserting new listing data and get a Foreign Key Constraint error.
I've double checked and my TD is up to date (10.3) installed and updated using pip

console output

To avoid further coding style issues, some kind of linting should be done.

Tested flake8, pylint and pep8 but all those report a lot of errors.
One could perhaps get some of them to work with some configuration.

I would like to have someway to do a proper install of this and if entry_points are defined one could get, for example, a trade.exe that works on windows.
If the package were published on pypi it could even be installed by pip (yay).

I can imagine that this will be difficult without moving code into a namespace for the package.

This is a wish from my side and a question if you would like some help with it.

Just updated to latest commit (24f26c6) and tried importing data and got the following error:

E:\Elite\TD>trade import -P eddblink
Traceback (most recent call last):
  File "E:\Elite\TD\trade.py", line 35, in <module>
    from tradedangerous.core import main
ModuleNotFoundError: No module named 'tradedangerous.core'

See the following excerpt from my TradeDangerous.prices. There are no slaves presently for sale at this station (HR 8208/Sturgeon Station)

   + Slavery
      Imperial Slaves                 17152   17510          ?   139659M  2018-09-17 13:02:48
      Slaves                          11039   11650          ?    54801M  2018-09-15 18:03:56

My best guess is that because "Slaves" is a subset of "Imperial Slaves" it did not get deleted when the new station data was uploaded, you can see the disparity of date/time information.

I'm working with a large multi-route calculator and recently it looks like the ls-penalty is no longer operating (in the last 6 months or so).

Example: This call...
trade.py run --fr Bonitou --to Gateway/Dublin --cap 100 --credits 50000000 --ly 16.38 --empty-ly 20 --start-jumps 0 --end-jumps 0 --hops 3 --jumps 3 --pad L --progress --no-planet --ls-penalty 5 --supply 5000 --demand 5000 --avoid Tobacco,Imperial,Narcotics,Slaves,@Shinrarta,"@plx 695", @Exbeur,Alkaid/Camus,Insulating,"@sol" --age 3 -vvv

Gives this station as stop 1:
Unload at Taliesin/Dunbar Platform (4.47Kls, BMk:N, Pad:L, Plt:N, Shp:Y,
With a 4.47Kls station distance.

This previously would never have occurred with ls-penalty = 5

If I then increase ls-penalty in magnitude to 50, 500, 5000, and 50000 none of these have any effect on the route at all.
Formerly an ls-penalty of 50 or 500 would have quickly killed any route with a stop of more than 50ls or so.

Note: "--ls-max 1500" still works perfectly well and the route changes when used, however I preferred ls-penalty in case of major profit just above the ls-max range limit.

Other thoughts:
Was this a previous bug that was fixed? I'm wondering if I'm completely up to date with the latest version. If so, perhaps check in current version, and, if not found, please disregard.

Obviously my first thought was that it's Willy's server, but I can browse that URL both from my PC and using lynx from the shell on the server, so the file is available, but for some reason, not when TD requests it.

I can only note that when I browse it via lynx, I'm seeing it as an x-gzip content type. Don't know if that helps. Otherwise, let me know what tests I can run.

Meantime, as there is the fallback ship index, I'll let TD/listener run from that.

NOTE: Checking for update to 'index.json'.
WARNING: Problem with download:
URL: https://beta.coriolis.io/data/index.json
Error: HTTP Error 403: Forbidden
NOTE: Using Default Ship Index.

But (e.g.) wget on same host/user/shell et al.

[elite@quoth ~]$ wget https://beta.coriolis.io/data/index.json
--2019-05-23 04:56:56--  https://beta.coriolis.io/data/index.json
Resolving beta.coriolis.io (beta.coriolis.io)... 104.24.116.234, 104.24.117.234, 2606:4700:30::6818:74ea, ...
Connecting to beta.coriolis.io (beta.coriolis.io)|104.24.116.234|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 869197 (849K) [application/json]
Saving to: ‘index.json’

100%[======================================>] 869,197     1.84MB/s   in 0.4s

2019-05-23 04:56:57 (1.84 MB/s) - ‘index.json’ saved [869197/869197]

On reflection I guess could be @willyb321 but only if his server is detecting and blocking the specific type of web request TD is using, but it does seem strange that other requests work perfectly.

The GUI seems to run fine, but threw the following errors (Win 10 x64, Python 3.7.3):

2019-06-29 17:31:53,019 appJar:ERROR [Line 9->2738/_fontHelper]: Failed to adjust inputFont to Courier.
2019-06-29 17:31:53,020 appJar:ERROR [Line 9->2738/_fontHelper]: Failed to adjust labelFont to Courier.
2019-06-29 17:31:53,022 appJar:ERROR [Line 9->2738/_fontHelper]: Failed to adjust buttonFont to Courier.
2019-06-29 17:31:53,057 appJar:WARNING [Line 9->4254/_removeContainer]: Closing empty container: req
2019-06-29 17:31:53,061 appJar:WARNING [Line 9->4254/_removeContainer]: Closing empty container: opt
2019-06-29 17:31:53,067 appJar:ERROR [Line 9->2556/configure]: Invalid config parameter: disabled, horizontal

Enable multiprocessing for calculations to run faster on multi-core CPUs.
Thought of this when running for trade routes with 500 LY jump range to see if FC trading is profitable.

Implementing this in python is an absolute PITA so understandable if wontfix or delayed indefinitely.

Olddata parameter does not work with --ls-max and --no-planet parameters.

When you trade with the biggest ships and want to make those real credits, its done within <1000ls stations and no planetary landings.

I would like to search oldest market prices and go update them, kind of like rejuvenating the markets, but at current database and without these parameters available... its just endless list of >1000ls stations and/or planetary outposts.

For reference: http://www.davek.com.au/td/editstation.asp allows you to filter ls distance (or missing distance) and planetary.

Minor (very minor) issue with the layout of a navigation command when including stations.

Alignment issues with "Shinrarta Dezhra/Gallum's Inheritance" as the name it too long for the layout as shown on the attached screen shot.

Running a query on the latest TD database gives a max system name length of 30 (Thor's Helmet Sector FB-X c1-5) and station name of 37 (Hipparque - Cartographers' University) ignoring the ones that are to be deleted. So the space for the station name probably needs to be expanded to allow a total name length of 5 + 30 + 1 + 37 + 1 or 73 which will probably overflow everything.

Alternatively remove the redundant system name before each station name.

I've just come back to Elite after a long time off, and pulled the latest copy of TD. I'm trying to get the edapi plugin to work, as that was my preferred method of pulling data for my current station. It's not obvious if it's still intended to work or not - I've found a few forks of the EDAPI tool itself, but even the latest one I can find won't actually let me log in.

Did all these tools break when they added the in-game trade tools?

Command line:
trade.py sell --near="Orang/Bessel Gateway" --ly=100 --pad=L --limit=42 -vv "Advanced Medicines, Basic Medicines"

Error report:
Traceback (most recent call last):
File "C:\Development\tradedangerous\trade.py", line 107, in <module>
main(sys.argv)
File "C:\Development\tradedangerous\trade.py", line 80, in main
results = cmdenv.run(tdb)
File "C:\Development\tradedangerous\commands\commandenv.py", line 81, in run
return self._cmd.run(results, self, tdb)
File "C:\Development\tradedangerous\commands\sell_cmd.py", line 78, in run
item = tdb.lookupItem(cmdenv.item)
File "C:\Development\tradedangerous\tradedb.py", line 1932, in lookupItem
val=lambda kvTup: kvTup[1]
File "C:\Development\tradedangerous\tradedb.py", line 2109, in listSearch
"Error: '%s' doesn't match any %s" % (lookup, listType)
LookupError: Error: 'Advanced Medicines, Basic Medicines' doesn't match any Item

Using the sell query with a single item works correctly.

Indentation

• Use 4 spaces per indentation level.

Suggested Maximum Line Length

• Limit all lines to a maximum of 79 characters. Feel free to ignore if inconvenient.

Blank Lines

• Blank lines must have the same indentation level as the immediately following non-blank line. This is an exception to the "No trailing whitespace" rule.

Whitespace in Expressions and Statements

Avoid extraneous whitespace in the following situations:

• Immediately inside parentheses, brackets or braces.

    Yes: spam(ham[1], {eggs: 2})
    No:  spam( ham[ 1 ], { eggs: 2 } )

• Between a trailing comma and a following close parenthesis.

    Yes: foo = (0,)
    No:  bar = (0, )

• Immediately before a comma, semicolon, or colon:

    Yes: if x == 4: print x, y; x, y = y, x
    No:  if x == 4 : print x , y ; x , y = y , x

• Immediately before the open parenthesis that starts the argument list of a function call:

    Yes: spam(1)
    No:  spam (1)

• Immediately before the open parenthesis that starts an indexing or slicing:

    Yes: dct['key'] = lst[index]
    No:  dct ['key'] = lst [index]

• More than one space around an assignment (or other) operator to align it with another.

  Yes:

    x = 1
    y = 2
    long_variable = 3

No:

    x             = 1
    y             = 2
    long_variable = 3

For everything else you might think of, refer to PEP8 as completely optional style advice except where it contradicts the above.

Only for your knowledge that I've installed the python 3.7 64bits and TD is not running with a message:
Requested Python version (3.6) is not installed

Regards!

Do we need to do some manual adding of ships after all?

trade.py import -P eddblink -O clean -w
.
.
.
NOTE: Simultaneously processing UpgradeVendors, this will take quite a while.
# ABUKUNIN/Reilly Hub has been added:
# ALRAI/Bounds Hub has been added:
# ANGLII/Starzl Plant has been added:
# BETA TRIANGULI AUSTRALIS/Lunan Hub has been added:
# BETA TRIANGULI AUSTRALIS/Hutchinson City has been added:
# BANDJIGALI/Teng-hui Station has been added:
# 39 TAURI/Porta has been added:
# AGARLA/Wilson Port has been added:
# BETA TRIANGULI AUSTRALIS/Vinge Gateway has been added:
# APATHAAM/Reamy Dock has been added:
# BRANI/Virtanen Hub has been added:
# 85 PEGASI/Buckland Gateway has been added:
# BAGAI/Humphreys' Progress has been added:
# AIABIKO/Maxwell Orbital has been added:
# AIABIKO/Gooch Terminal has been added:
# BHUTAS/Feynman Terminal has been added:
# AMISI/Sargent Escape has been added:
# AMISI/Brooks' Folly has been added:
# 78 URSAE MAJORIS/Townshend Hub has been added:
# BRANI/Akiyama Market has been added:
# ALLOWA/Moore Ring has been added:
# 85 PEGASI/Borman Terminal has been added:
# ANLAVE/Suri Park has been added:
# 44 B OPHIUCHI/Feustel Gateway has been added:
# AULIN/Aulin Enterprise has been added:
# AMAHU/Kondratyev Orbital has been added:
# AMARAK/Vela Port has been added:
# ATFERO/Patry Dock has been added:
# BLODER/Szebehely Station has been added:
# BHUMLIAURT/Lichtenberg Port has been added:
# BD+69 530/Burbank Gateway has been added:
# 78 URSAE MAJORIS/Read Gateway has been added:
# BD+69 530/Leoniceno Orbital has been added:
# 31 AQUILAE/Russo Dock has been added:
# AMIJARA/Shepherd Outpost has been added:
# BIDIONO/Wellman Dock has been added:
# APAM NAPAT/Langley Hub has been added:
# AGUANA/Webb Dock has been added:
# ANLAVE/Bain Dock has been added:
# ANLAVE/Hogg City has been added:
# BD+55 1519/Arber Colony has been added:
# AMALANGKAN/Doi Landing has been added:
# AREXE/Janes Horizons has been added:
# BELU/Covey Gateway has been added:
# BINAR/Ferguson Dock has been added:
# BRANI/Barsanti Enterprise has been added:
# BAKONIRTARI/Gregory Vision has been added:
# BRANI/Noakes Port has been added:
# ANDRUNGARRI/Disch Port has been added:
# BETA-1 TUCANAE/Adams Orbital has been added:
# 42 AQUILAE/Born Station has been added:
# AVIK/Bruce Prospect has been added:
# ATLANTIS/Kimura Terminal has been added:
# 42 AQUILAE/Ride Market has been added:
# 78 URSAE MAJORIS/Teller Terminal has been added:
# 110 HERCULIS/Jahn Hangar has been added:
# BALTAH'SINE/Baltha'Sine Station has been added:
# AHAUT/Goldberg Orbital has been added:
# ALTAIR/Solo Orbiter has been added:
# 78 URSAE MAJORIS/Rominger Dock has been added:
# ANLAVE/Brunton Gateway has been added:
# ANLAVE/Kobayashi City has been added:
# ALCOR/Birmingham Dock has been added:
# BD-04 4138/Shunn Dock has been added:
# BRANGLAL/Gibson Horizons has been added:
# BD+47 1236/Dana Relay has been added:
# BPM 89444/Szameit Ring has been added:
# ALIOTH/Irkutsk has been added:
Traceback (most recent call last):
  File "D:\Games\Game Tools\Elite\Trade Dangerous\trade.py", line 104, in <module>
    main(sys.argv)
  File "D:\Games\Game Tools\Elite\Trade Dangerous\trade.py", line 77, in main
    results = cmdenv.run(tdb)
  File "D:\Games\Game Tools\Elite\Trade Dangerous\commands\commandenv.py", line 81, in run
    return self._cmd.run(results, self, tdb)
  File "D:\Games\Game Tools\Elite\Trade Dangerous\commands\import_cmd.py", line 124, in run
    if not plugin.run():
  File "D:\Games\Game Tools\Elite\Trade Dangerous\plugins\eddblink_plug.py", line 795, in run
    self.importStations()
  File "D:\Games\Game Tools\Elite\Trade Dangerous\plugins\eddblink_plug.py", line 430, in importStations
    modified))
  File "D:\Games\Game Tools\Elite\Trade Dangerous\plugins\eddblink_plug.py", line 95, in execute
    result = cur.execute(sql_cmd, args)
sqlite3.IntegrityError: NOT NULL constraint failed: ShipVendor.ship_id

As per our hangouts discussion:

TD presently requires a content-length header or it bombs the download, per forum example from Gurunot.

This is likely caused by mod_deflate chunking up the data, so it cannot know how big the file is and so refrains from sending any content-length header at all.

There is no reliable/easy fix in the server configuration, so please workaround in transfers.py

In fact, the only clear fix for server would be to have files already gzipped, then use mod_rewrite to send that data directly (but pretend it's not sending a gzipped file). This can be done, but will still require (probably more) client side coding that a fix in transfers.py.

pull-requests should only be tested
This causes builds to show up as failed when semantic-release fails

Please can we change

BASE_URL = "http://elite.ripz.org/files/"

to

BASE_URL = "http://elite.tromador.com/files/"

in the next release. I am deprecating my services on the ripz.org domain and am in two minds as to whether or not I should keep paying for it. No particular rush, the site will respond on both urls for the forseeable future.

I'd push this myself (minor change), but as I'm unclear how git pushes (or is pulled from) downstream to pip. I don't really want to mess with the source tree at the moment.

In order to figure out a good value for the default lsp with the new formula, we need to collect data. Lots of data.

Yummy yummy data.

Submissions should include the following:

The route returned by TD at lsp == 0.

The lsp values at which the route returned by TD, with all other terms in the run command the same, is different than the lsp=0 route. More precise values are better (lsp=3.96 is more helpful than lsp=4).

Every route returned by TD where all commands except '--lsp' are the same, with the '--lsp' range in which that particular route is returned by TD. More precise values are better ('lsp<=3.96' is more helpful than 'lsp<=4'). It would also be helpful to have the score returned at the min and max lsp values for the route, although that's not as important.

Please don't fix this issue :)

running trade.py import -P eddblink -O help returns an error:

Traceback (most recent call last):
  File "trade.py", line 107, in <module>
    main(sys.argv)
  File "trade.py", line 80, in main
    results = cmdenv.run(tdb)
  File "S:\Elite Dangerous Programs\Trade Dangerous\eddblink-listener\commands\commandenv.py", line 81, in run
    return self._cmd.run(results, self, tdb)
  File "S:\Elite Dangerous Programs\Trade Dangerous\eddblink-listener\commands\import_cmd.py", line 119, in run
    plugin = pluginClass(tdb, cmdenv)
  File "S:\Elite Dangerous Programs\Trade Dangerous\eddblink-listener\plugins\eddblink_plug.py", line 62, in __init__
    super().__init__(tdb, tdenv)
  File "S:\Elite Dangerous Programs\Trade Dangerous\eddblink-listener\plugins\__init__.py", line 188, in __init__
    super().__init__(tdb, tdenv)
  File "S:\Elite Dangerous Programs\Trade Dangerous\eddblink-listener\plugins\__init__.py", line 52, in __init__
    raise SystemExit(self.usage())
  File "S:\Elite Dangerous Programs\Trade Dangerous\eddblink-listener\plugins\__init__.py", line 95, in usage
    text += tw.fill(options[opt].strip()) + "\n"
  File "C:\Python37\lib\textwrap.py", line 363, in fill
    return "\n".join(self.wrap(text))
  File "C:\Python37\lib\textwrap.py", line 354, in wrap
    return self._wrap_chunks(chunks)
  File "C:\Python37\lib\textwrap.py", line 314, in _wrap_chunks
    lines.append(indent + ''.join(cur_line))
TypeError: can only concatenate tuple (not "str") to tuple

For some reason the TextWrap module is splitting the string it's being passed into tuples?

In the setup guide you explain how to setup the initial database but there's a missing - in the code which causes an error even when cut and paste.

Existing text says:
$ trade.py import --plug=eddblink -opt=clean,skipvend

Needs to be:
$ trade.py import --plug=eddblink --opt=clean,skipvend

Command being run: trade import -P edapi -O tdh

The EDAPI plugin is not authorised when the command is run and the error received is pasted below. The Frontier Website opens but hangs with the address bar showing 127.2.0.1.

In order for this to work on Mac OS you have to issue the following command in the terminal:

sudo ifconfig lo0 alias 127.2.0.1

Doing this allows the system to work but in order to make the change persistent you have to do a whole heap of things (see the answer to the question posed here)

Does the address have to be 127.2.0.1?

Could it not be 127.0.0.1 which would avoid the problem?

Thanks

Traceback (most recent call last):
File "/usr/local/bin/trade", line 10, in
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/tradedangerous/cli.py", line 70, in main
trade(argv)
File "/usr/local/lib/python3.7/site-packages/tradedangerous/cli.py", line 125, in trade
results = cmdenv.run(tdb)
File "/usr/local/lib/python3.7/site-packages/tradedangerous/commands/commandenv.py", line 81, in run
return self._cmd.run(results, self, tdb)
File "/usr/local/lib/python3.7/site-packages/tradedangerous/commands/import_cmd.py", line 122, in run
if not plugin.run():
File "/usr/local/lib/python3.7/site-packages/tradedangerous/plugins/edapi_plug.py", line 743, in run
debug = tdenv.debug,
File "/usr/local/lib/python3.7/site-packages/tradedangerous/plugins/edapi_plug.py", line 138, in init
self.profile = self.query_capi("/profile")
File "/usr/local/lib/python3.7/site-packages/tradedangerous/plugins/edapi_plug.py", line 164, in query_capi
self._authorization_check()
File "/usr/local/lib/python3.7/site-packages/tradedangerous/plugins/edapi_plug.py", line 197, in _authorization_check
status_ok = self._authorization_login()
File "/usr/local/lib/python3.7/site-packages/tradedangerous/plugins/edapi_plug.py", line 260, in _authorization_login
oauth = OAuthCallbackServer(redirect_uri.hostname, redirect_uri.port, OAuthCallbackHandler)
File "/usr/local/lib/python3.7/site-packages/tradedangerous/plugins/edapi_plug.py", line 70, in init
self.httpd = myServer((hostname, port), handler)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socketserver.py", line 452, in init
self.server_bind()
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/server.py", line 137, in server_bind
socketserver.TCPServer.server_bind(self)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socketserver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 49] Can't assign requested address

Hi folks.

Just getting to grips with this - seems very useful so far.

Am wondering if there's a reason why commands like "buy" don't have a --ls-max option?
If not, I could have a go at patching it in myself...

I downloaded a fresh copy of TD and was running a clean install when I got the following error:

F:\>cd elite/td

F:\Elite\TD>trade import -P eddblink -O clean
NOTE: Rebuilding cache file: this may take a few moments.
NOTE: Missing "F:\Elite\TD\data\TradeDangerous.prices" file - no price data.
NOTE: Downloading file 'modules.json'.
NOTE: Requesting http://elite.ripz.org/files/modules.json
NOTE: Downloaded 292.3KB of gziped data 510.5KB/s
NOTE: Processing Upgrades: Start time = 2018-06-25 19:56:17.902780
NOTE: Finished processing Upgrades. End time = 2018-06-25 19:56:17.913750
NOTE: Downloading file 'index.json'.
NOTE: Requesting https://raw.githubusercontent.com/EDCD/coriolis-data/master/dist/index.json
NOTE: Downloaded   0.7MB of gziped data  10.9MB/s
NOTE: Processing Ships: Start time = 2018-06-25 19:56:18.237884
NOTE: Finished processing Ships. End time = 2018-06-25 19:56:18.244865
NOTE: Downloading file 'systems_populated.jsonl'.
NOTE: Requesting http://elite.ripz.org/files/systems_populated.jsonl
NOTE: Downloaded  20.5MB of gziped data   1.1MB/s
NOTE: Processing Systems: Start time = 2018-06-25 19:56:37.701863
NOTE: Finished processing Systems. End time = 2018-06-25 19:56:38.979448
NOTE: Downloading file 'stations.jsonl'.
NOTE: Requesting http://elite.ripz.org/files/stations.jsonl
NOTE: Downloaded 113.5MB of gziped data   3.2MB/s
NOTE: Processing Stations, this may take a bit: Start time = 2018-06-25 19:57:15.028101
NOTE: Simultaneously processing ShipVendors.
NOTE: Simultaneously processing UpgradeVendors, this will take quite a while.
NOTE: Finished processing Stations. End time = 2018-06-25 20:01:56.025085
NOTE: Downloading file 'commodities.json'.
NOTE: Requesting http://elite.ripz.org/files/commodities.json
NOTE: Downloaded  99.3KB of gziped data 213.7KB/s
NOTE: Processing Categories and Items: Start time = 2018-06-25 20:01:57.189972
NOTE: Finished processing Categories and Items. End time = 2018-06-25 20:01:57.194958
NOTE: F:\Elite\TD\data\Category.csv exported.
NOTE: F:\Elite\TD\data\Item.csv exported.
NOTE: F:\Elite\TD\data\Ship.csv exported.
NOTE: F:\Elite\TD\data\ShipVendor.csv exported.
NOTE: F:\Elite\TD\data\Station.csv exported.
NOTE: F:\Elite\TD\data\System.csv exported.
NOTE: F:\Elite\TD\data\Upgrade.csv exported.
NOTE: F:\Elite\TD\data\UpgradeVendor.csv exported.
NOTE: Downloading file 'listings.csv'.
NOTE: Requesting http://elite.ripz.org/files/listings.csv
NOTE: Downloaded 149.2MB of gziped data   4.0MB/s
NOTE: Processing market data from listings.csv: Start time = 2018-06-25 20:04:05.625704
Traceback (most recent call last):
  File "F:\Elite\TD\trade.py", line 104, in <module>
    main(sys.argv)
  File "F:\Elite\TD\trade.py", line 77, in main
    results = cmdenv.run(tdb)
  File "F:\Elite\TD\commands\commandenv.py", line 81, in run
    return self._cmd.run(results, self, tdb)
  File "F:\Elite\TD\commands\import_cmd.py", line 124, in run
    if not plugin.run():
  File "F:\Elite\TD\plugins\eddblink_plug.py", line 814, in run
    self.importListings(self.listingsPath)
  File "F:\Elite\TD\plugins\eddblink_plug.py", line 649, in importListings
    supply_price, supply_units, supply_level, from_live))
UnboundLocalError: local variable 'from_live' referenced before assignment

I get the following error:

ERROR Unrecognized item name: "AGRONOMICTREATMENT"

BD-01 1707.Khan Port.2020-04-18T21.54.10.prices.zip

If I run the command:

buy --near="Orang/Bessel Gateway" --ly=24 --pad=L --lim=0 -vv -P "Antiquities"

I get the following error:

Traceback (most recent call last):
[snip]
File "buy_cmd.py", line 245, in run
results.summary.avg = int(avgPrice)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

If I change the commodity to something else I do not get that error. For example:

buy --near="Orang/Bessel Gateway" --ly=24 --pad=L --lim=0 -vv -P "Aquaponic Systems"

I have not tried this with all the commodities or ships, just a handful and Antiquities is to only one that gives this error although there could be others.

I have the latest TD and updated the DB earlier today.

According to https://eddb.io/api the old v5 URLs are phased out very rapidly, and the old files are not updated anymore. It is used as a backup URL in the eddb plugin.
It also seems that http://elite.ripz.org/ is missing some important commodities (Void Opals for instance). These are present when the import is done from the v6 EDDB API, and seems to work as intended.

Currently, TD checks for numpy by using os.environ tradedb.py line 72. This doesn't always work since NUMPY can be installed without the string NUMPY being in the return of os.environ. However, that if isn't needed, since if numpy isn't installed, an ImportError is thrown, and that's caught in the try block anyway. As of now, NUMPY isn't really used (see below), but for correctness it should be fixed. Also, it may pay to do some timing tests, as for those of us with an installed numpy using a fast BLAS/LAPACK, the linalg.norm may be faster than the sqrt in the distanceTo function. I will provide a pull request with the correction shortly. Thank you,

E:\Elite\TD>grep -lr "all_distances"
tradedb.py
__pycache__/tradedb.cpython-36.pyc
__pycache__/tradedb.cpython-37.pyc

Add an option to ignore Fleet Carriers similar to --no-planet.

The progress bar in EDDBlink is wide and has a percent indicator at the end of the bar. In trade dangerous, however, the bar is small and has not percentage indicator.

Perhaps it would be good to make the TD progress bar match the EDDBlink bar.

I've been thinking of deploying Trade-Dangerous as an AWS Lambda service. It would be a really lightweight way to run it as a backend for any UI / app to consume. However, it wouldn't do to have the database being seeded on every invoke. I'm wondering if/how I could move TradeDangerous.db to cloud storage (S3 bucket), then do remote read/writes? That would allow the client to deploy quickly and scale to multiple instances reading from a single source.

I'm assuming it would be a case of replacing hard-coded references to the db file as a variable that can be overridden from a command line arg? Is there more to it, regarding the data dir setup or sqlite's ability to connect over http instead of a local file?

Hi, I'd like to ingest the output of Trade Dangerous in another tool. Would you be interested in adding a command flag to output results as JSON instead of the row format?

I can look at doing this in my own fork, but I'm not familiar enough with Python to be confident with a PR.

I'm getting an error on calling the edapi plugin as pasted below. The command is being run from a console window in the same folder as the TD data and tmp folders. I have updated TD to the latest version via pip. I have cut out the path to the python folder to improve readability.

This is with a fresh install of TD.

Thanks.

Error log:

C:\Development\TDHelper\TradeDangerousGUI\bin\Debug>trade import -P edapi -O tdh
Traceback (most recent call last):
File "...\python37\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "...\python37\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "...\Python37\Scripts\trade.exe\__main__.py", line 9, in <module>
File "...\python37\lib\site-packages\tradedangerous\cli.py", line 70, in main
trade(argv)
File "...\python37\lib\site-packages\tradedangerous\cli.py", line 125, in trade
results = cmdenv.run(tdb)
File "...\python37\lib\site-packages\tradedangerous\commands\commandenv.py", line 81, in run
return self._cmd.run(results, self, tdb)
File "...\python37\lib\site-packages\tradedangerous\commands\import_cmd.py", line 122, in run
if not plugin.run():
File "...\python37\lib\site-packages\tradedangerous\plugins\edapi_plug.py", line 748, in run
if tdh_path.exists():
AttributeError: 'str' object has no attribute 'exists'

The initial import of StationItems take forever.... Could it be speed up?

When I ran the initial update from eddb I got

...
NOTE: Processing market data from listings.csv: Start time = 2019-01-03 15:37:54.424876
NOTE: Finished processing market data. End time = 2019-01-03 15:58:23.628491
...

Processing market data took 21 minutes when starting from "empty" data folder.

Since the command line sqlite3 tool have support for importing csv files I made some tests, see attached script import-help.sql.txt .

2019-01-03 15:35:06|Importing eddb/listings.csv
2019-01-03 15:35:23|Generating Indexes
2019-01-03 15:35:29|Fixing brackets
2019-01-03 15:35:31|Creating 'simplified' StationItem table
2019-01-03 15:35:31|Insert or replace StationItems
2019-01-03 15:35:54|Done!

48 seconds......!
I ran this by doing rm test.db & sqlite3 test.db ".read import-help.sql.txt" in the Trade-Dangerouse folder after listings etc had been downloaded.

After hacking the code in eddblink_plug and importing using the sqlite3 command+script for listings.csv but not creating a modified StationItem table I got.

...
NOTE: Processing market data from listings.csv: Start time = 2019-01-03 17:23:15.991480
2019-01-03 16:23:16|Importing eddb/listings.csv
2019-01-03 16:23:37|Generating Indexes
2019-01-03 16:23:46|Fixing brackets
2019-01-03 16:23:50|Insert or replace StationItems
2019-01-03 16:27:32|Done!
NOTE: Finished processing market data. End time = 2019-01-03 17:27:32.298064
...

Worse with all the foreign key checking and existing related data but ~5 minutes is way better than 21.
(The 1 hour difference in timestamps depends on me being UTC+1)

Is there something I'm missing?

OS: Windows 10, 1809
CPU: i7-8650U
Python 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32