eyeonus / trade-dangerous Goto Github PK
View Code? Open in Web Editor NEWLicense: Mozilla Public License 2.0
License: Mozilla Public License 2.0
I know we previously discussed this, but given the change to a pip model, I think we should switch up from this hardcoded
self.dataPath = tdb.dataPath / Path("eddb")
to having a default (as above) but overridable with a $ENV. It will still only be relevant for server operation, I would think, but with automated upgrades, I think it's the right way to deal with it from now on.
I would like to have someway to do a proper install of this and if entry_points are defined one could get, for example, a trade.exe that works on windows.
If the package were published on pypi it could even be installed by pip (yay).
I can imagine that this will be difficult without moving code into a namespace for the package.
This is a wish from my side and a question if you would like some help with it.
I only ask because my first "run" started with selling at a 4k loss with the return trip purchasing a commodity not sold at that station I tried updating the station info but the route was unchanged
input:
trade run --cap 96 --cr 13m --from "diaguandri/ray gateway" --loop --ly-per 20 -v
output:
Diaguandri/Ray Gateway -> Diaguandri/Ray Gateway (score: 509872.828430)
Load from Diaguandri/Ray Gateway: 95 x Chemicals/Agronomic Treatment (@4669cr), 1 x Textiles/Military Grade Fabrics (@2241cr),
Dock at Wolf 1297/Ivins Depot
Load from Wolf 1297/Ivins Depot: 96 x Metals/Gold (@8986cr),
Dock at Diaguandri/Ray Gateway
Finish Diaguandri/Ray Gateway + 1,018,583cr (5,305cr/ton)=> 14,018,583cr
actual:
#! trade.py import -
# Created by E:D Market Connector 4.0.3 on Windows for Cmdr facadekitsune.
#
# <item name> <sellCR> <buyCR> <demand> <stock> <timestamp>
@ Wolf 1297/Ivins Depot
+ Chemicals
Tritium 42314 42776 ? 75L 2020-07-25 16:42:21
+ Machinery
Power Generators 1781 0 38300H - 2020-07-25 16:42:21
Water Purifiers 2043 0 13668H - 2020-07-25 16:42:21
+ Technology
H.E. Suits 1987 0 301200H - 2020-07-25 16:42:21
+ Waste
Chemical Waste 74 0 603H - 2020-07-25 16:42:21
Scrap 78 0 1059H - 2020-07-25 16:42:21
I'm trying to get away from Windows. TDH is written in .NET, so that doesn't help.
Hence this effort.
Coders and Graphics Designer requested for assistance, feedback, etc.
As per our hangouts discussion:
TD presently requires a content-length header or it bombs the download, per forum example from Gurunot.
This is likely caused by mod_deflate chunking up the data, so it cannot know how big the file is and so refrains from sending any content-length header at all.
There is no reliable/easy fix in the server configuration, so please workaround in transfers.py
In fact, the only clear fix for server would be to have files already gzipped, then use mod_rewrite to send that data directly (but pretend it's not sending a gzipped file). This can be done, but will still require (probably more) client side coding that a fix in transfers.py.
Here is my command line and the result, see Gain/Hop line.
python3 trade.py run --cr 10m --cap 788 --empty 21.29 --ly 12.07 --pad l --age 2 -vvv --prog --ls-max 2000 --avoid alioth,minmar/acropolis,Aganippe/Julian --insurance 3804342 --gpt 1500 --no-planet
* Hop 1: .....2,371 origins
* Hop 2: .......988 origins .. 1,525-3,750,880cr gain, 1,500-4,760cr/ton
HIP 6796/Ibold Port -> HIP 6978/Sterling Station (score: 5277727.942143)
Start CR: 6,195,658
Hops : 2
Jumps : 3
Gain CR : 4,992,768
Gain/Hop: 2.49638e+06
Final CR: 11,188,426
Not a very big issue but that has annoyed me since the beginning.
Currently, TD checks for numpy by using os.environ
tradedb.py line 72. This doesn't always work since NUMPY can be installed without the string NUMPY being in the return of os.environ
. However, that if
isn't needed, since if numpy isn't installed, an ImportError is thrown, and that's caught in the try block anyway. As of now, NUMPY isn't really used (see below), but for correctness it should be fixed. Also, it may pay to do some timing tests, as for those of us with an installed numpy using a fast BLAS/LAPACK, the linalg.norm may be faster than the sqrt in the distanceTo
function. I will provide a pull request with the correction shortly. Thank you,
E:\Elite\TD>grep -lr "all_distances"
tradedb.py
__pycache__/tradedb.cpython-36.pyc
__pycache__/tradedb.cpython-37.pyc
Minor (very minor) issue with the layout of a navigation command when including stations.
Alignment issues with "Shinrarta Dezhra/Gallum's Inheritance" as the name it too long for the layout as shown on the attached screen shot.
Running a query on the latest TD database gives a max system name length of 30 (Thor's Helmet Sector FB-X c1-5) and station name of 37 (Hipparque - Cartographers' University) ignoring the ones that are to be deleted. So the space for the station name probably needs to be expanded to allow a total name length of 5 + 30 + 1 + 37 + 1 or 73 which will probably overflow everything.
Alternatively remove the redundant system name before each station name.
pull-requests should only be tested
This causes builds to show up as failed when semantic-release fails
Indentation
Suggested Maximum Line Length
Blank Lines
Whitespace in Expressions and Statements
Avoid extraneous whitespace in the following situations:
Yes: spam(ham[1], {eggs: 2})
No: spam( ham[ 1 ], { eggs: 2 } )
Yes: foo = (0,)
No: bar = (0, )
Yes: if x == 4: print x, y; x, y = y, x
No: if x == 4 : print x , y ; x , y = y , x
Yes: spam(1)
No: spam (1)
Yes: dct['key'] = lst[index]
No: dct ['key'] = lst [index]
More than one space around an assignment (or other) operator to align it with another.
Yes:
x = 1
y = 2
long_variable = 3
No:
x = 1
y = 2
long_variable = 3
For everything else you might think of, refer to PEP8 as completely optional style advice except where it contradicts the above.
Please can we change
BASE_URL = "http://elite.ripz.org/files/"
to
BASE_URL = "http://elite.tromador.com/files/"
in the next release. I am deprecating my services on the ripz.org domain and am in two minds as to whether or not I should keep paying for it. No particular rush, the site will respond on both urls for the forseeable future.
I'd push this myself (minor change), but as I'm unclear how git pushes (or is pulled from) downstream to pip. I don't really want to mess with the source tree at the moment.
I've been thinking of deploying Trade-Dangerous as an AWS Lambda service. It would be a really lightweight way to run it as a backend for any UI / app to consume. However, it wouldn't do to have the database being seeded on every invoke. I'm wondering if/how I could move TradeDangerous.db
to cloud storage (S3 bucket), then do remote read/writes? That would allow the client to deploy quickly and scale to multiple instances reading from a single source.
I'm assuming it would be a case of replacing hard-coded references to the db file as a variable that can be overridden from a command line arg? Is there more to it, regarding the data dir setup or sqlite's ability to connect over http instead of a local file?
StationItem
has a foreign key to Item
with the definition FOREIGN KEY (item_id) REFERENCES Item(item_id)
but Item(item_id)
is not unique within Item.
Item has UNIQUE (item_id, fdev_id)
and not even a separate unique index on item_id.
It seems to be like that in reality it doesn't matter, until you try to export --all-tables or --table=StationItem and it's generally a bad practice to have a foreign key that is not unique by primary key or unique index.
When the exportcsv.exportTableToFile generates the sqlStmt it ends up with...
SELECT Station.station_id,Item.item_id,Item.fdev_id,StationItem.demand_price,StationItem.demand_units,StationItem.demand_level,StationItem.supply_price,StationItem.supply_units,StationItem.supply_level,StationItem.modified,StationItem.from_live FROM StationItem INNER JOIN Station USING(station_id) INNER JOIN Item USING(item_id) INNER JOIN Item USING(item_id) ORDER BY Station.station_id,Item.item_id,Item.fdev_id
Please note that there are 2 identical INNER JOIN Item USING(item_id)
which definitely would generate ambigous column names.
Only for your knowledge that I've installed the python 3.7 64bits and TD is not running with a message:
Requested Python version (3.6) is not installed
Regards!
The query trade.py market --sell -vv "Cemplangpa/Gorgani Orbital" shows the selling commodities but not the price data:
Item Selling Avg Supply Age/Days
-----------------------------------------------------------
+CHEMICALS
Explosives
Hydrogen Fuel
Surface Stabilisers
+INDUSTRIAL MATERIALS
Insulating Membrane
[snip]
The query trade.py market -vv "Cemplangpa/Gorgani Orbital" show all the available commodities but no selling price data is displayed:
Item Buying Avg Demand Selling Avg Supply Age/Days
----------------------------------------------------------------------------------------
+CHEMICALS
Explosives
Hydrogen Fuel
Hydrogen Peroxide 659 1,105 8,441H 0.52
Liquid Oxygen 598 441 0? 0.52
Mineral Oil 418 472 28,960H 0.52
Surface Stabilisers
Water 601 396 904H 0.52
[snip]
The query trade.py market --buy -vv "Cemplangpa/Gorgani Orbital" does not seem to be any different from the previous query except that the "Selling", "Ave" and "Supply" headers are not there as would be expected.
Item Buying Avg Demand Age/Days
----------------------------------------------------------------
+CHEMICALS
Explosives
Hydrogen Fuel
Hydrogen Peroxide 659 1,105 8,441H 0.52
Liquid Oxygen 598 441 0? 0.52
Mineral Oil 418 472 28,960H 0.52
Surface Stabilisers
Water 601 396 904H 0.52
[snip]
As with the market sell query I would expect that the market buy query would only show those commodities that are bought and not all the commodities.
Add an option to ignore Fleet Carriers similar to --no-planet
.
If I run the command:
buy --near="Orang/Bessel Gateway" --ly=24 --pad=L --lim=0 -vv -P "Antiquities"
I get the following error:
Traceback (most recent call last):
[snip]
File "buy_cmd.py", line 245, in run
results.summary.avg = int(avgPrice)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
If I change the commodity to something else I do not get that error. For example:
buy --near="Orang/Bessel Gateway" --ly=24 --pad=L --lim=0 -vv -P "Aquaponic Systems"
I have not tried this with all the commodities or ships, just a handful and Antiquities is to only one that gives this error although there could be others.
I have the latest TD and updated the DB earlier today.
Just updated to latest commit (24f26c6) and tried importing data and got the following error:
E:\Elite\TD>trade import -P eddblink
Traceback (most recent call last):
File "E:\Elite\TD\trade.py", line 35, in <module>
from tradedangerous.core import main
ModuleNotFoundError: No module named 'tradedangerous.core'
To avoid further coding style issues, some kind of linting should be done.
Tested flake8, pylint and pep8 but all those report a lot of errors.
One could perhaps get some of them to work with some configuration.
I'm getting an error on calling the edapi plugin as pasted below. The command is being run from a console window in the same folder as the TD data and tmp folders. I have updated TD to the latest version via pip. I have cut out the path to the python folder to improve readability.
This is with a fresh install of TD.
Thanks.
Error log:
C:\Development\TDHelper\TradeDangerousGUI\bin\Debug>trade import -P edapi -O tdh
Traceback (most recent call last):
File "...\python37\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "...\python37\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "...\Python37\Scripts\trade.exe\__main__.py", line 9, in <module>
File "...\python37\lib\site-packages\tradedangerous\cli.py", line 70, in main
trade(argv)
File "...\python37\lib\site-packages\tradedangerous\cli.py", line 125, in trade
results = cmdenv.run(tdb)
File "...\python37\lib\site-packages\tradedangerous\commands\commandenv.py", line 81, in run
return self._cmd.run(results, self, tdb)
File "...\python37\lib\site-packages\tradedangerous\commands\import_cmd.py", line 122, in run
if not plugin.run():
File "...\python37\lib\site-packages\tradedangerous\plugins\edapi_plug.py", line 748, in run
if tdh_path.exists():
AttributeError: 'str' object has no attribute 'exists'
Hi, I'd like to ingest the output of Trade Dangerous in another tool. Would you be interested in adding a command flag to output results as JSON instead of the row format?
I can look at doing this in my own fork, but I'm not familiar enough with Python to be confident with a PR.
The progress bar in EDDBlink is wide and has a percent indicator at the end of the bar. In trade dangerous, however, the bar is small and has not percentage indicator.
Perhaps it would be good to make the TD progress bar match the EDDBlink bar.
Please don't fix this issue :)
Currently running the olddata command lists stations within one jump’s range of a given location sorted by the age of the stations commodity prices. It doesn’t seem to take any account of the —pad filter so lists all stations. I don’t know if this is a bug or intended behaviour. Could the facility be added, please?
Even after fixing the exponentiation, I get the following error in the test case:
F:\Elite\TD>trade.py run -vvv --progress --summary --avoid imperial,slaves --insurance 1.51m --pad L --ly 15.06 --cap 264 --no-planet --credits 24m --from "tollan/gord" --to "LHS 2094/Patt" --hops 3
* Hop 1: .........1 origins
NOTE: Pruned 23 origins too far from any end stations
* Hop 2: ........63 origins .. 33,528-382,536cr gain, 127-1,449cr/ton
[===================== ] Traceback (most recent call last):
File "F:\Elite\TD\trade.py", line 104, in <module>
main(sys.argv)
File "F:\Elite\TD\trade.py", line 77, in main
results = cmdenv.run(tdb)
File "F:\Elite\TD\commands\commandenv.py", line 81, in run
return self._cmd.run(results, self, tdb)
File "F:\Elite\TD\commands\run_cmd.py", line 1220, in run
newRoutes = calc.getBestHops(routes, restrictTo=restrictTo)
File "F:\Elite\TD\tradecalc.py", line 979, in getBestHops
penalty = (-1 + 1 / (cruiseKls + 1) ** ((cruiseKls + 1) / 4)) / 2
OverflowError: (34, 'Result too large')
Possibly solved by going to logs. Will look into it if not fixed by this evening.
Enable multiprocessing for calculations to run faster on multi-core CPUs.
Thought of this when running for trade routes with 500 LY jump range to see if FC trading is profitable.
Implementing this in python is an absolute PITA so understandable if wontfix
or delayed indefinitely.
Does Python support SNI? (I believe yes per https://en.wikipedia.org/wiki/Server_Name_Indication but want to run it by you first).
Assuming it does -
Current best practice is to connect via SSL for all web traffic. Can TD be configured to connect to my server via https, rather than http?
I would suggest leaving the fallback unchanged as we don't have knowledge of the config of that server. (This is currently https anyway, so non-issue).
I downloaded a fresh copy of TD and was running a clean install when I got the following error:
F:\>cd elite/td
F:\Elite\TD>trade import -P eddblink -O clean
NOTE: Rebuilding cache file: this may take a few moments.
NOTE: Missing "F:\Elite\TD\data\TradeDangerous.prices" file - no price data.
NOTE: Downloading file 'modules.json'.
NOTE: Requesting http://elite.ripz.org/files/modules.json
NOTE: Downloaded 292.3KB of gziped data 510.5KB/s
NOTE: Processing Upgrades: Start time = 2018-06-25 19:56:17.902780
NOTE: Finished processing Upgrades. End time = 2018-06-25 19:56:17.913750
NOTE: Downloading file 'index.json'.
NOTE: Requesting https://raw.githubusercontent.com/EDCD/coriolis-data/master/dist/index.json
NOTE: Downloaded 0.7MB of gziped data 10.9MB/s
NOTE: Processing Ships: Start time = 2018-06-25 19:56:18.237884
NOTE: Finished processing Ships. End time = 2018-06-25 19:56:18.244865
NOTE: Downloading file 'systems_populated.jsonl'.
NOTE: Requesting http://elite.ripz.org/files/systems_populated.jsonl
NOTE: Downloaded 20.5MB of gziped data 1.1MB/s
NOTE: Processing Systems: Start time = 2018-06-25 19:56:37.701863
NOTE: Finished processing Systems. End time = 2018-06-25 19:56:38.979448
NOTE: Downloading file 'stations.jsonl'.
NOTE: Requesting http://elite.ripz.org/files/stations.jsonl
NOTE: Downloaded 113.5MB of gziped data 3.2MB/s
NOTE: Processing Stations, this may take a bit: Start time = 2018-06-25 19:57:15.028101
NOTE: Simultaneously processing ShipVendors.
NOTE: Simultaneously processing UpgradeVendors, this will take quite a while.
NOTE: Finished processing Stations. End time = 2018-06-25 20:01:56.025085
NOTE: Downloading file 'commodities.json'.
NOTE: Requesting http://elite.ripz.org/files/commodities.json
NOTE: Downloaded 99.3KB of gziped data 213.7KB/s
NOTE: Processing Categories and Items: Start time = 2018-06-25 20:01:57.189972
NOTE: Finished processing Categories and Items. End time = 2018-06-25 20:01:57.194958
NOTE: F:\Elite\TD\data\Category.csv exported.
NOTE: F:\Elite\TD\data\Item.csv exported.
NOTE: F:\Elite\TD\data\Ship.csv exported.
NOTE: F:\Elite\TD\data\ShipVendor.csv exported.
NOTE: F:\Elite\TD\data\Station.csv exported.
NOTE: F:\Elite\TD\data\System.csv exported.
NOTE: F:\Elite\TD\data\Upgrade.csv exported.
NOTE: F:\Elite\TD\data\UpgradeVendor.csv exported.
NOTE: Downloading file 'listings.csv'.
NOTE: Requesting http://elite.ripz.org/files/listings.csv
NOTE: Downloaded 149.2MB of gziped data 4.0MB/s
NOTE: Processing market data from listings.csv: Start time = 2018-06-25 20:04:05.625704
Traceback (most recent call last):
File "F:\Elite\TD\trade.py", line 104, in <module>
main(sys.argv)
File "F:\Elite\TD\trade.py", line 77, in main
results = cmdenv.run(tdb)
File "F:\Elite\TD\commands\commandenv.py", line 81, in run
return self._cmd.run(results, self, tdb)
File "F:\Elite\TD\commands\import_cmd.py", line 124, in run
if not plugin.run():
File "F:\Elite\TD\plugins\eddblink_plug.py", line 814, in run
self.importListings(self.listingsPath)
File "F:\Elite\TD\plugins\eddblink_plug.py", line 649, in importListings
supply_price, supply_units, supply_level, from_live))
UnboundLocalError: local variable 'from_live' referenced before assignment
In order to figure out a good value for the default lsp with the new formula, we need to collect data. Lots of data.
Yummy yummy data.
Submissions should include the following:
The route returned by TD at lsp == 0.
The lsp values at which the route returned by TD, with all other terms in the run command the same, is different than the lsp=0 route. More precise values are better (lsp=3.96 is more helpful than lsp=4).
Every route returned by TD where all commands except '--lsp' are the same, with the '--lsp' range in which that particular route is returned by TD. More precise values are better ('lsp<=3.96' is more helpful than 'lsp<=4'). It would also be helpful to have the score returned at the min and max lsp values for the route, although that's not as important.
When running the command trade import --plug=eddblink -O clean
I get to inserting new listing data and get a Foreign Key Constraint error.
I've double checked and my TD is up to date (10.3) installed and updated using pip
Command being run: trade import -P edapi -O tdh
The EDAPI plugin is not authorised when the command is run and the error received is pasted below. The Frontier Website opens but hangs with the address bar showing 127.2.0.1.
In order for this to work on Mac OS you have to issue the following command in the terminal:
sudo ifconfig lo0 alias 127.2.0.1
Doing this allows the system to work but in order to make the change persistent you have to do a whole heap of things (see the answer to the question posed here)
Does the address have to be 127.2.0.1?
Could it not be 127.0.0.1 which would avoid the problem?
Thanks
Traceback (most recent call last):
File "/usr/local/bin/trade", line 10, in
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/tradedangerous/cli.py", line 70, in main
trade(argv)
File "/usr/local/lib/python3.7/site-packages/tradedangerous/cli.py", line 125, in trade
results = cmdenv.run(tdb)
File "/usr/local/lib/python3.7/site-packages/tradedangerous/commands/commandenv.py", line 81, in run
return self._cmd.run(results, self, tdb)
File "/usr/local/lib/python3.7/site-packages/tradedangerous/commands/import_cmd.py", line 122, in run
if not plugin.run():
File "/usr/local/lib/python3.7/site-packages/tradedangerous/plugins/edapi_plug.py", line 743, in run
debug = tdenv.debug,
File "/usr/local/lib/python3.7/site-packages/tradedangerous/plugins/edapi_plug.py", line 138, in init
self.profile = self.query_capi("/profile")
File "/usr/local/lib/python3.7/site-packages/tradedangerous/plugins/edapi_plug.py", line 164, in query_capi
self._authorization_check()
File "/usr/local/lib/python3.7/site-packages/tradedangerous/plugins/edapi_plug.py", line 197, in _authorization_check
status_ok = self._authorization_login()
File "/usr/local/lib/python3.7/site-packages/tradedangerous/plugins/edapi_plug.py", line 260, in _authorization_login
oauth = OAuthCallbackServer(redirect_uri.hostname, redirect_uri.port, OAuthCallbackHandler)
File "/usr/local/lib/python3.7/site-packages/tradedangerous/plugins/edapi_plug.py", line 70, in init
self.httpd = myServer((hostname, port), handler)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socketserver.py", line 452, in init
self.server_bind()
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/server.py", line 137, in server_bind
socketserver.TCPServer.server_bind(self)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socketserver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 49] Can't assign requested address
I'm working with a large multi-route calculator and recently it looks like the ls-penalty is no longer operating (in the last 6 months or so).
Example: This call...
trade.py run --fr Bonitou --to Gateway/Dublin --cap 100 --credits 50000000 --ly 16.38 --empty-ly 20 --start-jumps 0 --end-jumps 0 --hops 3 --jumps 3 --pad L --progress --no-planet --ls-penalty 5 --supply 5000 --demand 5000 --avoid Tobacco,Imperial,Narcotics,Slaves,@Shinrarta,"@plx 695", @Exbeur,Alkaid/Camus,Insulating,"@sol" --age 3 -vvv
Gives this station as stop 1:
Unload at Taliesin/Dunbar Platform (4.47Kls, BMk:N, Pad:L, Plt:N, Shp:Y,
With a 4.47Kls station distance.
This previously would never have occurred with ls-penalty = 5
If I then increase ls-penalty in magnitude to 50, 500, 5000, and 50000 none of these have any effect on the route at all.
Formerly an ls-penalty of 50 or 500 would have quickly killed any route with a stop of more than 50ls or so.
Note: "--ls-max 1500" still works perfectly well and the route changes when used, however I preferred ls-penalty in case of major profit just above the ls-max range limit.
Other thoughts:
Was this a previous bug that was fixed? I'm wondering if I'm completely up to date with the latest version. If so, perhaps check in current version, and, if not found, please disregard.
I've just come back to Elite after a long time off, and pulled the latest copy of TD. I'm trying to get the edapi
plugin to work, as that was my preferred method of pulling data for my current station. It's not obvious if it's still intended to work or not - I've found a few forks of the EDAPI tool itself, but even the latest one I can find won't actually let me log in.
Did all these tools break when they added the in-game trade tools?
Do we need to do some manual adding of ships after all?
trade.py import -P eddblink -O clean -w
.
.
.
NOTE: Simultaneously processing UpgradeVendors, this will take quite a while.
# ABUKUNIN/Reilly Hub has been added:
# ALRAI/Bounds Hub has been added:
# ANGLII/Starzl Plant has been added:
# BETA TRIANGULI AUSTRALIS/Lunan Hub has been added:
# BETA TRIANGULI AUSTRALIS/Hutchinson City has been added:
# BANDJIGALI/Teng-hui Station has been added:
# 39 TAURI/Porta has been added:
# AGARLA/Wilson Port has been added:
# BETA TRIANGULI AUSTRALIS/Vinge Gateway has been added:
# APATHAAM/Reamy Dock has been added:
# BRANI/Virtanen Hub has been added:
# 85 PEGASI/Buckland Gateway has been added:
# BAGAI/Humphreys' Progress has been added:
# AIABIKO/Maxwell Orbital has been added:
# AIABIKO/Gooch Terminal has been added:
# BHUTAS/Feynman Terminal has been added:
# AMISI/Sargent Escape has been added:
# AMISI/Brooks' Folly has been added:
# 78 URSAE MAJORIS/Townshend Hub has been added:
# BRANI/Akiyama Market has been added:
# ALLOWA/Moore Ring has been added:
# 85 PEGASI/Borman Terminal has been added:
# ANLAVE/Suri Park has been added:
# 44 B OPHIUCHI/Feustel Gateway has been added:
# AULIN/Aulin Enterprise has been added:
# AMAHU/Kondratyev Orbital has been added:
# AMARAK/Vela Port has been added:
# ATFERO/Patry Dock has been added:
# BLODER/Szebehely Station has been added:
# BHUMLIAURT/Lichtenberg Port has been added:
# BD+69 530/Burbank Gateway has been added:
# 78 URSAE MAJORIS/Read Gateway has been added:
# BD+69 530/Leoniceno Orbital has been added:
# 31 AQUILAE/Russo Dock has been added:
# AMIJARA/Shepherd Outpost has been added:
# BIDIONO/Wellman Dock has been added:
# APAM NAPAT/Langley Hub has been added:
# AGUANA/Webb Dock has been added:
# ANLAVE/Bain Dock has been added:
# ANLAVE/Hogg City has been added:
# BD+55 1519/Arber Colony has been added:
# AMALANGKAN/Doi Landing has been added:
# AREXE/Janes Horizons has been added:
# BELU/Covey Gateway has been added:
# BINAR/Ferguson Dock has been added:
# BRANI/Barsanti Enterprise has been added:
# BAKONIRTARI/Gregory Vision has been added:
# BRANI/Noakes Port has been added:
# ANDRUNGARRI/Disch Port has been added:
# BETA-1 TUCANAE/Adams Orbital has been added:
# 42 AQUILAE/Born Station has been added:
# AVIK/Bruce Prospect has been added:
# ATLANTIS/Kimura Terminal has been added:
# 42 AQUILAE/Ride Market has been added:
# 78 URSAE MAJORIS/Teller Terminal has been added:
# 110 HERCULIS/Jahn Hangar has been added:
# BALTAH'SINE/Baltha'Sine Station has been added:
# AHAUT/Goldberg Orbital has been added:
# ALTAIR/Solo Orbiter has been added:
# 78 URSAE MAJORIS/Rominger Dock has been added:
# ANLAVE/Brunton Gateway has been added:
# ANLAVE/Kobayashi City has been added:
# ALCOR/Birmingham Dock has been added:
# BD-04 4138/Shunn Dock has been added:
# BRANGLAL/Gibson Horizons has been added:
# BD+47 1236/Dana Relay has been added:
# BPM 89444/Szameit Ring has been added:
# ALIOTH/Irkutsk has been added:
Traceback (most recent call last):
File "D:\Games\Game Tools\Elite\Trade Dangerous\trade.py", line 104, in <module>
main(sys.argv)
File "D:\Games\Game Tools\Elite\Trade Dangerous\trade.py", line 77, in main
results = cmdenv.run(tdb)
File "D:\Games\Game Tools\Elite\Trade Dangerous\commands\commandenv.py", line 81, in run
return self._cmd.run(results, self, tdb)
File "D:\Games\Game Tools\Elite\Trade Dangerous\commands\import_cmd.py", line 124, in run
if not plugin.run():
File "D:\Games\Game Tools\Elite\Trade Dangerous\plugins\eddblink_plug.py", line 795, in run
self.importStations()
File "D:\Games\Game Tools\Elite\Trade Dangerous\plugins\eddblink_plug.py", line 430, in importStations
modified))
File "D:\Games\Game Tools\Elite\Trade Dangerous\plugins\eddblink_plug.py", line 95, in execute
result = cur.execute(sql_cmd, args)
sqlite3.IntegrityError: NOT NULL constraint failed: ShipVendor.ship_id
Platform: TD 10.6.1 via pip, Python 3.8.0, Windows 10 18362.535
When doing an initial import, the download of the file http://elite.tromador.com/files/listings.csv never proceeds. That is, the last output on the console is: NOTE: Requesting https://elite.tromador.com/files/listings.csv
, the download progress bar never appears, not even after hours of waiting.
I tested downloading the file locally, and the file is available and downloads normally, so this doesn't appear to be an issue on the Tromador site.
I was able to work around downloading this by forcing EDDB to be used for the initial download (set TD_SERVER=https://eddb.io/archive/v6/
), but this results in a 404 when the -live
version of the listings.csv is requested. However, once the first listings.csv is downloaded,, the variable can be unset, allowing the -live file to be sourced from Tromador, and finally the import is complete.
The GUI seems to run fine, but threw the following errors (Win 10 x64, Python 3.7.3):
2019-06-29 17:31:53,019 appJar:ERROR [Line 9->2738/_fontHelper]: Failed to adjust inputFont to Courier.
2019-06-29 17:31:53,020 appJar:ERROR [Line 9->2738/_fontHelper]: Failed to adjust labelFont to Courier.
2019-06-29 17:31:53,022 appJar:ERROR [Line 9->2738/_fontHelper]: Failed to adjust buttonFont to Courier.
2019-06-29 17:31:53,057 appJar:WARNING [Line 9->4254/_removeContainer]: Closing empty container: req
2019-06-29 17:31:53,061 appJar:WARNING [Line 9->4254/_removeContainer]: Closing empty container: opt
2019-06-29 17:31:53,067 appJar:ERROR [Line 9->2556/configure]: Invalid config parameter: disabled, horizontal
Obviously my first thought was that it's Willy's server, but I can browse that URL both from my PC and using lynx from the shell on the server, so the file is available, but for some reason, not when TD requests it.
I can only note that when I browse it via lynx, I'm seeing it as an x-gzip content type. Don't know if that helps. Otherwise, let me know what tests I can run.
Meantime, as there is the fallback ship index, I'll let TD/listener run from that.
NOTE: Checking for update to 'index.json'.
WARNING: Problem with download:
URL: https://beta.coriolis.io/data/index.json
Error: HTTP Error 403: Forbidden
NOTE: Using Default Ship Index.
But (e.g.) wget on same host/user/shell et al.
[elite@quoth ~]$ wget https://beta.coriolis.io/data/index.json
--2019-05-23 04:56:56-- https://beta.coriolis.io/data/index.json
Resolving beta.coriolis.io (beta.coriolis.io)... 104.24.116.234, 104.24.117.234, 2606:4700:30::6818:74ea, ...
Connecting to beta.coriolis.io (beta.coriolis.io)|104.24.116.234|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 869197 (849K) [application/json]
Saving to: ‘index.json’
100%[======================================>] 869,197 1.84MB/s in 0.4s
2019-05-23 04:56:57 (1.84 MB/s) - ‘index.json’ saved [869197/869197]
On reflection I guess could be @willyb321 but only if his server is detecting and blocking the specific type of web request TD is using, but it does seem strange that other requests work perfectly.
running trade.py import -P eddblink -O help
returns an error:
Traceback (most recent call last):
File "trade.py", line 107, in <module>
main(sys.argv)
File "trade.py", line 80, in main
results = cmdenv.run(tdb)
File "S:\Elite Dangerous Programs\Trade Dangerous\eddblink-listener\commands\commandenv.py", line 81, in run
return self._cmd.run(results, self, tdb)
File "S:\Elite Dangerous Programs\Trade Dangerous\eddblink-listener\commands\import_cmd.py", line 119, in run
plugin = pluginClass(tdb, cmdenv)
File "S:\Elite Dangerous Programs\Trade Dangerous\eddblink-listener\plugins\eddblink_plug.py", line 62, in __init__
super().__init__(tdb, tdenv)
File "S:\Elite Dangerous Programs\Trade Dangerous\eddblink-listener\plugins\__init__.py", line 188, in __init__
super().__init__(tdb, tdenv)
File "S:\Elite Dangerous Programs\Trade Dangerous\eddblink-listener\plugins\__init__.py", line 52, in __init__
raise SystemExit(self.usage())
File "S:\Elite Dangerous Programs\Trade Dangerous\eddblink-listener\plugins\__init__.py", line 95, in usage
text += tw.fill(options[opt].strip()) + "\n"
File "C:\Python37\lib\textwrap.py", line 363, in fill
return "\n".join(self.wrap(text))
File "C:\Python37\lib\textwrap.py", line 354, in wrap
return self._wrap_chunks(chunks)
File "C:\Python37\lib\textwrap.py", line 314, in _wrap_chunks
lines.append(indent + ''.join(cur_line))
TypeError: can only concatenate tuple (not "str") to tuple
For some reason the TextWrap module is splitting the string it's being passed into tuples?
I get the following error:
ERROR Unrecognized item name: "AGRONOMICTREATMENT"
Olddata parameter does not work with --ls-max and --no-planet parameters.
When you trade with the biggest ships and want to make those real credits, its done within <1000ls stations and no planetary landings.
I would like to search oldest market prices and go update them, kind of like rejuvenating the markets, but at current database and without these parameters available... its just endless list of >1000ls stations and/or planetary outposts.
For reference: http://www.davek.com.au/td/editstation.asp allows you to filter ls distance (or missing distance) and planetary.
Maybe look into making TD able to handle that.
Command line:
trade.py sell --near="Orang/Bessel Gateway" --ly=100 --pad=L --limit=42 -vv "Advanced Medicines, Basic Medicines"
Error report:
Traceback (most recent call last):
File "C:\Development\tradedangerous\trade.py", line 107, in <module>
main(sys.argv)
File "C:\Development\tradedangerous\trade.py", line 80, in main
results = cmdenv.run(tdb)
File "C:\Development\tradedangerous\commands\commandenv.py", line 81, in run
return self._cmd.run(results, self, tdb)
File "C:\Development\tradedangerous\commands\sell_cmd.py", line 78, in run
item = tdb.lookupItem(cmdenv.item)
File "C:\Development\tradedangerous\tradedb.py", line 1932, in lookupItem
val=lambda kvTup: kvTup[1]
File "C:\Development\tradedangerous\tradedb.py", line 2109, in listSearch
"Error: '%s' doesn't match any %s" % (lookup, listType)
LookupError: Error: 'Advanced Medicines, Basic Medicines' doesn't match any Item
Using the sell query with a single item works correctly.
Why did this happen? Why add restrictive copyleft provisions? Also, technically, you have to get permission from EVERY contributor to change the license, unless you can show their contributions are no longer in the current state of the system. I don't this happened, so I'm not sure at this point that the TD system is properly licensed with any restrictions.
Hi folks.
Just getting to grips with this - seems very useful so far.
Am wondering if there's a reason why commands like "buy" don't have a --ls-max option?
If not, I could have a go at patching it in myself...
According to https://eddb.io/api the old v5 URLs are phased out very rapidly, and the old files are not updated anymore. It is used as a backup URL in the eddb plugin.
It also seems that http://elite.ripz.org/ is missing some important commodities (Void Opals for instance). These are present when the import is done from the v6 EDDB API, and seems to work as intended.
I wanted to update TD after a long absence, so I pulled from the github repository and ran -O clean
but got an error that Category.csv was missing. Just in case, I deleted the entire TD directory, cloned it afresh, and still got the following error. Any suggestions?
E:\Elite\TD>trade import -P eddblink
NOTE: Rebuilding cache file: this may take a few moments.
NOTE: Missing "E:\Elite\TD\data\TradeDangerous.prices" file - no price data.
NOTE: Downloading file 'modules.json'.
NOTE: Requesting http://elite.ripz.org/files/modules.json
NOTE: Downloaded 297.8KB of gziped data 2.5MB/s
NOTE: Processing Upgrades: Start time = 2018-12-30 20:45:29.579455
NOTE: Finished processing Upgrades. End time = 2018-12-30 20:45:29.588431
NOTE: Downloading file 'index.json'.
NOTE: Requesting https://raw.githubusercontent.com/EDCD/coriolis-data/master/dist/index.json
NOTE: Downloaded 0.7MB of gziped data 19.5MB/s
NOTE: Processing Ships: Start time = 2018-12-30 20:45:29.875662
NOTE: Finished processing Ships. End time = 2018-12-30 20:45:29.887630
NOTE: Downloading file 'systems_populated.jsonl'.
NOTE: Requesting http://elite.ripz.org/files/systems_populated.jsonl
NOTE: Downloaded 20.5MB of gziped data 6.3MB/s
NOTE: Processing Systems: Start time = 2018-12-30 20:45:33.659592
NOTE: Finished processing Systems. End time = 2018-12-30 20:45:35.524649
NOTE: Downloading file 'stations.jsonl'.
NOTE: Requesting http://elite.ripz.org/files/stations.jsonl
NOTE: Downloaded 115.0MB of gziped data 15.7MB/s
NOTE: Processing Stations, this may take a bit: Start time = 2018-12-30 20:45:43.387194
NOTE: Simultaneously processing ShipVendors.
NOTE: Simultaneously processing UpgradeVendors, this will take quite a while.
NOTE: Finished processing Stations. End time = 2018-12-30 20:52:48.068146
NOTE: Downloading file 'commodities.json'.
NOTE: Requesting http://elite.ripz.org/files/commodities.json
NOTE: Downloaded 100.6KB of gziped data 98.5MB/s
NOTE: Processing Categories and Items: Start time = 2018-12-30 20:52:48.563327
NOTE: Checking for missing items....
Traceback (most recent call last):
File "E:\Elite\TD\trade.py", line 107, in <module>
main(sys.argv)
File "E:\Elite\TD\trade.py", line 80, in main
results = cmdenv.run(tdb)
File "E:\Elite\TD\commands\commandenv.py", line 81, in run
return self._cmd.run(results, self, tdb)
File "E:\Elite\TD\commands\import_cmd.py", line 124, in run
if not plugin.run():
File "E:\Elite\TD\plugins\eddblink_plug.py", line 921, in run
self.importCommodities()
File "E:\Elite\TD\plugins\eddblink_plug.py", line 518, in importCommodities
with open(str(tdb.dataPath / Path("Category.csv")), "r") as fh:
FileNotFoundError: [Errno 2] No such file or directory: 'E:\\Elite\\TD\\data\\Category.csv'
See the following excerpt from my TradeDangerous.prices. There are no slaves presently for sale at this station (HR 8208/Sturgeon Station)
+ Slavery
Imperial Slaves 17152 17510 ? 139659M 2018-09-17 13:02:48
Slaves 11039 11650 ? 54801M 2018-09-15 18:03:56
My best guess is that because "Slaves" is a subset of "Imperial Slaves" it did not get deleted when the new station data was uploaded, you can see the disparity of date/time information.
Not really a TD bug, but it's missing from index.json so we need a temp workaround.
Would you mind doing it? I'm up to my eyeballs converting apache 2.2 config to 2.4 so we can migrate to the new server.
The initial import of StationItems take forever.... Could it be speed up?
When I ran the initial update from eddb I got
...
NOTE: Processing market data from listings.csv: Start time = 2019-01-03 15:37:54.424876
NOTE: Finished processing market data. End time = 2019-01-03 15:58:23.628491
...
Processing market data took 21 minutes when starting from "empty" data folder.
Since the command line sqlite3 tool have support for importing csv files I made some tests, see attached script import-help.sql.txt .
2019-01-03 15:35:06|Importing eddb/listings.csv
2019-01-03 15:35:23|Generating Indexes
2019-01-03 15:35:29|Fixing brackets
2019-01-03 15:35:31|Creating 'simplified' StationItem table
2019-01-03 15:35:31|Insert or replace StationItems
2019-01-03 15:35:54|Done!
48 seconds......!
I ran this by doing rm test.db & sqlite3 test.db ".read import-help.sql.txt"
in the Trade-Dangerouse folder after listings etc had been downloaded.
After hacking the code in eddblink_plug and importing using the sqlite3 command+script for listings.csv
but not creating a modified StationItem
table I got.
...
NOTE: Processing market data from listings.csv: Start time = 2019-01-03 17:23:15.991480
2019-01-03 16:23:16|Importing eddb/listings.csv
2019-01-03 16:23:37|Generating Indexes
2019-01-03 16:23:46|Fixing brackets
2019-01-03 16:23:50|Insert or replace StationItems
2019-01-03 16:27:32|Done!
NOTE: Finished processing market data. End time = 2019-01-03 17:27:32.298064
...
Worse with all the foreign key checking and existing related data but ~5 minutes is way better than 21.
(The 1 hour difference in timestamps depends on me being UTC+1)
Is there something I'm missing?
OS: Windows 10, 1809
CPU: i7-8650U
Python 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32
In the setup guide you explain how to setup the initial database but there's a missing -
in the code which causes an error even when cut and paste.
Existing text says:
$ trade.py import --plug=eddblink -opt=clean,skipvend
Needs to be:
$ trade.py import --plug=eddblink --opt=clean,skipvend
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.