opentripplanner / otpqa Goto Github PK

Our existing approach essentially tries the cross product of all modes, all times of day, and all endpoints. It scales this product down somewhat, but it's still an enormous number of combinations.

This approach was motivated by the possibility of stumbling upon some random origin point that might exhibit interesting characteristics only at a particular time of day, or with a particular mode. This produces an enormous amount of requests though. In practice, to achieve realistic run times we find ourselves drastically limiting the number of random endpoints.

For example, with 1000 random endpoints in non-fast mode we see an estimated run time of over 21 days in NYC.

Note that when holding the total number of requests constant, by virtue of the randomness of the search endpoints, this cross-product approach does not have a higher probability of finding an "interesting" endpoint for a particular set of search parameters, despite testing a much smaller number of locations.

Therefore, I propose that we grab the endpoints two by two rather than doing anything resembling a cross product. We can step over the random endpoints and the search parameter combinations in parallel, looping over the shorter list until both have been exhausted.

It may also be a good idea to take the full list of search parameter combinations from a CSV or JSON file rather than generating them automatically. Generating them by taking the cross product of a bunch of individual parameter values tends to over-represent unusual combinations. There should still be a script to generate "starter" files of request parameters that can then be edited down by the user. The current scripts already allow this approach, but it is not highlighted in the readme.

It is still a good idea to have a slow and a fast mode. Perhaps the slow mode should use the custom endpoints as origins, over the full range of search parameters, using the full range of random endpoints as destinations.

There are a lot of places in NY that are within 2km of a transit stop, but which are unreachable on foot from that transit stop. This is largely due to rivers.

I updated a little OTPProfiler:

• to add total_time/avg_time in seconds in otpprofiler
• to add real total_time (totalTime from OTP response without roundtrip time)
• to enable creating of graphs without the database
• to add optional support for OTP 0.11.x

What is missing is requirements.txt, option to ignore endpoints_custom.csv.

If this is useful I can do PR or just commit.

The currently workflow involves running a script that generates random endpoints in CSV, then another one that ingests the endpoints from CSV and outputs them unchanged to JSON. The final profiling script then consumes the JSON.

There does not seem to be any need for these multiple transformations -- the profiler script should just load the CSV directly, or the endpoints should be saved to JSON from the beginning. Given the tabular nature of endpoint lists, CSV seems like a good option.