openmobilityfoundation / mobility-data-specification Goto Github PK

@hunterowens, I have a few issues I discovered with error handling for the Trip endpoint. I'll be working on a solution. Please don't tag until I submit pull request tomorrow Sept 11th

Not an issue- just a few quick thoughts from SDOT.

Trip Data
route: could this specify an encoded polyline algorithm such as https://developers.google.com/maps/documentation/utilities/polylinealgorithm?

trip_duration: could this be calculated?

trip_distance: is this distance calculated or the actual route distance?

start_point/end_point: we will be geocoding to blockgroup. Would the companies be able to geocode this on their end, or can the API geocode this automatically?

start_time, end_time: we use unix timestamp a lot. Sometimes I prefer ISO 8601 because it’s human readable and doesn't require conversion. We usually screw up daylight savings time.

System Data / Availability: would we need to specify geography or time period granularity?

Realtime Data: The feed might be an easier way to measure availability. It might be good to specify which fields are required. I think these should be point coordinates, device_type, device_id. We would need to know status (rented or available). If there is no status, the feed should only return available bikes.

Need to decide what endpoints are going to be active in Version 1.0 (Jan)

I have concerns about having to call your endpoints in real time as the delays introduced by having to call multiple extra end points will affect our user experience. Also, if your endpoints are down for whatever reason, you might end up having incomplete data. May I know the reasoning behind the need for the v2 specifications when the v1 data requirements gives you everything you need from the v2 endpoints? Also what are the plans to address the issues i've stated above? Thanks!

Providing frequent and quick access to an ever growing set of data will obviously put pressure on provider's infrastructure, especially for new or smaller providers which may not have as sophisticated technology stacks. I think that we should establish explicit limits and expectations around QPS, latency, and data retention.

QPS/Load

• What kind of QPS should providers expect on these APIs?
• As a provider, I expect we will want to enforce rate limits, what are acceptable limits?

Latency

• What is an acceptable latency?
• Should we consider asynchronous implementations for providers that cannot meet latency expectations easily?

Data Retention

• What is the expected data retention? Indefinite?

Hunter and David, You guys mentioned in our last meeting that there were changes you made during implementation of the Provider side that need to be made to the Agency side. Could you make those changes before we start building?

First and last points in route are start and end; all points in route are marked with timestamps.

Trips

• Include vehicle ID
• Include driver / operator ID (if applicable)
• Request point (if different than start_point)
• Passenger count (if applicable)
• Payment type (CC, fare card, etc)
• I'm not sure what you mean by route type = line. In the SF taxi API, there's a separate reporting of telemetry data.
• White is route optional?
• Why is info on standard_cost included?

Availability

• Disabled access status of vehicle
• Not sure what associated_trips means

we may want to link the presentation from Wednesday in the README.md

@black-tea can you create a PR for this? We should probably only include the 2nd batch of slides unless @thecorconian feels otherwise.

I am concerned that many cities may not have the technical capacity to consume a dynamic, query-based API.

The /trips and /events endpoints both lend themselves to a flat file format. While LA will clearly want to require a dynamic API, it may be worth broadening MDS to allow for other cities to request the same data in flat file format (e.g. monthly/weekly CSVs).

Are LADOT folks amenable to including some guidance on "flat file MDS implementations" in the spec? I suspect this would make MDS more appealing to other cities.

Happy to do put in pull request of people think this is a good idea.

Should the location be a Point geometry or a route point feature with horizontalAccuracy?

https://github.com/CityOfLosAngeles/mobility-data-specification/blob/master/provider/README.md#status-changes

🎉

This is more of a point of clarification:

Currently, we use unavailable:low_battery to indicate that "A device is no longer available due to insufficient battery".

If a charger picks up said device, recharges it, and places back into service -- all in the same day -- is the second event available:maintenance_drop_off?

Either that, or we're missing the corresponding event_type_reason for placing a device that has been recharged back into service, separate from the initial placement of the day.

The MDS technical workshop made clear that LADOT wants precise location data as part of their interactions with providers. LADOT feels that can manage any real or perceived privacy risks that stem possessing anonymized, location-precise trip data, and that the benefits of this data are significant.

From comments made at the workshop, it seems that there is not universal agreement on this point, especially given the variability in state-level FOIA laws and the different attitudes individual cities have about the broad topic.

I'd like to propose that MDS be extended to allow for flexibility in how location data is reported for trips and routes. LA can choose to require precise data, but other jurisdictions would have the flexibility to request less detailed information. It is ultimately a policy decision, and a universal spec needs to support some level of policy variance across its users.

Two approaches to (intentionally) reducing the accuracy of GPS data:

1. Blurring data to report less precise coordinates. These would correspond to a radius of uncertainty (e.g. accurate to within 500m).

2. Aggregating data into defined boundary areas (e.g. census block, ZCTA, neighborhood, etc.).

Option 1 seems like the simplest approach. It could be done without change to the MDS specification. Providers and agencies would simply agree on the level of specificity in the route's GeoJSON FeatureCollection. With this approach, the specification can simply be modified to acknowledge the possibility of deliberately imprecise data and leave it up to the stakeholders to agree on the details.

Option 2 may be easier for cities to consume (since they're already used to aggregating data into boundaries like census block), though would necessitate more pre-processing by the providers. It also would require modification to the specification since GeoJSON does not have an obvious way to describe location within specify externally-defined boundary areas.

Thoughts on:

1. Whether we should modify the spec to support imprecise location sharing?
2. If yes, what method to use?

@hunterowens. Apologies if I'm missing something, but I don't see where we are specifying the required update period for Trip data in the Provider API. The Agency side arbitrarily mentions 5 seconds. Any thoughts around this?

Todd

The mobility spec already provides some nice narrative description of the inputs and outputs for the REST API. However, there is still some room for ambiguity, and it would be useful to be able to test providers/agencies against regressions and compliance with the spec.

One commonly-used approach to this problem is to use a JSON schema. This allows you to describe JSON data structures in a well-typed fashion. This allows for

• validation
• regression testing
• a source of truth for the spec to guard against different interpretations of the narrative.
• Easier tracking of spec versions

For instance, a schema for the response body of the /deregister-vehicle might look something like:

{
  "properties": {
    "unique_id": {
      "type": "string"
    },
    "device_id": {
      "type": "string"
    },
    "reason_code": {
      "type": "string",
      "enum": ["rebalancing", "maintenance"]
    }
  },
  "type": "object",
  "additionalProperties": false
}

High-quality JSON-schema validators should exist in most web-friendly languages.

I would like to clarify exactly what start_time and end_time mean for GET /trips

Does this refer to when the rider created a booking on the platform or when the route actually starts? What should the trip object look like for a canceled booking or a trip that otherwise didn't "start"?

Data Sharing Comments

1. Definition of real-time: in numerous places in the document you reference real-time. This is not well-defined.

• Recommendation: Use real time defined as "in as real time as possible", with the understanding that it may be minutes old and not at that very second.

2. Trip Data Query Parameters: It is quite complex to have the API be queryable and this is not currently the industry practice.

• Recommendation: To avoid system issues, the API should directly return /trips without the query parameters with a reasonable maximum number of records that can be returned (e.g. up to 1-day, or a few days)

3. Service area metrics: Similar to #2 above, it’s quite complex to allow querying by GeoJSON.

• Recommendation: It should be queried by service area ID.

4. Movement Plans: These appear to be specific to rideshare and would not be relevant to our industry.

• Recommendation: Remove this API requirement

5. Maintenance Reporting: This is not industry practice to convert maintenance records into API format.

• Recommendation: Operator should submit summary reports of maintenance tickets.

6. System Data / Availability Data: These are not clearly defined; for example, the difference between placement_reason and pickup_reason isn’t clear. Definitions should be more specific.

• Recommendation: Operators should provide the counts of available vehicles or list of unique IDs of vehicles that are available at time of query. We propose the use of the format we authored here for vehicle availability: https://bitbucket.org/limebike/dbfs/src/15d8a74b10ee1a0c5109d3c14bfd03cce02ece46/dbfs.md?at=master&fileviewer=file-view-default#get-bikes

7. Update-trip-data: We cannot guarantee this service level as the requested frequency granularity is very high.

• Recommendation: Focus on submitting a single record for each trip instead.

Fleet Size Comments

1. Maximum fleet size: We recognize the addition in the staff report permitting operators to add 2,500 vehicles in disadvantaged communities. Nonetheless, enforcing a cap on operators does not permit operators to scale based on demand nor does it incentivize high quality service.

• Recommendation: Institute an expansion plan of 1,000 vehicles per month as a baseline that operators can qualify for by being part of the pilot program. Operators should then be able to apply for expansion beyond this 1,000 vehicles/month based on several criteria, such as: meeting a threshold of 2 trips/bike/day for bicycles and electric-assist bicycles and 3 trips/scooter/day for electric scooters; meeting operational performance criteria including customer service response time; and meeting community engagement targets.

Parking Requirements

1. Locking mechanism requirement: Requiring every vehicle to have a locking mechanism to lock to a fixed object raises equity questions as not every community has proper bike parking infrastructure, while also posing potential obstacles from bikes locked to ADA rails, construction equipment, or other non-bike parking objects.

• Recommendation: Operators should be encouraged to add lock-to on their vehicles but not required.

Geofencing Comments

1. Geo-fencing requirement around Metro Bike Share and Downtown: Forcing operations to take place outside of areas near existing Metro Bike Share stations or the downtown area severely impacts our ability to serve the community on multiple levels. First, these are areas where traffic congestion and parking are the most severe. Also, solving first-last mile gaps becomes near impossible if riders cannot ride to a rail station or bus station where there is also a Metro Bike Share station; communites that live within 3 miles of a Metro Bike Share station but do not have a station nearby are also not able to enjoy dockless mobility, posing an equity challenge. As Metro Bike Share expands, this requirement will also make it near impossible to serve the City of Los Angeles in a meaningful way. In addition, there is no clear reason for why "dockless bike share shall not be allowed to directly compete with Metro Bike Share at this time" - dockless bike share systems have operated alongside dock-based bike share systems in other cities and have not been negatively impacted by it (see Washington, D.C.'s Capital Bikeshare system's record ridership during the same time dockless mobility companies have been in operation).

• Recommendation: Remove the geofence restriction to allow for equitable service of the City

Equity Comments

1. Low-income plan requirement: LADOT should better define how operators can qualify individuals as low-income and have the low-income plan match other low-income transportation discounts, rather than forcing free rides under 30 minutes.

• Recommendation: Define low-income plan qualifiers as an individual who qualifies for state or local low-income plan with the plan being a heavily discounted fare equivalent to current low-income transportation discounted fares.

in the provider API

• dockless_bike
• ebike
• scooter
• adaptive-device

Define CORS headers for provider. Sparked by #58

Similarly to how GBFS maintains a central systems.csv file, should MDS maintain a providers.csv (or similar)?

MDS regulators (municipalities) need a way to uniquely identify each of the different provider data feeds. On the other hand, a given provider may have a different MDS feed for many different municipalities.

Rather than forcing each provider to register with each municipality separately (to obtain a unique provider_id), the idea is to maintain a single, central source of truth.

An example schema:

Does it make sense to have the provider API URL as part of this registry? If that is the case, the API needs to support queries for a particular municipality (e.g. by geo-fenced area).

The current status changes don't account for vehicles which go missing (for any number of reasons). These vehicles are not available to be ridden, and most likely are also not in the public right of way. There are two ways this could go:

1. Add a new event_type of lost and exclude those vehicles from cap counts.
2. Add a new event_type_reason of lost to the removed event_type

Either way, there should then be an event_type_reason for available of found

We have scooters in our fleet that do not have their own GPS but piggyback off the user's bluetooth.

For these devices we do have a unique identifier for the scooter but it is not tied to the unit that transmits GPS.

In these cases how should we handle device_id?

Relevant spec

MDS defines the device as the unit that transmits GPS signals for a particular vehicle. A given device must have a UUID (device_id below) that is unique within the Provider's fleet.
Additionally, device_id must remain constant for the device's lifetime of service, regardless of the vehicle components that house the device.

In the provider API definition, it says that "pagination must comply with the JSON API specification", but the JSON API spec says:

Pagination links MUST appear in the links object that corresponds to a collection. To paginate the primary data, supply pagination links in the top-level links object.

This spec puts the first, last, prev, next URLs at the top level of the response, rather than inside in a links object. I feel like either this spec should use the links object or not say that paginations complies with JSON API.

In the current spec, trip_id is generated by the agency in response to a start_trip call. I think this creates an undesirable dependency on the agency’s infrastructure.

If I’m reading correctly, the trip_id is also used in end_trip and trips APIs. Now, imagine that the agency’s API server goes down for whatever reason (bugs, power outage, network outage, server maintenance, etc.). The provider will attempt to call start_trip and get a failure. Now they don’t have a trip ID with which to identify the trip, so they are in a pickle. If they go ahead and start the trip, it will never have an agency-approved trip ID. If they don’t start the trip, their service is effectively shut down.

I would suggest moving trip_id to be an argument to start_trip rather than a return value. This would eliminate the dependency.

Thanks for all the good work!

Currently it is difficult for us to differentiate between a mechanic dropping off a scooter after fixing it and a charger dropping off a scooter after charging it. It is also unclear whether the event_type_reason applies to a mechanic drop off or a charger drop off.

I'd love some clarification around the issue and to discuss ways to modify it.

Spec in question:

| maintenance_drop_off | Device introduced into service after being removed for maintenance |

Use underscore_case

The current MDS specification relies on using the request body in GET requests as the mechanism for specifying the resource being requested. This is generally considered to be an anti-pattern (see discussion here). More importantly, it makes caching such requests impossible, as caches operate based on the request URI only.

Fixing this is simple - either move the request parameters from the body to the query string or don't use GET for those requests. The fix depends on the intent and desired semantics for the endpoint. Right now, I'd propose the following:

• Move the Provider API endpoints (/trips and /status_changes) to POST requests. These queries will need to accept complex parameters (specifically thinking about the geographic shapes for narrowing the query)
• Change the Agency API /service_areas endpoint to accept a query param provider_id and support requesting specific service areas by id. e.g. GET /service_areas?provider_id=XXX would return all the active service areas for a provider and GET /service_areas/:id (where :id is the id of a specific service area) would return the data for a specific service area

I have no idea how it's supposed to be formatted, but it's not rendering correctly...

We strongly request that LADOT revise the Fleet Size C and D requirements to ensure those dockless companies with adaptive bikes, but who do not have e-bikes, can provide bike share services to those with disabilities who would not otherwise benefit from dockless bike share. Here is our proposed replacement language:

c) Operators must reserve a minimum of 50 percent of their fleet size for electric-assist bicycles, unless an operator provides adaptive bicycles for persons with disabilities.

d) Operators that do not provide 50 percent of their fleet size for electric-assist bicycles must reserve a minimum of 1 percent of their fleet size for adaptive bicycles for persons with disabilities. Adaptive bicycles must include locking mechanisms to securely affix the bicycles to bicycle racks. These adaptive bicycles must be accessible through the bike share operator’s app and not a third-party provider, and the experience of riding an adaptive bicycle should be similar to other vehicles in the operator’s fleet.

The bottom section of providers mentions that providers should expose a GBFS feed as well. Is it possible to provide more detail around the specifics and intention here.

There are number of parts of GBFS that don't apply clearly and directly to dockless mobility devices. It might be helpful to specify what specific pieces of GBFS should be implemented and in what manner that makes sense for dockless mobility devices.

Also, it's not clear what "exposed" means here. Would this fall under the same security as the proposed MDS endpoints?

Any clarification greatly appreciated!

Relevant section quotation:

Realtime Data

All MDS compatible provider APIs must expose a GBFS feed as well. For historical data, a time parameter should be provided to access what the GBFS feed showed at a given time.

These fields are provided by both dominant smartphone platforms, but they can be null. I believe the field names should be defined in the spec, but allowed to be empty. Proposed definitions:

• speed: Instantaneous speed in meters per second. Ref: iOS Android
• course: The direction in which the device is traveling, measured in degrees and relative to due north. Ref: iOS Android
• altitude: The altitude, measured in meters. Ref: iOS Android
• verticalAccuracy: The uncertainty of the altitude value, in meters. Ref: iOS Android

Example point:

    {
        "type": "Feature",
        "properties": {
            "timestamp": 1529968782.421409,
            "speed": 5.23,
            "course": 290,
            "altitude": 40,
            "verticalAccuracy": 30
        },
        "geometry": {
            "type": "Point",
            "coordinates": [
                -118.46710503101347,
                33.9909333514159
            ]
        }
    }

From an implementer's perspective, I'd argue that requiring that vehicles, devices, and trips all have UUIDs could be a bit of pain, and I'm not totally sure why it's required. Since a provider is presumably in charge of all their fleet data, they would have the ability to allocate unique IDs without using UUID. What do you think about just changing this to String type and requiring that it be unique by provider?

When a GPS device moves through an environment, especially an urban environment, the accuracy of the signal can vary tremendously. This means it's possible that some points on a GPS route may have an accuracy of a meter while others points in the route may have accuracy of 10 or 30 meters or whatever. In this kind of case, it's not entirely clear what to send in the accuracy field for a trip, since it varies so much.

I'd suggest that maybe it would be better to move the accuracy field into the route GeoJSON, like this:

"route": {
    "type": "FeatureCollection",
    "features": [{
        "type": "Feature",
        "properties": {
            "timestamp": 1529968782.421409,
            "accuracy": 1,
        },
        "geometry": {
            "type": "Point",
            "coordinates": [
                -118.46710503101347,
                33.9909333514159
            ]
        }
    },
    {
        "type": "Feature",
        "properties": {
            "timestamp": 1531007628.3774529
            "accuracy": 10,
        },
        "geometry": {
            "type": "Point",
            "coordinates": [
                -118.464851975441,
                33.990366257735
            ]
        }
    }]
}

Given that the provider API is intended for use by cities (and their vendors/patners) and NOT the general public, how should providers implement access control on their API endpoints?

Simple option would be for LADOT to issue a shared secret at the time when providers are granted an API.

Longer term, probably need a more cryptographic or account-oriented solution, since cities may want the ability to authorize 3rd parties (e.g. vendors) to access MDS data on their behalf.

We suggest that LADOT consider a regular time interval to evaluate increases in fleet size by a dockless company. Quarterly revisions makes sense to us. We suggest the following revised rules for Fleet Size rules E, F, and G:

e) All Operators are limited to a fleet-size of 500 Vehicles at launch. If a company wishes to deploy a combination of Vehicles (whether bikes, e-bikes, or scooters), the total number of vehicles that each company can launch cannot exceed a total of 500 Vehicles.

f) Operators will be allowed to increase the size of their fleets by 250 vehicles per quarter after reviewing bike placement plans with city officials and receiving approval.

g) The overall fleet size per Operator may not exceed 1,500 Vehicles.

First, I want to say that JUMP’s goal is to promote the safety of our riders while partnering with cities to provide access to the data required for city-planning. As noted during the workshop, sharing raw location data in the form of start and stop locations, route data and/or trip history presents a risk of re-identification that may endanger the privacy of our users.

Below, I have outlined a few approaches to lessening the privacy impact related to trip data, some of which have already been mentioned in issues #68, #66 & #61. JUMP would appreciate the opportunity to discuss any of these approaches in more detail with LADOT staff.

Trip Data Obfuscation Methods

• Limit location & time accuracy: limiting GPS precision based on population density of a particular city in combination with time aggregation is one means of anonymizing data. Depending on LA’s population density and trip count, providing GPS with two or three decimal places and on an hourly time aggregation may be a acceptable solution. One way to implement this within the MDS is to add an “accuracy” field in the collection of all timestamp and location data.

• Snap to nearest census block: operators can systematically fuzz the start and stop locations by “snapping” location data to the center of the nearest census block or predefined area. To maintain privacy, route location data should also be anonymized across location and time.

• Aggregate route data: operators may provide specific route data only once a minimum number of trips using that route occured in a specific time period. For example, route data would be provided for a trip from point A to point B only when at least 30 similar trips occured in a given time period. Start and stop locations could also be anonymized by either “snapping” or limiting GPS accuracy.

GBFS Feed - Real Time Availability

As designed, the GBFS feed does not adequately protect the privacy of its users. While the bikes “on trip” are not shown, anyone with access to the system can easily reconstruct a trip by tracking when bikes are reserved (at which time they disappear from the map) and when the reservation ends (when they reappear). This information can be used to build a database of historic trip information and ultimately re-identify users based on their trip history. One fix for this problem is to scramble the bike identifiers during a trip so it is impossible to determine which start and stops constitute a unique trip.

User Consent

Operators may offer users the choice to share trip data with cities without anonymization via opt-in. This could be in addition to a truly anonymized method above, which would not be subject to consent.

also swap ordering of device id and device type. then it's consistently big endian: company > device types > specific device

In the trips dataset, it would be useful to know how the user unlocked the dockless vehicle (e.g. smartphone, keypad, keycard, etc). Here in Phoenix, one of the dockless systems (GRiD/Social Bicycles) unlocks with a keypad or an RFID card.

Clarify that they cannot reassign these numbers. Per @jannasmith

Should the trip_duration value of the /trips endpoint include the time time between reservation and trip start (e.g. walk time)? In the context of rideshare, the data about time between request and arrival would be very important.

Incorporate a status for each endpoint to a meta endpoint for the Agency API

Happy to see y'all working on this. We're looking forward to contributing. Can we talk? [email protected].

Below, I suggest a (fairly minor) reorganization of the way this spec is presented. I will follow-up with a PR to introduce these changes, if one is warranted, after there's been a chance for some discussion.

Currently

From the README:

Currently, there are two implementations avaliable. Version v0.1 defines a data specification for providers to implement that the City can query in realtime, while v0.2 implements an API that providers can query for similar information and additional realtime parking, cost and trip permission information.
...
The specification will be versioned using Git tags

Given that this all still very early, the current demarcation into components v0.1 and v0.2 makes sense. I think what my proposal boils down to is re-labeling these components to provide for a little more clarity and maybe even more flexibility in the future.

Suggestion

• Rename Version v0.1 to the Provider spec (or similar)
• Rename Version v0.2 to the Agency spec (or similar)
• Use Git tags / semvar to version these component specs accordingly

Accessibility

The first goal in this reorg is to make explicit the different parties responsible for the different components of the spec. Using a name for the component that mirrors what data the component describes/contains, helps a newcomer jump in more easily.

This is consistent with how both GBFS and GTFS/RT name their feed components; both of which were specifically called out as inspiration for this spec.

Versioning

The second goal is to allow for independent versioning of these different components. Imagine for example a new field being required in trip data coming from the Provider -- the field could be added, version updated, without necessarily affecting the Agency side of the spec.

This naming scheme also eliminates possible confusion between the name of the component and the version of the component. In the current spec, we might have v2.0 of v0.1, and v1.0 of v0.2 -- this could get confusing quickly.

Future Growth

Finally, this naming scheme would allow for adding entirely new components, again more-or-less independently of the other components (versioning, naming, implementation, etc.) Maybe this isn't a concern right now, but at least in theory it would be easier to do so in the future.

Instead make a reference to the council motions in ReadMe.md

We request that LADOT remain firm on its commitment to locking mechanisms on dockless vehicles to prevent dockless bikes and scooters from blocking sidewalks for pedestrians, the elderly, and those with disabilities by removing the following sentence from Parking Rule B:

"However, locking mechanism requirement may be waived at LADOT discretion."

An alternative would be to require dockless operators to either have a lock-to mechanism, or to require dockless operators to include a notification mechanism in their app that reliably notifies users that their vehicle is properly parked upright in a geo-fenced, designated area by the city (similar to rules approved by the city of Austin).

Time fields like start_time, end_time, and route timestamp currently specify their type as Unix Timestamp. An example from route timestamp is 1531007628.3774529.

This leaves it open to interpretation and confusion what the correct format is.

GBFS spec specifies for instance

Integer POSIX timestamp

and provides an example 1434054678 or epoch seconds.

So using the example from route it should be either 1531007628 for epoch seconds. Or, if greater precision is required epoch milliseconds would be 531007628377.

In either case it would be good to clarify what the preferred format for providing time is. If you let me know your preference, I'll put out a PR.