Git Product home page Git Product logo

ka-mensa-fetch's Introduction

ka-mensa-fetch

CI Test Coverage Maintainability

Introduction

ka-mensa-fetch is one component in a three-part project whose goal it is to aggregate, process and visualize the Studierendenwerk Karlsruhe's canteen plans in ways superior to the official sources.

Disclaimer: This project is neither affiliated with nor endorsed by the Studierendenwerk Karlsruhe or the Karlsruhe Institute of Technology.

The entire project is written in JavaScript+TypeScript and is composed as follows:

  • ka-mensa-fetch: library package responsible for the fetching of raw plan data and conversion into canonical, easily digestible JSON documents
  • ka-mensa-api: NodeJS server that utilizes the fetcher to continuously collect meal plans and makes them available via REST API
  • ka-mensa-ui: single-page web app that loads meal plans from an API instance and displays them in a modern, responsive interface with filtering and color-coding capabilities

Installation

ka-mensa-fetch has a package on the NPM registry.

Make sure Node and NPM are available on your system, then use npm i ka-mensa-fetch to install it into your project's dependencies. Please note that it is packaged as an ECMAScript module and is incompatible with CommonJS.

TypeScript typings are available directly as part of the package.

Usage

Fetching a set of plans

You can call the fetcher as follows:

import { fetchMensa } from 'ka-mensa-fetch'

const plans = await fetchMensa(/* source, options */)
console.log(plans)

Arguments:

  • source: string: The data source (optional). Either 'simplesite' (the default) or 'jsonapi'.
  • options: object: Fetcher options (optional).
    • parallel: boolean: Optional, default: false. Whether to run all network requests in parallel. This speeds the fetch up significantly, but also increases server-side load and should therefore be used sparingly.

Additional options for 'simplesite' source:

  • canteens: string[]: Array of canteen ids for which plans are wanted. See src/data/canteens.ts for possible values.
  • dates: object[]: Array of date specifiers for which plans are wanted. They can have any one of the following forms (remember, JavaScript uses 0-indexed months):
    • { year: 2019, month: 11, day: 1 }
    • '2019-12-01'
    • new Date(2019, 11, 1)
    • 1575158400
  • sessionCookie: string: Optionally, a session cookie. Existence of the cookie could prevent redirects, see note below.

Additional options for 'jsonapi' source:

  • MANDATORY: auth: object = { user, password }: Authentication information for the API.

Return values:

The return value is a Promise resolving to an array of plans.

!!! WARNING !!! The result may contain all, some or none of the plans that were requested. It may also contain additional or even completely different plans. Handle with care.

Plan structure:

  • id: string: canteen identifier (e.g. 'adenauerring')
  • name: string: canteen name (e.g. 'Mensa Am Adenauerring'), may be null
  • date: object: plan date (e.g. { day: 2, month: 11, year: 2019 }) (note: month is 0-indexed)
  • lines: object[]: line array, containing objects of the following structure:
    • id: string: line id (e.g. 'l1'), may be null
    • name: string: line name (e.g. 'Linie 1'), may be null
    • meals: object[]: meal array, containing objects of the following structure:
      • name: string: meal name (e.g. 'Käseknacker mit Ketchup und Pommes')
      • price: string: human-readable price (e.g. '2,60 €'), may be empty
      • classifiers: string[]: meal classifiers (e.g. [ 'SAT' ])
      • additives: string[]: meal additives (e.g. [ '2', '3', 'ML' ])
Code example
fetchMensa('simplesite', { canteens: ['adenauerring', 'moltke'] })

Promise resolution value (shortened):

[
  {
    "id": "adenauerring",
    "name": "Mensa Am Adenauerring",
    "date": { "day": 2, "month": 11, "year": 2019 },
    "lines": [
      {
        "id": "l1",
        "name": "Linie 1",
        "meals": [
          {
            "name": "Käseknacker mit Ketchup und Pommes",
            "price": "2,60 €",
            "classifiers": [ "SAT" ],
            "additives": [ "2", "3", "ML" ]
          },
          //...
        ]
      },
      //...
    ]
  },
  {
    "id": "adenauerring",
    "name": "Mensa Am Adenauerring",
    "date": { "day": 3, "month": 11, "year": 2019 },
    "lines": [ /* ... */ ]
  },
  {
    "id": "adenauerring",
    "name": "Mensa Am Adenauerring",
    "date": { "day": 4, "month": 11, "year": 2019 },
    "lines": [ /* ... */ ]
  },
  {
    "id": "adenauerring",
    "name": "Mensa Am Adenauerring",
    "date": { "day": 5, "month": 11, "year": 2019 },
    "lines": [ /* ... */ ]
  },
  {
    "id": "adenauerring",
    "name": "Mensa Am Adenauerring",
    "date": { "day": 6, "month": 11, "year": 2019 },
    "lines": [ /* ... */ ]
  },
  {
    "id": "moltke",
    "name": "Mensa Moltke",
    "date": { "day": 2, "month": 11, "year": 2019 },
    "lines": [
      {
        "id": "wahl1",
        "name": "Wahlessen 1",
        "meals": [
          {
            "name": "Chicken Drum Sticks mit Sweet Chilli Soße",
            "price": "2,50 €",
            "classifiers": [],
            "additives": [ "5", "Se", "We" ]
          },
          //...
        ]
      },
      //...
    ]
  },
  {
    "id": "moltke",
    "name": "Mensa Moltke",
    "date": { "day": 3, "month": 11, "year": 2019 },
    "lines": [ /* ... */ ]
  },
  {
    "id": "moltke",
    "name": "Mensa Moltke",
    "date": { "day": 4, "month": 11, "year": 2019 },
    "lines": [ /* ... */ ]
  },
  {
    "id": "moltke",
    "name": "Mensa Moltke",
    "date": { "day": 5, "month": 11, "year": 2019 },
    "lines": [ /* ... */ ]
  },
  {
    "id": "moltke",
    "name": "Mensa Moltke",
    "date": { "day": 6, "month": 11, "year": 2019 },
    "lines": [ /* ... */ ]
  }
]

TypeScript Types

This library includes strong typing for its objects. The following types are available:

import {

  // types for canteen declarations (src/data/canteens.ts)
  Line,
  Canteen,

  // types for legend declaration (src/data/legend.ts)
  LegendItem,

  // types as they appear in fetch results
  // (plans contain lines, which contain meals)
  CanteenPlan,
  CanteenLine,
  CanteenMeal,
  
  // these types are used whenever dates are needed
  DateSpec,
  datelike,
  
  // types used for specifying fetcher options
  Options,
  SimpleSiteOptions,
  JsonApiOptions,
  AuthConfig

} from 'ka-mensa-fetch'

Caching

Because ka-mensa-fetch accesses sources not meant for automated processing, it cannot know or limit the data that it receives. Even if only a single day's plan is requested, a whole week or more might be obtained.

To be efficient, that data should not be thrown away but instead cached. Imagine Monday was requested but Monday-Friday were received, then Tuesday will already be available without making another request later.

Implementing caching is the polite and resourceful thing to do.

Avoiding redirects with session cookies

During the COVID-19 pandemic, the Studierendenwerk decided to redirect all requests site-wide to a press release informing about their measures. Obviously, this hinders data acquisition. The redirect can be averted though, by requesting a session cookie first and providing it with future requests. This is implemented in ka-mensa-fetch, to be used as follows:

import { fetchMensa, requestSessionCookie } from 'ka-mensa-fetch'

const session = await requestSessionCookie()
if (!session) {
  console.error('could not retrieve session cookie')
}

const plans = await fetchMensa('simplesite', {
  sessionCookie: session  // !!!
})

console.log(plans)

Multiple requests can be made with the same session cookie. Its lifetime is limited server-side, but I suspect it will be valid for at least 30 minutes. It might be valid much longer.

Again, exercise resourcefulness — obtaining a cookie has the overhead of an additional full request.

The JSON API source does not require a cookie to be set.

Packaged Data

ka-mensa-fetch includes some mappings to facilitate interaction with the API.

Canteens List

A list of canteens with their internal ids, display name, and lines. Every line also has an id and a display name.

Code example
import { canteens } from 'ka-mensa-fetch'
console.log(canteens)

Output (shortened):

[
  {
    "id": "adenauerring",
    "name": "Mensa Am Adenauerring",
    "lines": [
      { "id": "l1", "name": "Linie 1" },
      { "id": "l2", "name": "Linie 2" },
      //...
    }
  },
  //...
]

Legend

A list of meal qualifiers / additives / warnings / etc.

Code example
import { legend } from 'ka-mensa-fetch'
console.log(legend)

Output (shortened):

[
  { "short": "1", "label": "mit Farbstoff" },
  //...
  { "short": "VEG", "label": "vegetarisches Gericht" },
  { "short": "VG", "label": "veganes Gericht (ohne Fleischzusatz)" },
  //...
  { "short": "Er", "label": "Erdnüsse" },
  //...
]

Match Functions

Backwards-lookup of ids may be required in cases where only human-readable names are available. This functionality is used internally by the simplesite data source but also exposed as an export:

import { matchCanteenByName, matchLineByName } from 'ka-mensa-fetch'

console.log(matchCanteenByName('Mensa Am Adenauerring'))
// logs 'adenauerring' - the id used in the JSON API for that canteen

console.log(matchLineByName('adenauerring', '[pizza]werk Pizza 11-14 Uhr'))
// logs 'pizza' - the id used in the JSON API for that line

Data Sources

sw-ka simple site (web view for the visually impaired)

This is the default data source. The meal plan is extracted from the HTML page. This is relatively reliable as long as there are no structural changes.

The URL used is the following:

https://www.sw-ka.de/de/hochschulgastronomie/speiseplan/mensa_adenauerring/?view=ok&STYLE=popup_plain&c=adenauerring&p=1&kw=49

Here, adenauerring (appears twice!) is the canteen id and kw=49 indicates that the plan for the 49th calendar week is requested. The calendar weeks likely follow ISO-8601.

Unfortunately, the p parameter representing price category (1 for students, 2 for guests, 3 for employees, 4 for school children) has no effect. This appears to be a bug on the sw-ka site.

sw-ka JSON API

This source retrieves meal plans from the same API used by the official app. It requires authentication, which is why it is not the default source.

Endpoints:

  1. https://www.sw-ka.de/en/json_interface/general/ - used for obtaining up-to-date canteen and line names
  2. https://www.sw-ka.de/en/json_interface/canteen/ - used for obtaining the meal plans

There are no (known) parameters. The API returns all data from the beginning of the current week up to 2 weeks into the future, for all canteens.

Note that entries older than 1 day might be included, but they are always empty. This library filters those entries out.

Development

Contributions are welcome. Guidelines:

  • By contributing, you agree to make your changes available under the MIT license of this project.
  • Please write unit tests for as much code as possible.
    • To run: npm test
    • To create a coverage report: npm run coverage
  • Make sure to adhere to JS standard style and proper usage of TypeScript.
    • Linter: npm run lint
    • Automatic fixing of most style issues: npm run lint-fix

ka-mensa-fetch's People

Contributors

dependabot[bot] avatar github-actions[bot] avatar meyfa avatar renovate[bot] avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

ka-mensa-fetch's Issues

Retrieve current canteen+line names from JSON API

The jsonapi source does not take canteen and line names into account. It just assumes the standard name from bundled canteens.json. This information would be available via the https://www.sw-ka.de/json_interface/general/ endpoint.

Missing coverage for untested files

Files that are not tested, e.g. src/jsonapi/index.js among others, do not count towards test coverage. This makes it useless as a metric. NYC should be configured to generate coverage for everything.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

github-actions
.github/workflows/main.yml
  • actions/checkout v4
  • actions/setup-node v4
  • actions/checkout v4
  • actions/setup-node v4
  • actions/checkout v4
  • actions/setup-node v4
  • paambaati/codeclimate-action v5.0.0
.github/workflows/release-please.yml
  • google-github-actions/release-please-action v4
  • actions/checkout v4
  • actions/setup-node v4
npm
package.json
  • axios ^1.6.2
  • cheerio ^1.0.0-rc.12
  • moment ^2.29.4
  • @meyfa/eslint-config 5.1.0
  • @types/mocha 10.0.6
  • @types/node 20.12.2
  • axios-mock-adapter 1.22.0
  • c8 9.1.0
  • eslint 8.57.0
  • mocha 10.4.0
  • ts-node 10.9.2
  • typescript 5.4.5
  • node >=18.16.1

  • Check this box to trigger a request for Renovate to run again on this repository

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.