Node 14 got released and has some nice things like improved performance and the optional chaining feature. We could wait until it hits LTS in October or be early adopters. Thoughts?

We are a major version behind

If we don't find a match for episode titles on OpenSubtitles, look on OMDB

Vulnerabilities

DepShield reports that this application's usage of express:4.17.1 results in the following vulnerability(s):

• (CVSS 7.5) CWE-22: Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal')

_{This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.}

Vulnerabilities

DepShield reports that this application's usage of lodash.memoize:4.1.2 results in the following vulnerability(s):

• (CVSS 7.4) CWE-471: Modification of Assumed-Immutable Data (MAID)
• (CVSS 6.5) [CVE-2018-3721] lodash node module before 4.17.5 suffers from a Modification of Assumed-Immutabl...

Occurrences

lodash.memoize:4.1.2 is a transitive dependency introduced by the following direct dependency(s):

• ts-jest:24.3.0
        └─ lodash.memoize:4.1.2

_{This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.}

Vulnerabilities

DepShield reports that this application's usage of express:4.16.4 results in the following vulnerability(s):

• (CVSS 7.5) CWE-22: Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal')

_{This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.}

Maybe I'm using it wrong but if I see an error passed through to the client, I don't see an entry in pm2 logs, so it needs investigation

It should be a local MongoDB setup

Might as well try both places

I'm usually not a fan of mocking but the service is flaky af

We have this code in the client, that should probably be in the API instead:

			/**
			 * Sometimes if OpenSubtitles doesn't have an episode title they call it
			 * something like "Episode #1.4", so discard that.
			 */
			episodeName = StringEscapeUtils.unescapeHtml4(episodeName);
			if (episodeName.startsWith("Episode #")) {
				episodeName = "";
			}

Node14 now warns about circular dependencies. We have some warning printed on start now.

The issue in our case is from the mongodb native driver, which is used by Mongoose. It is in the process of being fixed, so it will be resolved once there is a new Mongoose version available with the mongo driver upgrade

Automattic/mongoose#8842

This is something we currently do in UMS so we should probably do that here before our first release too. I don't think it is used much but we support adding the IMDb ID into the filename itself and looking up from that.

Vulnerabilities

DepShield reports that this application's usage of lodash.get:4.4.2 results in the following vulnerability(s):

• (CVSS 7.4) CWE-471: Modification of Assumed-Immutable Data (MAID)
• (CVSS 6.5) [CVE-2018-3721] lodash node module before 4.17.5 suffers from a Modification of Assumed-Immutabl...

Occurrences

lodash.get:4.4.2 is a transitive dependency introduced by the following direct dependency(s):

• pm2:4.2.3
        └─ vizion:2.0.2
              └─ lodash.get:4.4.2

_{This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.}

Vulnerabilities

DepShield reports that this application's usage of lodash.sortby:4.7.0 results in the following vulnerability(s):

• (CVSS 7.4) CWE-471: Modification of Assumed-Immutable Data (MAID)

Occurrences

lodash.sortby:4.7.0 is a transitive dependency introduced by the following direct dependency(s):

• jest:24.9.0
        └─ jest-cli:24.9.0
              └─ jest-config:24.9.0
                    └─ jest-environment-jsdom:24.9.0
                          └─ jsdom:11.12.0
                                └─ data-urls:1.1.0
                                      └─ whatwg-url:7.0.0
                                            └─ lodash.sortby:4.7.0
                                └─ whatwg-url:6.5.0
                                      └─ lodash.sortby:4.7.0

_{This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.}

Allow lookups to OpenSubtitles via the OpenSubs-flavored hash, or search strings.

We could use our own code, or the author of https://github.com/vankasteelj/opensubtitles-api has said he would be willing to merge the change if we contribute it

Axios is terrible at handling errors as documented in this issue axios/axios#960
We can possibly do a workaround from that issue, or pick something that does what it should

I think there is a plugin that lets it use TypeScript directly. Research needed

1.22.4 is available on my server so we should bump to that

Our responses are JSON when there is a result, but strings when there isn't. We should change that to have predictable types. Something like:

{ message: "Metadata not found" }

for no results

At the moment it only returns from IMDb, but we should attempt to hydrate the data with OpenSubtitles too. The code is already mostly there but unreachable

We should refactor the episode batch part to run whenever an episode is searched for

In our tests we are getting impossible line numbers, possibly due to ts-node (that's a guess).
e.g.

console.error
    Error: title is required
        at Object.exports.getBySanitizedTitle (/home/travis/build/UniversalMediaServer/api/src/controllers/media.ts:3018:11)
        at /home/travis/build/UniversalMediaServer/api/src/routes/media.ts:314:25
        at dispatch (/home/travis/build/UniversalMediaServer/api/node_modules/koa-compose/index.js:42:32)
        at /home/travis/build/UniversalMediaServer/api/node_modules/koa-router/lib/router.js:368:16
        at dispatch (/home/travis/build/UniversalMediaServer/api/node_modules/koa-compose/index.js:42:32)
        at /home/travis/build/UniversalMediaServer/api/node_modules/koa-compose/index.js:34:12
        at dispatch (/home/travis/build/UniversalMediaServer/api/node_modules/koa-router/lib/router.js:373:31)
        at dispatch (/home/travis/build/UniversalMediaServer/api/node_modules/koa-compose/index.js:42:32)
        at bodyParser (/home/travis/build/UniversalMediaServer/api/node_modules/koa-bodyparser/index.js:95:11)
        at processTicksAndRejections (internal/process/task_queues.js:97:5)
        at /home/travis/build/UniversalMediaServer/api/src/app.ts:624:3
        at /home/travis/build/UniversalMediaServer/api/src/app.ts:584:5

https://www.universalmediaserver.com/api/media/a04cfbeafc4af7eb/884419440

TypeError: Cannot read property 'match' of undefined
    at LibID.extend (/Users/kyle-dev/umsapi/node_modules/opensubtitles-api/lib/identify.js:71:36)
    at /Users/kyle-dev/umsapi/node_modules/opensubtitles-api/index.js:209:42
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at Object.exports.getByOsdbHash (/Users/kyle-dev/umsapi/src/controllers/media.ts:89:14)
    at /Users/kyle-dev/umsapi/src/routes/media.ts:7:3
    at bodyParser (/Users/kyle-dev/umsapi/node_modules/koa-bodyparser/index.js:95:5)
    at /Users/kyle-dev/umsapi/src/app.ts:35:3
    at /Users/kyle-dev/umsapi/src/app.ts:23:5

Vulnerabilities

DepShield reports that this application's usage of lodash.findindex:4.6.0 results in the following vulnerability(s):

• (CVSS 7.4) CWE-471: Modification of Assumed-Immutable Data (MAID)

Occurrences

lodash.findindex:4.6.0 is a transitive dependency introduced by the following direct dependency(s):

• pm2:4.2.3
        └─ vizion:2.0.2
              └─ lodash.findindex:4.6.0

_{This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.}

The UMS client currently looks for a match to the OS hash, and if that fails it looks for a metadata match based on a sanitised title. We should allow that to happen through our API too.

Mongoose gives better stack traces when exec() is used with async/await

https://mongoosejs.com/docs/promises.html#should-you-use-exec-with-await?

I found this example of what a very good result from OMDb can be like, and it would be cool to support that. It's from http://www.omdbapi.com/?i=tt3896198 but you need your API key too, which is free.

I think we should store all of the fields. They can all make a richer web interface experience.

{"Title":"Guardians of the Galaxy Vol. 2","Year":"2017","Rated":"PG-13","Released":"05 May 2017","Runtime":"136 min","Genre":"Action, Adventure, Comedy, Sci-Fi","Director":"James Gunn","Writer":"James Gunn, Dan Abnett (based on the Marvel comics by), Andy Lanning (based on the Marvel comics by), Steve Englehart (Star-Lord created by), Steve Gan (Star-Lord created by), Jim Starlin (Gamora and Drax created by), Stan Lee (Groot created by), Larry Lieber (Groot created by), Jack Kirby (Groot created by), Bill Mantlo (Rocket Raccoon created by), Keith Giffen (Rocket Raccoon created by), Steve Gerber (Howard the Duck created by), Val Mayerik (Howard the Duck created by)","Actors":"Chris Pratt, Zoe Saldana, Dave Bautista, Vin Diesel","Plot":"The Guardians struggle to keep together as a team while dealing with their personal family issues, notably Star-Lord's encounter with his father the ambitious celestial being Ego.","Language":"English","Country":"USA","Awards":"Nominated for 1 Oscar. Another 14 wins & 52 nominations.","Poster":"https://m.media-amazon.com/images/M/MV5BNjM0NTc0NzItM2FlYS00YzEwLWE0YmUtNTA2ZWIzODc2OTgxXkEyXkFqcGdeQXVyNTgwNzIyNzg@._V1_SX300.jpg","Ratings":[{"Source":"Internet Movie Database","Value":"7.6/10"},{"Source":"Rotten Tomatoes","Value":"85%"},{"Source":"Metacritic","Value":"67/100"}],"Metascore":"67","imdbRating":"7.6","imdbVotes":"526,626","imdbID":"tt3896198","Type":"movie","DVD":"22 Aug 2017","BoxOffice":"$389,804,217","Production":"Walt Disney Pictures","Website":"N/A","Response":"True"}

Vulnerabilities

DepShield reports that this application's usage of debug:2.6.9 results in the following vulnerability(s):

• (CVSS 7.5) CWE-400: Uncontrolled Resource Consumption ('Resource Exhaustion')

Occurrences

debug:2.6.9 is a transitive dependency introduced by the following direct dependency(s):

• debug:2.6.9

• express:4.16.4
        └─ body-parser:1.18.3
              └─ debug:2.6.9
        └─ debug:2.6.9
        └─ finalhandler:1.1.1
              └─ debug:2.6.9
        └─ send:0.16.2
              └─ debug:2.6.9

• morgan:1.9.1
        └─ debug:2.6.9

_{This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.}

Do we want ESLint? Or is there a better thing?

The project could not be analyzed because of build errors. Please review the error messages here. Another build will be scheduled within 24 hours. If the build is successful this issue will be closed, otherwise the error message will be updated.

_{This is an automated GitHub Issue created by Sonatype DepShield. GitHub Apps, including DepShield, can be managed from the Developer settings of the repository administrators.}

If I run yarn test, the file media.spec.ts gets timeouts on all of the should statements. The other files then pass.

From 3.5.1 to 4.2.1

Old clients should continue to work forever, so we need a way to identify which version of the API the client was designed to consume

The express error handler at the moment doesn't suit an API project as it displays the error to the developer as an HTML page.

Errors should be logged to the console in development

@js-kyle Since we are using your fork of imdb-api and the owner doesn't seem to be paying attention to the things we have contributed, I suggest we add the support for private.omdbapi.com to your fork, what do you think?
It would be ideal to be able to pass that option as a parameter

We need to make sure that we are returning a different error/message depending on whether a lookup has failed due to a real failure to find it (both APIs returned no results, or the results were not complete enough), or just a temporary failure e.g. high server load from the third parties.
We should also make sure that those temporary failures don't result in a FailedLookup being written, since in those cases it would be better to try until a real result is received, even if that result is just that they don't have that record.

We may already do some of this so this story involves some review too.

This one is potentially tricky so I made a separate story. Most of our information never changes except to start existing, but ratings are changeable, do we care? It's probably pretty low stakes, so one option is to not really care about it. Another option is to periodically update it in a similar way to how we check for missing lookups.

Thoughts @js-kyle ?

Happy to discuss this but I think it would be good to remove the document expiry on MediaMetadata. I know it makes it convenient for progressively adding more fields but given that we are attempting to cache rate-limited services, it stops us from accumulating good results over time. Thoughts?

See https://github.com/UniversalMediaServer/api/pull/70/files#r392632619

I noticed this project by chance https://www.npmjs.com/package/imdb-api and it got me wondering if our data could be more reliable if we use another service specifically for that info. I noticed that genres seem less reliable on OpenSubtitles.

Maybe we could validate the OpenSubs responses and if there are any missing fields we look at the other service.

Vulnerabilities

DepShield reports that this application's usage of lodash.foreach:4.5.0 results in the following vulnerability(s):

• (CVSS 7.4) CWE-471: Modification of Assumed-Immutable Data (MAID)

Occurrences

lodash.foreach:4.5.0 is a transitive dependency introduced by the following direct dependency(s):

• pm2:4.2.3
        └─ vizion:2.0.2
              └─ lodash.foreach:4.5.0

_{This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.}

The project could not be analyzed because of build errors. Please review the error messages here. Another build will be scheduled within 24 hours. If the build is successful this issue will be closed, otherwise the error message will be updated.

_{This is an automated GitHub Issue created by Sonatype DepShield. GitHub Apps, including DepShield, can be managed from the Developer settings of the repository administrators.}

Our most immediate model should allow us to store metadata that we receive from OpenSubtitles, for easier and throttle-free use in UMS.
It should store data like the IMDb ID, associated genres, actors, episode title, season number, episode number, movie year, etc.

Sometimes if open subtitles can't find metadata based on a specific hash it will return an empty response - how should that be handled?

Options:

• create the empty record, so subsequent lookups return empty db documents
• avoid creating the incomplete document

At the moment our code has a concept of a TV episode, but not a TV series. OMDb has the concept of a show and I think we should too. It would let us make our web interface way nicer - if a user goes to a folder for a show, instead of just shows a list of seasons, we could list our nice data for the show that isn't tied to any particular episode.