metalwhale / newswaters Goto Github PK

What happened

skimmer cron job collect-item-urls encountered the following error message:

Container is terminated because of OOMKilled. It exited with exit code 137. It is not started and not ready.

Investigation

The skimmer process consumed excessive RAM, leading to the termination of the job. It appears that item ID 38115989 with url:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=cf8e8658100d4eae80ce9b21f7a81cb024dd5057

is the root cause.

Temporary fix

Create a fake record to ensure that the job ignores this item in the next run:

INSERT INTO item_urls (item_id, html, text, summary, status_code, status_note, created_at, updated_at)
    VALUES (38115989, NULL, NULL, NULL, -1, 'https://github.com/metalwhale/newswaters/issues/20', NOW(), NOW());

Suggestion

Option 1

Change text width at service.rs from std::usize::MAX to a smaller number (such as 4800, which is the SKIMMER_TEXT_MAX_TOTAL_LENGTH used for shortening the stories' texts)

html2text::from_read_with_decorator(html.as_bytes(), 4800, TrivialDecorator::new())

Option 2

Create a record at item_urls table with status_code = -1 before starting and update it after finished