Git Product home page Git Product logo

webcron's Introduction

Webcron

Continuous Integration License GitHub release GitHub issues

A server that will periodically download web content and store it in text, PDF, JPEG, and PNG format.

This service can be run via Docker container (recommended) or by npm.

Running

# Docker (recommended)
docker run --name webcron --interactive --tty -p 18080:3000 webcron:canary

# Manually
npm run --config=config.yaml

Development Notes

#Upgrade dependencies.
npm i npm-check-updates
ncu -u
npm install

Configuration

---
- name: "Websites"
  outputDirectory: "output/websites"
  format: ["txt", "pdf", "png", "jpg"]
  # https://crontab.guru/every-30-minutes
  schedule: "*/5 * * * *"
  browser:
    flags: []
    headless: true
    timeout: 60000
    tabs: 3
    width: 3840
    height: 2160
    # https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagegotourl-options
    waitUntil: "domcontentloaded"
  urls:
    - https://twitter.com/search?q=news&src=typd&lang=en
    - https://www.youtube.com/feed/trending

Example configuration to download what's trending on Twitter and YouTube.

Run with Docker and Custom Configuration

mkdir -p $HOME/webcron/output
# Places config.yaml in the output directory.
cp config.yaml $HOME/webcron/output/config.yaml
# NOTE: Don't change /home/webcron/output/config.yaml path. It is the mounted path of the file inside of the container. 
docker run --name webcron --interactive --tty --publish 8181:3000 --volume $HOME/webcron/output:/home/webcron/output jeremyje/webcron:canary serve -- --config=/home/webcron/output/config.yaml

# Run the container using a specific version of webcron.
docker rm -f webcron; docker run --name webcron --interactive --tty --publish 8181:3000 --volume $PWD/config.yaml:/home/webcron/config.yaml --volume $PWD/output:/tmp/output/news/ docker.io/jeremyje/webcron:v0.0.0-df10bb3 run serve -- --config=/home/webcron/config.yaml

Example Production Configuration Using Docker and Systemd

install.sh

#!/bin/bash

SERVICE_DIR=/etc/systemd/system/

chmod 644 *.service

InstallService() {
  service_file=$1
  chmod 644 ${service_file}
  sudo systemctl stop ${service_file}
  sudo systemctl disable ${service_file}
  sudo cp -f ${service_file} ${SERVICE_DIR}
  sudo systemctl enable ${service_file}
  sudo systemctl start ${service_file}
}

mkdir -p $HOME/webcron/output
cp -f $HOME/config.yaml $HOME/webcron/output/config.yaml

LOCAL_WORKSPACE=$HOME/webcron/output
CONTAINER_WORKSPACE=/home/webcron/output
DOCKER_IMAGE=jeremyje/webcron:canary
docker pull ${DOCKER_IMAGE}
docker rm webcron -f
docker create --name webcron --interactive --tty --publish 8181:3000 --volume ${LOCAL_WORKSPACE}:${CONTAINER_WORKSPACE} ${DOCKER_IMAGE} serve -- --config=${CONTAINER_WORKSPACE}/config.yaml
InstallService "webcron.service"

webcron.service

[Unit]
Description=webcron
Requires=docker.service
After=docker.service

[Service]
TimeoutStartSec=0
ExecStart=/usr/bin/docker start --attach webcron
ExecStop=/usr/bin/docker stop --time 2 webcron
Type=simple
User=jeremyje
Restart=always
RestartSec=1

[Install]
WantedBy=multi-user.target

eslint

# Activate environment
export PATH=$PWD/build/toolchain/nodejs/bin/:$PWD/node_modules/.bin/:$PATH
# Install eslint
npm install --save-dev @typescript-eslint/parser @typescript-eslint/eslint-plugin eslint typescript
npx eslint .

Based on https://typescript-eslint.io/getting-started/

webcron's People

Contributors

dependabot-preview[bot] avatar dependabot[bot] avatar jeremyje avatar lgtm-com[bot] avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

Forkers

lgtm-migrator

webcron's Issues

[DepShield] (CVSS 5.3) Vulnerability due to usage of kind-of:4.0.0

Vulnerabilities

DepShield reports that this application's usage of kind-of:4.0.0 results in the following vulnerability(s):


Occurrences

kind-of:4.0.0 is a transitive dependency introduced by the following direct dependency(s):

jest:25.5.4
        └─ @jest/core:25.5.4
              └─ jest-haste-map:25.5.1
                    └─ sane:4.1.0
                          └─ micromatch:3.1.10
                                └─ snapdragon:0.8.2
                                      └─ base:0.11.2
                                            └─ cache-base:1.0.1
                                                  └─ has-value:1.0.0
                                                        └─ has-values:1.0.0
                                                              └─ kind-of:4.0.0

This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.

[DepShield] (CVSS 4.3) Vulnerability due to usage of bl:4.0.2

Vulnerabilities

DepShield reports that this application's usage of bl:4.0.2 results in the following vulnerability(s):


Occurrences

bl:4.0.2 is a transitive dependency introduced by the following direct dependency(s):

puppeteer:5.2.1
        └─ tar-fs:2.1.0
              └─ tar-stream:2.1.3
                    └─ bl:4.0.2

This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.

[DepShield] (CVSS 7.5) Vulnerability due to usage of express:4.17.1

Vulnerabilities

DepShield reports that this application's usage of express:4.17.1 results in the following vulnerability(s):

This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.

[DepShield] (CVSS 5.3) Vulnerability due to usage of kind-of:5.1.0

Vulnerabilities

DepShield reports that this application's usage of kind-of:5.1.0 results in the following vulnerability(s):


Occurrences

kind-of:5.1.0 is a transitive dependency introduced by the following direct dependency(s):

jest:25.5.4
        └─ @jest/core:25.5.4
              └─ jest-haste-map:25.5.1
                    └─ sane:4.1.0
                          └─ micromatch:3.1.10
                                └─ snapdragon:0.8.2
                                      └─ define-property:0.2.5
                                            └─ is-descriptor:0.1.6
                                                  └─ kind-of:5.1.0

This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.

[DepShield] (CVSS 7.5) Vulnerability due to usage of hosted-git-info:2.8.8

Vulnerabilities

DepShield reports that this application's usage of hosted-git-info:2.8.8 results in the following vulnerability(s):


Occurrences

hosted-git-info:2.8.8 is a transitive dependency introduced by the following direct dependency(s):

concurrently:5.3.0
        └─ read-pkg:4.0.1
              └─ normalize-package-data:2.5.0
                    └─ hosted-git-info:2.8.8

This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.

[DepShield] (CVSS 7.4) Vulnerability due to usage of lodash.sortby:4.7.0

Vulnerabilities

DepShield reports that this application's usage of lodash.sortby:4.7.0 results in the following vulnerability(s):


Occurrences

lodash.sortby:4.7.0 is a transitive dependency introduced by the following direct dependency(s):

jest:25.5.4
        └─ @jest/core:25.5.4
              └─ jest-config:25.5.4
                    └─ jest-environment-jsdom:25.5.0
                          └─ jsdom:15.2.1
                                └─ whatwg-url:7.1.0
                                      └─ lodash.sortby:4.7.0

This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.

[DepShield] (CVSS 7.5) Vulnerability due to usage of acorn:6.4.1

Vulnerabilities

DepShield reports that this application's usage of acorn:6.4.1 results in the following vulnerability(s):


Occurrences

acorn:6.4.1 is a transitive dependency introduced by the following direct dependency(s):

jest:25.5.4
        └─ @jest/core:25.5.4
              └─ jest-config:25.5.4
                    └─ jest-environment-jsdom:25.5.0
                          └─ jsdom:15.2.1
                                └─ acorn-globals:4.3.4
                                      └─ acorn:6.4.1

This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.

[DepShield] (CVSS 7.4) Vulnerability due to usage of ini:1.3.8

Vulnerabilities

DepShield reports that this application's usage of ini:1.3.8 results in the following vulnerability(s):


Occurrences

ini:1.3.8 is a transitive dependency introduced by the following direct dependency(s):

pkg:5.3.0
        └─ prebuild-install:6.0.1
              └─ rc:1.2.8
                    └─ ini:1.3.8

This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.

[DepShield] (CVSS 9.8) Vulnerability due to usage of lodash:4.17.19

Vulnerabilities

DepShield reports that this application's usage of lodash:4.17.19 results in the following vulnerability(s):


Occurrences

lodash:4.17.19 is a transitive dependency introduced by the following direct dependency(s):

concurrently:5.3.0
        └─ lodash:4.17.19

jest:25.5.4
        └─ @jest/core:25.5.4
              └─ @jest/reporters:25.5.1
                    └─ istanbul-lib-instrument:4.0.1
                          └─ @babel/traverse:7.9.6
                                └─ lodash:4.17.19
              └─ @jest/transform:25.5.1
                    └─ @babel/core:7.9.6
                          └─ @babel/generator:7.9.6
                                └─ lodash:4.17.19
                          └─ @babel/helper-module-transforms:7.9.0
                                └─ lodash:4.17.19
                          └─ lodash:4.17.19
              └─ jest-config:25.5.4
                    └─ jest-environment-jsdom:25.5.0
                          └─ jsdom:15.2.1
                                └─ request-promise-native:1.0.8
                                      └─ request-promise-core:1.1.3
                                            └─ lodash:4.17.19
              └─ jest-snapshot:25.5.1
                    └─ @babel/types:7.9.6
                          └─ lodash:4.17.19

This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.

[DepShield] (CVSS 5.3) Vulnerability due to usage of kind-of:3.2.2

Vulnerabilities

DepShield reports that this application's usage of kind-of:3.2.2 results in the following vulnerability(s):


Occurrences

kind-of:3.2.2 is a transitive dependency introduced by the following direct dependency(s):

jest:25.5.4
        └─ @jest/core:25.5.4
              └─ jest-haste-map:25.5.1
                    └─ sane:4.1.0
                          └─ micromatch:3.1.10
                                └─ braces:2.3.2
                                      └─ fill-range:4.0.0
                                            └─ is-number:3.0.0
                                                  └─ kind-of:3.2.2
                                      └─ snapdragon-node:2.1.1
                                            └─ snapdragon-util:3.0.1
                                                  └─ kind-of:3.2.2
                                └─ snapdragon:0.8.2
                                      └─ base:0.11.2
                                            └─ cache-base:1.0.1
                                                  └─ has-value:1.0.0
                                                        └─ has-values:1.0.0
                                                              └─ is-number:3.0.0
                                                                    └─ kind-of:3.2.2
                                                  └─ to-object-path:0.3.0
                                                        └─ kind-of:3.2.2
                                            └─ class-utils:0.3.6
                                                  └─ static-extend:0.1.2
                                                        └─ object-copy:0.1.0
                                                              └─ kind-of:3.2.2
                                      └─ define-property:0.2.5
                                            └─ is-descriptor:0.1.6
                                                  └─ is-accessor-descriptor:0.1.6
                                                        └─ kind-of:3.2.2
                                                  └─ is-data-descriptor:0.1.4
                                                        └─ kind-of:3.2.2

This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.

[DepShield] (CVSS 7.4) Vulnerability due to usage of lodash.memoize:4.1.2

Vulnerabilities

DepShield reports that this application's usage of lodash.memoize:4.1.2 results in the following vulnerability(s):


Occurrences

lodash.memoize:4.1.2 is a transitive dependency introduced by the following direct dependency(s):

ts-jest:25.5.1
        └─ lodash.memoize:4.1.2

This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.

[DepShield] (CVSS 7.5) Vulnerability due to usage of debug:2.6.9

Vulnerabilities

DepShield reports that this application's usage of debug:2.6.9 results in the following vulnerability(s):


Occurrences

debug:2.6.9 is a transitive dependency introduced by the following direct dependency(s):

compression:1.7.4
        └─ debug:2.6.9

express:4.17.1
        └─ body-parser:1.19.0
              └─ debug:2.6.9
        └─ debug:2.6.9
        └─ finalhandler:1.1.2
              └─ debug:2.6.9
        └─ send:0.17.1
              └─ debug:2.6.9

jest:25.5.4
        └─ @jest/core:25.5.4
              └─ jest-haste-map:25.5.1
                    └─ sane:4.1.0
                          └─ micromatch:3.1.10
                                └─ extglob:2.0.4
                                      └─ expand-brackets:2.1.4
                                            └─ debug:2.6.9
                                └─ snapdragon:0.8.2
                                      └─ debug:2.6.9

morgan:1.10.0
        └─ debug:2.6.9

serve-index:1.9.1
        └─ debug:2.6.9

This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.

[DepShield] (CVSS 8.2) Vulnerability due to usage of y18n:4.0.0

Vulnerabilities

DepShield reports that this application's usage of y18n:4.0.0 results in the following vulnerability(s):


Occurrences

y18n:4.0.0 is a transitive dependency introduced by the following direct dependency(s):

concurrently:5.3.0
        └─ yargs:13.3.2
              └─ y18n:4.0.0

jest:25.5.4
        └─ @jest/core:25.5.4
              └─ jest-runtime:25.5.4
                    └─ yargs:15.3.1
                          └─ y18n:4.0.0
        └─ jest-cli:25.5.4
              └─ yargs:15.3.1
                    └─ y18n:4.0.0

This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.

Docker images should use the Makefile to build

There's currently build step duplication between the docker images and makefile. Just run it through the makefile. Also remove manual nodejs install since the makefile also installs a local copy of nodejs.

[DepShield] (CVSS 4.3) Vulnerability due to usage of bl:4.1.0

Vulnerabilities

DepShield reports that this application's usage of bl:4.1.0 results in the following vulnerability(s):


Occurrences

bl:4.1.0 is a transitive dependency introduced by the following direct dependency(s):

puppeteer:10.1.0
        └─ tar-fs:2.0.0
              └─ tar-stream:2.2.0
                    └─ bl:4.1.0

This is an automated GitHub Issue created by Sonatype DepShield. Details on managing GitHub Apps, including DepShield, are available for personal and organization accounts. Please submit questions or feedback about DepShield to the Sonatype DepShield Community.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.