Git Product home page Git Product logo

scheduled-task's Introduction

Scheduled task ๐Ÿ•’

Open in MATLAB Online

This is a prototype of scheduled task, a service aiming at scheduling data analysis, simulation and AI model training with MATLAB.

Table of Contents:

Create your first task

Schedule your task execution

Automate live script execution overnight

Under the hood: Github Action implementation

Configuration required

Document your work

Create your first task

The basic example will be checking Bitcoin prices daily, and storing it into a new row of an existing csv file.

mkdir Tasks
mkdir Data

First iteration of the task executed locally and interactively:

jason = webread("https://api.coinbase.com/v2/prices/BTC-USD/spot")
jason = struct with fields:
    data: [1x1 struct]

T = struct2table(jason.data)
amount base currency
1 63300.09 BTC USD
T.time = datetime("now")
amount base currency time
1 63300.09 BTC USD 26-Apr-2024 12:08:59
writetable(T,'Data/btc.csv')

Second iteration: Simply read the existing dataset as a timetable and add a row

% Read input data
T = readtable("Data/btc.csv");
jason = webread("https://api.coinbase.com/v2/prices/BTC-USD/spot");
Tnew = struct2table(jason.data);
Tnew.time = datetime("now");
%  Transform data to fit the input format
Tnew.amount = double(string(Tnew.amount));
Tnew.base = cellstr(Tnew.base);
Tnew.currency = cellstr(Tnew.currency);
T = [T;Tnew] % adding a row
amount base currency time
1 6.3300e+04 'BTC' 'USD' 26-Apr-2024 12:08:59
2 6.3404e+04 'BTC' 'USD' 26-Apr-2024 12:09:48
% Write output data
writetable(T,'Data/btc.csv')

Save this calculation into a script to run daily:

edit Tasks/task1.m
% Read input data
T = readtable("Data/btc.csv");
jason = webread("https://api.coinbase.com/v2/prices/BTC-USD/spot");
Tnew = struct2table(jason.data);
Tnew.time = datetime("now");
%  Transform data to fit the input format
Tnew.amount = double(string(Tnew.amount));
Tnew.base = cellstr(Tnew.base);
Tnew.currency = cellstr(Tnew.currency);
T = [T;Tnew]; % adding a row
% Write output data
writetable(T,'Data/btc.csv')

Schedule your task execution

addpath("code/");
filepath = 'Tasks/task1.m';
schedule = "*/5 * * * *"; % every 5 minutes
% output = "Data/"; % optionally specify output to zip and save
task = scheduleTask(filepath, schedule)

Automate live script execution overnight

This task is going to generate a report in different formats (md, html, ipynb, docx, pdf).

The results can be served up:

Create the mlx file to execute:

edit Tasks/task2.mlx

Schedule the execution:

filepath = 'Tasks/task2.mlx';
schedule = "0 0  * * *"; % every night at midnight
output = "Reports/"; % optionally specify output to zip and save
task = scheduleTask(filepath, schedule,output)

Under the hood: Github Action implementation

In the .github/workflows/ directory, create a new file called task.yml and add the following code.

edit .github/workflows/task.yml
name: MATLAB task
run-name: ${{ github.actor }} is scheduling a MATLAB task
on: 
  schedule:
    - cron: "*/5 * * * *"
  workflow_dispatch: {}

jobs:
  task:
    runs-on: ubuntu-latest
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v4
      
      # Sets up MATLAB on a GitHub-hosted runner
      - name: Set up MATLAB
        uses: matlab-actions/setup-matlab@v2

      # You can use "run-command" to execute custom MATLAB scripts, functions, or statements
      - name: Run custom testing procedure
        uses: matlab-actions/run-command@v2
        with:
          command: disp('Running my task!'); addpath('Tasks'); task1;

      # Commit and push the result of the MATLAB task
      - name: Commit and push changes
        run: |
          date > last-run.txt
          git config user.name github-actions
          git config user.email [email protected]
          git add .
          git commit -m "generated"
          git push

      # Save the result as artifact
      - name: Archive output data
        uses: actions/upload-artifact@v4
        with:
          name: bitcoin-price-history
          path: Data/btc.csv

This will run the task1.m on a specific schedule, here every 5 minutes

This follows the cron syntax:

# โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ minute (0โ€“59)
# โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ hour (0โ€“23)
# โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ day of the month (1โ€“31)
# โ”‚ โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ month (1โ€“12)
# โ”‚ โ”‚ โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ day of the week (0โ€“6) (Sunday to Saturday;
# โ”‚ โ”‚ โ”‚ โ”‚ โ”‚                                   7 is also Sunday on some systems)
# โ”‚ โ”‚ โ”‚ โ”‚ โ”‚
# โ”‚ โ”‚ โ”‚ โ”‚ โ”‚
# * * * * * <command to execute>

We are also adding a workflow_dispatch to enable a manual execution of the workflow in the actions tab:

image_0.png

Configuration required

This implementation saves the result of the task by pushing back the changes. For this ensure that you give Read and write permissions to the workflow in Settings/Actions

image_1.png

Document your work

export("GettingStarted.mlx", "README.md");

Resources

scheduled-task's People

Contributors

yanndebray avatar

Stargazers

Nicole Bonfatti avatar

Watchers

 avatar

Forkers

ppeeling

scheduled-task's Issues

Add second example

With timetable:

TT = readtimetable('Data/btc.csv')
...
writetimetable(TT,'Data/btc.csv')

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.