Git Product home page Git Product logo

Comments (5)

cowboyd avatar cowboyd commented on August 17, 2024 1

I can answer more on this later, but yes, this is normal behavior. The reason is because Effection Tasks, while not promises themselves can be consumed as promises. If you consume it as a promise, however, we need to let consumers know that a value will never be arriving because it the task was halted. As a result, whenever a Task is halted, its associated Promise is rejected. Again, you only see a HaltError if you are consuming a Task as a promise. E.g.

let task = run(function*() {
  yield; // just yield forever.
});

task.halt();

await task; //<== throws HaltError

It's annoying that the debugger is picking it up, and I think we could perhaps do better and only reject the promise if we know that we have consumers of the promise.

Thanks for reporting this, since it is confusing, and I think that we could make it less so.

from effection.

cowboyd avatar cowboyd commented on August 17, 2024 1

@kowan One thing I noted is that in most cases, you do not need to call spawn(). spawn() should only be used for code that you want to run asynchronously. For example, the code where you get your result:

let result = yield spawn(d.check_for_new_videos());

result is Task that will eventually return the result you want. In order to get the actual result from the task, you would say:

let resultTask = yield spawn(d.check_for_new_videos());
let result = yield resultTask;

However, you can just skip the spawn and access the result of the check_for_new_videos() operation directly.

let result = yield d.check_for_new_videos()

Does that make sense? I've created a gist out of your files so that we can have a common place to talk about it. Have a look: https://gist.github.com/cowboyd/a13897e6f8dbffbfb75bdd6a2a4b1eac

Also, feel free to drop by on Discord https://discord.gg/r6AvtnU

Gist
Example from https://github.com//issues/662#issuecomment-1243863041 - main.mjs
Discord
Check out the Frontside community on Discord - hang out with 193 other members and enjoy free voice and text chat.

from effection.

cowboyd avatar cowboyd commented on August 17, 2024

@kovan Would you mind posting your code that is using Effection? Because the promise is only created if someone is calling then(), catch(), or finally() on a Task, which means that someone is consuming the task as a promise.

from effection.

kovan avatar kovan commented on August 17, 2024

Sure, this is the code:

import { google } from 'googleapis'
import { Firestore } from '@google-cloud/firestore'
import { main, createQueue, spawn, all } from 'effection';
import "path"
import _ from "lodash"




class DB {
    constructor() {
        this.firestore = new Firestore()
    }
    *init() {
        const metaDoc = doc(this.firestore, "meta/meta")
        let snapshot = yield getDoc(metaDoc)
    }
    *get(path, id) {
        docRef = doc(this.firestore, path, id)
        docSnp = yield getDoc(docRef)
        return docSnp.exists ? docSnp : null
    }
}



function* dereferencePages(data, apiMethod, apiParams) {
    while (true) {
        if (!("nextPageToken" in data))
            break
        apiParams.pageToken = response.data.nextPageToken
        response = yield apiMethod(apiParams)
        data = { ...response.data }
    }
}


function* getWithCache(db, apiMethod, apiParams, id) {
    let headers = {}
    let cache = db.firestore.collection("cache")
    let cached_ref = cache.doc("id")
    let cached_doc = yield cached_ref.get()

    if (cached_doc.exists) {
        if (_.get(cached_doc, "videosdb.etag")) {
            headers["If-None-Match"] = cached_doc.videosdb.etag
        }
    }

    apiParams.headers = headers

    let response = yield apiMethod(apiParams)

    let result = null
    switch (response.status) {
        case 304:
            result = {
                doc: cached_doc,
                modified: false
            }
            break
        default:
            let full_data = dereferencePages(response.data, apiMethod, apiParams)
            let doc_ref = cache.doc(id)
            yield doc_ref.set(full_data.items)

            result = {
                doc: full_data,
                modified: true
            }
    }


    return result
}

class PlaylistRetriever {
    constructor(db, api, queue) {
        this.db = db
        this.queue = queue
        this.api = api

    }
    *start() {
        let channel_id = process.env.YOUTUBE_CHANNEL_ID
        let params = {
            "part": "snippet,contentDetails,statistics",
            "id": channel_id
        }
        console.info("Retrieving channel " + channel_id)
        let method = this.api.channels.list.bind(this.api)
        let channel_info = yield getWithCache(this.db,
            method,
            params,
            channel_id)
        console.debug(channel_info)

        this.YT_CHANNEL_NAME = channel_info.snippet.title
        let doc = this.db.firestore.doc("channel_infos/" + channel_id)

        yield doc.set(channel_info, { merge: true })
        let playlist_ids = null
        if (process.env.DEBUG) {
            let params = {
                "part": "contentDetails",
                "channelId": channel_id
            }
            let response = yield this.api.channelSections.list(params)

        }
        let playlistQueue = createQueue("Playlist queue")

    }

}

class Downloader {
    constructor(exclude_transcripts = false) {
        this.valid_video_ids = []
        this.db = new DB()
        this.api = google.youtube({
            version: 'v3',
            auth: process.env.YOUTUBE_API_KEY
        })
    }
    *check_for_new_videos() {
        console.info("Sync start")
        let retriever = new PlaylistRetriever(this.db, this.api)

        yield spawn(retriever.start())
        console.info("Sync finished")
        return true

    }
}



main(function* () {
    console.debug(process.env)
    try {
        let d = new Downloader()
        let result = yield spawn(d.check_for_new_videos())
        console.info("RESULT: " + result)
    } catch (err) {
        console.error("ERROR: " + err)
    }


})

And here is the package.json, in case you want to run it:

{
    "name": "backend",
    "version": "1.0.0",
    "main": "index.js",
    "license": "MIT",
    "private": true,
    "dependencies": {
        "@google-cloud/firestore": "^6.0.0",
        "effection": "^2.0.4",
        "googleapis": "^107.0.0",
        "lodash": "^4.17.21",
        "slugify": "^1.6.5"
    },
    "type": "module",
    "scripts": {
        "start": "node app.js"
    }
}

from effection.

kovan avatar kovan commented on August 17, 2024

Now it doesn't happen anymore. Also, I was wondering how to wait for the spawn() calls. So two issues solved at once! thank you.

As we said the behavior is a bit(a lot) obscure, but now it works for me and I understand why, so I'm happy.

See ya!

from effection.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.