apify / apify-cli Goto Github PK

Use envVars from apify.json as environment variable for local act.
It can be overridden by user e.g: MY_ENV_VAR=test apify run

Hey, I just installed apify-cli using npm -g i apify-cli
I'm running Windows 7 64bit

node -v v9.5.0
npm -v 6.0.0

Whenever I run apify, I get the following error messsage:

(node:9672) Error Plugin: apify-cli: files attribute must be specified in C:\Users\fisknils\AppData\Roaming\npm\node_modules\apify-cli\package.json

I figured it might not be optimized for windows, so I also tried it on a linux box running

node -v v10.1.0
npm -v 6.1.0

But I get the (basically) the same error.

I also tried downgrading, through nvm, to the same version as someone from your support staff, v9.3.0, but sadly, this didn't solve my problems either.

fisknils@linuxburk11:~$ apify
(node:28307) Error Plugin: apify-cli: files attribute must be specified in 
/home/fisknils/.nvm/versions/node/v9.3.0/lib/node_modules/apify-cli/package.json
module: @oclif/[email protected]
plugin: apify-cli
root: /home/fisknils/.nvm/versions/node/v9.3.0/lib/node_modules/apify-cli
See more details with DEBUG=*
Apify command line client to help you create, develop, build and run Actor acts.

VERSION
  apify-cli/0.1.9 linux-x64 node-v9.3.0

USAGE
  $ apify [COMMAND]

COMMANDS
  call    Runs the act remotely on the Apify platform.
  create  Creates a new act project directory from a selected template.
  info    Displays information about current Apify settings.
  init    Initializes an act project in an existing directory.
  login   Logs in to the Apify platform using the API token.
  logout  Logs out of the Apify platform.
  push    Uploads the act to the Apify platform and builds it there.
  run     Runs the act locally in the current directory.

I tried googling, but couldn't find any references to this problem.

I'm attaching log files from running with DEBUG=*
linux-debuglog.txt
win-debuglog.txt

Also, this is the output from tree on the linux box after running apify init in an empty directory
tree.txt

The tree structure on the windows machine looks the same.

Would it be possible to add option to pass an input data to the actor? i.e.:
apify call xxx --input-file=~/input.json

If I set up some Environment variable on Apify, then update something on local machine and push new version, the variable are deleted.

What if instead of an array we pass an object with public & secret keys as arrays?

{
  public: [],
  secret: []
}

Every day when user uses Apify CLI for the first time and is online we should check for newer version.

For example the user with CLI from the last week gets an error Error: The ‘APIFY_DEFAULT_REQUEST_QUEUE_ID’ environment variable is not defined.

So that people will know what it is, how to use it, info that they shouldn't add this to git etc.

When I run the act once, it creates a requestQueue, it fails on something, so I update the code and run it once more but it continues in the queue and might crash because of the updates.

I would suggest adding some param for Apify run, maybe something like apify run -c / -clean that would delete the folder of request-queues before the run, because it is also good to have the possibility to continue in the queue.

It should work witout apify_storage folder.

If I call my own Act, not to have to put my name before name of the act, so my call could be just

apify call name-of-act

and not

apify call myusername/name-of-act

But this is a minor issue.

Users want to override to env attribute in apify.json for local actors runs.

• add support for file apify.local.json, where user can override env for local runs
• decribe it into readme

I had a directory with .gitignore like this:

node_modules
.idea

Then I ran apify init and now the file contains:

node_modules
.ideaapify_local

If actor has an INPUT schema, then "apify run" should validate the input against the schema, using the exactly same code as Apify cloud is doing that.

With some reasonable defaults, e.g. apify_local, node_modules etc.

Enable to use for example --inspect option and others from https://nodejs.org/docs/latest-v9.x/api/cli.html#cli_command_line_options.

I think we can create new option node-options for apify run like:
apify run --node-options='--inspect=1234 --no-warnings'

When I want to start developing non-javascript Actor locally, I certainly don't need to any node_modules (or do I?) and possibly even apify_local storage.

It would be nice to have such a blank template that would still have the functionality like Apify push, apify run etc.

This happened to me a few months ago and now to one user:

Executing "apify login" and then inputing token doesn't work.
"apify -t ***" doesn't work either
"apify -- token ***" works

Add a parameter to apify init command, which can define docker file which actor will use.

When you run apify init in a normal Node.js project with other main file than 'main.js', it succeeds, but then when you run apify run you get this error:

Error: The "main.js" file not found in the current directory. Call "apify init" to create it.

We shouldn't hard-code "main.js" anywhere, user can use any file. Just read it from package.json

I called apify push, my build is still running but CLI returned an error:

Error: Build failed! Log:

Maybe a timeout?

https://github.com/apifytech/apify-cli/blob/f15ac5e6789d5ba3dd506fee1b7ce8c5017b1e8d/src/templates/puppeteer_crawler/Dockerfile#L4

This COPY command probably flat folder structure us
COPY . . instead

After update to the latest, cli creates the key-value-stores and it should be key_value_stores

Implement command apify secrets:ls to list all secrets. Just keys not value.

I did apify push and when the build failed, it returned me the log but without the last rows where the error was. The log on website was ok.

The release schedule of apify-cli is much slower than of apify NPM package, but we need to make sure new projects always use the latest version. We need to run npm update apify in the new projects.

https://github.com/apifytech/apify-shared-js/pull/9/files

It will be great if there will be possible by the command "apify push" to push also INPUT.json to Actor. There could be added a parameter to force it.

https://github.com/apifytech/apify-cli/blob/f15ac5e6789d5ba3dd506fee1b7ce8c5017b1e8d/src/hooks/pre_run.js#L9

It is not necessary checking latest version each run.

The warning that you're running an old version is shown only if you run some command, not when just run "apify". It would be nicer to show this all the time (if possible)

So that people can understand what's in it and how to edit it. For example, to show how they can define env vars.

We should test each of the templates when running integration tests. Also test that they use up-to-date apify package and maybe other packages.

Even though I'm not logged in:

$ apify info
Error: You are not logged in with your Apify account. Call "apify login" to fix that.

Running the actor works fine, only to fail later:

$ apify run
Run: npm start

> [email protected] start /Users/jan/Projects/actor-xxx
> node main.js

INFO: Snapshotter: Setting max memory of this run to 4096 MB. Use the APIFY_MEMORY_MBYTES environment variable to override it.
INFO: AutoscaledPool state {"currentConcurrency":0,"desiredConcurrency":2,"systemStatus":{"isSystemIdle":true,"memInfo":{"isOverloaded":false,"maxOverloadedRatio":0.2,"actualRatio":null},"eventLoopInfo":{"isOverloaded":false,"maxOverloadedRatio":0.2,"actualRatio":null},"cpuInfo":{"isOverloaded":false,"maxOverloadedRatio":0.4,"actualRatio":null}}}
ERROR: PuppeteerPool: browser launch failed {"id":0}
  ApifyClientError: Apify Proxy password must be provided as parameter or "APIFY_PROXY_PASSWORD" environment variable!

That's a confusing behavior. apify run should fail fast and tell users what to do, rather then let them figure it out themselves.

http.get('http://api.apify.com/v2/logs/[LOG_ID]?stream=1')
.on('response', (response) => {
    response.on('data', function (chunk) {
        console.log(chunk.toString());
    });
    response.on('end', function () {
        console.log('Finished');
    });
});

Stream gets finished when run/build gets finished/failed so you don't need to request the state.

It would be lovely to be able to login to the account and call Apify pull actId/actName and it would download the code to the folder where I am.

For example, if there's an error in INPUT_SCHEMA.json and I run "apify push", the build log is not shown.

See https://www.apify.com/docs/sdk/apify-client-js/latest, there's nothing...

Sometimes we faced an issue that apify run takes too much time to start actor.

This would be a nice UX improvement, especially on slower connections where it takes long time to download puppeteer.

For command apify run it runs by default main.js file.

User should have option how to override it.

We should add some warning message to Apify CLI saying that “The apify_storage directory contains a previous state, the actor will continue where it left off. To start from the initial state, use —purge parameter to clean the apify_storage directory”. Only show this if there are some files except for INPUT.json.

The point is to make sure users are not surprised when the crawler immediately finishes when they restart the actor using "apify run".

Sometimes it takes ages to install all the packages to create a new folder. It could be confusing for new users to watch their console sitting and doing nothing. Maybe at least display something like 'installing, please wait' or 'loading'.

Updated npm script and local storage but "apify run" crashes

$ apify run
Run: npm start
Error: spawn npm ENOENT

https://stackoverflow.com/questions/43230346/error-spawn-npm-enoent

cwd.split('/').pop() won't work on windows where is \ used instead.

Use path.delimiter from path package.

IMHO apify info shouldn't print out secrets like API token and proxy password. For example, when user is entering token in apify login, it's hidded, while here it's not. That's not consistent behavior and it's a potential security issue. Let's just skip these fields altogether.

Hi

I just run the command apify login and here is the output of the command:

Error: Logging into Apify failed, token or userId is not correct.
Success: Logged into Apify!

In the end I'm apparently not logged. I dont think it's an issue with wrong credentials, tought, I can try again.

Here is the results when I run the command apify push:

Error: You aren't logged call "apify login" to process login.
Info: Push act-name to Apify.
Run: Zipping all act files ...
Error: Cannot read property 'keyValueStores' of undefined

Lets change the packaging so that it will only ignore .git directory and all files and dirs listed in .gitignore.

I am able to do this

apify create superlongdescriptiontotesthtelengthofthename

but at apify push I get error that the name can be max 30 chars, would be nice have the error at create.