apify / apify-cli Goto Github PK
View Code? Open in Web Editor NEWApify command-line interface helps you create, develop, build and run Apify actors, and manage the Apify cloud platform.
Home Page: https://docs.apify.com/cli
Apify command-line interface helps you create, develop, build and run Apify actors, and manage the Apify cloud platform.
Home Page: https://docs.apify.com/cli
Use envVars from apify.json as environment variable for local act.
It can be overridden by user e.g: MY_ENV_VAR=test apify run
Hey, I just installed apify-cli using npm -g i apify-cli
I'm running Windows 7 64bit
node -v v9.5.0
npm -v 6.0.0
Whenever I run apify, I get the following error messsage:
(node:9672) Error Plugin: apify-cli: files attribute must be specified in C:\Users\fisknils\AppData\Roaming\npm\node_modules\apify-cli\package.json
I figured it might not be optimized for windows, so I also tried it on a linux box running
node -v v10.1.0
npm -v 6.1.0
But I get the (basically) the same error.
I also tried downgrading, through nvm, to the same version as someone from your support staff, v9.3.0, but sadly, this didn't solve my problems either.
fisknils@linuxburk11:~$ apify
(node:28307) Error Plugin: apify-cli: files attribute must be specified in
/home/fisknils/.nvm/versions/node/v9.3.0/lib/node_modules/apify-cli/package.json
module: @oclif/[email protected]
plugin: apify-cli
root: /home/fisknils/.nvm/versions/node/v9.3.0/lib/node_modules/apify-cli
See more details with DEBUG=*
Apify command line client to help you create, develop, build and run Actor acts.
VERSION
apify-cli/0.1.9 linux-x64 node-v9.3.0
USAGE
$ apify [COMMAND]
COMMANDS
call Runs the act remotely on the Apify platform.
create Creates a new act project directory from a selected template.
info Displays information about current Apify settings.
init Initializes an act project in an existing directory.
login Logs in to the Apify platform using the API token.
logout Logs out of the Apify platform.
push Uploads the act to the Apify platform and builds it there.
run Runs the act locally in the current directory.
I tried googling, but couldn't find any references to this problem.
I'm attaching log files from running with DEBUG=*
linux-debuglog.txt
win-debuglog.txt
Also, this is the output from tree on the linux box after running apify init in an empty directory
tree.txt
The tree structure on the windows machine looks the same.
Would it be possible to add option to pass an input data to the actor? i.e.:
apify call xxx --input-file=~/input.json
If I set up some Environment variable on Apify, then update something on local machine and push new version, the variable are deleted.
What if instead of an array we pass an object with public & secret keys as arrays?
{
public: [],
secret: []
}
Every day when user uses Apify CLI for the first time and is online we should check for newer version.
For example the user with CLI from the last week gets an error Error: The ‘APIFY_DEFAULT_REQUEST_QUEUE_ID’ environment variable is not defined.
So that people will know what it is, how to use it, info that they shouldn't add this to git etc.
When I run the act once, it creates a requestQueue, it fails on something, so I update the code and run it once more but it continues in the queue and might crash because of the updates.
I would suggest adding some param for Apify run, maybe something like apify run -c / -clean
that would delete the folder of request-queues before the run, because it is also good to have the possibility to continue in the queue.
It should work witout apify_storage folder.
If I call my own Act, not to have to put my name before name of the act, so my call could be just
apify call name-of-act
and not
apify call myusername/name-of-act
But this is a minor issue.
Users want to override to env attribute in apify.json for local actors runs.
I had a directory with .gitignore like this:
node_modules
.idea
Then I ran apify init
and now the file contains:
node_modules
.ideaapify_local
If actor has an INPUT schema, then "apify run" should validate the input against the schema, using the exactly same code as Apify cloud is doing that.
With some reasonable defaults, e.g. apify_local
, node_modules
etc.
Enable to use for example --inspect
option and others from https://nodejs.org/docs/latest-v9.x/api/cli.html#cli_command_line_options.
I think we can create new option node-options for apify run like:
apify run --node-options='--inspect=1234 --no-warnings'
When I want to start developing non-javascript Actor locally, I certainly don't need to any node_modules (or do I?) and possibly even apify_local storage.
It would be nice to have such a blank template that would still have the functionality like Apify push, apify run etc.
This happened to me a few months ago and now to one user:
Executing "apify login" and then inputing token doesn't work.
"apify -t ***" doesn't work either
"apify -- token ***" works
Add a parameter to apify init command, which can define docker file which actor will use.
When you run apify init
in a normal Node.js project with other main file than 'main.js', it succeeds, but then when you run apify run
you get this error:
Error: The "main.js" file not found in the current directory. Call "apify init" to create it.
We shouldn't hard-code "main.js" anywhere, user can use any file. Just read it from package.json
I called apify push
, my build is still running but CLI returned an error:
Error: Build failed! Log:
Maybe a timeout?
This COPY command probably flat folder structure us
COPY . .
instead
After update to the latest, cli creates the key-value-stores and it should be key_value_stores
Implement command apify secrets:ls
to list all secrets. Just keys not value.
I did apify push and when the build failed, it returned me the log but without the last rows where the error was. The log on website was ok.
The release schedule of apify-cli
is much slower than of apify
NPM package, but we need to make sure new projects always use the latest version. We need to run npm update apify
in the new projects.
It will be great if there will be possible by the command "apify push" to push also INPUT.json to Actor. There could be added a parameter to force it.
It is not necessary checking latest version each run.
The warning that you're running an old version is shown only if you run some command, not when just run "apify". It would be nicer to show this all the time (if possible)
So that people can understand what's in it and how to edit it. For example, to show how they can define env vars.
We should test each of the templates when running integration tests. Also test that they use up-to-date apify
package and maybe other packages.
Even though I'm not logged in:
$ apify info
Error: You are not logged in with your Apify account. Call "apify login" to fix that.
Running the actor works fine, only to fail later:
$ apify run
Run: npm start
> [email protected] start /Users/jan/Projects/actor-xxx
> node main.js
INFO: Snapshotter: Setting max memory of this run to 4096 MB. Use the APIFY_MEMORY_MBYTES environment variable to override it.
INFO: AutoscaledPool state {"currentConcurrency":0,"desiredConcurrency":2,"systemStatus":{"isSystemIdle":true,"memInfo":{"isOverloaded":false,"maxOverloadedRatio":0.2,"actualRatio":null},"eventLoopInfo":{"isOverloaded":false,"maxOverloadedRatio":0.2,"actualRatio":null},"cpuInfo":{"isOverloaded":false,"maxOverloadedRatio":0.4,"actualRatio":null}}}
ERROR: PuppeteerPool: browser launch failed {"id":0}
ApifyClientError: Apify Proxy password must be provided as parameter or "APIFY_PROXY_PASSWORD" environment variable!
That's a confusing behavior. apify run
should fail fast and tell users what to do, rather then let them figure it out themselves.
http.get('http://api.apify.com/v2/logs/[LOG_ID]?stream=1')
.on('response', (response) => {
response.on('data', function (chunk) {
console.log(chunk.toString());
});
response.on('end', function () {
console.log('Finished');
});
});
Stream gets finished when run/build gets finished/failed so you don't need to request the state.
It would be lovely to be able to login to the account and call Apify pull actId/actName and it would download the code to the folder where I am.
For example, if there's an error in INPUT_SCHEMA.json and I run "apify push", the build log is not shown.
See https://www.apify.com/docs/sdk/apify-client-js/latest, there's nothing...
Sometimes we faced an issue that apify run takes too much time to start actor.
This would be a nice UX improvement, especially on slower connections where it takes long time to download puppeteer.
For command apify run
it runs by default main.js file.
User should have option how to override it.
We should add some warning message to Apify CLI saying that “The apify_storage directory contains a previous state, the actor will continue where it left off. To start from the initial state, use —purge parameter to clean the apify_storage directory”. Only show this if there are some files except for INPUT.json.
The point is to make sure users are not surprised when the crawler immediately finishes when they restart the actor using "apify run".
Sometimes it takes ages to install all the packages to create a new folder. It could be confusing for new users to watch their console sitting and doing nothing. Maybe at least display something like 'installing, please wait' or 'loading'.
Updated npm script and local storage but "apify run" crashes
$ apify run
Run: npm start
Error: spawn npm ENOENT
https://stackoverflow.com/questions/43230346/error-spawn-npm-enoent
cwd.split('/').pop()
won't work on windows where is \
used instead.
Use path.delimiter
from path
package.
IMHO apify info
shouldn't print out secrets like API token and proxy password. For example, when user is entering token in apify login
, it's hidded, while here it's not. That's not consistent behavior and it's a potential security issue. Let's just skip these fields altogether.
Hi
I just run the command apify login
and here is the output of the command:
Error: Logging into Apify failed, token or userId is not correct.
Success: Logged into Apify!
In the end I'm apparently not logged. I dont think it's an issue with wrong credentials, tought, I can try again.
Here is the results when I run the command apify push
:
Error: You aren't logged call "apify login" to process login.
Info: Push act-name to Apify.
Run: Zipping all act files ...
Error: Cannot read property 'keyValueStores' of undefined
Lets change the packaging so that it will only ignore .git
directory and all files and dirs listed in .gitignore
.
I am able to do this
apify create superlongdescriptiontotesthtelengthofthename
but at apify push I get error that the name can be max 30 chars, would be nice have the error at create.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.