Additional actions about degit HOT 19 OPEN

I have made a project scaffolding project that I use all the time when I need to test something quickly. It contains the set of tools that I find myself always copying around anyway. It's already really useful, but I've had to include a self removing script into it, to do the bootstrapping. It can be seen here: https://github.com/gustavnikolaj/template-node

That means that I have to run it like so:

$ npx degit gustavnikolaj/template-node <targetDir> && node bootstrap.js

All I do in that script is to run some actions that execute npm commands to configure test and linting tools etc. As an example; I don't want to depend on a fixed version of say eslint, so instead of including it in a package.json file in the degit source repo, I run a command at install time that installs the latest version of it.

The same thing goes for package.json - if I include it in the repo, I'd have to go in and update it after setting it up - but running npm init -y in my bootstrap script does the right thing (picking information about the writer from git config, and picking the name of the package from the folder).

The obvious way of integrating this behavior would be to allow an action type that allowed the module to include such a boostrap script.

I understand why you might not want to allow arbitrary command execution, but maybe it would be okay if we required the user to answer y/N in a prompt before execution.

Another solution could be to add an action that would allow you to execute commands in the context of the directory.

[
    {
        "action": "shell",
        "shell": "git init"
    },
    {
        "action": "shell",
        "shell": "npm init -y"
    },
    {
        "action": "shell",
        "shell": "npm install --save-dev eslint ..."
    }
]

That would cover most of my use cases. I would also like a way to transform files after the fact though. Something that would allow me to define a script target in the package.json file that got created after running the npm init shell command.

I don't know how to do that with declarative actions, but maybe someone else has an idea? :-)

I'm happy to contribute PRs for any of these features if we can reach something that would work for us. I use my template project enough to justify improving it a little, and removing some of the ugly hacks :-)

from degit.

I was in need of this for rapidly creating some packages, so I made a wrapper over degit for this purpose specifically. it doesn't have the "exec" action proposed here to avoid the security issues mentioned, thought it might be useful to some other people checking this thread (like me).

from degit.

what if a simple action would be to display custom text/show post-actions?
[...]

That looks a lot like what @Rich-Harris suggested in #6 (although that predates the whole action concept).

Are we already unsafe today with postinstall scripts? (npm just runs them, it doesn't give me a chance to see what is going to do).

Yes, we are. This is not only a theory, it has been done already: https://eslint.org/blog/2018/07/postmortem-for-malicious-package-publishes (eslint-scope was taken over and had a new version released that harvested .npmrc files)

So, basically, how far should we worry?

People already run npm's postinstall hooks all the time without caring about it. And more often than not they will run npm install right after degit anyway. But I think we explored the subject enough - we need a decision from the module author. :-)

from degit.

You're right, did not see #6 . The only difference being: do it with actions and not a YAML file (seems better to me). Shell and/or script actions with prompts for safety

from degit.

I know I'm super late to this thread but, +1 on the "replace" action. Does anyone by any chance already have that action in a branch?

from degit.

I think this is great. For replace, what do you think of a templating library or were you trying to avoid that? I could see something like underscore's templates being useful in how it supports a variation for HTML-escaping (which is nice if you're calling replace on something going in the <title> as well as elsewhere) and also allows for custom syntax so someone could still use << foo >> if they wanted. In that case, could you get rid of the values key and by convention go with "the data loaded via your prompt" as the data? That brings up the question, could you load data via another method, such as an external data file? Or even modify degit.json to allow for a data key?

I wonder if template and replace are two separate actions. I could imagine it being useful to have something as simple as:

[
  {
    "action": "replace",
    "find": "foo",
    "replace": "bar",
    "files": ["src/**/*", "package.json"]
  }
]

from degit.

+1.

Maybe replace should just be a general transform configuration to pass the contents of the file to a function. This could be a generic JS file that exports a function, or it could be a specifically named module like degit-plugin-transformer.

from degit.

Ya. Or maybe also support for HTMLx template

from degit.

This is great @gustavnikolaj . Led me think about generic custom scripts too, maybe something like

[
    {
        "action": "script/run/node",
        "script": "bootstrap.js",
        "args": ""
    }
]

..if project has some special needs. This means degit could only have common actions, while specific ones would be customized by the repo itself.

from degit.

@bernardoadc That's a neat idea.

I think the main problem with a solution like that is security and trust. If you look at something like sveltejs/template's README degit is used to help people get started quickly - the example npx degit sveltejs/template svelte-app is featured at the very top.

If degit were to add features like the ones we have suggested, you would have to be much more careful about running degit-oneliners that you found online.

Instead of being a simple tool that is safe to run for everyone, it basically becomes equivalent to the curl ... | bash-pattern. Allowing people online to run arbitrary code on your machine is not a very good idea.

I completely understand if @Rich-Harris is not comfortable with a change like that, as the primary use case is to help people get going quickly - adding support for code-execution as we have asked for here would make it much less safe.

from degit.

Keeping the above in mind, I think that a shell action which prompts the user for confirmation before execution is about as far as we can reasonably push it. You would be able to use it to invoke custom code through node or any other available interpreter on the system, but at least the user would be aware.

I completely understand if even that is pushing it too far btw.

from degit.

How can we handle security properly when allowing to run custom scripts?

from degit.

How can we handle security properly when allowing to run custom scripts?

I guess the pedantic answer is that we can't.

Prompting the user (y/N) to confirm before executing a shell command is about as good as we can get, but it will still open the door for some dark-patterns, e.g. hiding malicious code in a bash script or non-obvious hooks. Consider this:

$ degit foo/bar
...

foo/bar wants to execute: $ npm install
Do you want to allow that? (y/N) > y

> npm install

It looks pretty benign on the surface. But with that in place, I could add a postinstall script to the package.json file that would harvest npm registry auth tokens and post them back to a server in my use, and then cleanly remove itself and the reference to the script from package.json.

You wouldn't know unless you read through the code before running the degit command.

That could be pretty disastrous if a malicious actor gets control of a repo that is frequently used as a degit template by lots of people.

from degit.

You're right @gustavnikolaj, I hadn't thought about security at all (so innocent of me). I agree we cannot make it more secure than to prompt the user, and that is not reasonably safe also.

(thinking out loud) what if a simple action would be to display custom text/show post-actions? Examples explain better:

$ degit foo/bar
...

foo/bar says: don't forget to bootstrap project
foo/bar bootstrap actions for you to run:
> npm install
> node bootstrap.js

$

This way nothing is done, it only shows what to do, in an easy copy paste operation. Would that be safer? Would people open scripts to see it's contents? Are we already unsafe today with postinstall scripts? (npm just runs them, it doesn't give me a chance to see what is going to do). Is it less safe a bootstrap.js file than a npm script? (which could call bootstrap.js anyway)

So, basically, how far should we worry?

from degit.

This is cool but I've been thinking it's awesome how minimal degit is. Perhaps you could make a new project with a different name and add the additional functionality to it?

from degit.

I also like the idea of being able to define custom-script to be run, but I understand the security concerns. I agree that it would probable be better to create a differente project like gitstrap or gitscript.

The use case I have in mind is to have a degit.yml on any folder defining scripts, pretty similar to package.json, and that you could run something like :

npx degit user/repo/folder run script-name param1 param2 param3

You could ask user for confirmation and also support a --force parameter.

from degit.

Was reading another issue on a totally different project, nonetheless the node_modules safety concern was also raised and I thought it would be good to quote them here (opinion from more people is good, hope you agree):

• microsoft/vscode-eslint#1012 (comment)
• microsoft/vscode-eslint#1012 (comment)

from degit.

the only action I think we really need is

{
        "action": "exec",
        "command": "npm", 
        "args": ["install"]
 }

because we can do any templating in pre and post install steps. You could fire off a full-on make command if you wanted to (npm even provides a lightweight cross-platform make cli), or just do some light m4-like templating with npm's own replace.

example:

"name": "my-foo-template",

"postinstall": "node postinstall.js"
...
"devDependencies":  {
  "replace": "latest"
}

postinstall.js

import exec from 'child-process'
exec("replace", ["'my-foo-template'", "'foo-instance'", "package.json"]

from degit.

Regarding the exec action proposed above… while that would be the all-powerful action to rule all actions, such power would make the security risks already raised in this thread become even more dire.

it doesn't necessary follow that exec has no place and should never be allowed, as it could prove very useful in repos that are under strict control and used only as intended by people who fully understand what's going on.

But given its potential as a footgun, let alone malicious acts, an exec action should at the very least require users to opt-in to its use, either by supplying an --allow-exec option to the degit command or by setting allowExec: true in a $HOME/.degit/config file.

Prompting the user to accept exec actions as they are encountered would help, but still would not be enough, as the concerns raised above also apply, with much severity due to the much greater power of exec.

from degit.