jerrysu / gulp-rsync Goto Github PK

Deploy breaks if new user requests the application files, while rsync is executed.

rsyncConf = {
progress: true,
incremental: true,
relative: true,
emptyDirectories: true,
recursive: true,
clean: true,
root: 'dist/',
archive: true,
};

I'm using the run-sequence package in tandem with gulp-rsync to sequentialize gulp tasks. I see this in the log:

[14:21:00] Finished 'rsync-assets' after 681 μs

Apparently, gulp-rsync returns right away and the rsync process is executed in the background, I guess.

How can I sequentialize gulp-rsync?

This plugin crashes if at least one filename contains round brackets.

gulp-rsync: /bin/sh: 1: Syntax error: "(" unexpected

events.js:141
      throw er; // Unhandled 'error' event
      ^
Error: Error: rsync exited with code 127
    at ChildProcess.<anonymous> (/Users/paddingme/Documents/paddingme's work/front/xsczda/client/node_modules/gulp-rsync/rsync.js:110:17)
    at emitTwo (events.js:87:13)
    at ChildProcess.emit (events.js:172:7)
    at maybeClose (internal/child_process.js:818:16)
    at Socket.<anonymous> (internal/child_process.js:319:11)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)
    at Pipe._onclose (net.js:469:12)

OS X EI Capitan

Currently I found out from the source code there's a workaround possible.
However, this would be confusing to anyone.

rsync({
    hostname: `rsync://${hostDetails.ip}`,
    destination: `${hostDetails.port}/volume`
})

// results in command:
// rsync -Rv file.txt rsync://0.0.0.0:000/volume

gulp-util is now deprecated. A guide to replace it can be found at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5

It would be nice if the chown was implemented. Sometimes settings permissions via chmod is not enough to ensure stuff works fine.

--chown=USER:GROUP
    This option forces all files to be owned by USER with group GROUP.
    This  is  a  simpler  interface  than  using  --usermap  and  --groupmap directly,
    but  it  is implemented using those options internally, so you cannot mix them.
    If either the USER or GROUP is empty, no mapping for the omitted user/group will
    occur.  If GROUP is empty, the trailing colon may be omitted, but if USER is
    empty, a leading colon must  be supplied.

    If you specify "--chown=foo:bar, this is exactly the same as specifying
    "--usermap=*:foo --groupmap=*:bar", only easier.

Thank you.

Is there any reason why #22 fix isn't already merged?

Tested on: Manjaro Linux

On our project we have filenames like "!$xyz.png". This is due internal requirements in the "RPG Maker MV" game engine. Normal rsync command on bash does copy them without any issues. But with gulp-rsync and these files, we have errors like the following:

gulp-rsync: rsync: [sender] link_stat "!.png" failed: No such file or directory (2)

all other files are synced without any problems. gulp then finishes with a non zero exit code and these error messages:

gulp-rsync: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1326) [sender=v3.2.4]

 Error in plugin "gulp-rsync"
Message:
    Error: rsync exited with code 23
    at ChildProcess.<anonymous> (node_modules/gulp-rsync/rsync.js:121:17)
    at ChildProcess.emit (node:events:527:28)
    at ChildProcess.emit (node:domain:537:15)
    at maybeClose (node:internal/child_process:1092:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)
    at Process.callbackTrampoline (node:internal/async_hooks:130:17)
Details:
    domainEmitter: [object Object]
    domainThrown: false

probably the escapeShellArgs function should be replaced by something like shell-escape

function escapeShellArg(arg) {
  if (!/(["'`\\\(\) ])/.test(arg)) {
    return arg;
  }
  arg = arg.replace(/([\\])/g, '/');
  return '"' + arg.replace(/(["'`\\])/g, '\\$1') + '"';
}

I used the following:

gulp.src('build/**')
  .pipe(rsync({
    hostname: 'myremotehost.com',
    destination: '/tmp',
    times: true,
    recursive: true,
    clean: true
  }));

So to test, I run it and it successfully copies over everything in my build directory to the proper place on the remote server.

Next, in order to test that the clean option is working, I go into my build directory and delete one of the subdirectories. I then run my task again. I check on the remote server and the subdirectory that I deleted is still in the target directory. Shouldn't it have been deleted with clean?

Am I misunderstanding something about how clean works?

如何用gulp-rsync把服务端的文件同步到本地
（how to use gulp-rsync let server data to local）Reverse synchronization

I am running the following command:

gulp.task('testRsync', function() {
    gulp.src('./_source/*.html')
        .pipe(rsync({
            options: {
                chmod: 'ugo=rwX',
                'r': true,
                'v': true,
                'delete': true,
                'verbose': true,
                'progress': true
            },
            username: 'root',
            root: '_source/',
            hostname: 'my_domain_here.com',
            destination: '/tmp',
            recursive: true
        }));
});

and all I get when I run it is:

[13:18:57] Using gulpfile C:\msweb\mswebsite_local\gulpfile.js
[13:18:57] Starting 'testRsync'...
[13:18:57] Finished 'testRsync' after 5.17 ms

now if I run my cygwin rsync, everything works fine...

any help is appreciated.

regards

Generates an “unknown filter rule” when a file to sync starts with a dash (i.e. “-file.txt” generates an “unknown file rule: `ile.txt’ error).

It would be good to prepend those files with a “./“.

Your first usage example has a return but the second one does not. That made all the difference for me. Without the return the rsync would not work. So maybe you can update your second usage example so that it has a return.

I had something that looked like this:

gulp.src(['dir1', 'dir2'])
  .pipe(rsync({...}));

This was causing an E2BIG error—I think because the number of matched files made the command too long. So I tried the recursive option, but that has no action. Tracing the issue, I found that it's short circuiting here because neither dir1 nor dir2 are the cwd.

Is this a bug? Why are those compared to cwd?

Thanks!

gulp-rsync crashes when it's output is redirected to a file:

$ gulp build > xxx    
/home/user/project/node_modules/gulp-rsync/log.js:8
  process.stdout.cursorTo(0);
                 ^
TypeError: Object #<SyncWriteStream> has no method 'cursorTo'
    at Console.log (/home/user/project/node_modules/gulp-rsync/log.js:8:18)
    at module.exports (/home/user/project/node_modules/gulp-util/lib/log.js:8:15)
    at module.exports (/home/user/project/node_modules/gulp-rsync/log.js:15:26)
    at /home/user/project/node_modules/gulp-rsync/index.js:112:13
    at Array.forEach (native)
    at /home/user/project/node_modules/gulp-rsync/index.js:111:29
    at Array.forEach (native)
    at Socket.handler (/home/user/project/node_modules/gulp-rsync/index.js:110:37)
    at Socket.EventEmitter.emit (events.js:95:17)
    at Socket.<anonymous> (_stream_readable.js:745:14)

I believe it should never ever call process.stdout.cursorTo(0). Removing that
line helps, as well as adding silent: true to arguments.

gulp.task('syncToServer', function() {
return gulp.src(uploadDir)
.pipe(rsync({
root: uploadDir,
username: 'web_server',
hostname: 'yun-s28.zw',
destination: '/data/web_server',
recursive: true,
exclude: [uploadDir + '/build', uploadDir + '/node_modules']
}));
});
,when i run 'gulp syncToServer' ,the directories in ‘exclude’ is still sync to my server.is there something wrong?

@jerrysu are you still managing this project, or is this just off to the wayside?

Is it possible to set a password for SSH? I didn't find the answer in the documentation.

Got this error:

[14:47:09] gulp-rsync: zsh:1: no matches found: --usermap=*:www-data --groupmap=*:www-data
[14:47:09] gulp-rsync:  [14:47:09] gulp-rsync: rsync: connection unexpectedly closed (0 bytes received so far) [sender]
[14:47:09] gulp-rsync:  [14:47:09] gulp-rsync: rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.2]

From:
Mac OS X Sierra, rsync 3.1.2

To:
Ubuntu 16.04, rsync 3.1.0

Tried config like this
chown: 'www-data:www-data',
and this:
chown: '"www-data:www-data"',

I'm syncing as root, so I'm fairly certain I have permissions. No idea what do do about this.

Hello.

Is there a way to sync with muliple servers (hostnames, really)?

If so, I can't for the life of me find it :)

Thanks for the plugin! It works great, but I have a small question about a part of the API that doesn't seem to gel with how gulp wants you to do things.

Initially I tried the following:

gulp.src('./path/to/assets/**')
  .pipe(rsync({
    hostname: 'host',
    destination: '/the/place/to/be'
  }));

My expectation was that, given a file ./path/to/assets/image.jpg, the file created on the remote would be /the/place/to/be/image.jpg. This would be consistent with gulp.dest and how the base option works:

options.base

Type: String
Default: everything before a glob starts (see [glob2base])

E.g., consider somefile.js in client/js/somedir:

gulp.src('client/js/**/*.js') // Matches 'client/js/somedir/somefile.js' and resolves `base` to `client/js/`
  .pipe(minify())
  .pipe(gulp.dest('build'));  // Writes 'build/somedir/somefile.js'

gulp.src('client/js/**/*.js', { base: 'client' })
  .pipe(minify())
  .pipe(gulp.dest('build'));  // Writes 'build/js/somedir/somefile.js'

Instead, I was surprised to find that the result file was /the/place/to/be/path/to/assets/image.jpg—as though I had set base to . in my gulp.src call. After some digging, I saw that gulp-rsync has a root option to accomplish what I wanted.

So it seems to me that gulp-rsync is essentially ignoring the base which, if it behaved as gulp expected, would make the root option redundant. It would be cool if the behavior were updated to behave more like gulp.dest and similar plugins.

Apologies in advance if I'm missing something!

Hi everybody! Maybe I'm too stupid to read the docs, but is it possible to rsync just the files from a directory 'dist/.' to a destination without the directory itself? I don't need the 'dist' folder on my server.

Thank you :)

here:
https://www.npmjs.com/package/gulp-rsync

text for chmod is equal to text for exclude

I've gotten the following gulp task to transfer files from a Windows 10 machine to a remote Linux server. However, files are being copied with a backslash "" vice forward slash "/" resulting in long file names like "js\myjs.js" instead of creating a folder called "js" with "myjs.js" in it. I haven't found an option or switch to fix the problem. Any ideas what I'm doing wrong?

gulp.task('deploy', function() {
return gulp.src('dist/*/')
.pipe(rsync({
options: {
chmod: 'ugo=rwX',
'r': true,
'v': true,
'delete': true,
'verbose': true,
'progress': true
},
username: "user",
password: "pass",
hostname: 'host.com',
destination: '~/test/',
recursive: true,
root: "dist/"
}));
});

By setting the exclude option a get an error, because in the rsync command all files from the source are listet. If i want to exclude some directories from the source an error occurs.

gulp-rsync: rsync error: some files could not be transferred (code 23)

I need to pass rsync-path in order to set the user to root, passing custom options like this doesn't work:

var gulp = require('gulp');
var rsync = require('gulp-rsync');

gulp.task('deploy', function() {
  gulp.src('build/**')
    .pipe(rsync({
      username: 'user',
      hostname: 'host',
      destination: '~/',
      options: {
        "rsync-path": "sudo rsync"
      }
    }));
});

Adding the following to index.js, after line 97 (https://github.com/jerrysu/gulp-rsync/blob/master/index.js#L97) picks up the options to add to the rsync command properly.

if (options.options) {
         for (var key in options.options) { config.options[key] = options.options[key]; }
}

Is there a better way to accomplish this?

Any possibility of getting chown support added ?

As described here it would be helpfull set this info on the parameters.

Hey,

I have set gulp-rsync to to sync a selection of files over to a server with the clean option enabled. If I delete a file then then run my gulp command to do the sync, the file is not deleted on the server.

I have tried using a glob to select the files like so: .src(config.paths.srcAssets + '**/*') as well as just the directory like this: .src(config.paths.srcAssets) but it seems to make no difference.

Is there a way to ensure that a local file deletion is replicated in the sync?

Thanks,
Neil

My configuration file is:
var gulp = require('gulp');
rsync = require('gulp-rsync');
gulp.task('rsync', function() {
return gulp.src('_site/**')
.pipe(rsync({
root: '_site',
hostname: 'ftp.domain.com',
username: '[email protected]',
destination: 'public_html',
incremental: true,
recursive: true,
compress: true,
times: true,
progress: true,
exclude: []
}));
});

And this is when I run the task:
[21:51:54] Starting 'rsync'...
[21:51:57] gulp-rsync: Starting rsync to [email protected]@ftp.web.com:/pub lic_html...
[21:51:57] gulp-rsync: 'rsync' is not recognized as an internal or external comm
[21:51:57] gulp-rsync:
[21:51:57] gulp-rsync: perable program or batch file.
[21:51:57] 'rsync' errored after 2.99 s
[21:51:57] Error in plugin 'gulp-rsync'
Message:
Error: rsync exited with code 1
at ChildProcess.<anonymous> (c:\Users\chris\desktop\fido\node_modules\gu lp-rsync\rsync.js:110:17)
at emitTwo (events.js:87:13)
at ChildProcess.emit (events.js:172:7)
at maybeClose (internal/child_process.js:818:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:211:5)
[21:51:57] gulp-rsync: Completed rsync.

Can you have a remote source and a local destination?

When the rsync fails, gulp-rsync doesn't log any errors in the terminal window.

Here's the task:
gulp.task('deploy', function() {
gulp.src('dist/**')
.pipe(rsync({
root: 'dist',
hostname: 'mydomain.com',
destination: '~/mypath',
recursive: true,
progress: true // the transfer progress for each file will be displayed in the console
}));
});

Here's the output:
$ gulp deploy
[19:28:36] Starting 'deploy'...
[19:28:36] Finished 'deploy' after 4.06 ms

It's probably failing because I'm not specifying login credentials / don't have a ssh key set up, but I can't tell because there's no logging.

I am not sure why I am running into this error, I have not seen them before with this module.

[23:06:48] Starting 'deploy'...
[23:06:51] gulp-rsync: Starting rsync to [email protected]:/path/to/server/directory...
internal/child_process.js:297
    throw errnoException(err, 'spawn');
          ^
Error: spawn E2BIG
    at exports._errnoException (util.js:846:11)
    at ChildProcess.spawn (internal/child_process.js:297:11)
    at exports.spawn (child_process.js:330:9)
    at Object.rsync.execute (/path/to/project/node_modules/gulp-rsync/rsync.js:92:22)

Here's the gulpfile config:

var rsync = require('gulp-rsync');
gulp.task('deploy', function() {
  return gulp.src('**')
    .pipe(rsync({
        hostname: 'myserver.com',
        destination: '/path/to/server/directory',
        username: 'tri',
        progress: true,
        times: true,
        exclude: ['node_modules', '.DS_Store']
    }));
});

This is the default for streams2 highWaterMark option.
I've fixed it by adding {highWaterMark: 100000000} as a parameter.
line 32: return through.obj({highWaterMark: 100000000}, function(file, enc, cb) {

I can rsync (using the same key I use to ssh), like this from the command line:
rsync -r -a -v -e "ssh -i ~/.ssh/my-key-local-dev" & etc

How would I add the path to my public key to gulp-rsync config option?

I have run into the following issue under Windows: hashicorp/vagrant#3256. When I call

rsync.exe -e ssh --chmod=ug=rwX,o=rxX -avtzP ...

my issue is fixed. However, I cannot find a way to pass the --chmod-option to gulp-rsync. Can you help or add a "free option string". That would be fantastic. Thank you.

I would like to copy a ../path/outside of the current directory with the gulpfile.js to a host. It is really ok in this case to have the directory above the current one. I get this error:
Error: Source contains paths outside of root

Other npm modules like 'del' offer a force: true option for this use case.

Oh, btw when I use ../path/outside as src, what is ok in rsync, the source path is reduced to . in the resulting rsync command, like this:
rsync -anRvz --progress --delete . host:/target/directory
so I used the path module to resolve the path gulp.src(path.resolve(dist))

gulp-rsync does not seem to honor the times configuration option.

To replicate:

1. Set the command option to true for debugging
2. Set the silent option to false for debugging
3. Set the archive option to either true or false (try both ways - it makes no difference)
4. Set the silent option to false (false overrides a behavior of archive but is not mutually-exclusive in rsync)

Since you set command to true, you will see the command that is being run, and it will not include the --no-times long option

Expected behavior:
When the times configuration parameter is set to either true or false the corresponding --times/--no-times command line parameter should be set for rsync (either -t or --times to enable, or --no-times to disable). Note that setting archive to true sets the --times command line parameter to true internally within rsync, so having --archive and --times set at the same time is non-harmful, but redundant. However, having --archive and --no-times set at the same time is useful and perfectly valid.

Here is my entire gulp task:

gulp.task( 'copy-sources',
    function( done )
    {
        gulp.src([
            srcDir
        ])
            .pipe(
                rsync({
                    command: true, // set this to true to see the rsync command that will run
                    root: srcDir, // when set to the source code root, rsync will show '.' (cwd) as the source directory
                    hostname: dstHost,
                    destination: dstDir,
                    archive: true,
                    times: false,
                    silent: false, // set to true to make the console less chatty
                    shell: 'ssh',
                    clean: false, // set to false to leave "missing" files from the orgin alone on the destination. true to delete them (sync)
                    dryrun: false, // set to true to see what will be done without doing anything
                    recursive: true,
                    exclude: [  '.DS_Store', '.buildpath', '.externalToolBuilders', '.git', 
								'.gitignore', '.project',, '.settings', 'logs' ] // directories you don't want to sync
              
                            })
            );
        done();
    }
);

Temporary workaround:
This is kludgy, and I'm sure you would do it differently, but I was able to insert this at line 81 of rsync.js to get the desired results:

	if( undefined !== this._options.t && false === this._options.t ) {
		args.push( "--no-times" );
	}

It may be worth noting that the rsync command does not change to include either --times or the --no-times option using any combination of true or false for the archive or the times configuration option. The times configuration option seems to have zero effect on anything.

My particular use case requires all of the options supplied by archive except that I need to drop modification times, which I was able to achieve using the workaround that I wouldn't have had to implement if the times configuration option were working as expected.

Error after updating version from 009 to 010

[22:12:04] gulp-rsync: Starting rsync to host:domen/www/...
[22:12:04] 'deploy' errored after 16 ms
[22:12:04] Error: Cannot find module 'gulp-util'
Require stack:

• /mnt/c/OSPanel/webwork/domen/node_modules/gulp-rsync/rsync.js
• /mnt/c/OSPanel/webwork/domen/node_modules/gulp-rsync/index.js
  at Module._resolveFilename (node:internal/modules/cjs/loader:1039:15)
  at Module._load (node:internal/modules/cjs/loader:885:27)
  at Module.require (node:internal/modules/cjs/loader:1105:19)
  at require (node:internal/modules/cjs/helpers:103:18)
  at rsync.execute (/mnt/c/OSPanel/webwork/domen/node_modules/gulp-rsync/rsync.js:94:5)
  at DestroyableTransform._flush (/mnt/c/OSPanel/webwork/domen/node_modules/gulp-rsync/index.js:154:19)
  at DestroyableTransform.prefinish (/mnt/c/OSPanel/webwork/domen/node_modules/readable-stream/lib/_stream_transform.js:138:10)
  at DestroyableTransform.emit (node:events:513:28)
  at DestroyableTransform.emit (node:domain:552:15)
  at prefinish (/mnt/c/OSPanel/webwork/domen/node_modules/readable-stream/lib/_stream_writable.js:619:14)

The rsync command would looks similar to this:
rsync -rave "ssh -i /path/to/my/key.pem" example/* [email protected]:/var/www/html/example/

I'm wondering how do I achieve the same thing on gulp-rsync ?