Git Product home page Git Product logo

gulp-merge-json's Introduction

gulp-merge-json

Version Downloads Build Status Coverage License

A gulp plugin for deep-merging multiple JSON files into one file. Export as JSON or a node module.

Usage

gulp.src('jsonFiles/**/*.json')
	.pipe(merge(options))
	.pipe(gulp.dest('./dist'));

Options

Key Type Default Description
fileName String combined.json Output filename
edit Function json => json Edit function (add/remove/edit keys during merge)
transform Function json => json Transform final merged object (similar to edit but applied at the end)
startObj Object/Array {} Starting object to merge into (useful for providing default values)
endObj Object/Array Object to merge after file merging complete (useful for overwriting with special values)
exportModule Boolean/String false Output module.exports = {MERGED_JSON_DATA}; or {exportModule} = {MERGED_JSON_DATA} when string passed
concatArrays Boolean false Whether to concatenate arrays instead of merging
mergeArrays Boolean true Whether to merge arrays or overwrite completely
customizer Function Custom merge function for use with mergeWith
jsonReplacer Function Custom JSON replacer function passed to stringify
jsonSpace String \t String used for white space by stringify
json5 Boolean false Use JSON5 instead of JSON for parse and stringify

Examples

var merge = require('gulp-merge-json');

/**
 * Basic functionality
 */
gulp.src('jsonFiles/**/*.json')
	.pipe(merge())
	.pipe(gulp.dest('./dist'));

/**
 * Edit JSON with function
 */
gulp.src('jsonFiles/**/*.json')
	.pipe(merge({
		fileName: 'file.json',
		edit: (parsedJson, file) => {
			if (parsedJson.someValue) {
				delete parsedJson.otherValue;
			}

			return parsedJson;
		},
	}))
	.pipe(gulp.dest('./dist'));

/**
 * Edit final JSON with transformer function
 */
gulp.src('jsonFiles/**/*.json')
	.pipe(merge({
		fileName: 'file.json',
		transform: (mergedJson) => {
			return {
				key: {
					type: 'data',
					...mergedJson,
				};
			};
		},
	}))
	.pipe(gulp.dest('./dist'));

/**
 * Provide a default object (files are merged in order so object values will be overwritten)
 */
gulp.src('jsonFiles/**/*.json')
	.pipe(merge({
		startObj: { someKey: 'defaultValue' },
	}))
	.pipe(gulp.dest('./dist'));

/**
 * Provide an overwriting object (merged at the end)
 */
gulp.src('jsonFiles/**/*.json')
	.pipe(merge({
		endObj: { someKey: 'specialValue' },
	}))
	.pipe(gulp.dest('./dist'));

/**
 * Output module.exports = {JSON_DATA}
 */
gulp.src('jsonFiles/**/*.json')
	.pipe(merge({
		exportModule: true,
	}))
	.pipe(gulp.dest('./dist'));

/**
 * Output a custom variable = {JSON_DATA}
 */
gulp.src('jsonFiles/**/*.json')
	.pipe(merge({
		fileName: 'dataModule.js',
		exportModule: 'const myVar',
	}))
	.pipe(gulp.dest('./dist'));

/**
 * Provide replacer and space options for JSON.stringify
 */
gulp.src('jsonFiles/**/*.json')
    .pipe(merge({
        jsonSpace: '  ',
        jsonReplacer: (key, value) => {/*...*/}
    })
    .pipe(gulp.dest('./dist'));

/**
 * Use a customizer function for custom merging behavior
 */
gulp.src('jsonFiles/**/*.json')
  .pipe(merge({
    customizer: (objA, objB) => {
      // Example: Concat arrays but only keep unique values
      if (Array.isArray(objA) && Array.isArray(objB)) {
        return objA.concat(objB).filter((item, index, array) => (
          array.indexOf(item) === index
        ));
      }

      return undefined;
    },
  }))
  .pipe(gulp.dest('./dist'));

/**
 * JSON5
 */
gulp.src('jsonFiles/**/*.json5')
	.pipe(merge({
		json5: true,
	}))
	.pipe(gulp.dest('./dist'));

Example Input

/*
	json/defaults.json
 */
{
	"key1": {
		"data1": "value1",
		"data2": "value2"
	},
	"key2": {
		"dataA": "valueA",
		"dataB": {
			"a": "b",
			"c": "d"
		}
	}
}

/*
	json/development.json
 */
{
	"key1": {
		"data1": "devValue"
	},
	"key2": {
		"dataB": {
			"c": "DEV MODE!"
		}
	},
	"key3": {
		"important": "value"
	}
}

Example Output

/*
	dist/combined.json
 */
{
	"key1": {
		"data1": "devValue",
		"data2": "value2"
	},
	"key2": {
		"dataA": "valueA",
		"dataB": {
			"dataA": "valueA",
			"dataB": {
				"a": "b",
				"c": "DEV MODE!"
			}
		}
	},
	"key3": {
		"important": "value"
	}
}

gulp-merge-json's People

Contributors

bendavis78 avatar cameronbosnic avatar dudeofawesome avatar ivan-gammel avatar joshswan avatar lpcmedia avatar mediafreakch avatar robinj avatar sawtaytoes avatar steverep avatar thedancingcode avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

gulp-merge-json's Issues

ConcatArrays option returns duplicated items

Hey, firstly, thanks for the great lib.

I know this is not an actual issue but, I am merging these 2 json:
a:

 ...
  "key": [
    "value_1"
  ]
}

b:

...
  "key": [
    "value_1",
    "value_2"
  ]
}

What I want to achieve is:

...
  "key": [
    "value_1",
    "value_2"
  ]
}

While with this lib it seems only does concat and returns:

...
  "key": [
    "value_1",
    "value_1",
    "value_2"
  ]
}

If you think this is okay to have an extra uniqArray option, I'd happy to send a PR.

Cheers

Merging multiple files keeps only keys from last file in array

I have 3 json files

  • config.json
    -- config-dev.json
    -- config-dev-locale.json

So what I did was adding

return gulp.src(['config.json', 'config-dev.json', 'config-dev-locale.json'])
       // gulp.src would return files in incosistent order thus using merge plugin https://www.npmjs.com/package/gulp-order
      .pipe(order([
        'config.json',
        'config-dev.json',
        'config-dev-locale.json'
      ]))
      .pipe(merge({
        fileName: 'resulting-config.json',
        contactArrays: true,
        mergeArrays: true
      }))
      .pipe(gulp.dest('./app'));

So this would actually only return the last file in order in this case config-dev-locale.json

Idea was to have majority of config in config.json and override it with -dev and -dev-locale files with only several keys that differ from original files, but it only returns last file that is in order

This also happens if I comment out order part

Make merges async

For better performance and simpler code:

  1. Consider using through2 package instead of through, and use callbacks instead of emitting events. Alternatively just implement a custom transform stream.
  2. Use asynchronous functions for the transform and flush.

Yet another array merging issue: root node array

I'm seen most of the issues are related to merging arrays but yet I didn't find the solution reading them.

The issue: root node array

root node array is not merging properly like is does when it's a sub-node array

file1.json

[
  {"title": "A"},
  {"title": "B"}
]

file2.json

[
  {"title": "C"},
  {"title": "D"}
]

expected result

[
  {"title": "A"},
  {"title": "B"},
  {"title": "C"},
  {"title": "D"}
]

with the 4 possible combinations of concatArrays, + mergeArrays

concatArrays mergeArrays output
true true
true false
false true
false false
{
  "0": {
    "title": "C"
  },
  "1": {
    "title": "D"
  }
}

Similar but not exactly the same issues

  • #19, #22, #18, #11 the array there is an element not the root node

Working example of sub-node array

file1.json

{
  "array": [
    {"title": "A"},
    {"title": "B"}
  ]
}

file2.json

{
  "array": [
    {"title": "C"},
    {"title": "D"}
  ]
}

concatArrays: true + mergeArrays: true

{
  "array": [
    {
      "title": "A"
    },
    {
      "title": "B"
    },
    {
      "title": "C"
    },
    {
      "title": "D"
    }
  ]
}

Possible to keep folder structure/original file name?

Let's say I multiple folders that I need to go through, and in each of those folders I'll have two JSON files with unique names, such as: Link.config.json and Link.css.json.

Is it possible to combine Link.css.json into Link.config.json for multiple folders?

Manually works great:

gulp.task('merge', function () {
  var options = [];
  return gulp.src('./src/components/**/*.json')
    .pipe(merge({
        fileName: 'Link.config.json'
    }))
    .pipe(gulp.dest('./src/components/01-Units/Links/Link/'));
});

But I need to remove to the specific fileName to be something like :name.config.json and the dest to be the same as the src.

So it would look something like:

gulp.task('merge', function () {
  var options = [];
  return gulp.src('./src/components/**/*.json')
    .pipe(merge({
        fileName: :name + '.config.json'
    }))
    .pipe(gulp.dest('./src/components/**/'));
});

Does not merge more than 3 files

I have a folder with 20 json files (arrays of objects) and the combined json output only has data from 3 of the files. My function looks like this:
function combineJson() {
return src('data/json/*.json')
.pipe(merge({
startObj: []
}))
.pipe(dest('./public/'))
}

Thank you!

Start object is mutated

The startObj option is unexpectedly mutated inside the plugin. The value of merged is initialized as let merged = options.startObj and then is reassigned during the merges, causing startObj to be mutated. The plugin should either clone startObj when creating the merged value or otherwise treat it as immutable.

problem handling arrays

case:
combining two files (in the order file1, file2) which both contain a key "a" which contains an array in both files (the array in file2 is a subset of the one in file1)

results:
1.
with the concatArrays option set to true the result is as expected: the "a" keys are combined and the joint array consists of all the entries from both source arrays in the order file1, file2.
2.
with the concatArrays option set to false however the result is not as I would expect.
my expectation is to find just the array entries of file 2 (overwriting the array from file1).
instead the result seems to be a either just the array from file 1 or maybe a merge of the two, which in this scenario happens to be the array from file 1

Replace deprecated dependency gulp-util

Replace deprecated dependency gulp-util

gulp-util has been deprecated recently. Continuing to use this dependency may prevent the use of your library with the latest release of Gulp 4 so it is important to replace gulp-util.

The README.md lists alternatives for all the components so a simple replacement should be enough.

Your package is popular but still relying on gulp-util, it would be good to publish a fixed version to npm as soon as possible.

See:

This merge deeply?

@joshswan

This library merge JSON in a "deep" way? Example:

If I have json A and B as below and I merge A with B:

{
  "mykey": {
     "a": "1",
     "c": {
         "d": 3
      }
   }
}
{
  "mykey": {
     "b": "1",
     "c": {
        "f": 5
      }
   }
}

The result is

{
  "mykey": {
     "a": "1",
     "b": "1",
     "c": {
         "d": 3,
        "f": 5
      }
   }
}

? Thanks 👍

npm test does not work on Windows 7

Output is as follows:

C:\git\gulp-merge-json\node_modules\.bin\_mocha.CMD:1
(function (exports, require, module, __filename, __dirname) { @IF EXIST "%~dp0
                                                              ^
SyntaxError: Unexpected token ILLEGAL
    at exports.runInThisContext (vm.js:73:16)
    at Module._compile (module.js:443:25)
    at Object.Module._extensions..js (module.js:478:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:501:10)
    at runFn (C:\git\gulp-merge-json\node_modules\istanbul\lib\command\common\ru
n-with-cover.js:122:16)
    at Object.run (C:\git\gulp-merge-json\node_modules\istanbul\lib\command\comm
on\run-with-cover.js:254:9)
    at TestCommand.Command.mix.run (C:\git\gulp-merge-json\node_modules\istanbul
\lib\command\test.js:27:22)
    at runCommand (C:\git\gulp-merge-json\node_modules\istanbul\lib\cli.js:78:19
)
npm ERR! Test failed.  See above for more details.

This is caused by _mocha not translating into a javascript files in Windows

See istanbul/issues/90

filename

Hi,

I've added merge-json to my build and the JSON merging is working fine.

However the destination directory seems to be overridden by fileName

gulp.task('merge_json', function() {

  // merge the data files
  return gulp.src('contents/pages/**/*.json')
              // merge json files into a single tests file for handlebars
              .pipe(merge({
                fileName: 'tests.json',
                concatArrays: true
              }))
              .pipe(gulp.dest('./contents/pages_temp/'));
              // desired output: /contents/pages_temp/folder1/folder2/tests.json
              //  actual output: /contents/pages_temp/tests.json
});

If I disable merge-json it works as expected:

gulp.task('merge_json', function() {

  // merge the data files
  return gulp.src('contents/pages/**/*.json')
              // merge json files into a single tests file for handlebars
              //.pipe(merge({
              //  fileName: 'tests.json',
              //  concatArrays: true
              //}))
              .pipe(gulp.dest('./contents/pages_temp/'));
              // desired output: /contents/pages_temp/folder1/folder2/test1.json
              //  actual output: /contents/pages_temp/folder1/folder2/test1.json
});

Is this a bug or is there an option I can toggle on to make this work?

Thanks,
Dan

[enhancement] JSON5 Support

Added support for JSON5 would allow lots of extra goodies for the files being merged/combined.

e.g. We like to use comments, TODO's, and FIXME's in our json (json5) files and use gulp-todo as a reporter to a TODO.md ).

I believe this could be beneficial to Hugo and it's users.

Still problem with array merging

// Sample one
{ "tags": [{"name": "store"} ] }
// Sample two
{ "tags": [{"name": "store2"} ] }
// Sample three
{ "tags": [{"name": "store3"} ] }

// Exacted result
{ "tags": [{"name": "store"}, {"name": "store2"}, {"name": "store3"}] }

//Got this
{ "tags": [{"name": "store3"} ] }

merge json array

If I try to merge 2 json arrays. Let's say I have 2 arrays with 2 objects in each.

[ { ... }, { ... } ]
[ { ... }, { ... } ]

I get something like that :

{
    "0": { ... },
    "1": { ... }
}

It outputs an object and not an array and this object is using the array index as property so I only get the last 2 object of my arrays.

mergeArrays: false

Hey. I'm using gulp-merge-json and it's great, so thanks but....

I'm using mergeArrays:false and my outputted Json contains key values that have been deleted from the source Json.

Am i correct in thinking mergeArrays:false should completely delete what was in the combined file then merge all Json, then re-create the Json object?

This is my gulp task:

gulp.task('combine-json', () => {
gulp.src('./src/js/data/*.json')
.pipe(merge({
fileName: '_combined.json',
mergeArrays: false
}))
.pipe(gulp.dest('./src/js/json'))
})

If i delete a key value from one of the Json files in ./src/js/data/ the key value still appears in my _combined.json

Am i doing something wrong or have i misunderstood mergeArrays: false?

Thanks in advance!

Cant merge an json Array

  • Actual behaviour
    {"modules": [{"id": 2}]}
  • Expected behaviour
    {"modules": [{"id": 1}, {"id": 2}]}
  • Reproduce
    file1.json
    {"modules": [{"id": 1}]}
    file2.json
    {"modules": [{"id": 2}]}
var
    gulp = require('gulp'),
    jsonMinify = require('gulp-jsonminify'),
    mergeJSON = require('gulp-merge-json'),
    rename = require('gulp-rename');

return gulp.src('config/**/*.json')
    .pipe(merge('manni.json'))
    .pipe(gulp.dest(this.opts.DIR_BUILD_DIST));

/*
        .pipe(jsonMinify())
        .pipe(rename('manni.min.json'))
        .pipe(gulp.dest(this.opts.DIR_BUILD_DIST))
        .pipe(gulp.dest(this.opts.DIR_BUILD_APP_WWW));
*/

Vinyl File type reference is broken

The Vinyl File type import is broken:

import { File } from 'vinyl';

TypeScript compiler throws the following error: node_modules/gulp-merge-json/index.d.ts(2,12): error TS2305: Module '.../node_modules/@types/vinyl/index"' has no exported member 'File'

Changing the import to import * as File from 'vinyl' fixes the issue.

Tested with:

  • gulp-merge-json v1.2.1
  • @types/vinyl v2.0.2
  • typescript v2.7.2

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.