Git Product home page Git Product logo

json.minify's Introduction

JSON minify

Minify blocks of JSON-like content into valid JSON by removing all white-space and C/C++ style comments.

JSON parsers (like JavaScript's JSON.parse() parser) generally don't consider JSON with comments to be valid and parseable. So, the intended usage is of this project is to minify development-friendly JSON (i.e with comments) to valid JSON before parsing, such as:

JSON.parse(JSON.minify(str))

Now you can maintain development-friendly JSON documents, where your source is formatted & commented, but minify them before parsing or before transmitting them over-the-wire.

As transmitting bloated (ie, with comments/white-space) JSON would be wasteful and silly, this JSON minify can also be used for server-side processing environments where you can strip comments/white-space from JSON before parsing a JSON document or before transmitting such over-the-wire from server to browser.

Though comments are not officially part of the JSON standard, this post from Douglas Crockford back in late 2005 helps explain the motivation behind this project.

A JSON encoder MUST NOT output comments. A JSON decoder MAY accept and ignore comments.

Basically, comments are not in the JSON generation standard, but that doesn't mean that a parser can't be taught to ignore them. Which is exactly what JSON minify is for.

The first implementation of JSON minify was in JavaScript (as JSON.minify), but the intent is to port the implementation to as many other environments and languages as possible/practical.

Using JSON Minify

Currently, JSON minify has been ported to multiple languages including PHP, Python, Objective C. Each of these ports live on a separate branch in the Github repository. Language specific instructions can be found in their respective branches.

json.minify's People

Contributors

bernhardg avatar felipou avatar getify avatar gstorer avatar pradyunsg avatar semanio avatar tib avatar tombyrer avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

json.minify's Issues

This minifier fails when JSON doesn't end with a newline

I was looking for JSON minifier libraries and this came up in my search. I tried it but surprisingly it wasn't working for some trivial JSON such as this:

{"a": 5}

The result it produces is still with a whitespace:

{"a": 5}

I started messing around and added a newline at the end of my trivial example and it started working!

{"a": 5}\n <- newline here

The result it produces now is correct:

{"a":5}

To make sure I wasn't missing something obvious, I edited tests.js and added these two test cases at the beginning of the file:

{
    source: "{\"a\": 5}\n",
    assert: "{\"a\":5}"
},
{
    source: "{\"a\": 5}",
    assert: "{\"a\":5}"
},

When I run the tests, the first example succeeds but the second one fails:

$ cat minify.json.js tests.js | node
Test 1 passed

[stdin]:125
                throw ("Test (" + (idx + 1) + ") failed:\n  " + res);
                                                          ^
Test (2) failed:
  {"a": 5}

I hope this was bug report was useful.

P.Krumins

Port to LabVIEW

Hey,
I want to port this code to LabVIEW.
I have already a working code and passing all tests in my machine.
Following the instructions I believe I should open an issue before doing the fork, right?
Waiting further instructions and possible discussion.
Regards,

Also stip trailing commas

In an effort to make JSON parsing a little more flexible, strip any trailing commas in objects or arrays.

Algorithm is accidentally quadratic

Hello there!

So today I tried to push a 2.5 megabyte file to my team’s Git server and it said “the pre-receive hook timed out after 120 seconds”.

I asked the CI priests (don’t laugh, it’s their official job description) what the 🔥 it could be doing, and they said “the only thing we do on JSON files is this” and supplied a small script involving the Python version of json_minify.

I ran it on my file. It finished in 76 minutes 12 seconds.

So I looked at the algorithm and found the cause.

  • On the top level, it has a loop going through “interesting” places in the input, where “interesting” is defined as “a potential string delimiter, comment delimiter, or a newline”. It also maintains several flags reflecting the syntax at current position.
  • If the current syntax is “inside string” and it finds a double quote, it might be a closing delimiter. So it checks if there is an even number (incl. zero) of backslashes preceding the double quote. If so, it is indeed a closing string delimiter.
  • The way it checks for an even number of backslashes is… curious. It searches for the regular expression \\*$ in the whole substring from position 0 to current. This succeeds always before the end, and matches the longest span of backslashes, but it tries every possible starting position from 0 to the current position, unless the regular expression engine is smart enough to notice the anchor at the end and match backwards. (Spoiler warning: Most of them are not.)

As a proof of concept, I made this change:

         if val == '"' and not (in_multi or in_single):
-            escaped = end_slashes_re.search(string, 0, match.start())
+            pos = match.start()
+            while pos > 0 and string[pos - 1] == '\\':
+                pos -= 1

             # start of string or unescaped quote character to end string
-            if not in_string or (escaped is None or len(escaped.group()) % 2 == 0):  # noqa
+            if not in_string or ((match.start() - pos) % 2 == 0):  # noqa
                 in_string = not in_string
             index -= 1  # include " character in next catch

and it passed all existing tests and processed my 2.5M file in 0.697 seconds. It is definitely a lot less pythonic than the original way, but it sure does the job.

You might want to review all ports where this check is done via the same regexp, and verify which of them exhibit quadratic complexity.

There is nothing peculiar about my JSON file; just generate one that has a moderately large number of double quote characters.

Set repo up on CI server

Is there interest to have continuous integration set up for this repo? Different languages could have their own configurations and badges could be added to the README files. For maven in particular it would be useful to have CI set up to enable CD on PR merges.

Create gpg keys for java code signing

For the java port, code deployed to maven central (see #36) is required to be signed. To this end, a gpg key should be created for the organization. This key should be controlled by the maintainers of the organization, and can be securely added to a CI server for automated deployment.

Whitespaces and tabs after last quote and newline are not stripped

This is an example that fails (neither the tab not the last space are stripped):

source: "{"foo": "bar"\t} ",
assert: "{"foo":"bar"}"

All tests in the suite end with \n, thus avoiding the problem, but I suppose that json without newlines are also intended to be supported?

I have tested javascript only, I don't know how other ports are behaving in this situation.

Example usage fails on node

node --version
v4.1.2

JSON.minify = JSON.minify || require("node-json-minify");
var myjson = JSON.minify( '{ /* comment */ "foo": 42 \n }' ); // {"foo":42}
^
TypeError: JSON.minify is not a function

Node port: export as UMD to support use in code

It would be great for this library to use UMD, allowing support for Node, browser and script injection.
For example, I'd like to use it in code like this:

var jsonMinify = require("node-json-minify");

// someJSONstring is a string in JSON format
let realJSON = jsonMinify(someJSONstring);

Invalid package.json

node.js/node_modules/json-minify/package.json has syntax errors.

  • "description" value must be single line.
  • "contributors" value array must be closed with ']'.
  • "version" value must have 3 numbers, like "0.1.0"

Retain newlines in comments

When running with strip-space=False it would be nice if newlines were retained.
Retaining newlines helps JSON errors being reported with the correct line numbers.
eg: "Missing , on line 1" should be "Missing , on line 2" if the first line contains a comment.

Here's inpiration for a fix:

@@ -44,9 +44,14 @@
             in_singleline_comment = True
         elif (match.group() == '\n' or match.group() == '\r') and not in_string and not in_multiline_comment and in_singleline_comment:
             in_singleline_comment = False
+            if not strip_space:
+                new_str.append("\n")
         elif not in_multiline_comment and not in_singleline_comment and (
              match.group() not in ['\n','\r',' ','\t'] or not strip_space):
                 new_str.append(match.group())
+        elif (in_multiline_comment or in_singleline_comment) and \
+            match.group() in ['\n','\r'] and not strip_space:
+                new_str.append(match.group())
     
     new_str.append(json[from_index:])
     return ''.join(new_str)

'module' object is not callable

>>> json_minify('{"test": "test"}')
Traceback (most recent call last):
  File "<pyshell#186>", line 1, in <module>
    json_minify('{"test": "test"}')
TypeError: 'module' object is not callable

Why is my Python code not working?

Incorrect strip last single line comment without line feed

Sample my.json:

["zzz"]
// last comment (no \n after it

Test script:

var fs= require("fs");
JSON.minify = require("node-json-minify");

var srcjson = fs.readFileSync("my.json").toString();
var minjson = JSON.minify(srcjson);
console.log(minjson);

Output:

["zzz"] last comment (no \n after it

Version Badges

Should we add version badges to the repository?

If yes, where?

  • In the master's README.md -- Most visible
  • In language-specific README.md -- Easiest from organization point of view
  • Both -- More work but Better than individual ones

Note: Should use http://shields.io/

Evolution of the PHP branch

As said in t1st3/php-json-minify#13, I made a duplicate of the PHP branch of this module some time ago. The project (t1st3/php-json-minify) has kind of evolved from then, bringing the following "goodies" to the original code:

  1. being published on Packagist
  2. being a class rather than a function (allowing namespaces, following PSR-4 spec.
  3. having tests run with PHPUnit
  4. having auto-generated API docs with PHPDocumentor

Yet, the original function has not changed at all since the beginning, and is still kind of a copy/paste of @getify 's original work.

This issue is merely a proposal to integrate all that has been done on my fork to the Kyle's project. This has been suggested by @emahuni and I personnaly think all these changes belong to this project rather than to a fork. So this issue should be seen as an open discussion rather than an absolute need for change :)

I'm willing to submit some PRs for these, but before that, please tell me if those PRs would be worth a shot!

@getify , I love your work btw...

Create credentials for OSSRH

As a developer, I want to be able to access the minify java library from maven central so that I can include it as a dependency in my projects.

To do this, an account needs to be made with OSSRH to create a maven organization (which will probably be called getify?).

This also will require a key for code signing of jars when they are built. (#37)

EDIT: a little pointer and some info on how this is done here. Basically just open a JIRA ticket specifying a domain you own as the organization name (com.getify probably).

Put different language implementations on different branches (Will break all other PRs)

I had a small suggestion, that you put the implementations in separate branches, one for each language. This way, language-specific packaging can be easily done, in a single repository.

Just what I think.... 😄

PS: I will happily make a PR for solving #21 if this is done.


  • Create a base branch from where someone can start work on a new language port. #25
  • Make language branches off base and populate them.
    • node
    • javascript
    • php
    • python #26
    • java #27
    • objective-c #28
  • Clean up master and update documentation.

Wow, great thing... but... how do I use this in practice ?

Sorry to sound stupid, but I need exactly your utility because of this

http://stackoverflow.com/questions/19342628/chromes-webstore-server-reject-an-extension-with-error-the-manifest-must-def

So I would had hope a very simple webpage where I can quickly remove the problem to see if the problem come frome here.

There is none ok. I'm not a javascript developer, but I know that this node.js and npm things are useful, so I install them on my macbook pro

but since there are no docs, I made an error
It's not https://npmjs.org/package/json-minify (doesn't allow comments)
It's https://npmjs.org/package/jsonminify (very different !)

ok, so I install the second one

but everything is a library there.
No command line utiliy that I can use to convert my manifest-with-comments.json file that a human can understand to manifest.json file that a computer can understand

Sure I should know python or node or... but I don't know enough to do that... and I'm by far not the only one that would benefit from your tool

OK, so for this time I will just use vim to remove comments, but I hope I can Read The Fine Documentation next time :-D

minify.json.php is incredibly slow

Just ran into an issue where using minify.json.php in my application results in ~4+ seconds of processing time on my localhost server before the client renders the page. Removing minify.json.php from my application eliminates the issue and the client renders a page in ~380ms.

I will be looking further into this issue, just wanted to document it here and see if others have seen this problem?

PyPI package?

Thanks for creating this utility. I was wondering if there are plans to make a python package to PyPI, so that clients can install via tools like pip. That would be pretty great.

why is slow in javascript project

The code is:

const fs = require('fs');
const path = require('path');
const minify = require('node-json-minify');

//读取广州地铁线路数据
let subwayJson = fs.readFileSync(path.join(__dirname, '../../public/static/ht/displays/subway.json'), 'utf-8');
let subwayData = minify(subwayJson);
fs.writeFileSync(path.join(__dirname, './subway1.json'), subwayData);

It take a long time to output the file

Port to Dart

I've ported the code to Dart over here. All the tests pass in Dart (which makes it usable in both server-side Dart and Flutter), and it cleanly transpiles to JS with the dart2js utility (though why anyone would want to do that instead of just using the JS library itself is beyond me).

JSON on a single line without line carriage is minify as an empty String

If you consider the following script:

var srcjson = "[{'a':12}]";
var minjson = JSON.minify(srcjson);

console.log(minjson);

You'll get an empty string logged in the console. In test.minify.html, it can be reproduced by making sure there is no new-line carriage before and after the json

Establish requirements for minifier ports

Wanted to start a conversation about establishing requirements for the JSON.minify core functionality and provide this outline in the README file.

As a minifier, I expect it will:

  • remove inline // ... and multiline /* ... */ comments

Though some people may come to expect it will also:

  • strip trailing commas
    • { "foo": "bar", "foo2": "bar2", }
  • remove multiple commas
    • { "foo": "bar", , "foo2": "bar2" }
  • minifiy inline JSON data
    • /* all my json data is inline >:D */{"this": "is just aweful..." // inline comment hell "please": "don't make me parse this :'("}

Since the goal is to parse development quality JSON and not necessarily validate it, I think the following expectations should be established to help define test cases for current and future ports:

  1. JSON should be relatively well formatted (Human readable is optional)
    • Would validate normally if comments were not embedded
    • No hard requirements regarding comma-first/comma-last notation
  2. JSON should be multi-line
  3. Inline comments should not precede JSON markup

Update Python readme

I think we could clarify in the readme, that the name to use to import the module should be json_minify.
Leading to:
import json_minify

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.