aheckmann / gridform Goto Github PK

Stream formidable uploads into MongoDB GridFS

Home Page: http://aheckmann.github.com/gridform/

License: MIT License

JavaScript 100.00%

gridform's Introduction

#gridform

Formidable streams uploads to the file system by default. If you're using GridFS to store files you'll then need to turn around and copy them off of the file system. Using gridform removes this burden.

Example:

var mongo = require('mongodb')
var gridform = require('gridform');

// assuming you've already created a db instance and opened it
gridform.db = db;
gridform.mongo = mongo;

// in your http server
var app = http.Server(function (req, res) {

  // create a gridform
  var form = gridform();

  // returns a custom IncomingForm
  assert(form instanceof formidable.IncomingForm);

  // optionally store per-file metadata
  form.on('fileBegin', function (name, file) {
    file.metadata = 'so meta'
  })

  // parse normally
  form.parse(req, function (err, fields, files) {

    // use files and fields as you do today
    var file = files.upload;

    file.name // the uploaded file name
    file.type // file type per [mime](https://github.com/bentomas/node-mime)
    file.size // uploaded file size (file length in GridFS) named "size" for compatibility
    file.path // same as file.name. included for compatibility
    file.lastModified // included for compatibility

    // files contain additional gridfs info
    file.root // the root of the files collection used in MongoDB ('fs' here means the full collection in mongo is named 'fs.files')
    file.id   // the ObjectId for this file

  });
});

install

npm install gridform

exports

The module exports a function which takes an options object.

var gridform = require('gridform');
var options = { db: db, mongo: mongo, filename: fn };
var form = gridform(options);

Available options:

db: an open node-mongodb-native db instance
mongo: the node-mongodb-native driver you are using
filename: function

The optional filename function is passed the file.name before streaming to MongoDB providing an opportunity to return a customized filename with a prefix etc.

db and mongo are required unless you've specified them on gridform itself.

var gridform = require('gridform');
gridform.db = db;
gridform.mongo = mongo;
var form = gridform(); // all good

The gridform function returns an instance of formidable.IncomingForm so you can process uploads without changing any code.

gridfs-stream

This module utilizes the gridfs-stream module which is exposed as require('gridform').gridfsStream.

tests

Run the tests with make test.

LICENCE

gridform's People

Contributors

Stargazers

Watchers

Forkers

james-voth paulwoodiii echorohit tonsoffun lewisjellis adammendoza 2bees-rd 4ker awk34 robbertwolfs mformarlon mchapman bryant1410 zevero lerit andreguerra123

gridform's Issues

Node 6.x - please update formidable to v1.1.1

I got this error

Error: Buffer.write(string, encoding, offset[, length]) is no longer supported

which has been solved for [email protected] lately. See node-formidable/formidable#364

Gridform is still using [email protected] please update to v1.1.1 - otherwise it does not work on Node 6+

Thank you

How get file

When file is saved, how can i get the file?

File upload problem using gridfs-stream 0.5.0

My code for uploading files was working with gridform until updating Mongoose to v4.x.x. When a file upload is attempted the following error occurs:

/Users/kenny/Dropbox/Dev/vqfolio/node_modules/gridform/node_modules/gridfs-stream/lib/writestream.js:44
this.id = new grid.mongo.BSONPure.ObjectID;
^
TypeError: Cannot read property 'ObjectID' of undefined
at Stream.GridWriteStream (/Users/kenny/Dropbox/Dev/vqfolio/node_modules/gridform/node_modules/gridfs-stream/lib/writestream.js:44:38)

It looks like similar issues have been seen for others using gridfs-stream: aheckmann/gridfs-stream#74

Does the version of gridfs-stream and mongodb that gridform currently use need to be updated to fix this issue? (current versions used by gridform 0.1.7 are gridfs-stream 0.5.0 and mongodb 1.4.3)

File id is not available on 'fileBegin', 'end', 'file', or while parsing

I have successfully setup a simple express app utilizing gridform for file uploads to GridFS. The problem I have is I would like to use the file object id for logging/enqueueing.

router.post('/files', function(req, res) {
      // Assume you have gridform setup
      var form = gridform();

      form.on('fileBegin', function (name, file) {
        console.log('on fileBegin\n');
        console.log(file);
        console.log('\n');
      });

      form.parse(req, function (err, fields, files) {
        console.log('while parsing\n');
        console.log(files.file);
        console.log('\n');
      });

      form.on('file', function (name, file) {
        console.log('on file\n');
        console.log(file);
        console.log('\n');
      });

      form.on('end', function (name, file) {
        console.log('on end:\n');
        console.log(file);
        console.log('\n');
      });
});

Gives me this

on fileBegin:

{ domain: null,
  _events: {},
  _maxListeners: 10,
  name: 'IMG_4073.jpg',
  path: 'IMG_4073.jpg',
  type: 'image/jpeg',
  size: 0,
  root: null,
  id: null,
  lastModified: Sat May 24 2014 16:28:33 GMT-0700 (PDT) }

on file:

{ domain: null,
  _events: {},
  _maxListeners: 10,
  name: 'IMG_4073.jpg',
  path: 'IMG_4073.jpg',
  type: 'image/jpeg',
  size: undefined,
  root: undefined,
  id: undefined,
  lastModified: Sat May 24 2014 16:28:33 GMT-0700 (PDT) }


while parsing:

{ domain: null,
  _events: {},
  _maxListeners: 10,
  name: 'IMG_4073.jpg',
  path: 'IMG_4073.jpg',
  type: 'image/jpeg',
  size: undefined,
  root: undefined,
  id: undefined,
  lastModified: Sat May 24 2014 16:28:33 GMT-0700 (PDT) }


on end:

undefined

I realize on end doesn't return the file, but I was desperate. Is there something I am missing?

How to append PNG files from GridForm using GraphicsMagic

I've got lots of PNG files stored in mongo using GridForm and am trying to read several and use GM's append() command to combine them before sending them out. The append() command seems to be suited toward physical files on the disk not streams read from GridForm.

I noticed @aheckmann contributes to both repos, so I'm hoping you'll be able to give me a hint. Here are the guts of the code:

var Async = require('async');
var GridForm = require('gridform');
var Mongoose = require ('mongoose');
var GM = require('gm').subClass({ imageMagick: true });

....

var fileNames = ['file1.png', 'file2.png', 'file3.png'];
var gfs = GridForm.gridfsStream(Mongoose.connection.db, Mongoose.mongo);

gfs.files.find({ filename: { $in: fileNames } }).toArray(function (err, files) {

    var combinedFile = null;
    Async.eachSeries(
        files,
        function(file, callback) {
            var readstream = gfs.createReadStream(file._id);
            var newFile = null;
            readstream.pipe(newFile);
            if(combinedFile == null) {
                combinedFile = newFile;
            } else {
                GM(combinedFile).append(newFile);
            }
            console.log('combined file ' + file._id);
            callback(null, null);
        },
        function(err, results) {
            console.log("Sent all files");
        }
    );
});

Its possible save in database from external file?

I want put a url and read sourcecode and save in gridfs. I use this function to read sourceCodes:

function getSourceCode(url, callback){

    // Obtenemos el código fuente
    http.get(url, function(res) {

    // Mientras va recibiendo añade el dato a una variable
    var body = '';
    res.on('data', function(chunk) {
        body += chunk;
    });

    // Cuando esté obtenido
    res.on('end', function() {

        // Devolvemos al callback el código fuente
        callback(null, body);

    });

    }).on('error', function(err) {

    // En caso de error
    myUtils.handError(err);

    // Devolvemos al callback
    callback({
        code: "GET SOURCE CODE",
        message: "Was imposible get surce code"
    });

    });

}

I can connect this as stream to gridform?

GridFS + Mongoose

Its possible use with "Mongoose", instead of "mongo" module?
Thanks.

Document doesn't exist when parse callback fires

In the form.parse callback, the new file's document is only available after a delay. I don't think this the intended behavior. Am I misusing gridform?

// Requires omitted. I don't think they're the
// problem as every other operation using
// node-mongo-native behaves as expected

app.post('/', function(req, res) {
  var form = gridform({
    db: db,
    mongo: mongo
  });

  form.parse(req, function(err, fields, files) {
    var file = files.upload;

    // version 1: this works
    setTimeout(function() {
      db.collection('fs.files', function(err, coll) {
        coll.findOne({_id: file.id}, function(err, doc) {
          // doc exists
        });
      });
    }, 30);

    // version 2: this doesn't work
    db.collection('fs.files', function(err, coll) {
      coll.findOne({_id: file.id}, function(err, doc) {
        // doc is null
      });
    });

  });
});

No answer when posting only text-fields (no files)

Hi,
I don't know if this is a formidable issue or a gridform issue, or maybe even more likely an issue with my code, but when posting a form without any files (only text fields), I get no answer. When only a file is provided everything works fine, the same when both files and fields are provided in the post. Do you have any idea what this problem might be?

var form = gridform(); //enters into the incoming form constructor (see longer down)
    assert(form instanceof formidable.IncomingForm); 
    form.parse(req, function (err, fields, files) { // this is not fired at all.....

I have tracked it down to the IncomingForm.handlePart method here:

IncomingForm.prototype.handlePart = function(part) {
...
...
    part.on('end', function() {
        console.log(value);//this prints
      self.emit('field', part.name, value);
    });
    return;

Enable passing existing Grid instance to gridform

I'm using gridform, and also using gridfs-stream itself. I have an existing Grid instance which I would like to be able to pass to gridform, but instead I have to pass the same db and mongo that I used to construct my Grid instance, and then gridform constructs its own Grid instance.

I've poked around a bit in both gridform and gridfs-stream and can't come up with any reason for gridform to need its own Grid instance. I'm happy to send a PR for this, but just want to have some discussion before doing so. I'm imagining just adding a grid field to options which, when provided, is used in place of db and mongo to populate __gridstream. Please let me know your thoughts.

Example without using the file system

An example of using gridform/gridfs-stream to save (from form upload) directly to mongodb and then read from mongo and send on a response would be really great. The examples I've seen on gridfs-stream all seem to involve a local file system, which I want to avoid as cloud web servers don't have that available.

I am using Mongoose.js, also.

Whatever you can provide would be much appreciated.

Thank you.

form.parse is not considering Ajax based upload.

Refer the following question in stack overflow.

http://stackoverflow.com/questions/13120385/how-to-parse-ajax-file-upload-using-gridform-in-nodejs-express

I am able to get data from ajax request. where in the case of form.parse the control is not going inside to the form.parse.