mscdex / connect-busboy Goto Github PK
View Code? Open in Web Editor NEWConnect middleware for busboy
License: MIT License
Connect middleware for busboy
License: MIT License
Tried using the example in the Readme, had to add some boilerplate, install express etc, but I'm seeing errors when posting:
[cmorgan@homepc busboy_test]$ node testapp.js
TypeError: Cannot call method 'on' of undefined
at app.use.busboy.immediate (/home/cmorgan/busboy_test/testapp.js:11:14)
at Layer.handle [as handle_request] (/home/cmorgan/busboy_test/node_modules/express/lib/router/layer.js:76:5)
at trim_prefix (/home/cmorgan/busboy_test/node_modules/express/lib/router/index.js:263:13)
at /home/cmorgan/busboy_test/node_modules/express/lib/router/index.js:230:9
at Function.proto.process_params (/home/cmorgan/busboy_test/node_modules/express/lib/router/index.js:305:12)
at /home/cmorgan/busboy_test/node_modules/express/lib/router/index.js:221:12
at Function.match_layer (/home/cmorgan/busboy_test/node_modules/express/lib/router/index.js:288:3)
at next (/home/cmorgan/busboy_test/node_modules/express/lib/router/index.js:182:10)
at /home/cmorgan/busboy_test/node_modules/connect-busboy/index.js:14:14
at Layer.handle [as handle_request] (/home/cmorgan/busboy_test/node_modules/express/lib/router/layer.js:76:5)
^C[cmorgan@homepc busboy_test]$
It looks like req doesn't have a .busboy element. Tried with expressjs 3.x and 4.x.
Is connect-busboy compatible with busboy 0.2.6? Been hacking for almost an hour, can't get it working.
Maybe update the examples to make the usage more obvious?
How to use busboy special event 'filesLimit' in nodejs?
Hello,
after updating connect-busboy from 0.0.2 to 1.0.0, I am having an issue. The problem is that this code sometimes works but sometimes does not. seems like something gets broken while uploading a file with API and just half of the file reaches the server and the close event does not trigger. if you call the API with the same file let's say 5-6 times, this error will occur just once.
Hi. I've got the following problem: http://stackoverflow.com/questions/33712020/my-application-using-busboy-freezes-when-no-files-uploaded
When uploading three files everything is ok, but when one or more files are missing, the application is freezing and "finish" section doesn't even start.
I was following this article to setup a nodejs server on my local machine (which has 16 gb memory and about 170gb free disk-space) and uploaded a 20 gb file, for the first couple of times the file got uploaded successfully, but after a while i started getting EPIPE error:
error FetchError: request to http://localhost:3200/upload failed, reason: write EPIPE
at ClientRequest.<anonymous> (/Volumes/FreeAgent GoFlex Drive/Test/multer-project/node_modules/node-fetch/lib/index.js:1455:11)
at ClientRequest.emit (events.js:327:22)
at Socket.socketErrorListener (_http_client.js:467:9)
at Socket.emit (events.js:315:20)
at emitErrorNT (internal/streams/destroy.js:100:8)
at emitErrorCloseNT (internal/streams/destroy.js:68:3)
at processTicksAndRejections (internal/process/task_queues.js:84:21) {
type: 'system',
errno: 'EPIPE',
code: 'EPIPE'
}
When i checked, the file got uploaded partially and was about 28mb in size. I tried uploading the file from both Postman, browser and a nodejs script, but got the same EPIPE error message. I am not sure why is this happening, googling the error message didn't help. I am not sure how to overcome this. Following is my server and client code.
// server.js
const express = require("express"); // Express Web Server
const busboy = require("connect-busboy"); // Middleware to handle the file upload https://github.com/mscdex/connect-busboy
const path = require("path"); // Used for manipulation with path
const fs = require("fs-extra");
const app = express(); // Initialize the express web server
app.use(
busboy({
highWaterMark: 2 * 1024 * 1024 // Set 2MiB buffer
})
); // Insert the busboy middle-ware
const uploadPath = path.join(__dirname, "uploads/"); // Register the upload path
fs.ensureDir(uploadPath); // Make sure that he upload path exits
/**
* Create route /upload which handles the post request
*/
app.route("/upload").post((req, res, next) => {
req.pipe(req.busboy); // Pipe it trough busboy
req.busboy.on("file", (fieldname, file, filename) => {
console.log(`Upload of '${filename}' started`);
// Create a write stream of the new file
const fstream = fs.createWriteStream(path.join(uploadPath, filename));
// Pipe it trough
file.pipe(fstream);
// On finish of the upload
fstream.on("close", () => {
console.log(`Upload of '${filename}' finished`);
res.send("ok");
});
});
});
/**
* Serve the basic index.html with upload form
*/
app.route("/").get((req, res) => {
res.writeHead(200, { "Content-Type": "text/html" });
res.write(
'<form action="upload" method="post" enctype="multipart/form-data">'
);
res.write('<input type="file" name="fileToUpload"><br>');
res.write('<input type="submit">');
res.write("</form>");
return res.end();
});
const server = app.listen(3200, function() {
console.log(`Listening on port ${server.address().port}`);
});
and my client code is:
// client.js
const fs = require("fs");
const FormData = require("form-data");
const fetch = require("node-fetch");
var formdata = new FormData();
formdata.append(
"file",
fs.createReadStream("/Users/phantom007/My Documents/35gb.myfile")
);
var requestOptions = {
method: "POST",
body: formdata,
redirect: "follow"
};
fetch("http://localhost:3200/upload", requestOptions)
.then(response => response.text())
.then(result => console.log(result))
.catch(error => console.log("error", error));
How do I know if the fileSize limit is hit?
For eg:
This works!
const registerBusboyFields = (req, res, next) => {
req.uploadBody = {};
req.busboy.on("field", function (name, value) {
req.uploadBody[name] = value;
console.log(name, value);
});
// Finished all busboy events
req.busboy.on("finish", () => {
// The finish event is triggered...
console.log(req.uploadBody);
next();
});
req.pipe(req.busboy);
};
This does not work :(
const registerBusboyFields = (req, res, next) => {
req.uploadBody = {};
req.busboy.on("file", function (name, file, info) {
req.uploadBody[name] = info;
console.log(info);
});
req.busboy.on("field", function (name, value) {
req.uploadBody[name] = value;
console.log(name, value);
});
// Finished all busboy events
req.busboy.on("finish", () => {
// The finish event is NOT triggered...
console.log(req.uploadBody);
next();
});
req.pipe(req.busboy);
};
PS: When i say "does not work", i mean, i just see the console.log happening on "file" event, but console.log does neither happen on "field" event not in "finish" event.
Am I doing something wrong?
Hello,
I have this piece of code:
router.route("/post")
.get(function(req, res) {
...
})
.post(authReq, function(req, res) {
...
// Get uploaded file
var fstream
req.pipe(req.busboy)
req.busboy.on('file', function (fieldname, file, filename, encoding, mimetype) {
...
// Start writing
fstream = fs.createWriteStream(tmpFileURL)
file.pipe(fstream)
fstream.on('close', function() {
...
})
})
})
})
This use to work perfectly. However, I don't know quite what happened, but I picked up my computer today and found this error occurring every time I attempt to post:
TypeError: Cannot call method 'on' of undefined
at IncomingMessage.Readable.pipe (_stream_readable.js:494:8)
at /Users/nathan/Library/Mobile Documents/com~apple~CloudDocs/Dev/LaughItUp/Code/Server/apiRoutes.js:139:8
at Layer.handle [as handle_request] (/Users/nathan/Library/Mobile Documents/com~apple~CloudDocs/Dev/LaughItUp/Code/Server/node_modules/express/lib/router/layer.js:82:5)
at next (/Users/nathan/Library/Mobile Documents/com~apple~CloudDocs/Dev/LaughItUp/Code/Server/node_modules/express/lib/router/route.js:100:13)
at authReq (/Users/nathan/Library/Mobile Documents/com~apple~CloudDocs/Dev/LaughItUp/Code/Server/apiRoutes.js:17:3)
at Layer.handle [as handle_request] (/Users/nathan/Library/Mobile Documents/com~apple~CloudDocs/Dev/LaughItUp/Code/Server/node_modules/express/lib/router/layer.js:82:5)
at next (/Users/nathan/Library/Mobile Documents/com~apple~CloudDocs/Dev/LaughItUp/Code/Server/node_modules/express/lib/router/route.js:100:13)
at next (/Users/nathan/Library/Mobile Documents/com~apple~CloudDocs/Dev/LaughItUp/Code/Server/node_modules/express/lib/router/route.js:94:14)
at Route.dispatch (/Users/nathan/Library/Mobile Documents/com~apple~CloudDocs/Dev/LaughItUp/Code/Server/node_modules/express/lib/router/route.js:81:3)
at Layer.handle [as handle_request] (/Users/nathan/Library/Mobile Documents/com~apple~CloudDocs/Dev/LaughItUp/Code/Server/node_modules/express/lib/router/layer.js:82:5)
_stream_readable.js:501
dest.end();
^
TypeError: Cannot call method 'end' of undefined
at IncomingMessage.onend (_stream_readable.js:501:10)
at IncomingMessage.g (events.js:180:16)
at IncomingMessage.emit (events.js:92:17)
at _stream_readable.js:943:16
at process._tickCallback (node.js:419:13)
In my index.js file, I have this code to use Busboy:
app.use(busboy())
Then this code to include my route:
app.use("/api", require("./apiRoutes.js")())
Thanks in advance!
I think this stackoverflow answer would be a great addition to the TROUBLESHOOTING section: http://stackoverflow.com/questions/24588408/busboy-not-firing-finish-event
node_modules/busboy/lib/utils.js:465
} catch {}
^
SyntaxError: Unexpected token {
from past week I am getting this error within the library.
any one else facing it?
Node V: 10.16.0
NPM: 6.9.0
lib version: Type definitions for connect-busboy 0.0
In package.json it saying 0.0.2
also tried: 1.0.0
Does this work with 3.0?
Using the raw busboy example, it works, but plugging raw busby into express and passing express headers doesn't get the files / fields. Also, using connect-busboy has the same effect.
Hi,
My understanding is that. finish event will trigger after file and field event completes?
Is their any other event that will fire after field and file event completes. I need to send the response to the client after all events completes.
This is the very critical one. Please let me know
Thanks,
Vivek
When the upload file is undefined,event handler is registered but event 'file' never happens. It will take along time for process to stop handling this request. When I keep on doing such thing for more than five times,the node process will halt for several minutes, it will come to normal after a few minutes.
Then how can I handle this situation in NodeJS process? Don't tell me never to upload undefined, when file upload failed, I think such situation will also take place.
The dependency is marked as *
and the new version released yesterday has breaking changes.
You, answer the question
http://stackoverflow.com/questions/26084373/error-while-process-req-with-charset-windows-1251
Can, you write a little example, for urlencoded, supports windows-1251?
I am directly streaming the FileStream to s3. In case of docx and xlx it is working perfectly. It upload pdf with size to s3 but the file doesn't open on download.
req.busboy.on(
"file",
async function (fieldname, file, filename, encoding, mimetype) {
file.on("limit", function () {
next(
new ErrorResponse(
{ contract: "ERR_FILE_CONTRACT_SIZE" },
"Validation Failed"
)
);
return;
});
if(file){
const stream = new PassThrough();
file.pipe(stream);
// Create file object
req.files[fieldname] = {
stream:stream,
name: filename,
encoding: encoding,
mimetype: mimetype,
};
}
}
);
I am using angular as a front end and Firebase cloud function (Express.js) REST API.
Busboy: 1.6.0 (also tried 1.5.0)
Node: 16
export async function uploadFile(req: Request) {
return new Promise<any>((resolve, reject) => {
const bb = busBoy({ headers: req.headers });
let uploadData = { file: "", type: "" };
const uploadId = db.collection("_").doc().id;
let fileExt: string = "";
let newFileName: string = "";
const uuidToken = uuidv4();
const bucketName = admin.storage().bucket().name;
let fbCollTicketData: any;
// busboy.on('file', (fieldname, file, filename, encoding, mimetype) => {
bb.on('file', (name, file, info) => {
console.log("Execution started, file Block");
const { filename, encoding, mimeType } = info;
fileExt = filename.substring(filename.lastIndexOf('.'));
newFileName = `${uploadId}${fileExt}`;
console.log(`File [${name}]: filename: ${filename}, encoding: ${encoding}, mimetype: ${mimeType}`);
const filepath = path.join(os.tmpdir(), newFileName);
uploadData = { file: filepath, type: mimeType };
const fstream = fs.createWriteStream(filepath);
file.pipe(fstream);
console.log("Execution Ended, file Block");
// file.on('data', function (data) {
// console.log('File [' + fieldname + '] got ' + data.length + ' bytes');
// });
});
//busboy.on('field', function (fieldname, val, fieldnameTruncated, valTruncated, encoding, mimetype) {
bb.on('field', (name, val, info) => {
console.log("Execution started, field Block");
//console.log('Field [' + fieldname + ']: value: ' + val);
console.log(`Field [${name}]: value: %j`, val);
if (name == 'data' && val) {
fbCollTicketData = <any>JSON.parse(val);
}
console.log("Execution Ended, field Block");
});
bb.on('error', (e) => {
console.log("Execution started, error Block");
console.log(e);
console.log("Execution Ended, error Block");
});
bb.on('close', () => {
console.log('Done parsing form!');
admin.storage().bucket().upload(uploadData.file, {
destination: `${STORAGE.CLEANINGPICS}/${newFileName}`,
resumable: false,
contentType: uploadData.type,
metadata: {
metadata: {
//cacheControl: 'public, max-age=31536000',
firebaseStorageDownloadTokens: uuidToken,
}
}
}).then(async (resp) => {
resolve(fbCollTicketData);
fs.unlinkSync(uploadData.file);
}).catch(err => {
reject(err);
})
});
req.pipe(bb);
});
};
In cloud function when i check Functions log it shows me Error "Unexpected end of form
at Multipart._final"
@mscdex - Please help.
Hello,
I have a simple route like this:
router.post('/locations/file', busboy(), function(req, res, next) {
console.log('here');
req.busboy.on('file', function (fieldname, file, filename) {
console.log('bar');
var filePath = path.join(os.tmpdir(), filename);
fstream = fs.createWriteStream(filePath);
file.pipe(fstream);
fstream.on('close', function () {
console.log('foo');
});
});
req.pipe(req.busboy);
});
And I have a weird behaviour: The file event is not triggered everytime (with the same file and the same upload form). but the 'here' log is displayed
I'm using express 4.6.1, connect-busboy 0.0.2 with busboy 0.2.9.
Thanks
Busboy do not parse fields. I submit the images with fields as multipart/formdata
curl http://localhost:3000/api/signup -F "[email protected]" -F "[email protected]" -F "email=bill" -F "password=hifsafafasd" -F "gender=true" -F "device_id=adsf" -F "platform=1" -v
I have the following controller code:
signup: function(req, res){
var data = {};
data.imagesInfo = {};
req.busboy.on('file', function(fieldname, file, filename, encoding){
file.on('data', function(binary) {
data.imagesInfo[fieldname + 'Encoding'] = encoding;
if(!data.imagesInfo[fieldname]) {
data.imagesInfo[fieldname] = new Buffer(binary);
s} else {
data.imagesInfo[fieldname] = Buffer.concat([data.imagesInfo[fieldname], binary]);
}
});
});
req.busboy.on('field', function(fieldname, val){
console.log(val);
data[fieldname] = val;
});
req.pipe(req.busboy);
req.busboy.on('finish', function() {
console.log(data);
req.services.users.create(data).then(function(user){
res.json(user);
}).catch(function(err){
res.json({ 'error' : err });
});
});
},
The values did not parsed while field name and files did. The log is:
{ imagesInfo:
{ imageEncoding: '7bit',
image: <Buffer ff d8 ff e0 00 10 4a 46 49 46 00 01 01 00 00 01 00 01 00 00 ff db 00 84 00 09 06 06 14 12 10 15 14 12 14 14 14 14 14 15 14 14 14 16 17 14 14 15 16 15 17 ...>,
imageSmallEncoding: '7bit',
imageSmall: <Buffer ff d8 ff e0 00 10 4a 46 49 46 00 01 01 00 00 01 00 01 00 00 ff db 00 84 00 09 06 07 14 13 12 15 14 13 14 15 16 14 17 14 17 18 17 18 17 17 17 17 18 16 18 ...> },
email: '',
password: '',
gender: '',
device_id: '',
platform: '1' }
I stumbled across this middleware because I am looking for a middleware that automatically grabs fields and attached files from a multipart upload but keeps the file contents available in memory (i.e. doesn't write them to disk). It looks like it is possible to extend this middleware to do the job. However, this seems like a very general problem and so I'm wondering if it makes sense to extend this middleware with this functionality. Thoughts?
This works:
app.post(`/`, async (req, res) => { /* busboy code */ });
This doesn't:
app.post(`/`, async (req, res) => { await (async () => {})(); /* busboy code */ });
Seems like the act of awaiting breaks how busboy works. I can't figure out why though.
Hi,
I am using connect-busboy to upload image files on my server. It works fine with all the desktop and android devices. But it is failing for the iPad/iPhone for bigger files >=1MB. On desktop,it is working with all types of files. On iPhone/iPad,it is working with small images. Is there some limitation using that? I am using angular-file-upload on the client. I tried using the native javascript XML HTTP request as well. There also, it fails for big files.
Regards,
Manik Mittal
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.