adonisjs / bodyparser Goto Github PK

what happened to FileJar reference in Adonis documentation https://adonisjs.com/docs/4.1/file-uploads#_multiple_file_uploads
link: https://github.com/adonisjs/adonis-bodyparser/blob/develop/src/Multipart/FileJar.js
does not exist.
and using multiple upload mode, returns object of undefined values! FileJar is missing.

The following post data:

items[0][volume][0][0]=foo
items[0][volume][1][0]=bar

Gets parsed like this:

"items": [{
    "volume" : [ 
        {
            "[0]" : "foo"
        }, 
        {
            "[0]" : "bar"
        }
    ]
}]

It should be:

"items": [{
    "volume" : [ 
        [
            ["foo"]
        ],
        [
            ["bar"]
        ]
    ]
}]

Package version

2.0.9

Node.js and npm version

Node v10.9.0 NPM v6.6.0

Sample Code (to reproduce the issue)

Multiparty options are hardcoded and not customizable in AdonisJs in node_modules\@adonisjs\bodyparser\src\Multipart\index.js

    this._multipartyOptions = {
      autoFields: true,
      autoFiles: false
    }

This prevents changing options like maxFields and now I'm stuck without being able to do anything because I need more than the default of 1000 fields with simple forms.

Package version

@adonisjs/[email protected]

Node.js and npm version

Node.js v12.16.0
npm 6.13.4

Sample Code (to reproduce the issue)

File validator works as required.
It's optional rule
'image': 'file|file_size:2mb|file_types:image',

@adonisjs\bodyparser\src\Bindings\Validations.js

  Validator.extend('file', async function (data, field, message, args, get) {
    if (get(data, field) instanceof File === false) {
      throw message
    }
  })

Need change something like:

  Validator.extend('file', async function (data, field, message, args, get) {
    const dataField = get(data, field)
    if (dataField && dataField instanceof File === false) {
      throw message
    }
  })

When uploading file, File class and validation function provides optional mime type validation.

But there are cases when mime type validation is not acceptable, e.g. when uploading CSV file, it can be recognized with many different mime types which is dependable on local user configuration. CSV file could be recognized as one of the following: text/csv, application/vnd.ms-excel, application/octet-stream etc.
Many files are recognized as application/octet-stream so enabling this mime type would allow users accidentally upload incorrect files.

It would be nice if support for filename extension validation could be added.

Brief Description

I tried stream files by following this https://adonisjs.com/docs/4.0/file-uploads docs. But it seems like "processManually" doesn't work when "autoProcess" set to "true".

Sample Code

Router.js:

...
Route.post('upload', 'UploadController.upload')
....

UploadController.js

...
    request.multipart.file('my-image', {}, async (file) => {
      await imageOptimizer(file.stream, file.fileName)
    })

    try {
      await request.multipart.process()
      console.log('works');
    } catch (error) {
      console.log(error);
    }
...

Code works correctly only if in config/bodyParser.js key autoProcess set to false (autoProcess: false)

But the same code doesn't work when:

autoProcess: true,
processManually: ['upload']

autoProcess: ['someurl'],
processManually: ['upload']

Error Stack

RuntimeException: E_CANNOT_PROCESS_STREAM: Cannot process multipart stream twice. Make sure to disable files {autoProcess} when manually calling multipart.process
    at Function.invoke (/Users/vlad/Downloads/nanoparser2/node_modules/@adonisjs/generic-exceptions/src/RuntimeException.js:83:12)
    at Multipart.process (/Users/vlad/Downloads/nanoparser2/node_modules/@adonisjs/bodyparser/src/Multipart/index.js:163:49)
    at UploadController.upload (/Users/vlad/Downloads/nanoparser2/app/Controllers/Http/UploadController.js:31:31)
    at Object.method (/Users/vlad/Downloads/nanoparser2/node_modules/@adonisjs/framework/src/Server/index.js:221:18)
    at Server._resolveMiddleware (/Users/vlad/Downloads/nanoparser2/node_modules/@adonisjs/framework/src/Server/index.js:110:28)
    at _resolveListItem (/Users/vlad/Downloads/nanoparser2/node_modules/co-compose/src/Runnable.js:50:28)
    at dispatch (/Users/vlad/Downloads/nanoparser2/node_modules/co-compose/src/Runnable.js:112:16)
    at _.once (/Users/vlad/Downloads/nanoparser2/node_modules/co-compose/src/Runnable.js:111:35)
    at /Users/vlad/Downloads/nanoparser2/node_modules/lodash/lodash.js:10067:25
    at AuthInit.handle (/Users/vlad/Downloads/nanoparser2/node_modules/@adonisjs/auth/src/Middleware/AuthInit.js:46:11)

Screenshots (if any)

Issue I face

• After I uploaded the file using the .moveToDisk method,When I tried to console log the file.toJSON(). It has returned the below object.

{
  fieldName: 'profile',
  clientName: 'Open Peeps - Avatar and Backdrop.png',
  size: 54105,
  filePath: '/var/www/html/adonis-crud/tmp/uploads/assets/users/profile/clhitg3ce0001xho51lto0kmd.png',
  fileName: 'clhitg3ce0001xho51lto0kmd.png',
  type: 'image',
  extname: 'png',
  subtype: 'png',
  state: 'moved',
  isValid: true,
  validated: true,
  errors: [],
  meta: {}
}

However, I found that file.toJSON() is a type of FileJSON.

But it is missing the fileName in the object. The current FileJSON object is look like this

{
        fieldName: string;
        clientName: string;
        size: number;
        filePath?: string;
        type?: string;
        extname?: string;
        subtype?: string;
        state: 'idle' | 'streaming' | 'consumed' | 'moved';
        isValid: boolean;
        validated: boolean;
        errors: FileUploadError[];
        meta: any;
    }

Please add the missing values from FileJSON type

Package version

"@adonisjs/bodyparser": "^8.1.7",
"@adonisjs/core": "^5.9.0",

Node.js and npm version

node version: v16.19.0

npm version: 8.19.3

Sample Code (to reproduce the issue)

BONUS (a sample repo to reproduce the issue)

How to increase parameterLimit on type of UrlEncoded?
I try to add parameterLimit: 100000 in config/bodyParser.js file and still not working

Executing this curl request crashes the entire server.

curl -X POST --url http://127.0.0.1:3333 \
  -H 'Content-Type: multipart/form-data; boundary=XXX' \
  -d $'--XXX\r\nContent-Disposition: form-data; name=""\r\n\r\nabc\r\n--XXX--\r\n'

Discussed in adonisjs/core#3231

I've the same issue with my content field, it can have a base64/binary, I'm also modified config/bodyparser.ts but still got this error. Any help?

File adonis-bodyparser/CONTRIBUTING.md contains a link to http://adonisjs.com/docs/2.0/contributing, which is a 404.

Package version

[1.0.0 .. 2.0.9]

Node.js and npm version

N/A

Sample Code (to reproduce the issue)

N/A

Package version

@adonisjs/[email protected]

Node.js and npm version

Node.js v12.16.0
npm 6.13.4

Sample Code (to reproduce the issue)

Validate file size, file extension and file types don't return correct messages when I use custom error messages.

get rules()
{
  return {
    photo: 'required|file|file_size:2mb|file_types:image',
  }
}

get messages()
{
  return {
    'photo.required': 'Главное фото обязательно.',
    'photo.file': 'Главное фото должно быть файлом.',
    'photo.file_size': 'Размер главного фото не должен превышать 2МБ.',
    'photo.file_types': 'Главное фото должно быть картинкой.',
  }
}

fixes are quite simple:

  Validator.extend('fileSize', async function (data, field, message, args, get) {
    const error = await validateFile(get(data, field), { size: args[0] })
    if (error) {
      throw error // <-----
    }
  })

to

  Validator.extend('fileSize', async function (data, field, message, args, get) {
    const error = await validateFile(get(data, field), { size: args[0] })
    if (error) {
      throw message // <-----
    }
  })

Package version

v10.0.1

Describe the bug

For 'application/x-www-form-urlencoded', we have an option to pass queryString: { allowDots: false } but there is no such option for 'multipart/form-data' and the fields are parsed using dot for nesting.

We have a form that allows users to submit a URL along with other inputs and files. We need to store some meta related to the submitted URL in an input field like this

<input type='hidden' name={`meta[${url}][some_key]`} value={'some value'} />

The URL will obviously contain dots.

Reproduction repo

No response

Hi Guys! I'd be grateful if you'd look into this and advise.

Package version

(Basically, it's app which recently have been created with adonis new)

"@adonisjs/ace": "^5.0.8",
"@adonisjs/auth": "^3.0.7",
"@adonisjs/bodyparser": "^2.0.5",
"@adonisjs/cors": "^1.0.7",
"@adonisjs/fold": "^4.0.9",
"@adonisjs/framework": "^5.0.9",
"@adonisjs/ignitor": "^2.0.8",
"@adonisjs/lucid": "^6.1.3",
"@adonisjs/validator": "^5.0.6",

Node.js and npm version

$ node --version
v10.15.0
$ npm --version
6.4.1

Sample Code (to reproduce the issue)

Ensure app/config/bodyParser.js has files.autoProcess set to true or matches upload route.

Add the following to the start/routes.js:

Route.post('/upload', async ({request, response}) => {
  return response.send();
});

Create any reasonable YAML file e.g.:

echo "a: 2" > file.yml

Send request via httpie and observe that app crashes

http -f POST http://localhost:3333/upload [email protected]

# application logs
TypeError: argument string is required
at Object.parse (/app/node_modules/media-typer/index.js:141:11)
at new File (/app/node_modules/@adonisjs/bodyparser/src/Multipart/File.js:148:36)
at Multipart.onPart (/app/node_modules/@adonisjs/bodyparser/src/Multipart/index.js:129:26)
at Form.form.on (/app/node_modules/@adonisjs/bodyparser/src/Multipart/index.js:172:14)
at Form.emit (events.js:182:13)
at Object.cb (/app/node_modules/multiparty/index.js:642:10)
at flushEmitQueue (/app/node_modules/multiparty/index.js:626:10)
at /app/node_modules/multiparty/index.js:602:5
at handlePart (/app/node_modules/multiparty/index.js:641:3)
at Form.onParseHeadersEnd (/app/node_modules/multiparty/index.js:527:5)
at Form._write (/app/node_modules/multiparty/index.js:319:24)
at doWrite (_stream_writable.js:410:12)
at writeOrBuffer (_stream_writable.js:394:5)
at Form.Writable.write (_stream_writable.js:294:11)
at IncomingMessage.ondata (_stream_readable.js:666:20)
at IncomingMessage.emit (events.js:182:13)

To my mind, the problem is multiparty sometimes doesn't set headers['content-type'] which causes unexpected exception to be thrown.

After further investigation I think the problem concerns all files with mime text/plain:

$ file --mime-type test.yml 
test.yml: text/plain
$ file --mime-type test2.txt 
test2.txt: text/plain

http -f POST http://localhost:3333/upload [email protected]
# same result

I am using adonis for uploading file on S3. I allowed some extensions to upload in which xls is also written but every time I upload xls file it show me the error Invalid file extention cfb.

Hi there, this isn't a bug report so much as me just searching for clues for how to solve a particular issue with my Adonis app.

I'm using Adonis v4 and my app logs are flooded with multiple errors that look like this:

BadRequestError: request aborted
    at IncomingMessage.onAborted (/app/api/node_modules/raw-body/index.js:231:10)
    at IncomingMessage.emit (events.js:400:28)
    at abortIncoming (_http_server.js:569:9)
    at socketOnClose (_http_server.js:562:3)
    at /app/api/node_modules/dd-trace/packages/dd-trace/src/scope/base.js:54:19
    at Scope._activate (/app/api/node_modules/dd-trace/packages/dd-trace/src/scope/async_resource.js:53:14)
    at Scope.activate (/app/api/node_modules/dd-trace/packages/dd-trace/src/scope/base.js:12:19)
    at Socket.bound (/app/api/node_modules/dd-trace/packages/dd-trace/src/scope/base.js:53:20)
    at Socket.emit (events.js:412:35)
    at TCP.<anonymous> (net.js:686:12)
    at TCP.callbackTrampoline (internal/async_hooks.js:130:17)

Based on docs for the raw-body package, this error occurs when the following happens:

request aborted

This error will occur when the request is aborted by the client before reading the body has finished. The received property will be set to the number of bytes received before the request was aborted and the expected property is set to the number of expected bytes. The status property is set to 400 and type property is set to 'request.aborted'.

It seems odd that I would see such a large volume of these errors in my logs and I'm trying to determine if this an issue with my app, or perhaps quite simply it's just my users losing an internet connection (e.g. train going into a tunnel) causing requests to be aborted in "mid-flight".

Package version

2.3.0

Node.js and npm version

Node 14

What happens when an unsupported content type is given?
What if I give an XML payload with content type "application/rss+xml"?
Can I register custom bodyparser when some mime types are not handled by default? or should I raise PR to modify main code to handle other mime types? (I don't think it is a good idea)

🚨 You need to enable Continuous Integration on all branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because we are using your CI build statuses to figure out when to notify you about breaking changes.

Since we did not receive a CI status on the greenkeeper/initial branch, we assume that you still need to configure it.

If you have already set up a CI for this repository, you might need to check your configuration. Make sure it will run on all new branches. If you don’t want it to run on every branch, you can whitelist branches starting with greenkeeper/.

We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

From @1001v on October 26, 2017 17:54

Hello. I've reproduced sample code from File uploads guide, and it seems that request.file method uses unixtime integer to name the tmp file. So the case when two different file upload instances get the same tmp file name isn't rare, i tested. Sometimes two-three upload requests would be enough to fire the issue. I bet the tmp file name should be more unique. Thank you.

Copied from original issue: adonisjs/core#685

This error was first reported here.
I currently have an app in production that constantly gets 502 because the backend crashes. For me this is very urgent.

If I can help I will. Need a description of what is wrong and where and I can make a pull request.

Package version

"@adonisjs/ace": "^5.0.8",
"@adonisjs/auth": "^3.0.7",
"@adonisjs/bodyparser": "^2.0.5",
"@adonisjs/cors": "^1.0.7",
"@adonisjs/drive": "^1.0.4",
"@adonisjs/fold": "^4.0.9",
"@adonisjs/framework": "^5.0.9",
"@adonisjs/ignitor": "^2.0.8",
"@adonisjs/lucid": "^6.1.3",
"@adonisjs/mail": "^3.0.10",
"@adonisjs/validator": "^5.0.6"

Node.js and npm version

node v12.16.2
npm v6.14.4

Sample Code (to reproduce the issue)

const file = request.file("image");

if (file && !(file.tmpPath == null)) {
    const res = await Cloudinary.v2.uploader.upload(file.tmpPath, {
    folder: "profile_pictures",
});
user.image_url = res.secure_url;
await network.users().save(user);

I am having the same problem like described in this post https://forum.adonisjs.com/t/using-stripe-webhook-problem-with-incoming-request/135

https://github.com/adonisjs/adonis-bodyparser/blob/5f566014b39c4f9ce26655a01b3f829cf976f00f/src/BodyParser/index.js
This script returns at the first match which is json (because of Content-Type application/json) and skips setting the raw part.

In some cases, you want to parse the raw part of a json request as well to validate signatures etc.

I think the raw part should always be set whatever Content-Type it is.

I am trying to manually process a file using the request.multipart.file() callback. Since I want to use the provided validation functions within the callback, I await file.runValidations() (found this from the source code). The validation for file type works, but it doesn't work for verifying file size (looks like its due to the async nature of how the File class gets the size). Is there another way to do this, or should the File class get size from file.stream.byteCount instead?

*edit: Actually the file.error() always returns an empty object unless a type error is specifically found.

Code:

const options = {
  size: '10kb',
  types: ['image', 'video']
}
request.multipart.file('file', options, async (file) => {
        await file.runValidations()
        const fileError = file.error()
        // if size validation is set, fileError will always return {}
        if (fileError) throw fileError
        // or else process file normally
})
await request.multipart.process()

When uploading a file:

const file = request.file("file")

The file is converted to the wrong file type.
Log:

fieldName: 'file',
  clientName: 'TestFile.rvt',
  headers: {
    'content-disposition': 'form-data; name="file"; filename="TestFile.rvt"',
    'content-type': 'application/octet-stream'
  },
  size: 59801600,
  errors: [],
  meta: {},
  state: 'consumed',
  extname: 'cfb',
  type: 'application',
  subtype: 'x-cfb',
  tmpPath: '/tmp/cl4rfvjj800008i0p76xf2pm1'

I can send you a test file. Github won't let me paste it in here.

Package version

@adonisjs/bodyparser:
"version": "8.1.3",

Node.js and npm version

node: "16.14.2"
npm: "8.12.1"

Similar issue:
#53 (comment)

Package version

"version": "4.1.0",
"adonis-version": "4.1.0",

@adonisjs/[email protected]
@adonisjs/[email protected]
@adonisjs/[email protected]
@adonisjs/[email protected]
@adonisjs/[email protected]
@adonisjs/[email protected]
@adonisjs/[email protected]
@adonisjs/[email protected]
@adonisjs/[email protected]
@adonisjs/[email protected]
@adonisjs/[email protected]

Node.js and npm version

NodeJS v10.9.0
npm 6.2.0

Sample Code (to reproduce the issue)

From documentation :

view

<form method="POST" action="upload" enctype="multipart/form-data">
  {{ csrfField() }}
  <input type="file" name="profile_pic" />
  <button type="submit"> Submit </button>
</form>

start/route

Route.post('upload', async ({ request }) => {
  const profilePic = request.file('profile_pic', {
    types: ['image'],
    size: '2mb'
  })

  await profilePic.move(Helpers.tmpPath('uploads'), {
    name: 'custom-name.jpg',
    overwrite: true
  })

  if (!profilePic.moved()) {
    return profilePic.error()
  }
  return 'File moved'
})

This 1st time you use this script, everything is going well.

When you try to upload another image to replace it, then there is an HTTP error 500: "dest already exists."