Git Product home page Git Product logo

querymen's Introduction

Querymen

JS Standard Style NPM version Build Status Coveralls Status Dependency Status Downloads

Querystring parser middleware for MongoDB, Express and Nodejs

Install

npm install --save querymen

Examples

Pagination

Querymen has a default schema to handle pagination. This is the most simple and common usage.

import { middleware as query } from 'querymen';

app.get('/posts', query(), ({ querymen: { query, select, cursor } }, res) => {

  Post.find(query, select, cursor).then(posts => {
    // posts are proper paginated here
  });
});

User requests /posts?page=2&limit=20&sort=-createdAt querymen will be:

querymen = {
  query: {},
  select: {},
  cursor: {
    limit: 20, 
    skip: 20, 
    sort: { createdAt: -1 }
  }
}

User requests /posts?q=term&fields=title,desc querymen will be:

When user requests /posts?q=term, querymen parses it to {keywords: /term/i}. It was designed to work with mongoose-keywords plugin, which adds a keywords field to schemas (check that out).

querymen = {
  query: {
    keywords: /term/i
  },
  select: {
    title: 1,
    desc: 1
  },
  cursor: {
    // defaults
    limit: 30, 
    skip: 0, 
    sort: { createdAt: -1 }
  }
}

User requests /posts?fields=-title&sort=name,-createdAt querymen will be:

querymen = {
  query: {},
  select: {
    title: 0
  },
  cursor: {
    limit: 30, 
    skip: 0, 
    sort: {
      name: 1,
      createdAt: -1
    }
  }
}

Custom schema

You can define a custom schema, which will be merged into querymen default schema (explained above).

import { middleware as query } from 'querymen';

app.get('/posts', query({
  after: {
    type: Date,
    paths: ['createdAt'],
    operator: '$gte'
  }
}), ({ querymen }, res) => {
  Post.find(querymen.query).then(posts => {
    // ...
  });
});

User requests /posts?after=2016-04-23 querymen will be:

querymen = {
  query: {
    createdAt: { $gte: 1461369600000 }
  },
  select: {},
  cursor: {
    // defaults
    limit: 30, 
    skip: 0, 
    sort: { createdAt: -1 }
  }
}

Reusable schemas

You can create reusable schemas as well. Just instantiate a Schema object.

import { middleware as query, Schema } from 'querymen';

const schema = new Schema({
  tags: {
    type: [String],
  }
});

// user requests /posts?tags=world,travel
// querymen.query is { tags: { $in: ['world', 'travel'] }}
app.get('/posts', query(schema));
app.get('/articles', query(schema));

Advanced schema

import { middleware as query, Schema } from 'querymen';

const schema = new Schema({
  active: Boolean, // shorthand to { type: Boolean }
  sort: '-createdAt', // shorthand to { type: String, default: '-createdAt' }
  term: {
    type: RegExp,
    paths: ['title', 'description'],
    bindTo: 'search' // default was 'query'
  },
  with_picture: {
    type: Boolean,
    paths: ['picture'],
    operator: '$exists'
  }
}, {
  page: false, // disable default parameter `page`
  limit: 'max_items' // change name of default parameter `limit` to `max_items`
});

app.get('/posts', query(schema), ({ querymen }, res) => {
  // user requests /posts?term=awesome&with_picture=true&active=true&max_items=100
  // querymen.query is { picture: { $exists: true }, active: true }
  // querymen.cursor is { limit: 100, sort: { createdAt: -1 } }
  // querymen.search is { $or: [{ title: /awesome/i }, { description: /awesome/i }]}
});

Dynamic advanced schema

import { middleware as query, Schema } from 'querymen';
const schema = new Schema();

schema.formatter('scream', (scream, value, param) => {
  if (scream) {
    value = value.toUpperCase() + '!!!!!!!';
  }
  return value;
});

schema.param('text', null, { type: String }); // { type: String }
schema.param('text').option('scream', true); // { type: String, scream: true }
schema.param('text').value('help');
console.log(schema.param('text').value()); // HELP!!!!!!!

schema.validator('isPlural', (isPlural, value, param) => {
  return {
    valid: !isPlural || value.substr(-1) === 's',
    message: param.name + ' must be in plural form.'
  };
});

schema.param('text').option('isPlural', true); // { type: String, scream: true, isPlural: true }
console.log(schema.validate()); // false
schema.param('text', 'helps');
console.log(schema.validate()); // true
console.log(schema.param('text').value()); // HELPS!!!!!!!

schema.parser('elemMatch', (elemMatch, value, path, operator) => {
  if (elemMatch) {
    value = { [path]: { $elemMatch: {[elemMatch]: {[operator]: value } }}};
  }
  return value;
});

schema.param('text', 'ivegotcontrols');
console.log(schema.param('text').parse()); // { text: 'IVEGOTCONTROLS!!!!!!!' }

schema.param('text').option('elemMatch', 'prop');
console.log(schema.param('text').parse()); // { text: { $elemMatch: { prop: { $eq: 'IVEGOTCONTROLS!!!!!!!'} }}}

Geo queries

Querymen also support geo queries, but it's disabled by default. To enable geo queries you just need to set near option to true in schema options.

import { middleware as query } from 'querymen';

app.get('/places', query({}, { near: true }), (req, res) => {
  
});

Its paths option is set to ['location'] by default, but you can change this as well:

import { middleware as query } from 'querymen';

app.get('/places', 
  query({
    near: { paths: ['loc'] }
  }, {
    near: true
  }), 
  (req, res) => {
  
  });

User requests /places?near=-22.332113,-44.312311 (latitude, longitude), req.querymen.query will be:

req.querymen.query = {
  loc: {
    $near: {
      $geometry: {
        type: 'Point',
        coordinates: [-44.312311, -22.332113]
      }
    }
  }
}

User requests /places?near=-22.332113,-44.312311&min_distance=200&max_distance=2000 (min_distance and max_distance in meters), req.querymen.query will be:

req.querymen.query = {
  loc: {
    $near: {
      $geometry: {
        type: 'Point',
        coordinates: [-44.312311, -22.332113]
      },
      $minDistace: 200,
      $maxDistance: 2000
    }
  }
}

You can also use legacy geo queries as well. Just set geojson option in param:

import { middleware as query } from 'querymen';

app.get('/places', 
  query({
    near: {
      paths: ['loc'],
      geojson: false
    }
  }, {
    near: true
  }), 
  (req, res) => {
  
  });

User requests /places?near=-22.332113,-44.312311&min_distance=200&max_distance=2000, req.querymen.query will be:

req.querymen.query = {
  loc: {
    $near: [-44.312311, -22.332113],
    // convert meters to radians automatically
    $minDistace: 0.000031,
    $maxDistance: 0.00031
  }
}

Error handling

// user requests /posts?category=world
import { middleware as query, querymen, Schema } from 'querymen';

const schema = new Schema({
  category: {
    type: String,
    enum: ['culture', 'general', 'travel']
  }
});

app.get('/posts', query(schema));

// create your own handler
app.use((err, req, res, next) => {
  res.status(400).json(err);
});

// or use querymen error handler
app.use(querymen.errorHandler());

Response body will look like:

{
  "valid": false,
  "name": "enum",
  "enum": ["culture", "general", "travel"],
  "value": "world",
  "message": "category must be one of: culture, general, travel"
}

Contributing

This package was created with generator-rise. Please refer to there to understand the codestyle and workflow. Issues and PRs are welcome!

License

MIT Β© Diego Haz

querymen's People

Contributors

chemitaxis avatar diegohaz avatar greenkeeper[bot] avatar jorgeluisrezende avatar tguelcan avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

querymen's Issues

Wrong pagination params on cursor

I have the following Schema


    new querySchema({
      type: String,
      garment: String,
      brand: String,
      supplier: String,
      name: RegExp
    })

Haven't touched anything regarding pagination or limit, however after updating to 2.1.3 to include the cache problem, which I was having as well, this happens:

http://localhost:9000/branch-products?sort=-createdAt&page=1&limit=10
Mongoose: branchproducts.find({}, { skip: 0, limit: 10, sort: { createdAt: -1 }, fields: {} })

http://localhost:9000/branch-products?sort=-createdAt&page=2&limit=10
Mongoose: branchproducts.find({}, { skip: 30, limit: 10, sort: { createdAt: -1 }, fields: {} })

http://localhost:9000/branch-products?sort=-createdAt&page=3&limit=10
Mongoose: branchproducts.find({}, { skip: 60, limit: 10, sort: { createdAt: -1 }, fields: {} })

http://localhost:9000/branch-products?sort=-createdAt&page=4&limit=10
Mongoose: branchproducts.find({}, { skip: 90, limit: 10, sort: { createdAt: -1 }, fields: {} })


As if the limit was 30. Any ideas?

Default settings without pagination

First, everything works like a charm, thanks!

I have one (minor) problem I can't seem to figure out. How to disable pagination and limits on default, unless some sets the limit query parameter?

So basically, if someone makes a request for a list without any limit, they should receive all. But if they set a limit, then pagination should be used.
Any suggestions?

Kind regards

[Question, Help needed] AND operator

Hello,
Is it possible, to have something like this:

query({
    attractions: { type: [String], paths: ['attractions'], multiple: true },
    categories: { type: [String], paths: ['category'], multiple: true },
    search: {
      type: RegExp,
      paths: ['name', 'description', 'city', 'street']
    }
  }),

And that would I like to have, is to match all of the rules, for example, if attractions and categories are specified, I'd like to have only objects which match both of the rules.

Same for search functionality, both filters should be applied (attractions and categories), then the search regex.

Improve README

Change examples to ES6, generate API reference and add a note about bodymen.

Handle errors

Currently it just set response status to 400 and pass error status to next()

Support injecting custom validators, parsers and formatters

Depending on #3

var menquery = require('menquery');

menquery.parser('exclamative', function(parserValue, paramValue, param) {
  return parserValue ? paramValue + '!' : paramValue;
});

var schema = new menquery.Schema({
  param: {
    exclamative: true
  }
});

schema.parser('exclamative', function(parserValue, paramValue, param) {
  return parserValue ? paramValue + '!' : paramValue;
});

schema.param('param').parser('exclamative', function(parserValue, paramValue, param) {
  return parserValue ? paramValue + '!' : paramValue;
});

Is there a way to paginate more than 30

I am having more than 3000 collections, how do I paginate in this case ?

/books?limit=100&page=31

{
"name": "max",
"param": "page",
"value": 31,
"max": 30,
"valid": false,
"message": "page must be lower than or equal to 30"
}

Multiple filters same path custom schema

Hi, if you add this to the middleware:

query({
    after: {
      type: Date,
      paths: ['DateInvoice'],
      operator: '$gte'
    },
    before: {
      type: Date,
      paths: ['DateInvoice'],
      operator: '$lte'
    }
  }),

Just the last is working, the first (after) is ignored... Any ideas? Thanks

Filter by case insensitive regex

Is it possible to filter records by case insensitive regex? My current implementation looks like:

router.get('/',
  token({ required: true }),
  query(),
  querymen.middleware({
    name: {
      type: String,
      paths: ['name'],
      operator: '$regex'
    }
  }),
  index)

Using custom schema causes wrong pagination

If a schema (either empty) is attached to routing,
and I set page and limit parameters in query, "skip" is calculated based on default "limit" value = 30 - not a value from query:

Bug

Skip should be 20 not 30.
In other words, if schema is attached 'skip' is calculated always based on default value.
With no schema attached - everything works fine.

I believe this is duplicate of #50
But that problem is still present.

querying ref / nested properties

Hi,

How can i use queryman to query references schema properties?
lets say i have City Model and Streets Model where

  id: 123,
  name: 'my city',
  streets: [..]
}

where streets is sub resource ref fields for example in the City Schema its defined like this:

    type: Schema.ObjectId,
    ref: 'Streets',
    field: 'cityId'
  }]

so when doing query man i want to search bit city name and streets name like so:

    term: {
      type: RegExp,
      paths: ['name', 'streets.name'],
        }
url?term="york"

so if i search for york it will give me cities that have "york" in their name and streets that have "york" in their name also.

hope i was clear,

Thanks!

keywords is returning an empty object

const express = require("express");
const qmen = require("querymen");

const app = express();

app.get("/", qmen.middleware(), ({ querymen: { query } }, res, next) => {
  res.send(query);
});

app.listen(8000, console.log.call(null, "Server running"));

As simple as that. when I request /?q=term I get an empty object:

{
    "keywords": {}
}

I don't know if I should enable something else or if it's really an issue.
Express version: 4.17.1
Node version: v12.16.1

Edit: It should be noted that all other features work well.

Update Lodash

Dependencies are shown to be insecure. Is it possible to update?

Thanks!

Get user ID from token in querymen

Hi Diego, I love your boilerplate (Yeoman Generator Rest), but I have a "question" that I can't solve...

I need to customize a Schema, something like this:

const customSchema = new QuerymenSchema({
  sort: '-LastModified', // this change the default order
  IdUser: '5819c860072c1d156cfec382' // this is the field I need to change
}, {
  page: false,
  limit: false
})

The purpose of that, if get the resource of an endpoint but JUST of the owner user... How can I solve this with querymen?

This is the output of Moongose:
myCollection.find({ IdUser: '5819c860072c1d156cfec382' }, { sort: { LastModified: -1 }, fields: {} })

Everything is ok, but I need to change the IdUser for the current user logged. I think this is a typical approach ;)

Thanks for your work, is excellent!

page must be lower than or equal to 30

How do I remove the max validation of the page. I am getting this error.
max:30
message:"page must be lower than or equal to 30"
name:"max"
param:"page"
valid:false

Adopt a pattern for get/set things

If we are going to use param.value() to get value and param.value('bar') to set value, it makes more sense to change schema.get('foo') and schema.set('foo', 'bar') to schema.param('foo') and schema.param('foo', 'bar').

Optional sort param

Hi @diegohaz, I'm thinking about mandatory sort. We are working with MongoDb, and some querys we don't need an index for sorting, but the "sort" param is always "one" because we have a default value:

sort: {
        type: [String],
        default: '-createdAt',
        bindTo: 'cursor',
        parse: (value) => {
          let fields = _.isArray(value) ? value : [value]
          let sort = {}
          fields.forEach((field) => {
            if (field.charAt(0) === '-') {
              sort[field.slice(1)] = -1
            } else if (field.charAt(0) === '+') {
              sort[field.slice(1)] = 1
            } else {
              sort[field] = 1
            }
          })
          return {sort: sort}
        }
      }

Can you please give some "light" to try this sort param in the Schema as "optional". Thanks!

Sort for referenced documents

If I have a model like below:

Authors:[
{_id:1
name:'Author 1'
}]

Books:[
{_id:1, authors:[1,2]
]

How can I perform sort on books.authors.name? Is it possible?

Support type: [Type]

Support array type. Think about the difference between this and multiple: true

Pass schema object to req

req.menquery.schema
req.menquery.query // {_q: /bla/i}
req.menquery.select // {field: 1}
req.menquery.cursor // {skip: 0, limit: 30, sort: {field: 1}}

Query not working (maybe I'm wrong)

I have a model, how do I query a field? Let's suppose I have the following model:

Timesheet : {
   month: 5,
   year: 2018
}

It is correct if i make a query like

url/timesheets?month=5&year=2018 ?
Because it is not working for me, maybe I'm doing something wrong

"limit" Paging error when using "Schema"

const querySchema = new QuerymenSchema({
  keywords: { search: true },
  joinedPlatform: { type: String, paths:['joined'], elementMatch: 'platform' },
  joinedGroup: { type: String, paths:['joined'], elementMatch: 'group' },
  joinedGrade: { type: String, paths:['joined'], elementMatch: 'grade' },
  createdAtDate: { type: Date, paths: ['createdAt'], duration: 'date', },
  status: status.type, 
  isAdult: status.isAdult, 
  activatedToEmail: activatedToEmail.type,
  activatedToMobile: activatedToMobile.type,
  activatedToIdentify: activatedToIdentify.type,
  activatedToBankAccount: activatedToBankAccount.type,
  allowMailing: allowMailing.type,
  allowSms: allowSms.type,
  page: { max: 100000 },
  limit: { max: 100000 }
})
querySchema.parser('search', (search, value, path, operator) => {
  if (!value) { return value }
  const regValue = new RegExp(value, "ig")
  if (search) {
    if(value*1 >= 0) {
      value = { $or: [{ mobile: regValue },{ accountNo: value*1 }] }
    } else {
      value = { $or: [{ email: regValue },{ name: regValue },{ realName: regValue },{ mobile: regValue },{ accountId: regValue }] }
    }
  }
  return value
})
querySchema.parser('elementMatch', (elementMatch, value, path, operator) => {
  if (!value) { return value }
  if (elementMatch) {
    value = { [path]: { $elemMatch: { [elementMatch]: value }, } }
  }
  return value
})
querySchema.parser('duration', (duration, value, path, operator) => {
  if (!value) { return value }
  if (duration == 'date') { 
    value = { [path]: { $gte: new Date(value).setHours(0,0,0,0), $lte: new Date(value).setHours(23,59,59,999), } }
  }
  return value
})
router.get('/',
  //token({ required: true, roles: ['admin'] }),
  query(querySchema),
  index)

then vscode debugs I saw an error like this.

{ query: {},
  select: {},
  cursor: { skip: 30, limit: 15, sort: { createdAt: -1 } } }
GET /?limit=15&page=2 200 1360.531 ms - -
{ query: {},
  select: {},
  cursor: { skip: 60, limit: 15, sort: { createdAt: -1 } } }
GET /?limit=15&page=3 200 25.560 ms - -
{ query: {},
  select: {},
  cursor: { skip: 0, limit: 15, sort: { createdAt: -1 } } }
GET /?limit=15&page=1 200 27.345 ms - -
{ query: {},
  select: {},
  cursor: { skip: 30, limit: 15, sort: { createdAt: -1 } } }
GET /?limit=15&page=2 200 27.026 ms - -
{ query: {},
  select: {},
  cursor: { skip: 60, limit: 15, sort: { createdAt: -1 } } }
GET /?limit=15&page=3 200 26.904 ms - -
{ query: {},
  select: {},
  cursor: { skip: 90, limit: 15, sort: { createdAt: -1 } } }
GET /?limit=15&page=4 200 30.536 ms - -
{ query: {},
  select: {},
  cursor: { skip: 120, limit: 15, sort: { createdAt: -1 } } }

Why is it an error here?

in querymen/index.js

    if (schema && schema.options && schema.options.near) {
      _schema = schema instanceof Schema
        ? _.clone(schema)
        : new Schema(schema, options)
    } else {
      _schema = schema instanceof Schema
        ? _.cloneDeep()
        : new Schema(schema, options)
    }

after

export function middleware (schema, options) {
  return function (req, res, next) {
    let _schema = schema instanceof Schema ? schema : new Schema(schema, options)

    _schema.validate(req.query, (err) => {
      if (err) {
        req.querymen = { error: err }
        res.status(400)
        return next(err.message)
      }

      req.querymen = _schema.parse()
      req.querymen.schema = _schema
      next()
    })
  }
}

This is how it works.

GET /?limit=15&page=3 200 27.436 ms - -
{ query: {},
  select: {},
  cursor: { skip: 0, limit: 15, sort: { createdAt: -1 } } }
GET /?limit=15&page=1 200 24.773 ms - -
{ query: {},
  select: {},
  cursor: { skip: 15, limit: 15, sort: { createdAt: -1 } } }
GET /?limit=15&page=2 200 25.139 ms - -
{ query: {},
  select: {},
  cursor: { skip: 30, limit: 15, sort: { createdAt: -1 } } }
GET /?limit=15&page=3 200 24.126 ms - -
{ query: {},
  select: {},
  cursor: { skip: 45, limit: 15, sort: { createdAt: -1 } } }

Why did this happen?

An in-range update of eslint-plugin-standard is breaking the build 🚨

Version 2.2.0 of eslint-plugin-standard just got published.

Branch Build failing 🚨
Dependency eslint-plugin-standard
Current Version 2.1.1
Type devDependency

This version is covered by your current version range and after updating it in your project the build failed.

As eslint-plugin-standard is β€œonly” a devDependency of this project it might not break production or downstream projects, but β€œonly” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this πŸ’ͺ


Status Details
  • ❌ continuous-integration/travis-ci/push The Travis CI build could not complete due to an error Details
Commits

The new version differs by 3 commits .

See the full diff.

Not sure how things should work exactly?

There is a collection of frequently asked questions and of course you may always ask my humans.


Your Greenkeeper Bot 🌴

Custom schema remember previous value

Hi:

I'm using a custom schema like this

new querymen.Schema({
    store: {
        type: RegExp,
        paths: ['store'],
        normalize: true,
    },
});

If I made a request like this:
myurl.com/stores/?store=amazon, everithing is ok, but if I made a request after, but without any params: myurl.com/stores/, for some reason i keep getting this { store: /amazon/i } in my query

Is there a way to fix this behavior? or maybe I'm missing somethings.

best regards.

question on querymen default schema

In the README:

User requests /posts?q=term&fields=title,desc req.querymen will be:

When user requests /posts?q=term, querymen parses it to {keywords: /term/i}. It was designed to work with mongoose-keywords plugin, which adds a keywords field to schemas (check that out).

I did look at mongoose-keywords but I have to admit that I don't really understand how it works. Anyway the querymen README says "When user requests /posts?q=term, querymen parses it to {keywords: /term/i}". My question would be what's 'i' in 'term/i'?

Non-Ascii characters in query are replaced

Lets assume simple search url like: /search?q=SΓ³l
By default, the query is parsed as /sol/i but this is not right (different word)
I tried to define my own schema:

let searchSchema = new querymen.Schema({
  q: {
    type: String
  }
})

But still non-ascii characters are replaced.
Can I somehow disable that escaping?

negative Query

How produce queries like this:

find({"$nor": [{"status": "inactive"}], "$and": [{"type": "x"}] })

?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.