CursorPaginator takes 2 parameters queryset and ordering and applies the ordering to the queryset on init.

However, if a query set is already adequately ordered for pagination purposes, either explicitly or on the model's Meta class, it would be convenient to be able to infer and user that ordering and to call CursorPaginator with a single queryset argument.

For example:

class CursorPaginator(object):

    def __init__(self, queryset, ordering=None):
        self.queryset = queryset
        self.ordering = ordering

        if ordering:
            self.queryset = queryset.order_by(*ordering)
        elif queryset.query.order_by:
            self.ordering = queryset.query.order_by
        elif queryset.query.get_meta().ordering
            self.ordering =  queryset.query.get_meta().ordering
        else:
            raise InvalidCursor('No ordering supplied of inferable') 

The code that I'm using is very similar to the example provided in the README:

qs = Vendor.objects.filter(client__pk=client_id)
page_size = 2
paginator = CursorPaginator(qs, ordering=('-completed_date', '-id'))
page = paginator.page(first=page_size, after=after)
data = {
    'objects': [p for p in page],
    'has_next': True,
    'last_cursor': paginator.cursor(page[-1])
}
return data

This is working for the first request with the after argument defaulted to None, but as soon as the after argument is added, I receive the following error row value misused. This error is caused mainly by the line queryset = queryset.annotate(_cursor=Tuple(*[o.lstrip('-') for o in self.ordering])). When accessing the queryset variable after this line, I receive the error row value misused. I'm using Django 2.1.5 and sqlite3 2.6.0.

Duplicate the exact "after" cursor item, when get next page.

The page() function evaluates querysets and will fail if called from within an async function. Will raise a SynchronousOnlyOperation exception.

In this project at the moment I have not used the offset approach which used by Django rest framework and inspired by David Cramer's blog post. The main reason for this is that it doesn't provide "globally unique" cursors. In particular you cannot guarantee that before=FIRST_ELEMENT_CURSOR will give the correct data set, as the offset of the first element cannot be calculated. This edge case could possibly be covered by a second query checking for uniqueness on that value, although this would be overkill in most situations.

From a bit of research, Disqus is the only implementation of cursor pagination "in the wild" which makes use of offsets. Both twitter and facebook use id or timestamp based approaches. By encoding the value of all fields used in the ordering into the cursor (rather than just the first in DRF's implementation), we don't need offsets and the resulting code is much simpler - in particular when calculating the cursor for every item on the page.

I've always tried to make sure that any paginated set has a truly unique ordering - for most applications either id or created is sufficient. It also means you can do nice things like .order_by('-score', '-created') where score may not be unique enough to guarantee that you don't need large offsets.

Thoughts?

Currently, our build step look like this:

python -m pip install build twine
python -m build
python -m twine check dist/*
python -m twine upload dist/*

I think we can migrate to a tool like flit, pdm or hatch, which won't require us to have a setup.py file (and we can migrate to pyproject.toml)

Given a set of 10 items, and the query "first 3 after 4", I'd expect the return value to have has_next=True and has_previous=True, but it only has has_next=True.

In this project at the moment I have not used the offset approach which used by Django rest framework and inspired by David Cramer's blog post. The main reason for this is that it doesn't provide "globally unique" cursors. In particular you cannot guarantee that before=FIRST_ELEMENT_CURSOR will give the correct data set, as the offset of the first element cannot be calculated. This edge case could possibly be covered by a second query checking for uniqueness on that value, although this would be overkill in most situations.

From a bit of research, Disqus is the only implementation of cursor pagination "in the wild" which makes use of offsets. Both twitter and facebook use id or timestamp based approaches. By encoding the value of all fields used in the ordering into the cursor (rather than just the first in DRF's implementation), we don't need offsets and the resulting code is much simpler - in particular when calculating the cursor for every item on the page.

I've always tried to make sure that any paginated set has a truly unique ordering - for most applications either id or created is sufficient. It also means you can do nice things like .order_by('-score', '-created') where score may not be unique enough to guarantee that you don't need large offsets.

Thoughts?

Since mongoengine doesn't support django annotate, the following exception occurs when trying to paginate.
'QuerySet' object has no attribute 'annotate'
in the following line:
queryset = queryset.annotate(_cursor=Tuple(*[o.lstrip('-') for o in self.ordering]))

I'm using django-cursor-pagination for queryset pagination, and for testing my code I'm using the django-mock-queries package for mock queryset , this packages not compatible with each other.

https://pypi.org/project/django-mock-queries/

Since the direction of the sorts are checked assuming they are strings (source), attempting to use an OrderBy object will cause a crash.

The use of an OrderBy object is required in order to change how null values are sorted (source)

qs.order_by(F('field_name').desc(nulls_last=True))

How to use forward pagination and backwards pagination in an API?

Can you please provide instructions on how to jump to a specific page? It seems I need to go thru the pages to work out the "after" from previous page.

👋 First of all, thank you for this awesome library!

I encountered this None/Null issue at my workplace (modified) where the last object of the previous page has NULL value for one of the ordering keys.

ERROR:  invalid input syntax for type smallint: "None" at character 1564

... AND ("product"."tier" > 'None' OR ("product"."tier" = 'None' AND "product"."id" > '555'))) ORDER BY "product"."tier" ASC, "product"."id" DESC LIMIT 101

I made a patch and that is working for me but now I am trying to update this library, so I can use it as a regular library.

This is what I have so far: https://gist.github.com/untidy-hair/9e432cb024c00dd8fa25ebcceda6a81a
And this is the new test: https://gist.github.com/untidy-hair/563c9efa55f14108fbb2199d68534cc5

Whether Null comes at the beginning or at the end of the order is DB dependent. In that sense, my implementation there is a little bit opinionated for always putting NULL at the end (except when "last" exists). I personally cannot think of any reasons why NULL should come first but that can be configurable, too, if we like. (Caveat on README should be updated with this)

I was going to make a PR but first I thought it might be better asking here! Thanks in advance!

[EDIT]
On a second thought, maybe it's easier to be able to see diffs than just the whole Gist file so I'm creating this PR as a draft for discussions 🙏 #45

In this project at the moment I have not used the offset approach which used by Django rest framework and inspired by David Cramer's blog post. The main reason for this is that it doesn't provide "globally unique" cursors. In particular you cannot guarantee that before=FIRST_ELEMENT_CURSOR will give the correct data set, as the offset of the first element cannot be calculated. This edge case could possibly be covered by a second query checking for uniqueness on that value, although this would be overkill in most situations.

From a bit of research, Disqus is the only implementation of cursor pagination "in the wild" which makes use of offsets. Both twitter and facebook use id or timestamp based approaches. By encoding the value of all fields used in the ordering into the cursor (rather than just the first in DRF's implementation), we don't need offsets and the resulting code is much simpler - in particular when calculating the cursor for every item on the page.

I've always tried to make sure that any paginated set has a truly unique ordering - for most applications either id or created is sufficient. It also means you can do nice things like .order_by('-score', '-created') where score may not be unique enough to guarantee that you don't need large offsets.

Thoughts?

We need to ensure that only the last field is __lt or __gt, the previous ones should be __lte.

As an example, order on ('a', 'b'), page at 2.

Data:

 a | b 
 1 | 2
 1 | 3
 1 | 4
 2 | 1

"second" page will return only the last element.

e.g.

  File "/Users/adam/coding/photocrowd/api/cursor.py", line 15, in connection_from_cursor_paginated
    edge = edge_type(node=item, cursor=paginator.cursor(item))
  File "/Users/adam/.virtualenvs/photocrowd/lib/python2.7/site-packages/cursor_pagination.py", line 119, in cursor
    return self.encode_cursor(self.position_from_instance(instance))
  File "/Users/adam/.virtualenvs/photocrowd/lib/python2.7/site-packages/cursor_pagination.py", line 114, in position_from_instance
    attr = getattr(instance, order.lstrip('-'))
AttributeError: 'Submission' object has no attribute 'judge_feedback__placing'

line 114 in cursor_paginator.py

Solution:
Split ordering on __ and recursively getattr

In this project at the moment I have not used the offset approach which used by Django rest framework and inspired by David Cramer's blog post. The main reason for this is that it doesn't provide "globally unique" cursors. In particular you cannot guarantee that before=FIRST_ELEMENT_CURSOR will give the correct data set, as the offset of the first element cannot be calculated. This edge case could possibly be covered by a second query checking for uniqueness on that value, although this would be overkill in most situations.

From a bit of research, Disqus is the only implementation of cursor pagination "in the wild" which makes use of offsets. Both twitter and facebook use id or timestamp based approaches. By encoding the value of all fields used in the ordering into the cursor (rather than just the first in DRF's implementation), we don't need offsets and the resulting code is much simpler - in particular when calculating the cursor for every item on the page.

I've always tried to make sure that any paginated set has a truly unique ordering - for most applications either id or created is sufficient. It also means you can do nice things like .order_by('-score', '-created') where score may not be unique enough to guarantee that you don't need large offsets.

Thoughts?

Hello,
This package is broken on Django version 3.1.7. Which is at least the latest stable release of Django.

Example code:

from cursor_pagination import CursorPaginator

Result:

"ImportError: cannot import name 'six' from 'django.utils'"

Expected Result:

CursorPaginator imported successfully.

In this project at the moment I have not used the offset approach which used by Django rest framework and inspired by David Cramer's blog post. The main reason for this is that it doesn't provide "globally unique" cursors. In particular you cannot guarantee that before=FIRST_ELEMENT_CURSOR will give the correct data set, as the offset of the first element cannot be calculated. This edge case could possibly be covered by a second query checking for uniqueness on that value, although this would be overkill in most situations.

From a bit of research, Disqus is the only implementation of cursor pagination "in the wild" which makes use of offsets. Both twitter and facebook use id or timestamp based approaches. By encoding the value of all fields used in the ordering into the cursor (rather than just the first in DRF's implementation), we don't need offsets and the resulting code is much simpler - in particular when calculating the cursor for every item on the page.

I've always tried to make sure that any paginated set has a truly unique ordering - for most applications either id or created is sufficient. It also means you can do nice things like .order_by('-score', '-created') where score may not be unique enough to guarantee that you don't need large offsets.

Thoughts?

Based on our conversation about this, I think your simpler implementation may well be a better approach to cursor pagination that the default provided by REST framework. It'd certainly be valuable to have this scheme easily available to REST framework users.

Any thoughts on if you'd consider a REST framework pagination class for this package?