Cursor on cluster - ERROR 1600 about python-arango HOT 11 CLOSED

Hi @Thijxx,

Thanks for reporting the issue. Unfortunately, python-arango is largely untested with clustered setup.
Can you provide me with the full stacktrace? Thanks.

Best,
Joohwan

from python-arango.

Per request by Kunal from Arango:
My cluster setup is de the default with 3 nodes. I started the first with 'arangodb' and then the other two with 'arango -join '. I took this from https://github.com/arangodb-helper/arangodb

I am not sure what the default cluster config is exactly. The collections I use are all created with 9 shards as advised in the documentation.

The requests I do with the cursor are ACID transactions with many statements, thousands, per transaction.

Trace (@joowani is this enough?)

/Users/tvanulden/VE_3.6/bin/python /Users/tvanulden/repos/inergy/inergy/dataloader/event_handlers/eventHandler_ypsilon.py
Traceback (most recent call last):
  File "/Users/tvanulden/repos/inergy/inergy/dataloader/event_handlers/eventHandler_ypsilon.py", line 118, in <module>
    doc = cursor.next()
  File "/Users/tvanulden/VE_3.6/lib/python3.6/site-packages/arango/cursor.py", line 124, in next
    raise CursorNextError(res)
arango.exceptions.CursorNextError: [HTTP 404][ERR 1600] cursor not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/tvanulden/repos/inergy/inergy/dataloader/event_handlers/eventHandler_ypsilon.py", line 179, in <module>
    print(sys.exc_info()[3])
IndexError: tuple index out of range

Process finished with exit code 1

from python-arango.

Thijs:
As covered in the documentation, ArangoDB currently supports single document level ACID operations in cluster mode of operation with sharding and replication. (multi-document, multi-collection etc. are not ACID in cluster mode).

Once a request is sent to a co-ordinator, the co-ordinator fetches the results in a cursor from the DBServer(s) and is served back as response .

However, the issue you see above could be Python specific when operating in cluster mode. I would let @joowani comment on that.

from python-arango.

@kunalak @Thijxx

Hmm the ArangoDB's HTTP REST API documentation does not seem to say anything different for cluster mode, so I'm not sure why this would fail. I will have to set up a cluster myself and inspect the response body coming from ArangoDB. Could you let me know what versions of ArangoDB and python-arango you are using? Thanks for your patience.

from python-arango.

It looks like the problem persists even with a single machine and the latest version of Arango (3.2.8). Your code is at 3.8.0 and I will upgrade it to 3.12.1 now to see if it changes anything.

[edit]
New version does not change anything.

from python-arango.

@Thijxx

Hmm this is strange. I am unable to reproduce the issue. So you are saying the issue is present even without the cluster mode at all (i.e. NOT using the arangodb helper command)?

I need more information:

1. Are you doing any sort of multi-threading/multi-processing in your code? Accessing the cursor object in parallel could lead to this issue.

2. Below is the script I ran while trying to reproduce your problem. Could you please try running something similar and see if you still get the same error?

from arango import ArangoClient

client = ArangoClient()  # Put in your server info here
db = client.db('_system')

if 'test_collection_1' not in [col['name'] for col in  db.collections()]:
    col = db.create_collection('test_collection_1')
else:
    db.delete_collection('test_collection_1')
    col = db.create_collection('test_collection_1')

col.insert_many([
    {'type': 'nieuw-inergy', 'value': 1}, 
    {'type': 'nieuw-inergy', 'value': 2},
    {'type': 'invalid', 'value': 3}, 
    {'value': 4}
])
cursor = db.aql.execute(
    'FOR doc IN test_collection_1 FILTER doc.type == @type RETURN doc', 
    bind_vars={'type': 'nieuw-inergy'}, 
    batch_size=1, 
    count=True
)

while cursor.has_more():
    doc = cursor.next()
    print(doc)

3. If above does not produce any errors, can you provide me an example set of test documents that can?

from python-arango.

1. Just serial processing
2. Your code runs fine for me.
3. My documents are large and complex but I cannot share them for legal reasons.

I notice that it is not a stable problem, sometimes it fails half way and sometimes it fails right at the start.
This morning I rewrote a copy of my code to run on Mongo and no problems there, I wanted to exclude the possibility I did wrong stuff in my Python code.

from python-arango.

I doubt this is a problem with your application code. This could be something in ArangoDB itself, so you might want to raise a github issue with them directly. You can also try turning on extra logging for more information on the HTTP requests (see here).

from python-arango.

from python-arango.

@Thijxx Please confirm if above helped solved the issues (and if there are any new ones).
We can close this issue if not relevant to Python anymore. Let us know.

from python-arango.

Yes, this error is produced by a time-out on the cursor connection. Solution is higher TTL or better; code that does not use a single cursor for a very long time.

from python-arango.