Issue with Requests 2.11.0 about betamax_matchers HOT 26 CLOSED

Is it possible the request body doesn't have any JSON in it? or something other than JSON?

from betamax_matchers.

Also, if you downgrade to 2.10.0, does it work again?

from betamax_matchers.

It passes when downgrading to 2.10.0. I'll see about verifying what's in the resposne body. I assume it is valid json as this is the test code and passing on 2.10.0.

response = session.request('PATCH', '/api/v1/me/prefs', json=json)
self.assertEqual('ja', response['lang'])
self.assertEqual(123, response['num_comments'])

from betamax_matchers.

Response:

$ base64 -D response.b64 | gunzip | python -m json.tool
{
    "default_theme_sr": null,
    "threaded_messages": true,
    "hide_downs": false,
    "show_stylesheets": true,
    "show_link_flair": true,
    "creddit_autorenew": false,
    "show_trending": true,
    "private_feeds": true,
    "monitor_mentions": true,
    "show_snoovatar": false,
    "research": false,
    "ignore_suggested_sort": false,
    "num_comments": 123,
    "clickgadget": true,
    "use_global_defaults": false,
    "label_nsfw": true,
    "affiliate_links": true,
    "over_18": true,
    "email_messages": false,
    "live_orangereds": true,
    "highlight_controversial": false,
    "no_profanity": true,
    "domain_details": false,
    "collapse_left_bar": true,
    "lang": "ja",
    "hide_ups": false,
    "public_server_seconds": false,
    "allow_clicktracking": true,
    "hide_from_robots": false,
    "compress": false,
    "store_visits": false,
    "threaded_modmail": false,
    "min_link_score": -4,
    "media_preview": "subreddit",
    "enable_default_themes": false,
    "content_langs": "all",
    "show_promote": null,
    "min_comment_score": -4,
    "public_votes": false,
    "organic": true,
    "collapse_read_messages": false,
    "show_flair": true,
    "mark_messages_read": true,
    "force_https": false,
    "hide_ads": false,
    "beta": false,
    "newwindow": false,
    "numsites": 25,
    "legacy_search": false,
    "media": "subreddit",
    "show_gold_expiration": false,
    "highlight_new_comments": true,
    "default_comment_sort": "confidence",
    "hide_locationbar": false
}

from betamax_matchers.

Oh you asked about the request body:

The first:

      "request": {
        "body": {
          "encoding": "utf-8",
          "string": "grant_type=password&password=<PASSWORD>&username=<USERNAME>"
        },

The second:

      "request": {
        "body": {
          "encoding": "utf-8",
          "string": "{\"num_comments\": 123, \"lang\": \"ja\"}"
        },

The first request's response is:

$ base64 -D response.b64 | gunzip | python -m json.tool
{
    "access_token": "uaeycrw_jcPlzhvR7fKCLrvnHsA",
    "token_type": "bearer",
    "expires_in": 3600,
    "scope": "*"
}

I know the first request is not an issue, because similar requests occur for many of the other cassettes and those continue to work fine.

from betamax_matchers.

I haven't tried to reproduce this yet. Can you patch betamax_matchers locally (assuming you can reproduce this locally) and print out which request.body it is trying? Looking at your traceback (reproduced below)

Traceback (most recent call last):
  File "/home/travis/build/praw-dev/prawcore/prawcore/requestor.py", line 46, in request
    return self._http.request(*args, **kwargs)
  File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/requests/sessions.py", line 471, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/requests/sessions.py", line 581, in send
    r = adapter.send(request, **kwargs)
  File "/home/travis/build/praw-dev/prawcore/.eggs/betamax-0.7.2-py3.5.egg/betamax/adapter.py", line 123, in send
    interaction = current_cassette.find_match(request)
  File "/home/travis/build/praw-dev/prawcore/.eggs/betamax-0.7.2-py3.5.egg/betamax/cassette/cassette.py", line 133, in find_match
    if not interaction.match(curried_matchers):
  File "/home/travis/build/praw-dev/prawcore/.eggs/betamax-0.7.2-py3.5.egg/betamax/cassette/interaction.py", line 58, in match
    return all(m(request) for m in matchers)
  File "/home/travis/build/praw-dev/prawcore/.eggs/betamax-0.7.2-py3.5.egg/betamax/cassette/interaction.py", line 58, in <genexpr>
    return all(m(request) for m in matchers)
  File "/home/travis/build/praw-dev/prawcore/tests/config.py", line 45, in match
    JSONBodyMatcher().match(to_compare, recorded_request)
  File "/home/travis/build/praw-dev/prawcore/.eggs/betamax_matchers-0.3.0-py3.5.egg/betamax_matchers/json_body.py", line 26, in match
    request_json = json.loads(request.body) if request.body else None
  File "/opt/python/3.5.2/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/opt/python/3.5.2/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/python/3.5.2/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/travis/build/praw-dev/prawcore/tests/test_sessions.py", line 79, in test_request__patch
    response = session.request('PATCH', '/api/v1/me/prefs', json=json)
  File "/home/travis/build/praw-dev/prawcore/prawcore/sessions.py", line 119, in request
    params)
  File "/home/travis/build/praw-dev/prawcore/prawcore/sessions.py", line 62, in _request_with_retries
    json=json, params=params)
  File "/home/travis/build/praw-dev/prawcore/prawcore/rate_limit.py", line 28, in call
    response = request_function(*args, **kwargs)
  File "/home/travis/build/praw-dev/prawcore/prawcore/requestor.py", line 48, in request
    raise RequestException(exc, args, kwargs)
prawcore.exceptions.RequestException: error with request Expecting value: line 1 column 1 (char 0)

If you look closely:

  File "/home/travis/build/praw-dev/prawcore/.eggs/betamax_matchers-0.3.0-py3.5.egg/betamax_matchers/json_body.py", line 26, in match
    request_json = json.loads(request.body) if request.body else None

So it looks like L26 in betamax_matchers/json_body.py is where you want to put a print before.

from betamax_matchers.

Requests 2.11.0:

b'{"num_comments": 123, "lang": "ja"}'

Requests 2.10.0

{"num_comments": 123, "lang": "ja"}

from betamax_matchers.

Aha! I expect if you printed the type you'd get bytes and str for each one. Passing bytes to json.decode will not work. I should be able to fix this tonight.

from betamax_matchers.

Actually it's even stranger:

<class 'str'>
b'{"lang": "ja", "num_comments": 123}'

The b'{"lang": "ja", "num_comments": 123}' is the actual string. Here is the repr(response.body):

'b\'{"num_comments": 123, "lang": "ja"}\''

from betamax_matchers.

uhhh @Lukasa we might have a bug in requests 2.11.0

from betamax_matchers.

I suspect this is actually the result of the change to the way we pass JSON to urllib3. I am about to go to bed so I don't have time to look at this now, but that's where I'd start looking first.

from betamax_matchers.

Yeah, just giving you a head's up. Not looking for you to fix it :)

from betamax_matchers.

Just to add a little more context: the test passes when generating the cassette. It's only when replaying the response that this issue occurs.

from betamax_matchers.

@bboe Out of interest, how did you build your JSON object? Is the string in the "string" field built from json.dumps, by any chance?

from betamax_matchers.

@Lukasa no, json is being passed to request.

Here's the test code:
https://github.com/praw-dev/prawcore/blob/3df51fe65c18e26960e0c09ec6e77f03cd521f72/tests/test_sessions.py#L77

That calls this request:
https://github.com/praw-dev/prawcore/blob/3df51fe65c18e26960e0c09ec6e77f03cd521f72/prawcore/sessions.py#L90

Which in turn calls this request that finally passes the request to requests:
https://github.com/praw-dev/prawcore/blob/3df51fe65c18e26960e0c09ec6e77f03cd521f72/prawcore/requestor.py#L43

from betamax_matchers.

@bboe Ooooh, hang on. Does betamax cast bodies to string anywhere? I wonder if it's getting caught and is expecting the body to be a native string.

from betamax_matchers.

Check out coerce_content in https://github.com/sigmavirus24/betamax/blob/72a8839474a90ce210c361b761a75e70d094c0c2/betamax/util.py.

I'm curious why such changes would cause requests 2.11.0 to fail though where it worked before.

from betamax_matchers.

@bboe Well, we changed the type of the json data we send to urllib3 on Python 3 in this release. If betamax calls str or repr on that data directly, that would cause this behaviour I suspect.

from betamax_matchers.

So the request object that is received here should be completely unmodified from what Requests gives us.

It goes into the adapter and is used to find a matching request here. We then pass it unmodified to the matcher. That falls through to the JSONBodyMatcher.

We never call str on things like this. That's why coerce_content exists in this case. I don't see where this is coming out of praw either though.

from betamax_matchers.

Edit: ignore this comment. I neglected to parse the data field from httpbin.

from betamax_matchers.

@bboe I'm confused on how that code snippet lead you to that conclusion, could you elaborate?

from betamax_matchers.

@sigmavirus24 please see my edit.

from betamax_matchers.

I just downloaded prawcore and found the problem because I noticed the test in question doesn't use the JSONBodyMatcher directly. It's used in another matcher that does: to_compare.body = str(to_compare.body)

from betamax_matchers.

@bboe this only affects Requests 2.11.0 because (as @Lukasa mentioned) it didn't always coerce the body to bytes and on Python 2 str == bytes but on Python 3 it isn't the same relation. I'm going to close this since the fix for your tests exists solely in your test code in prawcore.

Cheers!

from betamax_matchers.

🤦 Thanks for finding that. Sorry for the inconvenience.

from betamax_matchers.

No worries. If I had found the time sooner to dig into your tests, I probably would have saved us all a lot of time and discussion =D

from betamax_matchers.