Unexpected connection close during series of requests

Original discussion start: https://github.com/encode/httpx/issues/96#issuecomment-661923097

Moving it here, since it occurred to be a uvicorn issue.

Due to some issue, client making 2 subsequent requests gets an unexpected disconnect from uvicorn:

DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1:9093
send: b'POST /ep1/ HTTP/1.1\r\nHost: 127.0.0.1:9093\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Type: application/json\r\nContent-Length: 214\r\n\r\n'
send: b'some data'
reply: 'HTTP/1.1 201 Created\r\n'
header: date: Fri, 31 Jul 2020 13:53:22 GMT
header: server: uvicorn
header: content-length: 835
header: content-type: application/json
header: x-correlation-id: 23856694-cc1e-46a0-a9dc-b735eaa1ac9c
DEBUG:urllib3.connectionpool:http://127.0.0.1:9093 "POST /ep1/ HTTP/1.1" 201 835

send: b'PATCH /ep2/ HTTP/1.1\r\nHost: 127.0.0.1:9093\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 122\r\nContent-Type: application/json\r\n\r\n'
send: b'some other data'
reply: ''
Traceback (most recent call last):
  File ".../lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File ".../lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File ".../lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1332, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File ".../lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File ".../lib/python3.8/site-packages/urllib3/util/retry.py", line 403, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File ".../lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File ".../lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File ".../lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File ".../lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1332, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test-with-requests.py", line 39, in <module>
    resp = client.patch(base_url + '/ep2/',
  File ".../lib/python3.8/site-packages/requests/sessions.py", line 602, in patch
    return self.request('PATCH', url, data=data, **kwargs)
  File ".../lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File ".../lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File ".../lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

What I’ve found out so far:

  • this is definitely a uvicorn issue, running service under hypercorn is a workaround
  • this is related to Starlette’ background tasks: ep1 executes a heavy db query, then schedules another one in background task

Currently working on a testcase

So it looks like a Starlette bug: https://github.com/encode/starlette/issues/919

Disabling middleware also fixes the issue.