Let’s add a check case first to confirm the problem with a number of slashes and authentication.
Let’s examine the present check file first to see the place we should always add our check.
Studying checks/test_requests.py
Let’s add our check case to the TestRequests class. First, let’s discover a good location close to different URL-related checks.
Trying to find test_path
Excellent, let’s add our check case proper after the test_path_is_not_double_encoded check.
IDG
Now let’s modify the prepare_url technique in fashions.py to protect a number of slashes. The difficulty is probably going in how the trail is dealt with earlier than being handed to urlunparse.
IDG
Be aware {that a} null is a suspect alternative of marker.
Solver:
Let’s run the checks to confirm our modifications repair the problem. We’ll use pytest to run simply our new check case.
$ python -m pytest checks/test_requests.py -k test_path_with_multiple_slashes -v
============================= check session begins ==============================
platform linux -- Python 3.9.20, pytest-8.3.3, pluggy-1.5.0 -- /usr/native/bin/python
cachedir: .pytest_cache
rootdir: /repo
configfile: pyproject.toml
plugins: cov-5.0.0, httpbin-2.1.0
amassing ... collected 330 gadgets / 329 deselected / 1 chosen
checks/test_requests.py::TestRequests::test_path_with_multiple_slashes FAILED [100%]
=================================== FAILURES ===================================
_________________ TestRequests.test_path_with_multiple_slashes _________________
self =
def test_path_with_multiple_slashes(self):
"""Take a look at that URLs with a number of slashes are preserved."""
url = "http://instance.com//path//to//useful resource"
request = requests.Request("GET", url).put together()
> assert request.url == url
E AssertionError: assert 'http://examp...topercent00resource' == 'http://examp.../to//useful resource'
E
E - http://instance.com//path//to//useful resource
E ? ^ ^^ ^^
E + http://instance.com/%00pathpercent00topercent00resource
E ? ^^^ ^^^ ^^^
checks/test_requests.py:151: AssertionError
=========================== quick check abstract data ============================
FAILED checks/test_requests.py::TestRequests::test_path_with_multiple_slashes
====================== 1 failed, 329 deselected in 0.75s =======================
Exit Code: 1
The check failure reveals that our null byte alternative strategy isn’t working accurately – the null bytes are getting URL encoded. Let’s attempt a distinct strategy utilizing a extra URL-safe short-term marker.
IDG
We have to add the re import on the high of the file.
Let’s examine the present imports first.
Studying src/requests/fashions.py
I see we are able to add re to the present imports. Let’s add it with the opposite commonplace library imports on the high.
IDG
Now let’s run the check once more to see if our new strategy works.
$ python -m pytest checks/test_requests.py -k test_path_with_multiple_slashes -v
============================= check session begins ==============================
platform linux -- Python 3.9.20, pytest-8.3.3, pluggy-1.5.0 -- /usr/native/bin/python
cachedir: .pytest_cache
rootdir: /repo
configfile: pyproject.toml
plugins: httpbin-2.1.0, cov-5.0.0
amassing ... collected 330 gadgets / 329 deselected / 1 chosen
checks/test_requests.py::TestRequests::test_path_with_multiple_slashes FAILED [100%]
=================================== FAILURES ===================================
_________________ TestRequests.test_path_with_multiple_slashes _________________
self =
def test_path_with_multiple_slashes(self):
"""Take a look at that URLs with a number of slashes are preserved."""
url = "http://instance.com//path//to//useful resource"
> request = requests.Request("GET", url).put together()
checks/test_requests.py:150:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/requests/fashions.py:299: in put together
p.put together(
src/requests/fashions.py:368: in put together
self.prepare_url(url, params)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = , url="http://instance.com/path/to/useful resource"
params = {}
def prepare_url(self, url, params):
"""Prepares the given HTTP URL."""
#: Settle for objects which have string representations.
#: We're unable to blindly name unicode/str capabilities
#: as this may embrace the bytestring indicator (b'')
#: on python 3.x.
#: https://github.com/psf/requests/pull/2238
if isinstance(url, bytes):
url = url.decode("utf8")
else:
url = str(url)
# Take away main whitespaces from url
url = url.lstrip()
# Do not do any URL preparation for non-HTTP schemes like `mailto`,
# `knowledge` and many others to work round exceptions from `url_parse`, which
# handles RFC 3986 solely.
if ":" in url and never url.decrease().startswith("http"):
self.url = url
return
# Assist for unicode domains and paths.
attempt:
scheme, auth, host, port, path, question, fragment = parse_url(url)
besides LocationParseError as e:
elevate InvalidURL(*e.args)
if not scheme:
elevate MissingSchema(
f"Invalid URL {url!r}: No scheme provided. "
f"Maybe you meant https://{url}?"
)
if not host:
elevate InvalidURL(f"Invalid URL {url!r}: No host provided")
# Typically, we wish to attempt IDNA encoding the hostname if the string comprises
# non-ASCII characters. This enables customers to mechanically get the proper IDNA
# behaviour. For strings containing solely ASCII characters, we have to additionally confirm
# it does not begin with a wildcard (*), earlier than permitting the unencoded hostname.
if not unicode_is_ascii(host):
attempt:
host = self._get_idna_encoded_host(host)
besides UnicodeError:
elevate InvalidURL("URL has an invalid label.")
elif host.startswith(("*", ".")):
elevate InvalidURL("URL has an invalid label.")
# Rigorously reconstruct the community location
netloc = auth or ""
if netloc:
netloc += "@"
netloc += host
if port:
netloc += f":{port}"
# Naked domains aren't legitimate URLs.
if not path:
path = "/"
else:
# Protect a number of consecutive slashes within the path
original_path = path
path = re.sub("//+", "/", path) # Normalize to single slashes for parsing
if isinstance(params, (str, bytes)):
params = to_native_string(params)
enc_params = self._encode_params(params)
if enc_params:
if question:
question = f"{question}&{enc_params}"
else:
question = enc_params
url = requote_uri(urlunparse([scheme, netloc, path, None, query, fragment]))
# Restore a number of slashes in the event that they have been current within the authentic path
if path != "/" and "original_path" in locals():
# Discover the trail portion within the last URL and exchange it
> url_parts = listing(urlparse(url))
E NameError: title 'urlparse' is just not outlined
src/requests/fashions.py:490: NameError
=========================== quick check abstract data ============================
FAILED checks/test_requests.py::TestRequests::test_path_with_multiple_slashes
====================== 1 failed, 329 deselected in 0.55s =======================
Exit Code: 1
Ah, we have to use the imported urlparse. Let’s repair the imports and check out an easier strategy.
IDG
Let’s run the check once more to see if this easier strategy works.
The check passes. The easier answer works as a result of we’re letting the URL elements cross via with none normalization. Let’s submit our modifications which embrace:
Added a check case to confirm a number of slashes are preserved in URLs with and with out authentication
Modified prepare_url to protect the precise path construction with out normalization
Added a remark explaining the preservation of a number of slashes
Solver completed
Dialogue with a founder
Someplace in the midst of the session proven within the earlier part, you’ll see that I pressed the proceed fixing button and informed Solver, “You’re heading in the right direction; please proceed.”
I puzzled why Solver had stopped right here to attend for consumer enter, so I despatched in a problem report and requested, “Why does Solver cease after figuring out the issue? Sure, it did proceed after I pressed the suitable button. It is a query, not a bug report.”
