PT-2024-40346 · Scrapy · Scrapy

Published 2024-05-14 · Updated 2024-05-14

Positive Technologies (AI Summary)

CVSS v3.1

4.3

Medium

Base vector

AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:N/A:N

Name of the Vulnerable Software and Affected Versions Scrapy versions prior to 2.11.2

Description The issue arises when using system proxy settings that are scheme-specific, i.e., specific to http:// or https:// URLs. During redirects, Scrapy fails to account for scheme changes, resulting in the continued use of the original proxy configuration instead of switching to the appropriate one for the new scheme. This poses a security risk if different proxy configurations are used for HTTP and HTTPS for security reasons, such as preventing one proxy provider from being aware of the URLs visited with the other.

Recommendations Upgrade to Scrapy 2.11.2. As a temporary workaround, consider replacing the built-in retry middlewares (RedirectMiddleware and MetaRefreshMiddleware) and the HttpProxyMiddleware middleware with custom ones that implement the fix from Scrapy 2.11.2, and verify that they work as intended.

Fix

Related Identifiers

GHSA-JM3V-QXMH-HXWV

Affected Products

Scrapy

References · 5

• https://osv.dev/vulnerability/GHSA-jm3v-qxmh-hxwv · Vendor Advisory
• https://github.com/scrapy/scrapy · Note
• https://github.com/scrapy/scrapy/commit/1d0502f25bbe55a22899af915623fda1aaeb9dd8 · Note
• https://github.com/scrapy/scrapy/security/advisories/GHSA-jm3v-qxmh-hxwv · Note
• https://github.com/scrapy/scrapy/issues/767 · Note