Cookies not loaded in response.cookies but present in response.headers

[chemelli74]

Describe the bug

While working on a library swap from httpx to aiohttp, investigating why the latest was not working, I found out that cookies are not loaded.

Using the following code:

resp = await self.session.request(POST, url)         
        
_LOGGER.warning("Received cookies: %s", resp.cookies)
_LOGGER.warning("Received headers: %s", resp.headers)

I got the following outputs:

Cookies

2025-05-31 12:17:19.298 WARNING (MainThread) [aioamazondevices] Received cookies: Set-Cookie: UserPref=-; Domain=www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT; Path=/
Set-Cookie: ap-fid=""; Domain=amazon.it; expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=/ap/; Secure
Set-Cookie: at-acbit=-; Domain=www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT; Path=/
Set-Cookie: lc-acbit=-; Domain=www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT; Path=/
Set-Cookie: sess-at-acbit=-; Domain=www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT; Path=/
Set-Cookie: session-id=-; Domain=www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT; Path=/; Secure
Set-Cookie: session-id-time=-; Domain=www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT; Path=/; Secure
Set-Cookie: session-token=-; Domain=www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT; Path=/
Set-Cookie: ubid-acbit=-; Domain=www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT; Path=/
Set-Cookie: x-acbit=-; Domain=www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT; Path=/
Set-Cookie: x-wl-uid=-; Domain=www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT; Path=/

Headers (truncated for reporting)

2025-05-31 12:17:19.299 WARNING (MainThread) [aioamazondevices] Received headers <CIMultiDictProxy(
'Content-Type': 'text/html;charset=UTF-8', 
'Transfer-Encoding': 'chunked', 
'Connection': 'keep-alive', 
'Server': 'Server', 
'Date': 'Sat, 31 May 2025 12:17:19 GMT', 
'Set-Cookie': 'ap-fid=""; Domain=.amazon.it; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=/ap/; Secure', 
'Set-Cookie': 'session-id=260-2135165-7905862; Domain=.amazon.it; Expires=Sun, 31-May-2026 12:17:19 GMT; Path=/; Secure', 
'Set-Cookie': 'session-id-time=2379413839l; Domain=.amazon.it; Expires=Sun, 31-May-2026 12:17:19 GMT; Path=/; Secure', 
'Set-Cookie': 'session-id=-; path=/; domain=.www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT', 
'Set-Cookie': 'session-id-time=-; path=/; domain=.www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT', 
'Set-Cookie': 'session-token=-; path=/; domain=.www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT', 
'Set-Cookie': 'ubid-acbit=-; path=/; domain=.www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT', 
'Set-Cookie': 'at-acbit=-; path=/; domain=.www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT', 
'Set-Cookie': 'lc-acbit=-; path=/; domain=.www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT', 
'Set-Cookie': 'x-acbit=-; path=/; domain=.www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT', 
'Set-Cookie': 'x-wl-uid=-; path=/; domain=.www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT', 
'Set-Cookie': 'sess-at-acbit=-; path=/; domain=.www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT', 
'Set-Cookie': 'UserPref=-; path=/; domain=.www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT', 
'X-XSS-Protection': '1', 
[...]

Focusing on session-id, in the cookies (1st one) there is:

Set-Cookie: session-id=-; Domain=www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT; Path=/; Secure

while in the headers (2nd one):

'Set-Cookie': 'session-id=-; path=/; domain=.www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT', 
'Set-Cookie': 'session-id=260-2135165-7905862; Domain=.amazon.it; Expires=Sun, 31-May-2026 12:17:19 GMT; Path=/; Secure',

To Reproduce

"""Test script."""

import asyncio
import base64
import secrets

import orjson
from aiohttp import ClientSession

# Amazon APP info
AMAZON_APP_BUNDLE_ID = "com.amazon.echo"
AMAZON_APP_ID = "MAPiOSLib/6.0/ToHideRetailLink"
AMAZON_APP_NAME = "AioAmazonDevices"
AMAZON_APP_VERSION = "2.2.556530.0"
AMAZON_DEVICE_SOFTWARE_VERSION = "35602678"
AMAZON_DEVICE_TYPE = "A2IVLV5VM2W81"
AMAZON_CLIENT_OS = "16.6"

DEFAULT_HEADERS = {
    "User-Agent": (
        f"Mozilla/5.0 (iPhone; CPU iPhone OS {AMAZON_CLIENT_OS.replace('.', '_')} like Mac OS X) "  # noqa: E501
        "AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148"
    ),
    "Accept-Language": "en-US",
    "Accept-Encoding": "gzip",
    "Connection": "keep-alive",
}
URL = "https://www.amazon.it/ap/signin"


async def main() -> None:
    """Start script."""
    session = ClientSession(
        headers=DEFAULT_HEADERS,
        cookies=await _build_init_cookies(),
    )
    resp = await session.request("POST", URL)

    print("Cookies:")
    print(resp.cookies)
    print("--------")
    print("Headers:")
    print(resp.headers)


async def _build_init_cookies() -> dict[str, str]:
    """Build initial cookies to prevent captcha in most cases."""
    token_bytes = secrets.token_bytes(313)
    frc = base64.b64encode(token_bytes).decode("ascii").rstrip("=")

    map_md_dict = {
        "device_user_dictionary": [],
        "device_registration_data": {
            "software_version": AMAZON_DEVICE_SOFTWARE_VERSION,
        },
        "app_identifier": {
            "app_version": AMAZON_APP_VERSION,
            "bundle_id": AMAZON_APP_BUNDLE_ID,
        },
    }
    map_md_str = orjson.dumps(map_md_dict).decode("utf-8")
    map_md = base64.b64encode(map_md_str.encode()).decode().rstrip("=")

    return {"amzn-app-id": AMAZON_APP_ID, "frc": frc, "map-md": map_md}


if __name__ == "__main__":
    asyncio.run(main())

Expected behavior

resp.cookies should have all the cookies set by the server

Logs/tracebacks

N/A

Python Version

Python 3.13.3

aiohttp Version

Name: aiohttp
Version: 3.12.6
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: 
Author-email: 
License: Apache-2.0
Location: /workspaces/aioamazondevices/.venv/lib/python3.13/site-packages
Requires: aiohappyeyeballs, aiosignal, attrs, frozenlist, multidict, propcache, yarl
Required-by: aioamazondevices

multidict Version

Name: multidict
Version: 6.4.4
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /workspaces/aioamazondevices/.venv/lib/python3.13/site-packages
Requires: 
Required-by: aiohttp, yarl

propcache Version

Name: propcache
Version: 0.3.1
Summary: Accelerated property cache
Home-page: https://github.com/aio-libs/propcache
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache-2.0
Location: /workspaces/aioamazondevices/.venv/lib/python3.13/site-packages
Requires: 
Required-by: aiohttp, yarl

yarl Version

Name: yarl
Version: 1.20.0
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache-2.0
Location: /workspaces/aioamazondevices/.venv/lib/python3.13/site-packages
Requires: idna, multidict, propcache
Required-by: aioamazondevices, aiohttp

OS

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Related component

Client

Additional context

No response

Code of Conduct

• I agree to follow the aio-libs Code of Conduct

[bdraco]

more clear reproducer

"""Reproducer for aiohttp issue #11105: cookies not being loaded into resp.cookies.

This script demonstrates that cookies sent by the server via Set-Cookie headers
are not properly populated in the response.cookies attribute, even though they
are present in the response headers.
"""

import asyncio
import base64
import secrets

import orjson
from aiohttp import ClientSession

# Amazon APP configuration (needed to avoid captcha)
AMAZON_APP_BUNDLE_ID = "com.amazon.echo"
AMAZON_APP_ID = "MAPiOSLib/6.0/ToHideRetailLink"
AMAZON_APP_VERSION = "2.2.556530.0"
AMAZON_DEVICE_SOFTWARE_VERSION = "35602678"
AMAZON_CLIENT_OS = "16.6"

DEFAULT_HEADERS = {
    "User-Agent": (
        f"Mozilla/5.0 (iPhone; CPU iPhone OS {AMAZON_CLIENT_OS.replace('.', '_')} like Mac OS X) "
        "AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148"
    ),
    "Accept-Language": "en-US",
    "Accept-Encoding": "gzip",
    "Connection": "keep-alive",
}

URL = "https://www.amazon.it/ap/signin"


async def main() -> None:
    """Demonstrate the cookie loading issue."""
    # Create session with initial cookies
    session = ClientSession(
        headers=DEFAULT_HEADERS,
        cookies=await _build_init_cookies(),
    )
    
    async with session:
        # Make request
        async with await session.request("POST", URL) as resp:
            # Extract Set-Cookie headers
            set_cookie_headers = [
                (k, v) for k, v in resp.headers.items() 
                if k.lower() == "set-cookie"
            ]
            
            print("=== ISSUE #11105: Cookies not loaded into resp.cookies ===\n")
            
            print(f"Response status: {resp.status}")
            print(f"\nNumber of Set-Cookie headers: {len(set_cookie_headers)}")
            
            if set_cookie_headers:
                print("\nSet-Cookie headers found:")
                for i, (_, cookie) in enumerate(set_cookie_headers, 1):
                    # Show just the cookie name for clarity
                    cookie_name = cookie.split("=")[0] if "=" in cookie else cookie
                    print(f"  {i}. {cookie_name}...")
            
            print(f"\nNumber of cookies in resp.cookies: {len(resp.cookies)}")
            
            if resp.cookies:
                print("\nCookies in resp.cookies:")
                for cookie_name in resp.cookies:
                    print(f"  - {cookie_name}")
            else:
                print("\nBUG: resp.cookies is empty despite Set-Cookie headers!")
            
            print("\n=== Expected behavior: resp.cookies should contain all cookies from Set-Cookie headers ===")


async def _build_init_cookies() -> dict[str, str]:
    """Build initial cookies to prevent captcha."""
    # Generate random token
    token_bytes = secrets.token_bytes(313)
    frc = base64.b64encode(token_bytes).decode("ascii").rstrip("=")

    # Build device metadata
    map_md_dict = {
        "device_user_dictionary": [],
        "device_registration_data": {
            "software_version": AMAZON_DEVICE_SOFTWARE_VERSION,
        },
        "app_identifier": {
            "app_version": AMAZON_APP_VERSION,
            "bundle_id": AMAZON_APP_BUNDLE_ID,
        },
    }
    map_md_str = orjson.dumps(map_md_dict).decode("utf-8")
    map_md = base64.b64encode(map_md_str.encode()).decode().rstrip("=")

    return {"amzn-app-id": AMAZON_APP_ID, "frc": frc, "map-md": map_md}


if __name__ == "__main__":
    asyncio.run(main())

=== ISSUE #11105: Cookies not loaded into resp.cookies ===

Response status: 404

Number of Set-Cookie headers: 12

Set-Cookie headers found:
  1. session-id...
  2. session-id-time...
  3. session-id...
  4. session-id-time...
  5. session-token...
  6. ubid-acbit...
  7. at-acbit...
  8. lc-acbit...
  9. x-acbit...
  10. x-wl-uid...
  11. sess-at-acbit...
  12. UserPref...

Number of cookies in resp.cookies: 10

Cookies in resp.cookies:
  - session-id
  - session-id-time
  - session-token
  - ubid-acbit
  - at-acbit
  - lc-acbit
  - x-acbit
  - x-wl-uid
  - sess-at-acbit
  - UserPref

[bdraco]

one is Secure and one is not

[bdraco]

domains are different as well

'Set-Cookie': 'session-id=260-2135165-7905862; Domain=.amazon.it; Expires=Sun, 31-May-2026 12:17:19 GMT; Path=/; Secure', 
'Set-Cookie': 'session-id-time=2379413839l; Domain=.amazon.it; Expires=Sun, 31-May-2026 12:17:19 GMT; Path=/; Secure', 
'Set-Cookie': 'session-id=-; path=/; domain=.www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT', 
'Set-Cookie': 'session-id-time=-; path=/; domain=.www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT',

[bdraco]

the problem is we are storing cookies in SimpleCookie which doesn't support multiple cookies with the same name

[bdraco]

class MultiDomainCookies(SimpleCookie):
    """Cookie container that preserves all cookies, even with duplicate names.

    This is needed because servers may send multiple cookies with the same name
    but different domains or paths. SimpleCookie only keeps the last one.
    """

    def __init__(self) -> None:
        super().__init__()
        # Store all cookies as a list of (name, morsel) tuples
        self._all_cookies: List[Tuple[str, Morsel]] = []

    def __setitem__(self, key: str, morsel: Any) -> None:
        """Override to store all cookies, not just the last one with each name."""
        if not isinstance(morsel, Morsel):
            # Let parent class handle conversion to Morsel
            tmp = SimpleCookie()
            tmp[key] = morsel
            morsel = tmp[key]

        # Always add to our list
        self._all_cookies.append((key, morsel))

        # Also update the parent dict (this will overwrite duplicates)
        # This maintains compatibility with code expecting dict-like behavior
        super().__setitem__(key, morsel)

    def load(self, rawdata: str) -> None:
        """Override load to use our custom __setitem__."""
        # Create a temporary SimpleCookie to parse
        tmp = SimpleCookie()
        tmp.load(rawdata)
        # Add each cookie through our __setitem__
        for key, morsel in tmp.items():
            self[key] = morsel

    def items(self) -> List[Tuple[str, Morsel]]:
        """Return all cookies, including duplicates."""
        # Return our complete list instead of just the dict items
        return self._all_cookies.copy()

    def __len__(self) -> int:
        """Return the total number of cookies, including duplicates."""
        return len(self._all_cookies)

We could wrap SimpleCookie with something like this to return all the cookies... but I don't think that is what you want. I think the problem is that Secure cookies are not taking precedence of non-Secure cookies when its a secure domain

[bdraco]

I think the solution you want is to update cookie parsing to prefer Secure cookies over non-secure in HTTPS

[bdraco]

According to RFC 6265, multiple cookies with the same name, domain, and path, the most recently received cookie overwrites the previous one in the cookie store.

but the domain is different
one is www, one is not

'Set-Cookie': 'session-id=260-2135165-7905862; Domain=.amazon.it; Expires=Sun, 31-May-2026 12:17:19 GMT; Path=/; Secure', 
'Set-Cookie': 'session-id-time=2379413839l; Domain=.amazon.it; Expires=Sun, 31-May-2026 12:17:19 GMT; Path=/; Secure', 
'Set-Cookie': 'session-id=-; path=/; domain=.www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT', 
'Set-Cookie': 'session-id-time=-; path=/; domain=.www.amazon.it; expires=Fri, 31-May-2013 12:17:19 GMT', 

[bdraco]

https://datatracker.ietf.org/doc/html/rfc6265?#section-5.3

[bdraco]

11. If the cookie store contains a cookie with the same name,
    domain, and path as the newly created cookie:

    1. Let old-cookie be the existing cookie with the same name,
       domain, and path as the newly created cookie. (Notice that
       this algorithm maintains the invariant that there is at most
       one such cookie.)

    2. If the newly created cookie was received from a "non-HTTP"
       API and the old-cookie's http-only-flag is set, abort these
       steps and ignore the newly created cookie entirely.

    3. Update the creation-time of the newly created cookie to
       match the creation-time of the old-cookie.

    4. Remove the old-cookie from the cookie store.

[bdraco]

The RFC shows that cookies should be stored uniquely by (name, domain, path), not just by name.

[bdraco]

SimpleCookie just does it by name so its wrong

[bdraco]

CookieJar works correctly but we already loose the duplicates by the time it gets there

[bdraco]

SimpleCookie loses cookies with same name but different domain/path