SSL error with 8.2.2 or later when talking to github

[DJDevon3]

CircuitPython version

boot_out.txt

Adafruit CircuitPython 8.2.3 on 2023-08-11; Adafruit MatrixPortal S3 with ESP32S3
Board ID:adafruit_matrixportal_s3

INFO_UF2.TXT

TinyUF2 Bootloader 0.14.0-5-g45bc2fc - tinyusb (0.15.0-331-ge3b3229d6)
Model: Adafruit MatrixPortal S3
Board-ID: ESP32-S3-MatrixPortal-revB
Date: May 18 2023

Code/REPL

# SPDX-FileCopyrightText: 2020 Brent Rubell for Adafruit Industries
#
# SPDX-License-Identifier: MIT

import os
import ipaddress
import ssl
import wifi
import socketpool
import adafruit_requests

# URLs to fetch from
TEXT_URL = "http://wifitest.adafruit.com/testwifi/index.html"
JSON_QUOTES_URL = "https://www.adafruit.com/api/quotes.php"
JSON_STARS_URL = "https://api.github.com/repos/adafruit/circuitpython"

print("ESP32-S2 WebClient Test")

print(f"My MAC address: {[hex(i) for i in wifi.radio.mac_address]}")

print("Available WiFi networks:")
for network in wifi.radio.start_scanning_networks():
    print("\t%s\t\tRSSI: %d\tChannel: %d" % (str(network.ssid, "utf-8"),
                                             network.rssi, network.channel))
wifi.radio.stop_scanning_networks()

print(f"Connecting to {os.getenv('WIFI_SSID')}")
wifi.radio.connect(os.getenv("WIFI_SSID"), os.getenv("WIFI_PASSWORD"))
print(f"Connected to {os.getenv('WIFI_SSID')}")
print(f"My IP address: {wifi.radio.ipv4_address}")

ping_ip = ipaddress.IPv4Address("8.8.8.8")
ping = wifi.radio.ping(ip=ping_ip) * 1000
if ping is not None:
    print(f"Ping google.com: {ping} ms")
else:
    ping = wifi.radio.ping(ip=ping_ip)
    print(f"Ping google.com: {ping} ms")

pool = socketpool.SocketPool(wifi.radio)
requests = adafruit_requests.Session(pool, ssl.create_default_context())

print(f"Fetching text from {TEXT_URL}")
response = requests.get(TEXT_URL)
print("-" * 40)
print(response.text)
print("-" * 40)

print(f"Fetching json from {JSON_QUOTES_URL}")
response = requests.get(JSON_QUOTES_URL)
print("-" * 40)
print(response.json())
print("-" * 40)

print()

print(f"Fetching and parsing json from {JSON_STARS_URL}")
response = requests.get(JSON_STARS_URL)
print("-" * 40)
print(f"CircuitPython GitHub Stars: {response.json()['stargazers_count']}")
print("-" * 40)

print("Done")

Behavior

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
ESP32-S2 WebClient Test
My MAC address: ['obfuscated']
Available WiFi networks:
	SlurpNet		RSSI: -57	Channel: 10
Connecting to SlurpNet
Connected to SlurpNet
My IP address: obfuscated
Ping google.com: 0.0 ms
Fetching text from http://wifitest.adafruit.com/testwifi/index.html
----------------------------------------
This is a test of Adafruit WiFi!
If you can read this, its working :)
----------------------------------------
Fetching json from https://www.adafruit.com/api/quotes.php
----------------------------------------
[{'text': 'I don’t care that they stole my idea — I care that they don’t have any of their own', 'author': 'Nikola Tesla'}]
----------------------------------------

Fetching and parsing json from https://api.github.com/repos/adafruit/circuitpython
Traceback (most recent call last):
  File "adafruit_requests.py", line 515, in _get_socket
OSError: Failed SSL handshake

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "code.py", line 58, in <module>
  File "adafruit_requests.py", line 711, in get
  File "adafruit_requests.py", line 650, in request
  File "adafruit_requests.py", line 496, in _get_socket
RuntimeError: Sending request failed

Code done running.

Press any key to enter the REPL. Use CTRL-D to reload.

Description

Using Internet Test code from MatrixPortal S3 learn guide.

• succeeds SSL handshake with Adafruit HTTPS quotes url
• fails SSL handshake with HTTPS Github API

I have no problem viewing https://api.github.com/repos/adafruit/circuitpython with my browser. Shows all the JSON data fine. For whatever reason Github is failing to handshake with the MatrixPortal S3.

Additional information

This is only to report that a portion of code in a learn guide does not function correctly. I'm honestly not sure why it's failing.

Using adafruit_requests.mpy from adafruit-circuitpython-bundle-8.x-mpy-20230815

Matrix Portal is not connected to any matrix panels at this time. Using the board as I would a feather S3 with wifi.

I can get it to ignore the error with try/except but that's not the point in the case of learn guide code.

[DJDevon3]

Removed adafruit gets and focused only on the Github connection. It's not connecting to that API url. Rest of the script seems to work fine. It's specifically the Github handshake attempt.

[DJDevon3]

Here's a slimmer version that will fail gracefully during testing.

import os
import time
import ipaddress
import ssl
import wifi
import socketpool
import adafruit_requests

pool = socketpool.SocketPool(wifi.radio)

JSON_STARS_URL = "https://api.github.com/repos/adafruit/circuitpython"

# Connect to Wi-Fi
print("\n====== MatrixPortal S3 WebClient Test ======")
print("Connecting to WiFi...")
requests = adafruit_requests.Session(pool, ssl.create_default_context())
while not wifi.radio.ipv4_address:
    try:
        wifi.radio.connect(os.getenv('WIFI_SSID'), os.getenv('WIFI_PASSWORD'))
    except ConnectionError as e:
        print("Connection Error:", e)
        print("Retrying in 10 seconds")
    time.sleep(10)
print("Connected!\n")

while True:
    try:
        response = requests.get(JSON_STARS_URL).json()
        stargazers_count = response['stargazers_count']
        print(f"CircuitPython GitHub Stars: {stargazers_count}")
    except (ValueError, RuntimeError, OSError) as e:
        print("Failed to get data, retrying\n", e)
        time.sleep(60)
        continue
    response = None

    print("Done")

With the OSError handler in the mix it spits out

====== MatrixPortal WebClient Test ======
Connecting to WiFi...
Connected!

Failed to get data, retrying
 Sending request failed
Failed to get data, retrying
 Sending request failed
Failed to get data, retrying
 Sending request failed
Failed to get data, retrying
 Sending request failed

With continue statement it never reaches the print("done") line, it will if pass is used instead.

The script will attempt to connect indefinitely so no sense waiting for it to connect. I have no idea why it isn't working with Github. Did something change with Github's API? It's not a key:value pair error, it's a handshake error, which out of my comfort zone to track down.

[DJDevon3]

Tried on an Adafruit ESP32-S2 Feather on 8.2.3 with the same result.

Reverted the same S2 back to CP 8.2.0 and it works.

Adafruit CircuitPython 8.2.0 on 2023-07-05; Adafruit Feather ESP32S2 with ESP32S2
Board ID:adafruit_feather_esp32s2

====== Circuit Python Internet Simple Test ======
Connecting to WiFi...
Connected!

CircuitPython GitHub Stars: 3634

• 8.2.0 works
  Something between 8.2.0 and 8.2.3 broke SSL handshake (for some things but not all things). Attempting to bisect by installing older UF2's from S3 bucket.

8.2.3 broken (SSL Handshake error)
8.2.2 broken (SSL Handshake error)

8.2.1 I might be getting rate limited or there's some other issue. I am at least getting a handshake and response with 8.2.1

Adafruit CircuitPython 8.2.1 on 2023-07-25; Adafruit Feather ESP32S2 with ESP32S2
Board ID:adafruit_feather_esp32s2

====== Circuit Python Internet Simple Test ======
Connecting to WiFi...
Connected!

{'message': "API rate limit exceeded for *IPADDRESS* (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)", 'documentation_url': 'https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting'}
Traceback (most recent call last):
  File "code.py", line 30, in <module>
KeyError: stargazers_count

Code done running.

Accidentally left print(response) on and 8.2.0 flooded with data and crashed my S2 and Mu. It returned so much data that I immediately got throttled with 8.2.1 but the server does respond with a nice error message. At least it's connecting.

After that I went back to 8.2.0 just to see if I am getting rate limited and it'll throw the same error on a known good working version. It's giving the same rate limit error on 8.2.0 and 8.2.1 so I think we can call 8.2.1 good too.

The problem is after 8.2.1 to 8.2.2. There are no beta releases between 8.2.1 to 8.2.2.
The issue begins in 8.2.2. Bisection complete. Up to a developer to look into it.

I believe this issue could potentially affect any project (at least on S2/S3 modules) using Adafruit_Requests starting in 8.2.2
The way to recognize this bug is traceback will report OSError: Failed SSL handshake

[dhalbert]

Here is a roots.pem that I generated a couple of weeks ago using the procedure described in https://github.com/arduino/nina-fw/blob/master/README.md#build-a-new-certificate-list-based-on-the-google-android-root-ca-list.

roots.pem.zip

When I substitute this one for the one in lib/certificates/data/roots.pem, talking to github works again.
But I was unable to come up with an Adafruit IO example that breaks on the old roots.pem but not on this one, so I haven't made a PR yet.

I think we should stop depending on nina-fw as a submodule, and just keep our own roots.pem, documenting how we built it. We could even write makefile steps to do that and verify it.

[RetiredWizard]

I've been building the Pico W with the latest main branch bits and this evening was getting the "Sending request failed" message with one of my test programs. After building with the roots.pem file Dan posted the test program is working again.

That being said, my test program was running fine when I built from main last night so I'm thinking the error only occurs when you hit a particular server somewhere in your path.

I'll keep using the new roots.pem and if I hit any failures with it I'll update here.

[jepler]

The change to nina-fw that we think introduced this problem was initially done in the 8.2.x branch. However, it probably started affecting main today when we merged #8317.

After nina-fw is updated, the fix in CircuitPython will probably initially be made in 8.2.x and merged into main subsequent to that, though it depends on how Dan and/or Scott want to handle it.

[RetiredWizard]

@DJDevon3 Just curious, did you try building the MatrixPortal S3 using the roots.pem file that Dan posted above?

[DJDevon3]

No, building circuit python isn't something I've done in a long time. :/ I'm not good with make builds.

[RetiredWizard]

This is using the latest bits from main so it has the 1.19.1 Micropython merge included.

MatrixPortal S3 UF2 with updated certificates:
firmware.zip

[DJDevon3]

Is there a 9.0 alpha bundle? I'm getting

Traceback (most recent call last):
  File "code.py", line 7, in <module>
ValueError: incompatible .mpy file

with 9.0.0-alpha.1-8-gaf91625fb-dirty

It does not like the 8.2.3 adafruit_requests.mpy I have in lib

edit:
Grabbed the latest adafruit_requests.py and that works. Apparently alpha builds aren't using mpy.

import os
import time
import ssl
import wifi
import socketpool
import adafruit_requests

pool = socketpool.SocketPool(wifi.radio)

JSON_STARS_URL = "https://api.github.com/repos/adafruit/circuitpython"

# Connect to Wi-Fi
print("\n====== MatrixPortal S3 WebClient Test ======")
print("Connecting to WiFi...")
requests = adafruit_requests.Session(pool, ssl.create_default_context())
while not wifi.radio.ipv4_address:
    try:
        wifi.radio.connect(os.getenv('WIFI_SSID'), os.getenv('WIFI_PASSWORD'))
    except ConnectionError as e:
        print("Connection Error:", e)
        print("Retrying in 10 seconds")
    time.sleep(10)
print("Connected!\n")

while True:
    try:
        response = requests.get(JSON_STARS_URL).json()
        stargazers_count = response['stargazers_count']
        print(f"CircuitPython GitHub Stars: {stargazers_count}")
    except (ValueError, RuntimeError, OSError) as e:
        print("Failed to get data, retrying\n", e)
        time.sleep(60)
        continue
    response = None
    print("Finished")
    print("==============")
    time.sleep(60)

code.py output:

====== MatrixPortal S3 WebClient Test ======
Connecting to WiFi...
Connected!

CircuitPython GitHub Stars: 3643
Finished
==============

Works with that 9.0 alpha build. That issue is definitely fixed in the alpha. Good job; 👍

[DJDevon3]

@dhalbert @RetiredWizard Unsure if this can be closed now or not. Up to you.

[tannewt]

Is there a 9.0 alpha bundle? I'm getting

Use the .py files for now. We don't want to may a mpy bundle until 1.20 is merged because it changes the mpy format again.

[DJDevon3]

Thank you for the clarification. Already started transitioning the libraries on my 9.0 alpha boards to all py libraries and thus far everything is working as expected including the MatrixPortal S3.

[bill88t]

Just noting, this is still an issue in the the 9.0 pre-alpha builds (9.0.0-alpha.1-13-gc395e1f9f).
I was reworking my wget and tried wget https://raw.githubusercontent.com/adafruit/circuitpython/029c912bf0f19062426922962deaf12d9785c5bc/supervisor/shared/filesystem.c to test how it coped with the filename.

Traceback (most recent call last):
  File "adafruit_requests", line 1, in _get_socket
OSError: Failed SSL handshake

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "ljinux", line 1, in fpexec
  File "<string>", line 14, in <module>
  File "driver_wifi", line 1, in get
  File "adafruit_requests", line 1, in get
  File "adafruit_requests", line 1, in request
  File "adafruit_requests", line 1, in _get_socket
RuntimeError: Sending request failed

Adafruit_requests is an mpy for this specific commit. I build the mpy's directly for the version of circuitpython loaded on the board.

[RetiredWizard]

Yep, until the actual fix is decided on, if you want to use SSL Requests with the latest main, you need to manually replace lib/certificates/nina-fw/data/roots.pem with Dan's roots.pem file.

[anecdata]

...or use:

context = ssl.create_default_context()
context.load_verify_locations()

with a CA string or .pem file

[DJDevon3]

I suppose the only reason it worked for me is because I used the firmware.zip @RetiredWizard posted above in here which has the updated roots.pem in the build. Thank you for making that build for me. I can live on that build until it gets merged with a stable release.

[tyeth]

...or use:

context = ssl.create_default_context()
context.load_verify_locations()

with a CA string or .pem file

I'm admittedly not the sharpest python tool in the chest, but I assumed I'd be able to pass a bundle .pem file to load_verify_locations but I get an error about processing CRT (8015).

Fetching json from https://www.adafruit.com/api/quotes.php
Traceback (most recent call last):
  File "adafruit_requests.py", line 534, in _get_socket
OSError: Unhandled ESP TLS error 10368 0 8015 -1

I tried loading as bytes, string, and bunging a null terminator on the end, but no PEM bundle ever loaded (old-nina roots.pem 47kb, new-arduino-guide-nina 63kb, mozilla-full 216kb files). Is it only for loading a single chain/cert, or is there some mastery required for formatting etc?

I did replace the lib/nina-fw roots.pem with the 216kb mozilla copy (has cert labels and header comments) and built successfully, so quotes and github ssl both work, but looking at the build assets it's generated an ports/espressif/build/x509_crt_bundle.S file which looks like the equivalent bundle in c/assembly format.

So in relation to this issue, but specifically over the work-around, is there an example of a working certificate bundle and/or use of load_verify_locations for more than a single certificate?

It's also a bit misleading that load_verify_locations complains about extra arguments (with only one passed) unless you use the named argument cadata, lost a chunk of time on that one.

[anecdata]

load_verify_locations for more than a single certificate?

I'm not sure if it's intended for a bundle (docs could be modified to indicate), just that it's sometimes used to get around a missing cert for a site, without requiring a custom build.

[Myoldmopar]

Thanks for everything you all are doing. I struggled with this setting up a new Pico W today, trying to talk to the GitHub API. Based on the comments above, I simply downgraded to CircuitPython 8.2.0 and it immediately started working.

[dhalbert]

I plan to release an 8.2.x with updated root certificates soon.

[DJDevon3]

Tested and confirmed fixed with 8.2.5 released today on the following boards

• Feather ESP32-S2 with 4mb psram
• Feather ESP32-S3 with 4mb psram
• MatrixPortal S3

Thank you @dhalbert, issue closed.

[dhalbert]

@tyeth Could you open a new issue about the bundle loading? Also try it on regular CPython, and try just one cert and more than one.

[tyeth]

Done, opened #8381