SSL error with 8.2.2 or later when talking to github

[DJDevon3]

CircuitPython version

boot_out.txt

Adafruit CircuitPython 8.2.3 on 2023-08-11; Adafruit MatrixPortal S3 with ESP32S3
Board ID:adafruit_matrixportal_s3

INFO_UF2.TXT

TinyUF2 Bootloader 0.14.0-5-g45bc2fc - tinyusb (0.15.0-331-ge3b3229d6)
Model: Adafruit MatrixPortal S3
Board-ID: ESP32-S3-MatrixPortal-revB
Date: May 18 2023

Code/REPL

# SPDX-FileCopyrightText: 2020 Brent Rubell for Adafruit Industries
#
# SPDX-License-Identifier: MIT

import os
import ipaddress
import ssl
import wifi
import socketpool
import adafruit_requests

# URLs to fetch from
TEXT_URL = "http://wifitest.adafruit.com/testwifi/index.html"
JSON_QUOTES_URL = "https://www.adafruit.com/api/quotes.php"
JSON_STARS_URL = "https://api.github.com/repos/adafruit/circuitpython"

print("ESP32-S2 WebClient Test")

print(f"My MAC address: {[hex(i) for i in wifi.radio.mac_address]}")

print("Available WiFi networks:")
for network in wifi.radio.start_scanning_networks():
    print("\t%s\t\tRSSI: %d\tChannel: %d" % (str(network.ssid, "utf-8"),
                                             network.rssi, network.channel))
wifi.radio.stop_scanning_networks()

print(f"Connecting to {os.getenv('WIFI_SSID')}")
wifi.radio.connect(os.getenv("WIFI_SSID"), os.getenv("WIFI_PASSWORD"))
print(f"Connected to {os.getenv('WIFI_SSID')}")
print(f"My IP address: {wifi.radio.ipv4_address}")

ping_ip = ipaddress.IPv4Address("8.8.8.8")
ping = wifi.radio.ping(ip=ping_ip) * 1000
if ping is not None:
    print(f"Ping google.com: {ping} ms")
else:
    ping = wifi.radio.ping(ip=ping_ip)
    print(f"Ping google.com: {ping} ms")

pool = socketpool.SocketPool(wifi.radio)
requests = adafruit_requests.Session(pool, ssl.create_default_context())

print(f"Fetching text from {TEXT_URL}")
response = requests.get(TEXT_URL)
print("-" * 40)
print(response.text)
print("-" * 40)

print(f"Fetching json from {JSON_QUOTES_URL}")
response = requests.get(JSON_QUOTES_URL)
print("-" * 40)
print(response.json())
print("-" * 40)

print()

print(f"Fetching and parsing json from {JSON_STARS_URL}")
response = requests.get(JSON_STARS_URL)
print("-" * 40)
print(f"CircuitPython GitHub Stars: {response.json()['stargazers_count']}")
print("-" * 40)

print("Done")

Behavior

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
ESP32-S2 WebClient Test
My MAC address: ['obfuscated']
Available WiFi networks:
	SlurpNet		RSSI: -57	Channel: 10
Connecting to SlurpNet
Connected to SlurpNet
My IP address: obfuscated
Ping google.com: 0.0 ms
Fetching text from http://wifitest.adafruit.com/testwifi/index.html
----------------------------------------
This is a test of Adafruit WiFi!
If you can read this, its working :)
----------------------------------------
Fetching json from https://www.adafruit.com/api/quotes.php
----------------------------------------
[{'text': 'I don’t care that they stole my idea — I care that they don’t have any of their own', 'author': 'Nikola Tesla'}]
----------------------------------------

Fetching and parsing json from https://api.github.com/repos/adafruit/circuitpython
Traceback (most recent call last):
  File "adafruit_requests.py", line 515, in _get_socket
OSError: Failed SSL handshake

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "code.py", line 58, in <module>
  File "adafruit_requests.py", line 711, in get
  File "adafruit_requests.py", line 650, in request
  File "adafruit_requests.py", line 496, in _get_socket
RuntimeError: Sending request failed

Code done running.

Press any key to enter the REPL. Use CTRL-D to reload.

Description

Using Internet Test code from MatrixPortal S3 learn guide.

• succeeds SSL handshake with Adafruit HTTPS quotes url
• fails SSL handshake with HTTPS Github API

I have no problem viewing https://api.github.com/repos/adafruit/circuitpython with my browser. Shows all the JSON data fine. For whatever reason Github is failing to handshake with the MatrixPortal S3.

Additional information

This is only to report that a portion of code in a learn guide does not function correctly. I'm honestly not sure why it's failing.

Using adafruit_requests.mpy from adafruit-circuitpython-bundle-8.x-mpy-20230815

Matrix Portal is not connected to any matrix panels at this time. Using the board as I would a feather S3 with wifi.

I can get it to ignore the error with try/except but that's not the point in the case of learn guide code.

[DJDevon3]

Removed adafruit gets and focused only on the Github connection. It's not connecting to that API url. Rest of the script seems to work fine. It's specifically the Github handshake attempt.

[DJDevon3]

Here's a slimmer version that will fail gracefully during testing.

import os
import time
import ipaddress
import ssl
import wifi
import socketpool
import adafruit_requests

pool = socketpool.SocketPool(wifi.radio)

JSON_STARS_URL = "https://api.github.com/repos/adafruit/circuitpython"

# Connect to Wi-Fi
print("\n====== MatrixPortal S3 WebClient Test ======")
print("Connecting to WiFi...")
requests = adafruit_requests.Session(pool, ssl.create_default_context())
while not wifi.radio.ipv4_address:
    try:
        wifi.radio.connect(os.getenv('WIFI_SSID'), os.getenv('WIFI_PASSWORD'))
    except ConnectionError as e:
        print("Connection Error:", e)
        print("Retrying in 10 seconds")
    time.sleep(10)
print("Connected!\n")

while True:
    try:
        response = requests.get(JSON_STARS_URL).json()
        stargazers_count = response['stargazers_count']
        print(f"CircuitPython GitHub Stars: {stargazers_count}")
    except (ValueError, RuntimeError, OSError) as e:
        print("Failed to get data, retrying\n", e)
        time.sleep(60)
        continue
    response = None

    print("Done")

With the OSError handler in the mix it spits out

====== MatrixPortal WebClient Test ======
Connecting to WiFi...
Connected!

Failed to get data, retrying
 Sending request failed
Failed to get data, retrying
 Sending request failed
Failed to get data, retrying
 Sending request failed
Failed to get data, retrying
 Sending request failed

With continue statement it never reaches the print("done") line, it will if pass is used instead.

The script will attempt to connect indefinitely so no sense waiting for it to connect. I have no idea why it isn't working with Github. Did something change with Github's API? It's not a key:value pair error, it's a handshake error, which out of my comfort zone to track down.

[DJDevon3]

Tried on an Adafruit ESP32-S2 Feather on 8.2.3 with the same result.

Reverted the same S2 back to CP 8.2.0 and it works.

Adafruit CircuitPython 8.2.0 on 2023-07-05; Adafruit Feather ESP32S2 with ESP32S2
Board ID:adafruit_feather_esp32s2

====== Circuit Python Internet Simple Test ======
Connecting to WiFi...
Connected!

CircuitPython GitHub Stars: 3634

• 8.2.0 works
  Something between 8.2.0 and 8.2.3 broke SSL handshake (for some things but not all things). Attempting to bisect by installing older UF2's from S3 bucket.

8.2.3 broken (SSL Handshake error)
8.2.2 broken (SSL Handshake error)

8.2.1 I might be getting rate limited or there's some other issue. I am at least getting a handshake and response with 8.2.1

Adafruit CircuitPython 8.2.1 on 2023-07-25; Adafruit Feather ESP32S2 with ESP32S2
Board ID:adafruit_feather_esp32s2

====== Circuit Python Internet Simple Test ======
Connecting to WiFi...
Connected!

{'message': "API rate limit exceeded for *IPADDRESS* (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)", 'documentation_url': 'https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting'}
Traceback (most recent call last):
  File "code.py", line 30, in <module>
KeyError: stargazers_count

Code done running.

Accidentally left print(response) on and 8.2.0 flooded with data and crashed my S2 and Mu. It returned so much data that I immediately got throttled with 8.2.1 but the server does respond with a nice error message. At least it's connecting.

After that I went back to 8.2.0 just to see if I am getting rate limited and it'll throw the same error on a known good working version. It's giving the same rate limit error on 8.2.0 and 8.2.1 so I think we can call 8.2.1 good too.

The problem is after 8.2.1 to 8.2.2. There are no beta releases between 8.2.1 to 8.2.2.
The issue begins in 8.2.2. Bisection complete. Up to a developer to look into it.

I believe this issue could potentially affect any project (at least on S2/S3 modules) using Adafruit_Requests starting in 8.2.2
The way to recognize this bug is traceback will report OSError: Failed SSL handshake

[dhalbert]

Here is a roots.pem that I generated a couple of weeks ago using the procedure described in https://github.com/arduino/nina-fw/blob/master/README.md#build-a-new-certificate-list-based-on-the-google-android-root-ca-list.

roots.pem.zip

When I substitute this one for the one in lib/certificates/data/roots.pem, talking to github works again.
But I was unable to come up with an Adafruit IO example that breaks on the old roots.pem but not on this one, so I haven't made a PR yet.

I think we should stop depending on nina-fw as a submodule, and just keep our own roots.pem, documenting how we built it. We could even write makefile steps to do that and verify it.

[RetiredWizard]

I've been building the Pico W with the latest main branch bits and this evening was getting the "Sending request failed" message with one of my test programs. After building with the roots.pem file Dan posted the test program is working again.

That being said, my test program was running fine when I built from main last night so I'm thinking the error only occurs when you hit a particular server somewhere in your path.

I'll keep using the new roots.pem and if I hit any failures with it I'll update here.

[jepler]

The change to nina-fw that we think introduced this problem was initially done in the 8.2.x branch. However, it probably started affecting main today when we merged #8317.

After nina-fw is updated, the fix in CircuitPython will probably initially be made in 8.2.x and merged into main subsequent to that, though it depends on how Dan and/or Scott want to handle it.

[RetiredWizard]

@DJDevon3 Just curious, did you try building the MatrixPortal S3 using the roots.pem file that Dan posted above?

[DJDevon3]

No, building circuit python isn't something I've done in a long time. :/ I'm not good with make builds.

[RetiredWizard]

This is using the latest bits from main so it has the 1.19.1 Micropython merge included.

MatrixPortal S3 UF2 with updated certificates:
firmware.zip

[DJDevon3]

Is there a 9.0 alpha bundle? I'm getting

Traceback (most recent call last):
  File "code.py", line 7, in <module>
ValueError: incompatible .mpy file

with 9.0.0-alpha.1-8-gaf91625fb-dirty

It does not like the 8.2.3 adafruit_requests.mpy I have in lib

edit:
Grabbed the latest adafruit_requests.py and that works. Apparently alpha builds aren't using mpy.

import os
import time
import ssl
import wifi
import socketpool
import adafruit_requests

pool = socketpool.SocketPool(wifi.radio)

JSON_STARS_URL = "https://api.github.com/repos/adafruit/circuitpython"

# Connect to Wi-Fi
print("\n====== MatrixPortal S3 WebClient Test ======")
print("Connecting to WiFi...")
requests = adafruit_requests.Session(pool, ssl.create_default_context())
while not wifi.radio.ipv4_address:
    try:
        wifi.radio.connect(os.getenv('WIFI_SSID'), os.getenv('WIFI_PASSWORD'))
    except ConnectionError as e:
        print("Connection Error:", e)
        print("Retrying in 10 seconds")
    time.sleep(10)
print("Connected!\n")

while True:
    try:
        response = requests.get(JSON_STARS_URL).json()
        stargazers_count = response['stargazers_count']
        print(f"CircuitPython GitHub Stars: {stargazers_count}")
    except (ValueError, RuntimeError, OSError) as e:
        print("Failed to get data, retrying\n", e)
        time.sleep(60)
        continue
    response = None
    print("Finished")
    print("==============")
    time.sleep(60)

code.py output:

====== MatrixPortal S3 WebClient Test ======
Connecting to WiFi...
Connected!

CircuitPython GitHub Stars: 3643
Finished
==============

Works with that 9.0 alpha build. That issue is definitely fixed in the alpha. Good job; 👍