release builds using rustc 1.86.0 on macOS Ventura (intel) SDK exhibit incorrect behaviour

[spikegrobstein]

I’d like to report what appears to be a bug in the rust compiler which can be reproduced by using the reqwest crate, where it fails to include the http version string in the request and it only affects binaries that were:

• built targeting x86_64-apple-darwin
• using release mode
• using a macOS Ventura (13.x) SDK

The bug itself has nothing to do with the reqwest crate (looking at the code, it appears that this behaviour should be impossible) and specifically related to the built binary.

this bug does exist in release binaries cross-compiled from linux to macOS when using the macOS 13 (Ventura) SDK with osxcross.

the bug does not exist when creating a debug build or when cross-compiling from an AppleSilicon machine → intel and does not exist when performing a release build on intel → intel when the OS is not Ventura.

We tracked this down to being triggered by opt-level = 2

this bug does not exist in rustc 1.85.0 and appears to only affect 1.86.0 (and also occurs in nightly rust).

I have a reduction:

add reqwest with --features=blocking and this main.rs:

use reqwest::{blocking::Client, Method};

fn main() {
    Client::new().request(Method::GET, "http://localhost:8888/hello").send().unwrap();
}

in another window, start a nc server with: nc -l localhost 8888

then cargo run and you should see:

GET /hello HTTP/1.1
accept: */*
host: localhost:8888

If you then start a new nc server and cargo run --release you will see:

GET /hello
accept: */*
host: localhost:8888

(note the missing HTTP/1.1 on the first line)

when running this code in a debugger, I can see that the slice that’s supposed to add the version number to the request buffer has a length of 0, so it contains no data:

Process 79729 stopped
* thread #2, name = 'reqwest-internal-sync-runtime', stop reason = step in
    frame #0: 0x00000001000b380f reqwest-test`_$LT$hyper..proto..h1..role..Client$u20$as$u20$hyper..proto..h1..Http1Transaction$GT$::encode::h33cc17a167a456d1 at spec_extend.rs:61:18 [opt]
   58       #[track_caller]
   59       fn spec_extend(&mut self, iterator: slice::Iter<'a, T>) {
   60           let slice = iterator.as_slice();
-> 61           unsafe { self.append_elements(slice) };
   62       }
   63   }
Target 0: (reqwest-test) stopped.
(lldb) p self
(alloc::vec::Vec<unsigned char, alloc::alloc::Global> *) $0 = 0x0000000100504b90
(lldb) p self[0]
(alloc::vec::Vec<unsigned char, alloc::alloc::Global>) $1 = size=11 {
  [0] = 'G'
  [1] = 'E'
  [2] = 'T'
  [3] = ' '
  [4] = '/'
  [5] = 'h'
  [6] = 'e'
  [7] = 'l'
  [8] = 'l'
  [9] = 'o'
  [10] = ' '
}
(lldb) p slice
(*const [u8]) $2 = {
  data_ptr = 0x0000000000000000
  length = 0
}

a note from someone on my team who continued to look into it more deeply:

“it looks like it's computing the correct string HTTP/1.1 but for some odd reason it's computing a length of 0. as best I can tell it uses one lookup table of string pointers to find the string, and it uses another lookup table of 4-byte offsets that get added to the lookup table's base address to produce a new pointer, which looks like it's supposed to be the end of the string. Unfortunately that second lookup table has 3 identical values in it, meaning it will produce the correct end pointer for HTTP/1.0 but it produces the start pointer for HTTP/1.1, and so it ends up calculating a length of 0”

This always happens no matter what version of reqwest is used, and it seems to be caused by the rustc version.

I hope this is enough information.

[moxian]

Can you try running cargo bisect-rustc to narrow down what triggered the change? The command line would probably look like cargo bisect-rustc --start 1.85.0 --end 1.86.0 --prompt -- run

Edit: .. ah, you might need to replace the -- run with a more involved --script setup (I missed the fact that the issue requires cross-compilation)..

[jieyouxu]

this bug does exist in release binaries cross-compiled from linux to macOS when using the macOS 13 (Ventura) SDK with osxcross.

We may have to rely on your bisection to pinpoint a commit (range), because that setup is quite complicated...

[jieyouxu]

To minimize the variables:

• Another thing to try: does this also reproduce in the latest beta (I'd assume so)?
• Can you provide the exact reqwest version?
  • And transitively, which http version?

[spikegrobstein]

@moxian Sorry, I should have been more clear; this isn't only on cross compile. it happens if the macOS 13 SDK is used with the x86_64-apple-darwin target. both native and cross compiled.

I ran the bisect script and came up with this being offending commit: d88ffcd

output:

searched nightlies: from nightly-2025-01-03 to nightly-2025-02-15
regressed nightly: nightly-2025-02-15
searched commit range: a567209...d8810e3
regressed commit: d88ffcd

bisected with cargo-bisect-rustc v0.6.9

Host triple: x86_64-apple-darwin
Reproduce with:

cargo bisect-rustc --start 1.85.0 --end 1.86.0 --prompt -- run --release

@jieyouxu

confirmed that this bug occurs in the beta, as well.

I ran this with the following versions of reqwest and it occurred on each: 0.12.3, 0.12.5, 0.12.15

http crate version 1.3.1 is the one in my reduction project.

[jieyouxu]

cc @scottmcm

[spikegrobstein]

@moxian the tests in hyper all pass with the --release flag.

I ran tests in the reqwest crate for completeness. I get 2 failures in release mode that don't fail without the flag:

running 3 tests
test test_badssl_no_built_in_roots ... ok
test test_badssl_modern ... FAILED
test test_badssl_self_signed ... FAILED

failures:

---- test_badssl_modern stdout ----

thread 'test_badssl_modern' panicked at tests/badssl.rs:14:10:
called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: "https://mozilla-modern.badssl.com/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(Parse(Version))) }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- test_badssl_self_signed stdout ----

thread 'test_badssl_self_signed' panicked at tests/badssl.rs:55:10:
called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: "https://self-signed.badssl.com/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(Parse(Version))) }


failures:
    test_badssl_modern
    test_badssl_self_signed

test result: FAILED. 1 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.23s

these are the same errors that I'm reporting in this issue.

the code snippet you had me run does not error whether it's run with --release or not.

Could this be a very specific edge case that reqwest just happens to surface?

[apiraino]

cc: some people in the apple marker team for possibly a help reducing this regression on this platform.

(a bisection is at this comment)

@hkratz
@thomcc
@madsmtm
@BlackHoleFox

[madsmtm]

Thanks for the ping! I can reproduce the top-level example using:

DEVELOPER_DIR=/Applications/Xcode\ 14.3.1.app/Contents/Developer cargo run --release --target x86_64-apple-darwin

I'll try to investigate further.

[madsmtm]

I'm inclined to think that this is a linker bug? Can reproduce with just:

cargo rustc --release --target x86_64-apple-darwin -- -Clinker="/Applications/Xcode 14.3.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld"
./target/x86_64-apple-darwin/release/foo

Though still unsure of what the linker is actually doing wrong here, and if there's something rustc can do differently to avoid hitting this.

But then again, this is also present in the older Xcode 13.4.1, so it may also be a recent problem in rustc / LLVM.

[madsmtm]

I've minimized this to:

// dep.rs
#![crate_type = "lib"]

pub enum Version {
    Http09,
    Http10,
    Http11,
    H2,
    H3,
}

pub fn encode(version: Version, dst: &mut Vec<u8>) -> NonTrivialDrop {
    let has_drop = NonTrivialDrop;

    match version {
        Version::Http10 => dst.extend_from_slice(b"HTTP/1.0"),
        Version::Http11 => dst.extend_from_slice(b"HTTP/1.1"),
        Version::H2 => dst.extend_from_slice(b"HTTP/1.1"),
        _ => {}
    }

    has_drop
}

pub struct NonTrivialDrop;

impl Drop for NonTrivialDrop {
    #[inline(never)]
    #[export_name = "non_trivial_drop"]
    fn drop(&mut self) {
        let _ = Box::new(0);
    }
}

// foo.rs
extern crate dep;

use dep::{Version, encode};

fn main() {
    let mut dst = Vec::with_capacity(100);
    encode(Version::Http11, &mut dst);
    let res = std::str::from_utf8(&dst).unwrap();
    eprintln!("{res:?}");
}

Compile with:

rustc dep.rs --target x86_64-apple-darwin -C opt-level=1
rustc foo.rs --target x86_64-apple-darwin --extern dep=./libdep.rlib '-Clinker=/Applications/Xcode 14.3.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld'
./foo # Outputs "", where it should've output "HTTP/1.1"

Put it up as a GH repo at https://github.com/madsmtm/rustc-switch-older-xcode. Had to be in separate crates for some reason, possibly something relating to how the linker reads things from archives? Still unclear to me.

[madsmtm]

The bisected commit is indeed a red herring, I've pushed a commit to the repo above which demonstrates the problem with at least Rust 1.76 as well. Will try to minimize it further.

[madsmtm]

Now minimized to:

// foo.rs
#[inline(never)]
#[no_mangle]
pub unsafe fn slice_len_from_ptr_end(ptr: *const u8, end: *const u8) -> usize {
    core::hint::black_box(unsafe { end.offset_from(ptr) as usize })
}

#[inline]
pub fn slice_len(slice: &[u8]) -> usize {
    let ptr = slice.as_ptr();
    let end = unsafe { ptr.add(slice.len()) };
    unsafe { slice_len_from_ptr_end(ptr, end) }
}

pub struct NonTrivialDrop;

impl Drop for NonTrivialDrop {
    #[inline(never)]
    #[export_name = "non_trivial_drop"]
    fn drop(&mut self) {
        core::hint::black_box(());
    }
}

pub enum Version {
    Http09,
    Http10,
    Http11,
    H2,
    H3,
}

#[no_mangle]
fn broken() -> usize {
    let _has_drop = NonTrivialDrop;

    match core::hint::black_box(Version::Http11) {
        Version::Http10 => slice_len(b"HTTP/1.0"),
        Version::Http11 => slice_len(b"HTTP/1.1"),
        Version::H2 => slice_len(b"HTTP/1.1"),
        _ => 0,
    }
}

fn main() -> std::process::ExitCode {
    if broken() == 8 { std::process::ExitCode::SUCCESS } else { std::process::ExitCode::FAILURE }
}

Compile with:

rustc foo.rs -C opt-level=1 --target x86_64-apple-darwin -Clink-arg=-ld_classic

Seems like there's a bad interaction with the older linker (from Xcode 14.3.1 / when using ld_classic), matches/switches, unwinding and getting the end of data in static sections?

[madsmtm]

I converted the code to C++:

// Use any version of Xcode that has the old linker (-ld_classic).
// clang++ foo.cpp -fexceptions -O2 -target x86_64-apple-darwin -Wl,-ld_classic

#include <cstddef>
#include <cstdint>
#include <cstring>
#include <iostream>

// Helper to disable optimizations around a value.
static int black_box(int x) {
    volatile int temp = x;
    asm volatile("" ::: "memory");
    return temp;
}

__attribute__((noinline))
int slice_len_from_ptr_end(const char* ptr, const char* end) {
    return black_box(end - ptr);
}

static int slice_len(const char* ptr, int len) {
    const char* end = ptr + len;
    return slice_len_from_ptr_end(ptr, end);
}

__attribute__((noinline))
void non_trivial_destructor() {
    asm volatile("" ::: "memory");
}

struct NonTrivialDestructor {
    ~NonTrivialDestructor() {
        non_trivial_destructor();
    }
};

__attribute__((noinline))
int broken(int version) {
    NonTrivialDestructor _has_drop;

    switch (version) {
        case 1:
            return slice_len("aaaaaaa", 8);
        case 2:
            return slice_len("bbbbbbb", 8);
        case 3:
            return slice_len("bbbbbbb", 8);
        default:
            return 42;
    }
}

int main() {
    // Should return 8, unexpectedly returns 0 with old linker.
    return broken(2);
}

I think that's as far as I'll go, I've already procrastinated on other things for wayyy too long today.

I'd suggest that someone else reports this upstream to LLVM since it's not a Rust-specific issue?

@rustbot label A-llvm -regression-untriaged

[dianqk]

Amazing. I can continue the investigation.
Btw, you may have forgotten to add a break;.

[madsmtm]

Damn 🤦. Then that doesn't actually reproduce it. I've updated the code to be in C++ instead, since it seems to actually require the unwinding code.

[BlackHoleFox]

Wouldn't have been the first time unwinding broke in Apple's SDK tools sadly :/ #113783 was also a bad linker interaction with unwinding that impacted Rust.

[dianqk]

TL;DR
This is a bug in the apple linker. I didn't find any miscompiled with IR or ASM.

Here is a smaller example:

lib.rs

#![crate_type = "lib"]
#![allow(internal_features)]
#![feature(rustc_attrs)]

unsafe extern "Rust" {
    #[rustc_nounwind]
    safe fn slice_len_from_ptr_end(ptr: *const u8, end: *const u8) -> usize;
}

#[repr(i64)]
pub enum Version {
    A0 = 0,
    A1 = 1,
    A2 = 2,
}

static L1: &[u8] = &[0];
static L2: &[u8] = &[1, 2, 3, 4, 5, 6, 7, 8];
static L3: &[u8] = &[9];

#[no_mangle]
#[inline(never)]
pub unsafe fn broken(v: Version) -> usize {
    let ptr: *const u8;
    let end: *const u8;
    match v {
        Version::A0 => {
            ptr = L1.as_ptr();
            end = L1.as_ptr().add(1);
        }
        Version::A1 => {
            ptr = L2.as_ptr();
            end = L2.as_ptr().add(8);
        }
        Version::A2 => {
            ptr = L3.as_ptr();
            end = L3.as_ptr().add(1);
        }
    }
    slice_len_from_ptr_end(ptr, end)
}

main.rs

fn main() {
    unsafe {
        println!("{}", demo::broken(lookup::Version::A1));
    }
}

#[no_mangle]
pub unsafe fn slice_len_from_ptr_end(ptr: *const u8, end: *const u8) -> isize {
    end.offset_from(ptr)
}

It can also be reproduced in C by an altered IR.

foo.c

#include <stddef.h>
#include <stdint.h>
#include <stdio.h>

uint64_t slice_len_from_ptr_end(const uint8_t *ptr, const uint8_t *end);

const uint8_t L1[] = {0};
const uint8_t L2[] = {1, 2, 3, 4, 5, 6, 7, 8};
const uint8_t L3[] = {9};

size_t broken(uint64_t v) {
  const uint8_t *ptr;
  const uint8_t *end;

  switch (v) {
  case 0:
    ptr = L1;
    end = L1 + 1;
    break;
  case 1:
    ptr = L2;
    end = L2 + 8;
    break;
  case 2:
    ptr = L3;
    end = L3 + 1;
    break;
  default:
    __builtin_unreachable();
    break;
  }

  return slice_len_from_ptr_end(ptr, end);
}

main.c

#include <stddef.h>
#include <stdint.h>
#include <stdio.h>

uint64_t slice_len_from_ptr_end(const uint8_t *ptr, const uint8_t *end) {
  return end - ptr;
}

uint64_t broken(uint64_t v);

int main() {
  // Returns 0 unexpectedly with old linker
  printf("Expected: 8\nGot: %zu\n", broken(1));
  return 0;
}

Just need to add private to @L1 and @L3 and add private unnamed_addr to @L2. The IR can be simplified by opt -O1 | opt -pre-isel-intrinsic-lowering -S：

foo.ll

@L1 = private constant [1 x i8] zeroinitializer, align 1
@L2 = private unnamed_addr constant [8 x i8] c"\01\02\03\04\05\06\07\08", align 1
@L3 = private constant [1 x i8] c"\09", align 1
@reltable.broken = private unnamed_addr constant [3 x i32] [i32 trunc (i64 sub (i64 ptrtoint (ptr @L1 to i64), i64 ptrtoint (ptr @reltable.broken to i64)) to i32), i32 trunc (i64 sub (i64 ptrtoint (ptr @L2 to i64), i64 ptrtoint (ptr @reltable.broken to i64)) to i32), i32 trunc (i64 sub (i64 ptrtoint (ptr @L3 to i64), i64 ptrtoint (ptr @reltable.broken to i64)) to i32)], align 4
@reltable.broken.1 = private unnamed_addr constant [3 x i32] [i32 trunc (i64 sub (i64 ptrtoint (ptr getelementptr inbounds ([1 x i8], ptr @L1, i64 1, i64 0) to i64), i64 ptrtoint (ptr @reltable.broken.1 to i64)) to i32), i32 trunc (i64 sub (i64 ptrtoint (ptr getelementptr inbounds ([8 x i8], ptr @L2, i64 1, i64 0) to i64), i64 ptrtoint (ptr @reltable.broken.1 to i64)) to i32), i32 trunc (i64 sub (i64 ptrtoint (ptr getelementptr inbounds ([1 x i8], ptr @L3, i64 1, i64 0) to i64), i64 ptrtoint (ptr @reltable.broken.1 to i64)) to i32)], align 4

define i64 @broken(i64 noundef %0) {
  %reltable.shift = shl i64 %0, 2
  %2 = getelementptr i8, ptr @reltable.broken, i64 %reltable.shift
  %3 = load i32, ptr %2, align 4
  %4 = getelementptr i8, ptr @reltable.broken, i32 %3
  %5 = getelementptr i8, ptr @reltable.broken.1, i64 %reltable.shift
  %6 = load i32, ptr %5, align 4
  %7 = getelementptr i8, ptr @reltable.broken.1, i32 %6
  %8 = tail call i64 @slice_len_from_ptr_end(ptr noundef nonnull %4, ptr noundef nonnull %7)
  ret i64 %8
}

declare i64 @slice_len_from_ptr_end(ptr noundef, ptr noundef)

test.sh

TARGET="x86_64-apple-darwin"
CLANG="/usr/bin/clang"

$CLANG -c foo.ll -O1 -target $TARGET 
$CLANG -c main.c -O1 -target $TARGET 
$CLANG foo.o main.o -O1 -target $TARGET -o main -Wl,-ld_classic
./main
echo "output:"
echo "-Wl,-ld_classic: $?"
$CLANG foo.o main.o -O1 -target $TARGET -o main
./main
echo "$?"

# output:
# -Wl,-ld_classic: 0
# 8

Everything's fine, except that ld-classic is incorrectly assigning relative addresses.

	leaq	l_reltable.broken.1(%rip), %rcx
	movslq	(%rcx,%rdi,4), %rsi
	addq	%rcx, %rsi
...
l_reltable.broken.1:
	.long	(l_L1-l_reltable.broken.1)+1
	.long	(L_L2-l_reltable.broken.1)+8
	.long	(l_L3-l_reltable.broken.1)+1

The new linker output is:

0000000100000ef0 <_broken>:
...
100000efe: 48 8d 0d 4b 00 00 00        	leaq	0x4b(%rip), %rcx        ## 0x100000f50
100000f05: 48 63 34 b9                 	movslq	(%rcx,%rdi,4), %rsi
100000f09: 48 01 ce                    	addq	%rcx, %rsi
...
100000f4f: ff f1                       	pushq	%rcx
100000f51: ff ff                       	<unknown>
100000f53: ff 18                       	lcalll	*(%rax)
100000f55: 00 00                       	addb	%al, (%rax)
100000f57: 00 f2                       	addb	%dh, %dl
100000f59: ff ff                       	<unknown>
100000f5b: ff 00                       	incl	(%rax)
100000f5d: 00 00                       	addb	%al, (%rax)
100000f5f: 00 01                       	addb	%al, (%rcx)
100000f61: 02 03                       	addb	(%rbx), %al
100000f63: 04 05                       	addb	$0x5, %al
100000f65: 06                          	<unknown>
100000f66: 07                          	<unknown>
100000f67: 08                          	<unknown>

l_reltable.broken.1 is 0xffffff1,0x00000018,0xfffffff2, and %rsi is 0x100000f50+0x0000000000000018=(u32)0x100000f68 that also is L2+8.

But ld_classic output is：

0000000100000ee0 <_broken>:
...
100000eee: 48 8d 0d 4b 00 00 00        	leaq	0x4b(%rip), %rcx        ## 0x100000f40
100000ef5: 48 63 34 b9                 	movslq	(%rcx,%rdi,4), %rsi
100000ef9: 48 01 ce                    	addq	%rcx, %rsi
...
100000f3f: ff f1                       	pushq	%rcx
100000f41: ff ff                       	<unknown>
100000f43: ff 10                       	callq	*(%rax)
100000f45: 00 00                       	addb	%al, (%rax)
100000f47: 00 f2                       	addb	%dh, %dl
100000f49: ff ff                       	<unknown>
100000f4b: ff 00                       	incl	(%rax)
100000f4d: 00 00                       	addb	%al, (%rax)
100000f4f: 00 01                       	addb	%al, (%rcx)
100000f51: 02 03                       	addb	(%rbx), %al
100000f53: 04 05                       	addb	$0x5, %al
100000f55: 06                          	<unknown>
100000f56: 07                          	<unknown>
100000f57: 08                          	<unknown>

l_reltable.broken.1 is 0xffffff1,0x00000010,0xfffffff2, and %rsi is 0x100000f40+0x0000000000000010=(u32)0x100000f50 that should be L2+8, but L2.

lld triggers an assertion: out_of_range was thrown in -fno-exceptions mode with message "unordered_map::at: key not found. that I haven't investigated yet. Upstream issue: llvm/llvm-project#139439.

[madsmtm]

I've submitted this upstream to Apple via. the Feedback Assistant as FB17575521, but I haven't had much luck with that in the past, so an interesting question is whether LLVM could emit different codegen to avoid this bug?

[dianqk]

I've submitted this upstream to Apple via. the Feedback Assistant as FB17575521, but I haven't had much luck with that in the past, so an interesting question is whether LLVM could emit different codegen to avoid this bug?

Yes. It's possible. PR is rust-lang/llvm-project#181. I've also reached out to some folks at Apple, which might help the fix.

[madsmtm]

I got a response from Apple on my feedback request:

It looks like this has always been a problem with ld64, but we fixed it with the new linker. ld64 and the -ld_classic option are deprecated. Developers should be using the new linker if they haven’t already.

If there’s something that’s blocking rust from using the new linker we’ll be happy to take a look at that.

[BlackHoleFox]

Backread the issue but where does the -ld_classic usage come from in Rust by default? Rust should be passing either ld or cc. Then ld in modern SDKs should be their new linker.

[madsmtm]

Yup, rustc doesn't use -ld_classic. The issue appears with the macOS Ventura (13.x) SDK with osxcross, which uses the old ld64 (which is also what's available in Xcode 14.3.1).

The new linker source code hasn't been released, so this is currently the best way to cross-compile from Linux. See also tpoechtrager/osxcross#453 / tpoechtrager/osxcross#457.