Bevy picking fails after camera movment. Picking has wrong positions for entities after camera viewport is moved

[aDecoy]

Picking has wrong positions for entities after camera viewport is moved

Bevy version 0.16.0

Relevant system information

• cargo 1.86.0 (adf9b6ad1 2025-02-28)
• Windows 10,

[package]  
name = "bevyHelloWorld"  
version = "0.1.0"  
edition = "2024"  
  
[dependencies]  
  
bevy = { version = "0.16.0", features = ["dynamic_linking", "wayland" ,"bevy_dev_tools"] }  
  
[profile.dev]  
opt-level =1  

[profile.dev.package."*"]  
opt-level = 3

If your bug is rendering-related, copy the adapter info that appears when you run Bevy.

INFO bevy_render::renderer: AdapterInfo { name: "AMD Radeon RX 6800 XT", vendor: 4098, device: 29631, device_type: DiscreteGpu, driver: "AMD proprietary driver", driver_info: "24.12.1 (AMD proprietary shader compiler)", backend: Vulkan }

What you did

I have a window and a camera with a viewport + a button that allows me to drag the camera around on the window. The dragging is done with bevy picking (Trigger<Pointer<Drag>>). I simply move the camera around. And it only works once. Picking fails to find the button where it is rendered if the camera was moved.

Code to replicate this bug is lower down.

What went wrong

The first time i draged the button, the camera-moving works fine.

If the camera was moved away from the start position, it fails the second time.

• But it is possible to click and drag a "ghost-button" that is twice the distance away from the start position.
  • It seems like the magic Picking uses to locate entities based on screen position fails after a camera movment.

Pictures

The blue square is the button i am dragging. Note that the debugging find the correct entity in picture 1, but not in picture 2 after the camera has been draged to the right. Picture 3 shows the "ghost-button"

(i was not able to upload video, so it might be easiest to run the code and see)

What i suspect is wrong

I think the Picking is using the distance to the window left edge, instead of to the camera left edge.
Either that or the distance is being added twice. The ghost button always seems to be about twice the distance away.

Code to replicate bug

It is an extention of the https://github.com/bevyengine/bevy/blob/main/examples/picking/debug_picking.rs example, so it provides a lot of debug info. I changed it to 2d to make things a bit easier.

//! A simple scene to demonstrate picking events for UI and mesh entities,  
//! Demonstrates how to change debug settings  
  
use bevy::color::palettes::basic::BLUE;  
use bevy::dev_tools::picking_debug::{DebugPickingMode, DebugPickingPlugin};  
use bevy::prelude::*;  
use bevy::render::camera::Viewport;  
use bevy::window::{PrimaryWindow, WindowResized};  
use std::cmp::{max, min};  
  
fn main() {  
    App::new()  
        .add_plugins(DefaultPlugins.set(bevy::log::LogPlugin {  
            filter: "bevy_dev_tools=trace".into(), // Show picking logs trace level and up  
            ..default()  
        }))  
        .add_plugins((MeshPickingPlugin, DebugPickingPlugin))  
        .add_systems(Startup, setup_scene)  
        .insert_resource(DebugPickingMode::Normal)  
        // A system that cycles the debugging state when you press F3:  
        .add_systems(  
            PreUpdate,  
            (|mut mode: ResMut<DebugPickingMode>| {  
                *mode = match *mode {  
                    DebugPickingMode::Disabled => DebugPickingMode::Normal,  
                    DebugPickingMode::Normal => DebugPickingMode::Noisy,  
                    DebugPickingMode::Noisy => DebugPickingMode::Disabled,  
                }  
            })  
            .distributive_run_if(bevy::input::common_conditions::input_just_pressed(  
                KeyCode::F3,  
            )),  
        )  
        .add_systems(  
            Startup,  
            spawn_button_for_moving_camera_on_screen_inside_window_button.after(setup_scene),  
        )  
        .add_systems(  
            Update,  
            (  
                set_camera_viewports,  
                adjust_button_for_camera_dragging_on_window_resize,  
            ),  
        )  
        .run();  
}  
  
fn setup_scene(  
    mut commands: Commands,  
    mut meshes: ResMut<Assets<Mesh>>,  
    mut materials: ResMut<Assets<ColorMaterial>>,  
) {  
    commands.spawn((  
        Text::new("Click Me to get a box\nDrag cubes to rotate\nPress F3 to cycle between picking debug levels"),  
        Node {  
            position_type: PositionType::Absolute,  
            top: Val::Percent(12.0),  
            left: Val::Percent(12.0),  
            ..default()  
        },  
    )).observe(on_click_spawn_cube).observe(  
        |out: Trigger<Pointer<Out>>, mut texts: Query<&mut TextColor>| {  
            let mut text_color = texts.get_mut(out.target()).unwrap();  
            text_color.0 = Color::WHITE;  
        },  
    ).observe(  
        |over: Trigger<Pointer<Over>>, mut texts: Query<&mut TextColor>| {  
            let mut color = texts.get_mut(over.target()).unwrap();  
            color.0 = bevy::color::palettes::tailwind::CYAN_400.into();  
        },  
    );  
  
    // Base  
    commands.spawn((  
        Name::new("Base"),  
        Mesh3d(meshes.add(Circle::new(20.0))),  
        MeshMaterial2d(materials.add(Color::WHITE)),  
        Transform::from_rotation(Quat::from_rotation_x(-std::f32::consts::FRAC_PI_2)),  
    ));  
  
    // Light  
    commands.spawn((  
        PointLight {  
            shadows_enabled: true,  
            ..default()  
        },  
        Transform::from_xyz(4.0, 8.0, 4.0),  
    ));  
  
    // Camera  
    commands.spawn((  
        Camera2d::default(),  
        Camera {  
            viewport: Some(Viewport::default()),  
            ..default()  
        },  
    ));  
}  
  
fn set_camera_viewports(  
    primary_window: Query<&Window, With<PrimaryWindow>>,  
    mut resize_events: EventReader<WindowResized>,  
    mut kamera_query: Query<(&mut Camera)>,  
) {  
    for resize_event in resize_events.read() {  
        if let Ok(window) = primary_window.get(resize_event.window) {  
            let window_size = window.physical_size();  
            let halv_skjerm_størrelse = UVec2::new(window_size.x / 2, window_size.y);  
  
            for (mut camera) in &mut kamera_query {  
                camera.viewport = Some(Viewport {  
                    physical_position: UVec2::new(0, 0),  
                    physical_size: halv_skjerm_størrelse,  
                    ..default()  
                });  
            }  
        }  
    }  
}  
  
fn on_click_spawn_cube(  
    _click: Trigger<Pointer<Click>>,  
    mut commands: Commands,  
    mut meshes: ResMut<Assets<Mesh>>,  
    mut materials: ResMut<Assets<ColorMaterial>>,  
    mut num: Local<usize>,  
) {  
    commands  
        .spawn((  
            Mesh2d(meshes.add(Rectangle::new(80.0, 80.0))),  
            MeshMaterial2d(materials.add(Color::srgb_u8(124, 144, 255))),  
            Transform::from_xyz(0.0, 0.25 + 81.5 * *num as f32, 0.0),  
        ))  
        .observe(on_drag_rotate);  
    *num += 1;  
}  
  
fn on_drag_rotate(drag: Trigger<Pointer<Drag>>, mut transforms: Query<&mut Transform>) {  
    if let Ok(mut transform) = transforms.get_mut(drag.target()) {  
        transform.rotate_y(drag.delta.x * 0.02);  
        transform.rotate_x(drag.delta.y * 0.02);  
    }  
}  
  
#[derive(Component)]  
struct KameraEdgeMoveCameraInTheWindowDragButtonTag;  
  
pub fn spawn_button_for_moving_camera_on_screen_inside_window_button(  
    mut commands: Commands,  
    mut meshes: ResMut<Assets<Mesh>>,  
    mut materials: ResMut<Assets<ColorMaterial>>,  
    kamera_query: Query<(Entity, &Camera), With<Camera>>,  
) {  
    let material_handle = materials.add(Color::from(BLUE));  
  
    for (kamera_entity, camera) in kamera_query.iter() {  
        let mut parent_kamera = commands.get_entity(kamera_entity);  
  
        parent_kamera.unwrap().with_children(|parent_builder| {  
            parent_builder  
                .spawn((  
                    Mesh2d(meshes.add(Rectangle::new(51.5, 51.5))),  
                    MeshMaterial2d(material_handle.clone().into()),  
                    Transform::default(), // will be adjusted on window resize events  
                    KameraEdgeMoveCameraInTheWindowDragButtonTag,  
                ))  
                .observe(drag_camera_around_in_the_window);  
        });  
    }  
}  
  
fn drag_camera_around_in_the_window(  
    drag: Trigger<Pointer<Drag>>,  
    mut kamera_query: Query<(&mut Camera)>,  
    primary_window: Query<&Window>,  
) {  
    if let Ok(mut camera) = kamera_query.single_mut() {  
        // println!("moving camera in window");  
        let window_size = primary_window.single().unwrap().physical_size();  
  
        // move viewport  
        if let Some(viewport) = &mut camera.viewport {  
            let u32_vektor: &mut UVec2 = &mut viewport.physical_position;  
            drag_u32_vektor_med_potensielt_negative_i32_verdier(drag, u32_vektor);  
  
            u32_vektor.x = min(window_size.x - viewport.physical_size.x, u32_vektor.x);  
            u32_vektor.y = min(window_size.y - viewport.physical_size.y, u32_vektor.y);  
        }  
    }  
}  
  
fn drag_u32_vektor_med_potensielt_negative_i32_verdier(  
    drag: Trigger<Pointer<Drag>>,  
    u32_vektor: &mut UVec2,  
) {  
    let mut new_x_value = u32_vektor.x as i32;  
    new_x_value += drag.delta.x as i32;  
    u32_vektor.x = max(0, new_x_value) as u32;  
  
    let mut new_y_value = u32_vektor.y as i32;  
    new_y_value += drag.delta.y as i32;  
    u32_vektor.y = max(0, new_y_value) as u32;  
}  
  
/// Because the button is \"connected\" to camera edges, like a HUD element, instead of being connected to the "render of the world of entities", we update translation.  
/// (it gets a bit stretched away from the camera edge when scaling (adjusting window size) for some reason  ¯\_(ツ)_/¯ , and I feel like this looks better  
/// Bug still persist without this system   fn adjust_button_for_camera_dragging_on_window_resize(  
    kamera_query: Query<  
        (&Camera, &Children),  
        (  
            Changed<Camera>,  
            Without<KameraEdgeMoveCameraInTheWindowDragButtonTag>,  
        ),  
    >,  
    mut button_query: Query<&mut Transform, With<KameraEdgeMoveCameraInTheWindowDragButtonTag>>,  
    mut resize_events: EventReader<WindowResized>,  
) {  
    // moved it double?  
    if !resize_events.is_empty() {  
        println!("resizing window => moving HUD button since it is \"connected\" to camera dimensions, instead of being connected to the render of the world of entities ");  
  
        for (camera, barn_marginer) in kamera_query.iter() {  
            println!("updating button transform.translation");  
            if let Some(viewport) = &camera.viewport {  
                let view_dimensions = Vec2 {  
                    x: viewport.physical_size.x as f32,  
                    y: viewport.physical_size.y as f32,  
                };  
  
                // camera in top_left_corner has physical positon 0.0. Transform 0.0 is drawn at center of camera  
                let padding = 100.0;  
                let venstre_side_x = -(view_dimensions.x * 0.5) + padding;  
                let høyre_side_x = (view_dimensions.x * 0.5) - padding;  
                let bunn_side_y = -(view_dimensions.y * 0.5) + padding;  
                for barn_margin_ref in barn_marginer {  
                    // akkurat nå så er knappen alltid nede til høyre  
                    if let Ok((mut transform)) = button_query.get_mut(*barn_margin_ref) {  
                        transform.translation.x = venstre_side_x;  
                        transform.translation.y = bunn_side_y;  
                    }  
                }  
            };  
        }  
    }  
}

[aDecoy]

It seems that the problem is in the picking backend, since the PointerHits for the entity is missing after camera move.

(A PointerHits event produced by a picking backend after it has run its hit tests, describing the entities under a pointer. https://docs.rs/bevy/latest/bevy/picking/backend/struct.PointerHits.html)

How i got the debug info:

Info is from hovering the mouse over objects and using this system

fn listen_to_pointer_hits_on_keyboard_input(
    mut pointer_hits_events: EventReader<PointerHits>,
    keyboard_input: Res<ButtonInput<KeyCode>>,
) {
    if keyboard_input.pressed(KeyCode::KeyB) {
        info!("'B' currently pressed");
        for event in pointer_hits_events.read() {
            dbg!(&event);
        }
    }
}

When it does not work, it only finds the "default nothing event", and "hit default camera in primary window"-event.

2025-05-10T11:04:20.412353Z  INFO bevyHelloWorld: 'B' currently pressed
[src\main.rs:320:13] &event = PointerHits {
    pointer: Mouse,
    picks: [],
    order: 0.0,
}
[src\main.rs:320:13] &event = PointerHits {
    pointer: Mouse,
    picks: [
        (
            0v1#4294967296,
            HitData {
                camera: 0v1#4294967296,
                depth: 0.0,
                position: None,
                normal: None,
            },
        ),
    ],
    order: -inf,
}

When it works before the camera move i see the event when i hover over it :

2025-05-10T11:03:23.466149Z  INFO bevyHelloWorld: 'B' currently pressed
[src\main.rs:320:13] &event = PointerHits {
    pointer: Mouse,
    picks: [],
    order: 0.0,
}
[src\main.rs:320:13] &event = PointerHits {
    pointer: Mouse,
    picks: [
        (
            13v1#4294967309,
            HitData {
                camera: 9v1#4294967305,
                depth: 999.9999,
                position: Some(
                    Vec3(
                        -226.0,
                        -253.99998,
                        0.0,
                    ),
                ),
                normal: Some(
                    Vec3(
                        0.0,
                        0.0,
                        1.0,
                    ),
                ),
            },
        ),
    ],
    order: 0.0,
}
[src\main.rs:320:13] &event = PointerHits {
    pointer: Mouse,
    picks: [
        (
            0v1#4294967296,
            HitData {
                camera: 0v1#4294967296,
                depth: 0.0,
                position: None,
                normal: None,
            },
        ),
    ],
    order: -inf,
}

[aDecoy]

It seems like UI-elements/Node-elements are not impacted by this bug. In the example code i posted, the "click me to get a box..." text has correct hitbox, but the "box-entities" that are spawned are affected by the bug.

[aDecoy]

Further findings: It seems like the RayMap is wrong after a camera viewport move.

More specifically: The "origin" Vec3 of the Ray3D objects inside RayMap is wrong after a camera viewport move.

The (0,0) X,Y spot is in the center of the camera before move, but after the move, it has shifted the same direction and distance as the camera viewport move. (to the right in this example)

Debug systems

// My raycasting backend debugger
pub fn debug_ray_map(ray_map: Res<RayMap>, 
                   keyboard_input: Res<ButtonInput<KeyCode>>,
) {
    // for (&ray_id, &ray) in ray_map.iter() {
    if keyboard_input.pressed(KeyCode::KeyC) {
        info!("'C' currently pressed");
        for (ray) in ray_map.iter() {
            dbg!(&ray);
        }
    }
}

fn move_camera_a_bit_on_button_press(
    mut kamera_query: Query<&mut Camera>,
    keyboard_input: Res<ButtonInput<KeyCode>>,
) {
    if keyboard_input.pressed(KeyCode::KeyA) {
        info!("'A' currently pressed");
        if let Ok(mut camera) = kamera_query.single_mut() {
            // println!("moving camera in window");
            // move viewport
            if let Some(viewport) = &mut camera.viewport {
                let u32_vektor: &mut UVec2 = &mut viewport.physical_position;
                u32_vektor.x += 20
            }
        }
    }
}

RayMap docs : https://docs.rs/bevy/latest/bevy/picking/backend/prelude/struct.RayMap.html

[tomara-x]

i also ran into this yesterday, and looked at your investigation and went and fixed it, then found that someone already did the same thing in main :D 7f04906

[atlv24]

Fixed in main by #18870 and will be on patch release 0.16.1 (also, dupe of #18856)