Event handler sees no further pointer_hold after pointer_drag_start

[SaschaKoch]

Hi,

I work on a system with a 800x480 pxl touchscreen displaying several interaction buttons to operate the systems actuators. One of the interactions is changing an operating value, where the user either clicks a button to increase a value by one or holds the button pressed (button hold event) to continually increase the value to the desired level.

There have been complaints from the customer that the continuous increase stops and doesn't resume if a users finger slides a little on the button during this button hold event.

I have looked into the issue and from a user point of view, this can actually happen rather easily if you change the angle of your finger slightly and therefore "roll" it a little over the display. As a user I would say I didn't move my finger and would expect ongoing hold events to continue.

From a SW point of view I checked which events are triggered and sent to the widget handler and in fact each ever so slight movement on the touchscreen creates a raw_pointer_move. if enough raw_pointer_moves are triggered (and at this point I do not know how many are enough, this seems to depend on time/distance/etc) a pointer_drag_start is triggered. Once a pointer_drag_start is issued no further pointer_holds are being triggered no matter how long the finger stays still holding down the button. Only releasing the finger (triggering a pointer_drag_stop) and starting a new touch would enable new pointer_hold events as long as there is not too mich movement.

Looking deeper into the issue I found out that

• ImageButtons (and therefore probably all buttons) get triggered pointer_drags no matter where on the screen you slide as long as the drag started within the button. This ends with releasing the touch and triggering a pointer_drag_stop.
• RectangleWidgets only receive pointer_drags as long as you slide within their rectangle. As soon as you leave it no more evens are sent to the handler.
• ImageLabel only receive pointer_drags as long as you slide within their rectangle. As soon as you leave it no more evens are sent to the handler.

However, none of them receive further pointer_hold events as soon as the first pointer_drag (or pointer_drag_start) is issued.

So far my hot fix is rather complex and involves starting a timer which manually triggers a pointer_hold sending it to the button where the handler starts up the timer again.
Why can't there be subsequent pointer_hold events in parallel to the pointer_drags? To distinguish between drag and hold it would be very easy to do this in the handler where you only react to the event.id() you wish to.
Is this intended behavior or a bug?

I have a little test program to check the individual behavior and events triggered:

Image used (screenshot of an egt button):

#include <egt/ui>
#include <iostream>

const std::string picButton = "file:pictures/testPicButton3.png";

using namespace std;
using namespace egt;

int main(int argc, const char ** argv)
{

	Application app;

	TopWindow mainWin;
	mainWin.name("Topwindow");
	mainWin.color(Palette::ColorId::bg, Color(0xffffffff));

	auto heading = std::make_shared<Label>("Handler Test", Rect(300,100,200,30));


	Image empt(picButton);
	auto button1 = std::make_shared<ImageButton>(empt, "", Rect(150, 200, 100, 100));
	auto btn1Heading = std::make_shared<Label>("ImageButton", Rect(150,180,100,20));

	auto button2 = std::make_shared<RectangleWidget>(Rect(350,200,100,100));
	auto btn2Heading = std::make_shared<Label>("RectWdgt", Rect(350,180,100,20));

	auto button3 = std::make_shared<ImageLabel>(empt, "", Rect(550,200,100,100));
	auto btn3Heading = std::make_shared<Label>("ImageLabel", Rect(550,180,100,20));

	button1->on_event([](Event &event) {
			if(event.id() == EventId::pointer_hold)
			{
				cout << "ImageButton: " << "pointer_hold" << endl;
			}
			if(event.id() == EventId::pointer_drag_start)
			{
				cout << "ImageButton: " << "pointer_drag_start" << endl;
			}
			if(event.id() == EventId::pointer_drag_stop)
			{
				cout << "ImageButton: " << "pointer_drag_stop" << endl;
			}
			if(event.id() == EventId::pointer_drag)
			{
				cout << "ImageButton: " << "pointer_drag" << endl;
			}
			if(event.id() == EventId::raw_pointer_move)
			{
				cout << "ImageButton: " << "raw_pointer_move" << endl;
			}
			if(event.id() == EventId::pointer_click)
			{
				cout << "ImageButton: " << "pointer_click" << endl;
			}
	});

	button2->on_event([](Event &event) {
			if(event.id() == EventId::pointer_hold)
			{
				cout << "RectWdgt: " << "pointer_hold" << endl;
			}
			if(event.id() == EventId::pointer_drag_start)
			{
				cout << "RectWdgt: " << "pointer_drag_start" << endl;
			}
			if(event.id() == EventId::pointer_drag_stop)
			{
				cout << "RectWdgt: " << "pointer_drag_stop" << endl;
			}
			if(event.id() == EventId::pointer_drag)
			{
				cout << "RectWdgt: " << "pointer_drag" << endl;
			}
			if(event.id() == EventId::raw_pointer_move)
			{
				cout << "RectWdgt: " << "raw_pointer_move" << endl;
			}
			if(event.id() == EventId::pointer_click)
			{
				cout << "RectWdgt: " << "pointer_click" << endl;
			}
	});

	button3->on_event([](Event &event) {
			if(event.id() == EventId::pointer_hold)
			{
				cout << "ImageLabel: " << "pointer_hold" << endl;
			}
			if(event.id() == EventId::pointer_drag_start)
			{
				cout << "ImageLabel: " << "pointer_drag_start" << endl;
			}
			if(event.id() == EventId::pointer_drag_stop)
			{
				cout << "ImageLabel: " << "pointer_drag_stop" << endl;
			}
			if(event.id() == EventId::pointer_drag)
			{
				cout << "ImageLabel: " << "pointer_drag" << endl;
			}
			if(event.id() == EventId::raw_pointer_move)
			{
				cout << "ImageLabel: " << "raw_pointer_move" << endl;
			}
			if(event.id() == EventId::pointer_click)
			{
				cout << "ImageLabel: " << "pointer_click" << endl;
			}
	});


	mainWin.add(heading);
	mainWin.add(button1);
	mainWin.add(button2);
	mainWin.add(button3);
	mainWin.add(btn1Heading);
	mainWin.add(btn2Heading);
	mainWin.add(btn3Heading);


	mainWin.show();

	return app.run();
}

Regards
Sascha

[ldesroches]

Hi Sascha,

Thanks for the report and the investigation you did. I'll have a look as soon as possible.

Regards,
Ludovic

[ldesroches]

Hi,

Can you have a look to this branch: https://github.com/linux4sam/egt/commits/27-event-handler-sees-no-further-pointer_hold-after-pointer_drag_start/

I don't see any issue emitting the hold event while in drag mode. So I did the change. I have also provide a way to configure the 'sensibility' of the drag mode.

Regards,
Ludovic

[SaschaKoch]

Hi Ludovic,

Thanks for the commit. I gave it a try on my host system with the above test program and it looks good. Regular hold events are coming in in parallel to drags, that's perfect. Also the deletion of the pointer_click event at the end of a hold event is good (even though it means me having to undoing all the hot fixes :-) ).

I didn't try the adjustment of the drag sensibility yet. On which object do I change that property and can it be adjusted for each Widget individually or is it a global setting?

I will try out the changes on our target system over the next couple days and give you feedback again.

Regards
Sascha

[ldesroches]

Hi,

The drag sensibility is a global setting as it is a static parameter of MouseGesture, you can change it by calling egt::detail::MouseGesture::drag_enable_distance().

Regards,
Ludovic

[SaschaKoch]

Hi,

I tried the commit on my HW and it works perfect as expected. Thanks a lot for these improvements of the emitted events.
I did not change the drag sensitivity since I do have some elements on my UI (sliders) which use drag and are working fine as they are.

Feel free to close the issue as resolved.

Regards
Sascha

[SaschaKoch]

I have one additional remark or suggestion which is not exactly the original issue but it might be able to piggy back on it since it is closely related:

The drag events are split into a drag_start, drag, and drag_stop so that the program can react precisely on the beginning of the drag, each individual drag event or the end of the drag. This is a useful feature! The hold event however does not have this, there is only one hold event being emitted every 0,5s. It is therefore rather complicated to track if a hold event is finished and a new interaction starts or if the hold event is still ongoing and a parallel event (for example a drag event) happens.

To further elaborate the problem:
Related to the problem above, there shall be a button which can be either clicked to change a value by one increment or the button can be held pushed to continuously increment the value every 0,5s. Again as described above when clicking the users finger can slip slightly and trigger a drag_start/drag_stop rather than a click. Therefore, I would like to treat a drag_stop in the same way than a click.
When holding the button pushed the users finger could as well slip a bit creating a drag_start/drag_stop. Now the drag_stop is already been treated like a click leading to an additional click at the end of the hold events (similar to the situation before it was rectified in the commit by Ludovic). An easy way to fix this would be to neglect a drag_stop whenever there was a hold event before.

Now we get to the point: If a user actually produces a clean button hold without moving his finder (and this can be done easily) ends his button hold interaction and now wants to do a second interaction just a click and accidentally creates a drag_start/stop there is no way for the program to determine if the button hold event was finished and therefore the new drag_stop needs to be treated as a click or if the hold event was not finished and the drag_stop should be neglected.

The only way to work around this issue would be (very complicated) use a timer and check if the last hold event was more than 500ms away or to actually facilitate the raw pointers (raw_pointer_up) for example to detect the end of a stream of hold events.

If you add the raw_pointer_up in the above code snippet you can create the following three examples:

Case 1: User tries to click the button but slightly slides with his finger:

ImageButton: pointer_drag_start
ImageButton: raw_pointer_up
ImageButton: pointer_drag_stop

Case 2: User creates a couple consecutive hold events by keeping the button pushed and while releasing he accidentally slips with his finger

ImageButton: pointer_hold
ImageButton: pointer_hold
ImageButton: pointer_hold
ImageButton: pointer_hold
ImageButton: pointer_hold
ImageButton: pointer_drag_start
ImageButton: raw_pointer_up
ImageButton: pointer_drag_stop

Case 3: User creates a couple of clean consecutive hold events with no slipping at the end when releasing the button

ImageButton: pointer_hold
ImageButton: pointer_hold
ImageButton: pointer_hold
ImageButton: pointer_hold
ImageButton: pointer_hold
ImageButton: raw_pointer_up

without using the raw pointer or a timer the program cannot decide if there are two individual interactions of Case 3 followed by Case 1 or if it is a Case 2. Since it is mentioned in the documentation that it is preferred to use the normal pointers instead of the raw pointers it would be very helpful to get at least a pointer_hold_end event if not also a pointer_hold_start.

The whole thing is rather a small detail but it can make things easier for the programmer :-)

Regards
Sascha

[ldesroches]

Hi,

I think what you want to achieve is doable combining the hold event with raw pointer events. Nonetheless, even if I don't want to put too much logic with complex events, these ones probably make sense.
I have updated the branch, let me know if it fits your needs.

Regards,
Ludovic

[SaschaKoch]

Hi,

sorry for the late reply. I tested your latest implementations on the special 27-branch. The pointer_hold_start and pointer_hold_stop work as desired, thanks for that! However, if the finger slips during a hold event, (and a pointer_drag_start is emitted) there is no pointer_hold_stop anymore. It seems to be shadowed by the pointer_drag_stop in that case.

For the example code above:
simple hold event without finger/mouse movement triggers the following

ImageButton: pointer_hold_start
ImageButton: pointer_hold
ImageButton: pointer_hold
ImageButton: pointer_hold
ImageButton: pointer_hold
ImageButton: pointer_hold
ImageButton: pointer_hold
ImageButton: raw_pointer_up
ImageButton: pointer_hold_stop

which is as desired.
Hold event with an "accidental" drag results in that

ImageButton: pointer_hold_start
ImageButton: pointer_hold
ImageButton: pointer_hold
ImageButton: pointer_drag_start
ImageButton: pointer_drag
ImageButton: pointer_hold
ImageButton: pointer_hold
ImageButton: pointer_hold
ImageButton: raw_pointer_up
ImageButton: pointer_drag_stop

where you can see that the pointer_hold_stop is not there anymore. The drag stop can stay and in fact needs to stay for the slider widgets but it would be great if the pointer_hold_stop could also be emitted in that case.