Skip to content
This repository was archived by the owner on Feb 25, 2025. It is now read-only.

Support Scribble Handwriting #24224

Merged
merged 63 commits into from
Nov 8, 2021

Conversation

fbcouch
Copy link
Contributor

@fbcouch fbcouch commented Feb 5, 2021

Framework PR:
flutter/flutter#75472

This PR is the engine side of adding scribble handwriting support. I still need to clean up the NSLogs and add some tests, but looking for feedback on the overall approach. I did incorporate some feedback from @LongCatIsLooong on a previous draft, moving most of the logic of this into the framework and adding a widget there to handle most of the scribble-specific logic.

flutter/flutter#61278

Design doc: https://docs.google.com/document/d/1mjQbsSRQnHuAgMNdouaSgTS-Xv-w57fdKfOUUafWpRo

If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.

Pre-launch Checklist

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
  • I listed at least one issue that this PR fixes in the description above.
  • I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on
    writing and running engine tests.
  • I updated/added relevant documentation (doc comments with ///).
  • I signed the CLA.
  • All existing and new tests are passing.
  • The reviewer has submitted any presubmit flakes in this PR using the engine presubmit flakes form before re-triggering the failure.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

@fbcouch fbcouch changed the title Support Scribble Handrwriting Support Scribble Handwriting Feb 5, 2021
@fbcouch fbcouch force-pushed the feature/scribble+master branch from 7f5494d to 700fcd9 Compare April 8, 2021 20:26
fbcouch added 13 commits May 5, 2021 16:38
[scribble] Implement some more methods

[scribble] Clean up some things, add some logging and forward touch event selectors to superview to try to fix this issue where the cursor cant be moved easily

[scribble] Use UIScribbleInteractionDelegate to disable setMarkedText during scribble; add a reference to the viewController to send touch events to

[scribble] Do not send showAutocorrectionPromptRectForStart from firstRectForRange while scribble is in progress - that's what was causing the last character to always be selected

[scribble] Send 'showToolbar' message over the text input channel and add scribbleInProgress key to the editing state

[scribble] Add UIIndirectScribbleInteraction and Delegate

[scribble] Focuses, but not smoothly

[scribble] Able to focus in 'correct' position

[scribble] Track scribble rects on the framework side

[scribble] Clean up before rebase
…ods so that the withinRange version is called by the other
…e FlutterTextInputViewAccessibilityHider from views, as it prevents direct scribble interactions
…er so text doesn't flyaway to the end of the string for some reason
…now...not sure how that's the case, but OK; also fix some things so that the weird issue with zero width joiner characters seems to be fixed now as well
@fbcouch fbcouch force-pushed the feature/scribble+master branch from 65d40ac to b57c05a Compare May 5, 2021 21:40
@fbcouch fbcouch marked this pull request as ready for review May 6, 2021 01:27
@chinmaygarde
Copy link
Member

@chunhtai This seems ready for review. Are you able to take a look please?

@@ -289,6 +289,8 @@ - (void)setViewController:(FlutterViewController*)viewController {

- (void)attachView {
self.iosPlatformView->attachView();
_textInputPlugin.get().viewController = [self viewController];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if the view controller changes during the lifetime of the flutterengine?
also flutter holdsa weak ptr to the viewcontroller, I am not sure what will happen to the assign property of _textInputPlugin if the pointer is nulled in flutterengine, cc @gaaclarke

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chunhtai I also set this property below on like 476, I'm not sure whether or not that would fix the issue – how would I test that? I don't know when the view controller would change

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the viewcontroller of the textinputplugin need to be set in FlutterEngine.setViewController, and it also need to nil it in FlutterEngine.notifyViewControllerDeallocated. To test this, you can swap out view controller when you run the FlutterEngine and see if things still work

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chunhtai I may need some more guidance on how to test this...I tried the following, but the notifyViewControllerDeallocated method doesn't seem to get hit when swapping out the view controller – attachView does get called again, so I think it is "working":

@UIApplicationMain
@objc class AppDelegate: FlutterAppDelegate {
  override func application(
    _ application: UIApplication,
    didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?
  ) -> Bool {
    
    var controller = window.rootViewController as! FlutterViewController
    
    let criticChannel = FlutterMethodChannel(name: "test.viewController", binaryMessenger: controller as! FlutterBinaryMessenger)
    criticChannel.setMethodCallHandler { (call: FlutterMethodCall, result: @escaping FlutterResult) -> Void in
        if call.method == "swap" {
            let engine = controller.engine!
            engine.viewController = nil
            self.window.rootViewController = FlutterViewController(engine: engine, nibName: controller.nibName, bundle: controller.nibBundle)
            controller = self.window.rootViewController as! FlutterViewController
            result(nil)
        } else {
            result(FlutterMethodNotImplemented)
        }
    }
    
    GeneratedPluginRegistrant.register(with: self)
    return super.application(application, didFinishLaunchingWithOptions: launchOptions)
  }
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chunhtai I updated the setupIndirectScribbleInteraction method to take in the FlutterViewController instead of setting it directly in the engine, that way we're setting up that interaction every time the view controller changes. I also added a resetViewController method to nil the viewController from that notifyViewControllerDeallocated method, though I still haven't managed to actually have that get hit 🤷‍♂️

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you will need to hold a strong reference FlutterViewController somewhere and pass that into the flutter engine. and then nil the strong reference, this should cause garbage collection to dealloc the FlutterViewController, which will then call the notifyViewControllerDeallocated.

can you write a unit test for this? so that we don't break this unintentionally.

}

- (void)scribbleInteractionBegan {
[_textInputChannel.get() invokeMethod:@"TextInputClient.scribbleInteractionBegan" arguments:nil];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should there be a framework pr associated with the change? it may be easier to review if you can also create the framework part pr

Copy link
Contributor Author

@fbcouch fbcouch May 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chunhtai Yes, I should have put a link to it in the description. Here it is: flutter/flutter#75472

fbcouch added 2 commits May 20, 2021 17:07
…tPlugin to nil and set _hasScribbleInteraction to false when the view controller becomes nil
…leInteraction method and set up the interaction every time the viewController changes
Copy link
Member

@gaaclarke gaaclarke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I gave this a quick glance, seems promising thanks. In its current state there is still a fair bit of slapdash memory management. You're going to have to clean that up before merging.

@@ -471,6 +473,7 @@ - (void)setupChannels {

_textInputPlugin.reset([[FlutterTextInputPlugin alloc] init]);
_textInputPlugin.get().textInputDelegate = self;
[_textInputPlugin.get() setupIndirectScribbleInteraction:[self viewController]];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/[self viewController]/self.viewController; here and everywhere else.

- (void)focusElement:(UIScribbleElementIdentifier)elementIdentifier
atPoint:(CGPoint)referencePoint
result:(id)callback;
- (void)requestElementsInRect:(CGRect)rect result:(id)callback;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this parameter id? Please specify a specific type.


#pragma mark - Methods related to Scribble support

- (void)setupIndirectScribbleInteraction:(FlutterViewController*)viewController {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There already is a public setter for this via the property setViewController. If people are expected to use this instead considering putting this logic in that method or making the property readonly.

@@ -505,6 +557,9 @@ @implementation FlutterTextInputView {
const char* _selectionAffinity;
FlutterTextRange* _selectedTextRange;
CGRect _cachedFirstRect;
NSArray* _selectionRects;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use property instead of ivar so the memory management is explicit. I don't see a release for this array so it's probably leaking.

[NSNumber numberWithFloat:[rect[3] floatValue]], [NSNumber numberWithInt:[rect[4] intValue]]
]];
}
_selectionRects = rectsAsRect;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is leaking, making it a property as I mentioned above will help you fix that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to actually used this as a property to get the memory management benefits self.selectionRects = rectsAsRect;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, the issue is that this is the setter for that property – so if I reference it as self.selectionRects, I get an error about _selectionRects not being referenced:

../../flutter/shell/platform/darwin/ios/framework/Source/FlutterTextInputPlugin.mm:1182:1: error: ivar '_selectionRects' which backs the property is not referenced in this property's accessor [-Werror,-Wunused-property-ivar]```

So, it looks to me like `setSelectedTextRangeLocal` does a similar thing where it sets `_selectedTextRange` directly, so I'm not sure what the right thing to do is here – perhaps I just need to rename this method to not override the setter?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would adding autorelease to the rectsAsRect declaration help?

CGPointMake(rect.origin.x + rect.size.width, rect.origin.y + rect.size.height * 0.5);
float yDist = abs(pointForComparison.y - point.y);
float xDist = abs(pointForComparison.x - point.x);
if (yDist < _closestY - 1 ||
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like duplicated logic from above, more reason to make it a static function.

- (UITextPlaceholder*)insertTextPlaceholderWithSize:(CGSize)size {
[_textInputDelegate insertTextPlaceholderWithSize:size withClient:_textInputClient];
_hasPlaceholder = YES;
return [[FlutterTextPlaceholder alloc] init];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't return values that aren't released by the autorelease pool (read as this PR won't be accepted unless that change is made).

@@ -1270,9 +1549,11 @@ @interface FlutterTextInputPlugin ()

@implementation FlutterTextInputPlugin {
NSTimer* _enableFlutterTextInputViewAccessibilityTimer;
NSMutableDictionary<UIScribbleElementIdentifier, NSValue*>* _scribbleElements;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use a property so the memory management is explicit

// TODO(fbcouch): only do this on iPadOS?
// This is necessary to set up where the scribble interactable element will be
_activeView.frame =
CGRectMake([dictionary[@"transform"][12] intValue], [dictionary[@"transform"][13] intValue],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

12 and 13 should be annotated variables.

@chinmaygarde
Copy link
Member

@fbcouch Any updates based on the Aarons review?

@fbcouch
Copy link
Contributor Author

fbcouch commented Jun 3, 2021

@chinmaygarde I am hoping to address the new comments either tomorrow or early next week (thanks @gaaclarke for the feedback – I will readily admit I don't really know what I'm doing with memory management in objc, but I'll try to make the adjustments requested)

@fbcouch
Copy link
Contributor Author

fbcouch commented Oct 25, 2021

@gaaclarke @justinmc Yes, that's correct, that example should work just fine.

@gaaclarke
Copy link
Member

Okay so I tested it locally. It mostly works. Drawing a line through text to select it works, but the highlight of the region isn't rendered:
IMG_0005

If I background the app, then bring it back, the text is selected correctly.

@gaaclarke
Copy link
Member

Hey @justinmc since this mostly works, do you think we can land this behind a flag? So @fbcouch doesn't have to maintain these 2 big PR's? Then they could just land little patches to fix things up and when we are happy with it we can remove the flag?

@@ -397,6 +397,65 @@ static FlutterAutofillType autofillTypeOf(NSDictionary* configuration) {
return FlutterAutofillTypeNone;
}

static BOOL isApproximatelyEqual(float x, float y, float delta) {
return x >= y - delta && x <= y + delta;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fyi: fabsf(x - y) <= delta is less operations (3-5 versus 3)

static BOOL isScribbleAvailable() {
if (@available(iOS 14.0, *)) {
NSString* deviceModel = (NSString*)[UIDevice currentDevice].model;
if ([[deviceModel substringWithRange:NSMakeRange(0, 4)] isEqualToString:@"iPad"]) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if ( UI_USER_INTERFACE_IDIOM() == UIUserInterfaceIdiomPad ) is another way

@@ -625,14 +753,11 @@ - (instancetype)init {
_smartQuotesType = UITextSmartQuotesTypeYes;
_smartDashesType = UITextSmartDashesTypeYes;
}
_selectionRects = @[];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a potential crash, it should be _selectionRects = [@[] retain]; or _selectionRects = [[NSArray alloc] init]. The literal is on the autorelease pool and can be deleted before you access it.

action == @selector(delete:) || action == @selector(makeTextWritingDirectionLeftToRight:) ||
action == @selector(makeTextWritingDirectionRightToLeft:) ||
action == @selector(toggleBoldface:) || action == @selector(toggleItalics:) ||
action == @selector(toggleUnderline:) || action == NSSelectorFromString(@"_define:") ||
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_define: is a private selector? This may get flagged by iOS app review.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good point...we might have to make a whitelist instead of a blacklist to get around that...I'll look into that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I think we can just return NO for everything – I think the big thing I was worried about was saying no to the accessibility ones, but I just tried it with VoiceOver and it seems to still be able to navigate/read the TextInput.


_scribbleFocusStatus = FlutterScribbleFocusStatusUnfocused;
[self resetScribbleInteractionStatusIfEnding];
_selectionRects = copiedRects;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is leaking the old value of _selectionRects, use self.selectionRects = copiedRects; [copiedRects release];

@justinmc
Copy link
Contributor

I'm on board with merging this behind a flag 👍

@fbcouch
Copy link
Contributor Author

fbcouch commented Nov 5, 2021

@gaaclarke Thanks for the review and testing it out. This is interesting, because it seems that a MaterialTextField works correctly, but CupertinoTextField doesn't. I'll dig into that, but do you or @justinmc know off the top of your head if there's some difference in how they display that highlight?

I like the idea of landing behind a flag – how difficult will that be to enable in an app? (We're currently frozen on flutter 1.22.6 to avoid having to make multiple custom engine versions)

@fbcouch
Copy link
Contributor Author

fbcouch commented Nov 5, 2021

So the issue with the CupertinoTextField appears to be that focusing with the pencil doesn't trigger the _handleFocusChanged callback, so the widget doesn't get rebuilt knowing that it has focus. I'll see if I can figure out a quick fix for that.

@fbcouch
Copy link
Contributor Author

fbcouch commented Nov 5, 2021

Yep, it wasn't adding the focus listener in initState like the Material version does (flutter/flutter@f309302). So, @gaaclarke could you test again? Thanks!

@fbcouch fbcouch requested a review from gaaclarke November 5, 2021 17:00
Copy link
Member

@gaaclarke gaaclarke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested it out and it works for me. I looked through the code and it looks good now. I don't see any obvious errors. Thanks Jami, awesome job!

if (@available(iOS 14.0, *)) {
UIView* parentView = viewResponder.view;
if (parentView != nil) {
UIIndirectScribbleInteraction* _scribbleInteraction =
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/_scribbleInteraction/scribbleInteraction

@gaaclarke gaaclarke merged commit 1ae721c into flutter:master Nov 8, 2021
@fbcouch
Copy link
Contributor Author

fbcouch commented Nov 8, 2021

@gaaclarke Awesome, thanks for all your time giving feedback on this!

@morio77
Copy link

morio77 commented Dec 29, 2021

@fbcouch

I am using the flutter master channel.

Is it a bug that when I draw with Apple Pencil with scribble turned on in iPad settings, scribble trails are displayed in most widgets including AppBar widget?

Also, is this related to this PR?

By the way, the problem does not occur in flutter 2.0.0 to 2.8.1, only in the master channel.

@fbcouch
Copy link
Contributor Author

fbcouch commented Dec 29, 2021

@morio77 I think that is an issue that will be resolved once the framework PR is merged (see flutter/flutter#75472 ). If you use that branch instead, things should work properly. I think the issue is that the TextInputClient.focusElement method invocation is not being responded to properly by the framework, so the completion method is not getting called in a timely manner in the - focusElementIfNeeded method

@@ -331,6 +334,7 @@ - (void)setViewController:(FlutterViewController*)viewController {

- (void)attachView {
self.iosPlatformView->attachView();
[_textInputPlugin.get() setupIndirectScribbleInteraction:self.viewController];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Came across this while looking into refactoring attachView. Does it make sense to move this line somewhere else? setupIndirectScribbleInteraction:self doesn't sound like should be a part of "attachView"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe there's a better place for it...iirc it's purpose here is to make sure the indirect scribble interaction is set up along with the view – that is something that needs to always be there so that we can capture scribble interactions that happen outside of text areas

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants