-
Notifications
You must be signed in to change notification settings - Fork 6k
Conversation
7f5494d
to
700fcd9
Compare
shell/platform/darwin/ios/framework/Source/FlutterTextInputPlugin.mm
Outdated
Show resolved
Hide resolved
[scribble] Implement some more methods [scribble] Clean up some things, add some logging and forward touch event selectors to superview to try to fix this issue where the cursor cant be moved easily [scribble] Use UIScribbleInteractionDelegate to disable setMarkedText during scribble; add a reference to the viewController to send touch events to [scribble] Do not send showAutocorrectionPromptRectForStart from firstRectForRange while scribble is in progress - that's what was causing the last character to always be selected [scribble] Send 'showToolbar' message over the text input channel and add scribbleInProgress key to the editing state [scribble] Add UIIndirectScribbleInteraction and Delegate [scribble] Focuses, but not smoothly [scribble] Able to focus in 'correct' position [scribble] Track scribble rects on the framework side [scribble] Clean up before rebase
…rom TextInputState, update separately
…ods so that the withinRange version is called by the other
…oveTextPlaceholder to the framework
…e FlutterTextInputViewAccessibilityHider from views, as it prevents direct scribble interactions
…er so text doesn't flyaway to the end of the string for some reason
…; support focusing immediately
…now...not sure how that's the case, but OK; also fix some things so that the weird issue with zero width joiner characters seems to be fixed now as well
65d40ac
to
b57c05a
Compare
@chunhtai This seems ready for review. Are you able to take a look please? |
@@ -289,6 +289,8 @@ - (void)setViewController:(FlutterViewController*)viewController { | |||
|
|||
- (void)attachView { | |||
self.iosPlatformView->attachView(); | |||
_textInputPlugin.get().viewController = [self viewController]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what if the view controller changes during the lifetime of the flutterengine?
also flutter holdsa weak ptr to the viewcontroller, I am not sure what will happen to the assign property of _textInputPlugin if the pointer is nulled in flutterengine, cc @gaaclarke
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@chunhtai I also set this property below on like 476, I'm not sure whether or not that would fix the issue – how would I test that? I don't know when the view controller would change
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the viewcontroller of the textinputplugin need to be set in FlutterEngine.setViewController, and it also need to nil it in FlutterEngine.notifyViewControllerDeallocated. To test this, you can swap out view controller when you run the FlutterEngine and see if things still work
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@chunhtai I may need some more guidance on how to test this...I tried the following, but the notifyViewControllerDeallocated method doesn't seem to get hit when swapping out the view controller – attachView does get called again, so I think it is "working":
@UIApplicationMain
@objc class AppDelegate: FlutterAppDelegate {
override func application(
_ application: UIApplication,
didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?
) -> Bool {
var controller = window.rootViewController as! FlutterViewController
let criticChannel = FlutterMethodChannel(name: "test.viewController", binaryMessenger: controller as! FlutterBinaryMessenger)
criticChannel.setMethodCallHandler { (call: FlutterMethodCall, result: @escaping FlutterResult) -> Void in
if call.method == "swap" {
let engine = controller.engine!
engine.viewController = nil
self.window.rootViewController = FlutterViewController(engine: engine, nibName: controller.nibName, bundle: controller.nibBundle)
controller = self.window.rootViewController as! FlutterViewController
result(nil)
} else {
result(FlutterMethodNotImplemented)
}
}
GeneratedPluginRegistrant.register(with: self)
return super.application(application, didFinishLaunchingWithOptions: launchOptions)
}
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@chunhtai I updated the setupIndirectScribbleInteraction
method to take in the FlutterViewController
instead of setting it directly in the engine, that way we're setting up that interaction every time the view controller changes. I also added a resetViewController method to nil the viewController from that notifyViewControllerDeallocated
method, though I still haven't managed to actually have that get hit 🤷♂️
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you will need to hold a strong reference FlutterViewController somewhere and pass that into the flutter engine. and then nil the strong reference, this should cause garbage collection to dealloc the FlutterViewController, which will then call the notifyViewControllerDeallocated.
can you write a unit test for this? so that we don't break this unintentionally.
} | ||
|
||
- (void)scribbleInteractionBegan { | ||
[_textInputChannel.get() invokeMethod:@"TextInputClient.scribbleInteractionBegan" arguments:nil]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should there be a framework pr associated with the change? it may be easier to review if you can also create the framework part pr
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@chunhtai Yes, I should have put a link to it in the description. Here it is: flutter/flutter#75472
…tPlugin to nil and set _hasScribbleInteraction to false when the view controller becomes nil
…leInteraction method and set up the interaction every time the viewController changes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I gave this a quick glance, seems promising thanks. In its current state there is still a fair bit of slapdash memory management. You're going to have to clean that up before merging.
@@ -471,6 +473,7 @@ - (void)setupChannels { | |||
|
|||
_textInputPlugin.reset([[FlutterTextInputPlugin alloc] init]); | |||
_textInputPlugin.get().textInputDelegate = self; | |||
[_textInputPlugin.get() setupIndirectScribbleInteraction:[self viewController]]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/[self viewController]/self.viewController; here and everywhere else.
- (void)focusElement:(UIScribbleElementIdentifier)elementIdentifier | ||
atPoint:(CGPoint)referencePoint | ||
result:(id)callback; | ||
- (void)requestElementsInRect:(CGRect)rect result:(id)callback; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this parameter id
? Please specify a specific type.
|
||
#pragma mark - Methods related to Scribble support | ||
|
||
- (void)setupIndirectScribbleInteraction:(FlutterViewController*)viewController { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There already is a public setter for this via the property setViewController
. If people are expected to use this instead considering putting this logic in that method or making the property readonly.
@@ -505,6 +557,9 @@ @implementation FlutterTextInputView { | |||
const char* _selectionAffinity; | |||
FlutterTextRange* _selectedTextRange; | |||
CGRect _cachedFirstRect; | |||
NSArray* _selectionRects; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use property instead of ivar so the memory management is explicit. I don't see a release for this array so it's probably leaking.
[NSNumber numberWithFloat:[rect[3] floatValue]], [NSNumber numberWithInt:[rect[4] intValue]] | ||
]]; | ||
} | ||
_selectionRects = rectsAsRect; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is leaking, making it a property as I mentioned above will help you fix that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You need to actually used this as a property to get the memory management benefits self.selectionRects = rectsAsRect;
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So, the issue is that this is the setter for that property – so if I reference it as self.selectionRects
, I get an error about _selectionRects
not being referenced:
../../flutter/shell/platform/darwin/ios/framework/Source/FlutterTextInputPlugin.mm:1182:1: error: ivar '_selectionRects' which backs the property is not referenced in this property's accessor [-Werror,-Wunused-property-ivar]```
So, it looks to me like `setSelectedTextRangeLocal` does a similar thing where it sets `_selectedTextRange` directly, so I'm not sure what the right thing to do is here – perhaps I just need to rename this method to not override the setter?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would adding autorelease
to the rectsAsRect
declaration help?
CGPointMake(rect.origin.x + rect.size.width, rect.origin.y + rect.size.height * 0.5); | ||
float yDist = abs(pointForComparison.y - point.y); | ||
float xDist = abs(pointForComparison.x - point.x); | ||
if (yDist < _closestY - 1 || |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like duplicated logic from above, more reason to make it a static function.
- (UITextPlaceholder*)insertTextPlaceholderWithSize:(CGSize)size { | ||
[_textInputDelegate insertTextPlaceholderWithSize:size withClient:_textInputClient]; | ||
_hasPlaceholder = YES; | ||
return [[FlutterTextPlaceholder alloc] init]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You shouldn't return values that aren't released by the autorelease pool (read as this PR won't be accepted unless that change is made).
@@ -1270,9 +1549,11 @@ @interface FlutterTextInputPlugin () | |||
|
|||
@implementation FlutterTextInputPlugin { | |||
NSTimer* _enableFlutterTextInputViewAccessibilityTimer; | |||
NSMutableDictionary<UIScribbleElementIdentifier, NSValue*>* _scribbleElements; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
use a property so the memory management is explicit
// TODO(fbcouch): only do this on iPadOS? | ||
// This is necessary to set up where the scribble interactable element will be | ||
_activeView.frame = | ||
CGRectMake([dictionary[@"transform"][12] intValue], [dictionary[@"transform"][13] intValue], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
12 and 13 should be annotated variables.
@fbcouch Any updates based on the Aarons review? |
@chinmaygarde I am hoping to address the new comments either tomorrow or early next week (thanks @gaaclarke for the feedback – I will readily admit I don't really know what I'm doing with memory management in objc, but I'll try to make the adjustments requested) |
@gaaclarke @justinmc Yes, that's correct, that example should work just fine. |
@@ -397,6 +397,65 @@ static FlutterAutofillType autofillTypeOf(NSDictionary* configuration) { | |||
return FlutterAutofillTypeNone; | |||
} | |||
|
|||
static BOOL isApproximatelyEqual(float x, float y, float delta) { | |||
return x >= y - delta && x <= y + delta; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fyi: fabsf(x - y) <= delta
is less operations (3-5 versus 3)
static BOOL isScribbleAvailable() { | ||
if (@available(iOS 14.0, *)) { | ||
NSString* deviceModel = (NSString*)[UIDevice currentDevice].model; | ||
if ([[deviceModel substringWithRange:NSMakeRange(0, 4)] isEqualToString:@"iPad"]) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if ( UI_USER_INTERFACE_IDIOM() == UIUserInterfaceIdiomPad )
is another way
@@ -625,14 +753,11 @@ - (instancetype)init { | |||
_smartQuotesType = UITextSmartQuotesTypeYes; | |||
_smartDashesType = UITextSmartDashesTypeYes; | |||
} | |||
_selectionRects = @[]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a potential crash, it should be _selectionRects = [@[] retain]
; or _selectionRects = [[NSArray alloc] init]
. The literal is on the autorelease pool and can be deleted before you access it.
action == @selector(delete:) || action == @selector(makeTextWritingDirectionLeftToRight:) || | ||
action == @selector(makeTextWritingDirectionRightToLeft:) || | ||
action == @selector(toggleBoldface:) || action == @selector(toggleItalics:) || | ||
action == @selector(toggleUnderline:) || action == NSSelectorFromString(@"_define:") || |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
_define:
is a private selector? This may get flagged by iOS app review.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a good point...we might have to make a whitelist instead of a blacklist to get around that...I'll look into that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, I think we can just return NO for everything – I think the big thing I was worried about was saying no to the accessibility ones, but I just tried it with VoiceOver and it seems to still be able to navigate/read the TextInput.
|
||
_scribbleFocusStatus = FlutterScribbleFocusStatusUnfocused; | ||
[self resetScribbleInteractionStatusIfEnding]; | ||
_selectionRects = copiedRects; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is leaking the old value of _selectionRects
, use self.selectionRects = copiedRects; [copiedRects release];
I'm on board with merging this behind a flag 👍 |
@gaaclarke Thanks for the review and testing it out. This is interesting, because it seems that a MaterialTextField works correctly, but CupertinoTextField doesn't. I'll dig into that, but do you or @justinmc know off the top of your head if there's some difference in how they display that highlight? I like the idea of landing behind a flag – how difficult will that be to enable in an app? (We're currently frozen on flutter 1.22.6 to avoid having to make multiple custom engine versions) |
So the issue with the CupertinoTextField appears to be that focusing with the pencil doesn't trigger the |
Yep, it wasn't adding the focus listener in initState like the Material version does (flutter/flutter@f309302). So, @gaaclarke could you test again? Thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tested it out and it works for me. I looked through the code and it looks good now. I don't see any obvious errors. Thanks Jami, awesome job!
if (@available(iOS 14.0, *)) { | ||
UIView* parentView = viewResponder.view; | ||
if (parentView != nil) { | ||
UIIndirectScribbleInteraction* _scribbleInteraction = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/_scribbleInteraction/scribbleInteraction
@gaaclarke Awesome, thanks for all your time giving feedback on this! |
I am using the flutter master channel. Is it a bug that when I draw with Apple Pencil with scribble turned on in iPad settings, scribble trails are displayed in most widgets including AppBar widget? Also, is this related to this PR? By the way, the problem does not occur in flutter 2.0.0 to 2.8.1, only in the master channel. |
@morio77 I think that is an issue that will be resolved once the framework PR is merged (see flutter/flutter#75472 ). If you use that branch instead, things should work properly. I think the issue is that the |
@@ -331,6 +334,7 @@ - (void)setViewController:(FlutterViewController*)viewController { | |||
|
|||
- (void)attachView { | |||
self.iosPlatformView->attachView(); | |||
[_textInputPlugin.get() setupIndirectScribbleInteraction:self.viewController]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Came across this while looking into refactoring attachView
. Does it make sense to move this line somewhere else? setupIndirectScribbleInteraction:self
doesn't sound like should be a part of "attachView"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe there's a better place for it...iirc it's purpose here is to make sure the indirect scribble interaction is set up along with the view – that is something that needs to always be there so that we can capture scribble interactions that happen outside of text areas
Framework PR:
flutter/flutter#75472
This PR is the engine side of adding scribble handwriting support. I still need to clean up the NSLogs and add some tests, but looking for feedback on the overall approach. I did incorporate some feedback from @LongCatIsLooong on a previous draft, moving most of the logic of this into the framework and adding a widget there to handle most of the scribble-specific logic.
flutter/flutter#61278
Design doc: https://docs.google.com/document/d/1mjQbsSRQnHuAgMNdouaSgTS-Xv-w57fdKfOUUafWpRo
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist
writing and running engine tests.
///
).If you need help, consider asking for advice on the #hackers-new channel on Discord.