This repository was archived by the owner on Feb 25, 2025. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 6k
Support Scribble Handwriting #24224
Merged
Merged
Support Scribble Handwriting #24224
Changes from 62 commits
Commits
Show all changes
63 commits
Select commit
Hold shift + click to select a range
8278027
[scribble+master] [scribble] Very rough proof of concept
fbcouch 9104fe5
[scribble+master] Addressing PR feedback: remove scribbleInProgress f…
fbcouch abbdbf9
[scribble+master] Improve closestPositionToPoint algorithm
fbcouch 92268b9
[scribble+master] Add tests, refactor the closestPositionToPoint meth…
fbcouch 2af5fe9
[scribble+master] Run clang-format
fbcouch 0844910
[scribble+master] Pass calls to insertTextPlaceholderWithSize and rem…
fbcouch 3e87027
[scribble+master] Update scribble support to handle multi-byte charac…
fbcouch d230e1b
[scribble+master-updated] Clean up some misses during rebasing, remov…
fbcouch 6035793
[scribble+master-updated] Fixup my tests
fbcouch 9105c21
[scribble+master] Clear selection rects on insert text with placehold…
fbcouch 487bab4
[scribble+master] Add placeholder selection rects when inserting text…
fbcouch 7c784c3
[scribble+master] Add back autofill hider stuff, which seems to work …
fbcouch b57c05a
[scribble+master] Add Twin Sun, LLC to AUTHORS
fbcouch ed6d050
[scribble+master] Remove NSLogs and some TODOs
fbcouch 5746e60
[scribble+master] Add 'resetViewController' method to FlutterTextInpu…
fbcouch 97c67b4
[scribble+master] Pass FlutterViewController into setupIndirectScribb…
fbcouch deb3ae9
[scribble+master] Address PR feedback – memory management, extract co…
fbcouch 9d2e501
[scribble+master] Remove FlutterViewController import, remove unneces…
fbcouch 67d471a
[scribble+master] Use dynamic instead of synthesize for inherited pro…
fbcouch d10c767
[scribble+master] Remove some useless copy calls, extract logic of fi…
fbcouch 561935f
[scribble+master] Adding some (I think) missing autoreleases and clea…
fbcouch 4be3dd2
[scribble+master] Remove a couple more unnecessary synthesize calls, …
fbcouch 25f4100
[scribble+master] Swap back to synthesize for FlutterTextSelectionRec…
fbcouch 34db3dd
[scribble+master] Genericize selectionRects NSArray
fbcouch f6adcd9
[scribble+master] Merge branch 'master' into feature/scribble+master
fbcouch 82f9ce6
[scribble+master] Fix formatting issues and swap from _reusableInputV…
fbcouch 85f8665
Couple more memory things
fbcouch ea37dfa
Clear scribbleFocused flag in touchesBegan, insert/delete text rather…
fbcouch 671f29c
did a bit of cleanup (the easy stuff)
gaaclarke 6eed8a7
extract FlutterIndirectScribbleDelegate interface to handle requestEl…
fbcouch 39a3579
Merge branch 'master' into feature/scribble+master
fbcouch 6ac1cb6
Update new member of FlutterTextInputDelegate to pass caller as first…
fbcouch b123246
Refactor out FlutterViewResponder protocol, swap focused/focusing boo…
fbcouch 46bacea
Add position member to FlutterTextSelectionRect and use that to repre…
fbcouch a622e83
Fix linter errors
fbcouch 640267e
Fix tests to use new selection rects and delegate style
fbcouch f2d67b5
Add some more tests for scribble functions
fbcouch 7bfe2e9
Add new headers to licenses_flutter golden file
fbcouch 8fae879
Rename viewController to viewResponder; move viewResponder prop on th…
fbcouch 4008ea4
More PR changes; provide frame to inputHider so that UIScribbleIntera…
fbcouch c6aa05e
Swap scribbleInProgress over to enum, fix issue where cursor wouldn't…
fbcouch a39f759
Fix up tests, remove logs, prevent native toolbar from popping up
fbcouch 7a4e440
Add back errant removal of setSelectedTextRangeLocal call in replaceR…
fbcouch 3ec718b
Merge branch 'master' into feature/scribble+master
fbcouch a95f2d9
Add workarounds for UITextInteraction-related selection and gesture p…
fbcouch c0aeba0
Change up workaround a bit to move add the UITextInteraction in -begi…
fbcouch ae8f536
Move setup of UIScribbleInteraction to the init method of FlutterText…
fbcouch f106b91
Remove remaining NSLog
fbcouch a141a14
Add guard clause to selectionRectsForRange to prevent crash when chan…
fbcouch 2303259
Merge branch 'master' into feature/scribble+master
fbcouch fc8c9b1
Clean up some comments
fbcouch a9e02ae
Merge branch 'master' into feature/scribble+master
fbcouch 3966913
Add comment explaining isCloserVertically
fbcouch 7d75e7e
Merge branch 'master' into feature/scribble+master
fbcouch cbfd5b3
Fix compile error (need to add flutterTextInputView art to - updateEd…
fbcouch 8ea77a7
Update - updateEditingClient:withDelta: in tests
fbcouch 4ee848f
Merge branch 'master' into feature/scribble+master
fbcouch 2dd299f
Merge branch 'master' into feature/scribble+master
fbcouch 75e9b06
Merge branch 'feature/scribble+master' of github.com:twinsunllc/engin…
fbcouch 99de581
Merge branch 'master' into feature/scribble+master
fbcouch 07e195c
Incorporate feedback
fbcouch 975177b
Merge branch 'master' into feature/scribble+master
fbcouch ee11699
Rename scribbleInteraction variable
fbcouch File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -20,4 +20,5 @@ Hidenori Matsubayashi <[email protected]> | |
Sarbagya Dhaubanjar <[email protected]> | ||
Callum Moffat <[email protected]> | ||
Koutaro Mori <[email protected]> | ||
TheOneWithTheBraid <[email protected]> | ||
TheOneWithTheBraid <[email protected]> | ||
Twin Sun, LLC <[email protected]> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -20,6 +20,7 @@ | |
#import "flutter/shell/platform/darwin/common/command_line.h" | ||
#import "flutter/shell/platform/darwin/ios/framework/Source/FlutterBinaryMessengerRelay.h" | ||
#import "flutter/shell/platform/darwin/ios/framework/Source/FlutterDartProject_Internal.h" | ||
#import "flutter/shell/platform/darwin/ios/framework/Source/FlutterIndirectScribbleDelegate.h" | ||
#import "flutter/shell/platform/darwin/ios/framework/Source/FlutterObservatoryPublisher.h" | ||
#import "flutter/shell/platform/darwin/ios/framework/Source/FlutterPlatformPlugin.h" | ||
#import "flutter/shell/platform/darwin/ios/framework/Source/FlutterTextInputDelegate.h" | ||
|
@@ -42,7 +43,9 @@ @interface FlutterEngineRegistrar : NSObject <FlutterPluginRegistrar> | |
- (instancetype)initWithPlugin:(NSString*)pluginKey flutterEngine:(FlutterEngine*)flutterEngine; | ||
@end | ||
|
||
@interface FlutterEngine () <FlutterTextInputDelegate, FlutterBinaryMessenger> | ||
@interface FlutterEngine () <FlutterIndirectScribbleDelegate, | ||
FlutterTextInputDelegate, | ||
FlutterBinaryMessenger> | ||
// Maintains a dictionary of plugin names that have registered with the engine. Used by | ||
// FlutterEngineRegistrar to implement a FlutterPluginRegistrar. | ||
@property(nonatomic, readonly) NSMutableDictionary* pluginPublications; | ||
|
@@ -331,6 +334,7 @@ - (void)setViewController:(FlutterViewController*)viewController { | |
|
||
- (void)attachView { | ||
self.iosPlatformView->attachView(); | ||
[_textInputPlugin.get() setupIndirectScribbleInteraction:self.viewController]; | ||
} | ||
|
||
- (void)setFlutterViewControllerWillDeallocObserver:(id<NSObject>)observer { | ||
|
@@ -355,6 +359,7 @@ - (void)notifyViewControllerDeallocated { | |
platform_view->SetOwnerViewController({}); | ||
} | ||
} | ||
[_textInputPlugin.get() resetViewResponder]; | ||
_viewController.reset(); | ||
} | ||
|
||
|
@@ -514,6 +519,8 @@ - (void)setupChannels { | |
|
||
_textInputPlugin.reset([[FlutterTextInputPlugin alloc] init]); | ||
_textInputPlugin.get().textInputDelegate = self; | ||
_textInputPlugin.get().indirectScribbleDelegate = self; | ||
[_textInputPlugin.get() setupIndirectScribbleInteraction:self.viewController]; | ||
|
||
_platformPlugin.reset([[FlutterPlatformPlugin alloc] initWithEngine:[self getWeakPtr]]); | ||
|
||
|
@@ -720,22 +727,30 @@ - (void)notifyLowMemory { | |
|
||
#pragma mark - Text input delegate | ||
|
||
- (void)updateEditingClient:(int)client withState:(NSDictionary*)state { | ||
- (void)flutterTextInputView:(FlutterTextInputView*)textInputView | ||
updateEditingClient:(int)client | ||
withState:(NSDictionary*)state { | ||
[_textInputChannel.get() invokeMethod:@"TextInputClient.updateEditingState" | ||
arguments:@[ @(client), state ]]; | ||
} | ||
|
||
- (void)updateEditingClient:(int)client withState:(NSDictionary*)state withTag:(NSString*)tag { | ||
- (void)flutterTextInputView:(FlutterTextInputView*)textInputView | ||
updateEditingClient:(int)client | ||
withState:(NSDictionary*)state | ||
withTag:(NSString*)tag { | ||
[_textInputChannel.get() invokeMethod:@"TextInputClient.updateEditingStateWithTag" | ||
arguments:@[ @(client), @{tag : state} ]]; | ||
} | ||
|
||
- (void)updateEditingClient:(int)client withDelta:(NSDictionary*)delta { | ||
- (void)flutterTextInputView:(FlutterTextInputView*)textInputView | ||
updateEditingClient:(int)client | ||
withDelta:(NSDictionary*)delta { | ||
[_textInputChannel.get() invokeMethod:@"TextInputClient.updateEditingStateWithDeltas" | ||
arguments:@[ @(client), delta ]]; | ||
} | ||
|
||
- (void)updateFloatingCursor:(FlutterFloatingCursorDragState)state | ||
- (void)flutterTextInputView:(FlutterTextInputView*)textInputView | ||
updateFloatingCursor:(FlutterFloatingCursorDragState)state | ||
withClient:(int)client | ||
withPosition:(NSDictionary*)position { | ||
NSString* stateString; | ||
|
@@ -754,7 +769,9 @@ - (void)updateFloatingCursor:(FlutterFloatingCursorDragState)state | |
arguments:@[ @(client), stateString, position ]]; | ||
} | ||
|
||
- (void)performAction:(FlutterTextInputAction)action withClient:(int)client { | ||
- (void)flutterTextInputView:(FlutterTextInputView*)textInputView | ||
performAction:(FlutterTextInputAction)action | ||
withClient:(int)client { | ||
NSString* actionString; | ||
switch (action) { | ||
case FlutterTextInputActionUnspecified: | ||
|
@@ -799,15 +816,63 @@ - (void)performAction:(FlutterTextInputAction)action withClient:(int)client { | |
arguments:@[ @(client), actionString ]]; | ||
} | ||
|
||
- (void)showAutocorrectionPromptRectForStart:(NSUInteger)start | ||
end:(NSUInteger)end | ||
withClient:(int)client { | ||
- (void)flutterTextInputView:(FlutterTextInputView*)textInputView | ||
showAutocorrectionPromptRectForStart:(NSUInteger)start | ||
end:(NSUInteger)end | ||
withClient:(int)client { | ||
[_textInputChannel.get() invokeMethod:@"TextInputClient.showAutocorrectionPromptRect" | ||
arguments:@[ @(client), @(start), @(end) ]]; | ||
} | ||
|
||
#pragma mark - FlutterViewEngineDelegate | ||
|
||
- (void)flutterTextInputView:(FlutterTextInputView*)textInputView showToolbar:(int)client { | ||
[_textInputChannel.get() invokeMethod:@"TextInputClient.showToolbar" arguments:@[ @(client) ]]; | ||
} | ||
|
||
- (void)flutterTextInputPlugin:(FlutterTextInputPlugin*)textInputPlugin | ||
focusElement:(UIScribbleElementIdentifier)elementIdentifier | ||
atPoint:(CGPoint)referencePoint | ||
result:(FlutterResult)callback { | ||
[_textInputChannel.get() | ||
invokeMethod:@"TextInputClient.focusElement" | ||
arguments:@[ elementIdentifier, @(referencePoint.x), @(referencePoint.y) ] | ||
result:callback]; | ||
} | ||
|
||
- (void)flutterTextInputPlugin:(FlutterTextInputPlugin*)textInputPlugin | ||
requestElementsInRect:(CGRect)rect | ||
result:(FlutterResult)callback { | ||
[_textInputChannel.get() | ||
invokeMethod:@"TextInputClient.requestElementsInRect" | ||
arguments:@[ @(rect.origin.x), @(rect.origin.y), @(rect.size.width), @(rect.size.height) ] | ||
result:callback]; | ||
} | ||
|
||
- (void)flutterTextInputViewScribbleInteractionBegan:(FlutterTextInputView*)textInputView { | ||
[_textInputChannel.get() invokeMethod:@"TextInputClient.scribbleInteractionBegan" arguments:nil]; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. should there be a framework pr associated with the change? it may be easier to review if you can also create the framework part pr There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @chunhtai Yes, I should have put a link to it in the description. Here it is: flutter/flutter#75472 |
||
} | ||
|
||
- (void)flutterTextInputViewScribbleInteractionFinished:(FlutterTextInputView*)textInputView { | ||
[_textInputChannel.get() invokeMethod:@"TextInputClient.scribbleInteractionFinished" | ||
arguments:nil]; | ||
} | ||
|
||
- (void)flutterTextInputView:(FlutterTextInputView*)textInputView | ||
insertTextPlaceholderWithSize:(CGSize)size | ||
withClient:(int)client { | ||
[_textInputChannel.get() invokeMethod:@"TextInputClient.insertTextPlaceholder" | ||
arguments:@[ @(client), @(size.width), @(size.height) ]]; | ||
} | ||
|
||
- (void)flutterTextInputView:(FlutterTextInputView*)textInputView | ||
removeTextPlaceholder:(int)client { | ||
[_textInputChannel.get() invokeMethod:@"TextInputClient.removeTextPlaceholder" | ||
arguments:@[ @(client) ]]; | ||
} | ||
|
||
#pragma mark - Screenshot Delegate | ||
|
||
- (flutter::Rasterizer::Screenshot)takeScreenshot:(flutter::Rasterizer::ScreenshotType)type | ||
asBase64Encoded:(BOOL)base64Encode { | ||
FML_DCHECK(_shell) << "Cannot takeScreenshot without a shell"; | ||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
24 changes: 24 additions & 0 deletions
24
shell/platform/darwin/ios/framework/Source/FlutterIndirectScribbleDelegate.h
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
// Copyright 2013 The Flutter Authors. All rights reserved. | ||
// Use of this source code is governed by a BSD-style license that can be | ||
// found in the LICENSE file. | ||
|
||
#ifndef SHELL_PLATFORM_IOS_FRAMEWORK_SOURCE_FLUTTERINDIRECTSCRIBBLEDELEGATE_H_ | ||
#define SHELL_PLATFORM_IOS_FRAMEWORK_SOURCE_FLUTTERINDIRECTSCRIBBLEDELEGATE_H_ | ||
|
||
#import <Foundation/Foundation.h> | ||
|
||
NS_ASSUME_NONNULL_BEGIN | ||
@class FlutterTextInputPlugin; | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This should have |
||
@protocol FlutterIndirectScribbleDelegate <NSObject> | ||
- (void)flutterTextInputPlugin:(FlutterTextInputPlugin*)textInputPlugin | ||
focusElement:(UIScribbleElementIdentifier)elementIdentifier | ||
atPoint:(CGPoint)referencePoint | ||
result:(FlutterResult)callback; | ||
- (void)flutterTextInputPlugin:(FlutterTextInputPlugin*)textInputPlugin | ||
requestElementsInRect:(CGRect)rect | ||
result:(FlutterResult)callback; | ||
@end | ||
NS_ASSUME_NONNULL_END | ||
|
||
#endif // SHELL_PLATFORM_IOS_FRAMEWORK_SOURCE_FLUTTERINDIRECTSCRIBBLEDELEGATE_H_ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Came across this while looking into refactoring
attachView
. Does it make sense to move this line somewhere else?setupIndirectScribbleInteraction:self
doesn't sound like should be a part of "attachView"There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe there's a better place for it...iirc it's purpose here is to make sure the indirect scribble interaction is set up along with the view – that is something that needs to always be there so that we can capture scribble interactions that happen outside of text areas