Skip to content

feat: Update OMI Glass firmware files - Add battery setup and charging guide documentation, update firmware.ino, add build scripts and UF2 binary, update source configuration files #2565

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 36 commits into
base: development
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
55cb6f2
Remove pinecone plugin dependencies and disable typesense for local d…
cyhuman Jun 10, 2025
b0c8e0b
Re-enable typesense client initialization
cyhuman Jun 10, 2025
48bd5d6
Merge branch 'main' into openglass-integration
cyhuman Jun 10, 2025
6b1d614
Reapply "feat: Add OpenGlass AI image analysis for multimodal convers…
cyhuman Jun 10, 2025
90e07fa
Remove unused pinecone plugin dependencies from requirements.txt
cyhuman Jun 10, 2025
d272e2f
feat: Add OpenGlass AI image analysis for multimodal conversation int…
mdmohsin7 Jun 10, 2025
7994e2a
feat: Add OpenGlass AI image analysis for multimodal conversation int…
mdmohsin7 Jun 11, 2025
aa13f2e
bump packaging to 24.2
mdmohsin7 Jun 11, 2025
b13f2bf
fix: resolve conversation deadlock and improve UI layout - Add timeou…
cyhuman Jun 11, 2025
9d0bce7
Improve conversation capturing UI: remove image background boxes and …
cyhuman Jun 11, 2025
ba48510
Fix status priority: show 'Capturing' when glasses connected even wit…
cyhuman Jun 11, 2025
2d6309b
Fix/conversation deadlock and UI improvements (#2530)
mdmohsin7 Jun 11, 2025
e31385e
Improve device capture status logic and remove generic Openglass tags…
cyhuman Jun 11, 2025
7cae5c6
Fix auto-summarizer for photo-only conversations and improve device U…
cyhuman Jun 11, 2025
ff4383e
Fix auto-summarizer for photo-only conversations and improve device U…
mdmohsin7 Jun 11, 2025
0981062
refactor: Separate OpenGlass functionality and clean up conversation …
cyhuman Jun 11, 2025
2ac15b8
refactor(websocket): Extract WebSocket utilities from transcribe.py -…
cyhuman Jun 11, 2025
6ba1d27
fix(chat): Fix indentation error in upload_file_chat function - Fixed…
cyhuman Jun 11, 2025
2624bcc
Refactor/openglass conversation processing (#2532)
mdmohsin7 Jun 11, 2025
29a7dea
refactor: simplify OmiGlass system, remove over-engineering - reduced…
cyhuman Jun 11, 2025
dac0ae9
refactor: simplify Redis connection configuration - remove SSL settin…
cyhuman Jun 11, 2025
22a8202
Refactor/openglass conversation processing (#2535)
mdmohsin7 Jun 11, 2025
7779023
Fix image upload filename mismatch and indentation error - Fix Flutte…
cyhuman Jun 11, 2025
ba5f78e
Fix image upload filename mismatch (#2537)
mdmohsin7 Jun 11, 2025
708fe41
Restore chat session functionality and v1 routes for backward compati…
cyhuman Jun 11, 2025
46e5cff
refactor: Move openglass utilities and fix Redis access patterns - Mo…
cyhuman Jun 11, 2025
971c1f3
Refactor/openglass conversation processing (#2538)
mdmohsin7 Jun 11, 2025
3ae52c9
bump ver to 1.0.64+317
mdmohsin7 Jun 11, 2025
6eda5cc
downgrade deps
mdmohsin7 Jun 11, 2025
93d7e85
change device_info_plus ver
mdmohsin7 Jun 11, 2025
259a266
downgrade permision_handler
mdmohsin7 Jun 11, 2025
c1f7fa6
fix: use getattr() for safe photos access in ExternalIntegrationCreat…
cyhuman Jun 12, 2025
ad447fe
fix: use getattr() for safe photos access in ExternalIntegrationCreat…
mdmohsin7 Jun 12, 2025
58abb66
Merge branch 'main' into development
mdmohsin7 Jun 14, 2025
4f709c5
Merge branch 'main' of github.com:BasedHardware/omi into development
beastoin Jun 14, 2025
516545b
feat: Update OMI Glass firmware files - Add battery setup and chargin…
cyhuman Jun 19, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 40 additions & 40 deletions app/ios/Podfile.lock
Original file line number Diff line number Diff line change
Expand Up @@ -437,20 +437,20 @@ EXTERNAL SOURCES:
:path: ".symlinks/plugins/webview_flutter_wkwebview/darwin"

SPEC CHECKSUMS:
app_links: 76b66b60cc809390ca1ad69bfd66b998d2387ac7
app_links: f3e17e4ee5e357b39d8b95290a9b2c299fca71c6
AppAuth: d4f13a8fe0baf391b2108511793e4b479691fb73
audio_session: 9bb7f6c970f21241b19f5a3658097ae459681ba0
awesome_notifications: 0f432b28098d193920b11a44cfa9d2d9313a3888
awesome_notifications_core: 429c28df8746780a474de177e5acde33af87da63
connectivity_plus: cb623214f4e1f6ef8fe7403d580fdad517d2f7dd
device_info_plus: 71ffc6ab7634ade6267c7a93088ed7e4f74e5896
audio_session: 19e9480dbdd4e5f6c4543826b2e8b0e4ab6145fe
awesome_notifications: dd5518ff1c80be03d4f1c40f04da9d9cc2a37af5
awesome_notifications_core: d02eed89738fa362d56cbd372850e9adcd2c6bef
connectivity_plus: 2a701ffec2c0ae28a48cf7540e279787e77c447d
device_info_plus: bf2e3232933866d73fe290f2942f2156cdd10342
DKImagePickerController: 946cec48c7873164274ecc4624d19e3da4c1ef3c
DKPhotoGallery: b3834fecb755ee09a593d7c9e389d8b5d6deed60
file_picker: a0560bc09d61de87f12d246fc47d2119e6ef37be
file_picker: b159e0c068aef54932bb15dc9fd1571818edaf49
Firebase: 1fe1c0a7d9aaea32efe01fbea5f0ebd8d70e53a2
firebase_auth: 83bf106e5ac670dd3a0af27a86be6cba16a85723
firebase_core: 2d4534e7b489907dcede540c835b48981d890943
firebase_messaging: 75bc93a4df25faccad67f6662ae872ac9ae69b64
firebase_auth: 3f532201cbdc7cd6dfc3bfa89affc0c294111e20
firebase_core: 432718558359a8c08762151b5f49bb0f093eb6e0
firebase_messaging: 3b99522baf7480dfb4b7683d2b34e842d577c362
FirebaseAppCheckInterop: a92ba81d0ee3c4cddb1a2e52c668ea51dc63c3ae
FirebaseAuth: c4146bdfdc87329f9962babd24dae89373f49a32
FirebaseAuthInterop: e25b58ecb90f3285085fa2118861a3c9dfdc62ad
Expand All @@ -460,13 +460,13 @@ SPEC CHECKSUMS:
FirebaseInstallations: 9980995bdd06ec8081dfb6ab364162bdd64245c3
FirebaseMessaging: 2b9f56aa4ed286e1f0ce2ee1d413aabb8f9f5cb9
Flutter: e0871f40cf51350855a761d2e70bf5af5b9b5de7
flutter_archive: ad8edfd7f7d1bb12058d05424ba93e27d9930efe
flutter_background_service_ios: 00d31bdff7b4bfe06d32375df358abe0329cf87e
flutter_blue_plus: e5808fc4e5ebc58bb911635f8fdaf5e2b4da2754
flutter_foreground_task: a159d2c2173b33699ddb3e6c2a067045d7cebb89
flutter_native_splash: 6cad9122ea0fad137d23137dd14b937f3e90b145
flutter_silero_vad: 623c22e30420ae174857926385670b9f6e52e4b9
flutter_sound: b9236a5875299aaa4cef1690afd2f01d52a3f890
flutter_archive: cb3e0219e555897ba4b36f166baa1eca394890b9
flutter_background_service_ios: e30e0d3ee69e4cee66272d0c78eacd48c2e94aac
flutter_blue_plus: 4837da7d00cf5d441fdd6635b3a57f936778ea96
flutter_foreground_task: 21ef182ab0a29a3005cc72cd70e5f45cb7f7f817
flutter_native_splash: f71420956eb811e6d310720fee915f1d42852e7a
flutter_silero_vad: bcad5bcce50bd7f63b772ad3f46f9ce1995dd833
flutter_sound: 82aba29055d6feba684d08906e0623217b87bcd3
flutter_sound_core: 427465f72d07ab8c3edbe8ffdde709ddacd3763c
flutter_timezone: ffb07bdad3c6276af8dada0f11978d8a1f8a20bb
frame_sdk: 5d4d7ba36cf53cfdb6aebf563dc0a69e7920a727
Expand All @@ -478,46 +478,46 @@ SPEC CHECKSUMS:
GoogleUtilities: 00c88b9a86066ef77f0da2fab05f65d7768ed8e1
GTMAppAuth: 99fb010047ba3973b7026e45393f51f27ab965ae
GTMSessionFetcher: 5aea5ba6bd522a239e236100971f10cb71b96ab6
image_picker_ios: 7fe1ff8e34c1790d6fff70a32484959f563a928a
image_picker_ios: c560581cceedb403a6ff17f2f816d7fea1421fc1
Instabug: 97a4e694731f46bbc02dbe49ab29cc552c5e2f41
instabug_flutter: 0a2d35be020c80b2b63bd8337a94a3f2ffe65bc0
integration_test: 4a889634ef21a45d28d50d622cf412dc6d9f586e
instabug_flutter: 1d2781ee0b451dc2102fbe94802e310866d03e01
integration_test: 252f60fa39af5e17c3aa9899d35d908a0721b573
Intercom: bb29786287d8b4e23d171ccbe9ceeb5b74c38b36
intercom_flutter: 7d5f8d57b346a9ca644ad464d63549158519444b
intercom_flutter: 80e3f8f15cbee60ff772faa16f502368d10274f6
IosAwnCore: 653786a911089012092ce831f2945cd339855a89
iOSDFULibrary: 198c36ebe5a31bf2ca04ee8fb40837176a423e85
iOSMcuManagerLibrary: 4102a4595be1c69e5286d7f1520a733c82c30b0a
just_audio: 4e391f57b79cad2b0674030a00453ca5ce817eed
location: 155caecf9da4f280ab5fe4a55f94ceccfab838f8
manage_calendar_events: fe1541069431af035ced925ebd9def8b4b271254
map_launcher: fe43bda6720bb73c12fcc1bdd86123ff49a4d4d6
mcumgr_flutter: 8c4a598cb1b4d10a9adbc0f13288334297185506
just_audio: a42c63806f16995daf5b219ae1d679deb76e6a79
location: d5cf8598915965547c3f36761ae9cc4f4e87d22e
manage_calendar_events: 9b2889799340398027b3e3f5c4891d41599ec257
map_launcher: 5fde49ac9a52672bf99da746599f507b4490d7b5
mcumgr_flutter: 097e59bec5917b527ba6d095da57bdaed2de1fa2
Mixpanel-swift: 7b26468fc0e2e521104e51d65c4bbf7cab8162f8
mixpanel_flutter: a0b6b937035899cd01951735ad5f87718b2ffee5
mixpanel_flutter: c2bb8345c90bef15512a1b812ec800b52f8614b6
nanopb: fad817b59e0457d11a5dfbde799381cd727c1275
nordic_dfu: e4fb6f461f4a290b28ea4b1dfb69071665cdfa3e
nordic_dfu: 963e2e4e6afb04e515ff43c546e6f8abf3e04ed5
onnxruntime-c: e87399683ec19e3b812e13c6692882609a802b86
onnxruntime-objc: 57ae8f83779a4c32731065d50d02d042af581114
opus_flutter_ios: f16ed3599997ced564ad44509e87003159a86def
package_info_plus: af8e2ca6888548050f16fa2f1938db7b5a5df499
path_provider_foundation: 080d55be775b7414fd5a5ef3ac137b97b097e564
permission_handler_apple: 4ed2196e43d0651e8ff7ca3483a069d469701f2d
opus_flutter_ios: 0c103338d4f4316034e082857749a3514b9540d3
package_info_plus: c0502532a26c7662a62a356cebe2692ec5fe4ec4
path_provider_foundation: 2b6b4c569c0fb62ec74538f866245ac84301af46
permission_handler_apple: 9878588469a2b0d0fc1e048d9f43605f92e6cec2
PostHog: 77fa0e762ced56862eb9a520c46bda206ccf5f23
posthog_flutter: 631ab870f7daf1ed190deeb2414eb3e347445f37
posthog_flutter: 6d1041c7da77eb2032c27b72df103439af64cb8c
PromisesObjC: f5707f49cb48b9636751c5b2e7d227e43fba9f47
RecaptchaInterop: 11e0b637842dfb48308d242afc3f448062325aba
SDWebImage: f84b0feeb08d2d11e6a9b843cb06d75ebf5b8868
share_plus: 011d6fb4f9d2576b83179a3a5c5e323202cdabcf
shared_preferences_foundation: 9e1978ff2562383bd5676f64ec4e9aa8fa06a6f7
sign_in_with_apple: c5dcc141574c8c54d5ac99dd2163c0c72ad22418
sqflite_darwin: 20b2a3a3b70e43edae938624ce550a3cbf66a3d0
share_plus: 8875f4f2500512ea181eef553c3e27dba5135aad
shared_preferences_foundation: fcdcbc04712aee1108ac7fda236f363274528f78
sign_in_with_apple: f3bf75217ea4c2c8b91823f225d70230119b8440
sqflite_darwin: 5a7236e3b501866c1c9befc6771dfd73ffb8702d
SwiftCBOR: ce5354ec8b660da2d6fc754462881119dbe1f963
SwiftProtobuf: b7aa08087e2ab6d162862d143020091254095f69
SwiftyGif: 706c60cf65fa2bc5ee0313beece843c8eb8194d4
url_launcher_ios: 694010445543906933d732453a59da0a173ae33d
webview_flutter_wkwebview: 1821ceac936eba6f7984d89a9f3bcb4dea99ebb2
url_launcher_ios: 5334b05cef931de560670eeae103fd3e431ac3fe
webview_flutter_wkwebview: a4af96a051138e28e29f60101d094683b9f82188
ZIPFoundation: b8c29ea7ae353b309bc810586181fd073cb3312c

PODFILE CHECKSUM: 0ff3dedbc65a62aff6be5119a19cb4fd9e15742d

COCOAPODS: 1.16.2
COCOAPODS: 1.15.2
2 changes: 1 addition & 1 deletion app/lib/backend/http/api/conversations.dart
Original file line number Diff line number Diff line change
Expand Up @@ -407,4 +407,4 @@ Future<String> testConversationPrompt(String prompt, String conversationId) asyn
} else {
return '';
}
}
}
7 changes: 5 additions & 2 deletions app/lib/backend/schema/conversation.dart
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,11 @@ class CreateConversationResponse {
factory CreateConversationResponse.fromJson(Map<String, dynamic> json) {
return CreateConversationResponse(
messages: ((json['messages'] ?? []) as List<dynamic>).map((message) => ServerMessage.fromJson(message)).toList(),
conversation: json['memory'] != null ? ServerConversation.fromJson(json['memory']) : null,
conversation: json['memory'] != null
? ServerConversation.fromJson(json['memory'])
: json['conversation'] != null
? ServerConversation.fromJson(json['conversation'])
: null,
);
}
}
Expand Down Expand Up @@ -190,7 +194,6 @@ class ServerConversation {

String getTag() {
if (source == ConversationSource.screenpipe) return 'Screenpipe';
if (source == ConversationSource.openglass) return 'Openglass';
if (source == ConversationSource.sdcard) return 'SD Card';
if (discarded) return 'Discarded';
return structured.category.substring(0, 1).toUpperCase() + structured.category.substring(1);
Expand Down
82 changes: 78 additions & 4 deletions app/lib/backend/schema/structured.dart
Original file line number Diff line number Diff line change
Expand Up @@ -149,19 +149,93 @@ class Event {
class ConversationPhoto {
int id = 0;

String base64;
String? photoId; // Cloud storage photo ID
String? base64; // Legacy base64 data - optional for cloud photos
String description;

ConversationPhoto(this.base64, this.description, {this.id = 0});
String? thumbnailUrl; // Cloud storage thumbnail URL
String? url; // Cloud storage full-size URL
String? createdAt; // When photo was taken/added
String? addedAt; // When photo was added to conversation

ConversationPhoto(this.description, {
this.id = 0,
this.photoId,
this.base64,
this.thumbnailUrl,
this.url,
this.createdAt,
this.addedAt
});

factory ConversationPhoto.fromJson(Map<String, dynamic> json) {
return ConversationPhoto(json['base64'], json['description']);
return ConversationPhoto(
json['description'] ?? '',
id: json['id'] is int ? json['id'] : 0,
photoId: json['photo_id'] as String? ?? json['id'] as String?,
base64: json['base64'] as String?,
thumbnailUrl: json['thumbnail_url'] as String?,
url: json['url'] as String?,
createdAt: json['created_at'] as String?,
addedAt: json['added_at'] as String?,
);
}

toJson() {
return {
'id': id,
'photo_id': photoId,
'base64': base64,
'description': description,
'thumbnail_url': thumbnailUrl,
'url': url,
'created_at': createdAt,
'added_at': addedAt,
};
}

// Helper method to get createdAt as DateTime
DateTime get createdAtDateTime {
if (createdAt != null) {
try {
return DateTime.parse(createdAt!);
} catch (e) {
// Fallback to current time if parsing fails
return DateTime.now();
}
}
return DateTime.now();
}

// Helper method to get the best display URL (thumbnail first, then full URL)
String? getDisplayUrl() {
// Prefer thumbnail URL for display, fallback to full URL
if (thumbnailUrl != null && thumbnailUrl!.isNotEmpty) {
return thumbnailUrl;
}
if (url != null && url!.isNotEmpty) {
return url;
}
return null;
}

// Helper method to get full-size URL - prefer full URL, fallback to thumbnail, then base64
String? getFullUrl() {
if (url != null && url!.isNotEmpty) {
return url;
}
if (thumbnailUrl != null && thumbnailUrl!.isNotEmpty) {
return thumbnailUrl;
}
if (base64 != null && base64!.isNotEmpty) {
return 'data:image/jpeg;base64,$base64';
}
return null;
}

// Helper method to check if photo has any displayable content
bool hasDisplayableContent() {
return (thumbnailUrl != null && thumbnailUrl!.isNotEmpty) ||
(url != null && url!.isNotEmpty) ||
(base64 != null && base64!.isNotEmpty);
}
}
86 changes: 61 additions & 25 deletions app/lib/pages/capture/logic/openglass_mixin.dart
Original file line number Diff line number Diff line change
Expand Up @@ -15,61 +15,97 @@ mixin OpenGlassMixin {
StreamSubscription? _bleBytesStream;

// TODO: use connection directly
Future<BleAudioCodec> _getAudioCodec(String deviceId) async {
var connection = await ServiceManager.instance().device.ensureConnection(deviceId);
return connection?.getAudioCodec() ?? Future.value(BleAudioCodec.pcm8);
Future<BleAudioCodec> getAudioCodec(String deviceId) async {
final connection = await ServiceManager.instance().device.ensureConnection(deviceId);
if (connection == null) return BleAudioCodec.pcm8;
return connection.getAudioCodec();
}

Future<StreamSubscription?> _getImageListener(
String deviceId, {
required void Function(Uint8List base64JpgData) onImageReceived,
Future<StreamSubscription?> getImageListener({
required void Function(Uint8List) onImageReceived,
required String deviceId,
}) async {
var connection = await ServiceManager.instance().device.ensureConnection(deviceId);
return connection?.getImageListener(onImageReceived: onImageReceived) ?? Future.value(null);
final connection = await ServiceManager.instance().device.ensureConnection(deviceId);
if (connection == null) return null;
return connection.getImageListener(onImageReceived: onImageReceived);
}

Future _cameraStopPhotoController(String deviceId) async {
var connection = await ServiceManager.instance().device.ensureConnection(deviceId);
return connection?.cameraStopPhotoController() ?? Future.value(null);
Future<void> cameraStopPhotoController(String deviceId) async {
final connection = await ServiceManager.instance().device.ensureConnection(deviceId);
if (connection == null) return;
return connection.cameraStopPhotoController();
}

Future _cameraStartPhotoController(String deviceId) async {
var connection = await ServiceManager.instance().device.ensureConnection(deviceId);
return connection?.cameraStartPhotoController() ?? Future.value(null);
Future<void> cameraStartPhotoController(String deviceId) async {
final connection = await ServiceManager.instance().device.ensureConnection(deviceId);
if (connection == null) return;
return connection.cameraStartPhotoController();
}

Future<bool> _hasPhotoStreamingCharacteristic(String deviceId) async {
var connection = await ServiceManager.instance().device.ensureConnection(deviceId);
return connection?.hasPhotoStreamingCharacteristic() ?? Future.value(false);
Future<bool> hasPhotoStreamingCharacteristic(String deviceId) async {
final connection = await ServiceManager.instance().device.ensureConnection(deviceId);
if (connection == null) return false;
return connection.hasPhotoStreamingCharacteristic();
}

Future<void> refreshOpenGlassCamera(String deviceId) async {
// Stop current capture
await cameraStopPhotoController(deviceId);

// Wait briefly for camera to stop
await Future.delayed(Duration(milliseconds: 500));

// Clear any cached images
photos.clear();

// Start camera with single capture command to trigger immediate photo
await cameraStartPhotoController(deviceId);

// Wait briefly then send a single photo command to force immediate capture
await Future.delayed(Duration(milliseconds: 200));
final connection = await ServiceManager.instance().device.ensureConnection(deviceId);
if (connection != null) {
try {
// Send single photo command (-1 = immediate single capture)
await connection.performCameraStartPhotoController();
} catch (e) {
debugPrint('Error sending immediate capture command: $e');
}
}
}

Future<void> openGlassProcessing(
BtDevice device,
Function(List<Tuple2<String, String>>) onPhotosUpdated,
Function(bool) setHasTranscripts,
) async {
_bleBytesStream = await _getImageListener(
device.id,
_bleBytesStream = await getImageListener(
onImageReceived: (Uint8List completedImage) async {
if (completedImage.isNotEmpty) {
debugPrint('Completed image bytes length: ${completedImage.length}');
Tuple2<String, String> photo = Tuple2(base64Encode(completedImage), '');
photos.add(photo);
getPhotoDescription(completedImage).then((description) {
photos[photos.indexOf(photo)] = Tuple2(photo.item1, description);
if (photos.contains(photo)) {
int index = photos.indexOf(photo);
photos[index] = Tuple2(photo.item1, description);
onPhotosUpdated(photos);
debugPrint('photos: ${photos.length}');
setHasTranscripts(true);
}
});
onPhotosUpdated(photos);
}
},
deviceId: device.id,
);
await _cameraStopPhotoController(device.id);
await _cameraStartPhotoController(device.id);

// Initial camera setup
await cameraStopPhotoController(device.id);
await Future.delayed(Duration(milliseconds: 300));
await cameraStartPhotoController(device.id);
}

Future<bool> isGlassesDevice(String deviceId) async {
return await _hasPhotoStreamingCharacteristic(deviceId);
return await hasPhotoStreamingCharacteristic(deviceId);
}

void disposeOpenGlass() {
Expand Down
Loading