Skip to content

Failed to open DB: LOCK: already held by process #3967

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jzaccone opened this issue Oct 1, 2019 · 11 comments · Fixed by #3984
Closed

Failed to open DB: LOCK: already held by process #3967

jzaccone opened this issue Oct 1, 2019 · 11 comments · Fixed by #3984
Assignees

Comments

@jzaccone
Copy link

jzaccone commented Oct 1, 2019

Full error:

Fatal Exception: NSInternalInconsistencyException
Failed to open DB: Internal: Failed to open LevelDB database at 
/var/mobile/Containers/Data/Application/D5EA17E8-B506-4E84-84C2-
10A92D7A0AA4/Documents/firestore/__FIRAPP_DEFAULT/xxx/main: LevelDB
 error: IO error: lock /var/mobile/Containers/Data/Application/D5EA17E8-B506-4E84-84C2-
10A92D7A0AA4/Documents/firestore/__FIRAPP_DEFAULT/xxx/main/LOCK: 
already held by process

Xcode version: 10.3
Using Firebase (6.6.0)
Using FirebaseAnalytics (6.1.0)
Using FirebaseAnalyticsInterop (1.3.0)
Using FirebaseAuth (6.2.2)
Using FirebaseAuthInterop (1.0.0)
Using FirebaseCore (6.2.0)
Using FirebaseCoreDiagnostics (1.0.1)
Using FirebaseCoreDiagnosticsInterop (1.0.0)
Using FirebaseDatabase (6.0.0)
Using FirebaseDynamicLinks (4.0.2)
Using FirebaseFirestore (1.4.3)
Using FirebaseInstanceID (4.2.3)
Using FirebaseMessaging (4.1.2)
Using FirebaseStorage (3.4.0)
Using FirebaseUI (8.0.4)

Can't reproduce this problem. I am not deleting FIRApp instances, I'm only accessing the database via Firestore.firestore() or [[FIRFirestore firestore] which should provide the same instance back. The only thing I can think of is [FIRApp configure]; getting called twice with thread race, but that is supposed to be thread-safe as well.

Found this in another issue #3778 from Will. @wilhuff Is there any other scenarios which might cause this error?

@ChristianVinterly your case is different. Somehow you've managed to open Firestore more than once in a single process and the second one failed to open its database because the first one still had the database locked.

Normally, the Firestore SDK keeps a map of active instances so that [FIRFirestore firestore] will return the same instance. The only way I can think of that you might have encountered this is if you deleted FIRApp and then managed to create a new one and a new Firestore instance before the old one shut down.

If you're at all able to reproduce this, please create a new issue with the details of what you're doing--this is unrelated to the "Operation Not Permitted" issue going on in this discussion.

@google-oss-bot

This comment has been minimized.

@wilhuff
Copy link
Contributor

wilhuff commented Oct 1, 2019

Are you possibly accessing Firestore from multiple threads very early in your application?

@jzaccone
Copy link
Author

jzaccone commented Oct 2, 2019

Yes, right on launch on querying data from HealthKit, then saving the result into Firestore. This is sometimes called multiple times. Should I be accessing my Firestore instance from the main thread in this scenario?

@wilhuff
Copy link
Contributor

wilhuff commented Oct 2, 2019

It should be the case that this is all thread-safe, but given your description it seems like there may be a race in the initialization somewhere. I'll see if I can reproduce based on that.

This suggests that a workaround for now would be to call Firestore.firestore() (or [[FIRFirestore firestore]) once (likely early on the main thread) and reuse that instance. If the crash goes away after that it would also confirm the hypothesis.

@wilhuff wilhuff self-assigned this Oct 2, 2019
@jzaccone
Copy link
Author

jzaccone commented Oct 2, 2019

OK I will call once and reuse instance like you suggest and will let you know if the crash goes away.

wilhuff added a commit that referenced this issue Oct 2, 2019
Reported in #3967. Users sometimes seeing a crash when multiple threads
concurrently try to create Firestore instances. By inspection I found
that there's a race here where two threads can check for presence in the
cache, see nothing, proceed to creation, both store a value in the cache,
and both return their copies. This would lead to two Firestores being
created for a single App.

This replaces `dispatch_sync` on a serial queue with good old
`@synchronized` blocks. `dispatch_sync` will self-deadlock if called
from a thread already on the queue while `@synchronized` allows
recursion.

No new tests are added because several tests like
`testDependencyDoesntBlock` already cover this case.
@wilhuff
Copy link
Contributor

wilhuff commented Oct 3, 2019

I haven't been able to reproduce this issue locally, but tracing through the initialization logic I did find a race and I've proposed a fix in #3984.

You can try that out by adding the following to your Podfile:

pod 'FirebaseCore', :git => 'https://github.com/firebase/firebase-ios-sdk.git', :branch => 'pull/3984/head'

Note that any existing lines for Firebase/Core should remain unchanged.

@simonbengtsson
Copy link

simonbengtsson commented Oct 3, 2019

We released a new version yesterday and now this is our most common crash appearing for about 1% of users.

Our new release contains three things related to this:

  1. Upgraded firebase sdk form 6.8.1 to 6.9.0
  2. First build with Xcode 11
  3. Migrated parts of database from realtime to firestore

Based on environment posted in issue and above comments I would guess its #3 then since we also are using firestore early in the application.

ryanwilson pushed a commit that referenced this issue Oct 3, 2019
* Fix race condition in FIRComponentContainer instance creation

Reported in #3967. Users sometimes seeing a crash when multiple threads
concurrently try to create Firestore instances. By inspection I found
that there's a race here where two threads can check for presence in the
cache, see nothing, proceed to creation, both store a value in the cache,
and both return their copies. This would lead to two Firestores being
created for a single App.

This replaces `dispatch_sync` on a serial queue with good old
`@synchronized` blocks. `dispatch_sync` will self-deadlock if called
from a thread already on the queue while `@synchronized` allows
recursion.

No new tests are added because several tests like
`testDependencyDoesntBlock` already cover this case.

* format
ryanwilson pushed a commit that referenced this issue Oct 3, 2019
* Fix race condition in FIRComponentContainer instance creation

Reported in #3967. Users sometimes seeing a crash when multiple threads
concurrently try to create Firestore instances. By inspection I found
that there's a race here where two threads can check for presence in the
cache, see nothing, proceed to creation, both store a value in the cache,
and both return their copies. This would lead to two Firestores being
created for a single App.

This replaces `dispatch_sync` on a serial queue with good old
`@synchronized` blocks. `dispatch_sync` will self-deadlock if called
from a thread already on the queue while `@synchronized` allows
recursion.

No new tests are added because several tests like
`testDependencyDoesntBlock` already cover this case.

* format
ryanwilson added a commit that referenced this issue Oct 3, 2019
* Fix race condition in FIRComponentContainer instance creation (#3984)

* Fix race condition in FIRComponentContainer instance creation

Reported in #3967. Users sometimes seeing a crash when multiple threads
concurrently try to create Firestore instances. By inspection I found
that there's a race here where two threads can check for presence in the
cache, see nothing, proceed to creation, both store a value in the cache,
and both return their copies. This would lead to two Firestores being
created for a single App.

This replaces `dispatch_sync` on a serial queue with good old
`@synchronized` blocks. `dispatch_sync` will self-deadlock if called
from a thread already on the queue while `@synchronized` allows
recursion.

No new tests are added because several tests like
`testDependencyDoesntBlock` already cover this case.

* format

* Update Core CHANGELOG

* Update Core CHANGELOG again
@wilhuff
Copy link
Contributor

wilhuff commented Oct 3, 2019

@simonbengtsson have you been able to reproduce the issue locally? If so, does applying the fix as described above address it?

@simonbengtsson
Copy link

I have not been able to reproduce, but we will release a new version in a few days so can confirm then if noone have been able to before that.

@wilhuff
Copy link
Contributor

wilhuff commented Oct 3, 2019

OK. We haven't been able to reproduce either. The fix is based on inspection and the result is strictly better than what came before. My worry is that this may not have been the only cause. Please do let us know what you see.

In the meantime this fix will end up in the next Firebase release.

@jzaccone
Copy link
Author

jzaccone commented Oct 7, 2019

I have running a new version in production with the PR fix and I have yet to see the crash come up. This resolved the crashes I was seeing

ryanwilson added a commit that referenced this issue Oct 9, 2019
* Update versions for Release 6.10.0

* Cherrypick #3983 and include Database in release. (#3986)

* Remove log argument that causes SocketRocket crash (#3983)

* Add FirebaseDatabase to release.

* Update CHANGELOG with version.

* Cherrypick for #3984 along with CHANGELOG update. (#3988)

* Fix race condition in FIRComponentContainer instance creation (#3984)

* Fix race condition in FIRComponentContainer instance creation

Reported in #3967. Users sometimes seeing a crash when multiple threads
concurrently try to create Firestore instances. By inspection I found
that there's a race here where two threads can check for presence in the
cache, see nothing, proceed to creation, both store a value in the cache,
and both return their copies. This would lead to two Firestores being
created for a single App.

This replaces `dispatch_sync` on a serial queue with good old
`@synchronized` blocks. `dispatch_sync` will self-deadlock if called
from a thread already on the queue while `@synchronized` allows
recursion.

No new tests are added because several tests like
`testDependencyDoesntBlock` already cover this case.

* format

* Update Core CHANGELOG

* Update Core CHANGELOG again

* Fix merge from CHANGELOG
@firebase firebase locked and limited conversation to collaborators Nov 3, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants