[Rant] Full data loss in case of migration problems due to data storage in RocksDB

[codiflow]

As I already mentioned in this issue (#7431) the current way of storing data only in RocksDB increases the possibility of total data loss.

I'll give you an example:

I was using 0.7.6 on macOS for months now and had my data stored locally. Then I added a cloud account and tried to import my Notion stuff. This did not work so I just forgot about it.

Then after a few weeks I started to install 0.9.3 to see if things are working better with this version, but:

After the migration from 0.7.6 to 0.9.3 all local data was GONE.

I have several data folders now and I can clearly identify in which DB folder my data resides but I'm NOT able to access it anymore. I already followed this guide without any success.

Error messages differ, so just some examples I got while trying to recover anything:

• Can't find the user profile for user id: <REDACTED>, error: DatabaseError(Unknown, "no such column: user_table.workspace")
  => Database layout in flowy-database.db was missing tables

• {"msg":"[Flutter]: init workspace, current workspace: <REDACTED>, workspaces: (<REDACTED>), isCollabWorkspaceOn: false","time":"07-08 15:44:44","target":"dart_ffi"} 2025-07-08 15:44:44 ERROR flowy_user::event_handler: error: Not support yet to Failed to get workspace usage, error: code: NotSupportYet msg: Not support yet

• ERROR flowy_folder::manager_init: error: Local version not support

• ERROR flowy_folder::manager_init: error: code:Not support yet, message:Can not create default folder

• ERROR flowy_folder::event_handler: error: code:Internal error, message:folder is not initialized

• ERROR collab_plugins::local_storage::kv::doc: 🔴Insert update failed. Can't find the doc for 478892687370489856-"34de9b49-9412-571b-b474-b530235071d7"

I invested many hours now trying to recover the old database without any success. I was using the initial version 0.7.6 and also the most recent version 0.9.4, I cleanly initialized data folders, changed the IDs to the respective ones from the database whose data I want to access and imported the old database then, I tried to import the data folder within AppFlowy, I created a cloud account and tried to integrate the old database there:

Absolutely NO success with any of these actions.

And as RocksDB stores useless Key-Value pairs which look like this, a RocksDB database dump is completely useless too:

What I want to say here is that the current way AppFlowy stores local data is ‼️ not resilient ‼️ and users relying on local data storage without a manual Markdown export might experience FULL DATA LOSS in case migrations fail.

This definitely should be changed as user data should NEVER be put at risk.

[codiflow]

This issue seems to be related – the workflow with switching between cloud and local storage and loosing all local data afterwards sounds familiar...

https://reddit.com/r/AppFlowy/comments/1k4cs7l/lost_data_after_switching_to_cloud_need_help/

[codiflow]

And here's what happens after doing the following as described here¹:

• Starting AppFlowy-0.7.6-linux-x86_64.AppImage
• Clicking on Anonymous to have a clean local profile initialized
• Closing AppFlowy
• Editing data_aHR0cHM6Ly9iZXRhLmFwcGZsb3d5LmNsb3Vk/cache.db and change the values of database_storage_id and id to match the ID from the RocksDB and the workspace ID found within the old logs
• Saving the database
• Replacing folder collab_db within the new user ID folder with the data folder from one of the backups in collab_db_history folder

Result:

Note:
¹ I did replace the database itself and not the user folder like mentioned in the guide because the flowy-database.db within the old user folder is broken (missing table fields).

[codiflow]

Related issue with a possible "solution": #7872

[codiflow]

Luckily this procedure (done with v0.7.6) solved my data access problem: #7872 (comment)

But the main criticism of this post still remains – access to local user data should NEVER be this complicated.

[khorshuheng]

As you have rightly pointed out, the issue is not related directly to RocksDB, but a bug that was introduced when refactoring the offline workplace.

RocksDB is typically resilient, except when the same directory is also being used for some third party sync tool / network drive, in which data corruption is known to happen. But it is indeed much harder to extract the data from RocksDB than say, sqlite.

So why didn't we use sqlite? Frankly, it is indeed possible to design similar application using sqlite. But at this point, changing the data storage layer is risky enough that a complete data loss, such as what you observed, may happen with such changes.

Moreover, our raw data is actually not in markdown, or other human readable format. It is actually json, encoded in yrs/yjs binary format. So even if we make it easier to access the data by using something other than RocksDB (which I will wholeheartedly agree is not the easiest tool to work with), the data still can't be easily retrieved.

As to why isn't the raw data stored as pure markdown etc? Primarily because we need yjs/yrs for real time collaboration. If we don't support real time collaboration, then yes, it is indeed possible to design the app such that a human readable format is used and stored directly in the local database.

Meanwhile, we are working on a proper export / import system for the whole workspace. It won't completely address the root cause, but it should at least be better than what we have now.

[annieappflowy]

@codiflow if your local data folder is still there, we can help you recover it. @LucasXu0

[codiflow]

@khorshuheng thank you for taking my request seriously and explaining why you've choosen to use a "hard to recover" database like RocksDB.

I totally understand your points and want to give a +1 for implementing a proper export / import system asap. From my perspective this is a must have for every application holding (sensitive) user data as things may always break.

After messing around 6 hours with this bug I was able to access my data by following these instructions which technically make sense but are not intuitive to follow.

Long story short: The bug seems to be NOT resolved yet so people who are migrating from an old version like 0.7.6 (and maybe older) to for example 0.9.4 might still experience this kind of "data access loss" I described. Emptying the values within the SQLite as mentioned here as "official solution" did not help in my situation because the SQLite from the old data folder was broken too (missing table fields).

This issue should definitely gain more official attention and people should be warned proactively that local data access will be "lost" if they switch to a recent version and use the cloud feature making manual intervention necessary to regain access.

@annieappflowy @LucasXu0 Thanks for offering your help – as mentioned above I recovered access to my data already.

[LucasXu0]

@codiflow Thank you for your explanation and the important points you've raised.

We are currently working on the import and export feature that will enable users to export data from RocksDB in human-readable formats.

Additionally, we will address the 'data loss' issue related to the upgrade from version 0.7.x to the latest version asap (the missing table fields you mentioned).