Syncing Gone Wrong: The Day I Lost All My FocusPasta Data
Realm: 1, FocusPasta: 0. We are now on CoreData.
My worst nightmare just came true.
For months, I’ve been using FocusPasta — the productivity app I’ve been building — to track nearly every aspect of my life. Born out of my obsession with data (and pasta), it became my go-to app for tracking how I spend my time, from work to workouts, to even FocusPasta development itself.
After months of usage, the daily charts were starting to paint a clear picture of how I spent my days. The jar charts were overflowing with pasta, and the yearly habit tracker was starting to fill up and reveal meaningful patterns in my routine.
Then, in one fleeting moment of stupidity, it all vanished. In the blink of an eye, I went from admiring my beautifully filled charts to staring at an empty screen, paralyzed by denial.
Months of data — gone. Just like that.
The Sync Struggles
This tragedy is the result of my attempts at implementing iCloud sync capabilities for FocusPasta.
When I first started building FocusPasta, I chose Realm as my database over CoreData. There wasn’t any particular reason for this choice — FocusPasta doesn’t have complex data structures, and frankly, it didn’t really matter which option I went with at the time.
As I added more features to the app, I wanted to enable users to sync their data across devices via iCloud. That’s when the real challenge began.
Apple’s CloudKit Solution
Apple offers a native solution with CloudKit that allows developers to store and sync app data in iCloud. It handles all the heavy lifting from syncing, merging data, and conflict resolution.
The catch? CloudKit only works with CoreData, not Realm.
To make it work, I would need to migrate to CoreData, completely rewriting the backbone of FocusPasta. This introduces the risk of breaking the app, or you know — losing all my data.
Realm’s Cloud Sync Solution
Realm, on the other hand, offers its own cloud sync solution, but with a paid subscription which I didn’t want to pay for. (And as luck would have it, I obviously didn’t know at the time, but MongoDB recently announced that they’re deprecating Realm in 2025. Anyone using their cloud sync solutions would eventually have to migrate anyway).
The DIY Solution
Considering everything above, I decided to try implementing my own database sync solution. And here’s where things went… well, not quite as planned.
My First Approach to “iCloud Sync”
For my first attempt at implementing iCloud sync, I set up a basic manual backup and restore system. It wasn’t perfect, but it would allow users to transfer their data between devices via iCloud — ideal for cases like transferring data from one phone to another.
Apple provides applications with a dedicated container in iCloud, allowing them to store app-specific data. This feature gave me the opportunity to store a backup of the Realm database in the cloud, while keeping the main database securely stored locally on the device.
Here’s how it worked.
Backup: When the user taps “Backup”, the entire local Realm database is uploaded to the iCloud directory.
Restore: When the user taps “Restore”, the user is prompted with a warning that the local data will be overwritten. Upon confirmation, the local database is fully replaced with the iCloud backup.
This approach is straightforward and didn’t involve any conflict resolution or data merging — just a simple overwrite. Strictly speaking, it’s not even really a “sync”, more of a backup-and-restore feature.
Next Step: Achieving True Sync
While the backup-and-restore solution was functional, it didn’t provide the seamless experience that I wanted. I envisioned a true iCloud sync solution, where users can simply toggle a switch to enable or disable syncing, without any underlying backup or restore logic being exposed.
Hence, I decided to expand on my initial approach.
V1: Backup on Every Write, Restore on Every Read
Since the current backup and restore strategy works, I thought: Why not take the current implementation and make it automatic?
I figured that by triggering a backup every time data was written and triggering a restore whenever data was read, I could create the illusion of a true iCloud sync. This would take the manual process out of the equation and provide a smooth experience for users.
The Unexpected Disaster
Unfortunately, I didn’t fully think this through before testing it on my data. I launched FocusPasta on a simulator and logged into my iCloud, completely unaware of the disaster quietly unfolding in the background. As I logged in, a seamless exchange of files occurred between the simulator, the cloud, and my phone.
Unbeknownst to me, the simulator had uploaded a fresh Realm database to the cloud, and my actual device, without hesitation, restored that new database — completely overwriting my local data. Meanwhile, I, the unsuspecting user, sat there in frustration, wondering why my FocusPasta data wouldn’t restore to the simulator.
Once I realized what had happened, I spent the next hour in denial and disbelief, desperately trying to find a way to recover the lost data. But once the file was deleted, it was gone for good — it’s like using “rm -f” in the terminal, with no hope for recovery.
V2: An append-only Database
At this point, I was completely traumatized by the data loss.
It was devastating, and the fear of losing data again consumed me. I became so fixated on preventing future data losses that I skipped over other potential edge cases while thinking of an alternative solution.
That’s when it hit me — why not make the database append-only?
I felt like a genius. With an append-only database, data loss would be literally impossible — there would be no deletions! On top of that, it also seemed more efficient: I wouldn’t have to rewrite the entire database at each backup, just append the new entries.
With this brilliant plan in mind, I got to work and went to bed that night feeling victorious. I was proud of myself for solving what felt like the most difficult problem in the universe. Oh, my sweet summer child.
I thought I had everything figured out, but I missed out something completely basic. Take a moment and see if you can figure out what I overlooked.
Hello, It’s Me Again
The next day, fully confident that FocusPasta was now running flawlessly, I began cleaning up all the random test categories I had created — categories like “Blah”, “Test”, and “A”. I started using the app as per normal.
But when I reopened the app, to my dismay, those same test categories had returned!
Now, have you figured out the problem yet?
That’s right. That’s what happens when you have an append-only database.
V3: Adding a deleted_at Field, WAL implementation, AI-powered quantum migration?
By this point, I had officially entered the rabbit hole of overengineering.
My mind was racing with ideas on how to solve this issue, and I started sketching out possible solutions. One option was to introduce the “deleted_at” field, which would allow me to mark categories and events as deleted without actually removing them. I could filter these out in the UI and set up a TTL to automatically delete them after a set period.
Another option was inspired by the write-ahead log, which is a common technique used in database systems. What if, instead of syncing the entire Realm database to the cloud, I could just store the operations done on it? I would then apply those changes locally, keeping everything resilient, durable and efficient and… extremely complicated.
It didn’t take long for me to realize that I am starting to step into the realm (haha) of reinventing a distributed database. I had resisted a CoreData migration to save time, but here I was, spending even more time, energy and sanity on this overengineered mess that still didn’t work.
And let’s face it — I didn’t have any data left to lose.
The Big CoreData Migration
It became clear that migrating to CoreData was the way to go.
Doing it now, before FocusPasta actually launches, made perfect sense. It allows me to avoid the painstaking process of migrating real user data down the road, along with the burden of maintaining legacy code indefinitely. By cutting ties with Realm now and starting fresh, I could ensure a more stable foundation for the future of FocusPasta.
The migration itself wasn’t as complex as I feared. I owe my past self a big thank-you for organizing my database CRUD operations into clear, well-structured functions. This made the transition much more manageable, as I could feed small snippets of code into ChatGPT and request it to “Migrate this from Realm to CoreData.”. Before I knew it, FocusPasta was back up and running, as if nothing had ever happened.
And CoreData? It’s like it was made for this. CloudKit integration was a breeze with minimal setup, and updates made on one device synced effortlessly across all devices within seconds. Even better, the code remained nearly identical, whether or not cloud sync was enabled.
Of course, CoreData isn’t without its flaws.
With CloudKit integration, CoreData no longer supports unique constraints or non-optional fields and relationships. This means that I can’t enforce rules like ensuring every event has a parent category or that all data fields are filled at the database level. Instead, I have to manually validate these conditions in my code, introducing a fair number of guard let statements.
Annoying? Definitely.
But compared to reinventing the wheel with a DIY distributed database system packed with edge cases? I’ll take it any day.