Building data liberation infrastructure [see within blog graph]

You register your app, authorize it, get a token, and you are free to call various endpoints and fetch whatever you want.

I won't really elaborate on this as if you're reading this you probably have some idea how to use it. Otherwise, I'm sure there are tutorials out there that would help you.

if anyone knows of decent ones, please let me know and I'll add links!

Examples: thankfully, most services out there offer public API to some extent

Sometimes a service doesn't offer an API. But from the service developer's perspective, it's still very reasonable to have one if you've got backend/frontend communication.

So chances are the service just isn't exposing it, but you can spy on the token/cookies in your browser devtools and use them to access the API.

You can read more about handling such data sources here:

• Reverse engineering APIs using Chrome Developer Tools: an extremely comprehensive and beginner-friendly tutorial
• "How I used Python to find interesting people to follow on Medium": an example of reverse engineering Medium API and using devtools
• Starbucks should really make their API public: demo of reverse engineering Starbucks Android app, featuring using proxy and forging the signature

Some examples:

• for exporting Messenger data, I'm using fbchat library. It works by tricking Facebook into believing it's a browser and interacting with private API.
• even though Pocket has an API, to get highlights from it you need to spy on the API key they use in the web app

Sometimes a service doesn't offer an API, doesn't use it even internally and serves HTML pages directly instead. Or, reverse engineering the API is so painful scraping becomes a more attractive option.

In addition to the same difficulties you would experience during API exports, there are some extra caveats here:

• authorization is harder: you definitely need username/password and potentially even a 2FA token
• DDOS protection: captchas, Cloudflare, etc.
• or even deliberate anti-scraping measures

For Python the holy grail of scraping is scrapy:

• Advanced Web Scraping Tutorial: bypassing "403 Forbidden", captchas, and more
• self-contained minimum example script to run scrapy

I'm pretty sure there are similar libraries for other languages, perhaps you could start with awesome-web-scraping repo or Ask HN: What are best tools for web scraping?.

For dealing with authorization, my personal experience is that using a persistent profile directory in Selenium is sufficient in most cases: you can login once manually and, reuse the profile in your scripts.

Examples:

• even though Hackernews has an API for public data, there is no way of getting your upvotes/saves without scraping HTML.
• Amazon and Paypal have to be scraped if you want your data.
• my bank, HSBC doesn't have an API. Not that I expected it from HSBC, I don't live in a fairy tale; but even their manual transactions exports are in PDF which I have to parse.

It's great they exist, and it is the easiest way to get your data if you just want a backup. However it doesn't really help in the long run:

• it's very manual: usually requires requesting and clicking on an email link
• it's slow and asynchronous: normally takes at least a few days
• the takeout format usually differs from the API format, sometimes ends up as something neither machine friendly nor human friendly

That said, with some effort it can potentially be automated as well.

They can be useful to get the 'initial' bit of your data, past the API limits.

Examples:

• Google Takeout
• Twitter Archive
• Github data export

I don't have an iPhone, so will only be referring to Android in this section, but I'd imagine the situation is similar.

These days, a service might not even offer a desktop version at all and considering that scraping data off mobile apps is way harder getting it from the phone directly might be an easier option. The data is often kept as an sqlite database which in many ways is even more convenient than an API!

On Android the story is simple: apps keep their data in /data/data/ directory, which is not accessible unless you root your phone. These days, with magisk it's considerably easier; however, it's still definitely not something a typical Android user would be able to do. Rooting your phone can bring all sorts of trouble by triggering root detection (e.g. common in banking apps), so be careful. And of course, phones come unrooted for a reason, so do it at your own risk.

Once you have root you can write a script to copy necessary files from /data/data/ to your target directory, synchronized with your computer (e.g. via Dropbox or Syncthing).

Examples:

• you can export Whatsapp data by copying /data/data/com.whatsapp/databases/msgstore.db
• scripts for exporting mobile Chrome/Firefox browsing history
• exporting Bluemaestro environment sensor data

Here I am referring to standalone specific-purpose gadgets like sleep trackers, e-ink readers, etc. The distinguishing thing is the device doesn't have Internet access or doesn't talk to any API.

You've got some options here:

• the device is capable of synchronizing with your phone (e.g. via Bluetooth)

  It's probably easiest to rely on phone app exports here. If the sync has to be triggered manually, you can benefit from some UI automation.

• the device is running Linux and has Internet access

  That's often the case with e-ink readers.

  You can potentially run the export script on the device itself and send the data somewhere else. Another option is running an SSH server on the device and pulling data from it, but it's quite extreme.

• the device can mount to a computer

  Then, you can use udev to trigger export when the device is plugged in. If udev feels too complicated for you, even a cron script running every minute might be enough.

Examples:

• using kobuddy for semiautomatic exports from Kobo e-ink reader

Every time you want your data, go exhaustively through all the endpoints and fetch the data. The result is some sort of JSON file (reflecting the complete state of your data) which you can save to disk.

'Incremental' means that rerunning an export starts from the last persisted point and only fetches missing data.

Implementation wise, it looks like this:

• query previously exported data to determine the point (e.g. timestamp/message id) to continue from
• fetch missing data starting from that point
• merge it back with previously exported data, persist on disk

It's similar to a full export in the sense that there isn't that much data to retrieve: if you could, you would just fetch it in one go.

What makes it similar to the incremental export is that you don't have all the data available at once - only the latest chunk. The main motivation for a synthetic export is that no single export file will give you all of the data.

There are various reasons for that:

• API restrictions

  Many APIs restrict the number of items you can retrieve through each endpoint for caching and performance reasons.

  Example: Reddit limits your API queries to 1000 entries.

• Limited memory

  Example: autonomous devices like HR monitors or temperature monitors are embedded systems with limited memory.

  Typically, they use some kind of ring buffer so when you export data, you only get, say, the latest 10000 measurements.

• Disagreement on the 'state' of the system

  Example: Kobo reader uses an sqlite database for keeping metadata like highlights, which is awesome! However, when you delete the book from your reader, it removes your annotations and highlights from the database too.

  There is absolutely no reason to do this: I delete the book because I don't need it on my reader, not because I want to get rid of the annotations. So in order to have all of them my only option is having regular database snapshots and assembling the full database from these pieces.

• Security

  Example: Monzo bank API.

  After a user has authenticated, your client can fetch all of their transactions, and after 5 minutes, it can only sync the last 90 days of transactions. If you need the user’s entire transaction history, you should consider fetching and storing it right after authentication.

  So that means that unless you're happy with manually authorizing every time you export, you will only have access to the last 90 days of transactions.

  Note: I feel kind of sorry complaining at Monzo, considering they are the nicest guys out there in terms of being dev friendly; and I understand the security concerns. But that's the only example of such behavior I've seen so far, and it does complicate things.

One important difference from other types of exports is that you have to do them regularly/often enough. Otherwise you inevitably miss some data and in the best case scenario have to get it manually, or in the worst case lose it forever.

Now, you could deal with these complications the same way you would with incremental exports by retrieving the missing data only. The crucial difference is that if you do make a mistake in the logic, it's not just a matter of waiting to re-download everything. Some of the data might be gone forever.

So I take a hybrid approach instead:

• at export time, retrieve all the data I can and keep it along with a timestamp, like a full export.

  Basically, it makes it an 'append-only system', so there is no opportunity for losing data.

• at data access time, we dynamically build (synthesize) the full state of the data

  We go through all exported data chunks and reconstruct the full state, similarly to incremental export. That's where 'synthetic' comes from.

  The 'full export' only exists at runtime, and errors in merging logic are not problematic as you never overwrite data. If you do spot a problem you only have to change the code with no need for data migrations.

This may be obvious, but I still feel it has to be said. Unless retrieving data is trivial (i.e. single GET request), chances that someone has already invested effort in dealing with various API quirks. Bindings often deal with dirty details like rate limiting, retrying, pagination, etc. So if you're lucky you might end up spending very little effort on actually exporting data.

If there is something in bindings you don't like or lack, it's still easier to monkey patch or just fork and patch them up (don't forget to open a pull request later!).

Also if you're the author of bindings, I have some requests. Please:

• don't print in stdout, it's a pain to filter out and suppress. Ideally use proper logging modules

• don't be overly defensive, or allow to configure non-defensive behavior

  It's quite sad when the library silently catches all exceptions and replaces them with empty strings/nulls/etc., without you even suspecting it. It's especially problematic in Python, where "Ask forgiveness, not permission" is very common.

• expose raw underlying data (e.g. raw JSON/XML from the API)

  If you forget to handle something, or the user disagrees with the interpretation of data, they would still be able to benefit from the data bindings for retrieval and only alter the deserialization.

  Example of good data object:

  • pymonzo exposes programmer-friendly fields and also keeps raw data

• expose generic methods for handling API calls to make it easy to add new endpoints

  Same argument: if you forgot to handle some API calls, it makes it much easier for consumers to quickly add them.

Keep the data you retrieved as intact as possible.

That means:

• don't insert it in in a database, unless it's really necessary
• don't convert formats (e.g. JSON to XML)
• don't try to clean up and normalize

Instead, keep the exporter code simple and don't try to interpret data in it. Move data interpretation burden to the data access layer instead.

The rationale here is it's a potential source of inconsistencies. If you make a bug during data conversion, you might end corrupting your data forever.

I'm elaborating on this point here.

• never silently fallback on default values in case of errors, unless you're really certain about what you're doing

• don't add retry logic just in case

  In my experience, it's fair to assume that if the export failed, it's a random server-side glitch and not worth fine-tuning - it's easier to simply start the export all over again. I'm not dealing with that in the individual export scripts at all, and using arctee, to retry exports automatically.

  If you know what you're doing (e.g. some endpoint is notoriously flaky) and do need retries, I recommend using an existing library that handles that like backoff.

• you don't want them in your shell history or in crontabs

• keeping them in a file can potentially allow for fine access control

  E.g. with Unix permissions you could only allow certain scripts to read secrets. Note that I'm not a security expert and would be interested to know if there are better solutions to that

  Personally, I found it so boilerplaty I extracted this logic to a separate helper module. You can find an example here.

I find that the only important bit is if you keep multiple export files (e.g. synthetic), make sure their names include timestamps and the time order is consistent with lexicographic order.

This means the only acceptable date/time format is some variation of YYYY MM DD HH MM SS Z. Feel free to sprinkle in any separators you like, or use milliseconds if you are really serious. Any other date format, e.g. MM/DD/YY, using month names, or not using zero-padded numbers is going to bring you serious grief.

E.g.:

ls /exports/instapaper/ | tail -n 5

instapaper_20200101T000005Z.json
instapaper_20200101T040004Z.json
instapaper_20200101T080010Z.json
instapaper_20200101T120005Z.json
instapaper_20200101T160011Z.json

The reason is it's automatically sort/max friendly, which massively reduces the cognitive load when working with data.

To make timestamping automatic and less boilerplaty, I'm using a wrapper script.

Backups are trivial: I can just run borg against /exports. What is more, borg is deduplicating, so it's very friendly to incremental and synthetic exports.

I synchronize/replicate it across my computers with Syncthing, also used Dropbox in the past.

Some back of the envelope math arguing it shouldn't be a concern for you:

• the amount of data you generate grows linearly. That means that running exports periodically would take 'quadratic' space
• with time, your available storage grows exponentially (and only gets cheaper)

Hopefully that's convincing, but if this is an issue it can also be addressed with compression or even using deduplicating backup software like borg. Keep in mind that would come at the cost of slowing down access, which may be helped with caching.

I don't even bother compressing most of my exports, except for the few which arctee wrapper handles.

There are also ways to benefit from compression without having to do it explicitly:

• keeping data under borg and using borg mount to access it.

  You get deduplication for free, however this makes exporting and accessing data much more obscure. In addition, borg mount locks the repository so it's going to be read-only while you access it.

• using a filesystem capable of compressing on the fly

  E.g. ZFS/BTRFS.

  It seems straightforward enough, thought non-standard file systems might be incompatible with some software, e.g. Dropbox. I haven't personally tried it.

A natural question is: if you run through all your data snapshots each time you access it, wouldn't it be too slow?

First, it's somewhat similar to the worries about the disk space. Data grows at the quadratic rate; and while processing power doesn't seem to follow Moore's law anymore there is still some potential to scale horizontally and use multiple threads. In practice, for most data sources that I use this process is almost instantaneous without parallelizing anyway.

In addition:

• if you're using iterators/generators/coroutines (e.g. example), that overhead will be amortized and basically unnoticeable
• you can still use caching. Just make sure it doesn't involve boilerplate or cognitive overhead to use. E.g. cachew.

You can find more specific examples along with the motivation and explanations here:

• Reddit
• Instapaper/Endomondo
• Pocket
• Chrome

I run most of my data exports at least daily.

I wrote a whole post on scheduling and job running with respect to the personal infrastructure. In short:

• on desktop: at the moment, I'm mostly using cron (to be more specific, fcron).

  I'm still thinking of an alternative, but overall using cron is okay.

• on Android phone: I'm using Automate app and cron

This is a wrapper script I'm using to run most of my data exports.

Many things are very common to all data exports, regardless of the source. In the vast majority of cases, you want to fetch some data, save it in a file (e.g. JSON) along with a timestamp and potentially compress it.

This script aims to minimize the common boilerplate:

• path argument allows easy ISO8601 timestamping and guarantees atomic writing, so you'd never end up with corrupted exports.
• --compression allows to compress simply by passing the extension. No more tar -zcvf!
• --retries allows easy exponential backoff in case service you're querying is flaky.

Example:

arctee '/exports/rtm/{utcnow}.ical.zstd' --compression zstd --retries 3 -- /soft/export/rememberthemilk.py

1. runs /soft/export/rememberthemilk.py, retrying it up to three times if it fails

   The script is expected to dump its result in stdout; stderr is simply passed through.

2. once the data is fetched it's compressed as zstd
3. timestamp is computed and compressed data is written to /exports/rtm/20200102T170015Z.ical.zstd

The wrapper operates on regular files and is therefore, programming language agnostic as long as your export script simply outputs to stdout (or accepts a filename, so you can use /dev/stdout). It doesn't really matter how exactly (e.g. which programming language) it's implemented.

That said, it feels kind of wrong having an extra script for all these things since they are not hard in principle, just tedious and boring to do all over again. If anyone has bright ideas on simplifying this, I'd be happy to know!