Map of my personal data infrastructure

Well, it's been a year since I started the draft, so I guess it's about time to publish this! :)

This is a map of my personal data liberation infrastructure, with links to the scripts and tools used; and my blog posts elaborating on different parts of it.

My goal for data liberation is approximating the 'personal data mirror' concept, often despite crappy interoperability (or lack thereof) of different platforms.

I prepared this diagram for several reasons:

• to give more context for my blog posts about data liberation and tools around it
• to highlight the complexity and hoops we have to jump over because of the lack of interoperability
• it was also sort of fun :)

This time I won't write too much text and just let you explore it. Tips for exploring the diagram:

• perhaps open the full size SVG in a new tab
• make sure to read the legend
• links you can follow are marked with blue (and sometimes other colours)
• there is a bubble (💬) near some nodes/edges, you can hover it to see the comment
• some integrations are in progress: marked with WIP, construction signs (🚧🚧) and dashed edges
• arrows roughly represent the direction of data flow
• arrow colors roughly correspond to the data source (so it's easier to track how it flows)
• there are some rendering issues
  • it's probably not very mobile friendly (it's barely desktop friendly!)
  • SVG support varies among web browsers, so there might be some minor artifacts (chromium works better, but firefox works well enough)
  • navbar centering isn't broken on this page – it's just a temporary hack to fit in the diagram till I figure out wide pages properly

Some notes regarding the diagram:

• it's plotted via graphviz, and you can find the source here (although the code is quite domain specific)
• even though there is a lot of stuff on the diagram, it's still incomplete!
  • here there is an (also incomplete) list of data I collect/export
  • HPI modules are a good proxy for the data I'm using

• note that despite some platforms dying (e.g. Jawbone/Endomondo), I can still use data produced with them!

  E.g. after Endomondo was discontinued, I was able to quickly switch to open source RunnerUp app, while preserving complete data compatibility.

• note how many services are outright malicious with their anti-API/anti-scraping/anti-interoperability measures (yellow/red highlight for API nodes)
• probably more platforms have GDPR exports, I just haven't tried yet

• indirection is crazy

  Note how for some data, before I can get it on my computer, it goes as

  • device –> phone (over bluetooth)
  • phone –> cloud (over internet)
  • cloud –> computer (over internet)

• for many phone apps the only way I can sync the data is by rooting my phone in order to access the /data/data directory

  This is getting worse and worse with every Android version. I understand the security concerns, but this is ridiculous.

• some modules/packages (marked with^sb superscript) were developed by Sean Breckenridge

  He's forked my HPI package and working on it in parallel. For now, we decided to hack on it independently, in the hope that eventually we figure out what's a good model for cooperating and maintaining the modules.

  Also, he's done some cool work on automatic HTTP API for HPI!