⭐ Data exports & liberation

I need data exports to build tools around my personal data, and the actual process of exporting it from a silo is the first step.

After I export it I use it to build a 'data mirror'.

Here I mostly keep the notes about the data I haven't finished exporting.
The ones I have already/mostly finished are mentioned here:

Table of Contents

related exports

. exportssilo

. exportsinfra

. exportsbackup

. exportsdataliberation

[A] * motivation exports

Similar to backups.

[C] [2020-03-17] "GitHub blocked me and all my libraries" https://news.ycombinator.com/item?id=22593595 exports

TODO [B] motivation for using gdpr/takeouts: convenient when you're migrating off the service? don't have to worry about regular exports exports

CREATED: [2021-02-06]

[B] * implementation: goals/tips/practices exports

[A] [2019-10-03] another big goal is having little operational overhead. I'd rather set up a (potentially elaborate) system once and tthen never have to update it and think how it works exportsinfra

[2019-10-03] that involves automatic ci exportsinfraci

[2019-10-03] continuous cloud sync exportsinfracloud

[2019-10-03] automation/cron jobs for orger exportsinfradron

[A] [2021-03-04] Importance of agnostic exports: ofter you start backing up before you process the data exports

STRT [B] recommend checking the database to make sure it's got specific things you need exportsbackupchecker

CREATED: [2020-01-10]

TODO [B] synthetic style exports allow for defensive error handling – you can at least get data from the last state exportserrors

CREATED: [2020-06-06]

TODO [D] eh. maybe get rid of colored logs for export process? presumably no one would look at them often exports

CREATED: [2020-06-13]

TODO [D] use submodule for common files, but release as a standalone package? I guess it's the best of both exports

CREATED: [2020-07-05]

[C] [2020-01-01] ChromeDevTools/devtools-protocol: Chrome DevTools Protocol exportsscrape

TODO [D] backup-wrapper is a more generic tool… basically running arb command and saving output with pattern exports

CREATED: [2018-11-29]

TODO [B] thinking about data providers exportsdataliberation

CREATED: [2019-09-04]

Easiest option is just to have separate scripts to run regularly?

most users won't care about keeping historic data? Or maybe not keeping data at all? jsut provide lambda? exportsdataliberation

so the backup script could provide TODO

most users won't have cron set up? exportsdataliberation

so need a way to trigger backup from promnesia indexer itself? Fairly easy to achieve as it's all just python code?

to backup, use some python patternt library? exportsdataliberation

example how it could work: exportsdataliberation

in promnesia

def index_reddit():
    from exporters.reddit import export
    # TODO?

TODO [C] dashboard for tokens + expose json or something so any language can have bindings exportsinfra

CREATED: [2020-03-24]

might be annoying to implement token retrieval on JS only?

[2020-04-12] add this to myinfra repository?? exportsinfra

TODO [B] [2020-05-27] dunno, I'm a bit tired and not as motivated to build it… but could post so someone else picks up exportsinfratoblog

TODO [B] ok, exposing a stream is sort of good? and then filtering? makes it easier to use synthetic exports exportshpidal

CREATED: [2021-01-30]

STRT [B] * twitter exports

Twitter is a big pain in the ass, they've become very hostile towards API access.
Even the archives are somewhat incomplete (e.g. favorites lack some metadata).

E.g. from Apply for API — Twitter Developers

Be thorough
We need to completely understand your use case before we can approve it. So, please include as much detail as possible in your application.

TODO [A] hmm. links that you get through search or API are shortened? exportslinkrottwint

CREATED: [2020-04-28]

TODO [2020-04-28] shit.. also RTs are shortened?? so I need to get retweets properly? exportslinkrottwint

TODO [C] twint itself should work as incremental export… and then DAL should combine exportstwint

CREATED: [2020-11-30]

Even though Twint uses db, they seem to treat is as a temporary storage, so the schema might change.
I'm also not super convinced by how reliable the code is (from quick glance), so would worry about data loss.

TODO [D] talon databases (lots of them!) exportshpiandroid

CREATED: [2020-04-28]

STRT [D] [2019-07-29] taspinar/twitterscraper: Scrape Twitter for Tweets exports

One of the bigger disadvantages of the Search API is that you can only access Tweets written in the past 7 days. This is a major bottleneck for anyone looking for older past data to make a model from. With TwitterScraper there is no such limitation.

[2021-02-09] https://github.com/taspinar/twitterscraper/issues/344 broken as well exports

TODO [C] err, new twitter exports are half gig each? exports

CREATED: [2019-08-17]

TODO [C] twint: possibly missing reply things (with 'at') exportstwinthpi

CREATED: [2021-03-16]

compare tw-before.org (twint) and tw-after.org (twidump) in views
retweets in twint are def missing

STRT [B] * hackernews exportshackernews

TODO [B] [2020-04-05] Our plan is for the next version of HN's API to simply serve a JSON version of e… | Hacker News exportshackernews


Our plan is for the next version of HN's API to simply serve a JSON version of every page. I'm hoping to get to that this year.

TODO [D] [2020-04-07] Profile: karlicoss | Hacker News exportshackernews


user:	karlicoss
created:	August 25, 2016
karma:	757

capture HN karma? maybe on all comments

[2020-04-29] need to mirror HN… exportshackernews

TODO [C] materialistic – 'read' table exportshackernewspromnesia

CREATED: [2021-03-29]

could also have 'exact' time notion and 'approximate' time – when it's guessed from the file timestamp etc

[B] [2021-03-05] it's impresive that pretty much every tool for exporting has some flaws exportshackernews

don't have ci

TODO [B] HN data provider exportshackernewshpiorgerpromnesia

CREATED: [2021-01-31]

https://hacker-news.firebaseio.com/v0/user/karlicoss.json?print=pretty – get user data
extract 'submitted'
https://hacker-news.firebaseio.com/v0/item/25971799.json?print=pretty – comment
https://hacker-news.firebaseio.com/v0/item/25971380.json?print=pretty – type: "story"
dunno if useful to keep scores over time?
not sure if should dump everything in a single json? or split by files?
can change later I guess

STRT [C] * google takeout/other google data exportstakeout

Google Takeout doesn't have a proper API, and periodic expots are kind of annoying… would be good to automate them.

Another difficulty is that the data seems to have a certain retention,
so you can't just take the latest takeout, for some data you need to merge all of them.

TODO [B] wonder if it's possible to get watch position? exportstakeoutyoutube

CREATED: [2021-02-03]

TODO [C] automating login & downloading exportstakeout

TODO [B] [2019-09-28] life-vault/seleniumtakeout.py at master · ThorbenJensen/life-vault https://github.com/ThorbenJensen/life-vault/blob/master/src/takeout/selenium_takeout.py exportstakeout

STRT automating google drive exportstakeoutbackup

CREATED: [2019-02-22]
  • ocamlfuse + script to move to desired location
  • basically that only requires you to request new archive occasionally

STRT automate google takeouts? exportstakeout

CREATED: [2018-11-18]

maybe release my module for 2FA separately?

TODO [C] [2021-01-10] Hypothesis exportstakeout

Seriously, check out ratarmount if you haven't. Since the Google Takeout spans multiple 50GB tgz files (I'm at ~14, not including Google Drive in the takeout), ratarmount is brilliant. It merges all of the tgz contents into a single folder structure so /path/a/1.jpg and /path/a/1.json might be in different tgz folders but are mounted in to the same folder.

TODO [C] could sync mini-takeouts? with only necessary stuff picked from them exportstakeout

CREATED: [2019-06-11]

TODO [C] youtube watch history – should be accumulated from multiple takeouts exportstakeoutyoutube

CREATED: [2019-09-17]

[D] [2019-06-11] eh, recompressing to .tar.xz only saved 100 mb exportstakeout

TODO [D] ugh, also when it's too large, they split archive in two exportstakeout

CREATED: [2020-01-23]

TODO [D] also disappearing Disover/Myacvitiy?? exportstakeout

CREATED: [2020-04-24]
20180807 My Activity/Discover/MyActivity.html                                    20190523 20181015 My Activity/Discover/MyActivity.html                                    20190522 20181213 My Activity/Discover/MyActivity.html                                    20200122

[2020-04-23] I've found Google Takeouts to silently remove old data | beepb00p exportstakeout

huh, so with my script to search takeout duplicates, I've figured out that from 2015 at least Search/MyActivity.html hasn't been erased? interesting
but looks like Chrome/MyActivity.html still being removed

[2020-04-24] Takeout/My Activity/Search data is limited to last 10 years. Please remove limit - Google Search Community exportstakeout

Takeout/My Activity/Search data is limited to last 10 years. Please remove limit

TODO [C] [2020-04-29] > I’ve already pulled down my 2-300GB Google Photos archive How? I've tried sev… | Hacker News exportstakeout

cuu508 1 hour ago [-]
Takeout doesn't work in practice for bigger collections (archive creation routinely fails, timeouts while downloading, 50GB max size results in many splits)
I've used this 3rd party tool and it worked OK: https://github.com/gilesknap/gphotos-sync/
geekgonecrazy 1 hour ago [-]
I forgot to mention this. But yes the export failed several dozen times. I believe I ended up doing in chunks. It was hard to get them off

[C] [2020-05-04] I replied to a similar point about hashing here - https://news.ycombinator.com/i… | Hacker News exportstakeout

You're correct that the methods I described are a far cry from actually guaranteeing that the backup has no errors. In the same way that a unit test doesn't prove code is error-free, but _can_ justify increased confidence in the code, I'm interested in techniques that can justify increased confidence in my backups. Particularly in cases where I don't have direct access to the original data, and where exhaustively checking the data manually is too time-consuming to be worth it.


TODO [C] [2020-01-01] perkeep/gphotos-cdp: This program uses the Chrome DevTools Protocol to drive a Chrome session that downloads your photos stored in Google Photos. exportstakeoutscrape


In our original Perkeep issue, @bradfitz said that we might have to give up on APIs and resort to scraping, noting that the Chrome DevTools Protocol makes this pretty easy.

TODO [C] [2019-06-28] After hoarding over 50k YouTube videos, here is the youtube-dl command I settled on. : DataHoarder exportstakeout


After hoarding over 50k YouTube videos, here is the youtube-dl command I settled on.

[C] [2020-01-01] perkeep/gphotos-cdp: This program uses the Chrome DevTools Protocol to drive a Chrome session that downloads your photos stored in Google Photos. https://github.com/perkeep/gphotos-cdp exportstakeout

we'd like our photos mirrored in seconds or minutes, not weeks.

STRT [C] * emfit sleep tracker exportsemfit

Emfit QS is my sleep tracker.

[C] [2019-12-17] downloadEmfitAPI.py https://gist.github.com/vanne02135/6901cc2b92315881080d0ce0f07c1a17 exportsemfit

TODO [C] ugh. maybe autorefresh the token? Fuckig hell. exportsemfit

CREATED: [2020-08-27]

[2021-02-06] I think I ended up just using login + password. meh exportsemfit

TODO [D] hmm, with emfit can code some sort of feedback tool which signals me to move when emfit loses signal exportsemfit

CREATED: [2018-10-16]

[D] [2020-05-29] emfit API didn't work for about three days straight… exportsemfitbackup

[D] [2019-12-21] samuelmr/emfit-qs: Unofficial Node client for Emfit QS exportsemfit


Exchange username and password to a token (expires in 7 days). You can also log in to qs.emfit.com and check the ´remember_token´ parameter passed to API calls (e.g. with developer tools of your browser).

STRT [C] * bluemaestro temperature sensor exportsbluemaestro

STRT [C] figure out bluemaestro, make sure all merged exportsbluemaestro

CREATED: [2019-03-06]
  • State "STRT" from "TODO" [2019-03-12]

TODO [C] actually wonder if I can connect it to computer? exportsbluemaestro

CREATED: [2019-10-06]

STRT [C] merge bluemaestros, plot separate environmental dashboard? exportsbluemaestrodashboard

CREATED: [2020-07-06]

TODO [C] automate, about how I back up bluemaestro data exportsbluemaestrotoblog

CREATED: [2019-02-13]

[2019-09-29] yeah, could elaborate on backing up android data, could be quite generic? exportsbluemaestroandroid

STRT [C] * reddit exportsreddit

TODO [C] I think cool fact should just be converted into org mode from backups (merged!) but generally there is no point capturing them? exportsreddit

CREATED: [2019-01-27]

[2019-09-10] er, I guess for orger need to extract a simple reddit provider that just merges various timestamped backups? exportsreddit

TODO [C] I guess just rely on bleanser instead after all? Just make it less spammy exportsredditbleanser

CREATED: [2019-08-01]

STRT [D] Check for deleted favorites exportsreddit

CREATED: [2019-01-01]
  • State "STRT" from "TODO" [2019-03-23]

[2019-08-25] yep, it def happens; promnesia triggers it exportsreddit

DONE [C] shit. need to bleanse reddit properly, otherwise looks like it's too much data… exportsreddit

CREATED: [2019-04-12]

TODO [C] basically, just go through stuff that doesn't exist anymore but was in favorites ever (and suppress errors for some of them) exportsreddit

CREATED: [2019-01-27]

[C] [2020-01-11] Getting Started — PRAW 3.6.0 documentation exportsreddit


You may also have realized that the karma values change from run to run. This inconsistency is due to reddit’s obfuscation of the upvotes and downvotes. The obfuscation is done to everything and everybody to thwart potential cheaters. There’s nothing we can do to prevent this.

STRT [C] * browser history exports

TODO [D] compress databases as xz? would same about half of space at least, even more on firefox databases exportspromnesia

CREATED: [2019-10-12]

[2020-09-05] probably not necessary with pruning exportspromnesia

STRT [C] cleanup firefox phone exports… exports

CREATED: [2020-05-06]

STRT [C] firefox history – db format has changed?? exportshpiinfra

CREATED: [2020-08-28]

STRT [C] firefox history – could compress with zstd? seems like 30x compression exportspromnesia

CREATED: [2020-06-10]

[2020-06-10] to start with – simply compress locally once the db is synced, will think about doing something smarter later exportspromnesia

STRT [B] firefox dev history exportsphone

CREATED: [2020-12-03]

STRT [C] * hypothesis exportshypothesis

TODO [B] hmm, 9000 limit? might be necessary to do synthetic export instead… exportshypothesis

CREATED: [2019-12-30]

TODO [C] Hypothesis API are cloned as well. exportshypothesis

CREATED: [2020-01-19]

[2020-01-21] fix in hypexport? exportshypothesis

STRT [D] * github exportsgithub

[B] [2020-02-01] motivation for github backups exportsgithub

> if the official repo is taken down, your forks will disappear unless you have a copy.
I don't think that's true, I've personally recovered deleted repositories by finding its forks.
edit: Ah never mind it seems things work differently in the case of DMCA takedowns

TODO [C] warn about large repos? exportsgithub

CREATED: [2019-12-29]

TODO [D] ghexport – read times out exportsgithubghexport

CREATED: [2020-06-22]
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.github.com', port=443): Read timed out. (read timeout=15)

TODO [D] 500 error exportsgithubghexport

CREATED: [2020-06-22]
 File "/home/karlicos/.local/lib/python3.7/site-packages/github/Requester.py",
line 276, in requestJsonAndCheck
    return self.__check(*self.requestJson(verb, url, parameters, headers,
input, self.__customConnection(url)))
  File "/home/karlicos/.local/lib/python3.7/site-packages/github/Requester.py",
line 287, in __check
    raise self.__createException(status, responseHeaders, output)
github.GithubException.GithubException: 500 None

TODO [D] backport old github backups to new format? should be enough to just wrap in 'events' exportsgithubbackuptimelinepromnesia

CREATED: [2019-09-19]

TODO [C] github – starred repos aren't updated?? exportsgithub

CREATED: [2020-03-14]

TODO [D] * whatsapp exportswhatsapp

I don't really use it and it's pretty hostile so unlikely I'll bother.

TODO [B] /data/data/com.whatsapp/databases/msgstore.db exportswhatsapp

CREATED: [2020-01-11]

actually has messages!

[C] [2020-01-17] MasterScrat/Chatistics: 💬 Python scripts to parse your Messenger, Hangouts, WhatsApp and Telegram chat logs into DataFrames. exportswhatsapp


Unfortunately, WhatsApp only lets you export your conversations from your phone and one by one.

    On your phone, open the chat conversation you want to export
    On Android, tap on ⋮ > More > Export chat. On iOS, tap on the interlocutor's name > Export chat
    Choose "Without Media"
    Send chat to yourself eg via Email
    Unpack the archive and add the individual .txt files to the folder ./raw_data/whatsapp/

[C] [2019-07-13] tgalal/yowsup: The WhatsApp lib https://github.com/tgalal/yowsup exportswhatsapp

It seems that recently yowsup gets detected during registration resulting in an instant ban for your number right after registering with the code you receive by sms/voice. I'd strongly recommend to not attempt registration through yowsup until I look further into this. Follow the status of this here.

STRT [D] * stackexchange exportsstackexchange

TODO [C] [2019-09-01] Usage of /users/{ids}/favorites <span class='http-method' title='expects a GET HTTP method'>GET</span> - Stack Exchange API exportsstackexchangepromnesia


Usage of /users/{ids}/favorites GET
Get the questions that users in {ids} have favorited.
This method is effectively a view onto a user's favorites tab.
{ids} can contain up to 100 semicolon delimited ids. To find ids programmatically look for user_id on user or shallow_user objects.
The sorts accepted by this method operate on the following fields of the question object:
    activity – last_activity_date
    creation – creation_date
    votes – score
    added – when the user favorited the question
activity is the default sort.
It is possible to create moderately complex queries using sort, min, max, fromdate, and todate.
This method returns a list of questions.

[2019-09-16] shit. seems that no way to get upvoted posts… https://meta.stackexchange.com/questions/299264/how-to-get-the-list-of-all-posts-ive-upvoted-via-the-api exportsstackexchangepromnesia

TODO [2019-09-16] fuck. I guess I'm gonna have to scrape votes… https://stackoverflow.com/users/706389/karlicoss?tab=votes exportsstackexchangepromnesia

TODO [C] stackexchange – there are comments in GDPR requested data exportsstackexchange

CREATED: [2019-09-17]

TODO [C] stackexchange – shit exportsstackexchange

CREATED: [2019-09-21]
ERROR:stexport:Giving up fetch_backoff(...) after 1 tries (stackapi.stackapi.StackAPIError: ('https://api.stackexchange.com/2.2/users/706389/privileges/?pagesize=100&page=1&filter=%21LVBj2%28M0Wr1s_VedzkH%28VG&site=alcohol.meta', 502, 'throttle_violation', 'too many requests from this IP, more requests available in 50511 seconds')

TODO [C] stackexchange – need to figure out how to import remaining data… exportsstackexchange

CREATED: [2019-10-05]

TODO [C] Today I would probably have tried parsing the Stack Exchange Data Dump instead. exportsstackexchange

[2021-02-06] hmm, it's actual dump of all comments… bit too much I guess exportsstackexchange

TODO [C] hmm crashed on json decoding? exportsstackexchangestexport

CREATED: [2021-03-10]
[INFO stexport 2021-03-10 08:33:48,004 export.py:161] exporting dsp: users/{ids}/favorites
[INFO stexport 2021-03-10 08:33:48,302 _common.py:86] Backing off fetch_backoff(...) for 0.5s (stackapi.stackapi.StackAPIError: ('https://api.stackexchange.com/2.2/users/706389/comments/?pagesize=100&page=1&filter=%21LVBj2%28M0Wr1s_VedzkH%28VG&&site=dsp', "('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))", "('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))", "('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))"))
[ERROR stexport 2021-03-10 08:33:49,124 _common.py:101] Giving up fetch_backoff(...) after 2 tries (stackapi.stackapi.StackAPIError: ('https://api.stackexchange.com/2.2/users/706389/favorites/?pagesize=100&page=1&filter=%21LVBj2%28M0Wr1s_VedzkH%28VG&site=dsp', 'Expecting value: line 1 column 1 (char 0)', 'Expecting value: line 1 column 1 (char 0)', 'Expecting value: line 1 column 1 (char 0)'))
Traceback (most recent call last):
  File "/home/adhoc/.local/lib/python3.8/site-packages/stackapi/stackapi.py", line 204, in fetch
    response = response.json()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

TODO [D] * mastodon exportsmastodon

[C] [2020-01-11] kensanata/mastodon-backup: Archive your statuses, favorites and media using the Mastodon API (i.e. login required) exportsmastodon


Thus, if every request gets 20 toots, then we can get at most 6000 toots per five minutes.

TODO zigg/grabby: tools for scraping your Mastodon account data exportsmastodon

CREATED: [2020-01-13]


TODO [D] [2019-12-29] halcy/Mastodon.py: Python wrapper for the Mastodon ( https://github.com/tootsuite/mastodon/ ) API. exportsmastodon

TODO [C] tusky android app keeps some history in tuskyDb exportsmastodonhpi

CREATED: [2021-01-24]

STRT [D] * pinboard exportspinboard

TODO [D] huh looks like pinboard is quite unstable with regards to backup… unless the backup script is wrong or something? exportspinboardbleanser

CREATED: [2019-03-24]

[C] [2019-04-19] Pinboard on Twitter: "Next question is, does a raw API call give the same results as the website? The API and website search engine run off of different indexes.… https://t.co/CZrLE7YNWo" exportspinboard


Next question is, does a raw API call give the same results as the website? The API and website search engine run off of different indexes.

-------–— other data sources --------------- exports

STRT [B] Podcast addict data exports

CREATED: [2020-08-04]

TODO [B] [2020-07-31] alexattia/Maps-Location-History: Get, Concatenate and Process you location history from Google Maps TimeLine exportslocationtimelineqs

In order to export processed data from Google Maps website from a python script, you need to get your actual cookie.

fuck me! it actually exports kml files

TODO [B] ok, so need to preserve all (incl.older) versions of notebooks? dunno feels a bit excessive exportstimelineremarkable

CREATED: [2020-11-27]

[B] [2020-10-25] Garmin Connect exportsgarmin

Looks like you experienced a time change. This may cause some inaccuracies with today's data.


TODO [B] [2020-12-30] Notice: This project is unmaintained · Issue #613 · fbchat-dev/fbchat exports

STRT [C] Need my email mirrored exports

CREATED: [2019-02-06]

[2019-03-12] I guess I want continuous. Not sure how to achieve that, perhaps some mail client in the background? exports

STRT [C] [2019-06-13] joeyates/imap-backup: Backup GMail (or other IMAP) accounts to disk exports

TODO [C] Bandcamp history exports

CREATED: [2020-12-04]

[2020-12-13] https://bandcamp.com/developer no listening history though… exports

TODO [C] hmm memrise personal data request is neat! It's got all you training sessions + learned words and phrases exportspublish

CREATED: [2019-09-25]

TODO [C] do a full remarkable backup too? exportsremarkable

CREATED: [2020-11-27]

[2020-11-27] tech:ssh [reMarkableWiki] exportsremarkable

# the xochitl binary, if you plan on replacing or modifying it in any way
scp root@remarkable:/usr/bin/xochitl remarkable-backup/

TODO [C] better docs on what to do on expiry exportsmonzo

CREATED: [2020-05-05]
Traceback (most recent call last):
  File "pymonzo/monzo_api.py", line 209, in _get_response
    raise TokenExpiredError
oauthlib.oauth2.rfc6749.errors.TokenExpiredError: (token_expired

this is how token looks like after:

modified: .pymonzo-token
    "code": "internal_service",
    "message": "An error occurred processing the request"

TODO [C] huh, thriva uses an api… exports

CREATED: [2020-03-22]

STRT [C] call history from my old(er?) phones? (e.g. nokia) exports

CREATED: [2020-04-15]

TODO [C] increase sample rate to 10 seconds maybe? exportsarbtt

CREATED: [2020-06-07]

TODO [C] process old 'backups' repo? exports

CREATED: [2020-07-08]

TODO [C] reading hr data exportswahoo

CREATED: [2020-10-09]
import fitparse
ff = fitparse.FitFile('2020-10-02-161142-TICKR X 076C-1601655102-0.fit')
[m.get_value('timestamp') for m in ff.messages]

NOTE: not all messages are hr messages, there is also some metadata etc.

[2020-10-09] also tried gpsbabel, but it resulted in no data… weird exportswahoo

gpsbabel -i garmin_fit,allpoints=1 -f '2020-10-02-161142-TICKR X 076C-1601655102-0.fit' -o unicsv -F res.csv

[C] [2019-04-08] python - Steam API get historical player count of specific game - Stack Overflow exports

There is no Steam Web API method for historical player count of a specific game.

TODO [C] Feedbin starred stuff exports

CREATED: [2019-12-18]

TODO [C] [2019-07-14] fabianonline/telegrambackup: Java app to download all your telegram data. exports


Use --with-supergroups and / or --with-channels to also download all messages from the supergroups / channels you have joined that have been active in the last time.

TODO [C] eh, should include older account? compare oldest and one of newer files.. exportsmonzo

CREATED: [2019-10-15]

TODO [C] myshows: hmm, so looks like api v 1.8 is deprecated, for api 2.0 I'd need to email them. can just use raw jsons from existing backup script exports

CREATED: [2019-07-20]

TODO [C] compress chrome histories? would require backup script to compress it I suppose… maybe just go through them regularly and recompress exports

CREATED: [2019-08-31]

TODO [C] bookmarks limit through api??? exportsinstapaper

CREATED: [2020-01-04]

[2020-01-04] need to check historic exports and figure it out exportsinstapaper

TODO [C] gpslogger – add to backup checker?? exportslocation

CREATED: [2020-07-31]

TODO [C] [2020-10-03] Statify: Pull your playlist and listening data from the Spotify API to a Sqlite database /r/coolgithubprojects exports

STRT [C] monzo export: make sure it works with original repo.. exportsmonzo

CREATED: [2019-12-25]

[2019-12-29] huh, actually the problem might be in saving _token variable? exportsmonzo

TODO [2019-12-29] instead could just print it from disk? maybe even that is unnecessary? exportsmonzo

TODO [C] ugh, need to retrieve pinboard notes exportspinboard

CREATED: [2019-12-29]

e.g. motivational example of API discovery; I just assumed they all would be retrieved https://api.pinboard.in/v1/notes/ID

TODO [C] [2019-04-23] feedbin/feedbin-api: Feedbin API Documentation exportsfeedbin


The base URL for all requests is https://api.feedbin.com/v2/ Only https is supported.
The Feedbin API uses HTTP Basic authentication
curl -u 'example@example.com:password' https://api.feedbin.com/v2/subscriptions.json

STRT [C] [2020-11-27] Success Stories · tcgoetz/GarminDB Wiki exportsgarmin

TODO [C] [2020-12-19] Importing your Goodreads & Accessing them with Open Library’s APIs exports

Importing your Goodreads & Accessing them with Open Library’s APIs

TODO [C] [2020-06-24] Telegram Now Lets You Export Your Chats, View Notification Exceptions | Technology News exportstelegram

TODO [D] get off the messages stored in old format and make sure nothing is missing, dedup? exportsvk

CREATED: [2019-02-26]

TODO [D] shit, they stopped you from accessing messages api. fuck. exportsvk

CREATED: [2019-03-08]


[2019-03-08] that's very generic trend. I think ultimately we just need better tools to scrape that exportsvk

STRT [D] Headspace stats exportstimeline

CREATED: [2018-11-12]


STRT [D] .polar directory exportstimeline

CREATED: [2019-01-20]

TODO [D] [2019-09-02] vincaslt/memparse: A Memrise courses parser https://github.com/vincaslt/memparse exports

STRT [D] skype call history? exports

CREATED: [2020-04-15]

[2020-04-19] shit https://answers.microsoft.com/en-us/skype/forum/all/skype-api/e025d0f6-7ae4-4bc4-9d5a-b2d70136deab exports

I regret to inform you but we do not have API or a program in Skype that lets you export your chat history that will include dates, timestamps etc.

STRT [D] amazon orders history exports

  • State "STRT" from "TODO" [2019-02-23]

TODO [2018-05-04] ugh, order history report is broken for the UK version :( https://www.amazon.co.uk/gp/help/customer/display.html?nodeId=202119330 wrote to support exports

Then could connect to drebedengi and add comments (even with breakdown)

STRT [2019-02-23] backups/amazon exports

  • State "STRT" from [2019-02-23]

TODO ugh, bookmarks method in api is not exhaustive (elif item.get("type") == 'bookmark') exportsinstapaper

CREATED: [2020-01-04]

[D] [2019-04-01] Polar AccessLink Api Daily Activity Goal /r/Polarfitness exports

The API. You do need a session cookie for it and I didn't find an official documentation. The cookie can be retrieved by mimicking their login form. If you do have specific questions you can send me a DM but basically you just need to copy the requests their web app is making.

STRT [D] just reuse files dir? def no harm in it exportstelegram

CREATED: [2020-07-14]

DONE [B] blinkist: scrape off my highlights exports

[2019-08-13] eh, just copy responses manually? exports

[2019-08-13] huh, actually if webdriver could eavesdrop on json responses would be perfect exports

TODO [2019-08-13] post in on github… exportstoblog

DONE [B] export bitbucket exports

CREATED: [2020-01-12]

DONE [D] feedbin exports

CREATED: [2019-05-02]

CANCEL [2020-03-05] signalnerve/roam-backup: Automated Roam Research backups using GitHub Actions and AWS S3 exports


To use it, just fork this repo and add the following secrets to your repo (naming must match!):

---------------------------------------------------- exports

[A] [2020-02-03] Data lake - Wikipedia exportsdal

[B] [2020-04-21] fucking hell. so materialistic export stopped working exportsphone

… because I was copying sqlite file only
and the app suddenly decided to keep everything in WAL. it's been growing over the past week without ever writing into the database
what the fuck??? how do I deal with it???

TODO [B] start awesome-exports list? exportspublish

CREATED: [2020-02-21]

TODO [C] script to grab files from downloads and move accodingly? e.g. for oyster statements exports

CREATED: [2020-09-06]

TODO [C] TW at [2017-01-21] Играюсь с IMDB, думал придется beautiful soap доставать айтемы из вотчлиста, а там в стейте реакта лежит JSONка exports

STRT [C] Post about various ways of data handling exportstoblogdataliberation

CREATED: [2019-09-25]

Mock response from backend
Ignore method
Mypy literal
This is what rust does

[C] [2019-12-27] 'hostage model' is a good term exportstoblogdataliberationsadinfra

referring to software/services when data is held

[C] [2020-01-15] Hi, Camlistore author here. Andrew Gerrand worked with me on Camlistore too and… | Hacker News exportsinfra


bradfitz on Feb 21, 2017 | parent | favorite | on: Upspin – Another option for file sharing
Hi, Camlistore author here.
Andrew Gerrand worked with me on Camlistore too and is one of the Upspin authors.
The main difference I see is that Camlistore can model POSIX filesystems for backup and FUSE, but that's not its preferred view of the world. It is perfectly happy modeling a tweet or a "like" on its own, without any name in the world.
Upspin's data model is very much a traditional filesystem.
Also, upspin cared about the interop between different users from day 1 with keyservers etc, whereas for Camlistore that was not the primary design criteria. (We're only starting to work on that now in Camlistore).
But there is some similarity for sure, and Andrew knows both.

TODO [D] automatic date extraction? could work, e.g. for rescuetime exportsdatetimebackupchecker

CREATED: [2019-04-08]

–— last housekeeping on [2021-02-06] -------- exports

[D] [2020-04-13] twintproject/twint: An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations. exports

[D] [2020-04-23] MatthieuBizien/roam-to-git: Automatic RoamResearch backup to Git exports

 Format [[links]]
Format #links
Format attribute::
Format [[ [[link 1]] [[link 2]] ]]
Format ((link))

[D] [2020-04-28] timgrossmann/InstaPy: 📷 Instagram Bot - Tool for automated Instagram interactions exports

TODO [B] right, so if you enable sync it seems to suck in history on the phone database? eh. messy exportsfirefoxpromnesia

CREATED: [2021-03-29]

TODO [B] crap… android database has really high granulatity of events?? exportsrescuetime

CREATED: [2021-03-04]

[C] [2021-02-05] Chiaki/VKBK: Инструмент для создания и синхронизации локального бэкапа вашего профиля ВКонтакте (Profile backup & synchronization tool for Vk.com) exportsvk

ugh fuck.. apache & mysql? a bit much for me :(

TODO [C] [2021-02-07] Against developer terms of service? · Issue #171 · Tyrrrz/DiscordChatExporter exports

TODO [C] make it kinda smarter? exportsbackupchecker

CREATED: [2021-02-08]

if it's a single file, don't do anything just yet?
or treat it as 'simple' with month duration or something
just do it doesn't warn immediately. could be a takeout archive or something

TODO [C] hmm, takeout has all tcx files? exportsendomondo

CREATED: [2021-02-08]

TODO [C] hide praw logs unless interactive? too spammy in syslog exportsinfra

CREATED: [2021-02-23]

TODO [D] hm nice podcast addict simply backs up its database exports

CREATED: [2021-02-28]

(although it only maintains two?)

TODO [C] Hmm maybe need to check for similar dst problems… Basically mismatch between hr and sleep start/end? exportsemfit

CREATED: [2020-11-01]

[C] [2021-02-25] ryanmcgrath/twython: Actively maintained, pure Python wrapper for the Twitter API. Supports both normal and streaming Twitter APIs. exportspython

hmm still working? nice…

[C] [2021-02-04] Privacy Policy - October 15, 2020 - Reddit exportsreddit

ugh. gdpr takeout has to be emailed?

[C] [2021-02-05] Rapptz/discord.py: An API wrapper for Discord written in Python. exportsdiscord

[C] [2021-02-08] Oura ring vs. Emfit QS (My detailed comparison) - What do you think? - Quantified Self / Sports, Physical Activity, and Fitness - Quantified Self Forum exportsemfit

Can only store 10 hours of data on the device & 360 days in the cloud

huh? motivation for exports I guess

STRT [C] list the takeouts that are reduntant exportstakeoutpromnesia

CREATED: [2019-03-16]

TODO [C] runnerup database file? could use existing computations perhaps? exports

CREATED: [2021-03-28]

TODO [C] maybe for DAL, follow the pattern of exposing a method to read single export? exportshpi

CREATED: [2021-03-08]

so it could cooperate with HPI… egh not sure

TODO [C] Could utilize monzo categories for mine? I guess they could have errors.. Idk exportsmonzo

CREATED: [2021-03-24]

TODO [C] [2021-03-10] Quickstart — StackAPI 0.1.12 documentation exports

By default, StackAPI will return up to 500 items in a single call. It may be less than this, if there are less than 500 items to return. This is common on new or low traffic sites.
The number of results can be modified by changing the page_size and max_pages values. These are multiplied together to get the maximum total number of results. The API paginates the results and StackAPI recombines those pages into a single result.

[C] [2021-03-07] Exporting my own comment content from Disqus? · Discuss Disqus · Disqus exportsdisqus

seems hostile against exporting your own data

[C] [2021-03-23] Your eBay data exportsebay

can request data takeout here… takes ages to complete though, like a week

TODO [B] possible to have exactly same events with different API ids??? exportsgithub

CREATED: [2021-04-02]
vimdiff <(rg -A 363 -B 1 15538293160 events_20210317T120954Z.json) <(rg -A 363 -B 1 15538293166 events_20210317T120954Z.json)

[C] [2021-04-18] exobrain/data/exportsgdpr at master · seanbreckenridge/exobrain exportsgdpr

Some thoughts on how easy to parse/use GDPR/get data exports from different services. A lot of these I did just because I was curious what information/context I could glean into the past about

TODO [C] highlights are in UTC exportsremarkablekoreader

CREATED: [2021-04-29]
Jump to search, settings & sitemap