⭐ Data exports & liberation

https://github.com/ChromeDevTools/devtools-protocol

apply this to export helper…

Easiest option is just to have separate scripts to run regularly?

might be annoying to implement token retrieval on JS only?

Even though Twint uses db, they seem to treat is as a temporary storage, so the schema might change.
I'm also not super convinced by how reliable the code is (from quick glance), so would worry about data loss.

One of the bigger disadvantages of the Search API is that you can only access Tweets written in the past 7 days. This is a major bottleneck for anyone looking for older past data to make a model from. With TwitterScraper there is no such limitation.

compare tw-before.org (twint) and tw-after.org (twidump) in views
retweets in twint are def missing

https://news.ycombinator.com/item?id=22788526

Our plan is for the next version of HN's API to simply serve a JSON version of every page. I'm hoping to get to that this year.

https://news.ycombinator.com/user?id=karlicoss

user:	karlicoss
created:	August 25, 2016
karma:	757

capture HN karma? maybe on all comments

could also have 'exact' time notion and 'approximate' time – when it's guessed from the file timestamp etc

don't have ci

• https://github.com/davenicoll/hackernews
  
  • doesn't even have main??
    
• https://github.com/romaintailhurat/hns
  
  • uses pickle??
    
• https://github.com/amjd/HN-Saved-Links-Export
  
  • too defensive
    
  • writes to stdout
    
  • can't be used as API
    

https://github.com/HackerNews/API
https://hacker-news.firebaseio.com/v0/user/karlicoss.json?print=pretty – get user data
extract 'submitted'
https://hacker-news.firebaseio.com/v0/item/25971799.json?print=pretty – comment
https://hacker-news.firebaseio.com/v0/item/25971380.json?print=pretty – type: "story"
dunno if useful to keep scores over time?
not sure if should dump everything in a single json? or split by files?
can change later I guess

Seriously, check out ratarmount if you haven't. Since the Google Takeout spans multiple 50GB tgz files (I'm at ~14, not including Google Drive in the takeout), ratarmount is brilliant. It merges all of the tgz contents into a single folder structure so /path/a/1.jpg and /path/a/1.json might be in different tgz folders but are mounted in to the same folder.

20180807 My Activity/Discover/MyActivity.html                                    20190523 20181015 My Activity/Discover/MyActivity.html                                    20190522 20181213 My Activity/Discover/MyActivity.html                                    20200122

huh, so with my script to search takeout duplicates, I've figured out that from 2015 at least Search/MyActivity.html hasn't been erased? interesting
but looks like Chrome/MyActivity.html still being removed

Takeout/My Activity/Search data is limited to last 10 years. Please remove limit

cuu508 1 hour ago [-]
Takeout doesn't work in practice for bigger collections (archive creation routinely fails, timeouts while downloading, 50GB max size results in many splits)
I've used this 3rd party tool and it worked OK: https://github.com/gilesknap/gphotos-sync/
geekgonecrazy 1 hour ago [-]
I forgot to mention this. But yes the export failed several dozen times. I believe I ended up doing in chunks. It was hard to get them off

You're correct that the methods I described are a far cry from actually guaranteeing that the backup has no errors. In the same way that a unit test doesn't prove code is error-free, but _can_ justify increased confidence in the code, I'm interested in techniques that can justify increased confidence in my backups. Particularly in cases where I don't have direct access to the original data, and where exhaustively checking the data manually is too time-consuming to be worth it.

yes!

https://github.com/perkeep/gphotos-cdp

In our original Perkeep issue, @bradfitz said that we might have to give up on APIs and resort to scraping, noting that the Chrome DevTools Protocol makes this pretty easy.

https://www.reddit.com/r/DataHoarder/comments/c6fh4x/after_hoarding_over_50k_youtube_videos_here_is/

After hoarding over 50k YouTube videos, here is the youtube-dl command I settled on.

we'd like our photos mirrored in seconds or minutes, not weeks.

https://gist.github.com/harperreed/9d063322eb84e88bc2d0580885011bdd
https://gist.github.com/karlicoss/3361f6a239048a451daa2a02982ee180

https://github.com/samuelmr/emfit-qs

Exchange username and password to a token (expires in 7 days). You can also log in to qs.emfit.com and check the ´remember_token´ parameter passed to API calls (e.g. with developer tools of your browser).

https://praw.readthedocs.io/en/v3.6.0/pages/getting_started.html#connecting-to-reddit

You may also have realized that the karma values change from run to run. This inconsistency is due to reddit’s obfuscation of the upvotes and downvotes. The obfuscation is done to everything and everybody to thwart potential cheaters. There’s nothing we can do to prevent this.

> if the official repo is taken down, your forks will disappear unless you have a copy.
https://help.github.com/en/github/collaborating-with-issues-...
I don't think that's true, I've personally recovered deleted repositories by finding its forks.
edit: Ah never mind it seems things work differently in the case of DMCA takedowns

requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.github.com', port=443): Read timed out. (read timeout=15)

 File "/home/karlicos/.local/lib/python3.7/site-packages/github/Requester.py",
line 276, in requestJsonAndCheck
    return self.__check(*self.requestJson(verb, url, parameters, headers,
input, self.__customConnection(url)))
  File "/home/karlicos/.local/lib/python3.7/site-packages/github/Requester.py",
line 287, in __check
    raise self.__createException(status, responseHeaders, output)
github.GithubException.GithubException: 500 None

actually has messages!

https://github.com/MasterScrat/Chatistics

Unfortunately, WhatsApp only lets you export your conversations from your phone and one by one.

    On your phone, open the chat conversation you want to export
    On Android, tap on ⋮ > More > Export chat. On iOS, tap on the interlocutor's name > Export chat
    Choose "Without Media"
    Send chat to yourself eg via Email
    Unpack the archive and add the individual .txt files to the folder ./raw_data/whatsapp/

It seems that recently yowsup gets detected during registration resulting in an instant ban for your number right after registering with the code you receive by sms/voice. I'd strongly recommend to not attempt registration through yowsup until I look further into this. Follow the status of this here.

https://api.stackexchange.com/docs/favorites-on-users

Usage of /users/{ids}/favorites GET
Discussion
Get the questions that users in {ids} have favorited.
This method is effectively a view onto a user's favorites tab.
{ids} can contain up to 100 semicolon delimited ids. To find ids programmatically look for user_id on user or shallow_user objects.
The sorts accepted by this method operate on the following fields of the question object:
    activity – last_activity_date
    creation – creation_date
    votes – score
    added – when the user favorited the question
activity is the default sort.
It is possible to create moderately complex queries using sort, min, max, fromdate, and todate.
This method returns a list of questions.

ERROR:stexport:Giving up fetch_backoff(...) after 1 tries (stackapi.stackapi.StackAPIError: ('https://api.stackexchange.com/2.2/users/706389/privileges/?pagesize=100&page=1&filter=%21LVBj2%28M0Wr1s_VedzkH%28VG&site=alcohol.meta', 502, 'throttle_violation', 'too many requests from this IP, more requests available in 50511 seconds')

Todo promnesia
from ip Lessons learned from writing ShellCheck, GitHub’s now most starred Haskell project – Vidar's Blog

[INFO stexport 2021-03-10 08:33:48,004 export.py:161] exporting dsp: users/{ids}/favorites
[INFO stexport 2021-03-10 08:33:48,302 _common.py:86] Backing off fetch_backoff(...) for 0.5s (stackapi.stackapi.StackAPIError: ('https://api.stackexchange.com/2.2/users/706389/comments/?pagesize=100&page=1&filter=%21LVBj2%28M0Wr1s_VedzkH%28VG&&site=dsp', "('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))", "('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))", "('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))"))
[ERROR stexport 2021-03-10 08:33:49,124 _common.py:101] Giving up fetch_backoff(...) after 2 tries (stackapi.stackapi.StackAPIError: ('https://api.stackexchange.com/2.2/users/706389/favorites/?pagesize=100&page=1&filter=%21LVBj2%28M0Wr1s_VedzkH%28VG&site=dsp', 'Expecting value: line 1 column 1 (char 0)', 'Expecting value: line 1 column 1 (char 0)', 'Expecting value: line 1 column 1 (char 0)'))
Traceback (most recent call last):
  File "/home/adhoc/.local/lib/python3.8/site-packages/stackapi/stackapi.py", line 204, in fetch
    response = response.json()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

https://github.com/kensanata/mastodon-backup

Thus, if every request gets 20 toots, then we can get at most 6000 toots per five minutes.

https://github.com/zigg/grabby

https://github.com/halcy/Mastodon.py

https://twitter.com/Pinboard/status/1113807174717792256

Next question is, does a raw API call give the same results as the website? The API and website search engine run off of different indexes.

# the xochitl binary, if you plan on replacing or modifying it in any way
scp root@remarkable:/usr/bin/xochitl remarkable-backup/

gpsbabel -i garmin_fit,allpoints=1 -f '2020-10-02-161142-TICKR X 076C-1601655102-0.fit' -o unicsv -F res.csv

I regret to inform you but we do not have API or a program in Skype that lets you export your chat history that will include dates, timestamps etc.

https://www.amazon.co.uk/gp/b2b/reports
Then could connect to drebedengi and add comments (even with breakdown)