⭐ Backup cleanser

fdupes /data/exports/hypothesis/ -q
hm ok so it dumps
dunno if it's still better to rely on my own implementation… would be easier to test etc probably
maybe double check fdupes output after parsing

lower bound
specify data (fields/files etc) to preserve
if you only do that you might miss new useful data/schema changes like renames etc

. ideally they meet here
.. warn if we ended up here, i.e. dropping is not converting with picking. but keep the data

if you only do that you end up with too much garbage
specify data (fileds/files etc) to drop
upper bound

<toplevel> ::: {"album": "", "artist": "john lennon", "date": "1281296836", "name": "jealous guy"}
<toplevel> ::: {"album": "", "artist": "john lennon", "date": "1281296516", "name": "instant karma"}

might be useful as a sanity check? to ensure stuff isn't deleted by accident? (like foreign key triggers)
e.g.

• run extract query first to get a snapshot of data
  
• run cleanup query
  
• run extract query first to ensure the data we care about is there?
  

and make possible to override its path if the user doesn't want it in the same dir

e.g. useful for cleaning up files without unpacking

on the other hand, it might mean some legit change… e.g. post was edited and extra content added

• append only (e.g. foursquare, hypothesis)
  
• rolling (e.g. rescuetime, github, reddit)
  

either way you can think of it as as set of strings

unclear why glitches are happening – could be the backup tool, could be their api

• e.g. 'all files are same'
  
• only added data
  
• rolling data (some fake datetime stuff with 30d retention)
  
• error in cleaner script
  

2999 <toplevel> ::: {"album": "best of bowie", "artist": "david bowie", "date": "1527013173", "name": "space oddity - 1999 |    3029 <toplevel> ::: {"album": "best of bowie", "artist": "david bowie", "date": "1527013173", "name": "space oddity - 1999
3000 <toplevel> ::: {"album": "believe ep", "artist": "rob hes", "date": "1527012721", "name": "we rise"}                  |         ---------------------------------------------------------------------------------------------------------------------
3001 <toplevel> ::: {"album": "sviib", "artist": "school of seven bells", "date": "1527012721", "name": "open your eyes"}  |    3030 <toplevel> ::: {"album": "sviib", "artist": "school of seven bells", "date": "1527012721", "name": "open your eyes"}
     ----------------------------------------------------------------------------------------------------------------------|    3031 <toplevel> ::: {"album": "believe ep", "artist": "rob hes", "date": "1527012721", "name": "we rise"}

So your process with browser history files is to create a new sqlite version incrementally for each period T, and then have promnesia import them individually? I'm trying to figure out how I should "best" be storing the files locally; right now, I just have multiple machines scp the respective history files to a single location so that they can then be covered by index, but some of the history files (Safari in particular) are 40M and take a couple of minutes to process...

maybe keep all dirs that were there before – and only remove new empty dirs?