⭐ Backup cleanser
Github page: karlicoss/bleanser
Bleanser' stands for 'backup cleanser'.
The idea is figuring out 'redundant' backups and removing them to
- save on disk space
- same on data access time (see "data access layer")
This is the most relevant to incremental/synthetic style data exports.
It's not necessarily hard to implement for something specific, but the challenge is to do it in a data source agnostic way,
or at least with as minimum effort as possible.
This is possible for example for JSON: if the export from today is a superset of an export from yesterday, you can safely remove the old export. This actually works surprisingly well as is for many data sources.
For a few I've got slight adjustments that normalise them before comparing by removing certain fields that change often, but not very important. For example, Reddit upvotes/downvotes always jump, so I just exclude them from the comparison.
It's similar to extracting the useful fields, but instead it filters the useless ones. That makes it safer in case new fields are added by the backend, I'd rather keep extra data than potentially lose useful information.
Table of Contents
- related bleanserexportsbackupinfra
- [A] * motivation bleanser
- [A] * ideas bleanser
- [B] * specific data sources bleanser
- [C] * bugs bleanser
- STRT [C] compress backups? bleanserinfra
- bleanserinfra also compressing rtm
- TODO [D] move description to github bleanser
- ----------------------------------- bleanser
- TODO [C] generic sqlite databases export bleanser
- TODO [B] hmm. they serve sort of the same purpose??? bleanserbackupchecker
- TODO [B] maybe 'dynamic' optimizer for bleanser? and later can use it to actually delete stuff bleanserhpi
- bleanserhpi I guess HPI could import it as a dependency..
- TODO [C] github events via triples would be a good example bleanser
- TODO [C] Search results · PyPI bleanser
- [C] trailofbits/graphtage: A semantic diff utility and library for tree-like files such as JSON, JSON5, XML, HTML, YAML, and CSV. bleanser
- TODO [B] ok, triples for browser history are def gonna be impactful bleanser
- [C] Memory Filesystem — PyFilesystem 2.4.13 documentation bleanser
- TODO [B] warn about being disk/tmp intense? bleanser
- TODO [C] write about multiprocessing? bleanser
- TODO [B] 'extract' query bleanser
- TODO [D] hmm, for attributes that can change back and force in json, sorted strategy isn't the best… ugh bleanser
- TODO [A] sqlite: hmm….note sure about cascades… probably need to disable somehow? bleanser
- TODO [B] investigating diffs bleanser
- TODO [C] for properly impressive demo should prob run in single threaded mode? bleanser
- TODO [B] run tox first? to protect from crashes bleansersetup
- TODO [B] could artificially map jsons to line-based format (with full path to the entity?) bleanser
- TODO [B] kinds of snapshots bleansertoblog
- TODO [D] moving old files – not sure what to do about empty dirs? bleanser
- TODO [C] implement 'extract' mode later… after writing to blog definitely bleanser
- TODO [C] readme: gotcha about group boundaries not being removed (nad having empty diff) bleanser
- TODO [C] document what's happening in which case… with a literate test bleanser
- TODO [C] proper end2end test — could run against firefox? reinstalled at about 202006, could track by file size changes bleanser
- TODO [D] multiway is a bit more speculative bleansertoblog
- TODO [B] add for takeouts… I even had some script to compare it somewhere bleanser
- STRT [B] hypothesis, endomondo, pinboard – just use regular json processor bleanser
- TODO [B] old code for 'extract' bit bleanserpinboard
- TODO [C] json: sorting stuff might definitely make it more confusing when there is just one volatile attribute that has two values bleanser
- TODO [C] foursquare is a good motiation – lots of random changing crap even without the changes of underlying data? bleanser
¶related bleanserexportsbackupinfra
¶[A] * motivation bleanser
¶TODO [D] reddit processing takes quite a bit.. but I guess bleanser will optimize it bleanserhpireddit
¶[A] * ideas bleanser
¶TODO [A] pattern of handling unknown data sources bleansertoblog
lower bound
specify data (fields/files etc) to preserve
if you only do that you might miss new useful data/schema changes like renames etc
. ideally they meet here
.. warn if we ended up here, i.e. dropping is not converting with picking. but keep the data
if you only do that you end up with too much garbage
specify data (fileds/files etc) to drop
upper bound
¶[B] * specific data sources bleanser
¶STRT [B] firefox history? bleanserpromnesia
¶TODO [C] github-events – prune via triplet approach? bleanser
¶TODO [D] xml: smscalls bleanser
¶TODO [D] not sure, maybe ignore comment/link karma? it results in lots of differences… bleanserreddit
¶STRT [C] compress backups? bleanserinfra
started compressing reddit…
¶ also compressing rtm bleanserinfra
¶TODO [D] move description to github bleanser
¶----------------------------------- bleanser
¶TODO [C] generic sqlite databases export bleanser
- do not remove; move to killzone
- get all tables
- make sure all schemas match
- maybe convert it to json or something? and then compare jsons…
- checks that entries are dominated?
¶TODO [B] hmm. they serve sort of the same purpose??? bleanserbackupchecker
¶TODO [B] maybe 'dynamic' optimizer for bleanser? and later can use it to actually delete stuff bleanserhpi
¶ I guess HPI could import it as a dependency.. bleanserhpi
¶TODO [C] github events via triples would be a good example bleanser
¶TODO [C] Search results · PyPI bleanser
could name like this…
¶[C] trailofbits/graphtage: A semantic diff utility and library for tree-like files such as JSON, JSON5, XML, HTML, YAML, and CSV. bleanser
¶TODO [B] ok, triples for browser history are def gonna be impactful bleanser
maybe before comparison explicitly 'cleanup' stuff
¶[C] Memory Filesystem — PyFilesystem 2.4.13 documentation bleanser
could use for processing… maybe via option
¶TODO [B] warn about being disk/tmp intense? bleanser
¶TODO [C] write about multiprocessing? bleanser
¶TODO [B] 'extract' query bleanser
might be useful as a sanity check? to ensure stuff isn't deleted by accident? (like foreign key triggers)
e.g.
- run extract query first to get a snapshot of data
- run cleanup query
- run extract query first to ensure the data we care about is there?
¶TODO [D] hmm, for attributes that can change back and force in json, sorted strategy isn't the best… ugh bleanser
¶TODO [A] sqlite: hmm….note sure about cascades… probably need to disable somehow? bleanser
¶TODO [B] investigating diffs bleanser
shell globing nice for not typing too much
python3 -m bleanser.modules.firefox diff /path/to/database-201906{16,17}*.sqlite | less
¶TODO [C] for properly impressive demo should prob run in single threaded mode? bleanser
¶TODO [B] run tox first? to protect from crashes bleansersetup
¶TODO [B] could artificially map jsons to line-based format (with full path to the entity?) bleanser
that way might work more reliably… hmm
¶TODO [B] kinds of snapshots bleansertoblog
- append only (e.g. foursquare, hypothesis)
- rolling (e.g. rescuetime, github, reddit)
either way you can think of it as as set of strings
¶TODO [D] moving old files – not sure what to do about empty dirs? bleanser
maybe keep all dirs that were there before – and only remove new empty dirs?
¶TODO [C] implement 'extract' mode later… after writing to blog definitely bleanser
¶TODO [C] readme: gotcha about group boundaries not being removed (nad having empty diff) bleanser
¶TODO [C] document what's happening in which case… with a literate test bleanser
- e.g. 'all files are same'
- only added data
- rolling data (some fake datetime stuff with 30d retention)
- error in cleaner script
¶TODO [C] proper end2end test — could run against firefox? reinstalled at about 202006, could track by file size changes bleanser
¶TODO [D] multiway is a bit more speculative bleansertoblog
¶TODO [B] add for takeouts… I even had some script to compare it somewhere bleanser
¶STRT [B] hypothesis, endomondo, pinboard – just use regular json processor bleanser
¶TODO [B] old code for 'extract' bit bleanserpinboard
return pipe( '.tags |= .', '.posts |= map({href, description, time, tags})', # TODO maybe just delete hash? '.notes |= {notes: .notes | map({id, title, updated_at}), count}', # TODO hhmm, it keeps length but not content?? odd. )
¶TODO [C] json: sorting stuff might definitely make it more confusing when there is just one volatile attribute that has two values bleanser
e.g. on foursquare with isMayor: true – hmmm