⭐ Backup cleanser

Github page: karlicoss/bleanser

Bleanser' stands for 'backup cleanser'.

The idea is figuring out 'redundant' backups and removing them to

This is the most relevant to incremental/synthetic style data exports.

It's not necessarily hard to implement for something specific, but the challenge is to do it in a data source agnostic way,
or at least with as minimum effort as possible.

This is possible for example for JSON: if the export from today is a superset of an export from yesterday, you can safely remove the old export. This actually works surprisingly well as is for many data sources.
For a few I've got slight adjustments that normalise them before comparing by removing certain fields that change often, but not very important. For example, Reddit upvotes/downvotes always jump, so I just exclude them from the comparison.
It's similar to extracting the useful fields, but instead it filters the useless ones. That makes it safer in case new fields are added by the backend, I'd rather keep extra data than potentially lose useful information.

Table of Contents

related bleanserexportsbackupinfra

[A] * motivation bleanser

TODO [D] reddit processing takes quite a bit.. but I guess bleanser will optimize it bleanserhpireddit

CREATED: [2019-05-01]

[A] * ideas bleanser

TODO [A] pattern of handling unknown data sources bleansertoblog

CREATED: [2020-12-08]

lower bound
specify data (fields/files etc) to preserve
if you only do that you might miss new useful data/schema changes like renames etc

. ideally they meet here
.. warn if we ended up here, i.e. dropping is not converting with picking. but keep the data

if you only do that you end up with too much garbage
specify data (fileds/files etc) to drop
upper bound

[B] * specific data sources bleanser

STRT [B] firefox history? bleanserpromnesia

CREATED: [2020-06-10]

TODO [C] github-events – prune via triplet approach? bleanser

CREATED: [2020-09-05]

TODO [D] xml: smscalls bleanser

CREATED: [2020-12-08]

TODO [D] not sure, maybe ignore comment/link karma? it results in lots of differences… bleanserreddit

CREATED: [2019-09-29]

[C] * bugs bleanser

TODO [C] shit, dry run still left turds??? bleanser

CREATED: [2019-12-29]

STRT [C] compress backups? bleanserinfra

CREATED: [2018-07-31]

started compressing reddit…

[2019-03-25] also compressing rtm bleanserinfra

TODO [D] [2021-01-11] move description to github bleanser

----------------------------------- bleanser

TODO [C] generic sqlite databases export bleanser

CREATED: [2021-02-11]
  • do not remove; move to killzone
  • get all tables
  • make sure all schemas match
  • maybe convert it to json or something? and then compare jsons…
  • checks that entries are dominated?

TODO [B] hmm. they serve sort of the same purpose??? bleanserbackupchecker

CREATED: [2021-02-11]

TODO [B] maybe 'dynamic' optimizer for bleanser? and later can use it to actually delete stuff bleanserhpi

CREATED: [2021-02-24]

[2021-03-02] I guess HPI could import it as a dependency.. bleanserhpi

TODO [C] github events via triples would be a good example bleanser

CREATED: [2021-02-21]

TODO [C] [2021-03-02] Search results · PyPI bleanser

could name like this…

[C] [2021-02-27] trailofbits/graphtage: A semantic diff utility and library for tree-like files such as JSON, JSON5, XML, HTML, YAML, and CSV. bleanser

TODO [B] ok, triples for browser history are def gonna be impactful bleanser

CREATED: [2021-03-29]

maybe before comparison explicitly 'cleanup' stuff

[C] [2021-04-07] Memory Filesystem — PyFilesystem 2.4.13 documentation bleanser

could use for processing… maybe via option

TODO [B] warn about being disk/tmp intense? bleanser

CREATED: [2021-04-07]

TODO [C] write about multiprocessing? bleanser

CREATED: [2021-04-05]

TODO [B] 'extract' query bleanser

CREATED: [2021-04-09]

might be useful as a sanity check? to ensure stuff isn't deleted by accident? (like foreign key triggers)

  • run extract query first to get a snapshot of data
  • run cleanup query
  • run extract query first to ensure the data we care about is there?

TODO [D] hmm, for attributes that can change back and force in json, sorted strategy isn't the best… ugh bleanser

CREATED: [2021-04-08]

TODO [A] sqlite: hmm….note sure about cascades… probably need to disable somehow? bleanser

CREATED: [2021-04-08]

TODO [B] investigating diffs bleanser

CREATED: [2021-04-10]

shell globing nice for not typing too much

python3 -m bleanser.modules.firefox diff /path/to/database-201906{16,17}*.sqlite | less

TODO [C] for properly impressive demo should prob run in single threaded mode? bleanser

CREATED: [2021-04-10]

TODO [B] run tox first? to protect from crashes bleansersetup

CREATED: [2021-04-11]

TODO [B] could artificially map jsons to line-based format (with full path to the entity?) bleanser

CREATED: [2021-04-09]

that way might work more reliably… hmm

TODO [B] kinds of snapshots bleansertoblog

CREATED: [2021-04-05]
  • append only (e.g. foursquare, hypothesis)
  • rolling (e.g. rescuetime, github, reddit)

either way you can think of it as as set of strings

TODO [D] moving old files – not sure what to do about empty dirs? bleanser

CREATED: [2021-04-09]

maybe keep all dirs that were there before – and only remove new empty dirs?

TODO [C] implement 'extract' mode later… after writing to blog definitely bleanser

CREATED: [2021-04-11]

TODO [C] readme: gotcha about group boundaries not being removed (nad having empty diff) bleanser

CREATED: [2021-04-10]

TODO [C] document what's happening in which case… with a literate test bleanser

CREATED: [2021-04-10]
  • e.g. 'all files are same'
  • only added data
  • rolling data (some fake datetime stuff with 30d retention)
  • error in cleaner script

TODO [C] proper end2end test — could run against firefox? reinstalled at about 202006, could track by file size changes bleanser

CREATED: [2021-04-08]

TODO [D] multiway is a bit more speculative bleansertoblog

CREATED: [2021-04-07]

TODO [B] add for takeouts… I even had some script to compare it somewhere bleanser

CREATED: [2021-04-14]

STRT [B] hypothesis, endomondo, pinboard – just use regular json processor bleanser

CREATED: [2021-04-14]

TODO [B] old code for 'extract' bit bleanserpinboard

CREATED: [2021-04-14]
 return pipe(
     '.tags  |= .',
     '.posts |= map({href, description, time, tags})', # TODO maybe just delete hash?
     '.notes |= {notes: .notes | map({id, title, updated_at}), count}',  # TODO hhmm, it keeps length but not content?? odd.

TODO [C] json: sorting stuff might definitely make it more confusing when there is just one volatile attribute that has two values bleanser

CREATED: [2021-04-16]

e.g. on foursquare with isMayor: true – hmmm

TODO [C] foursquare is a good motiation – lots of random changing crap even without the changes of underlying data? bleanser

CREATED: [2021-04-16]
Jump to search, settings & sitemap