⭐ HPI (Human Programming Interface)

The repository: karlicoss/HPI and blog post.

This is a sort of todo-list with raw ideas and things not (yet?) worthy of github issues.

Table of Contents

[A] * similar projects & efforts hpi

TODO [B] [2020-03-21] KrauseFx/FxLifeSheet: Tracking the key metrics of my life hpiqs

wow, that looks quite elaborate and cool!

STRT [C] [2019-12-11] Solid (web decentralization project) - Wikipedia hpisolid

Solid (Social Linked Data)[1] is a web decentralization project led by Tim Berners-Lee, the inventor of the World Wide Web, developed collaboratively at the Massachusetts Institute of Technology (MIT). The project "aims to radically change the way Web applications work today, resulting in true data ownership as well as improved privacy"[2] by developing a platform for linked-data applications that are completely decentralized and fully under users' control rather than controlled by other entities. The ultimate goal of Solid is to allow users to have full control of their own data, including access control and storage location. To that end, Tim Berners-Lee formed a company called Inrupt to help build a commercial ecosystem to fuel Solid.

[2020-01-15] eh.. instructions are a bit mental even for technical users… https://solidproject.org/for-developers/pod-server hpisolid

[2020-01-16] https://solidproject.org/use-solid/apps apps are a bit disorganised and lots of duplicates hpisolid

[2020-01-17] Is it possible to use Solid offline (at least partially)? hpisolid

Yes, the Solid long term vision includes local first and a flexibility of different topologies of patch-passing sync networks. However, there are no implementations yet.

[2020-01-16] https://karlicoss.inrupt.net trying out pod hpisolid

apps are mostly very simplistic. Calendar/notepad/etc.

  • [2020-01-17] https://vincenttunru.gitlab.io/poddit eh, odd. saves bookmarks as xml, really?… hpisolid
  • [2020-01-17] tiddlywiki thing – didn't work hpisolid
  • [2020-01-17] I guess main question is – how do they interact with each other hpisolid

[2020-01-17] When it comes to malicious cyber attacks, an attack on a single source of many people’s data is generally more likely than on an individual level. hpisolid

good point

[2020-01-17] overall, it feels a bit raw and I don't see how it can assist existing infrastructure we have hpisolid

[C] [2020-01-17] I guess add this to HPI? hpisolid

Or maybe even separate post, e.g. my experience with solid/camlistore/etc?

TODO [C] [2019-12-18] ErikBjare/chatalysis: Analyse (group)chat messages hpi

Currently supports: Facebook Messenger. Planned: Slack, WhatsApp, Telegram, Signal, Wire

[C] [2020-05-08] How this site works | Jack Reid hpi

[2021-03-21] example of personal HTTP api – should be possible to do similar via HPIAPI hpi

[C] [2020-05-18] Quarantine Notes - Week 10 | Ben Congdon hpi

This probably warrants its own post, but I strongly agree with the philosophy of Dogsheep: everything lives in a SQLite database (that you own!), each exporter tool is its own separate CLI, and Datasette is an extremely flexible tool to explore data. The Dogsheep ecosystem is totally self-hosted (you own your data) and free (as in beer), unlike personal data aggregator platforms like Exist.io and Gyroscope.

TODO [C] [2021-02-28] Archivers hpi

[2021-02-28] qnzl-archivist/archivist hpi

STRT [B] [2021-03-19] LifeScope hpiqsdashboard

[2021-04-12] https://lifescope.io/getting-started hpiqsdashboard

ok, so not exactly clear what are they doing, but definitely worth checking out
seems centralized and talking via APIs?

TODO [2021-04-12] https://vimeo.com/519676585 hpiqsdashboardtowatch

[2021-04-12] I mean, looks cool https://lifescope.io/beta , would be good to chat with them hpiqsdashboard

TODO [B] need to outreach solid etc hpi

CREATED: [2021-04-12]

[A] * 3rd party integrations hpi

TODO [A] [2020-05-14] could caption "HPI meets X" hpitoblog

TODO [A] [2021-02-04] Apache Arrow 3.0 | Hacker News hpi

Not only in between processes, but also in between languages in a single process. In this POC I spun up a Python interpreter in a Go process and pass the Arrow data buffer between processes in constant time. https://github.com/nickpoorman/go-py-arrow-bridge

hmm would be pretty cool if possible to use

STRT [B] [2020-03-18] ricklamers/gridstudio: Grid studio is a web-based application for data science with full integration of open source data science frameworks and languages hpipandas

hmm, looks interesting, but it's all dockerized, so might be tricky to expose my data..

[2020-07-31] ugh, weird website with no infromation and only pricing.. hpipandas

[2020-11-01] hmm ok, soo tried out hpipandas

import sys
# TODO ugh. it's a mess, and struggled to launch it as non-root...
sys.path.insert(0, '/.local/lib/python3.7/site-packages/')
sys.path.insert(0, '/.local/lib/python3.8/site-packages/')

import os
os.environ['MY_CONFIG'] = '/root/.config/my'

import my.emfit as E
df = E.dataframe()
# NOTE had nan error?
xxx = df[['date', 'avg_hr']].dropna()
xxx['date'] = xxx['date'].astype(str)
# NOTE: for some reason just xxx didn't work..
sheet('A1', xxx['date'])
sheet('B1', xxx['avg_hr'])
  • I guess it could use nicer datatypes, e.g. dates etc. not sure if the cells are type aware though (they could 'remember' python type?)
    I guess ideally need to leverage libreoffice + publish to web??
  • nice that can just use python functions
  • I really don't wanna use their builtin editor… maybe best to rely on external files
  • very nice that plots update automatically on data updates
  • for data overlays could use a separate column

TODO [B] [2020-05-09] Live demo · andrey-utkin/taskdb Wiki hpi

it is pretty neat already for analysis with querying and visualization. But your stuff is orders of magnitude bigger. Possibly I will set up HPI for myself some day.

TODO [C] [2021-02-08] Bram Wiepjes / baserow · GitLab hpiexobrain

Open source online database tool and Airtable alternative.

TODO [B] Hpi to anki interface? hpisrs

CREATED: [2021-02-23]

That way would be able to easily import and remember lots of tgings. Just need stable IDs..

STRT [B] def try redash, seems to be a 'free' interface?? hpiinfluxdb

CREATED: [2021-02-12]


  • https://news.ycombinator.com/item?id=23860281

    I evaluated on-premise Redash as an alternative for engineers and analysts who don't want to learn tableau. It's harder to setup than Metabase but more intuitive to use (for someone with SQL expertise).

STRT [B] [2020-05-06] Find alternative to Google Data Studio · Issue #34 · KrauseFx/FxLifeSheet hpiinfluxdbspreadsheetdegoogle

Have you considered Redash? It's open source, self hosted and pretty solid.

google docs-like interface

TODO [B] [2020-10-31] Welcome to pyspread | pyspread hpipythonspreadsheets

pyspread expects Python expressions in its grid cells, which makes a spreadsheet specific language obsolete. Each cell returns a Python object that can be accessed from other cells. These objects can represent anything including lists or matrices.

TODO [B] [2020-01-10] Repl.it - Feed https://repl.it/talk/all?lang=python_turtle hpiprojectpromnesiademo

Repl from Repo
Instantly run any GitHub repository.

TODO [B] [2020-02-03] Foreign data wrappers - PostgreSQL wiki hpi


STRT [C] hpiobservable

CREATED: [2021-01-14]

STRT [C] trying out visidata hpivisidata

CREATED: [2020-07-31]

ok looks promising

tried https://www.visidata.org/docs/graph/ on bluemaestro

from my.bluemaestro import dataframe
df = dataframe()
import visidata

for all points, it was pretty slow… not sure why

STRT [C] demo: def need demonstration with Memacs… hpimemacs

CREATED: [2020-04-24]

STRT [C] [2019-12-20] Datasette — Datasette documentation tool for exploring data? hpi

Datasette is a tool for exploring and publishing data. It helps people take data of any shape or size and publish that as an interactive, explorable website and accompanying API.
Datasette is aimed at data journalists, museum curators, archivists, local governments and anyone else who has data that they wish to share with the world.

[2020-01-17] ok, so it's like a web UI for sqlite hpi

  • [2020-09-21] wonder if could reuse it for padnas? def could convert pandas to sqlite and use it hpi

[2020-11-14] tried with photos https://news.ycombinator.com/item?id=25090218 hpi

TODO [C] demo: with QS ledger: should be easy it is to integrate hpiqstoblog

TODO [C] [2021-01-01] List Of Virtual Tables hpisqlite

A virtual table is an object that presents an SQL table interface but which is not stored in the database file, at least not directly. The virtual table mechanism is a feature of SQLite that allows SQLite to access and manipulate resources other than bits in the database file using the powerful SQL query language.

TODO [C] use gephi as frontend? hpi

CREATED: [2020-12-02]

TODO [C] [2020-12-14] Simon Willison (@simonw): "sqlite-utils 3.1 adds a new command: sqlite-utils analyze-tables my.db It queries every column of every table and outputs useful statistics about them: https://sqlite-utils.readthedocs.io/en/stable/changelog.html#v3-1" | nitter hpi

sqlite-utils 3.1 adds a new command:
    sqlite-utils analyze-tables my.db
It queries every column of every table and outputs useful statistics about them

[C] [2021-02-14] influxdata/influxdb-python: Python client for InfluxDB hpiinfluxpandas

Additional dependencies are:
    pandas: for writing from and reading to DataFrames (http://pandas.pydata.org/

hmm this is useful.. wonder if could benefit from it

[C] [2020-06-16] A Jupyter Kernel for SQLite hpi

[2020-12-05] hmm, kinda nice, but not sure if it's better than sqlitebrowser when used locally hpi

TODO [C] grafana is good for gaps in data? hpigrafana

CREATED: [2021-02-09]

TODO [C] Make simplified data projections, so plots can be generated in couple of pandas commands hpitoblogdashboard

CREATED: [2019-12-25]

TODO [C] [2021-02-11] Repl.it - Hosting Apps with Always On hpipromnesiacomputing

As a reminder, Replit gives you most of what you need to rapidly build and ship apps in the cloud -- at lightning speed:
    A blazing fast online IDE
    Automatic Package Management
    Automatic hosting
    Automatic SSL/HTTPS
    Domain linking
    A simple and fast Database for persistence
    A secure way to store secrets

TODO [C] HPI should be able to generate blueprints for grafana; maybe even send to the API? hpigrafana

CREATED: [2021-02-13]

e.g. depending on the 'aspects' the data provider has, would be different plots/dashboards

TODO [C] maybe load all to postgres? would also work nice with #cachew …. hpigrafana

CREATED: [2021-02-12]

TODO [C] ok, guess I need to figure out a nice interface to connect to grafana? hpigrafana

CREATED: [2021-02-12]

TODO [C] upload stats; with coverage? hpiemfitgrafana

CREATED: [2021-02-18]

TODO [B] [2021-03-05] Simon Willison (@simonw) / Twitter hpi

finally need to cooperate with datasette…

[A] * documentation & user friendliness improvements hpi

CREATED: [2020-05-18]

STRT [B] Add data arbitration example for twitter hpitoblog

CREATED: [2020-04-15]

TODO [2020-05-06] myinfra: takeout merging/synthetic export is also a good example hpitoblog

STRT [B] overlay/extension docs: document how to 'overlay' parts of it? with RSS or twitter as examples hpi

CREATED: [2020-05-13]

maybe it's more of a platform to build your own layers etc
akin to spacemacs/doom

STRT [B] example of what you'd want to modify/hack in the module: localize datetimes hpi

CREATED: [2020-10-08]

STRT [C] rename all.py to main.py ? although a little misleading as well hpi

CREATED: [2020-10-05]

STRT [C] think about the guide to overriding sources? hpi

CREATED: [2020-05-22]

I guess ideally, the user could explicitly set exportdir = None for such source and it would be defensive? dunno.

STRT [C] [2020-09-20] Creating and discovering plugins — Python Packaging User Guide hpi

maybe doctor (or 'modules') could list that? with paths (e.g. so default/third party are easily distinguished apart)

STRT [B] specify export formats in docstrings and autogenerate? hpi

CREATED: [2020-01-07]

TODO [B] doctor diaginfo command? hpi

CREATED: [2020-12-07]

dump if install is editable or not
os/python version?

TODO [B] config: improve config helper to warn about the attributes? hpi

CREATED: [2020-05-10]
    return previsits_to_history(*args, **kwargs, src='whatever')[0] # TODO meh
src/promnesia/common.py:333: in previsits_to_history
    previsits = list(extr()) # TODO DEFENSIVE HERE!!!
src/promnesia/sources/takeout.py:105: in index
    from my.google.takeout.paths import get_takeouts
    from dataclasses import dataclass
    from ...core.common import Paths

    from my.config import google as user_config
>   class google(user_config):
        Expects [[https://takeout.google.com][Google Takeout]] data.
E       TypeError: no positional arguments expected

TODO [B] doc: core primitive: get_files hpi

CREATED: [2021-02-13]

can handle all sorts of things

STRT [B] readme: keep list of packages other people forked? suggest to send me links hpi

CREATED: [2020-01-19]

[A] [2020-08-26] readme: seanbreckenridge/HPI: Human Programming Interface hpi

Modified from karlicoss/HPI to more fit my life

STRT [C] list supported providers in readme hpi

CREATED: [2020-01-16]

STRT [C] Add check methods to each provider, expose in doctor hpi

CREATED: [2020-05-18]

TODO [2020-09-28] would be nice to add a meta-warning about missing stats method?? hpi

STRT [C] [2020-05-18] HPI/SETUP.org at master · karlicoss/HPI hpi


eh. not sure about this section…
can't have config/repos dir and config.py at the same time

STRT [C] [2020-05-08] intake.github.io/status https://intake.github.io/status hpiinspiration

[B] [2020-05-08] intake/README.md at master · intake/intake · GitHub hpiinspiration


Intake: A general interface for loading data
Intake [0] is another package that might help here. It organizes a set of data sources into
(1) plugins that actually connect to the data source and map the data to standard Python data structures like Data Frames
(2) catalogs that reference the plugins you want to use alongside project specific metadata like usernames/passwords/source URIs
(3) convenience functions that persistence, concatenation, etc
(4) a GUI for browsing data sources

[2020-05-12] ugh. struggled with installing gui https://intake.readthedocs.io/en/latest/gui.html hpiinspiration

[2020-05-12] anyways, I suppose it's more hpiinspiration

TODO [C] doc: I guess google takeout would be a good demonstration hpi

CREATED: [2020-05-04]

TODO [C] mention that using make_config isn't even necessary if you're not using default attributes hpi

CREATED: [2020-05-10]

also default attributes are pretty important because of caching, error handling policies, etc

[2020-05-10] ah shit, not quite true. works in runtime, but mypy complains… hpi

TODO [C] doc: giude with pyenv if they don't want to mess up their main environment, use docker hpi

CREATED: [2020-09-29]

TODO [C] add a doctor shortcut for profiling? hpi

CREATED: [2020-11-02]

TODO [C] avoid make_config as far as possible, and just use properties directly instead?? it's only necessary for truly complicated hackery hpi

CREATED: [2020-12-13]

TODO [C] doc: when I add strict mode, suggest HPISTRICT=true in troubleshooting guide? hpi

CREATED: [2021-02-19]

TODO [C] update doc about using hpi install [--user] my.modulename hpi

CREATED: [2021-02-18]

TODO [C] eh, doctor doesn't give meaningful res when running in /data/hpi?? hpi

CREATED: [2020-05-27]
my/core/init.py:40: UserWarning: 'my.config' package isn't found! (expected at /home/karlicos/.config/my). This is likely to result in issues.
See https://github.com/karlicoss/HPI/blob/master/doc/SETUP.org#setting-up-the-modules for more info.
✅ config file: my/config/__init__.py
❌ mypy check: failed
   Can't find package 'my.config'

TODO [C] [2019-12-24] inspiration: hugginn credentials inspiration: hpi


Your Credentials
Credentials are used to store values used by many Agents. Examples might include "twitter_consumer_secret", "user_full_name", or "user_birthday".

that's quite nice; would be cool to display credentials for my kron thing?

TODO [D] doc: Hmm i wonder if egg links can be used on systems without symlinks? hpipippython

CREATED: [2020-04-14]

[A] * patterns for writing concise and safe modules hpi

TODO [B] design: dateclasses – mutable vs immutable hpi

CREATED: [2021-02-13]
  • generally prefer mutable? errors would be rare, no one expects modifying data to do anything meaningful – and if they do, they really want it?
  • better to make immutable via Final (so it's possible to check statically, but have an escape hatch if necessary)

STRT [B] check for dataframe 'error' column hpierrorspandas

CREATED: [2020-09-15]

[2020-10-31] or maybe make the dashboard more robust towards it? not sure.. hpierrorspandas

STRT [C] functions with default inputs= argument hpi

CREATED: [2020-09-18]

better not to use it:

  • causes early inputs() calls
  • propagating all the way down wrappers (dataframes/aggregate functions etc) is unrealistic anyway
  • easy to override inputs() in tests, etc, sort of 'dependncy injection'


  • unclear how to cachew()???
    although could just have lazy inputs() key. not that bad.

[2020-10-13] handled in cachew? hpi

STRT [C] keep date as the first attribute to make sorting easier hpi

CREATED: [2020-05-11]

STRT [C] Could use properties for lazier paths hpi

CREATED: [2020-04-27]

STRT [C] top level decorator to check that some data was emitted? hpi

CREATED: [2020-05-24]

that could help for configuration mistakes

STRT [C] maybe have sensible fallbacks for configs? e.g. falling back to documents folder, builtin python packages/site dire/etc hpi

CREATED: [2020-04-14]

WAIT [C] use venv instead of repos symlinks hpi

CREATED: [2020-07-26]

TODO [C] Could use protocols (with Literal types) for pandas dfs??? hpipandasmypy

CREATED: [2020-10-19]

TODO [C] pattern: avoid importing config early so modules can work without it? hpi

CREATED: [2021-02-21]

Add an example, maybe with dynamic my.config module

STRT [D] hmm. could just use ripgreppy parsing for processing htmls?? hpipromnesia

CREATED: [2020-04-23]

[D] [2020-05-16] Lazy — MacroPy3 1.1.0 documentation hpipython

hmmm… nice
maybe could try it dith defensive behaviour…

TODO [D] avoid fromtimestamp? hpipython

CREATED: [2020-05-03]

[2020-09-28] eh? why? as long as the timezone is passed it's fine? maybe have a custom lint for that hpipython

[D] [2020-05-03] reddit: zstd vs lz4 comparison hpiredditexports

about 3803 files

  du -ch *.xz | tail -n 1
2.1G	total
  du -ch *.zstd | tail -n1
2.9G	total

[D] [2020-05-03] comparison of zstd vs lz4 hpireddit

(every tenth file, cache disabled)

lz4 : ./test  31.20s user 2.58s system 101% cpu 33.285 total
zstd: ./test  21.37s user 2.52s system 103% cpu 23.007 total

I mean, 1.5x is kinda nice…

TODO [D] wonder if can make Ipython force the iterators? hpipython

CREATED: [2020-04-24]

[B] [2021-03-15] config: extending base config which has Paths/Pathish and List as the default attribute hpi

e.g. in mycfg

class commits:
   roots: Sequence[PathIsh] = [L]

in my.commits

class commits_cfg(user_config):
    roots: Sequence[PathIsh] # --- this complains    ValueError: mutable default <class 'list'> for field roots is not allowed: use default_factory. shit
    emails: Optional[Sequence[str]] = None
    names: Optional[Sequence[str]] = None

huh, so adding roots equals field(default_factory=list) solved it?…

TODO [B] handles archived stuff via CPath helper hpi

CREATED: [2021-03-26]

requires a bit of cooperation by using isinstance check in DAL? … maybe inputs should take str, dunno

[B] * data providers/sources hpi

TODO [A] the fuck??? after about 16019960000803 the timestamps for individual measurements basically stopped updating?? hpibluemaestro

CREATED: [2021-02-14]


STRT [C] rescuetime: figure out utc or not hpi

CREATED: [2019-11-27]

STRT [C] [2020-07-31] mm, ok, so it's a bit unclear, but https://www.rescuetime.com/apidoc#analytic-api-reference mentions stuff like "defined by the user’s selected time zone"? hpi

so I guess it would mostly be London for me? And I guess I'd need to return dataclasses, 'thaw' them and patch up the timezone? ugh.

STRT [B] podcast addict hpi

CREATED: [2020-12-08]

has 'playbackDate' in episodes table
seems that only podcastAddict.db is useful, the rest is just crap

[2020-12-08] need to merge databases, I lost state at some point? hpi

STRT [B] gpslogger? hpi

CREATED: [2020-04-04]

TODO [B] [2019-09-17] jlumpe/pyorg: Python library for working with Emacs org mode. hpiorg

>>> org.orgdir  # Obtained automatically from org-directory variable in Emacs

huh that's quite mad!

[2020-09-11] could add it as an issue? maybe someone would be interested.. hpiorg

TODO [B] define reasonable ranges for exercises so I could detect errors.. hpiwledger

CREATED: [2019-03-29]

TODO [C] commits – need to cachew? hpicachew

CREATED: [2020-10-12]

TODO [C] reddit: maybe get rid of 'unfavorited'?… hpireddit

CREATED: [2020-07-28]

TODO [C] some ijson warnings hpilocation

CREATED: [2020-05-10]
  /home/karlicos/.local/lib/python3.7/site-packages/ijson/compat.py:47: DeprecationWarning:
  ijson works by reading bytes, but a string reader has been given instead. This
  probably, but not necessarily, means a file-like object has been opened in text
  mode ('t') rather than binary mode ('b').
    warnings.warn(_str_vs_bytes_warning, DeprecationWarning)

STRT [C] Hmm maybe filter reddit and only include every 10th or something to speed it up? hpihpi_patternsreddit

CREATED: [2020-04-13]

TODO [C] check frequencies?? hpibluemaestro

CREATED: [2020-07-11]

TODO [C] timezone provider issues hpilocation

CREATED: [2019-07-24]

huh, that's random

datetime.datetime(2012, 5, 8, 17, 37, 28, 181000, tzinfo=<DstTzInfo 'Europe/Moscow' MSK+4:00:00 STD>),
 (datetime.datetime(2012, 5, 8, 20, 46, 27, 16000, tzinfo=<DstTzInfo 'Asia/Novosibirsk' +07+7:00:00 STD>),
 (datetime.datetime(2012, 5, 8, 20, 50, 3, 274000, tzinfo=<DstTzInfo 'Asia/Novosibirsk' +07+7:00:00 STD>),

TODO [2019-07-24] what's up with LMT+2:30:00 STD? hpilocation

TODO [C] twitter archive trims the retweets (TODO put in the documentation?) hpi

CREATED: [2020-04-29]

TODO [C] [2020-10-15] wger/exercises.json at c70150b4850f2c7ab2fdc7a953c3c11f84d31e8c · wger-project/wger hpiexercise

"creation_date": null,
 "category": 12,
 "uuid": "7ce6b090-5099-4cd0-83ae-1a02725c868b",
 "muscles": [
 "license": 1,
 "name": "Pull-ups"

ok, nice it already has muscles involved.. I could use this data

[2020-10-17] could reuse the same exercise names as wger uses? hpiexercise

TODO [C] stackexchange gdpr has ip addresses hpi

CREATED: [2020-12-04]

TODO [C] location provider: foursquare/swarm hpi

CREATED: [2020-10-18]

TODO [C] twint doesn't get retweets hpitwint

CREATED: [2020-06-04]


twint -u karlicoss --retweets

TODO [C] need to switch all formats to json.. I guess it's still reasonable to have HTML parser because old takeouts didn't have it hpitakeout

CREATED: [2019-05-19]

TODO [B] [2019-09-10] that's also important before releasing hpitakeoutpromnesia

TODO [C] when searching for commits, make sure archived repos are present hpi

CREATED: [2020-10-12]

TODO [C] [2021-02-27] Upvoted submissions | Hacker News hpihackernewsorger

TODO [C] attempt to reverse geocache photos hpi

CREATED: [2020-05-03]

ugh. most libraries are outdated…
some hackery…

import geopy
from geopy.geocoders import Nominatim
from geopy.extra.rate_limiter import RateLimiter
locator = Nominatim(user_agent="myGeocoder")
# getloc = RateLimiter(locator.reverse, min_delay_seconds=0.001)
from functools import lru_cache

def query(p):
    print("UNCACHED!! ", p)
    return locator.reverse(p)

def getloc(p):
    lat, lon = p
    lat = round(lat, ndigits=3)
    lon = round(lon, ndigits=3)
    return query((lat, lon))

TODO [D] old last.fm + spotify + bandcamp arbitration hpi

CREATED: [2020-12-13]

TODO [D] switch pinboard to HPI hpipromnesia

CREATED: [2020-06-08]

TODO [D] hmm, keeping history of when I followed people on twitter could be useful.. hpipromnesia

CREATED: [2020-07-09]

TODO [D] how to make sure all providers are handled? hpitaplog

CREATED: [2020-10-12]

TODO [C] allow timestamp for likes? just make it optional and use in orger if possible hpi

CREATED: [2021-03-16]

TODO [C] pdfs: would be interesting to filter by author? dunno it's often not set/empty… hpi

CREATED: [2021-04-01]

TODO [C] pdfs: maybe extract some meta to build index? hpi

CREATED: [2021-04-03]

TODO [C] inactive windows are sort of useful too hmm hpiarbtt

CREATED: [2021-05-06]

TODO [C] cache pdfs by file? hpicachew

CREATED: [2021-05-07]

TODO [B] youtube playlists hpi

CREATED: [2021-05-20]

TODO [B] script to mirror 'liked' music hpispotify

CREATED: [2021-05-22]

one will be 'currently liked', another one is 'liked ever'?
so normally I'll use 'liked' playlist, but can safely delete stuff from it

[B] * improving testing & CI hpi

[A] also see J. Doe., a project aiming to create an artificial person/data narrative hpi

TODO [B] could test against twitter database hpi

CREATED: [2020-04-16]

actually could even commit it to github…
use some really really public account?

STRT [B] perhaps, 'fake data' generators belong to DAL/HPI as well? that way easy to keep in sync hpijdoe

CREATED: [2020-08-23]

TODO [C] sigh.. mimesis can't generate dates between two other, but works with json 'schemas' hpijdoe

CREATED: [2020-09-19]

faker can do that but doesn't support schemas out of the box..

  • mimesis: can't do gauss

TODO [C] use similar sample data download to bokeh? could be good for testing etc. hpidashboardproject

CREATED: [2020-08-23]

[B] * publicity & reaching out to other people hpi

TODO [B] describe what I've done by the commit history? and comments etc hpitoblog

CREATED: [2020-09-30]

TODO [B] one of the goals is demonstraing the architecture hpitoblog

CREATED: [2020-05-22]

I appreciate not everyone uses the same data as I do.
My point is showing that my private layer is actually pretty thin and you can implement something TODO suiting you by looking at mine as an example.

Same way as TODO think of some analogy? when you're using a todo list app, you've got your own unique pattern. Yet, we all benefit massively from sharing the same infrastructure

TODO [C] planetpython etc? hpipublish

CREATED: [2019-11-23]

TODO [C] [2021-02-15] (6) InfluxData (@InfluxDB) / Twitter hpipublish

could tweet at them/grafana?

TODO [C] about endomondo shutting down and using runnerup for workouts, show the commits hpitweetoutbox

CREATED: [2021-01-12]

TODO [D] publish the thing about HPI configs? hpitoblog

CREATED: [2020-05-10]

TODO [B] tweet about hpi + photos integration hpitotweetoutbox

CREATED: [2020-11-25]

have a screenshot

datasette .cache/my.photos.main:_photos  --config max_returned_rows:20000
  • [ ] can bookmark the query, so it's basically stateless
  • [ ] stretch the map height
  • [ ] need to add JS with file links (and possibly thumbnails? not sure if it's gonna work well)

[B] * enhancements hpi

STRT [B] get rid of individual cache paths? there is no point in them really… hpicachew

CREATED: [2020-07-26]

TODO [C] rename mcachew to cachew?? so it's less confusing hpicachew

CREATED: [2020-09-14]

TODO [C] for stats frequency (e.g. bluemaestro), also need to calculate 'reverse frequency'? e.g. how many 1 minute intervals are covered hpi

CREATED: [2020-08-23]

TODO [C] async interface (possibly async wrapper for the whole modules?) os it emits iterator of dataframes, and when thedata updated, new df is emitted hpi

CREATED: [2020-09-30]

STRT [C] use it in github provider after I support exceptions hpicachew

CREATED: [2019-12-19]

TODO [C] think about overlaying user config on top of default dynamically? not sure… hpi

CREATED: [2020-10-31]

WAIT [D] [2020-04-11] stephen-bunn/file-config: Attrs-like file config definitions inspired from https://github.com/hynek/environ_config hpiconfigs

TODO [C] treat deleted pinboard items as deleted/archived; expliclitly hpipinboard

CREATED: [2020-12-30]

maybe need some nicer algorithm, to prevent flickering (maybe it doesn't happen anymore though)

TODO [C] need to add hooks to config that get executed when HPI is imported? so people could hack without changing apps' code hpi

CREATED: [2020-09-17]

TODO [C] in v2.0, get rid of old my.config artifacts hpi

CREATED: [2020-07-26]

[2021-03-23] or maybe don't quite break it… but keep tests? and some dynamic magic to hack the code dynamically hpi

STRT [C] [2019-12-12] Re: [Scarygami/location-history-json-converter] Streaming parsing (#16) hpilocation

o Scarygami/location-history-json-converter, me, Author
Streaming parsing (--iterative) is now possible.
The functionality requires ijson to be installed.

TODO [C] ok, pipenv seems like a pleasant way of experimenting, maybe give it a shot hpi

TODO [C] maybe have a special tag, e.g. nohpi to shadow entries from hpi? although sometimes nice to retrieve all entries (but basically only one usecase for that, timeline) hpiorg

CREATED: [2020-10-10]

TODO [D] [2019-12-30] esnme/ultrajson: Ultra fast JSON decoder and encoder written in C with Python bindings hpi

make it optional dependency with fallback?

[C] [2020-01-16] didn't seem like it'd be massive speedup? hpi

[C] [2020-05-15] Type alias as a class member is not valid as a type · Issue #7866 · python/mypy hpimypy

Alias = NamedTuple("Alias", [("field", str)])

hmm, alias could be used as 'Like' type? for makeconfig

TODO [C] [2020-05-12] HPI/CONFIGURING.org at master · karlicoss/HPI defensive Protocol stub? hpi

so using it requires guarding the code with if typing.TYPE_CHECKING, which is a bit confusing and bloating.

could have a defensive import in my.core.typing

[2020-09-28] I guess my.core.compat makes more sense for such stuff hpi

[D] [2019-12-24] inspiration: credentials dashboard? Huginn hpi

Your Credentials
    Credentials are used to store values used by many Agents. Examples might include "twitter_consumer_secret", "user_full_name", or "user_birthday".

CNCL [D] 'hpi shell' command launches python shell hpi

CREATED: [2019-12-09]

[2020-03-14] wmy alias to start with hpi

[2020-09-21] and then what? seem complicated hpi

[2020-09-28] actully, pyenv shell in hpi env could make more sense? hpi

STRT [B] configuration hpi

CREATED: [2020-09-29]
from my.config import core as user_config # type: ignore[attr-defined]

maybe instead of defining dynamic bits, import stuff from my.module.config? and then override? not sure

TODO [B] entity guessing: in materialistic module, make sure hpi stat my.materialistic returns last entity time hpi

CREATED: [2021-03-08]

[B] automatic ids hpi

[2021-03-07] if there is a unique timestamp, use it as implicit id? hpi

[2021-03-07] for ID: implement an uid function? not sure how to type it reliably though… hpi

[2021-03-07] ids hpi

IDS should be optional, in that case could use implicit ID like timestamps or even 'index' of the item in sequence. in practice it would be stable enought?
or use hash of some other metadata

[2021-03-08] automatic ID: guess from increasing field? hpi

[C] * bugs/issues hpi

[C] github issues hpi

STRT [C] For demos of my tools generate fake data? Automatically map over fields; maybe env var? hpijdoe

CREATED: [2019-07-11]

STRT [C] wonder if could make http api with smth automatic? not sure how feasible it'd be though.. hpi

CREATED: [2020-01-15]

STRT [C] Have defensive policy for me/other people hpi

CREATED: [2020-01-16]

TODO [C] windows compatibility.. hpi

CREATED: [2021-02-16]
  • [ ] hpi doctor fails with some inf recursion??
  • [ ] my.location.takeout module is mentioned but unavailable??

TODO [C] hmm, iterator in iter_tzs never exhausts? cause it doesn't go over the last year. so it won't cache things?? hpi

CREATED: [2020-10-08]

and then, location caching also never properly happens. uhoh

TODO [C] MYCONFIG should resolve path to absolute? hpi

CREATED: [2020-12-02]

TODO [D] tzabbr hack attempt hpi

CREATED: [2020-10-26]
for zone in pytz.all_timezones:
    tz = pytz.timezone(zone)
    infos = getattr(tz, '_tzinfos', [])
    for _, _, x in infos:
        tz_lookup[x] = tz

TODO [C] doctor: for compileall check need to use tmp path? hpi

CREATED: [2021-03-16]
✅ config file : /.config/my/my/config/__init__.py
Compiling '/.config/my/my/config/__init__.py'...
*** OSError: [Errno 30] Read-only file system: '/.config/my/my/config/__pycache__'

TODO [C] hmm stats look a bit off? hpi

CREATED: [2021-03-26]
✅ OK  : my.reading.goodreads
✅     - stats: {'books': {'books': {'count': 222, 'last': datetime.datetime(2013, 7, 14, 2, 58, 4, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=61200)))}}, 'events': {'events': {'count': 222, 'last': datetime.datetime(2013, 7, 14, 2, 58, 4, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=61200)))}}, 'inputs': {'inputs': {'count': 934}}, 'reviews': {'reviews': {'count': 222}}}

TODO [C] this is when forget parens after @cachew hpi

CREATED: [2021-03-09]
 ❯ python3 xx.py
Traceback (most recent call last)
  File "xx.py", line 6, in <module>
    for x in M.bookmarks():
TypeError: <lambda>() missing 1 required positional argument: 'realf'

[D] * misc hpi

Uncategorized stuff

STRT [B] integrate with private overlay hpi

CREATED: [2020-10-09]

need to test behaviour w.r.t order of running local install?

TODO [B] Automatic sqlite interface.. People seem to like it? hpicachew

CREATED: [2020-05-02]

[2020-09-21] yep, again would be easy to map from pandas? hpicachew

[2020-11-01] that would def be very useful for automatic discovery… maybe start with some markers hpicachew

[2020-11-24] maybe promnesia sqlite module, similarly to doctor.. hpicachew

TODO [B] [2020-12-07] CLI Guidelines – A guide to help you write better command-line programs | Hacker News hpi

if you are displaying tabular data, present an ncurses interface

feed into visidata?

[B] [2020-11-14] Personal Data Warehouses: Reclaiming Your Data | Hacker News hpi

I believe all data warehouses are limited by the quality of their data model. Most start with good relational intentions over a small domain, but eventually get bogged down arguing how semantic angels might dance on ontological pins. The parts that work become ossified and impossible to change. The system starts to fragment into multiple federated datastores or unstructured file dumps (“big data!”) where you have to build your own integration every time you want to use the data. Someone comes along and proposes a unifying model (“everything is an event!”) and rebuilds the whole thing but with an extra layer of complexity. Someone suggests buying an industry data model instead - surely the data experts will have solved all these problems for us? A skunkworks project spins up and starts implementing the bought model with good relational intentions over a small domain...
I don’t think personal data warehouses are immune to any of these forces.

TODO [C] github traffic dashboard for each repo? hpitoblog

CREATED: [2020-10-01]

TODO [C] could write a tool to process historic stats for github referrers etc? hpitoblogcoding

CREATED: [2020-04-29]

[C] [2021-02-23] Yet another Тарантога hpi

[C] [2020-10-05] mention data gathering libraries · seanbreckenridge/HPI@fbe4ffc hpi

Disregarding tools which actively collect data (like [`ttt`](https://github.com/seanbreckenridge/ttt)/[`window_watcher`](https://github.com/seanbreckenridge/aw-watcher-window)), I have some other libraries I've created for this project, to provide more context to some of the data.
- [`ipgeocache`](https://github.com/seanbreckenridge/ipgeocache) - for any IPs gathered from data exports, provides geolocation info, so I have location info going back to 2013 (thanks facebook)

TODO [D] [2020-09-19] iterable -> iterator · seanbreckenridge/HPI@90a16bb hpi

wonder why did he do that?
Iterable needs to be iter(), e.g. you can't return list as Iterator

TODO [D] Slots vs dataclasses hpipythonmypy

CREATED: [2020-02-12]
Good writeup. A couple points.
`zope.interface` is more explicit and scalable than `typing.Protocol`s, and more flexible than `abc.ABC`. There's a mypy plugin for it: https://github.com/Shoobx/mypy-zope
> The drawback is that code that changes the representation of its data a lot tends not to be fast code.
That's not a very convincing reason to avoid dataclasses except in the most performance-constrained environments -- and even then I'm doubtful it'd help. Especially with `slots=True`, dataclasses can take less resources.

TODO [2020-09-28] this could be some sort of meta-attribute? e.g. if someone needs to override, they can monkey patch slots=False? would be nice to add to some FAQ hpipythonmypy

TODO [D] [2020-01-01] John Stultz on Twitter: "random idea: Want something that I can point it at various services (imap/rss/other web services like gphotos,twitter) or takeout archives and it will import/dedup/index/archive locally on my system." / Twitter https://twitter.com/johnstultz_work/status/1156691692772196352 hpiwebarchive

John Stultz
random idea: Want something that I can point it at various services (imap/rss/other web services like gphotos,twitter) or takeout archives and it will import/dedup/index/archive locally on my system.

[D] [2020-05-16] User awal | Lobsters hpi

Anyway, thanks a lot for building all this stuff. Definitely gonna explore and it also helped me refine some of my thoughts on the subject!

DONE [A] [2020-05-02] hyfen.net/memex/updates/putting-the-memex-into-a-container-shazam-other-memex-sightings hpi

My main objective right now is packaging what I’m working on into something that I can easily get to beta testers.

[B] [2021-03-09] tried using monkeypatch to infer output types.. hpitypes

tried with my.pinboard module for bookmarks() function:

  • can infer List[Bookmark]
  • can infer List[Union[Bookmark, RuntimeError]]
  • doesn't seems to be able to infer yield… pretty annoying

so all in all seems that it would be easier to assume Res[X] and try to guess X?

TODO [B] "data repository" is a good description? hpi

CREATED: [2021-03-10]

TODO [C] shit Url alias doesn't work… hpi

CREATED: [2021-03-09]
Url = NewType('Url', str)

what's up with this Bookmark thing??

[ins] In [24]: inspect.signature(Bookmark.url.fget)
Out[24]: <Signature (self) -> <function NewType.<locals>.new_type at 0x7f9c8626ce50>>

TODO [C] [2021-03-07] Hypothesis hpi

update hypothesis link?

STRT [C] [2021-03-08] Instagram/MonkeyType: A system for Python that generates static type annotations by collecting runtime types hpicachew

MonkeyType collects runtime types of function arguments and return values, and can automatically generate stub files or even add draft type annotations directly to your Python code based on the types collected at runtime.

could use for some magic caching/inference of serialization? not sure

[C] [2021-04-12] Extract, transform, load - Wikipedia hpi

I guess it's kind of 'auto'? Although ETL sounds a bit magical

TODO [C] [2021-03-13] qsledger/instapaperdownloader.ipynb at master · markwk/qsledger hpi

see what's their approach to credentials

-------------------------------------------- hpi

TODO [D] classify modules by ease of setup? hpi

CREATED: [2021-04-14]

TODO [B] I'm not inventing anything new – trying to apply simplicity, decomposition and resilience principles hpi

CREATED: [2021-04-14]

TODO [C] ids are useful to mark some stuff as public (e.g. annotations) hpimemex

CREATED: [2021-04-29]

TODO [C] could implement some helper that prompts visidata for column types hpivisidata

CREATED: [2021-05-04]

TODO [B] why ids important: ranking items after overview hpi

CREATED: [2021-05-18]
Jump to search, settings & sitemap