⭐ HPI (Human Programming Interface)

pretty nice format; perhaps I should do that?

wow, that looks quite elaborate and cool!

Solid (Social Linked Data)[1] is a web decentralization project led by Tim Berners-Lee, the inventor of the World Wide Web, developed collaboratively at the Massachusetts Institute of Technology (MIT). The project "aims to radically change the way Web applications work today, resulting in true data ownership as well as improved privacy"[2] by developing a platform for linked-data applications that are completely decentralized and fully under users' control rather than controlled by other entities. The ultimate goal of Solid is to allow users to have full control of their own data, including access control and storage location. To that end, Tim Berners-Lee formed a company called Inrupt to help build a commercial ecosystem to fuel Solid.

Currently supports: Facebook Messenger. Planned: Slack, WhatsApp, Telegram, Signal, Wire

This probably warrants its own post, but I strongly agree with the philosophy of Dogsheep: everything lives in a SQLite database (that you own!), each exporter tool is its own separate CLI, and Datasette is an extremely flexible tool to explore data. The Dogsheep ecosystem is totally self-hosted (you own your data) and free (as in beer), unlike personal data aggregator platforms like Exist.io and Gyroscope.

https://github.com/LifeScopeLabs

Not only in between processes, but also in between languages in a single process. In this POC I spun up a Python interpreter in a Go process and pass the Arrow data buffer between processes in constant time. https://github.com/nickpoorman/go-py-arrow-bridge

hmm would be pretty cool if possible to use

hmm, looks interesting, but it's all dockerized, so might be tricky to expose my data..

it is pretty neat already for analysis with querying and visualization. But your stuff is orders of magnitude bigger. Possibly I will set up HPI for myself some day.

made a super simple CLI plotting thingy, reads numbers on stdin, draws svg to stdout

Open source online database tool and Airtable alternative.

That way would be able to easily import and remember lots of tgings. Just need stable IDs..

https://news.ycombinator.com/item?id=13597068

• https://news.ycombinator.com/item?id=23860281
  

  I evaluated on-premise Redash as an alternative for engineers and analysts who don't want to learn tableau. It's harder to setup than Metabase but more intuitive to use (for someone with SQL expertise).
  

pyspread expects Python expressions in its grid cells, which makes a spreadsheet specific language obsolete. Each cell returns a Python object that can be accessed from other cells. These objects can represent anything including lists or matrices.

Repl from Repo
Instantly run any GitHub repository.

hook it right into open humans

Twitter

ok looks promising

tried https://www.visidata.org/docs/graph/ on bluemaestro

from my.bluemaestro import dataframe
df = dataframe()
import visidata
visidata.view_pandas(df.reset_index()[-1000:])

for all points, it was pretty slow… not sure why

https://datasette.readthedocs.io/en/stable
Datasette is a tool for exploring and publishing data. It helps people take data of any shape or size and publish that as an interactive, explorable website and accompanying API.
Datasette is aimed at data journalists, museum curators, archivists, local governments and anyone else who has data that they wish to share with the world.

e.g. https://github.com/markwk/qs_ledger/blob/master/instapaper/instapaper_data_analysis.ipynb
or lastfm ipynb?

A virtual table is an object that presents an SQL table interface but which is not stored in the database file, at least not directly. The virtual table mechanism is a feature of SQLite that allows SQLite to access and manipulate resources other than bits in the database file using the powerful SQL query language.

sqlite-utils 3.1 adds a new command:
    sqlite-utils analyze-tables my.db
It queries every column of every table and outputs useful statistics about them

Additional dependencies are:
    pandas: for writing from and reading to DataFrames (http://pandas.pydata.org/

hmm this is useful.. wonder if could benefit from it

https://blog.jupyter.org/a-jupyter-kernel-for-sqlite-9549c5dcf551

As a reminder, Replit gives you most of what you need to rapidly build and ship apps in the cloud -- at lightning speed:
    A blazing fast online IDE
    Automatic Package Management
    Automatic hosting
    Automatic SSL/HTTPS
    Domain linking
    A simple and fast Database for persistence
    A secure way to store secrets

integrate with it?

e.g. depending on the 'aspects' the data provider has, would be different plots/dashboards

https://github.com/typesense/typesense

finally need to cooperate with datasette…

e.g.
maybe it's more of a platform to build your own layers etc
akin to spacemacs/doom

dump if install is editable or not
os/python version?

    return previsits_to_history(*args, **kwargs, src='whatever')[0] # TODO meh
src/promnesia/common.py:333: in previsits_to_history
    previsits = list(extr()) # TODO DEFENSIVE HERE!!!
src/promnesia/sources/takeout.py:105: in index
    from my.google.takeout.paths import get_takeouts
    from dataclasses import dataclass
    from ...core.common import Paths

    from my.config import google as user_config
    @dataclass
>   class google(user_config):
        '''
        Expects [[https://takeout.google.com][Google Takeout]] data.
E       TypeError: no positional arguments expected

can handle all sorts of things

~/.config/my/my/config.py

eh. not sure about this section…
can't have config/repos dir and config.py at the same time

also default attributes are pretty important because of caching, error handling policies, etc

my/core/init.py:40: UserWarning: 'my.config' package isn't found! (expected at /home/karlicos/.config/my). This is likely to result in issues.
See https://github.com/karlicoss/HPI/blob/master/doc/SETUP.org#setting-up-the-modules for more info.
  """.strip())
✅ config file: my/config/__init__.py
❌ mypy check: failed
   Can't find package 'my.config'

http://localhost:3000/user_credentials

Your Credentials
Credentials are used to store values used by many Agents. Examples might include "twitter_consumer_secret", "user_full_name", or "user_birthday".

that's quite nice; would be cool to display credentials for my kron thing?

• generally prefer mutable? errors would be rare, no one expects modifying data to do anything meaningful – and if they do, they really want it?
  
• better to make immutable via Final (so it's possible to check statically, but have an escape hatch if necessary)
  

better not to use it:

• causes early inputs() calls
  
• propagating all the way down wrappers (dataframes/aggregate functions etc) is unrealistic anyway
  
• easy to override inputs() in tests, etc, sort of 'dependncy injection'
  

downsides:

• unclear how to cachew()???
  although could just have lazy inputs() key. not that bad.
  

that could help for configuration mistakes

Add an example, maybe with dynamic my.config module

hmmm… nice
maybe could try it dith defensive behaviour…

about 3803 files

  du -ch *.xz | tail -n 1
2.1G	total
  du -ch *.zstd | tail -n1
2.9G	total

(every tenth file, cache disabled)

lz4 : ./test  31.20s user 2.58s system 101% cpu 33.285 total
zstd: ./test  21.37s user 2.52s system 103% cpu 23.007 total

I mean, 1.5x is kinda nice…

e.g. in mycfg

class commits:
   roots: Sequence[PathIsh] = [L]

in my.commits

@dataclass
class commits_cfg(user_config):
    roots: Sequence[PathIsh] # --- this complains    ValueError: mutable default <class 'list'> for field roots is not allowed: use default_factory. shit
    emails: Optional[Sequence[str]] = None
    names: Optional[Sequence[str]] = None

huh, so adding roots equals field(default_factory=list) solved it?…

requires a bit of cooperation by using isinstance check in DAL? … maybe inputs should take str, dunno

wtf?????

has 'playbackDate' in episodes table
seems that only podcastAddict.db is useful, the rest is just crap

>>> org.orgdir  # Obtained automatically from org-directory variable in Emacs
OrgDir('/home/jlumpe/org/')

huh that's quite mad!

tests/takeout.py::test_location_perf
  /home/karlicos/.local/lib/python3.7/site-packages/ijson/compat.py:47: DeprecationWarning:
  ijson works by reading bytes, but a string reader has been given instead. This
  probably, but not necessarily, means a file-like object has been opened in text
  mode ('t') rather than binary mode ('b').
    warnings.warn(_str_vs_bytes_warning, DeprecationWarning)

huh, that's random

datetime.datetime(2012, 5, 8, 17, 37, 28, 181000, tzinfo=<DstTzInfo 'Europe/Moscow' MSK+4:00:00 STD>),
  'Europe/Moscow'),
 (datetime.datetime(2012, 5, 8, 20, 46, 27, 16000, tzinfo=<DstTzInfo 'Asia/Novosibirsk' +07+7:00:00 STD>),
  'Asia/Novosibirsk'),
 (datetime.datetime(2012, 5, 8, 20, 50, 3, 274000, tzinfo=<DstTzInfo 'Asia/Novosibirsk' +07+7:00:00 STD>),
  'Asia/Novosibirsk'),

"creation_date": null,
 "category": 12,
 "uuid": "7ce6b090-5099-4cd0-83ae-1a02725c868b",
 "muscles": [
     12
 ],
 "license": 1,
 "name": "Pull-ups"

ok, nice it already has muscles involved.. I could use this data

https://github.com/twintproject/twint/issues/786

twint -u karlicoss --retweets

ugh. most libraries are outdated…
https://github.com/thampiman/reverse-geocoder
some hackery…

import geopy
from geopy.geocoders import Nominatim
from geopy.extra.rate_limiter import RateLimiter
locator = Nominatim(user_agent="myGeocoder")
# getloc = RateLimiter(locator.reverse, min_delay_seconds=0.001)
#
from functools import lru_cache

@lru_cache(None)
def query(p):
    print("UNCACHED!! ", p)
    return locator.reverse(p)

def getloc(p):
    lat, lon = p
    lat = round(lat, ndigits=3)
    lon = round(lon, ndigits=3)
    return query((lat, lon))

one will be 'currently liked', another one is 'liked ever'?
so normally I'll use 'liked' playlist, but can safely delete stuff from it

actually could even commit it to github…
use some really really public account?

faker can do that but doesn't support schemas out of the box..

• mimesis: can't do gauss
  

I appreciate not everyone uses the same data as I do.
My point is showing that my private layer is actually pretty thin and you can implement something TODO suiting you by looking at mine as an example.

Same way as TODO think of some analogy? when you're using a todo list app, you've got your own unique pattern. Yet, we all benefit massively from sharing the same infrastructure

could tweet at them/grafana?

have a screenshot

datasette .cache/my.photos.main:_photos  --config max_returned_rows:20000

• [ ] can bookmark the query, so it's basically stateless
  
• [ ] stretch the map height
  
• [ ] need to add JS with file links (and possibly thumbnails? not sure if it's gonna work well)
  

https://github.com/stephen-bunn/file-config

maybe need some nicer algorithm, to prevent flickering (maybe it doesn't happen anymore though)

o Scarygami/location-history-json-converter, me, Author
Streaming parsing (--iterative) is now possible.
The functionality requires ijson to be installed.

see https://stackoverflow.com/questions/41573587/what-is-the-difference-between-venv-pyvenv-pyenv-virtualenv-virtualenvwrappe

make it optional dependency with fallback?

Alias = NamedTuple("Alias", [("field", str)])

hmm, alias could be used as 'Like' type? for make_config

so using it requires guarding the code with if typing.TYPE_CHECKING, which is a bit confusing and bloating.

could have a defensive import in my.core.typing

Your Credentials
    Credentials are used to store values used by many Agents. Examples might include "twitter_consumer_secret", "user_full_name", or "user_birthday".

from my.config import core as user_config # type: ignore[attr-defined]

maybe instead of defining dynamic bits, import stuff from my.module.config? and then override? not sure

• [ ] hpi doctor fails with some inf recursion??
  
• [ ] my.location.takeout module is mentioned but unavailable??
  

and then, location caching also never properly happens. uhoh

for zone in pytz.all_timezones:
    tz = pytz.timezone(zone)
    infos = getattr(tz, '_tzinfos', [])
    for _, _, x in infos:
        tz_lookup[x] = tz

✅ config file : /.config/my/my/config/__init__.py
Compiling '/.config/my/my/config/__init__.py'...
*** OSError: [Errno 30] Read-only file system: '/.config/my/my/config/__pycache__'

✅ OK  : my.reading.goodreads
✅     - stats: {'books': {'books': {'count': 222, 'last': datetime.datetime(2013, 7, 14, 2, 58, 4, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=61200)))}}, 'events': {'events': {'count': 222, 'last': datetime.datetime(2013, 7, 14, 2, 58, 4, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=61200)))}}, 'inputs': {'inputs': {'count': 934}}, 'reviews': {'reviews': {'count': 222}}}

 ❯ python3 xx.py
Traceback (most recent call last)
  File "xx.py", line 6, in <module>
    for x in M.bookmarks():
TypeError: <lambda>() missing 1 required positional argument: 'realf'

need to test behaviour w.r.t order of running local install?

if you are displaying tabular data, present an ncurses interface

feed into visidata?

I believe all data warehouses are limited by the quality of their data model. Most start with good relational intentions over a small domain, but eventually get bogged down arguing how semantic angels might dance on ontological pins. The parts that work become ossified and impossible to change. The system starts to fragment into multiple federated datastores or unstructured file dumps (“big data!”) where you have to build your own integration every time you want to use the data. Someone comes along and proposes a unifying model (“everything is an event!”) and rebuilds the whole thing but with an extra layer of complexity. Someone suggests buying an industry data model instead - surely the data experts will have solved all these problems for us? A skunkworks project spins up and starts implementing the bought model with good relational intentions over a small domain...
I don’t think personal data warehouses are immune to any of these forces.

https://twitter.com/LeahNeukirchen/status/1335669406588923905

I put my IRC logs of the last decade into that, here is a dot for all 489398 lines I wrote:

Disregarding tools which actively collect data (like [`ttt`](https://github.com/seanbreckenridge/ttt)/[`window_watcher`](https://github.com/seanbreckenridge/aw-watcher-window)), I have some other libraries I've created for this project, to provide more context to some of the data.
- [`ipgeocache`](https://github.com/seanbreckenridge/ipgeocache) - for any IPs gathered from data exports, provides geolocation info, so I have location info going back to 2013 (thanks facebook)

What features would I want in my HPI?

wonder why did he do that?
Iterable needs to be iter(), e.g. you can't return list as Iterator

Good writeup. A couple points.
`zope.interface` is more explicit and scalable than `typing.Protocol`s, and more flexible than `abc.ABC`. There's a mypy plugin for it: https://github.com/Shoobx/mypy-zope
> The drawback is that code that changes the representation of its data a lot tends not to be fast code.
That's not a very convincing reason to avoid dataclasses except in the most performance-constrained environments -- and even then I'm doubtful it'd help. Especially with `slots=True`, dataclasses can take less resources.

John Stultz
@johnstultz_work
random idea: Want something that I can point it at various services (imap/rss/other web services like gphotos,twitter) or takeout archives and it will import/dedup/index/archive locally on my system.

Anyway, thanks a lot for building all this stuff. Definitely gonna explore and it also helped me refine some of my thoughts on the subject!

My main objective right now is packaging what I’m working on into something that I can easily get to beta testers.

tried with my.pinboard module for bookmarks() function:

• can infer List[Bookmark]
  
• can infer List[Union[Bookmark, RuntimeError]]
  
• doesn't seems to be able to infer yield… pretty annoying
  

so all in all seems that it would be easier to assume Res[X] and try to guess X?

Url = NewType('Url', str)

what's up with this Bookmark thing??

[ins] In [24]: inspect.signature(Bookmark.url.fget)
Out[24]: <Signature (self) -> <function NewType.<locals>.new_type at 0x7f9c8626ce50>>

update hypothesis link?

MonkeyType collects runtime types of function arguments and return values, and can automatically generate stub files or even add draft type annotations directly to your Python code based on the types collected at runtime.

could use for some magic caching/inference of serialization? not sure

I guess it's kind of 'auto'? Although ETL sounds a bit magical

see what's their approach to credentials