Axol: search alerts

Eventually I'll write about it here

Table of Contents

[A] Motivation axol

TODO [B] I don't necessarily want to read everything found by 'scott alexander', but it's still interesting to run search to see the overlap between people? axol

CREATED: [2020-06-25]

TODO why not hnrss? axol

CREATED: [2020-11-30]

it's very likely more convenient to use if you only want a few HN queries, and don't care about historic ones

TODO why axol over rss bridges? axol

CREATED: [2020-12-05]

rss is awesome! downsides

  • might be trickier to do various post-filtering
  • with axol, you can compare results across queries (user summary)
  • can be used with promnesia maybe

STRT motivation: I don't understand how google search alerts work. e.g. try on openbci query (see my old emails from google alert) axol

CREATED: [2018-11-10]

TODO [D] [2019-12-30] Ask HN: Do you still use RSS? | Hacker News axolrsstoblog

I've just started using Feedbin about a month ago, and although my HN firehose feed is at like 1100 something, it definitely limits the rest of the HN feeds. The show and ask feeds are both stuck around 400 something.

[D] [2018-11-24] problems with diff approach axol

random errors, resulting in empty diff axol

small differences in output (e.g. google search) axol

not always interested in items disappearing from query axol

the downside – having to keep the state :(

[C] TW at [2018-10-02] А кто-нибудь знает тулы типа , но чтобы туда можно было вбить грубо говоря любую поисковую кверю, или API (например reddit/github); и оно отслеживало результаты? axol

[2019-10-02] huh axol

[C] [2020-06-22] I feel the same. So many cool things I'd love to learn about, but not enough tim… | Hacker News axolpkm

I feel the same. So many cool things I'd love to learn about, but not enough time.

[B] Similar/existing projects axol

TODO [C] [2019-12-26] awesome-selfhosted/awesome-selfhosted: A list of Free Software network services and web applications which can be hosted locally. Selfhosting is the process of hosting and managing applications instead of renting from Software-as-a-Service providers axol

DONE [B] trackreddit only two subscriptions axol

CREATED: [2018-07-15]

wanted lifelogging

DONE [D] tool to search on reddit or even custom services? special ordering ('least likely' for showing least occuring subreddits). could also do it on rust? axolpkm

CREATED: [2018-09-29]

searched as 'keyword monitoring tool'
tried searching on reddit, but nothing really useful.. – unclear what it's doing

keyword tracking (SERP) – not sure if an overkill..

DONE [2018-11-06] just implement a provider for kibitzr? axolpkm

CANCEL rust? axolpkm

[B] * Make it more user friendly axol

TODO add axol doctor config axolproject

CREATED: [2020-11-30]

also axol doctor to check individual providers + reuse in tests

TODO [B] rely on user config dirs axol

CREATED: [2020-05-25]

TODO [C] provide an asci diagram for crawler + report + feed reader? axol

CREATED: [2020-03-10]

[B] * Blacklisting axol

TODO [B] maybe button to ban user? it would write to config or something? maybe I can even use some public API constructor? axol

CREATED: [2019-08-17]

TODO [B] I suppose pouchdb would be perfect for blacklisting axolcouchdb

CREATED: [2019-09-02]

TODO [B] for blacklisting, instead could just apply custom per-user classes? or even edit them. that would allow to highlight properly axol

CREATED: [2019-09-16]

TODO [C] yeah, blacklisting could both update backend and hide locally axol

CREATED: [2019-08-17]

TODO [C] [2019-04-15] axol results for redditpkm, rendered at Fri 12 Apr 2019 05:07 axol

shit. need to ignore the weapons subreddits
I think generally, my tools needs to have a database…

[C] shit, top lifelogging tweets are on japanese… axol

CREATED: [2019-07-29]

TODO [C] would be interesting to ignore links I already visited from results. It can even be done automatically…. axolpromnesia

CREATED: [2019-07-20]

TODO [D] huh, quite a few bots on reddit? axolreddit

CREATED: [2019-07-27]

bprogramming even maybe?

[C] huh, lots of stuff from twitter is just garbage. need a good way of suppressing it… axol

CREATED: [2019-07-29]

[2020-01-01] twittermypy (211) - Miniflux axol

[2020-01-01] twittermypy (211) - Miniflux axol

/aymk_mypy/status/1211970059205107712 All
twitter_mypy 7 hours ago Original @Witch_Astaroth みどりさん!この垢にしてから相互になった方の中では割と話せたと思ってます笑 来年もよろしくお願いします!

[2020-01-01] twittermypy (111) - Miniflux axol

/mypy2424/status/1211845733210443778 All
twitter_mypy 7 hours ago Original 事実でも噂でも、クズとかいうやつお前はその人より努力してからいえよな〜って思うよ!!!!! 好きな

[2020-01-01] twittermypy (111) - Miniflux axol

/soe1113/status/741281801323175936 All
   twitter_mypy 7 hours ago O

[2020-01-03] twitterlifelogging (20) - Miniflux axol

/jager_atami/status/24390787028 All
twitter_lifelogging 2 days ago Original #udetate #lifelogging 陶房で壺割り 12 個 201

[2020-01-03] twitterquantifiedself (36) - Miniflux axol

/hiperesoterismo/status/1212803558203985920 All
    twitter_quantified_self 4 hours ago Original mis únicos 4 ★

[B] What would be a good UI for axol? axol

TODO [B] I really need some sort of proper frontend browser for it… axol

CREATED: [2020-10-26]

TODO [C] would be nice to have some html dashboard, so it's easy to blacklist terms? axol

CREATED: [2020-01-03]

STRT [B] need a UI to easily add items to axol. e.g. Alexei Kitaev axol

CREATED: [2019-07-18]

maybe some simple cmdline available from anywhere. or org mode as source?

TODO [C] use metabase or something? could use a column to mark as seen? would be much easier than rss axol

CREATED: [2020-12-10]

TODO [B] dunno about rss interface… really need a more efficient way of processing content, reordering, etc axol

CREATED: [2020-05-21]

[C] Queries axol

TODO [A] search for 'data export' or something? axol

CREATED: [2019-09-23]

[2019-12-07] not much on reddit for 'data liberation: axol

[2020-03-10] 'data export' looks promising on github axol

TODO [C] [2020-01-12] - Twitter Search / Twitter axolself

[2020-03-10] right, it looks quite reasonable to have axolself

  • [2020-11-30] very few results though axolself

STRT [C] [2020-01-30] my. package | beepb00p axolpostprivacyqstoread

Interesting experiment! Thanks for sharing :-) You might find this person's musings about such experiments interesting:

TODO [2020-03-01] axol it axolpostprivacyqstoread

STRT [B] [2019-02-15] What Universal Human Experiences Are You Missing Without Realizing It? | Slate Star Codex axolmind

[2019-04-22] actually even found something interesting on gh.. axolmind

[2019-04-22] so trying to google that query axolmind

if looking for past month, that basically results in random keywords
what universal human experiences are you missing without realizing it

[2019-06-13] yeah, twitter feed is not too huge, so could subscribe to it axolmind

TODO [D] [2019-06-29] - Twitter Search axol

[2019-08-09] axol this? axol

  • [2019-08-25] or aaxol for twitter? although doesn't seem to be posted often axol

[C] [2020-01-09] karlicoss/cachew - Twitter Search / Twitter axolcachew

TODO [B] [2020-08-24] All | Search powered by Algolia Noon Universe search axol

STRT [C] mypy – exclude mypython; prioritize topics axolmypy

CREATED: [2020-06-24]

TODO [C] sleep tracking axolsleepqs

CREATED: [2018-12-31]

STRT [C] add bret victor? axolbretvictor

CREATED: [2019-05-20]

[2019-06-13] uh. need a proper interface for it axolbretvictor

  • STRT [2019-06-13] what's the quickest possible way to create guis? still gonna be python config, right? perhaps self-checking! axolbretvictor
    • [2019-06-15] ok, just main function sounds ok.. axolbretvictor

TODO [C] ted chiang – pretty nice to search on twitter axoltedchiang

CREATED: [2018-12-31]

TODO [C] complex numbers group; argonov; transhumanism? axolargonov

CREATED: [2018-11-10]

STRT [B] [2019-06-15] axolargonov

can find some interesting stuff on twitter..

DONE [2019-07-28] "виктор аргонов" got some good results on twitter axolargonov

STRT [C] kobo; spaced repetition? axolspacedrep

CREATED: [2018-11-16]

[2019-12-07] eh, kobo not so interesting.. axolspacedrep

STRT [C] [2018-08-25] scott alexander unsong - Twitter Search axol

TODO could add this to my twitter poller thing (again, via API) or kibitzr? axol

STRT [C] karlicoss! axolself

CREATED: [2018-12-31]

[2019-06-15] doesn't look much on pinboard… axolself

[2019-12-07] not much interesting axolself

STRT [C] cancel scott alexander search alert axol

CREATED: [2020-06-22]

TODO [D] set up alerts for nutrition stuff axol

CREATED: [2018-11-09]

TODO [B] add "lagrangian mechanics"??? axollagrangian

CREATED: [2020-03-09]

[2020-11-30] or 'Hamiltonian'? at least on HN axollagrangian

[C] [2020-03-09] #promnesia axol

GitHub - karlicoss/promnesia - Another piece of your extended mind

search on pinboard? or even axol..

STRT [A] kedr livansky axolkedr

CREATED: [2020-04-27]

STRT [B] exobrain? axolexobrain

CREATED: [2020-04-28]

TODO [D] [2020-05-01] Pinboard bookmarks tagged eeg axol

TODO [D] [2020-05-01] Pinboard bookmarks tagged km axolpkm

STRT [B] memex? esp github axolmemex

CREATED: [2020-05-19]

STRT [B] george hotz? axol

CREATED: [2020-10-26]

DONE [C] add mypy to search?? axol

CREATED: [2019-11-23]

[D] [2019-10-01] tried aaxol for axol

"pocket export" axol

"data liberation" axol

TODO [C] pkm for twitter can probably be removed… axol

CREATED: [2020-06-22]

STRT [C] initial query… axolmypy

CREATED: [2019-10-29]

mypy -from:mypy2424 -from:mypy1031 -from:aymkmypy -to:aymkmypy -from:mypy0229

ugh, not sure how convenient it'd be to filter this shit

TODO cleanup 'extended mind' – certainly lots of crap in the database axol

CREATED: [2020-11-30]

TODO hmm, isn't resolving anything? axolself

CREATED: [2020-11-30]

[D] [2019-12-02] axol results for hackernewspkm, rendered at 02 Dec 2019 11:05 axol


Personal Knowledge database

[D] [2019-12-02] axol results for hackernewspkm, rendered at 02 Dec 2019 11:05 axol


Personal knowledge base

DONE [B] subscribe to more news on QS, BCI and gadgets axolqs

CREATED: [2018-04-29]

DONE regular? axolqs

TODO brain-computer interface axolqsbci

[C] Sources axol

STRT [C] wonder if I could search among hypothesis users… axolhypothesis

CREATED: [2019-04-19]

[2019-06-15] eh, search is a bit weird… axolhypothesis

TODO [D] could add google search too I suppose.. but that's def lowest priority axol

CREATED: [2019-01-02]

STRT [C] implement for reddit. release reddit/github searchers (as library, then import and use) axol

CREATED: [2018-11-27]

STRT [C] youtube? could search quantified self at least axol

CREATED: [2019-06-15]

[2019-07-20] eh, tried few queries and does't look that result appear that often… axol

TODO [C] World be great to search in comments axolreddit

CREATED: [2019-01-11]

TODO [C] hypothesis axol

CREATED: [2019-01-05]

[2019-07-28] not that many results on pkm/quantified self.. axol

[2019-07-28] more on spaced repetition and ted chiang axol

TODO [C] [2019-07-28] Schedule - axol

Current Schedule
    April comments should be available around May 20 ,2018.

TODO [C] [2019-07-28] New API endpoint – Now you can search comments! : redditdev axol

TODO [D] for google search, only notify about new results; not about changes. wonder how? axol

CREATED: [2018-11-11]

[C] [2019-12-28] Search Reddit Comments by User axol

Search through comments of a particular reddit user.

TODO [C] [2020-01-11] pushshift/api: Pushshift API axol

TODO [C] duckduckgo? axol

CREATED: [2019-11-07]

[C] [2019-12-15] hacker-news-favorites-api/main.js at master · reactual/hacker-news-favorites-api axol

const x = require('x-ray')()

hmm, it's got 'paginate'?

TODO [B] [2020-05-18] Hypothesis axol

eh need to run orger I guess? or axol!

TODO [C] could run HN more often axolhackernews

CREATED: [2020-06-03]

also use more generic hooks?

[2020-05-03] [[[lang][0]=Python][import my\..*$ -]] axol

[C] CI/testing axol

TODO HN is very quick, so prob really good to test on (even on CI) axol

CREATED: [2020-11-30]

TODO [B] Sort tags by number of total occurences? axol

CREATED: [2019-07-16]

TODO [B] Use cachew and keep stuff as blobs with id axolcachew

CREATED: [2020-01-31]

Not sure if I should overwrite or update? Could decide later and query with unique ids to start with?

TODO [B] warn when there are too many atom items? axol

CREATED: [2020-06-21]

TODO [B] suppress some feeds in the config? axol

CREATED: [2020-07-09]

TODO [B] [2020-11-21] Show HN: I made an alternative to Google Alerts that listens to social media axol

[2020-12-05] eh, demands to register etc axol

STRT [C] shit, seems that the timestamps are wrong and also I got the link wrong axol

CREATED: [2020-07-18]

might need to work on this: axol/databases/twitterextendedmind.sqlite

TODO [C] Maybe record a video on the phone ? axoldemo

CREATED: [2020-04-19]

STRT [C] maybe check crawled pinboard users for interesting tags/links? axol

CREATED: [2019-01-02]

[2019-06-15] yeah, need to make this bit more effecient.. axol

STRT [C] maybe, summary and 'rendered' are really sort of the same page? just different sorting… axol

CREATED: [2019-07-16]

STRT [C] Def interesting to see user stats axol

CREATED: [2019-07-16]

TODO [C] Sort tags by number of total occurences? axol

CREATED: [2019-07-16]

TODO [C] Maybe better way of normalising? E.g. look at tedchiang and gq article. Display 'bumped' entries separately? Like a different way of sorting axol

CREATED: [2019-07-16]

TODO [C] prepend # in tag? axol

CREATED: [2019-07-28]

TODO [C] could search for interesting tags occurence without them actually being scraped axol

CREATED: [2019-07-30]

TODO [C] might be good to do some sort of fuzzy grouping? axol

CREATED: [2019-08-17]

wonder what's an effecient way of doing it? sort of similarity connected components?
Can Google's AI-powered Clips make people care about lifelogging? - TechCrunch
2017-10-05 01:28 by TheGoogleDotCom
Can Google’s AI-powered Clips make people care about lifelogging?
2017-10-05 01:28 by gauravndhankar
Can Google’s AI-powered Clips make people care about lifelogging?

TODO [C] would be interesting to have explorer for users that looks for some relevant taks/keywords? axolpinboard

CREATED: [2019-11-21]

TODO [C] Hmm also need real-time search and notify I guess? axolhackernews

CREATED: [2019-12-02]

TODO [C] Eh, better idea would be a tag subscription… axolmypy

CREATED: [2020-01-31]

STRT [C] would be nice to have some efficient frontend + backend thing axoltimeline

CREATED: [2019-12-02]

[2019-12-02] hmmm. actually could do it in a twitter account?? axoltimeline

TODO [2019-12-04] could ask on HN? axoltimelineoutbox

TODO [C] [2019-12-24] Edit Feed: - Miniflux axol

Scraper Rules
Rewrite Rules
Title Filter
Content Filter

TODO [C] [2019-12-24] Command Line Usage - Documentation axol

miniflux -config-file /etc/miniflux.conf

TODO [C] could make a filter to release items slowly? e.g. tweets with more than 10 likes, if update pops it up, then it ends up in the feed. although I need 'processed' entries axol

CREATED: [2020-03-10]

[C] [2020-05-27] Axol: Personal automatic news feed – crawl Reddit/Twitter/HN and read as RSS | Hacker News axol

TODO [C] perhaps redefine everything in entities? and have relations – people, subreddits, urls, tags, etc axol

CREATED: [2019-04-15]

TODO [C] rename adhoc to 'search'? axol

CREATED: [2020-07-18]

TODO [C] think about a special tag to mark stuff that should be autoimported in a similar manner my kibitzr thing worked axol

TODO [C] some todos axol

CREATED: [2020-11-29]
  • [ ] move individual data sources to files within the repo.. not even submodules, too much hassle
    if someone needs, they can just import axol.sources.src directly
  • [ ] cleanup the json shit.. ideally use some proper library
  • [ ] not sure what to do with RSS feeds.. but could start with HTML report generation
  • [ ] query language:
    might be better to adopt
    github:some query
    not sure what to do with colons though.. but maybe think about this later. most won't support searching them anyway

TODO [C] def should keep original results in the DB as far as possible axol

CREATED: [2020-11-30]

TODO [C] to start with, only support exact queries? e.g. demand them in queries and mention that support for fuzzier might be added later axol

CREATED: [2020-11-30]

TODO [C] think about multiple small databases vs one huge? axol

CREATED: [2020-11-30]

multiple small:

  • easier to mess with/explore
  • easier concurrency
  • easier to remove from reports (although for that need to make sure it's really 1-1 correspondence with source and query? dunno)

single db:

  • easier to bulk clean/somewhat easier to bulk normalise
    although this would be kind of useless if I store raw json outputs
  • easier to do queries across multiple (e.g. associating users?)

TODO [C] thinking about query language axol

CREATED: [2020-11-30]

how it could look in adhoc mode
github:'scott alexander' twitter:'scott alexander'

in config, allow something nicer like
[twitter,github,reddit]:'scott alexander'

or [twitter,github,reddit, pinboard]:['scott alexander', 'quantified self']

  • [ ] NOTE: echo twitter:'scott alexander' – this is gonna get swallowed by bash… suggest to always quote?
  • [ ] NOTE: treat " and ' the same? twitter does it…
  • [ ] TODO: make sure that query parsing is defensive

TODO [C] for people to try it out it really needs a simplest service possible they can run with docker? ideally without auth etc axol

CREATED: [2020-12-05]

STRT [D] Track most active pinboard users? They might have interesting other stuff axol

CREATED: [2018-11-14]

[2019-07-20] maybe, try to intersect known user's tags and see what they got in common? axol

TODO [D] running under docker results in /app/axol/js/sorttable axol

CREATED: [2020-06-17]

TODO [D] use different font? axol

CREATED: [2019-12-02]

TODO [D] might need two pass algorithm? One for crawling, second for filtering? axol

CREATED: [2019-12-02]

e.g. I crawled quite a bit of pokemon crap, would be good to filter it?

related axolpkmsearchdegoogle

[C] [2019-04-15] Pinboard: network for karlicoss axolpinboard
shit… too many tweets. I need a way to filter the network…

[2021-01-16] in fact it's the most common request to pinboard author apparently axolpinboard

TODO [C] spinboard: something's not right. e.g. try axol

CREATED: [2018-11-29]

querying t:quantified-self

spinboard gives 220 total results. however, on the first page there are 50…
scraper is missing something?

eh. sooo, there are no dupes even!! BS4 actually sees only 20 per page (pinboard still gives us '50' in the next url).
whereas chrome does show up 50 entries; but if you go to the second page they are gonna overlap.

TODO must be some pinboard bug?? axolpinboard

STRT [D] [2019-11-06] classes — classes 0.1.0 documentation axol

[2020-02-15] hmm, somethihg I was trying to do in axol?… axol

doesn't look active. all top results are from 2017 axolupspin

CREATED: [2020-04-02]

STRT [D] [2019-09-04] ScriptSmith/socialreaper: Social media scraping / data collection library for Facebook, Twitter, Reddit, YouTube, Pinterest, and Tumblr APIs axolredditscrape

Get the top 10 comments from the top 50 threads of all time on reddit

[C] [2020-05-16] ok, seems to be using real APIs, so overall I'm skeptical. but it's got a nice panel for tokens axolredditscrapeexportsjdoe

TODO [B] pruning – for now via sqlitedbbrowser? make sure it locks the db? axol

CREATED: [2021-02-09]
Jump to search, settings & sitemap