⭐ Cannon

let alone all the tracking/etc crap

e.g. subreddit-post/user-comment/user-tweet, etc.

https://web.archive.org/web/20090902224414/http://reason.com/news/show/119237.html

however we can sort of get away with normalizing on server only?

• [2019-08-27] not very insigntful, example of msdn with weird characters in urls
  

so you can't connect your annotations on instapaper to notes etc

perhaps not super useful if we can't access them, but still

• [2020-04-29] write about it? the future?
  

• [2021-03-10] although I guess it needs to fetch the page anyway so "rel=canonical" works we ll enough
  

e.g. if the link is not present in archive.org, it doesn't mean it's not archived under a different canonical

e.g. blockers, various highlighters, hypothesis, etc

Once ClearURLs has cleaned the address, it will look like this: https://www.amazon.com/dp/exampleProduct

compareUrls('HTTP://sindresorhus.com/?b=b&a=a', 'sindresorhus.com/?a=a&b=b');

hmm could use this for better extraction…

for the rest

• allow user to customize
  
• allow user to submit normalization errors (where?)
  

Could also be useful for Archive.org/archivebox/etc. But a bit out of scope for this project..

e.g. twitter.com/user/status/statusid

maybe normalise to this?
twitter.com/i/web/status/1053151870791835649

reddit.com/comments/5ombk8 – huh, normalise to this?
TODO m.readdit/old.reddit

en.m.wikipedia/ru.m.wikipedia
maybe stripp off subdom completely?

youtube.com/watch?v=xAy—wp_DQ&list=PL0k_{yDgrqAiUEF5d7krLIds1ebhTxCjm}&shuffle=221
youtube.com/watch?v=Woa3MPijE3s&list=PL0k_{yDgrqAiXKspaa1GIS0jbbLrsAa3sk}&spfreload=10

consider https://www.youtube.com/watch?v=wHrCkyoe72U&list=WL
basically

• cut of protocol just merely for simplicity? I guess makes everything much easier
  
• the result is always 'composed of' inputs. e.g. maps to youtube/wHrCkyoe72U, both parts are in the original link
  might not be the case if domain names are remapped though.. e.g. youtu.be
  
• sort query parts alphabetically
  (although might make sense to make it hierarchy aware?)
  
• treat parts & query the same way, parts are query with None keys
  
• to handle domain names better, replace dots before first / with : e.g. www.youtube.com -> www/youtube/com
  then cat treat the same way as subpaths
  i.e. we get
  None www | drop
  None youtube | keep
  None com | drop
  None watch | drop
  list WL | keep? – actually this could be considered a 'tag'? unclear
  v wHrCkyoe72U | keep
  

ok so how do we generalize from two examples?
e.g. say we also have
youtube.ru/watch?v=abacaba -> youtube/abacaba
we get
youtube | keep
ru | drop
watch | drop
v abacaba | keep
I suppose it could guess that if we keep a query parameter once, we'll keep it always?
and if we extracted a certain substring without a query parameter, we'll also always keep it as is?

TODO how about this?
https://news.ycombinator.com/reply?id=25100810&goto=item%3Fid%3D25099862%2325100810
it's a reply to https://news.ycombinator.com/item?id=25100035
which is a comment to https://news.ycombinator.com/item?id=25099862

http get 'http://archive.org/wayback/available?url=https://stackoverflow.com/questions/1425892/how-do-you-merge-two-git-repositories'
{
    "archived_snapshots": {
        "closest": {
            "available": true,
            "status": "200",
            "timestamp": "20210219235548",
            "url": "http://web.archive.org/web/20210219235548/https://stackoverflow.com/questions/1425892/how-do-you-merge-two-git-repositories"
        }
    },
    "url": "https://stackoverflow.com/questions/1425892/how-do-you-merge-two-git-repositories"
}

abstract_id

eh, rules might be a bit complicated. E.g. if both v and list are present, we wanna ditch list, otherwise keep list

Ah sure: This DOI: https://doi.org/10.1073/pnas.1211902109  should lead to this paper: https://pnas.org/content/109/48/E3324 .

url normalising… this is an example where fragments are important

support for archive.org and test on this page

https://wiki.debian.org/SecureBoot#MOK_-_Machine_Owner_Keycanonical: wiki.debian.org/SecureBootsources : notes[[https://wiki.debian.org/SecureBoot][SecureBoot - Debian Wiki]]

'sid' matters here

ru.wikipedia.org/wiki/Грамматикализация

eh, urls can have commas… e.g. http://adit.io/posts/2013-04-17-functors,_applicatives,_and_monads_in_pictures.html
so, for csv need a separate extractor.

ValueError: netloc ' +79869929087, mak34@gmail.com' contains invalid characters under NFKC normalization

id here is important

e.g. this is very likely to be mapped to normal py docss
file:///usr/share/doc/python3/html/library/contextlib.html

fragments are often random and useless
even default org-mode is guilty

huh, so reddit seems to normalise to the main page, and displays annotations as 'orphaned' for comment views?