Dron: my job scheduler

Motivating blog post: https://beepb00p.xyz/scheduler.html

Github project: https://github.com/karlicoss/dron

Table of Contents

related droninfracronsystemd

[A] * motivation dron

mostly in the blog post, some extra links there too

DONE [B] [2020-02-01] Rethinking Cron dron

[2020-02-29] good wishlist for missing cron features dron

STRT [C] hmm it's kind of nice that in cron you can embed snippets of code… a bit harder in systemd dron

CREATED: [2020-04-13]

[B] * improvements dron

STRT [D] [2019-08-25] maybe forbid creating pycache? dron

TODO [A] I think I need to enable/start regardless changes; just in case dron

CREATED: [2020-03-08]

TODO [B] managed view – would be nice to display timers only; and a command on a keypress. like cron dron

CREATED: [2020-03-16]

TODO [B] conflicting jobs, e.g. promnesia generator and db backer dron

CREATED: [2018-05-31]

maybe there should be multiple tags? if two jobs got same tags, they can't run at the same time
implemented via flocks on files named same as tags

TODO [C] create separate target (instead of timers.target)? that way it'd be less messy dron

CREATED: [2020-03-08]

TODO [C] jcu got lots of potentially useful stuff.. dron

CREATED: [2020-01-24]
-t --identifier=STRING     Show entries with the specified syslog identifier
 -p --priority=RANGE        Show entries with the specified priority

TODO [C] Wonder if there is a way to overview all systems job failures over past boot? dronsystemd

CREATED: [2020-01-25]

TODO [C] mode to confirm systemd files diff first? dron

CREATED: [2020-11-02]

TODO [C] for running now – show time it's been running for dron

CREATED: [2020-05-29]

TODO [D] show desktop notification on failure dron

CREATED: [2020-02-27]

[B] * bugs dron

STRT [B] shit, monitor is consuming quite a bit of CPU. should run less often I guess.. wonder if possible to only run when I look at it? dron

CREATED: [2020-06-12]
  • [2021-01-25] ugh. did some work, but it seems that it's just bus communication that still takes cpu and it's sort of inevitable?

TODO [C] next time should be local instead of utc… dron

CREATED: [2020-06-14]

TODO [D] might fail if manually disabled the timer? dron

CREATED: [2020-11-09]
Traceback (most recent call last):
  File "/home/karlicos/.local/bin/dron", line 11, in <module>
    load_entry_point('dron', 'console_scripts', 'dron')()
  File "/code/dron/dron.py", line 1168, in main
  File "/code/dron/dron.py", line 966, in cmd_monitor
    _cmd_monitor(managed, params=params)
  File "/code/dron/dron.py", line 849, in _cmd_monitor
    [service, timer] = gr
ValueError: not enough values to unpack (expected 2, got 1)

TODO [B] [2020-05-29] I guess what I was confused about was that if the job is running simultaneouslyu to previous instance, it's silently not running dron

[2020-05-29] wonder if could email on this! dron

[C] * publicity dronpublish

TODO [C] lobsters maybe and elsewhere? when it gets a bit more mature dronpublish

CREATED: [2020-01-26]

STRT [C] post about it? really need a proper circleci test… dronpublish

CREATED: [2020-01-25]

[B] * readme/post amendemens? drontoblog

[2020-03-18] eh, I'm finding that copy paste in crontabs is not too bad as long as you align everything… drontoblog

[D] [2020-04-08] it's easier to align in python, because of string concatenation drontoblog


export_jobs = [
     job(at('02:08'), arctee(backups / 'feedbin'   /   'feedbin_{utcnow}.json', '--', 'python3.7', soft / 'backup/misc/feedbin.py'            ), unit_name='export-feedbin'),
     job(at('01:07'), arctee(backups / 'myshows'   /   'myshows_{utcnow}.json', '--', 'python3.7', soft / 'backups/myshows/myshows_backup.py' ), unit_name='export-myshows'),
     job(at('01:08'), arctee(backups / 'goodreads' / 'goodreads_{utcnow}.json', '--', 'python3.7', soft / 'backups/myshows/myshdows_backup.py'), unit_name='export-myshows'),


0     0  * * *          $K -s "backup-goodreads"       --low            -- arctee      '/backups/goodreads/goodreads_{utcnow}.xml'           -- /soft/backup/goodrexport/run
01   01  * * *          $K -s 'backup-pinboard'        --low -c         -- arctee      '/backups/pinboard/bookmarks_{utcnow}.json'           -- /soft/backup/pinbexport/run
05   01  * * *          $K -s 'backup-lastfm'          --low -c         -- arctee      '/backups/lastfm/lastfm_{utcnow}.json'                -- python3.7 /soft/backup/lastfm/lastfm_backup.py
07   01  * * *          $K -s "backup-spotify"         --low -c         -- arctee      '/backups/spotify/spotify_{utcnow}.json'              -- /soft/backup/spotify/backup

TODO [D] motivation: crontabs are hard to keep under version control. if you got different on different machines, hard to reconcile drontoblog

CREATED: [2020-04-08]

TODO [D] why drontoblog

CREATED: [2020-01-25]
  • running custom jobs is easier
  • you can just start any jobs (+autocomplete)

TODO [B] simple frontend for Systemd, drontoblog

CREATED: [2020-05-13]

Not frontend, generator?

STRT [D] [2020-01-18] watchdog? dronsystemd


    Configures the watchdog timeout for a service. The watchdog is activated when the start-up is completed. The service must call sd_notify(3) regularly with "WATCHDOG=1" (i.e. the "keep-alive ping")

CNCL [D] [2020-02-01] schedule python library dron

nope, just a completely custom scheduler, definitely not ideal

--------------------------------------- dron

TODO [B] a perspective a year later? dron

CREATED: [2021-03-13]

For dron, thinking about it a year later, a large chunk of the convenience in this dron tool would probably be from

  • allowing multiple services/timers in the same file (and then splitting them up in the actual systemd files)
  • allowing a different format (something more yaml-like) to make it possible to align jobs vertically. Then it's even fine not to have a general purpose language – much easier to keep copypasta if you can block edit it

TODO [B] hmm, managed status drops after reinstalling crontabs… I guess, timers? dron

CREATED: [2020-03-16]

TODO [C] if there is an invalid systemd file, it fails to load state hence you have to fix manually dron

CREATED: [2020-05-28]

and impossible to fix. e.g. add some garbage to onCalendar and try applying

TODO [B] [2019-10-24] to enable the systemd service you can run the following. dronsystemd

systemctl --user enable --now matrixcli

STRT [B] [2021-02-24] linux - Proper way to use OnFailure in systemd - Unix & Linux Stack Exchange dron

In the code you run at ExecStopPost=, you can use one of $SERVICE_RESULT, $EXIT_CODE or $EXIT_STATUS to determine the failure condition and act accordingly. See the documentation on those environment variables to check which one is appropriate for you.

hmm can run this on success

[C] [2020-01-28] arch linux - systemd: cpu usage of services - Super User dron

If you enabled the cpuacct cgroup subgroup in the kernel. You can try systemd-cgtop to identify which systemd service causes high cpu usage.

[C] [2020-01-28] systemd - systemctl status not showing CPU/Memory usage? - Ask Ubuntu dron

CPUAccounting = yes
MemoryAccounting = yes

[C] [2020-01-26] Systemd: How to make two services mutually exclusive but run both? - Super User dronsystemd

[C] [2020-01-26] systemd.service https://www.freedesktop.org/software/systemd/man/systemd.service.html#TimeoutStartSec= dron

wonder if this is useful wrt to dependencies and conflicts

[C] [2020-01-28] systemd-analyze https://www.freedesktop.org/software/systemd/man/systemd-analyze.html dron

systemd-analyze verify FILE...
The following errors are currently detected:
    unknown sections and directives,
    missing dependencies which are required to start the given unit,
    man pages listed in Documentation= which are not found in the system,
    commands listed in ExecStart= and similar which are not found in the system or not executable.

STRT [C] using fzf to start service dron

CREATED: [2020-11-25]
systemctl --user start $(ls ~/.config/systemd/user/ | fzf)

[D] [2020-01-28] systemd-cgtop dron

systemd-cgtop — Show top control groups by their resource usage

wonder if I could use it?

TODO [D] [2020-05-04] systemd, 10 years later: a historical and technical retrospective dron

Yes, we currently handle socket-triggered, bus-triggered, file-triggered, mount-triggered, automount-triggered, device-triggered

[2020-05-29] ok, so timeouts via RuntimeMaxSec work as expected dronsystemd

TODO [C] dron: one year later drontoblog

CREATED: [2020-11-30]

what worked:

  • systemd integration is excellent, can hack on systemd to experiment, and then reapply changes properly via dron
  • while mailing is sort of hacky, don't think it has ever let me down

what didn't work

  • barely used dron edit, mainly dron apply

what to improve:

  • speed up dron apply?
  • monitor too slow?

TODO bug: try stoping an active timer with systemctl explicitly; it will be displayed as some weird big number (should be 'never') dron

CREATED: [2021-02-08]

TODO [C] feature: for 'next' if the date is today, hide it dron

CREATED: [2021-02-08]

TODO [B] [2021-04-18] [QUESTION] Why is `systemd-email` so complex ? · Issue 16 · karlicoss/dron dron

Hi ! I'm reading your sources with great interest, and i came out to systemd-email, and it looks like to me that the same output can be achieved with this single command: systemctl --user status myFailingUnit.service --lines 99999 -o cat
Is it really equivalent ?

TODO [C] systemd-email – need full path to make sure it won't fail to deliver email.. dron

CREATED: [2021-05-04]
Jump to search, settings & sitemap