⭐ Cachew

• compact
  
• fast
  
• easy iteration
  
• possible to use as data interface by querying the db directly
  

why using separate columns at all

• actually appart from potentially nicer queries I'm not sure. I wonder if more time is spent serializing as opposed to simply using json?
  

https://www.reddit.com/r/Python/comments/kz7vj/is_it_possible_to_use_a_namedtuple_with_sqlalchemy/

sqlalchemy needs to attach attributes onto a mapped class, which include one for each column mapped, as well as an attribute _sa_instance_state for bookkeeping. It also, as someone mentioned, needs to create a weak reference to the object. So a SQLAlchemy-mapped class needs to be a "new style class", extending from object.
If you look at how namedtuple works, it has quite a bit of trickery going on, not the least of which is that it sets __slots__ to (). Then there's some super-nasty frame mutation stuff after that to somehow make it pickleable.
That said it's straightforward to make your own class that is SQLAlchemy compatible and acts more or less the same as a namedtuple. Just define __iter__, __len__, __repr__, other methods as appropriate. Plus it will be pickleable without injecting variables into stack frames.

When you're handed unstructured data (by your network, file system, database...), cattrs helps to convert this data into structured data.
When you have to convert your structured data into data types other libraries can handle, cattrs turns your classes and enumerations into dictionaries, integers and strings.

https://github.com/unmade/apiwrappers

Modern - decode JSON with no effort using dataclasses and type annotations

hmm kind of similar to what I'm doing/want for axol?

• caching diffs
  

  • reasonable perf boost
    
  • relatively easy? just ignore 'emitted'?
    
  • automatically works for changed prefix (bleanser)
    

  ? requires changes to cachew key handling
  

  • might be still slowish
    
• explicitly querying for cached prefix
  
  • best performance
    
  • fairly easy
    
  • almost no changes to cachew
    
  • requires restructuting code in a specific way, mcachew thing might be harder
    
  • gonna be tricky if the prefix can cahnge (bleanser) (although if we can probe for a cached key, it can work?)
    
• single cachew decorator (not sure if possible?)
  
  • best performance
    
  • pretty simple
    
  • still requires 'hack' in the caller for detecting if something was cached or not
    
  • if prefix changes won't work
    
  • requires cachew changes? some sort of global function context? pretty unclear how to implement
    
    • with cached(f) as cf:
      return cf(some_args)
      
    • ok, maybe by default cachew is 'recursive'?
      when we enter the function, we memorize the argument that needs to be cached, but we don't lock the database yet?
      so, let's consider
      @cachew
      def factorials(n: int) -> int:
      last = 1
      for prev in factorials(n - 1):
      yield prev
      yield prev * n
      say, we've run factorials(3) before, the cache has [1, 2, 6]
      factorials(5), cachew memorizes {factorials: 5}, it's the one being computed, goes inside the function
      factorials(4) – not in the cache. so it goes inside and tries evaluating factorials(3)
      factorials(3) – in the cache, cachew opens the db ans starts emitting?
      factorials(4) shouldn't be writing becase it's not the one being computed
      factorials(5) on the other hand should start writing
      kind of a problem however is that it reads and writes at the same time.. I guess that could work with transactions?
      

feels a bit more atomic…

• maybe allow using app specific subcaches?
  e.g. make_cachew(prefix='my')
  
• [2021-03-20] ok done https://github.com/karlicoss/cachew/commit/161ce25304bfda77b484f886ff9c0b4b23924563
  

https://pyfiddle.io/fiddle/4de2f70f-e421-4326-bbb8-b06d5efa547d/?i=true
yeah really need to give a demo

https://twitter.com/pycoders/status/1214956434519154688

perhaps with more processing difference would be even more striking…

$ time with_my python3 -c 'from my.bluemaestro import get_dataframe; print(get_dataframe())'
USING CACHEW!!!
                     temp
dt
2018-07-15 02:57:00  24.3
2018-07-15 02:58:00  24.3
2018-07-15 02:59:00  24.3
2018-07-15 03:00:00  24.3
2018-07-15 03:01:00  24.3
...                   ...
2019-07-27 10:42:00  23.8
2019-07-27 10:43:00  23.8
2019-07-27 10:44:00  23.8
2019-07-27 10:45:00  23.8
2019-07-27 10:46:00  23.8

[549054 rows x 1 columns]
with_my python3 -c   3.32s user 0.36s system 111% cpu 3.296 total

$ time with_my python3 -c 'from my.bluemaestro import get_dataframe; print(get_dataframe())'
                     temp
dt
2018-07-15 02:57:00  24.3
2018-07-15 02:58:00  24.3
2018-07-15 02:59:00  24.3
2018-07-15 03:00:00  24.3
2018-07-15 03:01:00  24.3
...                   ...
2019-07-27 10:42:00  23.8
2019-07-27 10:43:00  23.8
2019-07-27 10:44:00  23.8
2019-07-27 10:45:00  23.8
2019-07-27 10:46:00  23.8

[549054 rows x 1 columns]
with_my python3 -c   16.03s user 0.37s system 102% cpu 16.019 total

During reading cache all that happens is reading rows from sqlite and mapping them onto your target datatype, so the only overhead would be from reading sqlite, which is quite fast.

ugh, grammar is a bit odd

attemps to cause

to call

caching for

globally?

Maybe even dal is fine if I illustrate it by integration test?

add a super simple, trivial example. just with some dictionaries maybe?

from sqlalchemy.interfaces import PoolListener # type: ignore
# TODO ugh. not much faster...
class MyListener(PoolListener):
    def connect(self, dbapi_con, con_record):
        pass
        # eh. doesn't seem to help much..
        # dbapi_con.execute('PRAGMA journal_mode=MEMORY')
        # dbapi_con.execute('PRAGMA synchronous=OFF')
# self.db = sqlalchemy.create_engine(f'sqlite:///{db_path}', listeners=[MyListener()])

test_dbcachemany

de8b67cd0896e0b7512d276a5bb0fc9784ea9a49

100K: about  3.0 seconds
500K: about 15.5 seconds
  1M: about 29.4 seconds

after updating to nice binders

100K: about  3.2 seconds
500K: about 15.6 seconds
  1M: about 31.5 seconds

logger.debug('inserting...')
from sqlalchemy.sql import text # type: ignore
from sqlalchemy.sql import text
nulls = ', '.join("(NULL)" for _ in bound)
st = text("""INSERT INTO 'table' VALUES """ + nulls)
engine.execute(st)
shit. so manual operation is quite a bit faster??
but we still want serialization :(
ok, inserting gives noticeable lag
thiere must be some obvious way to speed this up...
pylint: disable=no-value-for-parameter
logger.debug('inserted...')

~/.cache/my/my.core.core_config:test_cachew.\<locals\>.cf