python - Structure for large volume of semi-persistent data? -

i need track large volume of inotify messages set of files that, during lifetime, move between several specific directories, inodes intact; need track movement of these inodes, create/delete , changes file's content. there many hundreds of changes per second.

because of limited resources, cant store in ram (or disk, or database).

luckily, of these files deleted in short order; file content- , movement-history need stored later analysis. files not deleted end staying in particular directory known period of time.

so seems me need data structure partially stored in ram, , partially saved disk; part of portion saved disk need recalled (the files not deleted), not. not need query data, access identifier (the file name, [a-z0-9]{8}). helpful able configure when file data flushed disk.

does such beast exist?

edit: i've asked related question.

why not database? sqlite.

while sqlite isn't efficient storage mechanism in terms of space there number of advantages -- first , foremost is sql rdbms. amount of memory sqlite uses (to temporarily cache data) can configured through cache_size pragma.

if sqlite isn't option, 1 of "key value stores"? range distributed client/server in-memory (e.g. memcached) local embedded disk-based (e.g bdb) memory-with-a-persistent-backing-for-overflow , anywhere in-between, etc. not have sql ddl/dql (although might allow relationships), efficient @ -- store keys , values.

of course, 1 implement lru structure (say basic sorted list limit) overflow simple extensible disk-based hash implementation... but... consider above first :) [there may micro-kv libraries/source out there].

happy coding.