php - Atomicity in Map/Reducing over new records (MongoDB) -


here's situation: i've got mongodb cluster , web-app, pretty intensive map/reduce query. query happens periodically (every 5min) in cron job, , results stored (using $merge) collection.

what works: currently, query performs on every record in collection. said collection growing millions of rows, , each time runs, takes little longer.

the obvious solution run map/reduce on new records, , use reduce function on old stored values calculate correct value. mongodb great, lets specify reduce option instead of merge that.

what can't figure out: how correctly perform m/r on new records in initial collection. see 2 potential solutions, neither of good. ideas?

  1. i flag records have been processed. problem how flag same records m/r'd over?
  2. i query matching items, pass list of ids $in: [id1, id2, ...] query map/reduce, , send update set flag using same $in. that's inelegant, , don't know how that's going perform when list of records huge.

tl;dr: how select new records in map/reduce query reduces result collection.

a kind soul on #mongodb irc channel helped me figure 1 out. simple solution have state machine field, , following (in pseudo-code):

set {state:'processing'} {state:{$exists:false}} mapreduce {...} {state:'processing'} set {state:'done'} {state:'processing'}

now, suboptimal because wastes lot of disk space on collection millions of records. real question is, why did not think of sooner?


Comments

Post a Comment

Popular posts from this blog

c# - SharpSVN - How to get the previous revision? -

i trying find efficient way previous revision of file text comparison using sharpsvn. using (svnclient c = new svnclient()) { c.authentication.defaultcredentials = new networkcredential( configurationmanager.appsettings.get("svnserviceusername") , configurationmanager.appsettings.get("svnservicepassword") , configurationmanager.appsettings.get("svnservicedomain") ); c.authentication.sslservertrusthandlers += new eventhandler<svnsslservertrusteventargs>(authentication_sslservertrusthandlers); collection<svnfileversioneventargs> fileversioncollection = new collection<svnfileversioneventargs>(); svnrevisionrange range = new svnrevisionrange(0, this.hooks.revision); svnfileversionsargs args = new svnfileversionsargs(); args.retrieveproperties = true; args.range = range; foreach (svnchangeitem item in log.changedpaths) { string path = this.repositorypath + it...
Read more

c++ - Is it possible to compile a VST on linux? -

for class project i'm attempting write vst plugin backed cuda. current cuda workflow on linux box, i'd prefer compile , link there. according wikipedia , should possible (i couldn't find steinberg documentation relevant linux) can't find makefile or instructions on how build if aren't using xcode or visual studio. i'm vst 3 sdk doesn't support linux. when try compile plugin under linux, error: ./base/source/fatomic.cpp:39:30: fatal error: libkern/osatomic.h: no such file or directory this issue caused following code in "vst3 sdk/base/source/fatomic.cpp" #if mac #include <libkern/osatomic.h> #if mac_os_x_version_min_required > mac_os_x_version_10_4 #define native_atomic_type (volatile int32_t*) #else #define native_atomic_type (int32_t*) #endif #elif windows #include <windows.h> #endif but hope compiling under linux work vst sdk 2.4. reading. jvstwrapper seems run on lin...
Read more

url - Querystring manipulation of email Address in PHP -

i using third party application (email2http) send me emailid, body , message. unable retrieve email id due name appended on , enclosed < , >. here querystring getting. test.php?from=ajay+reddy &subject=cx&body=somebody when print variable in php returns name , not email id. $from = $_get['from']; echo $from; //output ajay reddy please suggest me how can emailid also. i dont want use post method due testing , debugging. it's working fine. forgot use htmlentities() in debugging.
Read more