python - Threading HTTP requests (with proxies) -


i've looked @ similar questions, there seems whole lot of disagreement on best way handle threading http.

what want do: i'm using python 2.7, , want try , thread http requests (specifically, posting something), socks5 proxy each. code have works, rather slow since it's waiting each request (to proxy server, web server) finish before starting another. each thread making different request different socks proxy.

so far i've purely been using urllib2. looked modules pycurl, extremely difficult install python 2.7 on windows, want support , coding on. i'd willing use other module though.

i've looked @ these questions in particular:

python urllib2.urlopen() slow, need better way read several urls

python - example of urllib2 asynchronous / threaded request using https

many of examples received downvotes , arguing. assuming commenters correct, making client asynchronous framework twisted sounds fastest thing use. however, googled ferociously, , not provide sort of support socks5 proxies. i'm using socksipy module, , try like:

socks.setdefaultproxy(socks.proxy_type_socks5, ip, port) socks.wrapmodule(twisted.web.client)

i have no idea if work though, , don't know if twisted want use. go threading module , work current urllib2 code, if going slower twisted, may not want bother. have insight?

perhaps easier way rely on gevent (or eventlet) let open lots of connections server. these libs monkeypatch urllib make async, whilst still letting write code sync-ish. smaller overhead vs threads means can spawn lots more (1000s not unusual).

ive used loads (plagiarized here):

urls = ['http://www.google.com', 'http://www.yandex.ru', 'http://www.python.org']  import gevent gevent import monkey  # patches stdlib (including socket , ssl modules) cooperate other greenlets monkey.patch_all()  import urllib2   def print_head(url):     print ('starting %s' % url)     data = urllib2.urlopen(url).read()     print ('%s: %s bytes: %r' % (url, len(data), data[:50]))  jobs = [gevent.spawn(print_head, url) url in urls]

Comments

Post a Comment

Popular posts from this blog

c# - SharpSVN - How to get the previous revision? -

i trying find efficient way previous revision of file text comparison using sharpsvn. using (svnclient c = new svnclient()) { c.authentication.defaultcredentials = new networkcredential( configurationmanager.appsettings.get("svnserviceusername") , configurationmanager.appsettings.get("svnservicepassword") , configurationmanager.appsettings.get("svnservicedomain") ); c.authentication.sslservertrusthandlers += new eventhandler<svnsslservertrusteventargs>(authentication_sslservertrusthandlers); collection<svnfileversioneventargs> fileversioncollection = new collection<svnfileversioneventargs>(); svnrevisionrange range = new svnrevisionrange(0, this.hooks.revision); svnfileversionsargs args = new svnfileversionsargs(); args.retrieveproperties = true; args.range = range; foreach (svnchangeitem item in log.changedpaths) { string path = this.repositorypath + it...
Read more

c++ - Is it possible to compile a VST on linux? -

for class project i'm attempting write vst plugin backed cuda. current cuda workflow on linux box, i'd prefer compile , link there. according wikipedia , should possible (i couldn't find steinberg documentation relevant linux) can't find makefile or instructions on how build if aren't using xcode or visual studio. i'm vst 3 sdk doesn't support linux. when try compile plugin under linux, error: ./base/source/fatomic.cpp:39:30: fatal error: libkern/osatomic.h: no such file or directory this issue caused following code in "vst3 sdk/base/source/fatomic.cpp" #if mac #include <libkern/osatomic.h> #if mac_os_x_version_min_required > mac_os_x_version_10_4 #define native_atomic_type (volatile int32_t*) #else #define native_atomic_type (int32_t*) #endif #elif windows #include <windows.h> #endif but hope compiling under linux work vst sdk 2.4. reading. jvstwrapper seems run on lin...
Read more

url - Querystring manipulation of email Address in PHP -

i using third party application (email2http) send me emailid, body , message. unable retrieve email id due name appended on , enclosed < , >. here querystring getting. test.php?from=ajay+reddy &subject=cx&body=somebody when print variable in php returns name , not email id. $from = $_get['from']; echo $from; //output ajay reddy please suggest me how can emailid also. i dont want use post method due testing , debugging. it's working fine. forgot use htmlentities() in debugging.
Read more