javascript - Process a continuous stream of JSON -

the (now defunct) page http://stream.twitter.com/1/statuses/sample.json used return continuous , endless stream of json data.

i'd process using jquery (or javascript, preferably jquery) inside own web page able display visual effects based on live feed of tweets.

since far know jquery parsejson function execute callback function after data has been sent server, continuous stream of data. how can process data "as happens" still keep connection running?

update 2

this sort of thing best done using websockets now.

it looks this:

var connection = new websocket('ws://html5rocks.websocket.org/echo', ['soap', 'xmpp']); 

attaching event handlers connection allows know when connection opened, when you've received incoming messages, or if error has occurred.

sending messages becomes easy this:

connection.send('your message'); connection.send(binarydata); 

see introducing websockets: bringing sockets web full explanation on how this.

you may want consult browser support websockets.

asp.net developers: if reason need support older browsers , don't want figure out how deal don't support websockets, consider using library such signalr.

update 1

most browsers implement eventsource api, makes long polling easy, long stream can delivered content-type text/event-stream. older browsers or developers reason can't engineer stream have content-type can use helper script same thing.

here's example:

var jsonstream = new eventsource('https://example.com/yourstreamingservice') jsonstream.onmessage = function (e) {    var message = json.parse(e.data);   // handle message }; 

this full-fledged version of exact thing outline below.

the old service streaming answer

what want called long polling. you'll need custom ajax onreadystatechange handling function. instead of waiting until entire stream has completed (since never will), you'll need examine contents periodically. note you'll need heavy lifting work in ie, using iframe.

roughly:

• respond each onreadystatechange event , examine stream you've been given current character see if there enough data consume 1 or more discrete events. you'll need parse stream javascript string-handling functions. combination of split, indexof, regular expressions, looping, , on can used accomplish task.
• if there's not enough content yet, exit , wait next event.
• i pretty sure each time onreadystatechange handler fires, responsetext data has been received far. define persistent variable hold position of first character hasn't been processed yet.
• once there enough content 1 or more discrete events appear in stream, take them out 1 @ time , pass them json parser render text objects. use them normally.

check out http streaming @ ajax patterns discussion of exact topic (it covers service streaming you're doing). stated, if must support ie, you'll need use iframe method that.

in summary, service streaming makes http streaming approach more flexible, because can stream arbitrary content rather javascript commands, , because can control connection's lifecycle. however, combines 2 technologies aren't consistent across browsers, predictable portability issues. experiments suggest page streaming technique work on both ie , firefox , service streaming works on firefox, whether xmlhttprequest or iframe used. in first case ie suppresses response until complete, iframe works if workaround used: ie accepts message server after first 256 bytes thing send 256 dummy bytes before sending messages. after messages arrive expected. full service streaming possible in ie, too!

security issues

normal ajax cannot go cross-domain, meaning (now pay attention fact want stream twitter) won't able you're asking. can worked around jsonp, jsonp nature can't service streamed , isn't offered twitter anyway. there cross-origin resource sharing (cors) twitter's not going set you--that's kind of thing they'd domains affiliated them. , cors requires modern browser.

your option create proxy service on web server performs requests twitter , hands out data. can done same domain main page served from. doing allow create version work ie using iframe technique. if don't care old ie versions, can implement cors defeat domain restriction, if know domain making requests.

if have full control of client software (like if corporate intranet) there option: hosting web browser inside of compiled locally-executed application's user form. have done using c# imagine it's possible other languages. when use right browser object, because it's hosted inside c# application, c# application can defeat cross-domain security restrictions, reading , writing page content no matter domain comes from. doubt situation 1 wanted put option here others might appreciate it.