iphone - iOS CVImageBuffer distorted from AVCaptureSessionDataOutput with AVCaptureSessionPresetPhoto -

at high level, created app lets user point or iphone camera around , see video frames have been processed visual effects. additionally, user can tap button take freeze-frame of current preview high-resolution photo saved in iphone library.

to this, app follows procedure:

1) create avcapturesession

capturesession = [[avcapturesession alloc] init]; [capturesession setsessionpreset:avcapturesessionpreset640x480]; 

2) hook avcapturedeviceinput using back-facing camera.

videoinput = [[[avcapturedeviceinput alloc] initwithdevice:backfacingcamera error:&error] autorelease]; [capturesession addinput:videoinput]; 

3) hook avcapturestillimageoutput session able capture still frames @ photo resolution.

stilloutput = [[avcapturestillimageoutput alloc] init]; [stilloutput setoutputsettings:[nsdictionary     dictionarywithobject:[nsnumber numberwithint:kcvpixelformattype_32bgra]     forkey:(id)kcvpixelbufferpixelformattypekey]]; [capturesession addoutput:stilloutput]; 

4) hook avcapturevideodataoutput session able capture individual video frames (cvimagebuffers) @ lower resolution

videooutput = [[avcapturevideodataoutput alloc] init]; [videooutput setvideosettings:[nsdictionary dictionarywithobject:[nsnumber numberwithint:kcvpixelformattype_32bgra] forkey:(id)kcvpixelbufferpixelformattypekey]]; [videooutput setsamplebufferdelegate:self queue:dispatch_get_main_queue()]; [capturesession addoutput:videooutput]; 

5) video frames captured, delegate's method called each new frame cvimagebuffer:

- (void)captureoutput:(avcaptureoutput *)captureoutput didoutputsamplebuffer:(cmsamplebufferref)samplebuffer fromconnection:(avcaptureconnection *)connection {     cvimagebufferref pixelbuffer = cmsamplebuffergetimagebuffer(samplebuffer);     [self.delegate processnewcameraframe:pixelbuffer]; } 

6) delegate processes/draws them:

- (void)processnewcameraframe:(cvimagebufferref)cameraframe {     cvpixelbufferlockbaseaddress(cameraframe, 0);     int bufferheight = cvpixelbuffergetheight(cameraframe);     int bufferwidth = cvpixelbuffergetwidth(cameraframe);      glclear(gl_color_buffer_bit);      glgentextures(1, &videoframetexture_);     glbindtexture(gl_texture_2d, videoframetexture_);     gltexparameteri(gl_texture_2d, gl_texture_min_filter, gl_linear);     gltexparameteri(gl_texture_2d, gl_texture_mag_filter, gl_linear);     gltexparameteri(gl_texture_2d, gl_texture_wrap_s, gl_clamp_to_edge);     gltexparameteri(gl_texture_2d, gl_texture_wrap_t, gl_clamp_to_edge);      glteximage2d(gl_texture_2d, 0, gl_rgba, bufferwidth, bufferheight, 0, gl_bgra, gl_unsigned_byte, cvpixelbuffergetbaseaddress(cameraframe));      glbindbuffer(gl_array_buffer, [self vertexbuffer]);     glbindbuffer(gl_element_array_buffer, [self indexbuffer]);      gldrawelements(gl_triangle_strip, 4, gl_unsigned_short, buffer_offset(0));      glbindbuffer(gl_array_buffer, 0);     glbindbuffer(gl_element_array_buffer, 0);     [[self context] presentrenderbuffer:gl_renderbuffer];      gldeletetextures(1, &videoframetexture_);      cvpixelbufferunlockbaseaddress(cameraframe, 0); } 

this works , leads correct results. can see video preview of 640x480 processed through opengl. looks this:

however, if capture still image session, resolution 640x480. want high resolution, in step 1 change preset line to:

[capturesession setsessionpreset:avcapturesessionpresetphoto]; 

this correctly captures still images @ highest resolution iphone4 (2592x1936).

however, video preview (as received delegate in steps 5 , 6) looks this:

i've confirmed every other preset (high, medium, low, 640x480, , 1280x720) previews intended. however, photo preset seems send buffer data in different format.

i've confirmed data being sent buffer @ photo preset valid image data taking buffer , creating uiimage out of instead of sending opengl:

cgcolorspaceref colorspace = cgcolorspacecreatedevicergb(); cgcontextref context = cgbitmapcontextcreate(cvpixelbuffergetbaseaddress(cameraframe), bufferwidth, bufferheight, 8, bytesperrow, colorspace, kcgbitmapbyteorder32little | kcgimagealphapremultipliedfirst);  cgimageref cgimage = cgbitmapcontextcreateimage(context);  uiimage *animage = [uiimage imagewithcgimage:cgimage]; 

this shows undistorted video frame.

i've done bunch of searching , can't seem fix it. hunch it's data format issue. is, believe buffer being set correctly, format line doesn't understand:

glteximage2d(gl_texture_2d, 0, gl_rgba, bufferwidth, bufferheight, 0, gl_bgra, gl_unsigned_byte, cvpixelbuffergetbaseaddress(cameraframe)); 

my hunch changing external format gl_bgra else help, doesn't... , through various means looks buffer in gl_bgra.

does know what's going on here? or have tips on how might go debugging why happening? (what's super weird happens on iphone4 not on iphone 3gs ... both running ios4.3)

this doozy.

as lio ben-kereth pointed out, padding 48 can see debugger

(gdb) po pixelbuffer <cvpixelbuffer 0x2934d0 width=852 height=640 bytesperrow=3456 pixelformat=bgra # => 3456 - 852 * 4 = 48 

opengl can compensate this, opengl es cannot (more info here opengl subtexturing)

so here how i'm doing in opengl es:

(cvimagebufferref)pixelbuffer   // pixelbuffer containing raw image data passed in  /* ... */ glactivetexture(gl_texture0); glbindtexture(gl_texture_2d, videoframetexture_);  int framewidth = cvpixelbuffergetwidth(pixelbuffer); int frameheight = cvpixelbuffergetheight(pixelbuffer);  size_t bytesperrow, extrabytes;  bytesperrow = cvpixelbuffergetbytesperrow(pixelbuffer); extrabytes = bytesperrow - framewidth*4;  glubyte *pixelbufferaddr = cvpixelbuffergetbaseaddress(pixelbuffer);  if ( [[capturesession sessionpreset] isequaltostring:@"avcapturesessionpresetphoto"] ) {      glteximage2d( gl_texture_2d, 0, gl_rgba, framewidth, frameheight, 0, gl_bgra, gl_unsigned_byte, null );      for( int h = 0; h < frameheight; h++ )     {         glubyte *row = pixelbufferaddr + h * (framewidth * 4 + extrabytes);         gltexsubimage2d( gl_texture_2d, 0, 0, h, framewidth, 1, gl_bgra, gl_unsigned_byte, row );     } } else {     glteximage2d(gl_texture_2d, 0, gl_rgba, framewidth, frameheight, 0, gl_bgra, gl_unsigned_byte, pixelbufferaddr); } 

before, using avcapturesessionpresetmedium , getting 30fps. in avcapturesessionpresetphoto i'm getting 16fps on iphone 4. looping sub-texture not seem affect frame rate.

i'm using iphone 4 on ios 5.