UIImage and IplImage

In Cocoa Touch, UIImage is the class we use for images, however, in OpenCV, the corresponding class is IplImage. So we need to convert between the two classes:

We can create IplImage from UIImage with this function in Objective-C:


// NOTE you SHOULD cvReleaseImage() for the return value when end of the code.
- (IplImage *)CreateIplImageFromUIImage:(UIImage *)image {
  // Getting CGImage from UIImage
  CGImageRef imageRef = image.CGImage;


  CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
  // Creating temporal IplImage for drawing
  IplImage *iplimage = cvCreateImage(
    cvSize(image.size.width,image.size.height), IPL_DEPTH_8U, 4
  );
  // Creating CGContext for temporal IplImage
  CGContextRef contextRef = CGBitmapContextCreate(
    iplimage->imageData, iplimage->width, iplimage->height,
    iplimage->depth, iplimage->widthStep,
    colorSpace, kCGImageAlphaPremultipliedLast|kCGBitmapByteOrderDefault
  );
  // Drawing CGImage to CGContext
  CGContextDrawImage(
    contextRef,
    CGRectMake(0, 0, image.size.width, image.size.height),
    imageRef
  );
  CGContextRelease(contextRef);
  CGColorSpaceRelease(colorSpace);


  // Creating result IplImage
  IplImage *ret = cvCreateImage(cvGetSize(iplimage), IPL_DEPTH_8U, 3);
  cvCvtColor(iplimage, ret, CV_RGBA2BGR);
  cvReleaseImage(&iplimage);


  return ret;
}


And vice versa, we can create UIImage from IplImage in Objective-C:

// NOTE You should convert color mode as RGB before passing to this function
- (UIImage *)UIImageFromIplImage:(IplImage *)image {
  CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
  // Allocating the buffer for CGImage
  NSData *data =
    [NSData dataWithBytes:image->imageData length:image->imageSize];
  CGDataProviderRef provider =
    CGDataProviderCreateWithCFData((CFDataRef)data);
  // Creating CGImage from chunk of IplImage
  CGImageRef imageRef = CGImageCreate(
    image->width, image->height,
    image->depth, image->depth * image->nChannels, image->widthStep,
    colorSpace, kCGImageAlphaNone|kCGBitmapByteOrderDefault,
    provider, NULL, false, kCGRenderingIntentDefault
  );
  // Getting UIImage from CGImage
  UIImage *ret = [UIImage imageWithCGImage:imageRef];
  CGImageRelease(imageRef);
  CGDataProviderRelease(provider);
  CGColorSpaceRelease(colorSpace);
  return ret;
}

Using OpenCV on the iPhone

In my project, I'm doing a lot of image processing on the GPU, however, we still need OpenCV to do a lot of stuff. Here I'm writing down my experience of using OpenCV on the iPhone:

Building OpenCV 2.2 with Xcode 4 and iOS SDK 4.3
OpenCV 2.2 started to support Android SDK, but not iPhone. So we have to build it on our own. As iOS SDK doesn't support dynamic ".framework", the solution is to build it as a static ".a" lib file, and link it against the app statically

1. Install CMake. I used Homebrew as a package manager. I highly recommend it. As compared to Macports, it's much cleaner and more elegant.

2. Download OpenCV 2.2 from http://sourceforge.net/projects/opencvlibrary/files/opencv-unix/2.2/OpenCV-2.2.0.tar.bz2/download

3. xjvf OpenCV-2.2.0.tar.bz2

4.
% cd OpenCV-2.2.0
% patch -p1 < ../OpenCV-2.2.0.patch

5.

% cd .. # Back to the top of demo project directory.
% mkdir build_simulator
% cd build_simulator
% ../opencv_cmake.sh Simulator ../OpenCV-2.2.0
% make -j 4
% make install

6.

% cd .. # Back to the top of demo project directory.
% mkdir build_device
% cd build_device
% ../opencv_cmake.sh Device ../OpenCV-2.2.0
% make -j 4
% make install

Using OpenCV library in the project
Actually it is more complicated than I thought. I googled for a long time and finally got it working.

Add libopencv_core.a etc, from OpenCV lib directory for either simulators or devices. Actually Xcode doesn’t care which one is for devices or simulators at this point because it is selected by the library search path.
Add Accelerate.framework which is used internally from OpenCV library.
Select your active build target, then open the build tab in the info panel by Get Info menu.
Add -lstdc++ and -lz to Other Linker Flags
Add path to OpenCV include directory to Header Search Paths for both simulators and devices.
Add path to OpenCV lib directory to Library Search Paths for both simulators and devices.

Ok. For now we're happy to use OpenCV on the iPhone!

OpenGL ES Presentation

OpenGL ES Presentation

View more presentations from Eric Cheng
Demo 1: My Rotating Cube based on Xcode OpenGL ES 2.0 template

Demo 2: Bumpy from iPhone 3D Programming

Final Project alpha presentation

Final Project Alpha presentation

View more presentations from Eric Cheng

What this blog is about and final project pitch

Dear visitors,

This semester I'm taking the CIS565 course in Penn taught by Joe Kider. Joe is a wonderful tutor and he gave us various options to choose for the final project. And we're required to write blogs to update our development progress. And that's why this blog is born.

I thought about the final project for several weeks. I've been strongly interested in iOS development for 2 years, so I have the idea of combining iOS dev with GPU. Right now, the graphics processor in iOS devices is getting better and better for generations. In iPhone 4, Apple used a PowerVR SGX 535 GPU embedded in their A4 chip, which had decent graphics performance. This article compared iPhone 4 with the popular game handhelds, and iPhone 4 is obviously outstanding. Here's a chart below from Wikipedia describing the technical details of the iPhone GPU:

Model	Year	Die Size (mm2)[1]	Config core[2]	Fillrate (@ 200 MHz)		Bus width (bit)	API (version)		GFLOPS(@ 200 MHz)
				MTriangles/s[1]	MPixel/s[1]		DirectX	OpenGL
SGX520	Jul 2005	2.6@65 nm	1/1	7	250	64	N/A	N/A	0.8?
SGX530	Jul 2005	7.2@90 nm	2/1	14	500?	64	N/A	2.0	1.6
SGX531	Oct 2008	?	2/1	14?	500?	128	N/A	N/A	1.6?
SGX535	Nov 2007	?	2/2	14	1000?	64	9.0c	2.1	1.6
SGX540	Nov 2007	?	4/2	28	1000	64	N/A	2.1	3.2
SGX545	Jan 2010	12.5@65 nm	4/2	40	1000	64	10.1	3.2	3.2?

We can learn from above, though mobile GPUs might not have stunning GFLOPS as desktop GPUs, they're already capable of handling a lot of things. Since 2007, the Khronos Group introduced OpenGL ES 2.0, which had a major upgrade over OpenGL ES 1.1. OpenGL ES 2.0 eliminates most of the fixed function rendering pipeline in favor of a programmable one. Almost all rendering features of the transform and lighting pipelines, such as the specification of materials and light parameters formerly specified by the fixed-function API, are replaced by shaders written by the graphics programmer. So this opened up all the possibility for the programmers to leverage the power of the iPhone GPU.

Right now, there're tons of apps on the iOS platform. Last summer, in my internship in SAP US Newtown Square, I was doing iOS development under Martin Lang. Martin always showed some cool apps to fellow colleagues, and two of them are very impressive in my opinion. One is called Layar, which shows the names and details information of the building around you, in your camera scene

The other app is called WordLens. This app detects text information from the camera in real time and translate it, which is really amazing

However, none of these apps are open source, and they're not using GPU to do the calculation. So here my idea comes around. I wanted to do an Augmented Reality app on the iOS, and leveraging the GPU at the same time.

Apple had done such thing in their latest iOS 4.3. When apple introduced iPad 2, they brought Photo Booth from Mac to iOS.

Photo Booth uses Core Image API to process the video frames, which have GLSL underneath. So, GPU powered video processing is doable on iOS.

About what I want to implement specifically, I read this paper from CVPR, and the algorithm introduced inside is cool and efficient. Therefore I want to do it on the iOS, which use the GPU to process the video frames, and fetching texts from within the frames. Ideally if I can speak out in iOS with a TTS(Text-to Speech) engine, that would be perfect, but right now I have no clue whether that's possible.

So my plan for the final project is listed below:

1. A simple app on the iPhone just to capture the video, and get the video frames.

2. Implement the algorithms from the paper, get the texts within frames

3. Speed up Step 2 with GLSL/Core Image to get better frame rates, ideally real time performance

And I think my project might be very useful in the future, like bind people can use their iPhones to read information from public places, and it's fun to play with for a normal person

That's it. Also suggested by the final project write-up, I use the twitter account nlyrics2 for updating news. Wish myself good-luck in future development.