GB2458636A - Combining Low Resolution Video to Produce a High Resolution Still Image - Google Patents

Abstract

A camera operable under control of a processing means records a series of low resolution images forming a low resolution video stream, or a high resolution still image. The processing means performs a real-time chroma-key operation on the low resolution video stream and a chroma-key operation on the high resolution still image to produce respective composite video and still images and to output representations of the composite still and/or video streams via the display output. Also disclosed is adjusting the composition of the still image to match that of the combined video stream, and automatically providing a user with a reference ID as part of an authentication process to provide access to the high resolution combined image.

Description

1 2458636 Photographic Terminal and Processor This invention relates to hardware and software which allows high resolution, chroma-key, stills photography to be carried out by untrained operators in environments such as tourist attractions and shopping centre promotions where subjects can be photographed and superimposed onto backgrounds suitable to the promotion Promotions may need the ability to use Chromakey Green screen' style setups where participants have their photo taken against a green or blue background which is then instanUy removed and replaced with a therned background.

Chroma-key or green screen video has been available for many years In principle people or other objects are filmed against a green or blue background, the video is fed into a mixing desk with an embedded processor that removes the green screen in real time and overlays the new background such as used for superimposing a weather forecaster over a weather map during televised weather forecasts In this application, the camera is typically producing 25 or 30 low resolution frames (720x576 pixels) per second. Since television is a low resolution medium, these images are adequate for television broadcast but for photographic purposes they produce only very small images at adequate resolution and cannot be used to produce studio quality prints.

Green screen still photography has also been around for many years In principle a digital camera is used to take a single high resolution photograph of the subject which is then loaded into a software application that removes the green background on the single high resolution frame and superimposes it over a new background for printing An example of this type of arrangement is Express Digital A camera is tethered to the computer, the operator takes a photo against a green screen background that is immediately imported into the application, the green removed and replaced with the

selected foreground and background

US Patent Publication Number 2007/0065 143 describes the production of a chroma-keyed video for distribution on a DVD or over a web interface. The chroma-keyed composite video sequence may be displayed in real-time to the person being videoed. It proposes using a video camera and associated equipment to produce a chroma-keyed video sequence against a moving video background and using a separate stills camera to separately record a single high resolution still version of the subject against the green screen. It also proposes the alternative of a still photograph produced from the composite video sequence.

For stills photography, in practice a Digital Single Reflex Camera (DSLR) is used.

The photographer looks through the camera view finder and composes the subject against the green screen with no physical reference to the background onto which they are later to be superimposed For a professional photographer placement of the subject is achieved using skill and judgment borne out of training and experience, but for an amateur or untrained promotional staff this operation is almost impossible and the results are typically of poor quality This is particularly so if the subject needs to

interact with the background.

Systems to be used in fixed settings such as tourist attractions or promotions maylrely on a fixed camera setup where the operator often cannot see through the view finder. The camera is therefore set at a wide angle, the shot is taken and then subsequently corrected by adjusting the positions of the background and/or newly taken image in a software application.

Some digital compact cameras (Canon in particular) offer an "electronic view finder" function that allows the image which would normally be displayed on screen on the back of the camera to be transmitted by cable to a PC to be rendered in a window on the PC display, hence allowing the camera operator easily to see the composition of a picture.

The viewfinder function sends back a series of low resolution video style frames to the computer that are adequate to use to compose a photograph Once composed, the camera takes a photo at standard (high) resolution and a single high resolution still frame is thus downloaded to the computer for processing in a software application. In this case whilst the operator can see through the view finder" during composition, they still can only see the subject against the green screen with no reference to the background into which they are to be superimposed.

The problem to be overcome is how to make it easy for an untrained operator to take a perfectly composed green screen photograph quickly and to keep the unit cost of a system down without recourse to expensive DSLRs cameras and studio lighting or time spent post-processing the image Hitherto this sort of promotion has not been widely available because of the cost of setup and operation using professional photographers and equipment According to a first aspect of the invention, there is provided image processing apparatus comprising processing means, a display output couplable to a display screen, and a camera, the camera being selectably operable under control of the processing means, to sample a series of low resolution non-interlaced, images forming a low resolution video stream, or a high resolution still image, the processing means being arranged to perform a real-time chroma-key operation on the said low resolution video stream and a chroma-key operation on the high resolution still image to produce respective combined video and still images and to output representations of the combined still and/or video streams via the display output In a second aspect, the invention provides a method of producing a high resolution still image comprising instructing a camera to provide a low resolution, non-interlaced sequence of images samples forming a low resolution video stream, performing a real-time chroma-key operation on the said low resolution video stream, combining the output of the chroma key operation with another image to form a combined video stream, displaying the combined video stream, taking user input to adjust the composition of the combined video stream and/or initiate recordal of a high resolution still image using the camera, which corresponds to the same composition as the low resolution combined video stream In this way, it is possible to provide the seamless integration of low resolution video preview images with high resolution photographic images in a single application for the purpose of producing photographic quality images for printing and web upload.

In a third aspect, the invention provides a method of distributing a high resolution still image comprising instructing a camera to provide a low resolution, non-interlaced sequence of images samples forming a low resolution video stream, performing a real-time chroma-key operation on the said low resolution video stream, combining the output of the chroma key operation with another image to form a combined video stream, displaying the combined video stream, taking user input to adjust the composition of the combined video stream and/or initiate recordal of a high resolution still image using the camera, which after combining with said another image, corresponds to the same composition as the low resolution combined video stream, automatically generating a reference ID, providing the user with the reference ID and providing a user interface which takes the reference ID as at least part of an authentication process and provides access to electronic and/or printed copies of the said high resolution combined image If payment is to be taken, typically it will be taken at the time of producing the reference ID or the copies of the high resolution combined image.

Embodiments of the invention will now be described y way of example and with reference to the drawings in which Figure 1 is a schematic diagram of apparatus in accordance with the invention, Figure 2 is a screen shot showing a normal mode of operation in which a photographer sees a subject against a green screen; Figure 3 shows a screen shot of the prior to a photograph being taken, showing a real time preview against a green screen: Figure 4 is a screen shot showing the resultant shot from Figure 2; and Figure 5 is an example of an output image in accordance with the invention With reference to Figure 1, a camera 2 is coupled to a computer 4 (in this case a personal computer having user input means and a display) The camera 2 is arranged to take a sequence of low resoftition images of a person 6, against background of predetermined colour (not shown) such as a predetermined shade of green or blue.

The images are fed back along a cable 8, to the computer 4. The images are processed by the computer 4 to display a set of images of the type shown in the screenshot of Figure 2 on the computer display With reference to Figure 2, a real-time preview 10 of the image seen by the camera 2, is shown on the dispiay of the computer 4. A set of earlier still images 12 are shown on the left of the screen display for user reference Figure 3 shows a real-time moving preview 10' including chroma-keying in real-time Figure 4 shows a fully rendered and chroma-keyed image image 10" after pressing the shoot button 14 as described below.

An enlarged view of the final high resolution combined image is shown in Figure 5.

In use a user views the computer screen and instructs the person 6 to move until the image is suitable composed by view in the real-time preview 10' Preferably, the screen is touch sensitive and operation may be carried out by touching the screen in the area of the buttons 14 This area includes a button marked shoot ". Pressing this button causes the camera 2 to take a high resolution image which is suitable for enlarging up to normal photographic size This is then instantly processed by chrorna-key (against the selected background colour e.g. green or blue) and adding whatever additional images have been chosen. In this way, composition of the image is greatly simplified and the resulting high-resolution composite image may then be distributed or printed for example by.- * Instant print * Secure web upload * Shown live on a screen * Copied onto a memory stick The printing and other distribution methods listed above may be deferred. The system may provide a user reference such as a 6 digit number which may then subsequently be used as part of a log-in procedure to an online (e.g. Internet) user interface The number may be randomly assigned to improve security Payment may optionally be taken via the interface (not shown) prior to providing the distribution materials.

The above-described architecture has been developed specifically for untrained operators to use in situations such as Tourist attractions and Shopping centre promotions/locations where subjects can be photographed and superimposed into

backgrounds suitable to the promotion.

The challenge faced in developing this application was how to make it easy for an untrained operator to take a perfectly composed green screen photograph in just seconds and to keep the unit cost of a system down without recourse to expensive DSLRs cameras and studio lighting. In particular, how to find a way to integrate a live camera preview giving raw images against a green screen, removing the background in real time to give a live video-style preview using only software and seamlessly integrating this with high resolution stills from the same camera to produce a high resolution image for printing Using a series of low resolution still images and by optimizing the size of the incoming video preview frames and adapting a modified green screen algorithm we are able to remove the green screen background in real time from the preview and overlay it onto the background in exactly the same position as the photo itself So the operator selects the background they want to use, looks at a real time preview of the subject in the frame, they can then move the subject around and change their pose until they are in the correct position, they then press shoot" and take the photo.

The resulting high resolution image is placed in exactly the same place as the preview frame, hence creating a perfectly aligned photograph.

Additionally in the application when each background is setup a place holder is located to show exactly where the image is to be placed, in many cases it is only over a specific area of the background. The viewfinder preview is placed in this position and not over the whole background, ensuring a match between the final image and the preview.

Initially, processing software was entirely implemented in C# This was the technology of preference, because it enabled proper portability of the code around different platforms, and rapid implementation of additional functionality (primarily because of the vast amount of functionality that comes bundled with the Microsoft NET 2.0 Framework) However, this implementation provide inadequate performance as described below and the current implementation therefore combines bespoke libraries coded in C++, that are called from wrapper functions in C# Green Screen As noted briefly above, chroma key is a technique for mixing two images together, in which a colour (or a small colour range) from one image is removed (or made transparent), revealing another image behind it This technique is also referred to as colour keying, colour-separation overlay green screen, and blue screen Matte Separation Mattes are used in photography and special effects filmmaking to combine two or more image elements into a single, final image Usually, mattes are used to combine a foreground image (such as actors on a set, or a spaceship) with a background image (a scenic vista, a field of stars and planets) In this case, the matte is the

background painting.

Whilst the ideal scenario is an evenly lit set with no shadows, this is often not possible in a mobile setup such as a shopping centre Therefore the methodology chosen to remove the green background must be robust enough to cope with shadows on the background Thus a photographic approach is taken rather than the video-based approach in the prior art In this approach, a single frame is taken which is frozen with the use of powerful flash lights brighter than ambient light used by video technology. The resultant image is much clearer and crisper and is suitable for producing high resolution prints from and as such is not susceptible to problems with ambient light.

For example in a shopping centre where the sunlight is shining through the glass ceiling the prior art video solutions will not work unless housed in a blacked out area or at least a controlled lighting area, making it very difficult to implement in any situation, However, the camera with flash systems can be set to levels that are brighter than the sunlight hence eliminating such problems. In practice this is an important part of the solution to produce high quality images In contrast, in video a series of frames is captured every second, typically between to 29 (PAL / NTSC) every second with a shutter speed of approx 1/50th of a second using ambient light (that includes general video lights). This has the effect of capturing a series of frames that blend together to form a motion picture. If a still frame is captured from such a video it will have an inherent blur due to the specific nature of the method of capture. In order to use a frame from an HD video, the video must first be captured, then rendered then the still frame captured / saved to disk It is not saved to the computer as a single frame from the camera as in a still photo.

Captured frames from HD cameras are 1920x1080 pixels, the max print resolution for these is around 6"x4" without interpolating the image to increase its size By creating the matte one can distinguish between foreground and background, and thus remove the green or blue background and add a new one In order for the matte creation to be simpler it is necessary that the background is in a predefined colour (usually green or blue, because these two colours are not part of the human flesh and thus they are easier to distinguish from the foreground) There are several requirements for the screen that need to be fulfilled in order for the chroma keying to be more successful The screen needs to be evenly lit in order to produce relatively the same colour as background, and without any sharp shadows. in order to reduce shadows the person who is being shot should be several feet away from the screen. Because the background colour is removed from the image, it is important that the person is not

wearing clothing the same colour as the background

There are different approaches for keying such as colour keying that removes the background colour of the image to reveal another "behind" it or luma or also called alpha keying, that applies transparency to regions in an image which falls into a particular range of brightness We investigated algorithms like Ultimatte, Primatte, RGBCMYL keyer, Keylight etc that are used in major commercial applications After evaluating these we came to the conclusion that the most suitable starting point for implementation was an algorithm proposed in a paper by F van den Bergh, & V Lalioti -"Software Chroma Keying in an Immersive Virtual Environment" South African Computer Journal, (24):155-162, Nov. 1999. The algorithm was optimised specifically to be used in a real time software environment on a PC not encoded in hardware. It uses very simple calculations, keeping everything in integer arithmetic (no floating points) In contrast, the Primatte algorithm uses a 128-faced polyhedron which is a very complex structure This algorithm is based on the RGB colour space and expects a couple of input parameters (a colour range for example) It evaluates each pixel against the colour of the estimated background and makes all matching pixels transparent Pixels that come on the border of the foreground and the background are coloured with different opacity in order to produce some kind of anti-aliasing and prevent the rough edges that are produced otherwise This algorithm gives good quality results compared to others, since it has 3-5 calculations per pixel and it is not nearly as memory intensive as others are It is less memory intensive because it doesn't need to support any complex structures in memory. As explained below, the algorithm uses a plane which crosses the RGB colour space (this space can be thought of as a cube with a 255 units wide side) and divides it in two subspaces, using a simple alpha function In this wayall that is required to be stored in memory is the image to be processed and a discrete alpha function Preliminary Stages Initial Implementation The first stage of the green screen involved selecting a shade of the green and an allowable range round this shade. Then a comparison of each pixel against the allowable values gave an estimate if it should be treated as transparent or left it as it was.

This approach used some heuristics in the sense of "if a pixel is green then the green component should be the dominant one, i.e. the green pixel's component should be greater than the red and the blue ones'. However, the "allowable values" value was usually subject to number of external factors that accounted for usually poor results.

After the algorithm had been applied it left some green areas visible.

This initial version of the algorithm did in other words treat the image in monochromic way, and was either removing (making the pixel transparent) or leaving the pixel as it is. This additionally led to rough edges and reduced the quality of the final image.

Transition to Hue, Lightness and Saturation (HLS) Colour Space To deal with these problems, a HUE-based representation of the colour palette was used. In this way, the algorithm needed only to specify what shade of the green is to be removed and then specify an angle to the left and to the right of the resulting HLS space. Any pixel whose components meet the constraints of the values in the specified range of the HLS hexagon were to be made transparent, while others were to be left intact.

This once again produced results that were not satisfying enough It still led to rough edges. Another downside was that the maths involved while estimating the ranges consumed too much CPU effort -primarily because the RGB colour space needed to be transferred to HLS colour space As result the green screen removal could not be performed in real time Optimized Euclidian distance approach The article mentioned earlier noted that one may use a plane to cross the RGB colour space and depending on the sub space in which the pixel is -it is evaluated as transparent or left as it is This however led to rough edges as well Therefore, the improvement proposed was to use a special alpha function that takes as an argument the distance of the currently processed pixel to the crossing plane. The "farther" the pixel is from this plane the more transparent it should be. The result of this Improvement was that the rough edges disappeared and a smooth anti-aliasing effect was achieved.

The RGB colour space may be thought of as a cube with, for example, a 255 units wide side. In this system, you may have a plane that crosses this cube, i.e crosses the RGB colour space. Thus 2 half-spaces are created, i e. the cube is divided into two pieces, which when glued together form the original cube. Every image pixel has four components red, green, blue and alpha. Using an 8 bit representation, all of these components have a value in the interval 0-255, i.e every pixel is a point in the RGB cube When you notionally draw the plane that crosses the cube, the pixel will be situated in one of the two pieces, i.e. in one of the two half-spaces of the RGB colour space If this pixel is in one of the pieces where C < B or G < R then we say it is from the foreground, so keep its alpha component as it is. If it is in the other piece make its alpha = 0, i e. make this pixel transparent If it is in the first piece and the G > R and G > B check the distance of the current pixel to the cube's main diagonal and choose an appropriate value for the pixel's alpha component (these pixels are on

the border of the fore and background)

Classical Euclidean distance calculation involves operations like squaring and square rooting which are known to be time consuming Therefore a simpler distance function is used. One that uses summing and subtraction, thus preserving the calculations in the integer arithmetic to provide a real-time processing capability Initial Implementation The initial implementation of the alpha keying algorithm was performed in C#. It was based on the Windows GDI+ functions provided by the Microsoft NET Framework for standard processing. These were benchmarked against set of testing It was shown that operations the algorithm had to perform on these images, were not optimally executed by the.NET based API This was due to the fact that images in NET are not internally kept as a sequence of binary data, but as arrays of Pixel objects Thus meaning that the iteration over each pixel that had to be performed became a complex task, because it involved a number of consequent calculations of indices and almost unpredictable access to the physical memory The conclusion was that basic NET implementation processed each pixel much slower than needed. This is because the image library of NET is not intended for live processing of large amounts of image data.

This caused extreme delays when processing large images. Therefore, the algorithm has been implemented in C++, taking into account the physical representation of an image in a computer's memory, i e a successive array of memory cells.

Implementation Optimizations Using pointers, the C++ implementation enables the algorithm to be implemented in a way that allows a memory pointer to move from the currently processed pixel directly to the next one without having to calculate indices from the beginning of the image This results in a performance gain of more than 20 times, and made possible the processing of a regular 1600x1200 image in less than half a second.

In the original C# prototype version a GDI+ function was used in the following way: Color pixel = bitmap_object GetPixel(int x, mt y), that returns the colour components (red, green, blue and alpha) of the pixel located at (x, y). This function does not know that a whole image is to be processed and so it calculates the pixel to return by starting from the beginning of the image and moving to the requested pixel. To process every pixel of an image (as is required for this system) you get the following: Color pixell = bitmap_object GetPixel(0, 0), Color pixel2 = bitmap_object GetPixel(1, 0), Color pixel3 = bitmap_object.Getpixel(2, 0), etc. These calls do not move the pointer of the bitmap to the adjacent pixel but rather start from the beginning of the bitmap and search for the requested pixel.

Instead, the C++ implementation does the following: Cotor* pixel = GetStartOflmage(bitmap object) Color pixell = *pixel pixel = pixel + 1, Color pixel2 = *pixel pixel pixel + 1, Color pixel3 = *pixel pixel = pixel + 1, etc. Thus the C++ implementation does not lose time searching for the next pixel to process by starting from the beginning of the image but simply increments the current pointer ready for the next pixel.

Another improvement introduced in the C++ Implementation was the use of four pointers (one for each of the four pixel components -red, green, blue and alpha) This way every pixel was processed in a single operation and not in four In order to speed things up furthermore we availed ourselves of the NET compiler option to compile the code for a specific platform (e g x86), thus producing even a higher performance Another major low-level optimization was facilitating the simplified Euclidian distance algorithm mentioned earlier and having it implemented in the C++ version as well.

Applications for the Implementation The algorithm is used to remove a range of green shades (the background of the image) making the corresponding pixels transparent. This way the background image will be displayed in their place and the resultant composed image could well become a piece of the art.

The fast processing of images supported by the algorithm, provides the application with the unique option to process frames received from the camera in real time, i.e camera live view.

Further optimization of live previews The green screen removals in camera preview mode may be further optimised. In order to make the preview display frames at a reasonable speed we had to balance the quality of frames with the CPU power consumed After tests and we ended with a well-balanced approach -the frame received from the camera is decreased in size, then the green screen algorithm iterates over a smaller version of the frame and the processed frame is then resized to the size of the panel it is going to be displayed in.

Although the quality of the frames may not be perfect, it gives the photographer an adequate impression of how things are going to look like when the final photo is

taken and blended with the predefined background

Options A further improvement of the algorithm could be the removal of any colour such as blue shades from images This is possible because of the nature of the algorithm -it makes the specified colour transparent if it is the dominant component of the currently processed pixel Another improvement may be implementing an option to dynamically set the amount of resizing to take place. Depending on what amount of CPU power is consumed the application may dynamically estimate the maximum possible size of the resized image before the green screen separation takes place.

Thus in summary, the above-describe arrangement provides a system for the real-time preview of green screen images from a non-interlaced source such as the preview function of a digital stills camera to provide identical high resolution stills of the same image.

Claims (3)

1. Claims 1. Image processing apparatus comprising processing means, a display output couplable to a display screen, and a camera, the camera being selectably operable under control of the processing means, to sample a series of low resolution non-interlaced, images forming a low resolution video stream, or a high resolution still image, the processing means being arranged to perform a real-time chroma-key operation on the said low resolution video stream and a chroma-key operation on the high resolution still image to produce respective combined video and still images and to output representations of the combined still and/or video streams via the display output.
2. 2 Apparatus according to claim 1, where in the processing means is further arranged to reduce the resolution of each or selected frames of the lower resolution video stream prior to performing the real-time chroma-key operation on the video stream and wherein the processing means is further arranged to perform a second re-sizing operation after the chroma-key operation and before outputting the representation of the video stream.
3. 3. Apparatus according to claim 3 wherein the processing means is arranged to dynamically set the amount of resizirg to take place dependent on the amount of CPU power which is consumed by the chroma-key operation to achieve a balance between quality and speed 4 Apparatus according to any preceding claim wherein during the video chroma-key operation, the processing means is arranged to determine the location of pixels in RGB colour space in relation to a predetermined plane in the ROB colour space and to mark each pixel as transparent or non-transparent dependent which side of the plane each pixel lies 5. Apparatus according to claim 4 wherein the transparency level is set dependent on the distance of the currently processed pixel from the said plane.6. Apparatus according to any preceding claim, wherein the processing means is implemented as a digital computer operable to execute software instructions 7. Apparatus according to claim 6, wherein the software instructions include the use of four pointers respectively for each of the four pixel components -red, green, blue and alpha, to allow each pixel to be processed more quickly 8 A method of producing a high resolution still image comprising instructing a camera to provide a low resolution, non-interlaced sequence of images samples forming a low resolution video stream, performing a real-time chroma-key operation on the said low resolution video stream, combining the output of the chroma key operation with another image to form a combined video stream, displaying the combined video stream, taking user input to adjust the composition of the combined video stream and/or initiate recordal of a high resolution still image using the camera, which corresponds to the same composition as the tow resolution combined video stream 9. A method according to claim 8 further comprising reducing the resolution of each or selected frames of the lower resolution video stream prior to performing the real-time chroma-key operation on the video stream and performing a second re-sizing operation after the chroma-key operation and before displaying the said combined video stream 10 A method according to claim 9, wherein the amount of resizing is automatically adjusted dependent on the amount of CPU power which is consumed by the chroma-key operation in order to achieve a balance between quality and speed.11 A computer program product directly loadable into the internal memory of a digital computer, comprising software code portions for performing the steps of any of claims 8 to 10 when said product is run on a computer 12. Image processing apparatus constructed and arranged as described herein with reference to the drawings 13 A method of distributing a high resolution still image comprising instructing a camera to provide a low resolution, non-interlaced sequence of images samples forming a low resolution video stream, performing a real-time chroma-key operation on the said low resolution video stream, combining the output of the chroma key operation with another image to form a combined video stream, displaying the combined video stream, taking user input to adjust the composition of the combined video stream and/or initiate recordal of a high resolution still image using the camera, which after combining with said another image, corresponds to the same composition as the low resolution combined video stream, automatically generating a reference ID, providing the user with the reference ID and providing a user interface which takes the reference ID as at least part of an authentication process and provides access to electronic and/or printed copies of the said high resolution combined image.14. A method according to claim 13 including providing a printed copy of the reference ID generally at the same time as producing the said high resolution combined image.