KR20070008687A - Creating an output image - Google Patents

Abstract

A method of creating an output image (108) on basis of a sequence of temporally consecutive input images is disclosed. The method comprises: identifying a particular part of a particular object (100) in a first one of the input images (102); fetching a first group of pixels (110) from the first one of the input images (102), the first group of pixels (110) corresponding to the particular part of the particular object (100); localizing the particular part of the particular object (100) in a second one of the input images (104); fetching a second group of pixels (110) from the second one of the input images (104), the second group of pixels (110) corresponding to the particular part of the particular object (100); and appending the second group of pixels (110) to the first group of pixels (110) to form the output image. ® KIPO & WIPO 2007

Description

CREATING AN OUTPUT IMAGE}

The present invention relates to a method for generating an output image based on a sequence of input images that are temporally continuous.

The invention further relates to a computer program product loaded by a computer device comprising instructions for generating an output image based on a sequence of temporally successive input images.

The invention further relates to an image processing apparatus arranged to generate an output image based on a sequence of input images that are temporally contiguous.

The advantage of showing a sequence of input images in time is that dynamic events can be visualized, such as the movement of an object relative to the background. For example, a sports game such as soccer may be shown in which the actual movement of the ball is involved. It is a common feature to repeat a portion of an image sequence corresponding to a soccer game during a broadcast relay. Typically these parts correspond to the most interesting moments of the game. However, when it is desired to exemplify such an interesting moment, for example in newspapers or some other kind of print media, much of the attraction to the event will be lost. This is because the images in the newspaper do not represent the movement of the event.

It is an object of the present invention to provide a method of the kind described at the outset for summarizing dynamic events in an output image.

This object of the invention is that the method,

Identifying a specific portion of the particular purpose in the first image of the input image;

Obtaining from the first image of the input image a first group of pixels corresponding to a particular portion of the particular object;

Localizing the particular portion of the particular object in a second of the input images;

Obtaining from the second of the input images a second group of pixels corresponding to a particular portion of a particular object; And

-Attaching a second group of pixels to a first group of pixels to form the output image.

An obvious existing approach to illustrating dynamic events is to create a schematic drawing, for example an artificial graphical representation. The method according to the invention differs in that a sequence of temporally successive images, ie, spatial-temporal data, is used to produce a spatial image at a different time in time, including a static image, i.e. an input image.

The portions are selected from the dynamic (x, y, t) input images and combined to form a single still (x, y) output image. This is done so that the static output image illustrates dynamic events, such as the movement of an object, for example. The feature is that the output image contains image data of a particular portion of a particular object sampled at different moments in time. In other words, certain parts of a particular object appear multiple times in the output image. This is because the second pixel group is typically added to the first pixel group adjacent to the second pixel group.

It is distinct from the prior-art, which creates one large panoramic picture by concatenating other smaller images. In such a case, spatial image data sets are used to generate a larger output image. In this method according to the prior art, parts of successive images are otherwise combined. Typically the pixels of each of the spatially overlapping image areas are merged. The result is that each object appears only once in the output image.

In the method according to the invention, data representative of single objects at different moments in time is explicitly used.

In one embodiment of the method according to the invention, the appending comprises a weighted summation of the values of respective pixels of the first pixel group and the second pixel group. The advantage of the weight sum is that the transition in luminance and / or color proceeds smoothly from the first pixel group to the second pixel group. Alternatively, the second pixel group lies adjacent to the first pixel group. Typically, a combination of locating pixel groups and using weighted sums for transitions is used. Thus parts of the two images are selected and combined through some form of interpolation either by weighted average or by simply placing the parts adjacent to each other.

In one embodiment according to the method according to the invention, the first pixel group corresponds to the pixels of many column pixels of the first image of the input images. In this embodiment of the method according to the invention, the first pixel group, and also consecutive pixel groups, extend over the full height of the pixel matrix corresponding to the input image. This means that all pixels located in a column containing pixels representing a particular portion of a particular object are selected and used as a kind of slice to construct the output image. In other words, the output image contains a set of slices taken from successive input images. Each of the slices represents a particular portion of a particular object in separate input images. Typically, the slices also represent a background, in which certain objects move in front of the background. This embodiment according to the invention has the advantage of generating an output image illustrating the horizontal movement of the object.

In an embodiment of the method according to the invention, the pixels of the first group correspond to the pixels of the large number of row pixels of the first image of the input images. In this embodiment of this method according to the invention the first pixel group and also the continuous pixel groups extend over the full width of the pixel matrix corresponding to the input image. That means that all the pixels located in the row containing the pixels representing a particular part of a particular object are selected and used as a kind of slice to compose the output image. In other words, the output image includes a set of slices taken from consecutive input images. Each of the slices represents a particular portion of a particular object in a separate input image. Typically, the slices also represent a background, with a particular object moving in front of the background. This embodiment according to the invention has the advantage of generating an output image illustrating the vertical movement of the object.

In an embodiment of the method according to the invention, the pixels of the first group correspond to the pixels of the plurality of column pixels of the first image of the input images. The number of columns of pixels is based on tracking of a particular object. The movement of a particular object is estimated. The estimated motion determines the dimension of the pixels of the first groups. For example, if the estimated motion of a particular portion of the particular object estimated is equal to 20 pixels, the number of columns of pixels is also 20.

In an embodiment of the method according to the invention, the first pixel groups correspond to the pixels of a large number of rows of pixels of the first image of the input images, the number of rows of pixels being based on the tracking of the measurement object. The movement of a particular object is estimated. The estimated motion determines the dimension of the first pixel group. For example, if the estimated motion of a particular portion of a particular object is equal to 20 pixels, then the number of pixel rows is also 20.

In one embodiment according to the present invention, tracking is based on evaluation of a large number of motion vector candidates, which includes establishing a minimum match error. This technique is commonly known as motion estimation. Preferably, the match error corresponds to a difference between respective pixel values corresponding to a particular object in the first image of the input images and / or the second image of the input images.

The movement is relatively positive. The motion can be represented with respect to the pixel matrix of successive input images. If successive input images are obtained following a fixed position camera, such an approach is appropriate. It can be used directly to calculate the motion of the object, the coordinates of the particular part of the particular object in the first image of the input images and the coordinates of the measuring part of the particular object in the second image of the input images. In many cases, however, the camera pans and / or zooms while capturing moving objects. If a sequence of temporally successive input images is based on such capture, a modification to this camera movement is preferred. In a preferred embodiment according to the invention the number of pixel columns is based on the tracking motion of the background in the first image of the input images and / or the second image of the input images. Alternatively, the number of pixel rows is based on the tracking movement of the background in the first image of the input images and / or the second image of the input images. In the general compensation according to the background, the motion model is realized. This may be a so-called pan-zoom model. It models the background model with a combination of translation and scaling, but it is also more complex and can also deal with other aspects such as perspective projection and rotations. .

As mentioned above, the number of columns / rows taken is based on the movement. This movement is relative to the background where the object is moving in the foreground. In the case of fixed position cameras these movements correspond to movements relative to various pixel metrics.

As an alternative to tracking a specific object by motion estimation based on the estimation of the motion vectors, the specific object may also be tracked semi-manually. In that case, the number of columns of pixels is

Determining first pixel coordinates based on identifying a particular portion of a particular object in the first of the input images;

Determining a second pixel coordinate based on identifying a particular portion of a particular object in a third of the input images;

Determining the number of consecutive input images located in time between the first image of the input images and the third of the input images; And

And determining the number of columns based on the first pixel coordinates, the second pixel coordinates, and the number of consecutive input images.

In this embodiment according to the invention, the user must indicate where a particular part of a particular object is located in a large number of images. This can be done by moving the cursor with respect to the displayed input images.

It is a further object of the present invention to provide a computer program product of the kind described at the outset for summarizing dynamic events in the output image.

It is an object of the present invention that after a computer program product is loaded in a computer device comprising processing means and a memory,

Receiving a position of a particular part of the particular object in the first of the input images;

Taking, from one of the input images, a first group of pixels corresponding to a particular portion of a particular purpose;

Localizing the measurement portion of the particular object in a second of the input images;

Taking a second pixel group from a second one of the input images, the second pixel group corresponding to a particular portion of a particular object; And

It can be achieved in that it provides the processing means with the ability to perform the step of adding a second pixel group to the first pixel group to form an output image.

It is a further object of the present invention to provide an image processing apparatus of the kind described at the outset for summarizing dynamic events in the output image.

The object of the present invention is to provide an image processing apparatus,

Receiving a position of a particular portion of a particular object in the first of the input images;

Taking from the first of the input images a first group of pixels corresponding to a particular part of the particular object;

Localizing a particular part of a particular object in a second of the input images;

Taking from the second of the input images a second group of pixels corresponding to a particular part of the particular object; And

It can be achieved in that it comprises processing means having the ability to perform the step of adding a second pixel group to the first pixel group to form an output image.

Changes and variations in the method may correspond to variations and variations in the image processing apparatus and computer program product to be described.

These and other aspects of the image processing apparatus, the method and the image processing apparatus of the computer program product according to the present invention will become apparent from the embodiments and embodiments described hereinafter and with reference to the accompanying drawings.

1 is a schematic diagram illustrating a method according to the invention wherein the camera is stationary while capturing an input image.

2A is a schematic diagram illustrating a method according to the invention wherein the camera pans while capturing an input image.

2B is a schematic diagram illustrating multiple output images in accordance with the present invention.

3 is a schematic diagram showing a plurality of input images of a soccer game and an output image generated according to the present invention based on these input images.

4 is a schematic diagram showing a second embodiment of an image processing apparatus according to the present invention;

5 is a schematic diagram showing a second embodiment of an image processing apparatus according to the present invention;

Like reference numerals are used to designate like parts throughout all the figures.

1 schematically shows a method according to the invention, wherein the camera is stationary while capturing input images 102, 104, 106. Input images 102, 104, 106 represent an object, ie a ball 100, moving in front of a uniform background. The camera does not move while capturing input images 102, 104, 106. It can be clearly seen that the ball 100 moves from left to right with respect to the pixel matrix corresponding to the input images 102, 104, 106. The output image 108 based on the input images 102, 104, 106 includes a plurality of slices 110, 112, 114 of each input image 102, 104, 106. Slice refers to a set of pixels corresponding to multiple columns (or rows) of the input image. Arrows in FIG. 1 depict the relationship between slices taken from input images 102, 104, 106 and slices that are combined to form output image 108. The size of these slices is based on the movement of the ball 100 with respect to the pixel matrix. The output image 108 also includes a start portion 116 of the first input image 102 and an end portion 118 of the last input image 106. The size of the starting portion 116 and the ending portion 118 is not related to the movement of the ball 100.

2a schematically illustrates the method according to the invention, wherein the camera pans while capturing the input image. Input images 102, 104, 106 represent an object, a ball 100 moving in front of a house. The camera pans while capturing the input images 102, 104, 106. The direction of camera and ball movement is the same. The speed of the camera movement is faster than the speed of the ball 100. The output image 208 based on the input images 102, 104, 106 includes a plurality of slices 110, 112, 114 of each input image 102, 104, 106. The arrows in FIG. 2A depict the relationship between slices taken from input images 102, 104, 106 and combined slices from output image 208. The size of these slices is based on the movement of the ball 100 with respect to the background. The output image 208 also includes a start portion 116 of the first input image 102 and an end portion 118 of the final input image 106. The size of start 116 and end 118 are not related to the movement of ball 100. By comparing the output image 208 with the input images 102, 104, 106, it becomes apparent that the output image is even larger. The output image 208 shows a complete house where the other input images show part of the house. That means that the method according to the invention also selectively combines spatially related image data, resulting in a relatively large output image.

It will be apparent that whenever a new slice of the input image is added to the output image as configured so far, a new output image is created. In other words, the first output image added to the slice becomes the second output image. Showing such a series of output images under construction gives the user the impression of a lively dynamic event combined with the history of the elapsed portion of the event. The user sees a series of output images that differ in size, ie, the subsequent output image is larger than their preceding image.

Alternatively, a relatively large full image is first constructed based on a sequence of input images, which represents the entire scene captured by the input image. However, there is no duplicate described above. This is preferably done using strips of pixels that do not contain pixels representing moving objects. Typically these strips are located at the boundaries of the input images. The size of these strips is not related to the movement of the particular object to be tracked, but to the movement of the background relative to the camera. After generating such a large overall image the method according to the invention is applied. The intermediate result of the method, ie the subsequent output images, is combined into the whole image. Basically this means that subsequent output images are added to each part, ie the remaining parts of the whole image. 2B schematically illustrates a number of output images 202, 204, 208 constructed according to this approach. The first image of the output image 202 shows the entire image where the ball 100 is shown only once. The ball 100 can be seen twice in the second image of the output image 204, and the ball 100 can be seen three times in the third image of the output image 208.

FIG. 3 schematically illustrates a number of input images 102, 104, 106 of a soccer game and an output image 308 generated in accordance with the present invention based on these input images 102, 104, 106. It should be noted that the illustrated input images 102, 104, 106 are part of a long sequence of consecutive input images. These input images 102, 104, 106 represent a soccer game. It can be seen that the player kicks the ball 100 in the first image of the input image 102. See the circle. It can be seen in the second image of the input image 104 that the ball 100 flies towards the sky. See the original again. It can be seen that the ball 100 reaches the goal within the third image of the input image 106. 3 also shows an output image 308 based on the shown input images 102, 104, 106 and approximately 40 input images not shown. The actual trajectory of the ball is clearly seen in the output image 308.

4 shows a first embodiment of an image processing apparatus 100 according to the present invention. The image processing apparatus 400 is arranged to receive a sequence of input images at its image input connector 410 and to provide a sequence of intermediate output images and a final output image at its image output connector 414. Preferably, the image processing apparatus according to the present invention is provided by location information provided by a user dialogue, for example by a user instructing an object of interest in a plurality of input images. Image processing apparatus 100,

Receiving by the location information input interface 412 the location of a particular part of the particular object in the first image of the input image;

Taking, by the pixel processor 404 from the first one of the input images temporarily stored in the input memory device 402, a first group of pixels corresponding to a particular portion of the particular object;

Localizing by localization unit 408 a particular portion of a particular object in a second image of the input image;

Obtaining, by the pixel processor 404 from the second image of the input images temporarily stored in the input memory device 402 after the first one of the input images, a second group of pixels also corresponding to a particular portion of the particular object. step; And

Processing means having the ability to perform the step of adding a second pixel group to the first pixel group to form an output image. The pixel processor 404 is arranged to copy the accessed second group pixel value and to write a copy of the pixel value at the appropriate location in the output memory device 406.

5 schematically shows a second embodiment of an image processing apparatus 500 according to the present invention. This embodiment 500 is basically the same as the embodiment 400 described with reference to FIG. 4. The difference is that this embodiment 500 is arranged to compensate for camera movement. This embodiment of the image processing apparatus is arranged to perform motion estimation of the background to compensate for the effects of camera movement. This embodiment 5400 includes an additional memory device for temporal storage of the second input image. The localization unit 408 is provided with the target of interest, i.e., location information for the particular object to be tracked within the sequence of input images. In addition, the localization unit 408 is arranged to calculate a global motion vector against the background in which the target is moving in front. The global motion vector is calculated by combining a number of motion vectors to be calculated based on a pair of input images. The motion vector is calculated by a standard motion estimator which is preferably integrated into localization unit 408. Motion estimators are described, for example, by G. de Haan et al., In the October 1993 IEEE Bulletin on Circuits and Systems for Video Technology (pages 368-379, Volume 3, No. 5), "True-Motion Estimation with 3-D Recursive Search Block Matching ".

Alternatively, J.S.Kim and R.H., Vol. 25, pages 29-30 of Electronics Letters. As disclosed in the paper by Park, "feature-based block matching algorithm integral projections", motion vectors for the entire image are calculated based on the average image-row (x-component) and the average image-column (y-component). .

The pixel processor 404 and localization unit 408 may be implemented using one processor. Normally, these functions are performed under a software program product. During execution, normally the software program product is loaded into memory such as RAM and executed from there. The program may be loaded from a ROM, hard disk, or background memory such as magnetic and / or optical storage, or over a network such as the Internet. Optionally, application specific integrated circuits (ASICs) provide the functionality disclosed.

Operation of an embodiment of an image processing apparatus as depicted in FIG. 5 will be described using an example comprising an input image sequence representing a free kick in soccer. Some input images, ie video frames, are shown in FIG. 3. The camera will pan to the goal at the position of the kick at a constant speed. The dynamic event to be captured in the output image is the ball flying towards the goal, so the ball must be tracked in the sequence of the input image.

The motion of the ball can be approximated by using a constant velocity in the x direction (which exists along the left-right axis in the input image). This is a reasonable estimate of the ball's movement between the kick and the first subsequent contact with an object such as a goal net. In this example, the movement in the y direction is ignored (top-bottom axis in the input image). For the x position of the soccer ball, the following equation is derived.

n ₀ is the reference input image number, where x _screen , the x position of the ball on the pixel matrix, and the x _camera relative position (x _camera ) are considered known. The actual position of the ball is given by the sum of the screen position and the camera position.

For example, if the ball moves to the right in the "real" world, it is possible that the camera moves to the right faster than the ball moves, in which case the ball appears to move left on the screen. To compensate for these effects, the camera position is included in equation (1).

If the second screen position in the input image n ₁ is known, the actual speed v can be calculated using the following equation.

In such an embodiment, the user may not only provide the start and end points of the event, but also be able to determine the velocity, two or more space-temporal locations with respect to the input image n _i {X _screen (n _i }} Is required.

Using a global motion estimation algorithm, the relative camera position {x _camera (n)} for each input image n is automatically calculated from the video sequence. V then slices containing multiple columns of input images in the region of interest, i.e., screen coordinates, that are horizontal for the event and for each input image n, can be computed by using x _screen ( n) Center around.

These regions of interest, or slices, are copied into the appropriate portion of the output image.

The embodiments presented herein are limited to certain aspects that can be overcome with more advanced processing techniques. In particular, it relies on the user dialogue to provide start and end input images as well as start and end positions of the "interesting object". This can be more generalized (“ball tracking”) using intelligent automatic selection of start and end frames for motion estimation (object-based) and events.

The method, computer program product and image processing apparatus according to the present invention are beneficial for some of the following applications:

Professional image processing, such as in film studios, broadcast studios or newspapers and other types of print media;

Household appliances devices such as TVs, three-top boxes and personal video recording devices;

Educational purposes; And

For example, home video processing software for producing home video.

It should be noted that the above embodiments illustrate rather than limit the invention, and that those skilled in the art can design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The term "comprising" does not exclude the presence of elements or steps not listed in a claim. Singular elements do not exclude plural elements. The invention can be implemented by means of hardware comprising several individual elements and by means of a suitable programmed computer. In the device claim enumerating several means, several of these means may be implemented following one and the same hardware item. The use of terms such as first, second and third does not indicate any order. These words will be interpreted as names.

The present invention relates to a method for generating an output image based on a sequence of input images temporally continuous, and to a computer device comprising instructions for generating the output image based on a sequence of input images temporally continuous. It is available for computer program products and image processing devices that are loaded by.

Claims (14)

A method for generating an output image 108 based on a sequence of temporally successive input images, Identifying a particular portion of the particular object 100 in the first image of the input image 102; Fetching, from the first image of the input image 102, a first group of pixels 110 corresponding to a particular portion of the particular object 100; Localizing a particular portion of a particular object 100 in a second image of the input image 104; Obtaining from the second image of the input image 104 a second group of pixels 110 corresponding to a particular portion of the particular object 100; And Adding a second pixel group 110 to the first pixel group 110 to form an output image. And a method for generating an output image. 2. The method of claim 1, wherein the further step comprises weighting the respective pixel values of the first group pixel (110) and the second pixel group (110). The method of claim 1, wherein the first group of pixels (110) corresponds to pixels of a plurality of pixel columns of the first image of the input image (102). The method of claim 1, wherein the first pixel group (110) corresponds to pixels of a plurality of rows of pixels of the first image of the input image (102). The method of claim 3, wherein the number of pixel columns is based on tracking of the particular object (100). 5. A method according to claim 4, wherein the number of pixel rows is based on the tracking of the particular object (100). 7. The method of claim 5 or 6, wherein the tracking is based on evaluation of a plurality of motion vector candidates, the evaluation comprising establishing a minimum match error. The method of claim 7, wherein the match error is based on a difference between respective pixel values corresponding to the particular object 100 in the first image of the input image 102 and / or the second image of the input image 104. Correspondingly, a method for generating an output image. The output image according to claim 5, wherein the number of pixel columns is based on tracking of the movement of the background in the first image of the input image 102 and / or the second image of the input image 104. Method for generating. The output image of claim 6, wherein the number of pixel rows is based on tracking the movement of the background in the first image of input image 102 and / or the second image of input image 104. Way. The method of claim 5, wherein the number of pixel columns is: Determining coordinates of a first pixel based on identifying a particular portion of a particular object (100) in the first image of the input image (102); Determining second pixel coordinates based on identifying a particular portion of a particular object (100) in the third image of the input image; Determining the number of consecutive input images located in time between the first image of the input image 102 and the third image of the input image; And A method of producing an output image calculated by calculating the number of columns based on the first pixel coordinate, the second pixel coordinate, the number of consecutive input images. A computer program product loaded by a computer device, Instructions for generating an output image 108 based on a sequence of temporally successive input images, wherein the computer device comprises processing means and a memory, the computer program product, after being loaded, the processing means on: Receive a position of a particular portion of a particular object (100) in the first image of the input image (102); -From the first image of the input image 102, a first group of pixels corresponding to a particular part of a particular object 100; Localize a particular portion of a particular object 100 in the second image of the input image 104; From the second image of the input image 104, a second group of pixels 110 corresponding to a particular portion of the particular object 100; Adding a second pixel group 110 to the first pixel group 110 to form the output image. A computer program product that provides the ability to perform. An image processing apparatus arranged to produce an output image 108 based on a sequence of temporally continuous input images, Receive a position of a particular portion of a particular object (100) in a first image of the input image (102); Obtaining a first group of pixels (110) corresponding to a particular portion of a particular object (100) from a first image of the input image (102); Localizing a particular portion of a particular object 100 in the second image of the input image 104; From the second image of the input image 104, a second group of pixels 110 corresponding to a particular portion of a particular object 100; Adding a second pixel group 110 to the first pixel group 110 to form the output image. And processing means having the ability to perform. 15. The image processing apparatus of claim 13, further comprising a display device for displaying the output image.

Images