GB2330266A

GB2330266A - Generating camera image data for an image processing system

Info

Publication number: GB2330266A
Application number: GB9721592A
Authority: GB
Inventors: Kenneth Philip Appleby
Original assignee: Harlequin Group PLC; Harlequin Group Ltd
Current assignee: Harlequin Group PLC; Harlequin Group Ltd
Priority date: 1997-10-10
Filing date: 1997-10-10
Publication date: 1999-04-14
Also published as: TW390091B; AU9634798A; WO1999019837A1; GB9721592D0

Abstract

To produce camera data α, #, #, A and Á (as defined in figure 3) for a sequence of images two images i and j are selected 500, from the sequence of images. A portion or sample, P, of pixels from the first image i of the two images is selected 502. The initial estimates of the camera image data parameters are set 504 as α i , # i , #', A and Á. The initial parameters are used to establish a first estimate of the matrix T (figure 4). The pixels of the sample P are transformed, using the current estimate of matrix T into the second image to produce a second set of sample pixels P'. The images i and j are compared 508,510 to determine the degree of similarity or alignment between them by determining the difference using the function #C where #C=(#)#((#r)<SP>2</SP>+(#g)<SP>2</SP>+(#b)<SP>2</SP>) for all N in which #r, #g, #b represent the differences between the colour components of the two images. A determination is made 512 as to whether a minimum of #C has been reached and if not the estimates of α i , # i , #', A and Á are altered in accordance with a Simplex method. Once a minimum has been reached the values of α, #, #, A and Á are stored. This process is then repeated for each pair of images in the sequence and an iterative process used on the processed frames to find the values of A and Á. These are then used in the above described process to find the values of α, # and #.

Description

Image processing system, method and computer program product The present invention relates to an image data processing system, method, and computer program product.

There exists a significant quantity of recorded image data such as, for example, video, television, films, CDROM images and computer generated images.

Typically once the images have been recorded or generated and then recorded, it is very time consuming and very difficult to give effect to any changes to the images.

It is often the case that a recorded sequence of images may contain artefacts which were not intended to be included within the sequence of images.

Alternatively, artefacts may be added to a sequence of images. For example, it may be desirable to add special effects to the images. The removal or addition of such artefacts is referred to generically as postproduction processing and involves a significant degree of skill and time.

Producing a mosaic or modifying images which have already been recorded is a very difficult task. The degree of effort required to modify or mosaic images or a sequence of images can be reduced if information is available which relates to at least one of the orientation or the position of a camera at the time when the images were produced. Deriving image camera data from a sequence of images is disclosed in, for example, "Creating Full View Panoramic Image Mosaics and Environment Maps", Szeliski R., Shum, HY, Computer Graphics Proceedings, Annual Conference Series, pp 251258, 1997. However, the processing required by the techniques disclosed within the above paper are very complex and require a significant degree of numerically intensive processing in order to solve equations containing eight unknowns.

Often, special effects may be added to an image or other edits may be made either digitally or manually.

The placement of such special effects typically relies upon the judgment of the artist or operator giving effect to the edits. Clearly, there is scope for some error in the placement of the special effects as they are made to a clean plate.

It is therefore an object of the present invention to at least mitigate some of the problems of the prior art.

Accordingly the present invention provides a method for determining camera image data for first and second images of a sequence of images within an image processing system comprising memory for storing said first and second images, said method comprising the steps of storing in said memory the first and second images; and deriving, as a consequence of a comparison between the first and second images, camera image data for the first and second images.

Advantageously, the camera image data for each image within a sequence thereof can be determined. The determined camera image data can then be used in order to give effect to further image processing such as, for example, producing a mosaic of selected images.

A second aspect of the present invention provides an image processing system for determining camera image data for first and second images of a sequence of images, the system comprising memory for storing said first and second images, means for storing in said memory the first and second images; and means for deriving, as a consequence of a comparison between the first and second images, camera image data for the first and second images.

It will be appreciated that the means referred to above and in the claims can be preferably realised using a computer such as that schematically depicted in figure 1 and suitable program code.

Accordingly, a third aspect of the present invention provides a computer program product for determining camera image data for first and second images of a sequence of images within an image processing system comprising memory for storing said first and second images, the product comprising computer program code means for storing in said memory the first and second images; and computer program code means for deriving, as a consequence of a comparison between the first and second images, camera image data for the first and second images.

Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings in which: figure 1 illustrates a computer suitable for implementing an image processing system or method according to the embodiments of the present invention; figure 2 depicts a sequence of source images from which at least one generated image can be produced; figure 3 shows a world co-ordinate system having an image i positioned therein according to corresponding image camera data ai, ssi, Oi and ki; figure 4 illustrates several equations utilised by the present invention; figure 5 there is shown a flow chart for deriving camera image data, ai ,ssi ,0 , A(=ln(ki)) and p; figure 6 illustrates a further equation utilised by the present invention.

Referring to figure 1 there is shown a computer 100 suitable for implementing embodiments of the present invention. The computer 100 comprises at least one microprocessor 102, for executing computer instructions to process data, a memory 104, such as ROM and RAM, accessible by the microprocessor via a system bus 106. Mass storage devices 108 are also accessible via the system bus 106 and are used to store, off-line, data used by and instructions for execution by the microprocessor 102. Information is output and displayed to a user via a display device 110 which typically comprises a VDU together with an appropriate graphics controller. Data is input to the computer using at least one of either of the keyboard 112 and associated controller and the mass storage devices 108.

Referring to Figure 2, there is illustrated schematically several images 1 to N of a sequence of images 200. The images may be derived from a film, video tape, CD-ROM, video disc or other mass storage medium. Typically, the image data varies as between images as a consequence of a combination of motion of object within an image and as a consequence of, for example, variations in the position or orientation of a camera during image capture. Typically, only once the images are recorded. Therefore, the camera orientation information is lost.

Referring to figure 3 there is shown schematically a world co-ordinate system 300 comprising three axes 302, 306 and 308 which correspond to the x, y, and z axes respectively. The position of one image, image j, with reference to another image, image i, can be described in terms of the orientation of a unit vector, u, within the world co-ordinate system in conjunction with a degree of rotation, Oi, about that vector. The orientation of the unit vector within the world coordinate system 300 can be specified using two angles, ai and ssi. The combination of ai, ssi and #i define a rotation which maps image i onto image j. The distance Xj represents the distance from the origin to the centre of the image i. Similarly, the distance #j represent the distance from the origin to the centre of image j.

Conceptually, the camera is positioned such that the optical centre thereof is as close as possible to the origin. The line of sight 308 of the camera is normal to the centre of the image i, which represents in reality the field of view of the camera at a focal length determined by ki- It will be appreciated that the camera image data ai, ssi, Oi, and ki is not available when retrieving images from some mass storage medium such as, for example, a video disc or CD-ROM or the like.

The relationship between a point Q=(x,y,z), in terms of world co-ordinates, within an image frame or field of view at a first orientation of the camera and the same point Q=(x',y',z') within the field of view at a second orientation of the camera is given by the equations shown in figure 4 where ux, uy and uz are the components of the unit vector u, 6 is the degree of rotation about the unit vector, Qr=(xr,y,zr) represents the rotation of the point Q about the unit vector u and x', y' and z' represent the location, in terms of world co-ordinates, of the projection of point Or onto the second field of view.

Referring to figure 5 there is shown a flow chart for deriving camera image data, ai ,ssi ,6 and A(=ln(ki)), describing the orientation within a world co-ordinate system of a camera which captured each of the images in the sequence of images shown in figure 2, wherein 0 decouples o and # using the equation shown in figure 6.

Essentially, the determination of the above parameters utilises a function which compares the colour differences between pixels of any two given images. The first image of the two images is addressed using conventional pixel row and address information.

The second image of the two images is addressed using co-ordinates which have been transformed using the matrix T as shown in figure 4. The second image is preferably addressed using sub-pixels so that colour interpolation can be performed. The function which is used to measure the colour difference and hence derive optimal values for ai ,ssi ,O' and A is as follows: AC=(1/N) r)2+(6g)2+(6b)2) for all N where 5r = ri - rjl 5g = gi - gj, 6b = bi - bj represent the differences between the RGB colour components any given pixels i and j, and N is the number of pixels for which the transformed coordinates fall inside the image frame boundary of the second image.

The minimisation process preferably uses the simplex method as is well known within the art (see, for example, "Numerical Recipes in C", The Art of Scientific Computing, second edition, Press by Teukolsky, Vetterling, Flannery, CUP, IUSBN 0 521 431085, the entire content of which is incorporated herein by reference).

Referring more particularly to figure 5, two images i and j are selected at step 500 from a sequence of images 200. A portion or sample, P, of pixels from the first image i of the two images is selected at step 502. The sample P preferably comprises 2000 pixels which are expressed in terms of pixel rows and columns.

The initial estimates of the camera image data parameters are set at step 504. Preferably, at least three of the parameters ai ,ssi ,O,A and p should be initialised. The initialised parameters are used to establish a first estimate of the matrix T illustrated in figure 4.

At step 506, the pixels of the sample P are transformed, using the current estimate of matrix T, into the second image to produce a second set of sample pixels P . The two images i and j are compared at steps 508 and 510 to determine the degree of similarity or alignment therebetween. Preferably, effect is given to the comparison by extracting image data, such as, for example, rgb colour data, from the pixels at sample points P in image i and at sample points P in image j and determining the differences between the colour data. Still more preferably, the differences are evaluated using the function given above for calculating AC.

A determination is made at step 512 as to whether or not a minimum value of AC has been reached. If not, processing continues at step 514 in which the estimates or current values of cri ,ssi ,O,A and > are altered in After the above iteration of the processing shown in the flow chart, each image pair has a set of camera image data governing the orientation, focal length and lens distortion of the camera at the instant the pair of images were captured.

For each pair of images a (3x3) relative rotation matrix, Rij, is calculated, using the estimated values of a, ss, 6, and #. A relative rotation matrix governs the relative rotation between the two images. The orientation of the first or a reference image of the sequence of images is selected and the relative rotation matrices are used to produced a set of matrices of each image relative to the first or reference image by combining or compositing the relative rotation matrices.

The derived rotation matrices or the relative rotation matrices can be used to define the absolute or relative positions in a world co-ordinate system of corresponding images.

Preferably, once the relative rotation matrices R have been derived for each pair of images i and j, a set of non-contiguous images, which are preferably substantially evenly distributed throughout the sequence of images, is selected such that there is some overlap between pairs of the selected images. For each consecutive pair of selected images i and j an estimate of the camera image data, a, ss, 0, x and , is determined as above to produce rotation matrices Si > .

The rotation matrix Sij between any two images i and j is calculated from the relative rotation matrices R of the images which are between images i and j in the sequence. The difference between Sij and Rjj represents an error matrix EiJ such that Sij =Eij.Rij. Therefore, Si.RiJl = Eij The inverse matrix Rij is calculated by negating the value of 0 in the matrix Ri > . The matrix EiJ, that is the error, is then divided or spread across the relative rotation matrices for all or a selected number of the images which are between the two images i and image j. Assuming that the error is to be spread across n images, the error Eij is expressed in terms of a, ss and o and a rotation defined by a, ss and O/n is applied to each relative rotation matrix for all images between images i and j. The value of n may represent all images which are between the two images i and j or n may present a selected number of the images between the two images i and j.

The resulting matrices can then be used for further image processing, such as producing a mosaic of images, clean plate generation, improving the image quality or resolution and the like.

Image capture includes capture of images using camera such as, for example, charge coupled devices, or computer generated images within, for example, a virtual reality context or computer animation context using a virtual camera.

Claims

CLAIMS 1. A method for determining camera image data for first and second images of a sequence of images within an image processing system comprising memory for storing said first and second images, said method comprising the steps of storing in said memory the first and second images; and deriving, as a consequence of a comparison between the first and second images, camera image data for the first and second images.

2. A method as claimed in claim 1, wherein the step of deriving comprises the steps of iteratively comparing selectable portions of the first and second images; and producing, in response to the comparison, a first estimate of the camera image data; 3. A method as claimed in claim 2, wherein the camera image data comprises a plurality of parameters, and the method further comprises the steps of producing a fixed estimate of at least a selectable one, preferably, two, of the parameters; and re-iterating the steps of deriving, producing and comparing in order to produce a refined estimate of the camera image data.

4. A method as claimed in claim 3, wherein the step of producing comprises the steps of repeating the step of deriving for all or a selected portion of the images in the sequence of images, preferably, the selected portion of images which satisfy predeterminable criteria; and calculating an average value, preferably a weighted average value, of said at least one of the parameters.

6. A method as claimed in claim 5, wherein the predeterminable criteria relates to the distance between the centres of selected pairs of all or the selected portion of images.

7. A method as claimed in any preceding claim, further comprising the steps of selecting a subset of images from the sequence of images which satisfy further predeterminable criteria; and re-iterating the step of deriving using the subset of images as the sequence of images from which the first and second images are selected to produce for each pair or for selected pairs of images subset camera image data.

8. A method as claimed in claim 7, further comprising the steps of determining a set of error values from the subset camera image data; and refining the camera image data for the first and second images.

9. A method as claimed in claim 8, wherein the step of refining the camera image data for the first and second images comprises the step of dividing the error between any images which fall between the first and second within the sequence of images.

10. A method substantially as described herein with reference to, and/or as illustrated in, the accompanying drawings.

11. An image processing system for determining camera image data for first and second images of a sequence of images, the system comprising memory for storing said first and second images; means for storing in said memory the first and second images; and means for deriving, as a consequence of a comparison between the first and second images, camera image data for the first and second images.

12. A system as claimed in claim 11, wherein the means for deriving comprises means for iteratively comparing selectable portions of the first and second images; and producing, in response to the comparison, a first estimate of the camera image data; 13. A system as claimed in claim 12, wherein the camera image data comprises a plurality of parameters, and the system further comprises means for producing a fixed estimate of at least a selectable one, preferably, two, of the parameters; and means for re-iterating the steps of deriving, producing and comparing in order to produce a refined estimate of the camera image data.

14. A system as claimed in claim 13, wherein the means for producing comprises means for repeating the step of deriving for all or a selected portion of the images in the sequence of images, preferably, the selected portion of images which satisfy predeterminable criteria; and means for calculating an average value, preferably a weighted average value, of said at least one of the parameters.

15. A system as claimed in claim 14, wherein the predeterminable criteria relates to the distance between the centres of selected pairs of all or the selected portion of images.

16. A system as claimed in any preceding claim, further comprising means for selecting a subset of images from the sequence of images which satisfy further predeterminable criteria; and means for re-iterating the step of deriving using the subset of images as the sequence of images from which the first and second images are selected to produce for each pair or for selected pairs of images subset camera image data.

17. A system as claimed in claim 15, further comprising means for determining a set of error values from the subset camera image data; and means for refining the camera image data for the first and second images.

18. A system as claimed in claim 9, wherein the means for refining the camera image data for the first and second images comprises means for dividing the error between any images which fall between the first and second within the sequence of images.

19. An image processing system substantially as described herein with reference to, and/or as illustrated in, the accompanying drawings.

20. A computer program product for determining camera image data for first and second images of a sequence of images within an image processing system comprising memory for storing the first and second images, the product comprising computer program code means for storing in said memory the first and second images; and computer program code means for deriving, as a consequence of a comparison between the first and second images, camera image data for the first and second images.

21. A product as claimed in claim 20, wherein the computer program code means for deriving comprises computer program code means for iteratively comparing selectable portions of the first and second images; and producing, in response to the comparison, a first estimate of the camera image data; 22. A product as claimed in claim 21, wherein the camera image data comprises a plurality of parameters, and the product further comprises computer program code means for producing a fixed estimate of at least a selectable one, preferably, two, of the parameters; and computer program code means for re-iterating the steps of deriving, producing and comparing in order to produce a refined estimate of the camera image data.

23. A product as claimed in claim 22, wherein the computer program code means for producing comprises computer program code means for repeating the step of deriving for all or a selected portion of the images in the sequence of images, preferably, the selected portion of images which satisfy predeterminable criteria; and computer program code means for calculating an average value, preferably a weighted average value, of said at least one of the parameters.

24. A product as claimed in claim 23, wherein the predeterminable criteria relates to the distance between the centres of selected pairs of all or the selected portion of images.

25. A product as claimed in any of claims 20 to 24, further comprising means for selecting a subset of images from the sequence of images which satisfy further predeterminable criteria; and means for re-iterating the step of deriving using the subset of images as the sequence of images from which the first and second images are selected to produce for each pair or for selected pairs of images subset camera image data.

26. A product as claimed in claim 25, further comprising computer program code means for determining a set of error values from the subset camera image data; and computer program code means for refining the camera image data for the first and second images.

27. A product as claimed in claim 26, wherein the computer program code means for refining the camera image data for the first and second images comprises computer program code means for dividing the error between any images which fall between the first and second within the sequence of images.

28. An image processing system substantially as described herein with reference to, and/or as illustrated in, the accompanying drawings.