WO1999019836A1 - Systeme, procede et produit programme informatique de traitement d'images - Google Patents

Systeme, procede et produit programme informatique de traitement d'images Download PDF

Info

Publication number
WO1999019836A1
WO1999019836A1 PCT/GB1998/003038 GB9803038W WO9919836A1 WO 1999019836 A1 WO1999019836 A1 WO 1999019836A1 GB 9803038 W GB9803038 W GB 9803038W WO 9919836 A1 WO9919836 A1 WO 9919836A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
images
data
computer program
camera
Prior art date
Application number
PCT/GB1998/003038
Other languages
English (en)
Inventor
Kenneth Philip Appleby
Original Assignee
Harlequin Limited
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from GB9721592A external-priority patent/GB2330266A/en
Priority claimed from GB9721591A external-priority patent/GB2330265A/en
Application filed by Harlequin Limited filed Critical Harlequin Limited
Priority to AU94496/98A priority Critical patent/AU9449698A/en
Publication of WO1999019836A1 publication Critical patent/WO1999019836A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/97—Determining parameters from multiple pictures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20228—Disparity calculation for image-based rendering

Definitions

  • the present invention relates to an image data processing system, method, and computer program product.
  • a recorded sequence of images may contain artefacts which were not intended to be included within the sequence of images.
  • artefacts may be added to a sequence of images. For example, it may be desirable to add special effects to the images.
  • the removal or addition of such artefacts is referred to generically as post-production processing and involves a significant degree of skill and time.
  • image carrier such as a film, video tape, cd-rom, video disc, mass storage medium or the like.
  • complex digital filtering is typically required which is, again, both time consuming and expensive in terms of supporting hardware.
  • the present invention provides a method for producing a first image from a second image and a third image within an image processing system comprising storage means for storing the first image and first image camera data governing the orientation (a, b k , q k ) and the first focal length (l k ) of the first image, and for storing the second and third images with second and third image camera data corresponding to the second and third orientations ( x bi q , a. b.
  • the method comprising the steps of setting the first image camera data; and deriving first image data for the first image selectively from at least the second and third images using the first, second and third image camera data.
  • a second aspect of the present invention provides an image processing system for producing a first image from a second image and third image, the system comprising means for storing the first image and first image camera data governing the first orientation (a k , b k , _,.) of the first image and storing the second and third images with second and third image camera data corresponding to the second and third orientation (a ⁇ b L q a 3 b. q.,) and second and third focal lengths (l 1 : ) of a camera when the second and third images were captured; the system comprising means for setting the first image camera data; and means for deriving first image data for the first image selectively from at least the second and third images using the first, second and third image camera data .
  • a third aspect of the present invention provides a computer program product for producing a first image from a second image and third image within an image processing system comprising storage means for storing the first image and first image camera data governing the first orientation (a k , b k , q k ) and first focal length (l k ) of the first image and for storing the second and third images with second and third image camera data corresponding to the second and third orientation (a x h ⁇ q lf a. b.
  • the computer program product comprising computer program means for setting the first image camera data; and computer program code means for deriving first image data for the first image selectively from at least the second and third images using the first, second and third image camera data.
  • figure 1 illustrates a computer suitable for implementing an image processing system and/or method according to the embodiments of the present invention
  • figure 2 depicts a sequence of source images from which at least one generated image can be produced
  • figure 3 shows the production of a first image using image data derived from two source image frames i and j
  • figure 4 illustrates a frame co-ordinate system for an image
  • figure 5 shows the mapping of the frame co-ordinate system of figure 4 into a world co-ordinate system
  • figure 6 shows in terms of a world co-ordinate frame of reference how a vector may intersect several image frames
  • figure 7 illustrates from another perspective the intersection of a vector with several images frames
  • figure 8 depicts a frame co-ordinates system and image frame of a virtual image or image to be generated
  • figure 9 shows the extrapolation of a portion of the image frame of figure 8 onto a source image
  • figure 10 illustrates a flowchart for mapping the image portion of an
  • the computer 100 comprises at least one microprocessor 102, for executing computer instructions to process data, a memory 104, such as ROM and RAM, accessible by the microprocessor via a system bus 106.
  • Mass storage devices 108 are also accessible v a the system bus 106 and are used to store, off-line, data used by and instructions for execution by the microprocessor 102.
  • Information is output and displayed to a user via a display device 110 which typically comprises a VDU together with an appropriate graphics controller. Data is input to the computer using at least one of either of the keyboard 112 and associated controller and the mass storage devices 108.
  • Data for a first image 200 is produced by selecting portions of image data from at least two images 202 and 204 of a sequence of images 206.
  • Each image 1 to N of the sequence of images 206 and the first image to be generated 200 has associated therewith image camera data 208 to 218.
  • the image camera data 208 to 218 governs the position with reference to, for example, a world co-ordinates frame of reference of the images 1 to N and the first image 200.
  • the image camera data represents or is related to the focal length of a camera which was used to capture the images 1 to N as well as the orientation of the camera when the images were captured.
  • the associated image camera data comprises a x and b, which determine the orientation within the world co-ordinates reference of a frame, and hence indirectly the camera, by way of an axis of rotation.
  • q t represents a degree of rotation about that axis relative to a reference within the world co-ordinate system.
  • the focal length l x of image i 202 governs the distance from the origin, which coincides with the optical centre of the camera to the centre of the image frame i 202.
  • image k 200 is derived from second and third images, image i 202 and image j 204.
  • image k 200 is derived from image i, that is to say, image k is identical to image i but for a first image portion 300.
  • the first image portion 300 is derived from a corresponding portion 302 of the second image 204. Therefore, rather than the first image 200 containing the image data contained within a correspondingly located portion 304 of the second image 202, the image data for first image portion 300 is derived from the image data contained within portion 302 of the third image.
  • Each image 1 to N of the sequence of images 206 is stored with reference to a corresponding image frame coordinate system such as that shown in Figure 4.
  • the image frame co-ordinate system 400 in the present embodiment, comprises three mutually orthogonal axes 402, 404 and 406 which correspond to the x, y and z axis in a right handed Cartesian co-ordinate system respectively.
  • An image 408 is centred on the z-axis 406 at a distance l - which represents the focal length of the camera at the time when the image 403 was captured, generated or retrieved.
  • FIG. 5 there is shown the image 408 of Figure 4 and the corresponding image frame coordinate system when that co-ordinate system has been mapped into, for example, a world co-ordinate system 500 which also comprises three mutually orthogonal axes 502, 504 and 506 forming a right handed Cartesian co-ordinate system.
  • a pixel Vp within image 408 is mapped from its position within the image frame co-ordinate system to a corresponding position, Vp 1 , within the world co-ordinate system by a suitable matrix S. Therefore, the position, in terms of the world co-ordinate system 500, of any pixel within the image 408 can be determined by multiplying the image frame 5 reference co-ordinates of that pixel by the matrix S.
  • the point of intersection 604 is illustrated as a single pixel on image frame number 74. It will be appreciated that the frame co-ordinates of the point
  • Figure 7 shows a sequence of consecutive but noncontiguous images 700 having identified within each of the images 702 to 708 a corresponding portion 710 of the images
  • FIG. 8 there is shown a first image to be generated or modified 800 as positioned within a corresponding frame co-ordinate system 802 comprising three mutually orthogonal axes 804, 806 and 808 which form a right 35 handed Cartesian co-ordinate system.
  • the focal length l k of the image 800 is shown along the z axis.
  • a portion 810 of the image 800 to be generated is selected.
  • the portion 810 may correspond to a single pixel of the image 800 or to a group of pixels.
  • the location, or co-ordinates, of the portion 800 with respect to the frame co-ordinate system 802 is determined.
  • the co-ordinates of the portion 810 are mapped, using an appropriate transformation, into corresponding world co-ordinates of the world co-ordinates system illustrated in Figure 5.
  • a determination or selection from the sequence of images 206 is then made in order to establish those images from which image data for the first portion 810 can be derived.
  • Each of the source images 1 to N also has associated therewith a matrix R L to R N which transforms that source image from corresponding frame co-ordinates into world co-ordinates as well as the inverse of such a matrix R x ' ⁇ to R N _1 .
  • the inverse matrices map the images 1 to N from the world co-ordinate system 500 into the corresponding frame co-ordinate systems.
  • FIG. 10 there is shown schematically a flow chart which implements the mapping of a first image portion 810 of a first image onto a second image portion 900 of a second image, for example image i, in order to determine second image data of the second image portion 900 from which first image data can be derived for the first image .
  • any orientation of the co-ordinate system shown in Figure 8 within the world co-ordinate system may map the first image portion onto several images from which image data can be derived in order to produce the first image data for the first portion 810. Therefore, there will be several projections of the first image portion onto prospective images from which data can be derived. In such a case, the steps 1006 to 1004 of Figure 10 are repeated for all or a selectable portion of the eligible source images 206.
  • An eligible source image is an image which is intersected by the vector, or the extrapolation thereof, produced by the co-ordinates of the first image portion 810 in terms of corresponding frame co-ordinates.
  • One embodiment determines the distance of each of the projections 900 of the first portion 810 onto the eligible source frames 206 and selects image data according to the distances.
  • the location of the point of intersection or the projection of that point onto the image is given by l t ) .
  • the distance of the projections 900 of the first image portion 810 onto the eligible source images from the corresponding centres of the eligible source images is determined.
  • the image data of the projection 900 which is closest to its corresponding image frame centre is used as the basis for deriving first image data for the first image portion 810.
  • the image data of the projections 900 of the first image portion 810 onto the eligible source frames 206 is averaged and that average value is used to determine or derive the first image data for the first image portion 810.
  • the above calculated average is a weighted according to the distance of any given projection from its corresponding image frame centre.
  • the image data retrieved from the projections 900 represents the colour or RGB component at the location of the projections within the eligible source images .
  • the colour data may be derived or sampled from an eligible source image using, for example, a bilinear interpolation technique or some other more sophisticated sampling technique.
  • the resolution of the generated image can be increased. This has the consequence of improving the image quality, that is, an increase in the resolution of the generated image as compared to the eligible source images can be realised.
  • the noise of images which is attributable to the grain of, for example, a film medium or a video tape used to record the source images can be reduced by generating images using the present invention.
  • the noise introduced during image capture can be reduced if the data for any given pixel, or portion of the image co be generated, is derived from corresponding portions of several images .
  • FIG. 11 there is shown an image 1100 which has been generated from a plurality of eligible source images in which the image data has been selected from the projections which were closest to the centre of corresponding eligible source images . It can be seen from within the defined area 1102 that this image data selection strategy results in a smearing or blurring of foreground or moving images. It will be appreciated that the source image sequence from which the image 1100 was derived illustrates a person walking past the steps of an entrance to a house.
  • FIG. 12 These individual eligible source images or at least a selection of the individual eligible source images from which the image 1100 in Figure 11 can be derived are shown in Figure 12, 13, 14 and 15.
  • Figure 12 there is shown a region 1202 from which background image data can be derived in order to remove the smearing which is depicted in Figure 11 within box 1102.
  • the image data derived from the portion of Figure 12 defined by the region 1202 will be sufficient to remove the smearing defined within box 1104 of image 1100, that is the three right most images of the person can be removed by incorporating into the image 1100 the background image data contained within or derived from the identified region 1202 of Figure 12.
  • the left most person depicted in Figure 11 can be removed by copying into that region of Figure 11 the image data defined or contained within, for example, the region 1502 of Figure 15. Therefore, by firstly combining the image data of Figures 12, 13, 14 and 15 and then by combining the resultant image 1100 shown in Figure 11 with selected portions of the source images an image 1600 as shown in Figure 16 can be produced.
  • the image 1600 shown in Figure 16 represents a clean plate, that is, an image which contains only background image data.
  • Figure 17 represents the clean plate 1600 which results from the above processing without the defining box 1102.
  • FIG. 18 there is shown a sequence of four consecutive but non-contiguous source images 1800 to 1806.
  • a selected portion of one frame 1800 is edited in order to remove, for example, the eagle 1808 using either an appropriate tool for editing or a combination of selectable portions of source image 1800 and, for example, source image 1806 in substantially the same manner as defined above with reference to the clean plate generation process.
  • the edited frame 1810 is utilised in conjunction with the remaining source images 1802 to 1806 in order to generate new images 1812 to 1818.
  • eagle 1808 has been removed from the source frame 1800 and that removal has been propagated throughout subsequently generated images 1812 to 1818.
  • a source frame could be edited to include some new artefact and that new artefact can be propagated readily throughout subsequently generated images in substantially the same manner as described above with reference to object removal.
  • images can be generated which have a different camera format as compared to the format in which the source images were captured.
  • FIG 19 there is shown a single generated image 1900 having a significantly different aspect ratio as compared to that of the source images of Figure 18.
  • the generated image of 1900 of Figure 19 has been derived from several source images. In this way the aspect ratio of a generated image can be set.
  • the smearing depicted in Figure 11 is also apparent in Figure 19.
  • the image data for the area of the larger aspect ratio image 1900 contained within the box 1904 can be derived from at least one of any of the source images 1800 to 1806. Therefore, the larger aspect ratio image comprises the background of several source images together with the foreground or action derived from at least one of the source images .
  • the above derivation can be repeated in order co produce a plurality of larger aspect ratio source images or a sequence of larger aspect ratio images .
  • FIG. 20 there is shown a sequence of consecutive but non-contiguous source frames 2002 to 2006 in which one of the source images 2004 comprises a scratch 2008.
  • An image frame 2010 is generated in which the image data within a pre-defined area 2012 of the source frame is derived from a corresponding area of, for example, a source image 2006 in substantially the same manner as described above. Therefore, the scratch 2008 can be removed from the image, thereby improving the quality of the images.
  • An image or clean plate 2110 can be generated from all or a selectable subset of the source images as follows. For each pixel within the image 2110 to be generated, corresponding image data is derived or extracted from all or a selectable subset of the eligible source images. It will be appreciated that the derived image data for those regions of each source image which do not comprise moving images will remain substantially constant .
  • the image data derived from the source images for any given pixel or portion of the image to be generated is arranged within, for example, a histogram according to, for example, predeterminable bands of the colour information. The histogram is used to determine the most frequently occurring colour or predeterminable range of colours within the corresponding portions of the source frames . That most frequently occurring colour is then used for the appropriate portion of the generated image 2110. In this way an image or clean plate 2110 is generated which is completely free of any moving or changing image data .
  • the term captured refers to the generation or recording of digital images. It will therefore be appreciated that a CCD camcorder captures digital images and stores the capture image on a suitable tape. It will also be appreciated that capturing an image includes the generation of a digital image within or by a computer or other image/data processing environment .
  • the images which are generated as a consequence of the present invention may be output together with the image camera data for further processing such as, for example, incorporation within a virtual reality environment or to an environment which combines computer generated images with the generated images.
  • camera data taken from a computer animation environment together with the corresponding images be utilised in order to combine the animated images with the captured images .
  • the setting of the image camera data for an image includes retrieving the image camera data for an existing image or generating image camera data or retrieving image camera data from some storage medium or receiving image camera data via some transmission medium.
  • the first image to be produced may in fact be the second image, that is, the first image and the second image are one and the same.
  • the term "producing an image” includes modifying that image.
  • image data for an image from ac least one other image preferably two images,- can take place either substantially concurrently or sequentially.
  • image data may be derived from several source images, each source image may be processed one at a time.
  • a suitable transformation for mapping data between images or for locating data within an image may be the rotation given below:
  • T is the rotation matrix
  • ⁇ represents the focal length of an image and R is given by cos ⁇ ) + u y sin ⁇ cos ⁇ ) - u x sin ⁇ : ) cos ⁇ + u 2 2
  • u represents the orientation of an arbitrary axis with respect to a reference frame
  • ⁇ represent a degree of rotation about the arbitrary axis u with respect to a reference , for example a common reference or another image frame .
  • u sin ⁇ .cos ⁇
  • U y sin ⁇
  • u z cos ⁇ .cos ⁇
  • cc is the angle between the projection of u onto the xz and the z axis
  • ⁇ is the angle between the projection of u and u.
  • the rotation may represent the mapping between a source image and a target image or may represent the positions of the source and target images within a frame of reference relative to some common reference.
  • the rotation is used to identify the locations within source and target images of corresponding image data, that is, image data which represents the same object or part of an object or the same background feature
  • the image camera data may be derived from a sequence of images camera using the invention which s the subject of UK patent application no. GB 9721592.5, the content of which is incorporated herein by reference for all purposes and a copy of which is included herein m the appendix.
  • the present invention relates to an image processing system, method, and computer program product.
  • a recorded sequence of images may contain artefacts which were not intended to be included within the sequence of images.
  • artefacts may be added to a sequence of images. For example, it may be desirable to add special effects to the images.
  • the removal or addition of such artefacts is referred to generically as post-production processing and involves a significant degree of skill and time.
  • the present invention provides a method for determining camera image data for at least one of first and second images of a sequence of images within an image processing system comprising memory for storing said first and second images, said method comprising the steps of storing in said memory the first and second images; and deriving, as a consequence of a comparison between the first and second images, camera image data for at least one of the first and second images.
  • the camera image data for each image within a sequence thereof can be determined.
  • the determined camera image data can then be used in order to give effect to further image processing such as, for example, producing a mosaic of selected images .
  • a second aspect of the present invention provides an image processing system for determining camera image data for at least one of first and second images of a sequence of images, the system comprising memory for storing said first and second images, means for storing in said memory the first and second images; and means for deriving, as a consequence of a comparison between the first and second images, camera image data for at least one of the first and second images .
  • a third aspect of the present invention provides a computer program product for determining camera image data for at least one of first and second images of a sequence of images within an image processing system comprising memory for storing said first and second images, the product comprising computer program code means for storing in said memory the first and second images; and computer program code means for deriving, as a consequence of a comparison between the first and second images, camera image data for at least one of the first and second images .
  • figure 1 illustrates a computer suitable for implementing an image processing system or method according to the embodiments of the present invention
  • figure 2 depicts a sequence of source images from which at least one generated image can be produced
  • figure 3 shows a world co-ordinate system having an image i positioned therein according to corresponding image camera data ⁇ supervise ⁇ 1( ⁇ 1 and ⁇ 1( - figure 4 illustrates several equations utilised by the present invention
  • figure 6 illustrates a further equation utilised by the present invention.
  • the computer 100 comprises at least one microprocessor 102, for executing computer instructions to process data, a memory 104, such as ROM and RAM, accessible by the microprocessor via a system bus 106.
  • Mass storage annex devices 108 are also accessible via the system bus 106 and are used to store, off-line, data used by and instructions for execution by the microprocessor 102.
  • Information is output and displayed to a user via a display device 110 which typically comprises a VDU together with an appropriate graphics controller. Data is input to the computer using at least one of either of the keyboard 112 and associated controller and the mass storage devices 108.
  • the images may be derived from a film, video tape, CD- ROM,- video disc or other mass storage medium.
  • the image data varies as between images as a consequence of a combination of motion of an object within an image and as a consequence of, for example, variations in the position or orientation of a camera during image capture.
  • the camera orientation information is lost.
  • FIG. 3 there is shown schematically a world co-ordinate system 300 comprising three axes 302, 306 and 308 which correspond to the x, y, and z axes respectively.
  • the position of one image, image j, with reference to another image, image i, can be described in terms of the orientation of a unit vector, u, within the world co-ordinate system in conjunction with a degree of rotation, ⁇ 1( about that vector.
  • the orientation of the unit vector within the world co-ordinate system 300 can be specified using two angles, ⁇ t and .
  • the combination of ⁇ ⁇ r ⁇ L and ⁇ L define a rotation which maps image i onto image j or which can be used to identify corresponding portions of images i and j .
  • the distance ⁇ represents the distance from the origin to the centre of the image i.
  • the distance ⁇ . represent the distance from the origin to the centre of image j .
  • the camera is positioned such that the optical centre thereof is as close as possible to the origin.
  • the line of sight 308 of the camera is annex normal to the centre of the image i, which represents in reality the field of view of the camera at a focal length determined by ⁇ x .
  • the camera image data ⁇ x , ⁇ 1( ⁇ ⁇ ; and ⁇ x is not available when retrieving images from some mass storage media such as, for example, a video disc or CD-ROM or the like.
  • the determination of the above parameters utilises a function which compares the colour differences between pixels of any two given images .
  • the first image of the two images is addressed using conventional pixel row and address information.
  • the second image of the two images is addressed using co-ordinates which have been transformed annex using the matrix T as shown in figure 4.
  • the second image is preferably addressed using sub-pixels so that colour interpolation can be performed.
  • the function which is used to measure the colour difference and hence derive optimal values for , ⁇ x , ⁇ ' and ⁇ is as follows:
  • N is the number of pixels for which the transformed coordinates fall inside the image frame boundary of the second image .
  • the minimisation process preferably uses the simplex method as is well known within the art (see, for example, "Numerical Recipes in C” , The Art of Scientific Computing, second edition, Press by Teukolsky, Vetterling, Flannery, CUP, IUSBN 0 521 431085, the entire content of which is incorporated herein by reference) .
  • two images i and j are selected at step 500 from a sequence of images
  • a portion or sample, P, of pixels from the first image i of the two images is selected at step 502.
  • the sample P preferably comprises 2000 pixels wnich are expressed in terms of pixel rows and columns .
  • the initial estimates of the camera image data parameters are set at step 504.
  • at least three of the parameters ⁇ , , ⁇ x , ⁇ ' , ⁇ and ⁇ should be initialised.
  • the initialised parameters are used to establish a first estimate of the matrix T illustrated in figure 4.
  • the pixels of the sample P are transformed, using the current estimate of matrix T, into annex the second image to produce a second set of sample pixels P ' .
  • the two images i and j are compared at steps 508 and 510 to determine the degree of similarity or alignment therebetween.
  • effect is given to the comparison by extracting image data, such as, for example, rgb colour data, from the pixels at sample points P in image i and at sample points P ' in image j and determining the differences between the colour data.
  • the differences are evaluated using the function given above for calculating ⁇ C.
  • step 530 The re-iteration of steps 500 to 518 for selected or annex all pairs of eligible images is commenced at step 530. Once it is determined that all eligible images have been processed, average values for ⁇ and ⁇ are determined at step 530.
  • the averages are weighted according to the reciprocals of the final colour difference ⁇ C between the pairs of images .
  • each image pair has a set of camera image data governing the orientation, focal length and lens distortion of the camera at the instant the pair of images were captured.
  • a (3x3) relative rotation matrix, R i: is calculated, using the estimated values of ⁇ , ⁇ , ⁇ , and ⁇ .
  • a relative rotation matrix governs the relative rotation between the two images.
  • the orientation of the first or a reference image of the sequence of images is selected and the relative rotation matrices are used to produced a set of matrices of each image relative to the first or reference image by combining or compositing the relative rotation matrices .
  • the derived rotation matrices or the relative rotation matrices can be used to define the absolute or relative positions in a world co-ordinate system of corresponding images relative to a reference image or relative to an arbitrary common reference.
  • a set of noncontiguous images which are preferably substantially evenly distributed throughout the sequence of images, is selected such that there is some overlap between pairs of the selected images.
  • an estimate of the camera image data, ⁇ , ⁇ , ⁇ , ⁇ and ⁇ is determined as above to produce rotation matrices S 13 .
  • the rotation matrix S ⁇ :) between any two images i and j is calculated from the relative rotation matrices R of the images which are between images i and j in the sequence.
  • the inverse matrix R ⁇ "1 is calculated by negating the value of ⁇ in the matrix R ⁇ : .
  • the matrix E ⁇ :) that is the error, is then divided or spread across the relative rotation matrices for all or a selected number of the images which are between the two images i and image j .
  • the error E 13 is expressed in terms of ⁇ , ⁇ and ⁇ and a rotation defined by ⁇ , ⁇ and ⁇ /n is applied to each relative rotation matrix for all images between images i and j .
  • the value of n may represent all images which are between the two images i and or n may present a selected number of the images between the two images i and j .
  • the resulting matrices can then be used for further image processing, such as producing a mosaic of images, clean plate generation, improving the image quality or resolution and the like.
  • Image capture includes capture of images using camera such as, for example, charge coupled devices, or computer generated images within, for example, a virtual reality context or computer animation context using a virtual camera .
  • camera such as, for example, charge coupled devices, or computer generated images within, for example, a virtual reality context or computer animation context using a virtual camera .
  • annex
  • a method for determining camera image data for at least one of first and second images of a sequence of images within an image processing system comprising memory for storing said first and second images, said method comprising the steps of storing in said memory the first and second images; and deriving, as a consequence of a comparison between the first and second images, camera image data for at least one of the first and second images.
  • step of deriving comprises the steps of iteratively comparing selectable portions of the first and second images ; and producing, in response to the comparison, a first estimate of the camera image data;
  • step of producing comprises the steps of repeating the step of deriving for all or a selected portion of the images in the sequence of images, preferably, the selected portion of images which satisfy predeterminable criteria; and calculating an average value, preferably a weighted average value, of said at least one of the parameters.
  • a method as claimed in any preceding claim, further 5 comprising the steps of selecting a subset of images from the sequence of images which satisfy further predeterminable criteria; and re-iterating the step of deriving using the subset of images as the sequence of images from which the first and 0 second images are selected to produce for each pair or for selected pairs of images subset camera image data.
  • step of refining the camera image data for the first and second images comprises the step of dividing the error between any images which fall between the first and second within the sequence of images.
  • An image processing system for determining camera image data for at least one of first and second images of a sequence of images, the system comprising memory for storing said first and second images; means for storing in said memory the first and second 35 images ; and means for deriving, as a consequence of a comparison between the first and second images, camera image data for at least one of the first and second images.
  • the means for deriving comprises means for iteratively comparing selectable portions of the first and second images ; and producing, in response to the comparison, a first estimate of the camera image data;
  • a system as claimed in claim 12, wherein the means for producing comprises means for repeating the step of deriving for all or a selected portion of the images in the sequence of images, preferably, the selected portion of images which satisfy predeterminable criteria; and means for calculating an average value, preferably a weighted average value, of said at least one of the parameters.
  • a system as claimed in claim 13 wherein the predeterminable criteria relates to the distance between the centres of selected pairs of all or the selected portion of images .
  • a system as claimed in claim 16, wherein the means for refining the camera image data for the first and second images comprises
  • a computer program product for determining camera image data for at least one of first and second images of a sequence of images within an image processing system comprising memory for storing the first and second images,
  • the product comprising computer program code means for storing in said memory the first and second images; and computer program code means for deriving, as a consequence of a comparison between the first and second 30 images, camera image data for at least one of the first and second images .
  • 35 code means for iteratively comparing selectable portions of the first and second images ; and producing, in response to the comparison, a first estimate of the camera image data; annex
  • the computer program code means for producing comprises computer program code means for repeating the step of deriving for all or a selected portion of the images in the sequence of images, preferably, the selected portion of images which satisfy predeterminable criteria; and computer program code means for calculating an average value, preferably a weighted average value, of said at least one of the parameters.
  • a product as claimed in claim 25, wherein the computer program code means for refining the camera image data for the first and second images comprises computer program code means for dividing the error between any images which fall between the first and second within the sequence of images.
  • the present invention relates to an image processing system, method and computer program product for processing or generating an camera image data from given at least two source images by iteratively estimating the parameters constituting the camera image data and using a predeterminable comparison between the first and second images as the basis for determining whether or not to continue said iterative estimating.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

Cette invention concerne un système, un procédé et un produit programme informatique de traitement d'images pour le traitement ou la création de données d'image à partir d'au moins deux images sources. La méthode couverte par cette invention permet d'obtenir une première image à partir d'une deuxième et d'une troisième image à l'intérieur d'un système de traitement d'images comprenant des moyens servant au stockage de la première image et des données caméra qui règlent l'orientation (ak, bk, qk) et une première distance focale (lk) de la première image, et servant au stockage des deuxième et troisième images avec les données caméras de ces dernières images correspondant aux deuxième et troisième orientations (ai bi ci, aj bj cj) et aux deuxième et troisième distances focales (li, lj) d'une caméra au moment ou les deuxième et troisième images ont été saisies. Le procédé comporte les étapes consistant à, d'une part définir les données caméra de la première image et, d'autre part, dériver des données de première image pour la première image par sélection à partir d'au moins une des deuxième et troisième images à l'aide des données caméra des première, deuxième et troisième images.
PCT/GB1998/003038 1997-10-10 1998-10-12 Systeme, procede et produit programme informatique de traitement d'images WO1999019836A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
AU94496/98A AU9449698A (en) 1997-10-10 1998-10-12 Image processing system, method and computer program product

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
GB9721592A GB2330266A (en) 1997-10-10 1997-10-10 Generating camera image data for an image processing system
GB9721591.7 1997-10-10
GB9721592.5 1997-10-10
GB9721591A GB2330265A (en) 1997-10-10 1997-10-10 Image compositing using camera data

Publications (1)

Publication Number Publication Date
WO1999019836A1 true WO1999019836A1 (fr) 1999-04-22

Family

ID=26312414

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/GB1998/003038 WO1999019836A1 (fr) 1997-10-10 1998-10-12 Systeme, procede et produit programme informatique de traitement d'images

Country Status (2)

Country Link
AU (1) AU9449698A (fr)
WO (1) WO1999019836A1 (fr)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0509208A2 (fr) * 1991-03-11 1992-10-21 Director-General Of The Agency Of Industrial Science And Technology, Procédé de détection du déplacement d'une caméra
EP0514266A2 (fr) * 1991-05-13 1992-11-19 Sony Corporation Appareil et procédé de transformation d'image

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0509208A2 (fr) * 1991-03-11 1992-10-21 Director-General Of The Agency Of Industrial Science And Technology, Procédé de détection du déplacement d'une caméra
EP0514266A2 (fr) * 1991-05-13 1992-11-19 Sony Corporation Appareil et procédé de transformation d'image

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
PARK J -I ET AL: "Robust estimation of camera parameters from image sequence for video composition", SIGNAL PROCESSING. IMAGE COMMUNICATION, vol. 9, no. 1, November 1996 (1996-11-01), pages 43-53, XP004071578 *
SRINIVASAN M V ET AL: "Qualitative estimation of camera motion parameters from video sequences", PATTERN RECOGNITION, vol. 30, no. 4, April 1997 (1997-04-01), pages 593-606, XP004059156 *
ZHUANG H ET AL: "DEPTH ESTIMATION FROM A SEQUENCE OF MONOCULAR IMAGES WITH KNOWN CAMERA MOTION", ROBOTICS AND AUTONOMOUS SYSTEMS, vol. 13, no. 2, 1 July 1994 (1994-07-01), pages 87 - 95, XP000454414 *

Also Published As

Publication number Publication date
AU9449698A (en) 1999-05-03

Similar Documents

Publication Publication Date Title
GB2330265A (en) Image compositing using camera data
US6278460B1 (en) Creating a three-dimensional model from two-dimensional images
JP4898459B2 (ja) 奥行きマップの生成
EP0930585B1 (fr) Appareil pour la traitement d'images.
US7511730B2 (en) Image processing apparatus and method and image pickup apparatus
US7084875B2 (en) Processing scene objects
US7907793B1 (en) Image sequence depth enhancement system and method
KR101168384B1 (ko) 깊이 맵을 생성하는 방법, 깊이 맵 생성 유닛, 이미지 처리 장치, 및 컴퓨터 프로그램 제품
US6429875B1 (en) Processing image data
EP0638875A2 (fr) Appareil et méthode pour génération d'animation en 3 dimensions
EP1008956A1 (fr) Système de montage automatique d'image
EP0930583A1 (fr) Appareil et méthode d'enregistrement d'une image utilisant des champs lumineux pour traquer une position et une orientation
KR20030036747A (ko) 원래 영상에 사용자 영상을 수퍼임포징하는 방법 및 장치
Klein et al. Simulating low-cost cameras for augmented reality compositing
EP0903695B1 (fr) Appareil de traitement d'images
JP2005122361A (ja) 画像処理装置及び方法、コンピュータプログラム、記録媒体
US4899295A (en) Video signal processing
JPH0721365A (ja) 画像処理方法および装置
WO1999019836A1 (fr) Systeme, procede et produit programme informatique de traitement d'images
JP7328942B2 (ja) ３ｄモデル生成装置ならびに仮想視点映像生成装置、方法およびプログラム
JP2973432B2 (ja) 画像処理方法および装置
WO1999019837A1 (fr) Systeme, procede et produit programme informatique de traitement des images
JP3318248B2 (ja) 画質改善装置及び記録媒体
KR20040042470A (ko) 합성영상 컨텐츠 저작도구를 위한 카메라 정보추출 방법
JP4560441B2 (ja) 画像合成装置、画像合成方法、及び画像合成プログラム

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AL AM AT AU AZ BA BB BG BR BY CA CH CN CU CZ DE DK EE ES FI GB GE GH GM HR HU ID IL IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MD MG MK MN MW MX NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT UA UG US UZ VN YU ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GH GM KE LS MW SD SZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF BJ CF CG CI CM GA GN GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
NENP Non-entry into the national phase

Ref country code: KR

REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase

Ref country code: CA