JP2007025863A - Photographing system, photographing method, and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a photographing system which very precisely estimates the posture of a photographing means whose posture is changeable without using feature points of a fixed object in an observation environment. <P>SOLUTION: An initial proofreading part 31 computes an epipole and epipolar homography between cameras when initial proofreading is made. Extraction parts 21-2n extract silhouette images from images which are photographed by a camera, whose posture is to be estimated, and another camera when the posture of the camera changes. A first epipolar line computation part 32 computes the epipolar line of another camera from the silhouette image of another camera and the epipole between the cameras, and a second epipolar line computation part 33 computes the epipolar line of the camera from the epipolar line of another camera and the epipolar homograph between the cameras. A movement quantity computation part 34 finds a conversion P<SB>i</SB>in which the distance between the epipolar line of the camera and the silhouette image posterior to the movement is minimum, and a posture estimation part 35 estimates the posture of the camera from the conversion P<SB>i</SB>. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a photographing system and a photographing method for photographing an image of an object using a plurality of photographing means, and an image processing program for processing an image of an object photographed using a plurality of photographing means.

Multi-viewpoint imaging systems that capture images of objects using multiple cameras have been applied to various systems such as security systems, non-contact type interface systems, and three-dimensional shape restoration systems. Proposed. For example, regarding the generation of a three-dimensional model from multiple viewpoints, many systems using the visual volume intersection method have been proposed. In Non-Patent Document 1, the visual volume intersection method and the stereo matching method are obtained by synchronous observation of 22 cameras. A three-dimensional model generation method using the above is disclosed.
Kajiyama et al., “Method of generating a three-dimensional moving object from multi-viewpoint images using visual volume intersection method and stereo matching method”, Journal of the Institute of Image Information and Television Engineers, 2004, Vol. 58, no. 6, p. 797-p. 806

  However, in order to restore 3D shape information by a multi-viewpoint imaging system, it is necessary to perform camera calibration by obtaining the positions and orientations of all cameras. Because a single color background is used to facilitate extraction, feature points cannot be obtained in the background, and conventional camera calibration methods that use feature points of fixed objects in the observation environment should be used. I can't.

  Several camera calibration methods using silhouettes have also been proposed, but these methods are intended for fixed cameras, so they should be applied to multi-viewpoint shooting systems using movable cameras whose posture changes. I can't.

  An object of the present invention is to provide a photographing system, a photographing method, and an image processing program capable of accurately estimating the posture of a photographing unit whose posture can be changed without using a feature point of a fixed object in an observation environment. That is.

  An imaging system according to the present invention includes a plurality of imaging units configured to change an attitude of at least one imaging unit and imaging an object, an epipole between one imaging unit and another imaging unit at a predetermined time, and Acquisition means for acquiring epipolar homography, and extraction means for extracting a silhouette image representing a silhouette of an object from images taken by one photographing means and another photographing means when the posture of one photographing means changes The epipolar line of the other photographing means is calculated based on the silhouette image of the other photographing means extracted by the extracting means and the epipole between one photographing means and the other photographing means obtained by the obtaining means. A first calculation unit, an epipolar line of another imaging unit calculated by the first calculation unit, one imaging unit acquired by the acquisition unit, and another imaging unit Based on the epipolar homography, the second calculating means for calculating the epipolar line of the one photographing means, and the extracting means so as to match the epipolar line of the one photographing means calculated by the second calculating means Moving means for moving the extracted silhouette image of the one photographing means.

  In the imaging system according to the present invention, when the epipole and epipolar homography between one imaging unit and another imaging unit at a predetermined time are acquired and the attitude of the one imaging unit changes, the one imaging unit and A silhouette image representing the silhouette of the object is extracted from an image photographed by another photographing means, the extracted silhouette image of the other photographing means, and between the obtained one photographing means and the other photographing means Based on the epipole, the epipolar line of the other photographing means is calculated, and the calculated epipolar line of the other photographing means and the acquired epipolar homography between the one photographing means and the other photographing means. The epipolar line of one imaging means is calculated for each of the imaging means, and one of the imaging means extracted when the attitude of the one imaging means changes to match the calculated epipolar line of the one imaging means. Silhouette image means is moved.

  Here, the silhouette image after movement is one photographing means calculated from the epipolar line of another photographing means at the time when the posture changes and the epipolar homography between the one photographing means and the other photographing means. Therefore, it can be regarded as a silhouette image when it is assumed that one photographing means has photographed the object in the posture before the posture change at the time when the posture is changed. Therefore, by obtaining the amount of movement of the silhouette image extracted at the time when the posture changes, that is, the silhouette image after the posture change, with respect to the silhouette image after the movement, that is, the silhouette image before the posture change, the posture of one photographing means is determined. Since it can be estimated, the posture of the photographing means capable of changing the posture can be estimated with high accuracy without using the feature point of the fixed object in the observation environment.

  The moving means moves the silhouette image of one photographing means extracted by the extracting means so as to match the epipolar line of the one photographing means calculated by the second calculating means. It is preferable to further include an estimation unit that calculates the movement amount of the image and estimates the posture of one photographing unit based on the movement amount calculated by the movement unit.

  In this case, the amount of movement of the silhouette image of one photographing means when the silhouette image of one photographing means extracted so as to match the calculated epipolar line of the other photographing means is calculated and calculated. Since the posture of one photographing means can be estimated based on the amount of movement, the posture of the photographing means whose posture can be changed can be estimated with high accuracy by simple processing.

The moving means is H _kl for epipolar homography between one imaging means and other imaging means, l _{lk, 1} , l _{lk, 2} for the epipolar lines of the other imaging means, The silhouette image of the photographing means is S _k ⁽ⁱ⁾ , the transformation for moving the silhouette image S _k ⁽ⁱ⁾ on the image is P, and the silhouette image transformed by the transformation P is P (S _k ⁽ⁱ⁾ ), And when the point on the silhouette image S through which the tangent parallel to the epipolar line l of one photographing means calculated by the second calculating means passes is p _{S, l} , the epipolar line l and the point p _S, If the distance between _l expressed in D (S, l), it is preferable to obtain the conversion P _i to the following formula (1) and the minimum.

In this case, it is possible to calculate the movement amount of the silhouette images of one imaging unit from the conversion P _i obtained, is possible to calculate the movement amount of the silhouette image at high speed and with high accuracy by using a simple formula it can.

  The acquisition means preferably acquires epipole and epipolar homography between one imaging means and other imaging means at the time of initial calibration.

  In this case, since the epipole and epipolar homography between one imaging means and other imaging means at the time of initial calibration are acquired, it is only necessary to acquire the epipole and epipolar homography once at the time of initial calibration. Thereafter, it is possible to continuously estimate the posture of the photographing means whose posture changes variously.

  An imaging method according to the present invention includes a plurality of imaging means, an acquisition means, an extraction means, a first calculation means, a second calculation means, and a movement, in which at least one imaging means is configured to be capable of changing its posture An imaging method using an imaging system comprising means for acquiring an epipole and epipolar homography between one imaging means and another imaging means at a predetermined time, and an extracting means Extracting the silhouette image representing the silhouette of the object from the images photographed by the one photographing means and the other photographing means when the posture of the photographing means is changed, and the first calculating means is the extracted other Determining an epipolar line of another imaging means based on the silhouette image of the imaging means and the acquired epipole between the one imaging means and the other imaging means; The calculation means calculates the epipolar line of one imaging means based on the determined epipolar line of the other imaging means and the acquired epipolar homography between the one imaging means and the other imaging means. And a step of moving the silhouette image of the one photographing unit extracted by the extracting unit so that the moving unit matches the calculated epipolar line of the one photographing unit.

  An image processing program according to the present invention is an image processing program in which at least one photographing unit is configured to be able to change its posture and processes an image photographed by a plurality of photographing units that photograph an object. Acquisition means for acquiring epipole and epipolar homography between the imaging means and other imaging means, and images taken by one imaging means and other imaging means when the posture of one imaging means changes Extraction means for extracting a silhouette image representing the silhouette of the object, silhouette images of other photographing means extracted by the extracting means, and epipole between one photographing means and other photographing means obtained by the obtaining means Based on the above, the first calculation means for calculating the epipolar line of the other imaging means, and the epipolar of the other imaging means calculated by the first calculation means And a second calculation unit that calculates an epipolar line of the one imaging unit based on the epipolar homography between the one imaging unit and the other imaging unit acquired by the acquisition unit, and a second calculation The computer functions as a moving means for moving the silhouette image of one photographing means extracted by the extracting means so as to match the epipolar line of one photographing means calculated by the means.

  According to the present invention, since the epipole and epipolar homography between one photographing means and another photographing means and the silhouette image after the posture change, the silhouette image before the posture change of the one photographing means can be obtained. By obtaining the amount of movement of the silhouette image after the posture change with respect to the silhouette image before the posture change, the posture of the photographing means capable of changing the posture can be estimated with high accuracy without using the feature point of the fixed object in the observation environment. can do.

  Hereinafter, an imaging system according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of an imaging system according to an embodiment of the present invention.

  The imaging system shown in FIG. 1 includes N movable cameras 11 to 1n (N is an integer of 2 or more) and an image processing device 20. The image processing apparatus 20 includes a plurality of extraction units 21 to 2n, an initial calibration unit 31, a first epipolar line calculation unit 32, a second epipolar line calculation unit 33, a movement amount calculation unit 34, and a posture estimation unit 35.

  Note that the image processing device 20 may be configured by dedicated hardware circuits for each function shown in the drawing, and is not particularly limited to this example, and is not limited to ROM (Read Only Memory), CPU (Central Processing Unit), A normal computer including a RAM (Random Access Memory), an external storage device, an input device, an image interface device, a display device, and the like may be used. In this case, each of the above functions can be realized by executing an image processing program for executing a posture estimation process, which will be described later, on the CPU or the like. Moreover, when using software, you may make it perform one part or all part of each said function using one or several computers.

  The movable cameras 11 to 1n are composed of color cameras that can change the shooting angle in the pan direction and the tilt direction. For example, a pantilter integrated color video camera EVI-G20 manufactured by Sony Corporation can be used. The movable cameras 11 to 1n are fixed to a predetermined position in the observation space, for example, a ceiling or wall of a room or a corridor, and synchronously shoot a target object to be imaged and output image data to the extraction units 21 to 2n. To do. In addition, the movable cameras 11 to 1n are automatically controlled in posture in the pan direction and the tilt direction so as to automatically track an object by using a known object tracking method by an attitude control unit (not shown). Yes. The image captured by the movable camera is not particularly limited to a color image, and other images such as a black and white image may be used as long as a silhouette image can be extracted.

  The extraction units 21 to 2n are provided for the movable cameras 11 to 1n, respectively, and extract a silhouette image representing the contour of the object from the image data output from the connected movable cameras 11 to 1n for each frame. The data is output to the calibration unit 31, the first epipolar line calculation unit 32, and the movement amount calculation unit 34. In addition, the structure of the extraction parts 21-2n is not specifically limited to said example, When the several image data output from the several movable cameras 11-1n can be processed within one frame, by one extraction part Each image data may be processed sequentially.

  The initial calibration unit 31 acquires silhouette images of the movable cameras 11 to 1n from the extraction units 21 to 2n at the time of initial calibration, and uses these silhouette images to perform initial calibration processing (camera calibration processing) of all the movable cameras 11 to 1n. ). As this calibration process, various calibration processes can be used. N. Sinha, M .; Pollefeys and L. McMillan: “Camera network calibration from dynamic silhouettes”, Proc. of CVPR2004, Vol. 1, pp. 195-202 (2004) can be used.

  The initial calibration unit 31 calculates the relative relationship between the movable cameras 11 to 1n using the above-described initial calibration result, acquires the epipole and epipolar homography between the movable cameras 11 to 1n, and obtains between the movable cameras 11 to 1n. The epipole is output to the first epipolar line calculation unit 32, and the epipolar homography between the movable cameras 11 to 1n is output to the second epipolar line calculation unit 33.

  In addition, the method of acquiring the epipole and epipolar homography between the movable cameras 11 to 1n is not particularly limited to the above example, and the epipole and the epipolar homography between the movable cameras 11 to 1n calculated separately are directly acquired. Various changes are possible. Further, the acquisition timing of the epipole and epipolar homography is not particularly limited to the above example, and the epipole and epipolar homography may be acquired at an arbitrary time during observation by the movable cameras 11 to 1n.

  Further, the postures of the movable cameras 11 to 1n at the time of the initial calibration become reference postures of the movable cameras 11 to 1n in the posture estimation process described later, and the initial calibration unit 31 includes the movable cameras 11 to 11 at the time of initial calibration. The angle in the pan direction and the tilt direction of 1n is output to the posture estimation unit 35 as a reference posture. The initial calibration process is a process that needs to be performed only once when the movable cameras 11 to 1n are installed.

  The first epipolar line calculation unit 32 is between the movable cameras 11 to 1n and the movable cameras 11 to 1n output from the initial calibration unit 31 and the silhouette images of cameras other than the one camera that is the target of posture estimation. Epipolar lines of other cameras are calculated using the epipole and output to the second epipolar line calculating unit 33.

  The second epipolar line calculation unit 33 uses the epipolar line of another camera output from the first epipolar line calculation unit 32 and the epipolar homography between the movable cameras 11 to 1n output from the initial calibration unit 31. The epipolar line of one camera that is the target of posture estimation is calculated and output to the movement amount calculation unit 34.

  The movement amount calculation unit 34 outputs the silhouette image of one camera extracted by the extraction units 21 to 2n when the posture of the one camera changes, and the epipolar line of one camera output from the second epipolar line calculation unit 33 The movement amount of the silhouette image of one camera before and after the posture change is calculated and output to the posture estimation unit 35.

Specifically, the movement amount calculation unit 34 performs epipolar homography between one camera and another camera as H _kl , epipolar lines of other cameras as l _{lk, 1} , l _{lk, 2} , when the posture changes. The extracted silhouette image of one camera is S _k ⁽ⁱ⁾ , the transformation for moving the silhouette image S _k ⁽ⁱ⁾ on the image is P, and the silhouette image transformed by the transformation P is P (S _k ^{( i) When} the point on the silhouette image S that passes through a tangent parallel to the epipolar line l of one camera calculated by the second epipolar line calculation unit 33 is represented by p _{S, l} , the epipolar line l and the point When the distance from p _{S, l} is represented by D (S, l), a transformation P _i that minimizes the following equation (1) is obtained. The transformation P _i obtained here represents the amount of movement (including movement direction information) of the silhouette image due to the posture change of one camera.

Posture estimation unit 35 includes a reference posture that is output from the initial calibration unit 31, and a posture change before and after the movement amount of conversion P _i Namely silhouette image calculated by the movement amount calculating section 34, the posture i.e. pan of one camera Estimate the angle of the direction and tilt direction. The above processing is executed for all the movable cameras 11 to 1n, and the postures of all the movable cameras 11 to 1n are estimated.

  In the present embodiment, the movable cameras 11 to 1n correspond to an example of an imaging unit, the initial calibration unit 31 corresponds to an example of an acquisition unit, the extraction units 21 to 2n correspond to an example of an extraction unit, and the first epipolar unit. The line calculation unit 32 corresponds to an example of a first calculation unit, the second epipolar line calculation unit 33 corresponds to an example of a second calculation unit, the movement amount calculation unit 34 corresponds to an example of a movement unit, and the posture The estimation unit 35 corresponds to an example of an estimation unit.

  Next, posture estimation processing by the imaging system configured as described above will be described. FIG. 2 is a flowchart for explaining posture estimation processing by the imaging system shown in FIG.

  First, in step S1, the movable cameras 11 to 1n capture an object and output image data to the extraction units 21 to 2n. The extraction units 21 to 2n extract silhouette images from the image data, and the initial silhouettes are extracted. The image is output to the initial calibration unit 31 as an image.

  Here, the relationship between the silhouette shape (contour of the silhouette image) and the frontier point, which is the background of the posture estimation processing by the imaging system, will be described. In the following description, for ease of explanation, one of the movable cameras 11 to 1n is described as camera K, and the other camera as camera L.

FIG. 3 is a schematic diagram for explaining the relationship between the silhouette shape and the frontier point. As shown in FIG. 3, the frontier point is a point P _f on the object surface projected on the contours of both silhouette images obtained by the two cameras K and L. From FIG. 3, the frontier point P _f It can be seen that the epipolar lines at are obtained as silhouette-shaped tangents (each passing through each other's epipole).

_Further , the following relationship is established for the two epipolar lines l _kl and l _lk obtained from the two images I _k and I _l .

al _lk = H _kl l _kl (2)
Here, H _kl is an epipolar homography between the camera K and the camera L, a is a constant, and the epipolar homography is _obtained from the basic matrix F _kl and the epipoles e _kl and e _lk between the camera K and the camera L. Can be calculated. In this imaging system, it estimates the common observation point is by utilizing the property of the frontier point P _f postures of the cameras between two cameras K, L defined on the contour of the silhouette image.

  Next, in step S2, the initial calibration unit 31 acquires silhouette images of the movable cameras 11 to 1n from the extraction units 21 to 2n at the time of initial calibration, and uses all of the movable cameras 11 to 1n using the acquired silhouette images. Perform initial calibration. Using this initial calibration result, the initial calibration unit 31 calculates the relative relationship between the movable cameras 11 to 1n to calculate the epipole and epipolar homography between the movable cameras 11 to 1n, and between the movable cameras 11 to 1n. Are output to the first epipolar line calculation unit 32, and epipolar homography between the movable cameras 11 to 1 n is output to the second epipolar line calculation unit 33.

  Here, the above-described initial calibration processing will be described in detail using the cameras K and L as an example. In the initial calibration stage of each camera K, L, the position of each camera K, L is estimated based on the calibration method of the multi-viewpoint system using the silhouette image proposed by Sinha et al. Seeking a relationship.

  First, a tangent group having an inclination of 0 to 360 (deg) with respect to the X axis is calculated for silhouette images extracted from images obtained by the cameras K and L. When the contact coordinates at each angle are recorded as the feature amount of each silhouette image, for example, a silhouette image (white area in the figure) and a tangent group (white line in the figure) shown in FIG. 4 are obtained.

Next, two tangent lines are randomly selected for each camera K and L from the corresponding set of silhouette image tangents obtained by the two cameras K and L, and the intersection points of each are epipole. Assuming that Subsequently, a corresponding set of tangents is extracted from another silhouette image set of the cameras K and L using the assumed epipole. By using the three tangents and two epipoles obtained here, the basic matrix F _kl between the two cameras K and L can be calculated.

Next, the basic matrix F _kl obtained using the observation sequences other than the two observation sets used in the above processing is evaluated, and the evaluation is sufficiently high (the projection error is small for the observation sequences other than the two sets). The process is repeated until a thing is obtained, and the basic matrix F _kl is determined. Finally, the epipoles e _kl and e _lk and the epipolar homography H _kl between the cameras K and L are calculated from the calibration results of the cameras K and L calculated by the above processing.

  Next, in step S3, the movable cameras 11 to 1n change their postures while tracking the object. In addition, the tracking operation | movement of the movable cameras 11-1n is continuously performed in the observation period after an initial calibration process, and the attitude | position of at least 1 movable camera shall change.

  Next, in step S4, the movable cameras 11 to 1n photograph the object and output image data to the extraction units 21 to 2n. The extraction units 21 to 2n change the posture in the same manner as the initial calibration process. A silhouette image is extracted from the subsequent image data and output to the first epipolar line calculation unit 32 and the movement amount calculation unit 34.

  Next, in step S <b> 5, the first epipolar line calculation unit 32 outputs the silhouette images of cameras other than the one camera that is the target of posture estimation among the movable cameras 11 to 1 n and the initial calibration unit 31. Epipolar lines of other cameras are calculated using the epipole between the movable cameras 11 to 1n and output to the second epipolar line calculation unit 33.

  Next, in step S <b> 6, the second epipolar line calculation unit 33 is connected between the epipolar lines of other cameras output from the first epipolar line calculation unit 32 and the movable cameras 11 to 1n output from the initial calibration unit 31. Epipolar lines in the reference posture of one camera that is the target of posture estimation are calculated using epipolar homography and output to the movement amount calculation unit 34.

Next, in step S7, the movement amount calculation unit 34 obtains a transformation P _i that minimizes the above equation (1), and uses this transformation P _i as a movement amount of a silhouette image due to a posture change of one camera. To 35.

  Next, in step S8, the posture estimation unit 35 determines the posture of one camera, that is, the pan direction and the panning direction from the reference posture output from the initial calibration unit 31 and the movement amount of the silhouette image calculated by the movement amount calculation unit 34. The angle in the tilt direction is estimated, and then the process returns to step S3 and the subsequent processing is continued.

  Next, the processes in steps S4 to S8 will be described in detail using the cameras K and L as an example. First, silhouette images are extracted from the images obtained by the cameras K and L in the same manner as in the initial calibration process described above, and tangent groups each having an inclination of 0 to 360 (deg) with respect to the X axis of the image plane are calculated. The contact coordinates at each angle are recorded as the feature amount of each silhouette image.

  Next, the posture of each camera is estimated as follows using the obtained silhouette image. FIG. 5 is a schematic diagram for explaining the relationship between the object and the silhouette image, and FIG. 6 is a schematic diagram for explaining the relationship between the silhouette image and the epipolar geometry. In the following, in order to simplify the description, a case where only the posture of the camera K is changed will be considered.

As shown in FIG. 5, paying attention to the two cameras K and L, the postures of both the cameras K and L at the time of initial calibration are R _k ⁽⁰⁾ and R _l , and the posture of the camera K in the frame i is R _k ^{( i),} and let K ⁽⁰⁾ , L, K ⁽ⁱ⁾ be the cameras corresponding to each posture. Camera ^{K (0), K} the ⁽ⁱ⁾ posture variation between [Delta] R _k ⁽ⁱ⁾ is assumed to be small, the camera ^{K (0),} the silhouette image _S ^{k (0)} to be taken by ^{K (i),} _{S k} Regarding ⁽ⁱ⁾ , it can be assumed that the projection position and orientation change greatly, but the change in shape is minute.

Here, as shown in FIG. 6, assuming that silhouette images S _k ⁽ⁱ⁾ and S _l ⁽ⁱ⁾ are obtained on the cameras K ⁽ⁱ⁾ and L in the frame i, the camera K is subjected to the initial calibration process. ^Since the epipoles e _kl and e _lk between ⁽⁰⁾ and L are known, the epipolar line on the camera L regarding the frontier point obtained between the cameras K ⁽⁰⁾ and L is the tangent of the silhouette image S _l ⁽ⁱ⁾ It is obtained as _two tangents l _{lk, 1} , l _{lk, 2} passing through the epipole e _{lk of the} group.

On the other hand, for the camera K ⁽⁰⁾ , if the silhouette image S _k ⁽⁰⁾ is known, the epipolar line on the camera K ⁽⁰⁾ is the epipole e _{kl in} the tangent line group of the silhouette image S _k ^(0). However, in practice, the posture of the camera K changes in the frame i, and the silhouette image S _k ⁽⁰⁾ cannot be directly observed by the camera K ⁽⁰⁾ . .

Therefore, when attention is paid to the epipolar homography H _kl between the cameras K ⁽⁰⁾ and L, the epipolar lines l _{lk, 1} , l _{lk, 2} and the epipolar homography H _kl at the camera L are used in the camera K ⁽⁰⁾ . Epipolar lines l _{kl, 1} , l _{kl, 2} can be calculated. Further, when attention is paid to the silhouette image S _k ⁽ⁱ⁾ obtained by the camera K ⁽ⁱ⁾ , the difference in silhouette shape between the cameras K ⁽⁰⁾ and K ⁽ⁱ⁾ is assumed to be very small, and the camera K ⁽⁰⁾ The silhouette image S _k ⁽⁰⁾ obtained in the above is given as a translation and rotation of the silhouette image S _k ⁽ⁱ⁾ on the image.

Here, when the transformation P is defined and the silhouette image S _k transformed by the transformation P is represented by P (S _k ), the epipolar on the camera K ⁽⁰⁾ is obtained from the silhouette image P (S _k ) and the epipole e _kl. Lines l ′ _{kl, 1} , l ′ _{kl, 2} are determined. Similarly, since the epipolar lines l _{kl, 1} , 1 _{kl, 2} on the camera K ⁽⁰⁾ can be calculated from the epipolar homography H _kl and the epipolar lines l _{lk, 1} , 1 _{lk, 2} , the epipolar lines l _{kl, 1} , L _{kl, 2} and l ′ _{kl, 1} , l ′ _{kl, 2} are calculated to calculate the transformation P _i so that the silhouette projection position on the camera K ⁽⁰⁾ can be determined.

Actually, since the camera K shares observations with a plurality of other cameras including the camera L, it pays attention only to the shape of the silhouette image obtained by each camera, and it is initial for the camera group of the reference posture. The silhouette image S _k ⁽⁰⁾ obtained by the camera K ⁽⁰⁾ can be estimated by searching for the projection position and posture of each silhouette that best satisfies the relative relationship between the cameras at the time of calibration. This search is calculated by finding the transformation P _i that minimizes equation (1) above.

FIG. 7 is a schematic diagram for explaining the distance between the silhouette image and the epipolar line. In the above equation (1), D (S, l) is the distance between the straight line 1 and the point p _{S, l} shown in FIG. 7, and the point p _{S, l} has a tangent line parallel to the straight line l on the silhouette image S. It is a passing point. Therefore, the silhouette image S is a silhouette image converted by the conversion P _i, that is, the silhouette image P (S _k ⁽ⁱ⁾ ) after movement, and the straight line l is an epipolar line l _{kl, 1} , 1 on the camera K ^(0). _{kl, 2,} ie, H _kl l _{lk, 1} and H _kl l _{lk, 2} , D (S, l) is moved with the epipolar line l _{kl, 1} , l _{kl, 2} on camera K ⁽⁰⁾ Represents the distance from the silhouette image P (S _k ⁽ⁱ⁾ ).

Since the transformation P _i obtained in this manner represents the amount of movement of the silhouette image due to the posture change between the cameras K ⁽⁰⁾ and K ⁽ⁱ⁾ , the transformation K _i to the cameras K ⁽⁰⁾ and K ⁽ⁱ⁾ A posture change amount ΔR _k ⁽ⁱ⁾ can be calculated, and the posture of the camera K for each frame, that is, the angle in the pan direction and the tilt direction can be calculated.

  In the above description, the case where there is one camera for which the posture is to be estimated has been described. However, the same applies to the case where the postures of a plurality of cameras are estimated. Furthermore, the postures of all the cameras that are subject to posture estimation can be estimated by executing the processing of each step in parallel or by repeating the processing of steps S4 to S8 in sequence. .

With the above processing, in the present embodiment, the epipole and epipolar homography between the movable cameras 11 to 1n at the time of initial calibration are calculated, and the posture of the camera that is the posture estimation target among the movable cameras 11 to 1n is changed. A silhouette image is extracted from the camera captured by the posture estimation target and other cameras, and the other silhouette image of the other camera extracted and the epipole between the camera of the posture estimation target and the other camera are extracted. The epipolar line of the other camera is calculated, and the epipolar line of the target camera is calculated from the calculated epipolar line of the other camera and the epipolar homography between the target camera and the other camera, Posture so that the distance between the calculated epipolar line of the target camera and the silhouette image after movement is minimized Converting P _i for moving silhouette image is obtained after reduction, the posture i.e. the angle in the pan direction and the tilt direction of the transformation P _i from the pose estimation target camera is calculated.

  Therefore, the postures of the movable cameras 11 to 1n whose posture can be changed can be estimated with high speed and high accuracy without using the feature points of the fixed object in the observation environment. As a result, even when the postures of the movable cameras 11 to 1n change, it is possible to execute camera calibration processing that follows the posture changes using the silhouette images obtained by the movable cameras 11 to 1n.

  Next, attitude estimation accuracy of the above-described imaging system will be described. As the imaging system shown in FIG. 1, four movable cameras were used, and only one of the four movable cameras was changed in posture, and the other three movable cameras were used as fixed cameras. FIG. 8 is a diagram illustrating an environment in which photographing is performed, and a self-propelled toy is arranged as an object in a blue background without arranging any fixed objects or the like in the observation environment.

  In the above observation environment, first, the postures of the four movable cameras were fixed, the initial calibration process was performed, and the epipole and epipolar homography between the movable cameras were calculated. Next, the self-propelled toy moving in the scene was observed with four movable cameras while changing the posture of only one movable camera. At this time, synchronous observation was performed with four movable cameras, and an image sequence of 130 frames was obtained in an observation period of about 20 seconds.

  FIG. 9 is a diagram illustrating an example of an observed image sequence. FIG. 9 illustrates an image sequence photographed by one movable camera whose posture is changed by the posture change camera in the drawing, and the fixed cameras 1 to 3 are changed in posture. The image sequence image | photographed with three movable cameras which are not made to show is shown. The posture of the posture change camera was controlled by a computer, and the posture of the posture change camera was estimated using four images obtained in each frame.

  10 is a diagram illustrating an example of a silhouette projection position and posture estimation result in the reference posture of the posture change camera, FIG. 11 is a diagram illustrating an example of the posture estimation result, and the horizontal axis in FIG. 11 represents the number of frames. The vertical axis represents the attitude value, the solid line represents the estimated attitude value by the present observation system, and the broken line represents the attitude control value of the attitude change camera given by the computer. The upper left image of FIG. 10 shows the silhouette projection position and posture estimation results of the posture change camera, and the other three images show the silhouette projection positions and posture estimation results of the fixed cameras 1 to 3. Further, as shown in FIG. 11, the estimation error is an average of 0.2 degrees, and it has been found that the posture of the posture change camera can be accurately estimated.

  In the above description, the process up to the estimation of the posture of the movable camera has been described. However, the three-dimensional shape information of the object may be restored using the estimated posture. In this case, a known visual volume intersection method or the like can be used as the three-dimensional shape information restoration processing. For example, a multi-viewpoint image using the visual volume intersection method and the stereo matching method described in Non-Patent Document 1 above. 3D moving object generation method from "Parallel view volume intersection method using inter-plane perspective projection" (U et al., 3 people, Journal of Information Processing Society of Japan: Computer and Image Media, 2001, Vol. 42, SIG ( CVIM2), p.33-p.43), etc. can be used.

  In this embodiment, the attitude of the movable camera is explicitly estimated. However, the present invention is not particularly limited to this example, and the parameters corresponding to the attitude of the movable camera can be set without explicitly estimating the attitude of the movable camera. It may be estimated and the three-dimensional shape information of the object may be restored using this parameter. In this embodiment, the angles of the pan direction and the tilt direction of the camera are estimated. However, the present invention is not particularly limited to this example, and the present invention can estimate the posture of the camera, that is, the angles of all directions.

1 is a block diagram illustrating a configuration of an imaging system according to an embodiment of the present invention. 3 is a flowchart for explaining posture estimation processing by the imaging system shown in FIG. 1. It is a schematic diagram for demonstrating the relationship between a silhouette shape and a frontier point. It is a figure which shows an example of the relationship between a silhouette image and a tangent group. It is a schematic diagram for demonstrating the relationship between a target object and a silhouette image. It is a schematic diagram for demonstrating the relationship between a silhouette image and epipolar geometry. It is a schematic diagram for demonstrating the distance of a silhouette image and an epipolar line. It is a figure which shows the environment which image | photographed. It is a figure which shows an example of an observation image sequence. It is a figure which shows an example of the estimation result of the silhouette projection position and attitude | position in the reference | standard attitude | position of an attitude | position change camera. It is a figure which shows an example of an attitude | position estimation result.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11-1n Movable camera 20 Image processing apparatus 21-2n Extraction part 31 Initial calibration part 32 1st epipolar line calculation part 33 2nd epipolar line calculation part 34 Movement amount calculation part 35 Posture estimation part

Claims (6)

A plurality of photographing means configured to photograph at least one photographing means and an object;
An acquisition means for acquiring epipole and epipolar homography between the one imaging means and the other imaging means at a predetermined time;
Extracting means for extracting a silhouette image representing a silhouette of an object from images photographed by the one photographing means and the other photographing means when the posture of the one photographing means is changed;
Based on the silhouette image of the other photographing means extracted by the extracting means and the epipole between the one photographing means and the other photographing means obtained by the obtaining means. First calculating means for calculating an epipolar line;
Based on the epipolar line of the other imaging means calculated by the first calculating means, and the epipolar homography between the one imaging means and the other imaging means acquired by the acquisition means, Second calculating means for calculating an epipolar line of the one photographing means;
And a moving means for moving the silhouette image of the one photographing means extracted by the extracting means so as to match the epipolar line of the one photographing means calculated by the second calculating means. Shooting system to do. The moving means moves the silhouette image of the one photographing means extracted by the extracting means so as to match the epipolar line of the one photographing means calculated by the second calculating means. Calculate the amount of movement of the silhouette image of one shooting means,
The imaging system according to claim 1, further comprising an estimation unit that estimates an attitude of the one imaging unit based on a movement amount calculated by the movement unit. The moving means includes H _kl for epipolar homography between the one photographing means and the other photographing means, and l, _{k, 1} , l _{lk, 2} for the epipolar lines of the other photographing means and the extracting means. The extracted silhouette image of the one photographing means is S _k ⁽ⁱ⁾ , the transformation for moving the silhouette image S _k ⁽ⁱ⁾ on the image is P, and the silhouette image transformed by the transformation P is P When the point on the silhouette image S, which is represented by (S _k ⁽ⁱ⁾ ) and passes through a tangent parallel to the epipolar line l of the one photographing unit calculated by the second calculating unit, is represented by p _{S, l.} The transformation P _i that minimizes the following formula (1) is obtained when the distance between the epipolar line l and the point p _{S, l} is represented by D (S, l). Shooting system.
  The imaging system according to claim 1, wherein the acquisition unit acquires epipole and epipolar homography between one imaging unit and another imaging unit at the time of initial calibration. An imaging system is used in which at least one imaging unit is configured to be capable of changing the posture and includes a plurality of imaging units for imaging an object, an acquisition unit, an extraction unit, a first calculation unit, a second calculation unit, and a movement unit. A shooting method,
The acquisition means acquires epipole and epipolar homography between the one imaging means and another imaging means at a predetermined time; and
The extraction means extracting a silhouette image representing a silhouette of an object from images photographed by the one photographing means and the other photographing means when the posture of the one photographing means is changed;
The first calculating unit is configured to detect the other image capturing unit based on the extracted silhouette image of the other image capturing unit and the acquired epipole between the one image capturing unit and the other image capturing unit. Determining an epipolar line;
The second calculation unit is configured to determine the one of the one imaging unit based on the determined epipolar line of the other imaging unit and the acquired epipolar homography between the one imaging unit and the other imaging unit. Calculating an epipolar line of the imaging means;
The moving means includes a step of moving the silhouette image of the one photographing means extracted by the extracting means so as to match the calculated epipolar line of the one photographing means. . An image processing program for processing an image captured by a plurality of imaging means configured to change an attitude of at least one imaging means and image an object,
An acquisition means for acquiring epipole and epipolar homography between the one imaging means and the other imaging means at a predetermined time;
Extracting means for extracting a silhouette image representing a silhouette of an object from images photographed by the one photographing means and the other photographing means when the posture of the one photographing means is changed;
Based on the silhouette image of the other photographing means extracted by the extracting means and the epipole between the one photographing means and the other photographing means obtained by the obtaining means. First calculating means for calculating an epipolar line;
Based on the epipolar line of the other imaging means calculated by the first calculating means, and the epipolar homography between the one imaging means and the other imaging means acquired by the acquisition means, Second calculating means for calculating an epipolar line of the one photographing means;
Causing the computer to function as a moving unit that moves the silhouette image of the one photographing unit extracted by the extracting unit so as to match the epipolar line of the one photographing unit calculated by the second calculating unit. A characteristic image processing program.