JP5672112B2 - Stereo image calibration method, stereo image calibration apparatus, and computer program for stereo image calibration - Google Patents

Description

  The present invention relates to a stereo image calibration method, a stereo image calibration apparatus, and a stereo image calibration computer program that calibrate at least one of two images obtained by photographing the same object from different directions for a stereo image.

  Conventionally, research for reproducing a three-dimensional image has been made. As one method for reproducing a three-dimensional image, two images taken from different directions with respect to the same object are displayed side by side, and each of the two images is displayed on the left and right eyes of the observer. The method of showing is known. A set of two images used in such a method is called a stereo image.

  Since the two images included in the stereo image are observed by the left and right eyes of the observer, in order to reproduce a high-quality three-dimensional image, the images projected on the two images are It is preferable that the image is taken under the same conditions as the conditions under which an observer generally sees an object. However, a camera that captures an image for the left eye and a camera that captures an image for the right eye may not be appropriately arranged. As a result, the images in the two images are shifted from their proper positions in the vertical or horizontal direction, or the image in one image is converted into the image in the other image. On the other hand, it may rotate on the image. In such a case, in order to generate a stereo image capable of reproducing a good three-dimensional image, a calibration process for obtaining a calibration parameter for correcting the position of the image shown in at least one of the images is performed.

  Conventionally, in the calibration process, the position of the image for the left eye or the position of the image for the right eye is adjusted while the user sees with the eyes, and this operation is very troublesome for the user. It was. In particular, when the user is a beginner, it is not easy for the user to appropriately perform the calibration process.

On the other hand, a technique for automatically aligning the positions of the images on the two images based on the images of the two images has been proposed (see, for example, Patent Documents 1 to 3). For example, in Patent Document 1, a difference in display position of a specific target in each captured image is adjusted based on a difference vector between representative coordinates of detection marks of a specific target in a first captured image and a second captured image. Thus, a technique for adjusting the parallax amount of the specific target is disclosed.
Further, in Patent Document 1, instead of a detection mark, a specific target part, for example, an eye or a mouth that is a part of a face is detected from each captured image, and the display position of each part on the image is detected. It is also disclosed that the amount of parallax is adjusted based on the difference between the two.

  Furthermore, in Patent Document 2, in order to adjust the deviation of the optical axes of the two imaging units, a face is detected from each image captured by each imaging unit, and the amount of deviation of the position of the face between the two images A technique for calculating the correction angle of the optical axis based on the above has been proposed. Further, in Patent Document 3, in order to adjust the shift of the optical axis of the stereo camera, one of the images from the two cameras is based on each of a far region and a near region whose distances are known in advance. There has been proposed a technique for obtaining the other translational correction amount and rotation angle with respect to.

JP 2010-147940 A JP 2008-252254 A JP-A-11-259632

However, in the technique disclosed in Patent Document 1 or 2, a correction amount for aligning the image is obtained only from a point on the subject such as a face that is relatively close to the camera. Therefore, in these techniques, proper alignment may not be performed with respect to the entire image, and as a result, a three-dimensional reproduced image of an object located far away on the image may be unnatural.
Further, in the technique disclosed in Patent Document 3, since a plurality of points whose distance from the camera is known in advance are used for optical axis adjustment, when there is no known point from the camera, Technology is not applicable.

  Therefore, an object of the present specification is to provide a stereo image calibration method capable of calculating a calibration parameter for correcting the position of the entire image shown in the image by each camera so as to be appropriate as a stereo image.

  According to one embodiment, a stereo image calibration method is provided. In this stereo image calibration method, a first image generated by shooting a region including a subject with a first camera and a second image generated by shooting the region with a second camera are used. A first subject area in which the subject is captured from the first image, and a second subject area in which the subject is photographed is detected from the second image, and the first subject area and the first subject area are detected. At least one set of subject feature points corresponding to the same point on the subject is extracted from the two subject areas, and a first parallel to the image plane is extracted based on the at least one set of subject feature points. The difference between the rotation angle of the first camera and the rotation angle of the second camera around each of the axis and the second axis orthogonal to the first axis is obtained, and the difference between the rotation angle of the second camera and the subject on the first image is determined. Correct the difference between the position and the position of the subject on the second image The calibration parameter of 1 is obtained, and the object reflected in the first background area obtained by removing the first subject area from the first image and the second background area obtained by removing the second subject area from the second image. From the image, a second calibration parameter for correcting a rotation between the first image and the second image due to a difference between a rotation angle around the optical axis of the first camera and a rotation angle around the optical axis of the second camera is obtained. Including seeking.

The objects and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims.
It should be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention as claimed.

  The stereo image calibration method disclosed herein can calculate calibration parameters for correcting the position of the entire image shown in the image from each camera so as to be appropriate as a stereo image.

It is a schematic block diagram of the stereo image calibration apparatus by one embodiment. It is a functional block diagram of a processing part. (A)-(c) is a schematic diagram showing the extraction procedure of the group of the face feature point by this embodiment. It is a figure showing the coordinate system on real space defined about each camera. (A) is a schematic diagram showing an example of a set of face feature points in the left image and the right image. (B) is a schematic diagram showing the relationship between the difference in the vertical direction from the image center to the face feature point and the rotation component from the horizontal plane. (C) is a schematic diagram showing the relationship between the difference in the horizontal direction from the image center to the face feature point and the rotation component from the vertical plane. (A) is a figure explaining the positional relationship of the some pixel used for calculation of local gradient, (b) is a schematic diagram of distribution of orientation of the local gradient calculated | required from the background area | region of a left image and a right image It is. It is an operation | movement flowchart of a stereo image calibration process. (A) is a schematic diagram showing the range concealed by the head in a right image among the background area | regions reflected on the left image. Moreover, (b) is a schematic diagram showing the range concealed by the head in the left image in the background region shown on the right image. (A) And (b) is a schematic diagram showing the area | region excluded from a background area | region in a left image and a right image, respectively regarding extraction of a background feature point, respectively. (A) And (b) is a schematic diagram of another example showing the area | region excluded from a background area | region in the left image and the right image regarding extraction of a background feature point, respectively. It is a schematic diagram showing the relationship between the imaging range of the camera which produces | generates a left image, and the imaging range of the camera which produces | generates a right image. The camera when the distance from the two cameras to the plane where the range shown in the left image and the range shown in the right image are the same is larger than the distance from the two cameras to the farthest object in the left and right images It is a schematic diagram of the imaging range. A camera in which the distance from the two cameras to the plane where the range shown in the left image and the range shown in the right image are the same is less than the distance from the two cameras to the farthest object in the left and right images It is a schematic diagram of the imaging range. (A) And (b) is a schematic diagram of the further another example showing the area | region excluded from a background area | region in the left image and the right image regarding extraction of a background feature point, respectively.

  A stereo image calibration apparatus according to one embodiment or a modification thereof will be described with reference to the drawings. This stereo image calibration device detects a set of feature points corresponding to the same point on the subject based on information on the subject located relatively near the camera from the images generated by the two cameras. . Then, the stereo image calibration apparatus corrects the displacement of the subject position caused by the difference in the rotation angle around the horizontal axis parallel to the image plane between the cameras and the difference in the rotation angle around the vertical axis parallel to the image plane. A calibration parameter is obtained based on the amount of deviation between the feature points. Further, the stereo image calibration device detects feature points corresponding to the same object from the regions on the image where the background outside the subject is reflected, and based on these feature points, the rotation angle around the optical axis of the camera. A calibration parameter for correcting the image rotation due to the difference is obtained. Thereby, even if the distance from each camera to the subject is unknown, this stereo image calibration apparatus can calculate a calibration parameter capable of aligning the entire image shown in the image from each camera for a stereo image.

  In this embodiment, the stereo image calibration device is applied to a videophone system that displays a three-dimensional face image, and the camera included in the stereo image calibration device captures the head of a person who is going to talk as a subject. And However, this stereo image calibration apparatus can be applied to various apparatuses that generate stereo images other than the videophone system.

  FIG. 1 is a schematic configuration diagram of a stereo image calibration apparatus according to an embodiment. As shown in FIG. 1, the stereo image calibration apparatus 1 includes two cameras 2-1 and 2-2, an input unit 3, a storage unit 4, and a processing unit 5. Furthermore, the stereo image calibration apparatus 1 transmits a corrected image obtained by correcting the images generated by the cameras 2-1 and 2-2 using calibration parameters described later to other devices via a communication network. An interface circuit (not shown) for outputting may be included.

The camera 2-1 is a camera that generates an image for the left eye, and the camera 2-2 is a camera that generates an image for the right eye. For this purpose, the cameras 2-1 and 2-2 have an image sensor having an array of solid-state imaging elements arranged two-dimensionally, and an imaging optical system that forms an image of a subject on the image sensor. .
The cameras 2-1 and 2-2 capture the area including the same subject, and generate a stereo image for reproducing a three-dimensional image of the subject and the background. It is preferable that the optical axes are substantially parallel and arranged at a predetermined interval in a substantially horizontal direction. For this purpose, the cameras 2-1 and 2-2 may be accommodated in a predetermined position in one casing (not shown), for example. However, as will be described later, a calibration parameter for correcting the position of an image shown in at least one image is calculated by a calibration process executed by the processing unit 5. Therefore, the arrangement of the cameras 2-1 and 2-2 does not have to be adjusted so precisely that the images generated by these cameras become stereo images as they are. In addition, the size of the subject image on the image generated by the camera 2-1 is substantially equal to the size of the same subject image on the image generated by the camera 2-2. The focal length of the imaging optical system 2-2 and the number of pixels of the image sensor are the same. However, the focal lengths of the imaging optical systems of the cameras 2-1 and 2-2 and the number of pixels of the image sensor may be different from each other.
Each of the cameras 2-1 and 2-2 transmits the generated image to the input unit 3 each time an image is generated. In the following, for the sake of convenience, the image for the left eye generated by the camera 2-1 is referred to as a left image, and the image for the right eye generated by the camera 2-2 is referred to as a right image.

The input unit 3 receives the left image and the right image from the cameras 2-1 and 2-2, and delivers each image to the processing unit 5. For this purpose, the input unit 3 is an interface circuit in accordance with a serial bus standard such as a universal serial bus (USB) for connecting the cameras 2-1 and 2-2 to the processing unit 5, for example. Or a video interface circuit.
The input unit 3 passes the acquired left image and right image to the processing unit 5.

  The storage unit 4 includes, for example, a readable / writable volatile or nonvolatile semiconductor memory circuit, or a magnetic recording medium or an optical recording medium. The storage unit 4 stores the left image and the right image received from the input unit 3. Moreover, when each function which the process part 5 of the stereo image calibration apparatus 1 has is implement | achieved by the computer program run on the processor which the process part 5 has, you may memorize | store the computer program. Furthermore, the storage unit 4 may store calibration parameters.

  The processing unit 5 includes at least one processor and its peripheral circuits. Then, the processing unit 5 obtains a calibration parameter for correcting the position of the image on at least one of the images in order to generate a stereo image from the left image and the right image.

  FIG. 2 shows a functional block diagram of the processing unit 5. As illustrated in FIG. 2, the processing unit 5 includes a face detection unit 11, a face feature point extraction unit 12, a background feature point extraction unit 13, a calibration parameter calculation unit 14, and a determination unit 15. Each of these units included in the processing unit 5 is implemented as a functional module realized by a computer program executed on a processor included in the processing unit 5, for example. Alternatively, each of these units included in the processing unit 5 may be mounted on the stereo image calibration device 1 as a separate arithmetic circuit, or the stereo image calibration device 1 as one integrated circuit that realizes the function of each unit. May be implemented.

The face detection unit 11 is an example of a subject region detection unit, and extracts a region where a face is shown from each of the left image and the right image. Hereinafter, for convenience, an area in which a face is shown is referred to as a face area. The face area is an example of a subject area.
For example, the face detection unit 11 detects a face area from each of the left image and the right image using a discriminator based on machine learning. Such a discriminator can be, for example, an Adaboost discriminator, a multilayer perceptron or a support vector machine. These classifiers detect facial regions on images by supervised learning using multiple images whose faces are known in advance and multiple images whose faces are known not to be captured. To be learned. For example, when an Adaboost classifier is used as the classifier, the face detection unit 11 divides each of the right image and the left image into a plurality of small areas, and sequentially inputs the small areas to the classifier. For example, the classifier detects a Haar-like feature useful for detecting a face region from a small region, and determines whether the small region is a face region based on the Haar-like feature.
Similarly, when another classifier such as a multilayer perceptron or a support vector machine is used, the face detection unit 11 identifies a small area obtained by dividing the right image and the left image, or various feature amounts extracted from the small area. It is determined whether or not the small area is a face area.

  The face detection unit 11 may detect the face area according to any of various methods for detecting a face area other than the classifier. For example, the face detection unit 11 detects pixels having a color component corresponding to the skin color from the right image and the left image, and obtains a region where the pixels are gathered as a face candidate region by labeling processing. When the color component of a pixel is represented in the HSV color system, if the H component has a value included in the range of about 0 to about 30, the pixel has a color component corresponding to the skin color. be able to. The face detection unit 11 includes, for example, the size of the face candidate area included in the range of general face sizes on the right image and the left image, and the circularity of the face candidate area is The face candidate area may be set as a face area when the threshold value is equal to or greater than a predetermined threshold corresponding to the contour of a simple face. The face detection unit 11 obtains the total number of pixels located on the contour of the face candidate area as the perimeter of the face candidate area, and multiplies the total number of pixels in the face candidate area by 4π as the square of the perimeter. The circularity can be calculated by dividing.

  Alternatively, the face detection unit 11 may approximate the face candidate region to an ellipse by applying the least square method by applying the coordinates of each pixel on the contour of the face candidate region to the elliptic equation. The face detection unit 11 may use the face candidate area as a face area when the ratio of the major axis to the minor axis of the ellipse is included in the range of the ratio of the major axis to the minor axis of the face. Note that, when the shape of the face candidate region is evaluated by ellipse approximation, the face detection unit 11 may detect an edge pixel corresponding to an edge by performing an inter-pixel calculation on the luminance component of each pixel of the image. . In this case, the face detection unit 11 connects the edge pixels using, for example, a labeling process, and uses the edge pixels connected to a certain length or more as the contour of the face candidate region.

  Alternatively, the face detection unit 11 performs template matching between the face candidate region and a template corresponding to a general face shape, calculates a normalized cross-correlation value between the face candidate region and the template, When the normalized cross correlation value is equal to or greater than a predetermined value, the face candidate area may be determined as the face area.

In addition, there may be a plurality of people within the shooting range of the cameras 2-1 and 2-2. In such a case, a plurality of face regions are detected from the left image or the right image. Therefore, when detecting a plurality of face areas from the left image, the face detection unit 11 focuses on the face of one person for calculating the calibration parameters. For example, the face detection section 11 is the face area closest to the center of the left image or the largest. Select one face area. Similarly, when detecting a plurality of face areas from the right image, the face detection unit 11 selects one face area closest to the center of the right image or the largest face area.
When the stereo image calibration apparatus has a plurality of microphones arranged at different positions, the processing unit 5 estimates the direction of the person who spoke out of the plurality of persons from the difference in time of the sound reaching each microphone, The face detection unit 11 may select the face area closest to the estimated direction.

When detecting the face area, the face detection unit 11 generates information representing the face area and a background area that is an area obtained by removing the face area from the image for each of the left image and the right image. For example, the information can be a binary image having the same size as the right image or the left image and having a pixel value in which the pixels in the face area and the pixels in the background area are different. Note that, when a plurality of face areas are detected from one image as described above, the face detection unit 11 may include a face area other than the selected one in the background area.
Then, the face detection unit 11 passes information representing the face region and the background region to the face feature point extraction unit 12 and the background feature point extraction unit 13.

  The face feature point extraction unit 12 is an example of a subject feature amount extraction unit, and at least a set of face feature points corresponding to the same point of the face from each of the face region on the left image and the face region on the right image. Extract one. In the present embodiment, the face feature point extraction unit 12 extracts a first candidate point that is a candidate for a face feature point from one face area of the left image and the right image, and the face feature point is extracted from the face area of the other image. A second candidate point that matches the candidate point is searched. Then, the facial feature point extraction unit 12 finds a third candidate point that matches the second candidate point detected in the other image on the image from which the first candidate point has been extracted. When the first candidate point and the third candidate point can be regarded as substantially the same, the face feature point extraction unit 12 determines the first candidate point or the third candidate point and the second candidate point as the face A set of feature points. Thereby, the face feature amount extraction unit 12 can accurately extract the face feature points on the left image and the face feature points on the right image corresponding to the same part of the face.

  First, in order to extract the first candidate point from the face area of one image, the face feature point extraction unit 12 is detected by applying a corner detector to the face area on the left image, for example. A plurality of points are set as first candidate points. The face feature point extraction unit 12 can use, for example, a Harris detector as such a corner detector. The face feature point extraction unit 12 may use a detector that detects a characteristic point other than the corner detector in order to extract the first candidate point from the face region. As such a detector, for example, the face feature point extraction unit 12 may use a Scale-invariant feature transform (SIFT) detector.

  At that time, the face feature point extraction unit 12 can predict a range in which a characteristic part such as an eye, a nose, or a mouth may exist in the face region, so that the characteristic part may exist. The first candidate point may be extracted by applying the detector as described above within a certain range. Alternatively, the face feature point extraction unit 12 reduces the size of the region input to the detector in order to determine whether or not to extract the target pixel as the first candidate point as the size of the face region is smaller. Also good. Furthermore, the face feature point extraction unit 12 obtains a normalized cross-correlation value by performing template matching while changing the relative position between the template representing the characteristic part of the face and the face area of the left image. Also good. Then, the face feature point extraction unit 12 may extract any pixel in the face region that overlaps the template when the normalized cross-correlation value is the highest as the first candidate point.

  Next, the face feature point extraction unit 12 sets, for each first candidate point extracted from the left image, a predetermined area centered on the first candidate point as a template. Then, the face feature point extraction unit 12 performs template matching while changing the relative position between the template and the face area on the right image, and obtains, for example, a normalized cross-correlation value. Then, the facial feature point extraction unit 12 obtains, as a second candidate point, a pixel on the right image corresponding to the center of the template when the normalized cross-correlation value is maximized, that is, the template matches most.

Similarly, the face feature point extraction unit 12 sets a predetermined region centered on the second candidate point as a re-search template. Then, the face feature point extraction unit 12 performs template matching while changing the relative position between the re-search template and the face area on the left image, and obtains a normalized cross-correlation value, for example. Then, the face feature point extraction unit 12 determines the pixel on the left image corresponding to the center of the re-search template when the normalized cross-correlation value is maximum, that is, the best match with the re-search template, as the third candidate. Find as a point. The face feature point extraction unit 12 obtains a distance between the third candidate point and the original first candidate point, and if it is equal to or smaller than a predetermined distance threshold, the first candidate point or the third candidate on the left image Let the point and the second candidate point on the right image be a set of face feature points corresponding to the same part. It is preferable that the predetermined area has such a size that the degree of coincidence of the structures of the faces around the candidate points can be examined and that the influence of the difference in the photographing direction can be reduced. For example, the predetermined area can be a rectangular area having a length of about 1/10 to 1/4 of the horizontal width of the face area in both the horizontal and vertical directions. Further, the distance threshold is set to, for example, the maximum distance that can be regarded as a feature point corresponding to a part having the same distance.
On the other hand, when the maximum value of the normalized cross-correlation value between the template and the face area on the right image is less than a predetermined threshold, the face feature point extraction unit 12 determines the first candidate point corresponding to the template and If the matching second candidate point does not exist in the right image, the first candidate point may be excluded from the search target of the set of face feature points. Similarly, the face feature point extraction unit 12 responds to the re-search template even when the maximum normalized cross-correlation value between the re-search template and the face area on the left image is less than a predetermined threshold. The first candidate point and the second candidate point may be excluded from the search target of the set of face feature points. The higher the predetermined threshold value is set, the more the face feature point extraction unit 12 can improve the certainty that the set of face feature points corresponds to the same part. For example, the predetermined threshold is set to 0.9 to 0.95. Alternatively, the face feature point extraction unit 12 may increase the predetermined threshold as the number of first candidate points extracted from the left image increases. Thereby, the face feature point extraction unit 12 can extract only a set of face feature points that are highly likely to correspond to the same part when the number of candidate points extracted from the face area of one image is large. . Further, even if the number of face feature points extracted from the face area of one image is small, the face feature point extraction unit 12 can extract a sufficient number of sets of face feature points for obtaining calibration parameters.

FIG. 3A to FIG. 3C are schematic views showing a procedure for extracting a set of face feature points according to the present embodiment. In FIG. 3A, a plurality of first candidate points 301 are extracted in the face area of the left image 300. At this time, no processing is performed on the right image 310.
Next, as illustrated in FIG. 3B, a template 302 centered on a target candidate point 301 a among the plurality of first candidate points extracted from the left image 300 is set. In the right image 310, template matching with the template 302 is performed, and second candidate points 311 are extracted.
Thereafter, as shown in FIG. 3C, a re-search template 312 centering on the second candidate point 311 is set based on the right image 310. In the left image 300, the re-search template 312 and Template matching with the face area is performed. As a result, the third candidate point 303 is extracted, and if the distance between the third candidate point 303 and the first candidate point 301a is equal to or smaller than the distance threshold, the first candidate point 301a (or the third candidate) The point 303) and the second candidate point 311 are a set of face feature points.

The face feature point extraction unit 12 first extracts a first candidate point from the face area of the right image, and searches for a second candidate point corresponding to the first candidate point in the face area of the left image. Also good.
The face feature point extraction unit 12 stores the horizontal coordinate value and the vertical coordinate value on the image of the two face feature points in the storage unit 4 for each set of obtained face feature points.

The background feature point extraction unit 13 extracts at least one set of background feature points corresponding to the same point on the same object in the background region from the background region of the left image and the right image.
Therefore, similarly to the face feature point extraction unit 12, the background feature point extraction unit 13 extracts a plurality of first candidate points that are candidates for background feature points from one of the background regions of the left image and the right image. The background feature point extraction unit 13 performs template matching for each first candidate point while changing the relative position between the template centered on the candidate point and the background region of the other image. A second candidate point that most closely matches the first candidate point is obtained. Then, the background feature point extraction unit 13 performs template matching between the re-search template centered on the second candidate point and the background region of the image from which the first candidate point is extracted, thereby obtaining the second candidate point. The third candidate point that most closely matches is obtained. And if the distance between the 3rd candidate point and the 1st candidate point is below a distance threshold, background feature-value extraction part 13 will be the 1st candidate point or 3rd candidate point in one picture, and the other The second candidate point in the image is extracted as a set of background feature points.
The background feature point extraction unit 13 stores the horizontal coordinate value and the vertical coordinate value on the image of the two background feature points in the storage unit 4 for each set of obtained background feature points.

  The calibration parameter calculation unit 14 calculates a calibration parameter for correcting the position of the image shown on the image for at least one of the left image and the right image. In the present embodiment, the calibration parameter calculation unit 14 calculates the rotation angle of the camera 2-1 around the horizontal axis parallel to the image plane and the rotation angle of the camera 2-2 from a set of facial feature points corresponding to the same part. A horizontal calibration parameter for correcting the positional deviation of the image of the subject on the image due to the difference between the two is obtained. Similarly, the calibration parameter calculation unit 14 calculates the difference between the rotation angle of the camera 2-1 around the vertical axis parallel to the image plane and the rotation angle of the camera 2-2 from the set of face feature points corresponding to the same part. The vertical calibration parameter for correcting the image positional deviation due to the above is obtained. Further, the calibration parameter calculation unit 14 determines an image on the right image based on the difference between the rotation angle of the camera 2-1 and the rotation angle of the camera 2-2 around the optical axis from a set of background feature points corresponding to the same object. And a rotation direction calibration parameter for correcting the rotation between the images on the left image.

Here, for the convenience of expressing the calibration parameters, a coordinate system in the real space defined for the cameras 2-1 and 2-2 is defined. FIG. 4 is a schematic diagram showing such a coordinate system. The coordinate system 401 is a coordinate system defined for the camera 2-1 that generates the left image. The x-axis represents the horizontal direction parallel to the image plane of the camera 2-1, and the y-axis represents the vertical direction. The z-axis is the optical axis of the imaging optical system included in the camera 2-1. The x-axis is positive toward the face 410 of the person who is the subject, the y-axis is positive upward, and the z-axis is positive toward the face 410. The center (X _C , Y _C ) of the left image 402 corresponds to the origin of the coordinate system 401. Therefore, the horizontal line of the vertical coordinate Y _C on the left image corresponds to the x axis in real space. Further, the vertical line of the horizontal coordinate X _C on the left image 402 corresponds to the y-axis in real space. Similarly, the coordinate system 403 is a coordinate system defined for the camera 2-2 that generates the right image. The x axis is a horizontal direction parallel to the image plane of the camera 2-2, and the y axis is a vertical direction parallel to the image plane. Represents a direction. The z-axis is the optical axis of the imaging optical system that the camera 2-2 has. The center (X _C , Y _C ) of the right image 404 corresponds to the origin of the coordinate system 403.

The orientation of one camera may differ from the orientation of the other camera due to assembly errors when the camera 2-1 and the camera 2-2 are installed. This difference in orientation is represented by rotation around the x axis, rotation around the y axis, and rotation around the z axis. In addition, even when there is a difference in the attachment positions of the left and right cameras, the difference in the positions is expressed in a form included in the difference in the rotation amount around each axis.
Therefore, in this embodiment, the calibration parameter calculating unit 14 calculates the rotation amount of the difference theta _x about the x-axis as a horizontal calibration parameters from the set of facial feature points, the difference between the rotation amount about the y-axis as the vertical direction calibration parameters Find θ _y .

With reference to FIGS. 5A to 5C, the relationship between the set of face feature points and the calibration parameters θ _x and θ _y will be described. FIG. 5A shows an example of a set of face feature points in the left image and the right image. 5A, the left image 500 detects a face feature point 501 included in a set of i-th (i is an integer of 1 or more) face feature points at coordinates (X _{L, i} , Y _{L, i} ). In the right image 510, a face feature point 511 included in a set of i-th face feature points is detected at coordinates (X _{R, i} , Y _{R, i} ). In this case, the horizontal direction difference and the vertical direction difference on the left image from the image center 502 to the face feature point 501 corresponding to the optical axis of the camera 2-1 located at the coordinates (X _C , Y _C ) are respectively ( X _{L, i} -X _C , Y _{L, i} -Y _C ). Similarly, the horizontal difference and the vertical difference on the right image from the image center 512 to the face feature point 511 corresponding to the optical axis of the camera 2-2 located at the coordinates (X _C , Y _C ) are respectively ( X _{R, i} -X _C , Y _{R, i} -Y _C ). This vertical difference corresponds to the rotational component from the horizontal plane passing through the optical axes of the cameras 2-1 and 2-2 to the feature point, while the horizontal difference is the light of the cameras 2-1 and 2-2. This corresponds to the rotation component from the vertical plane passing through the axis to the feature point. The rotation component from the horizontal plane to the feature point corresponds to the rotation angle around the x axis, and the rotation component from the vertical plane to the feature point corresponds to the rotation angle around the y axis.

ここでfは、カメラ２−１、２−２が有する結像光学系の焦点距離である。 FIG. 5B is a schematic diagram showing the relationship between the difference in the vertical direction from the image center to the face feature point and the rotation components θ _{x, L, i} and θ _{x, R, i} from the horizontal plane. It is assumed that the image plane is located at the focal length of the camera from the camera viewpoint. Therefore, as shown in FIG. 5B, the rotation components θ _{x, L, i} and θ _{x, R, i} are calculated according to the following equations, respectively.

Here, f is the focal length of the imaging optical system that the cameras 2-1 and 2-2 have.

The calibration parameter calculation unit 14 obtains the rotation components θ _{y, L, i} , θ _{y, R, i} according to the equation (1) for each set of face feature points, and calculates the difference between the rotation components (θ _{x, L, i} -θ _{x, R, i} ) The calibration parameter calculator 14 sets the average value of the differences as the horizontal calibration parameter θ _x .

FIG. 5C is a schematic diagram showing the relationship between the difference in the horizontal direction from the image center to the face feature point and the rotation components θ _{y, L, i} and θ _{y, R, i} from the vertical plane. As shown in FIG. 5C, the rotation components θ _{y, L, i} and θ _{y, R, i} are respectively calculated according to the following equations.

The calibration parameter calculation unit 14 obtains the rotation component θ _{y, L, i} and the rotation component θ _{y, R, i} according to the equation (2) for each set of face feature points, and calculates the difference between the rotation components (θ _{y, L, i} -θ _{y, R, i} ) is obtained. Then, the calibration parameter calculation unit 14 sets the average value of the differences as the vertical direction calibration parameter θ _y .

Note that the calibration parameter calculation unit 14 may obtain the horizontal direction calibration parameter θ _x and the vertical direction calibration parameter θ _y by performing weighted averaging of the difference in rotation angle for each set of face feature points. At that time, the calibration parameter calculation unit 14 calculates the rotation calculated for the set of face feature points as the average distance from the image center (X _c , Y _c ) to the face feature points included in the set of face feature points decreases. It is preferable to reduce the weighting coefficient for the corner difference. Since the image center corresponds to the optical axis, the face feature point closer to the image center has a smaller absolute value of the rotation component with respect to the horizontal and vertical planes. For this reason, as the set of face feature points closer to the center of the image, the error included in the difference in rotational components between the left and right images becomes relatively large.

ここで(x_R,j,y_R,j)(jは1以上の整数)は、j番目の背景特徴点の組に含まれる、右画像上の背景特徴点の水平座標及び垂直座標である。また(x'_R,j,y'_R,j)は、j番目の背景特徴点の組に含まれる、右画像上の背景特徴点の補正後の水平座標及び垂直座標である。なお、右画像から抽出された各背景特徴点の位置を補正する代わりに、左画像上の各背景特徴点の位置を補正する場合、較正パラメータ算出部１４は、（３）式においてθ_x、θ_yをそれぞれ(-θ_x)、(-θ_y)とすればよい。また較正パラメータ算出部１４は、各画像の背景特徴点をそれぞれ較正パラメータθ_x、θ_yの半分ずつ移動させてもよい。この場合、左画像上の背景特徴点について、（３）式におけるθ_x、θ_yは、それぞれ(-θ_x/2)、(-θ_y/2)に置き換えられる。また右画像上の背景特徴点について、（３）式におけるθ_x、θ_yは、それぞれ(θ_x/2)、(θ_y/2)に置き換えられる。 Further, the calibration parameter calculation unit 14 obtains the rotation direction calibration parameter θ _z based on the set of background feature points. Therefore, first, the calibration parameter calculation unit 14 cancels the shift of the position of the feature point on the image due to the difference in the rotation angle of the camera around the x axis and the difference in the rotation angle of the camera around the y axis for the set of background feature points. . For this purpose, the calibration parameter calculation unit 14 corrects the position of each background feature point extracted from the right image using the calibration parameters θ _x and θ _y according to the following equation.

Where (x _{R, j} , y _{R, j} ) (j is an integer greater than or equal to 1) is the horizontal and vertical coordinates of the background feature points on the right image included in the set of the jth background feature point . Further, (x ′ _{R, j} , y ′ _{R, j} ) is the corrected horizontal coordinate and vertical coordinate of the background feature point on the right image included in the set of the j-th background feature point. In addition, when correcting the position of each background feature point on the left image instead of correcting the position of each background feature point extracted from the right image, the calibration parameter calculator 14 calculates θ _x , the theta _y respectively (-θ _x), - may be a (θ _y). The calibration parameter calculation unit 14 may move the background feature points of each image by half of the calibration parameters θ _x and θ _y , respectively. In this case, the background feature points on the left image, the theta _x, theta _y in (3), respectively (- [theta] _x / 2), - is replaced by (θ _y / 2). For the background feature point on the right image, θ _x and θ _y in equation (3) are replaced with (θ _x / 2) and (θ _y / 2), respectively.

Thereafter, the calibration parameter calculation unit 14 sets each background feature point on the left image and each background feature point on the right image on the image in the opposite directions with the image center (X _c , Y _c ) as the rotation center. To rotate the angle by Δθ / 2. When the calibration parameter calculation unit 14 changes the rotation angle Δθ of the right image with respect to the left image little by little, the rotation angle when the average value of the difference between the left and right vertical coordinates is minimized for each set of background feature points. Let Δθ be the rotation direction calibration parameter θ _z .
The calibration parameter calculation unit 14 may obtain the rotation direction calibration parameter θ _z by performing weighted averaging of the vertical coordinate difference for each set of background feature points. Also in this case, the calibration parameter calculation unit 14 calculates a set of background feature points as the average value of the distances from the image center (X _c , Y _c ) to the background feature points included in the set of background feature points is small. It is preferable to reduce the weighting factor for the difference between the vertical coordinates.

Note that the local change in the pixel value in the background region may be small, as in the case where only the wall exists behind the person who is the subject. In such a case, the number of background feature point sets to be extracted is reduced, and as a result, the number of background feature point sets may be insufficient to obtain the rotation direction calibration parameter θ _z . Therefore, the calibration parameter calculation unit 14 obtains the local gradient orientation distribution in the background region from the left image and the right image, respectively, and calculates the difference in peak angle in the local gradient orientation distribution between the left image and the right image in the rotational direction calibration. The parameter θ _z may be used.

ここでV_i,j、V_i-1,j及びV_i,j-1は、それぞれ、画素(i,j)、(i-1,j)及び(i,j-1)の画素値である。なお、左画像及び右画像がカラー画像である場合、これらの画素値は、例えば、何れか一つの色成分の値とすることができる。較正パラメータ算出部１４は、背景領域内の各画素について、（４）式に従って方位角を求め、そして方位角ごとの画素数を求める。 FIG. 6A is a diagram for explaining the positional relationship between a plurality of pixels used for calculation of the local gradient. As shown in FIG. 6A, for the pixel of interest (i, j) in the background region 600, the calibration parameter calculation unit 14, for example, has the pixel (i−1, j) adjacent to the left side and the upper side. Refer to the adjacent pixel (i, j-1). Then, the calibration parameter calculation unit 14 obtains the orientation φ _ij of the local gradient of the pixel (i, j) according to the following equation.

Where V _{i, j} , V _{i−1, j} and V _{i, j−1} are the pixel values of pixels (i, j), (i−1, j) and (i, j−1), respectively. is there. In addition, when the left image and the right image are color images, these pixel values can be set to the value of any one color component, for example. The calibration parameter calculation unit 14 obtains an azimuth angle for each pixel in the background area according to the equation (4), and obtains the number of pixels for each azimuth angle.

FIG. 6B is a schematic diagram of the orientation distribution of the local gradient obtained from the background regions of the left image and the right image. The histogram 610 represents the distribution of the azimuth of the local gradient obtained from the background region of the left image, and the histogram 620 represents the distribution of the azimuth of the local gradient obtained from the background region of the right image. The horizontal axis represents the azimuth angle, and the vertical axis represents the number of pixels. For example, in the histogram 610, the number of pixels of the azimuth angle phi _a most frequently, i.e., azimuth angle phi _a is peak angle. In the histogram 620, the azimuth angle φ _b is the peak angle. In this case, the calibration parameter calculation unit 14 calculates the azimuth angle difference (φ _a −φ _b ) as the rotation direction calibration parameter θ _z .
The calibration parameter calculation unit 14 passes the set of calibration parameters (θ _x , θ _y , θ _z ) to the determination unit 15.

  The determination unit 15 determines whether the calibration parameter calculated by the calibration parameter calculation unit 14 is appropriate. In order to correctly display a stereo image in three dimensions, the height of the same object on the left image and the right image corrected using the calibration parameters is equal, that is, the height of the epipolar line for the same object is the same. It is important that they are equal in both images. Therefore, the determination unit 15 determines whether or not the calibration parameter is appropriate based on the magnitude of the vertical deviation of each set of face feature points or each set of background feature points in the corrected left image and right image. The epipolar line is a line obtained by projecting a line connecting one camera and a point of interest on the subject onto an image photographed by the other camera.

ここで、(x_R,k,y_R,k)(kは1以上の整数)は、k番目の背景特徴点または顔特徴点の組に含まれる、x軸及びy軸周りのカメラの回転角の差による位置のずれが補正された後の右画像上の背景特徴点または顔特徴点の水平座標及び垂直座標である。また(x"_R,k,y"_R,k)は、k番目の背景特徴点または顔特徴点の組に含まれる、光軸周りのカメラの回転角の差による位置のずれを補正した後の右画像上の背景特徴点または顔特徴点の水平座標及び垂直座標である。なお、右画像から抽出された各特徴点の位置を補正する代わりに、左画像上の各特徴点の位置を補正する場合、判定部１５は、（３）及び（５）式においてθ_x、θ_y、θ_zをそれぞれ(-θ_x)、(-θ_y)、(-θ_z)とすればよい。また判定部１５は、各画像の特徴点をそれぞれ較正パラメータθ_x、θ_y、θ_zの半分ずつ移動させてもよい。この場合、左画像上の特徴点については、（３）及び（５）式におけるθ_x、θ_y、θ_zは、それぞれ(-θ_x/2)、(-θ_y/2)、(-θ_z/2)に置き換えられる。また右画像上の特徴点については、（３）及び（５）式におけるθ_x、θ_y、θ_zは、それぞれ(θ_x/2)、(θ_y/2)、(θ_z/2)に置き換えられる。 For example, the determination unit 15 converts the coordinates of the face feature point and the background feature point of the right image using the calibration parameter set described above, and obtains the coordinates of each feature point on the corrected right image. For this purpose, the determination unit 15 corrects the position of each feature point according to, for example, the above equation (3). After that, the determination unit 15 converts the corrected coordinates of the feature points according to the following expression, so that the position around the z axis, that is, the difference in rotation angle between the camera 2-1 and the camera 2-2 around the optical axis. Correct the deviation.

Where (x _{R, k} , y _{R, k} ) (k is an integer greater than or equal to 1) is the rotation of the camera around the x-axis and y-axis included in the set of the kth background feature point or face feature point It is the horizontal coordinate and the vertical coordinate of the background feature point or the face feature point on the right image after correcting the position shift due to the difference in corners. Also, (x " _{R, k} , y" _{R, k} ) is after correcting the position shift due to the difference of the camera rotation angle around the optical axis included in the kth background feature point or face feature point set The horizontal coordinate and the vertical coordinate of the background feature point or the face feature point on the right image. In addition, when correcting the position of each feature point on the left image instead of correcting the position of each feature point extracted from the right image, the determination unit 15 determines that θ _x , θ _y, θ _z, respectively _{(-θ x), (- θ} y), - may be a (θ _z). The determination unit 15 may move the feature points of each image by half of the calibration parameters θ _x , θ _y , and θ _z , respectively. In this case, for feature points on the left image, θ _x , θ _y , and θ _z in equations (3) and (5) are (−θ _x / 2), (−θ _y / 2), (− replaced by θ _z / 2). For feature points on the right image, θ _x , θ _y , and θ _z in equations (3) and (5) are (θ _x / 2), (θ _y / 2), and (θ _z / 2), respectively. Is replaced by

If the average value of absolute deviations in the vertical coordinate shift between the feature points included in each feature point set is within a predetermined allowable range, the determination unit 15 sets the calibration parameter set (θ _x , θ _y , θ _z ) is determined to be appropriate. On the other hand, if the average value is out of the allowable range, the determination unit 15 determines that the set of calibration parameters is inappropriate. For example, when a stereo image is displayed by the interlace method, since the shift of one pixel in the vertical direction is not perceived, the allowable range is set to ± 1 pixel.

In addition, the allowable range for the average absolute value of the vertical coordinate shift between the feature points in the set of face feature points is the average of the absolute value of the vertical coordinate shift between the feature points in the set of background feature points. May be different from the allowable range for. For example, when the quality of the 3D display image of the face is more important than the quality of the 3D display image of the object in the background area, the allowable range for the set of face feature points is the allowable range for the set of background feature points. It may be set narrower than the range. For example, the allowable range for the set of face feature points may be ± 1 pixel, and the allowable range for the set of background feature points may be ± 2 pixels.
Further, in order to evaluate the vertical coordinate shift between the feature points included in each set of feature points, the determination unit 15 calculates the average of the absolute values of the shifts instead of calculating the average value of the squares of the shifts. A value may be obtained. In this case, the determination unit 15 determines that the set of calibration parameters (θ _x , θ _y , θ _z ) is appropriate when the average value of the squares of deviations in the vertical coordinate is equal to or less than a predetermined threshold. .
Further, the determination unit 15 obtains an average value of the vertical coordinate shift between the feature points using only the set of face feature points or only the set of background feature points as described above, and the average value falls within the allowable range. Whether or not a set of calibration parameters (θ _x , θ _y , θ _z ) is appropriate may be determined depending on whether they are included.

When the determination unit 15 determines that the calibration parameter set (θ _x , θ _y , θ _z ) is appropriate, the determination unit 15 stores the calibration parameter set in the storage unit 4 and stores the calibration parameter set in the processing unit 5. Notify that the pair has been obtained. Thereafter, each time a pair of left and right images is obtained from the cameras 2-1 and 2-2, the processing unit 5 uses at least one of the calibration parameter sets according to the equations (3) and (5). A set of stereo images is obtained by converting the coordinates of each pixel of the image.
On the other hand, when the determination unit 15 determines that the set of calibration parameters (θ _x , θ _y , θ _z ) is inappropriate, the determination unit 15 discards the set of calibration parameters and notifies the processing unit 5 to that effect. Then, the processing unit 5 obtains the left image and the right image again from the cameras 2-1 and 2-2, and obtains a set of calibration parameters again based on the newly obtained left image and right image.

Further, the determination unit 15 uses the calibration parameter set (θ _x , θ _y , θ _z ) to generate a corrected image of the right image or the left image obtained after calculation of the calibration parameter set, and the correction A set of face feature points and a set of background feature points may be extracted again based on the image. Then, the determination unit 15 determines whether or not the set of calibration parameters is appropriate depending on whether or not the average value of the absolute value of the deviation of the vertical coordinate between the feature points in the set of feature points is included in the predetermined allowable range. You may judge.

FIG. 7 is an operation flowchart of a calibration process (stereo image calibration process) executed by the processing unit 5.
The processing unit 5 acquires the left image from the camera 2-1, and acquires the right image from the camera 2-2 (step S101). Then, the processing unit 5 passes the left image and the right image to the face detection unit 11, the face feature point extraction unit 12, and the background feature point extraction unit 13. The face detection unit 11 detects each face area from the left image and the right image (step S102). Then, the face detection unit 11 passes information representing the face region and the background region to the face feature point extraction unit 12 and the background feature point extraction unit 13.

The face feature point extraction unit 12 extracts at least one set of face feature points corresponding to the same point on the face from each face area of the left image and the right image (step S103). Then, the face feature point extraction unit 12 stores the coordinates of each feature point included in the set of face feature points in the storage unit 4.
Further, the background feature point extraction unit 13 extracts at least one set of background feature points corresponding to the same point on the same object from the respective background regions of the left image and the right image (step S104). Then, the background feature point extraction unit 13 stores the coordinates of each feature point included in the set of background feature points in the storage unit 4.

The calibration parameter calculation unit 14 calculates calibration parameters θ _x and θ _y in the horizontal direction and the vertical direction on the image from the set of face feature points (step S105). The calibration parameter calculation unit 14 calculates a rotation direction calibration parameter θ _z from the set of background feature points (step S106). Then, the calibration parameter calculation unit 14 passes the set of calibration parameters (θ _x , θ _y , θ _z ) to the determination unit 15.

The determination unit 15 determines whether or not the set of calibration parameters (θ _x , θ _y , θ _z ) is appropriate (step S107). If the set of calibration parameters (θ _x , θ _y , θ _z ) is not appropriate (No at Step S107), the processing unit 5 repeats the processes after Step S101. On the other hand, if the set of calibration parameters (θ _x , θ _y , θ _z ) is appropriate (Yes in step S107), the determination unit 15 stores the set of calibration parameters in the storage unit 4, and then performs a calibration process. Exit. Note that the processing unit 5 may change the order of the processes in steps S103 and S104.

  As described above, this stereo image calibration apparatus detects a face that is known as a subject in advance from the left and right images using the feature, and based on the feature point in the region in which the face is reflected. Thus, the horizontal and vertical calibration parameters are obtained. Therefore, this stereo image calibration apparatus can correct the position of the face as the subject in the left and right images so as to be suitable for the stereo image. Further, the stereo image calibration apparatus obtains a calibration parameter in the rotation direction based on a set of background feature points in the background region. Since the background area is usually away from the center of the image corresponding to the optical axis, it easily reflects the difference in rotation angle around the optical axis between cameras. Therefore, this stereo image calibration apparatus can appropriately correct the rotation of the image between the left and right images.

According to the modification, the calibration parameter calculation unit may obtain the rotation direction calibration parameter θ _z using not only the set of background feature points but also the set of face feature points. Thus, the calibration parameter calculating unit, it is possible to refer to the set of feature points distributed in a wider range in the image can be determined more appropriately rotating direction calibration parameter theta _z. In particular, there may be a small number of background feature point sets to be extracted, as in the case where a wall having no pattern appears in the background area. In such a case, the calibration parameter calculation unit can obtain the rotation direction calibration parameter with high accuracy by using a set of face feature points.

According to still another modification, the background feature point extraction unit may limit the region from which the background feature point is extracted to a region where the same object is likely to appear in both the left image and the right image. Good.
In the vicinity of the face area, an occlusion occurs in which an object behind the face area is concealed by the head of the person reflected in the face area. In the present embodiment, since the positions of the camera 2-1 and the camera 2-2 are different, the range hidden by the head in the left image is different from the range hidden by the head in the right image. Therefore, the background feature point extraction unit removes a region within a predetermined range from the face region from the background region, and extracts a background feature point from the remaining region.

This will be described with reference to FIGS. 8A and 8B. FIG. 8A is a schematic diagram showing a range concealed by the head in the right image in the background area shown on the left image. FIG. 8B is a schematic diagram showing a range concealed by the head in the left image in the background area shown on the right image.
8A and 8B, points 801 and 802 represent the viewpoints of the camera 2-1 and the camera 2-2, respectively. Lines 803 and 804 represent the imaging surface of the camera 2-1 and the imaging surface of the camera 2-2, respectively. A line 805 represents the range of the head in real space. In this example, the head 805 is approximately represented by a plane for simplification. A line 806 represents an object farthest from the cameras 2-1 and 2-2. In this case, an object in the area 807 is not shown in the right image but hidden in the head 805 although it is shown in the left image. Therefore, in the left image, the background feature point extraction unit from the horizontal coordinate x _{L, 0, max} on the left image corresponding to the left end of the region 807 to the horizontal coordinate x _{L, 0} on the left image corresponding to the right end of the region 807 Is excluded from the range from which background feature points are extracted.

In this case, since the right end of the area 807 is defined by the head 805 appearing in the face area, the horizontal coordinate x _{L, 0} on the left image is the left end coordinate of the face area. The horizontal coordinate x _{L, 0, max} is a position corresponding to the left end coordinate x _{R, 0} of the face area on the right image. If the distance Z _max from the cameras 2-1 and 2-2 to the farthest object is known, the distance between the camera 2-1 and the camera 2-2 is usually known. Based on this, the horizontal coordinate x _{L, 0, max} on the left image corresponding to the coordinate x _{R, 0} on the right image is calculated. When the distance Z _max is unknown, the background feature point extraction unit extracts Z _max = ∞, that is, a line passing through the viewpoint 802 of the camera 2-2 and the coordinate x _{R, 0} and the viewpoint 801 of the camera 2-1. Assuming that the lines passing through the horizontal coordinates x _{L, 0, max} are parallel, the horizontal coordinates x _{L, 0, max} are set to the coordinates x _{R, 0} .

Similarly, a region that is hidden in the head image 805 but is not reflected in the left image even though it appears in the right image is the region 808 shown in FIG. 8B. Therefore, in the right image, the background feature point extraction unit from the horizontal coordinate x _{R, 1} on the right image corresponding to the left end of the region 808 to the horizontal coordinate x _{R, 1, max} on the right image corresponding to the right end of the region 808. Is excluded from the range from which background feature points are extracted. Since the left end of the area 808 is defined by the head 805 that appears in the face area, the horizontal coordinate x _{R, 1} is the coordinate of the right end of the face area. On the other hand, the horizontal coordinate xR _{, 1, max} is a position corresponding to the coordinate xL _{, 1} of the right end of the face area on the left image. If the distance Z _max from the cameras 2-1 and 2-2 to the farthest object is known, the horizontal coordinate x _R is based on the principle of triangulation as well as the horizontal coordinate x _{L, 0, max. , 1, max} are calculated. When the distance Z _max is unknown, the background feature point extraction unit assumes that Z _max = ∞ and sets the horizontal coordinate x _{R, 1, max} to the coordinate x _{L, 1} .

FIG. 9A and FIG. 9B are schematic diagrams showing regions excluded from the background region in the left image and the right image, respectively, regarding the extraction of background feature points. As shown in FIG. 9A, in the left image 900, a range 902 from the horizontal coordinates x _{L, 0, max} to x _{L, 0} adjacent to the left side of the face area 910 is extracted from the background feature points. It is excluded from the background area 901 as a non-performed area. Also, as shown in FIG. 9B, in the right image 905, the range 907 from the horizontal coordinates xR _{, 1} to xR _{, 1, max} adjacent to the right side of the face area 915 is extracted from the background feature points. Is excluded from the background area 906 as an area where no image processing is performed. Thereby, for example, in the left image 900, the object point 920 concealed by the head is out of the background feature point extraction target area in the right image 905. Incidentally, when the face region is not a rectangular area, the background feature point extraction unit, x _{L, 0} to regard the left image, the leftmost coordinate of the face area and x _{L, 0} for each row, obtained as above _{, the} left coordinate for the width from _max to x _{L, 0} may be x _{L, 0, max} . Similarly, with respect to the right image, the background feature point extraction unit sets the right end coordinate of the face area to x _{R, 1} for each row, and obtains x _{R, 1} to x _{R, 1, max} as described above. The coordinates on the right side of the width may be set to _{xR, 1, max} .

  FIG. 10A and FIG. 10B are schematic diagrams of other examples showing areas excluded from the background area in the left image and the right image, respectively, regarding the extraction of the background feature points. As shown in FIG. 10A, in the left image 1000, an object shown in a region 1001 having a width d near the left end is not shown in the right image like an object point 1020. Further, as shown in FIG. 10B, in the right image 1010, an object shown in a region 1011 having a width d near the right end is not shown in the left image. Therefore, the background feature point extraction unit may not extract the background feature points from the left image area 1001 and the right image area 1011.

ここで、Wは左画像の水平方向の画素数である。同様に、右画像において背景特徴点が抽出されない右画像の右端からの距離dも（６）式により算出される。 A method for determining the width d will be described with reference to FIGS. FIG. 11 is a schematic diagram showing the relationship between the shooting range of the camera 2-1 that generates the left image and the shooting range of the camera 2-2 that generates the right image. In this example, it is assumed that the optical axis of the imaging optical system of the camera 2-1 is parallel to the optical axis of the imaging optical system of the camera 2-2.
In FIG. 11, points 1101 and 1102 represent the viewpoints of the camera 2-1 and the camera 2-2, respectively. Lines 1103 and 1104 represent the imaging plane of the camera 2-1 and the imaging plane of the camera 2-2, respectively. The horizontal angles of view of the cameras 2-1 and 2-2 are both assumed to be θ. A line 1105 represents an object farthest from the cameras 2-1 and 2-2. In this case, the horizontal width of the object 1105 shown in the left image is (2Z _max tan (θ / 2), where Z _max is the distance from the camera 2-1 to the object 1105. Similarly, the right The horizontal width of the object 1105 in the image is (2Z _max tan (θ / 2). If the distance between the camera 2-1 and the camera 2-2 is T, the object corresponding to the left end on the left image An object existing in an area 1107 including a point from 1106 on the point 1105 to a point from a distance T is not shown in the right image, so that a background feature point is not extracted and a distance d from the left end of the left image is calculated by the following equation. .

Here, W is the number of pixels in the horizontal direction of the left image. Similarly, the distance d from the right end of the right image from which the background feature point is not extracted in the right image is also calculated by the equation (6).

Further, the optical axis of the imaging optical system of the camera 2-1 may not be parallel to the optical axis of the imaging optical system of the camera 2-2.
FIG. 12 shows the distance Z ′ from the cameras 2-1 and 2-2 to the plane where the range shown in the left image and the range shown in the right image are the same. it is a schematic view of a photographic range of the camera 2-1 and 2-2 is larger than the most distant distance Z _max to the object caught on.
In FIG. 12, points 1201 and 1202 represent the viewpoints of the cameras 2-1 and 2-2, respectively. Lines 1203 and 1204 represent the imaging plane of the camera 2-1 and the imaging plane of the camera 2-2, respectively. The horizontal angles of view of the cameras 2-1 and 2-2 are both assumed to be θ. An object 1205 is an object farthest from the cameras 2-1 and 2-2. Further, the optical axis of the imaging optical system of the camera 2-1 and the optical axis of the imaging optical system of the camera 2-2 intersect at the object side at an angle α, and α / Suppose that it is tilted by 2. This α is, for example, a design value.

Here, P is a point corresponding to the left end of the right image in the object 1205. At this time, a distance l between the point P2 vertically lowered from the viewpoint 1202 of the camera 2-2 to the object 1205 and P is Z _max tan (θ / 2 + α / 2). On the other hand, an angle β formed by a straight line passing through the viewpoint 1201 and P of the camera 2-1 and the line of sight is (α / 2 + tan ⁻¹ ((lT) / Z _max )). Note that T is the distance between the camera 2-1 and the camera 2-2. Accordingly, if the number of pixels in the horizontal direction of the left image and the right image is W, the width d from the image end in the other image of the region not shown in one image is obtained by

FIG. 13 shows the distance Z ′ from the cameras 2-1 and 2-2 to the plane where the range shown in the left image and the range shown in the right image are the same. it is a schematic view of a photographic range of the camera 2-1 and 2-2 is smaller than the distance Z _max of the most up to distant objects caught on.
In FIG. 13, points 1301 and 1302 represent the viewpoints of the cameras 2-1 and 2-2, respectively. Lines 1303 and 1304 represent the imaging plane of the camera 2-1 and the imaging plane of the camera 2-2, respectively. The horizontal angles of view of the cameras 2-1 and 2-2 are both assumed to be θ. An object 1305 is an object that is farthest from the cameras 2-1 and 2-2 (distance Z _max ). Further, the optical axis of the imaging optical system of the camera 2-1 and the optical axis of the imaging optical system of the camera 2-2 intersect on the object side at an angle α, and α / Suppose that it is tilted by 2.
Here, Q is a point corresponding to the left end of the right image in the object 1305. At this time, a distance m between the point Q1 lowered vertically from the viewpoint 1301 of the camera 2-1 to the object 1305 and Q is Z _max tan (θ / 2−α / 2). On the other hand, the angle γ formed between the straight line passing through the viewpoint 1301 of the camera 2-2 and Q and the optical axis 1306 of the camera 2-2 is (tan ⁻¹ ((m + T) / Z _max ) −α / 2). Become. Note that T is the distance between the camera 2-1 and the camera 2-2. Accordingly, if the number of pixels in the horizontal direction of the left image and the right image is W, the width d from the image end in the other image of the region not shown in one image is obtained by

Further, the background feature point extraction unit removes both the above-mentioned region near the left end of the left image and the region near the right end of the right image and the region near the face region where occlusion occurs due to the head from the background region. Background feature points may be extracted only from the region.
FIG. 14A and FIG. 14B are schematic diagrams of still another example showing areas excluded from the background area in the left image and the right image with respect to extraction of the background feature points, respectively. As shown in FIG. 14A, in the left image 1400, the horizontal coordinate x _{L, 0, max} to x _{L in the} vicinity of the left end of the area 1402 and the face area 1403 from the background area to the left end of the left image from the background area. _{, 0} , the remaining area 1405 excluding the area 1404 is the background feature point extraction target. 14B, in the right image 1410, horizontal coordinates x _{R, 1} to x _R, from the background region to the region 1412 having a width d near the right end of the right image and the right end of the face region 1413 _{, The} remaining area 1415 excluding the area 1414 up to _{1, max} is the background feature point extraction target.

The setting of the exclusion range of background feature point extraction as described above can also be applied to an apparatus that performs three-dimensional measurement with a calibrated stereo camera.
For example, since the interval between the camera 2-1 and the camera 2-2 is known, the processing unit determines whether the pair of background feature points is from the cameras 2-1 and 2-2 based on the principle of triangulation. The distance to the object corresponding to the set of background feature points can be obtained. Similarly, for each set of facial feature points, the processing unit calculates the distance from the cameras 2-1 and 2-2 to the point on the face corresponding to the set of facial feature points based on the principle of triangulation. Can be sought.
At this time, if a feature point concealed by the subject in an image taken with one camera or a feature point that can only be seen by one camera is included, the amount of processing is wasted and the feature point is mishandled. A measurement error occurs. At this time, as described above, by setting a region where the background feature points are not extracted, the processing unit can realize an improvement in processing efficiency and accuracy.

In some cases, the focal length of the imaging optical system of the camera 2-1 may be different from the focal length of the imaging optical system of the camera 2-2. In such a case, the magnification of the image shown in the left image (hereinafter referred to as the camera magnification) is different from the magnification of the image shown in the right image. According to the modification, the processing unit can also use a set of feature points corresponding to the same point of the same object for calibration of the difference in camera magnification.
Here, the average value of the distance between the two feature points on the left image and the average value of the distance between the two feature points on the right image are estimated to reflect the difference in camera magnification. In particular, for a point on an object that is relatively far from the camera, the distance from both cameras to that point is almost equal, so the difference between each camera and the distance to that point is negligibly small.
Therefore, for example, for each of the left image and the right image, the calibration parameter calculation unit selects any two background feature points from a plurality of background feature points that correspond relatively to the object from the camera, and the two backgrounds The distance between feature points (positional difference on the image) is calculated. The calibration parameter calculation unit calculates various distances between the background feature points by changing various combinations of background feature points to be selected, and obtains an average value of the distances. Then, the calibration parameter calculation unit obtains, as a scale ratio S, a ratio of the average value of the distance between the background feature points obtained for the right image to the average value of the distance between the background feature points obtained for the left image. When the scale ratio S is determined, the calibration parameter calculation unit determines X _{R, i} in the equations (1) and (2) when determining the calibration parameter θ _x around the x axis and the calibration parameter θ _y around the y axis. By multiplying YR _{, i} by the scale ratio S, the difference in camera magnification can be corrected.

According to still another modification, a set of face feature points and a set of background feature points are obtained from a right image or a left image corrected based on a set of calibration parameters (θ _x , θ _y , θ _z ) once obtained. By obtaining, the distance from the camera to the face and the distance to the object corresponding to each set of background feature points are obtained. As a result, the relative distance from the face to the object corresponding to each set of background feature points can be measured. Based on the measured distance, the background feature point extraction unit calculates a template matching search range when obtaining a set of background feature points to an area on the image that is a distance within a predetermined range centered on the distance. Can be limited. The background feature point extraction unit can obtain a set of background feature points that are more likely to correspond to the same point of the same object by performing template matching again within a limited range. Become. The calibration parameter calculation unit can obtain a more accurate calibration parameter by calculating the calibration parameter again using the set of background feature points. In particular, in the above-described estimation of the scale ratio, the calibration parameter calculation unit selectively uses a distant background feature point to thereby determine the distance from the camera 2-1 to the object and the camera 2-2 to the object. Measurement error of the scale ratio due to the difference in distance can be reduced.

  According to still another modification, the subject may be a person's whole body or upper body, a vehicle, a plant, or the like instead of a face. In this case, the processing unit includes a subject region detection unit that detects a subject region in which such a subject is captured from the left image and the right image. For example, the subject area detection unit, like the face detection unit, uses supervised learning using a plurality of images whose subject is known to be captured in advance and a plurality of images whose subject is known not to be captured. Thus, the subject region is extracted using the discriminator learned to extract the subject region on the image. Then, after the subject area is extracted, the processing unit obtains a set of subject feature points representing the same point of the subject from the subject areas of the left image and the right image, as in the above-described embodiment or the modification thereof, and the background. A set of background feature points is obtained from the region. Then, the processing unit may obtain horizontal and vertical calibration parameters from the set of subject feature points, and obtain a rotation direction calibration parameter from the set of background feature points.

  The computer program that realizes the function of the processing unit according to the above-described embodiment or its modification may be provided in a form recorded on a computer-readable recording medium such as a magnetic recording medium or an optical recording medium.

  All examples and specific terms listed herein are intended for instructional purposes to help the reader understand the concepts contributed by the inventor to the present invention and the promotion of the technology. It should be construed that it is not limited to the construction of any example herein, such specific examples and conditions, with respect to showing the superiority and inferiority of the present invention. Although embodiments of the present invention have been described in detail, it should be understood that various changes, substitutions and modifications can be made thereto without departing from the spirit and scope of the present invention.

The following supplementary notes are further disclosed regarding the embodiment described above and its modifications.
(Appendix 1)
Obtaining a first image generated by shooting the area including the subject with the first camera and a second image generated by shooting the area with the second camera;
Detecting a first subject area in which the subject is reflected from the first image, and detecting a second subject area in which the subject is reflected from the second image;
Extracting at least one set of subject feature points corresponding to the same point on the subject from the first subject region and the second subject region;
Based on the set of at least one subject feature point, the rotation angle of the first camera around each of a first axis parallel to the image plane and a second axis orthogonal to the first axis And a first calibration for correcting a difference between the position of the subject on the first image and the position of the subject on the second image due to the difference. Find the parameters
From an image of an object shown in a first background area obtained by removing the first subject area from the first image and a second background area obtained by removing the second subject area from the second image, A second calibration parameter for correcting rotation between the first image and the second image due to a difference between a rotation angle around the optical axis of the first camera and a rotation angle around the optical axis of the second camera; Seeking
Stereo image calibration method comprising:
(Appendix 2)
Further comprising extracting at least one set of background feature points corresponding to the same point on the same object from the previous first background region and the second background region;
Obtaining the second calibration parameter means that at least one of the first image and the second image is rotated, and the background included in the first image for each set of background feature points. The distance between the feature point and the background feature point included in the second image is obtained to obtain an average value of the distance, and the second image is rotated with respect to the first image when the average value is minimized. The stereo image calibration method according to appendix 1, wherein an angle is the second calibration parameter.
(Appendix 3)
Extracting the set of background feature points extracts the background feature points from an area excluding the range corresponding to the area hidden by the subject for the second camera from the first background area. The stereo image calibration method according to appendix 2.
(Appendix 4)
In extracting the background feature point, the background feature point is extracted from an area excluding a range outside the shooting range of the second camera from the first background area. A stereo image calibration method as described.
(Appendix 5)
The second calibration parameter is obtained by obtaining the first azimuth angle at which the frequency of the local gradient is maximized by obtaining the azimuth angle of the local gradient of the pixel value in each pixel in the first background region. And, by obtaining the azimuth angle of the local gradient of the pixel value in each pixel in the second background region, the second azimuth angle at which the frequency of the local gradient is maximized is obtained, and the first azimuth angle and The stereo image calibration method according to appendix 1, wherein the difference in the second azimuth is the second calibration parameter.
(Appendix 6)
The corrected feature obtained by converting the coordinates of the subject feature point for at least one of the first image and the second image using the first calibration parameter and the second calibration parameter. The difference between the vertical coordinate of the subject feature point and the vertical coordinate of the subject feature point corresponding to the same point on the subject for the other of the first image and the second image is a predetermined value. The stereo image calibration method according to any one of appendices 1 to 5, wherein the first calibration parameter and the second calibration parameter are stored in a storage unit when the tolerance is within an allowable range.
(Appendix 7)
Extracting at least one set of subject feature points;
Detecting a first candidate point on the first subject area using a predetermined detector;
A second partial area in the second subject area that most closely matches the first partial area in the first subject area that includes the first candidate point is determined, and the location in the second partial area is determined. Let the fixed point be the second candidate point,
A third partial region in the first subject region that most closely matches the second partial region is obtained, and a distance between the third candidate point in the third partial region and the first candidate point is determined. If it is equal to or less than a predetermined threshold, the first candidate point or the third candidate point and the second subject candidate point are set as a set of the subject feature points. A stereo image calibration method according to claim 1.
(Appendix 8)
Determining the first calibration parameter includes
For each set of subject feature points,
According to a first distance from a point corresponding to the optical axis of the first camera on the first image to a first subject feature point on the first image included in the set of subject feature points In addition, the second feature point included in the set of subject feature points from the point corresponding to the rotation angle of the first subject feature point with respect to the first direction and the optical axis of the second camera on the second image. Determining a first difference between a rotation angle of the second subject feature point with respect to the first direction according to a second distance to the second subject feature point on the image of
A rotation angle of the first subject feature point with respect to the second direction according to the first distance, and a rotation angle of the second subject feature point with respect to the second direction according to the second distance; Find the second difference of
The stereo image calibration method according to any one of appendices 1 to 7, wherein an average of the first difference and an average of the second difference are used as the first calibration parameter.
(Appendix 9)
Obtaining a first image generated by shooting the area including the subject with the first camera and a second image generated by shooting the area with the second camera;
Detecting a first subject area in which the subject is reflected from the first image, and detecting a second subject area in which the subject is reflected from the second image;
Extracting at least one set of subject feature points corresponding to the same point on the subject from the first subject region and the second subject region;
Based on the set of at least one subject feature point, the rotation angle of the first camera around each of a first axis parallel to the image plane and a second axis orthogonal to the first axis And a first calibration for correcting a difference between the position of the subject on the first image and the position of the subject on the second image due to the difference. Find the parameters
From an image of an object shown in a first background area obtained by removing the first subject area from the first image and a second background area obtained by removing the second subject area from the second image, A second calibration parameter for correcting rotation between the first image and the second image due to a difference between a rotation angle around the optical axis of the first camera and a rotation angle around the optical axis of the second camera; Seeking
A computer program for stereo image calibration that causes a computer to execute this.
(Appendix 10)
A first camera that generates a first image by photographing an area including a predetermined subject;
A second camera arranged at a different position from the first camera and generating a second image by photographing the region;
A subject region detection unit that detects a first subject region in which the subject is captured from the first image and detects a second subject region in which the subject is captured from the second image;
A subject feature point extraction unit that extracts at least one set of subject feature points corresponding to the same point on the subject from the first subject region and the second subject region;
Based on the set of at least one subject feature point, the rotation angle of the first camera around each of a first axis parallel to the image plane and a second axis orthogonal to the first axis And a first calibration for correcting a difference between the position of the subject on the first image and the position of the subject on the second image due to the difference. Parameters are obtained, and are reflected in a first background area obtained by removing the first subject area from the first image and a second background area obtained by removing the second subject area from the second image. The rotation between the first image and the second image due to the difference between the rotation angle around the optical axis of the first camera and the rotation angle around the optical axis of the second camera is corrected from the image of the moving object A calibration parameter calculator for obtaining a second calibration parameter to be
Stereo image calibration apparatus having

DESCRIPTION OF SYMBOLS 1 Stereo image calibration apparatus 2-1 and 2-2 Camera 3 Input part 4 Memory | storage part 5 Processing part 11 Face detection part 12 Face feature point extraction part 13 Background feature point extraction part 14 Calibration parameter calculation part 15 Determination part

Claims (6)

Obtaining a first image generated by shooting the area including the subject with the first camera and a second image generated by shooting the area with the second camera;
Detecting a first subject area in which the subject is reflected from the first image, and detecting a second subject area in which the subject is reflected from the second image;
Extracting at least one set of subject feature points corresponding to the same point on the subject from the first subject region and the second subject region;
Based on the set of at least one subject feature point, the rotation angle of the first camera around each of a first axis parallel to the image plane and a second axis orthogonal to the first axis And a first calibration for correcting a difference between the position of the subject on the first image and the position of the subject on the second image due to the difference. Find the parameters
From an image of an object shown in a first background area obtained by removing the first subject area from the first image and a second background area obtained by removing the second subject area from the second image, A second calibration parameter for correcting rotation between the first image and the second image due to a difference between a rotation angle around the optical axis of the first camera and a rotation angle around the optical axis of the second camera; Seeking
Stereo image calibration method comprising: Further comprising extracting at least one set of background feature points corresponding to the same point on the same object from the previous first background region and the second background region;
The second to determine the calibration parameter, the first image and while rotating at least one of said second image, the background included in the prior SL first image for each set of said background feature points seeking distance between background feature points included in the second image feature points, an average value of those said distance, said second image with respect to the first image when the average value is minimized The stereo image calibration method according to claim 1, wherein the rotation angle is set as the second calibration parameter.   The second calibration parameter is obtained by obtaining the first azimuth angle at which the frequency of the local gradient is maximized by obtaining the azimuth angle of the local gradient of the pixel value in each pixel in the first background region. And, by obtaining the azimuth angle of the local gradient of the pixel value in each pixel in the second background region, the second azimuth angle at which the frequency of the local gradient is maximized is obtained, and the first azimuth angle and The stereo image calibration method according to claim 1, wherein a difference between the second azimuth angles is used as the second calibration parameter. Extracting at least one set of subject feature points;
Detecting a first candidate point on the first subject area using a predetermined detector;
A second partial area in the second subject area that most closely matches the first partial area in the first subject area that includes the first candidate point is determined, and the location in the second partial area is determined. Let the fixed point be the second candidate point,
A third partial region in the first subject region that most closely matches the second partial region is obtained, and a distance between the third candidate point in the third partial region and the first candidate point is determined. If it is less than a predetermined threshold value, and said first candidate point or the third candidate points and the second candidate point, a set of object feature point, one of claims 1 to 3 one The stereo image calibration method according to Item. Obtaining a first image generated by shooting the area including the subject with the first camera and a second image generated by shooting the area with the second camera;
Detecting a first subject area in which the subject is reflected from the first image, and detecting a second subject area in which the subject is reflected from the second image;
Extracting at least one set of subject feature points corresponding to the same point on the subject from the first subject region and the second subject region;
Based on the set of at least one subject feature point, the rotation angle of the first camera around each of a first axis parallel to the image plane and a second axis orthogonal to the first axis And a first calibration for correcting a difference between the position of the subject on the first image and the position of the subject on the second image due to the difference. Find the parameters
From an image of an object shown in a first background area obtained by removing the first subject area from the first image and a second background area obtained by removing the second subject area from the second image, A second calibration parameter for correcting rotation between the first image and the second image due to a difference between a rotation angle around the optical axis of the first camera and a rotation angle around the optical axis of the second camera; Seeking
A computer program for stereo image calibration that causes a computer to execute this. A first camera that generates a first image by photographing an area including a predetermined subject;
A second camera arranged at a different position from the first camera and generating a second image by photographing the region;
A subject region detection unit that detects a first subject region in which the subject is captured from the first image and detects a second subject region in which the subject is captured from the second image;
A subject feature point extraction unit that extracts at least one set of subject feature points corresponding to the same point on the subject from the first subject region and the second subject region;
Based on the set of at least one subject feature point, the rotation angle of the first camera around each of a first axis parallel to the image plane and a second axis orthogonal to the first axis And a first calibration for correcting a difference between the position of the subject on the first image and the position of the subject on the second image due to the difference. Parameters are obtained, and are reflected in a first background area obtained by removing the first subject area from the first image and a second background area obtained by removing the second subject area from the second image. The rotation between the first image and the second image due to the difference between the rotation angle around the optical axis of the first camera and the rotation angle around the optical axis of the second camera is corrected from the image of the moving object A calibration parameter calculator for obtaining a second calibration parameter to be
Stereo image calibration apparatus having

Images