JP2016156702A - Imaging device and imaging method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device and an imaging method using a stereo camera with which the accuracy in determination of abnormal pixels included in distance images can be improved.SOLUTION: An imaging device comprises an imaging part 2, a three-dimensional measurement part 3, a face area extraction part 4, an area division part 5, an effective range setting part 6, an abnormal pixel determination part 7, an interpolation part 8, and a smoothing part 9. The abnormal pixel determination part 7 determines, for each of divided areas constituting distance images obtained through division by the area division part 5, of a plurality of pixels included in each of the divided areas, a pixel having a distance value falling out of an effective range set in each of the divided areas as an abnormal pixel.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an imaging apparatus and imaging method using a stereo camera.

  2. Description of the Related Art In recent years, stereo three-dimensional measurement apparatuses that measure the distance to an object, the shape of the object, and the like using a plurality of cameras (stereo cameras) or a three-dimensional sensor have been put into practical use. As a stereo three-dimensional measurement method, two cameras (for example, left and right) are spaced apart to simultaneously photograph an object, and a pair of corresponding pixels in the left and right images from the obtained pair of left and right images, that is, correspondence This is a measurement method in which a point is searched, the distance between corresponding points in the left and right images is determined in the left-right direction, that is, a parallax is obtained, and the distance to the object is calculated from the parallax using the principle of triangulation. The spread of such a three-dimensional measuring apparatus is progressing in various fields.

  A conventional general camera can only store luminance information and color information of each pixel in a two-dimensional image data format. In contrast, the three-dimensional measuring device measures the distance to the person or object that is the object to be photographed, and stores the distance value of each pixel in the form of three-dimensional shape data such as distance image data or point cloud data. can do.

  One of the main uses of three-dimensional measurement is to measure the shape of a human face. Since the shape of a person's face has characteristics and information unique to each individual, development of various security systems and medical-related systems that utilize the characteristics or information of a person's face measured in three dimensions is expected. Has been.

  A part of the distance image generated by the three-dimensional measurement apparatus may include a pixel having an erroneous measurement value different from the actual distance value. There are various causes for this, for example, the quality of the captured image is low, or the processing method or parameters used for the three-dimensional measurement are inappropriate.

  An example of a conventional three-dimensional measuring apparatus aimed at solving such deficiencies during three-dimensional measurement is disclosed in Patent Document 1. This conventional apparatus determines a position in a three-dimensional coordinate system of a three-dimensional coordinate system that has been scale-converted so as to fit a three-dimensional distance image of a person, and further interpolates three-dimensional shape data from the three-dimensional model whose position has been specified. A background surface to be a processing standard for the determination is determined. Thereby, the three-dimensional shape data missing in the three-dimensional measurement is interpolated.

JP 2001-12922 (published January 19, 2001)

  The present invention has been made to solve the above-described problems of the present invention. And the objective is to provide the imaging device and imaging method which can raise the determination precision of the abnormal pixel contained in a distance image more.

  In order to solve the above-described problem, an imaging apparatus according to the present invention captures a subject from a plurality of different viewpoints, thereby generating an imaging unit that generates a plurality of images corresponding to each of the viewpoints, and the plurality of images. A three-dimensional measurement unit that generates a distance image by performing a three-dimensional measurement process, an extraction unit that extracts a determination region in the distance image, a division unit that divides the determination region into a plurality of different divided regions, and For each divided region, a setting unit that sets an effective range according to the divided region, and for each divided region, out of the effective range set for the divided region among a plurality of pixels included in the divided region And a determination unit that determines that a pixel having a distance value is an abnormal pixel.

  In order to solve the above problems, an imaging method according to the present invention captures a subject from a plurality of different viewpoints, thereby generating a plurality of images corresponding to each of the viewpoints, and the plurality of images. A three-dimensional measurement process for generating a distance image by performing a three-dimensional measurement process; an extraction process for extracting a determination area in the distance image; a division process for dividing the determination area into a plurality of different divided areas; A setting step for setting an effective range according to the divided area for each divided area, and out of the effective range set for the divided area among the plurality of pixels included in the divided area for each divided area And determining a pixel having a distance value as an abnormal pixel.

  According to one aspect of the present invention, there is an effect that the determination accuracy of abnormal pixels included in a distance image can be further increased.

It is a block diagram which shows the principal part structure of the imaging device which concerns on Embodiment 1 of this invention. It is a block diagram which shows the detailed structure of an imaging part and a dimension measurement part. It is explanatory drawing which shows the method of performing a corresponding point search process with respect to two parallelized images in Embodiment 1 of this invention. (A) is explanatory drawing which shows each pixel on the parallel image of the left camera in Embodiment 1 of this invention, (b) is obtained with respect to one attention pixel in Embodiment 1 of this invention. It is a graph which shows the relationship curve of a value and parallax. It is a figure explaining the reduction process of the erroneous measurement value which the imaging device which concerns on Embodiment 1 of this invention performs. It is a figure explaining the setting process of the effective range which the imaging device which concerns on Embodiment 3 of this invention performs. It is a figure explaining the detection process of the face area | region which the imaging device which concerns on Embodiment 5 of this invention performs.

[Embodiment 1]
One embodiment of the present invention will be described below with reference to FIGS.

  FIG. 1 is a block diagram showing a main configuration of an imaging apparatus 1 according to Embodiment 1 of the present invention. As shown in this figure, the imaging device 1 includes an imaging unit 2, a three-dimensional measuring unit 3, a face area extracting unit 4 (extracting unit), an area dividing unit 5 (dividing unit), and an effective range setting unit (setting unit, average). A distance value calculation unit, a reference value calculation unit, a focus distance range calculation unit) 6, an abnormal pixel determination unit (determination unit) 7, an interpolation unit 8, and a smoothing unit 9.

  FIG. 2 is a block diagram illustrating detailed configurations of the imaging unit 2 and the three-dimensional measurement unit 3. As shown in this figure, the imaging unit 2 includes a right camera 2a and a left camera 2b. The three-dimensional measuring unit 10 includes a parallelizing unit 11, a prefilter unit 12, a corresponding point searching unit 13, and a distance image generating unit 14.

  The imaging unit 2 captures a subject such as a human face from a plurality of different viewpoints, thereby generating a plurality of images corresponding to each of the viewpoints. The right camera 2a and the left camera 2b in the imaging unit 2 are cameras for imaging a subject. These may be a CCD (Charge Coupled Device) camera or a CMOS (Complementary Metal-Oxide Semiconductor) camera with a lens, and may be either a color camera or a monochrome camera. In this embodiment, a monochrome camera that requires a small amount of data is used.

  The three-dimensional measurement unit 3 generates a distance image of the subject by three-dimensionally measuring a plurality of images obtained by the imaging unit 2.

  The face area extraction unit 4 extracts the face area (determination area) of the subject from the distance image. Details of the face area detection will be described later.

  The area dividing unit 5 divides the face area extracted from the distance image into a plurality of different divided areas based on the characteristics of the human face shape.

  The effective range setting unit 6 sets an effective range corresponding to each divided region for each divided region. The effective range means a range of distance values that can be assumed by each pixel in the divided area when a portion corresponding to the divided area in the subject is normally three-dimensionally measured.

  For each divided region, the abnormal pixel determination unit 7 determines that all of the pixels included in the divided region that have a distance value outside the effective range set in the divided region are abnormal pixels. .

  For each divided region, the interpolation unit 8 performs effective interpolation in which the distance value of the abnormal pixel is set in the divided region by interpolation processing using each distance value of the pixel group around the pixel determined to be an abnormal pixel. Change to a value within the range.

  For each divided region, the smoothing unit 9 performs a smoothing process on the distance values of all the pixels included in the divided region.

  The imaging device 1 includes a processor and a memory (not shown). In this memory, a program necessary for the processor to execute each process such as the above-described three-dimensional measurement process, temporary data, calibration data of the right camera 2a and the left camera 2b, and data such as a focus distance, etc. Is stored.

  A procedure for three-dimensional measurement executed by the imaging apparatus 1 according to the present embodiment will be described below.

  In the imaging apparatus 1 according to the present embodiment, a three-dimensional measurement process is executed by the three-dimensional measurement unit 10 on the captured images taken by the right camera 2a and the left camera 2b.

  In the three-dimensional measuring unit 10, first, the parallelizing unit 11 performs a stereo parallelization process on the captured images taken by the right camera 2 a and the left camera 2 b. In the stereo parallelization process, in each image taken by a plurality of cameras, for example, when a wide-angle lens or the like is used, the distortion increases toward the outside of the image. Correcting the distortion is called parallelization processing. The stereo parallelization process is obtained by camera calibration performed in advance. For this reason, stereo parallelization processing is performed on the captured image of the right camera 2a and the captured image of the left camera 2b based on the calibration data 15 stored in the memory. The calibration data 15 is data for correcting distortion and angular deviation of the lenses of the two right cameras 2a and 2b in advance before executing measurement.

  Next, prefiltering is performed by the prefilter unit 12 on the stereo-parallelized image (hereinafter referred to as “parallelized image”), and then the corresponding point search unit 13 performs corresponding point search. Executed.

  First, in the prefilter unit 12, in order to absorb a luminance difference between the left and right images, a local luminance level variation in the image, and the like, differentiation processing is performed as a prefilter and edge enhancement processing is performed. For example, a Sobel filter or a Laplacian filter is used as the prefilter.

  Next, the corresponding point search process of the corresponding point search unit 13 will be described based on FIGS. 3 and 4A and 4B. FIG. 3 is an explanatory diagram illustrating a method for executing corresponding point search processing on two parallelized images according to the first embodiment of the present invention. 4A is an explanatory diagram showing each pixel on the parallelized image of the left camera 2b according to the first embodiment of the present invention, and FIG. 4B is a diagram illustrating one pixel according to the first embodiment of the present invention. It is a graph which shows the relationship curve of the SAD value and parallax obtained with respect to an attention pixel. Note that the imaging apparatus 1 performs a process of searching for a corresponding pixel from the parallelized image of the right camera 2a for each pixel on the parallelized image of the left camera 2b with the left camera 2b as a reference.

  As shown in FIG. 3, the corresponding point search process is first called a square-shaped “correlation window” of p × p (p is an integer) pixels in the vertical and horizontal directions centering on the target pixel of the parallelized image of the left camera 2b. Set the area. Subsequently, a correlation window of the same size is set in the same vertical pixel row in the parallelized image of the right camera 2a, and the pixels included in the correlation window of both images are slid in the horizontal direction of the pixel row. An evaluation value as a determination accuracy indicating the degree of correlation between the two is calculated. As the evaluation value indicating the degree of correlation, for example, an amount that can be calculated by pixel calculation such as SAD (Sum of Absolute Differrence) is used. In SAD represented by the following formula, the smaller the value, the higher the degree of correlation.

  Here, for example, as shown in FIGS. 4A and 4B, from the left end to the right end of the predetermined corresponding point search range (in FIGS. 4A and 4B, the left end of the parallel image of the left camera 2b). To the right end), the center pixel position of the correlation window of the parallelized image of the right camera 2a in the state where the degree of correlation is highest (the SAD value is the smallest) when searching is the target pixel of the parallelized image of the left camera 2b Specify a corresponding point for. In the present embodiment, as shown in FIG. 4B, the position xk (k is an integer) is the corresponding position.

  Subsequently, a parallax value is calculated from the information on the obtained corresponding points. Parallax is defined as the distance between the pixel of interest of the parallelized image of the left camera 2b and the corresponding pixel of the parallelized image of the right camera 2a expressed in units of pixels in the horizontal direction. That is, as shown in (b) of FIG. 4, the parallax “xi−xk (where i and k are integers)” is represented.

  Next, distance image generation processing by the distance image generation unit 14 is performed. Specifically, the three-dimensional coordinates (X, Y, Z) of the object are calculated from the obtained parallax value by the principle of triangulation. Note that the three-dimensional coordinates (X, Y, Z) are a distance X in the horizontal direction, a distance Y in the vertical direction, and a distance in the depth direction from the center of the imaging surface of the left camera 2b as a reference.

  For example, when measuring the distance Z in the depth direction to the measurement object,

Can be calculated.

  The three-dimensional measuring unit 3 obtains a distance value Z for each pixel of the parallelized image of the left camera 2b, which is a reference, and generates a distance image of the subject by using the pixel value of the pixel. As described above, the distance image may include a pixel having an erroneous measurement value (error distance value) different from the actual distance value (Z value). A method for reducing such erroneous measurement values will be described below with reference to FIG. FIG. 5 is a diagram for explaining the erroneous measurement value reduction process executed by the imaging apparatus 1 according to the first embodiment of the present invention.

  FIG. 5A shows an example of a distance image of the subject (human face) generated by the three-dimensional measurement process. Each pixel included in the distance image has a three-dimensional coordinate value (X value, Y value, and Z value). In FIG. 5A, the magnitude of the Z value of each pixel is grayscale. It shall be expressed as The smaller the Z value of a pixel, the closer the location of the subject corresponding to that pixel is to the imaging unit 2. If a pixel having a smaller Z value is expressed in white on the drawing, the distance image shown in FIG. 5A originally represents the nose portion more white (brighter) than the cheek portion. It should be simplified in this figure. In FIG. 5A, a pixel having an erroneous measurement value in the distance image is shown as measurement noise Ni (i is a natural number).

  In the example shown in FIG. 5A, the measurement noise Ni exists in both the face area and other areas in the distance image. First, the imaging apparatus 1 reduces all measurement noises included in regions other than the face region by extracting the face region from the distance image. This procedure will be described below.

  In the present embodiment, the face area extraction unit 4 extracts a face area from the distance image. At that time, the face area extraction unit 4 first detects a two-dimensional face area from at least one of the plurality of captured images obtained by the imaging unit 2, and then uses the detection result to obtain the distance. A three-dimensional face region is extracted from the image. Below, the detail of the face area extraction by the face area extraction part 4 is demonstrated first.

  The face area extraction unit 4 detects the face area of the subject from one of a plurality of captured images obtained by the imaging unit 2. Hereinafter, a captured image obtained by the left camera 2b is used. Since the captured image is a two-dimensional image before the three-dimensional measurement process is applied, each pixel in the captured image has color information representing the color of the subject. The face area extraction unit 4 detects the face area of the subject from the captured image based on the color information of each pixel in the captured image. Specifically, skin color detection processing is executed to detect a face area from the captured image. More specifically, the face area extraction unit 4 has skin color information included in the distance image based on a color information pattern (for example, a ratio of R value, G value, and B value) characteristic of human skin color. A face area is detected from the captured image by selecting each pixel. FIG. 5B shows an example of the face area 31 detected from the captured image by the face area extracting unit 4. As shown in FIG. 5C, the face area extraction unit 4 extracts an area corresponding to the face area 31 detected from the captured image in the distance image as a face area 32 in the distance image.

  Note that the face area extraction unit 4 may detect a face area from the parallelized image obtained by the parallelizing unit 11 instead of the captured image obtained by the imaging unit 2. For example, since color information remains in the parallelized image obtained by the left camera 2b as in the captured image, the face area extraction unit 4 detects the face area from the parallelized image based on this color information. Further, the face area extraction unit 4 may directly extract the three-dimensional face area 32 from the distance image without using the two-dimensional face area 31.

  As shown in FIG. 5C, by extracting the face area 32 from the distance image, all measurement noise included in the area other than the face area 32 in the distance image is reduced. On the other hand, a plurality of measurement noises Ni still remain in the face area 32 extracted from the distance image. A method for reducing these measurement noises will be described below.

In general, there are individual differences in the shape of a person's face, but there is no individual difference in the arrangement of characteristic parts included in the face, such as eyes, nose, and cheeks. Is also located within a certain area of the face. The area dividing unit 5 divides the face area 32 into a plurality of divided areas 33 respectively corresponding to the human eyes, nose, and others by using such a human face feature pattern. In the memory of the imaging apparatus 1 according to the embodiment, an area division pattern for dividing the face area 32 into a plurality of different divided areas 33 as shown in FIG. In the example of FIG. 5D, the area division pattern defines a plurality of different divided areas. Each divided region 33 corresponds to a characteristic part (region) included in a human face. The area dividing unit 5 adjusts the size of the area dividing pattern so as to match the contour of the face area 32 extracted from the distance image, and then applies the template to the face area 32. The size adjustment includes, for example, matching the contour of the face region 32 with the outline of the region division pattern. The area dividing unit 5 divides the face area 32 into a plurality of divided areas 33a to 33y by applying an area dividing pattern to the face area 32 as shown in FIG.

  In order to simplify the notation, in FIG. 5E, corresponding member numbers are shown only in the divided regions 33a, 33c, and 33y among the divided regions 33a to 33y. Measurement noise Ni is included in a part of each of the divided regions 33a to 33y. Hereinafter, a method for reducing measurement noise included in each divided region 33 will be described.

  The area dividing unit 5 outputs the face area 32 divided into the plurality of divided areas 33 a to 33 y to the effective range setting unit 6. The effective range setting unit 6 sets an effective range corresponding to each divided region 33 for each divided region 33 constituting the face region 32. In the present embodiment, the effective range corresponding to each divided region 33 is recorded in advance in the memory of the imaging apparatus 1. Therefore, the effective range setting unit 6 may set an effective range corresponding to the divided area 33 recorded in the memory for the divided area 33.

  As described above, each divided region 33 included in the region divided pattern is defined based on each characteristic portion included in the human face shape. Therefore, if the subject is imaged in a state where the distance between the imaging unit 2 and the subject is kept constant, a range of probable (effective) distance values of each pixel in each divided region 33 can be calculated in advance. it can. Therefore, for each divided region 33 included in the region divided pattern, an effective distance value range corresponding to the feature portion of the human face shape corresponding to the divided region 33 is associated with the divided region 33, and the imaging apparatus It may be recorded in advance in one memory.

  An example of the effective range set in each divided region 33 will be described with reference to FIG. FIG. 5F shows a cross-sectional image of the face region 32 based on the one-dot chain line (A-A ′ line) shown in FIG. The A-A ′ line is superimposed on the divided areas 33 b, 33 d, 33 f, 33 j, 33 m, 33 n, 33 q, 33 v, and 33 y among the divided areas 33 a to 33 y. Therefore, (f) of FIG. 5 shows a cross-sectional image of these divided regions 33b, 33d,. FIG. 5F further shows the actual contour 35 of the face shape based on the statistical data of the human face shape.

  The effective range 34 recorded in advance in the memory corresponding to each divided region 33 is determined based on the contour 35. For example, since the divided region 33q is a region corresponding to the periphery of the nose, a relatively wide range is recorded in the memory as the effective range 34 corresponding to the divided region 33q. On the other hand, since the divided area 33v is an area corresponding to the periphery of the mouth, a relatively narrow range is recorded in the memory as the effective range 34 corresponding to the divided area 33v. The effective range setting unit 6 sets the effective range 34 determined based on the contour 35 in each divided region 33, thereby comparing the divided region 33v corresponding to the periphery of the nose as shown in FIG. A relatively wide effective range 34 is set, while a relatively narrow effective range 34 is set in the divided region 33y corresponding to the periphery of the mouth.

  The effective range setting unit 6 notifies the abnormal pixel determination unit 7 of the face region 32 in which the effective range 34 is set in each divided region 33. The abnormal pixel determination unit 7 is, for each divided region 33, a pixel having a distance value out of the effective range 34 set in the divided region 33 among the plurality of pixels included in the divided region 33 is an abnormal pixel. Is determined. For example, in the example of FIG. 5F, the distance value of the measurement noise Ni included in the divided area 33d is out of the effective range 34 set in the divided area 33d, so this measurement noise Ni is an abnormal pixel. It is determined. On the other hand, since the distance value of the measurement noise N1 included in the divided area 33j is within the effective range 34 set in the divided area 33j, the measurement noise N1 is not determined to be an abnormal pixel. In the example of FIG. 5F, the measurement noise Ni included in each of the divided areas 33b, 33f, 33m, 33q, and 33v is determined to be an abnormal pixel, while the measurement noise included in the divided area 33j. N1 and the measurement noise N2 included in the divided region 33y are not determined to be abnormal pixels.

  The abnormal pixel determination unit 7 executes the above-described abnormal pixel determination process for all the divided regions 33a to 33y constituting the face region 32. Thereby, each measurement noise Ni included in the divided regions 33j, 33l, 33r, and 33y shown in FIG. 5E is not determined to be an abnormal pixel. On the other hand, the measurement noise Ni included in each divided region 33 other than these divided regions 33 is determined to be an abnormal pixel.

  The abnormal pixel determination unit 7 outputs the face region 32 divided into the plurality of divided regions 33 and the abnormal pixel determination result in each divided region 33 to the interpolation unit 8. The interpolation unit 8 calculates the distance value of the measurement noise Ni by the interpolation process using the distance values of the pixel groups around the pixel determined to be the measurement noise Ni (abnormal pixel) for each divided region 33. The value is changed to a value within the effective range set in the divided area 33. Examples of the interpolation processing include expansion processing and contraction processing. By this interpolation processing, all measurement noises Ni determined to be abnormal pixels are reduced from the face area 32. FIG. 5G shows the face area 32 after the interpolation process is performed. In the example shown in this figure, all measurement noises Ni other than the measurement noises Ni included in the divided regions 33j, 33l, 33r, and 33y are reduced.

  The interpolation unit 8 outputs the face area 32 after performing the interpolation process on each measurement noise Ni determined to be an abnormal pixel to the smoothing unit 9. Pixels having distance values included in the effective range 34, such as the measurement noise N1 included in the divided region 33j and the measurement noise N2 included in the divided region 33y, are not determined to be abnormal pixels, and thus remain in the face region 32. Has been. The smoothing unit 9 reduces the measurement noise N1 and the like remaining in the face area 32 by the smoothing process for each divided area 33. Examples of the smoothing process include a Gaussian spatial filter process and a median spatial filter process.

  Through the processes described above, a three-dimensional face area 32 (face area image) in which all the measurement noise Ni included in the distance image is reduced can be acquired.

  As described above, in this embodiment, the subject is a person, and the region extracted from the distance image is a face region. However, the embodiment of the present invention is not limited to this example. For example, the subject can be any subject such as an object other than a person. In addition, the region extracted from the distance image and divided into a plurality of divided regions can be any authorized region for determining the presence or absence of abnormal pixels.

[Embodiment 2]
Embodiment 2 according to the present invention will be described below. Each member common to the above-described embodiment is denoted by the same reference numeral, and detailed description thereof is omitted.

  The effective range setting unit 6 according to the first embodiment sets the effective range 34 corresponding to each divided region 33 recorded in advance in the memory in each divided region 33. On the other hand, the effective range setting unit 6 according to the present embodiment is not the effective range 34 recorded in advance in such a memory, but a reference for setting the effective range by a predetermined calculation for each divided region 33. A value is calculated, and an effective range based on this reference value is set in the corresponding divided region 33.

  This setting method will be described below. The effective range setting unit 6 first calculates an average distance value of the distance values of each pixel included in the divided region 33 as a reference value for setting the effective range for each divided region 33, and this average distance value Is set to the divided area 33. In the present embodiment, the effective range setting unit 6 sets a predetermined range (for example, a certain range before and after) having the average distance value as the center value in the corresponding divided region 33. As a result, no matter what the subject is, an appropriate effective range according to the actual shape of the subject in each divided region 33 can be set in each divided region 33. Furthermore, it is not necessary to record the effective range 34 for each divided region 33 corresponding to the subject in the memory in advance.

[Embodiment 3]
Embodiment 3 according to the present invention will be described below with reference to FIG. The members common to the above-described embodiments are denoted by the same reference numerals, and detailed description thereof is omitted.

  The effective range setting unit 6 according to the first embodiment sets the effective range 34 in each divided region 33 on the assumption that the distance between the camera and the subject is constant (the distance is known). On the other hand, the effective range setting unit 6 according to the present embodiment sets the effective range 34 in each divided region 33 by a method corresponding to the case where the distance is unknown. The method will be described below.

  Most of the measurement noise in the distance image is generated at random positions in the three-dimensional space. Therefore, the effective range setting unit 6 obtains the maximum value and the minimum value among the distance values of each pixel included in the divided region 33 for each divided region 33 smoothed by a spatial filter such as a Gaussian spatial filter. A rough distance from the camera of each pixel in the divided area 33 can be calculated. The effective range setting unit 6 sets an effective range in the divided region 33 based on these maximum and minimum values.

  This will be described in more detail with reference to FIG. FIG. 6 is a diagram for explaining an effective range setting process executed by the imaging apparatus 1 according to the third embodiment of the present invention.

  FIG. 6A shows a cross-sectional image of the face area 32 after being divided into a plurality of divided areas 33, as in FIG. This cross-sectional image is an image before each divided region 33 is subjected to the smoothing process. In the present embodiment, the effective range setting unit 6 first instructs the smoothing unit 9 to perform the smoothing process on each divided region 33 before setting the effective range on each divided region 33.

  In response to this instruction, the smoothing unit 9 smoothes each divided region 33 before the effective range is set. FIG. 6C shows a cross-sectional image of the face area 32 after each divided area 33 has been subjected to the smoothing process. Due to the influence of measurement noise included in the distance image, the cross-sectional shape 36 in each divided region 33 after smoothing is distorted before and after the actual contour 35 of the face shape based on the statistical data of the human face shape. . The effective range setting unit 6 acquires the maximum value and the minimum value of the distance values of the pixels corresponding to the cross-sectional shape 36 in the divided region 33 for each smoothed divided region 33. Then, as shown in FIG. 6C, an effective range 37 based on these maximum and minimum values is set in the corresponding divided region 33. Thereby, an effective range 37 close to the actual face shape is set in each divided region 33.

  Processing such as abnormal pixel determination after the effective range 37 is set is the same as that in the first embodiment, and a detailed description thereof will be omitted.

[Embodiment 4]
Embodiment 4 according to the present invention will be described below. Each member common to the above-described embodiment is denoted by the same reference numeral, and detailed description thereof is omitted.

  In the present embodiment, the effective range setting unit 6 calculates the focus distance range of the camera (the right camera 2a or the left camera 2b) provided in the imaging apparatus 1, and is effective for each divided region 33 based on the calculated focus distance range. Set the range.

  A method for calculating the focus distance range will be described below. For example, s is the subject distance (the distance to focus), f is the lens focal length, F is the F value, and δ is the allowable circle of confusion. In this case, the effective range in focus within the allowable circle of confusion with respect to the subject distance s is between the near point and the far point. However, the near point is a position closer to the camera by the distance s from the distance s, and the far point is the position far from the camera by the distance s from the distance s: Df.

At this time, the near point distance: D _n and the far point distance: D _f are shown below.

The effective range setting unit 6 calculates the range between s + D _f and s−D _n as the focus distance range of the camera. Further, the effective range setting unit 6 sets the calculated focus distance range as the effective range 34 in each divided region 33. As a result, in this embodiment, the effective range 34 with high reliability and the same range is uniformly set for all the divided regions 33. The abnormal pixel determination unit 7 determines that a pixel having a distance value outside the set effective range 34 is an abnormal pixel, as in the first embodiment.

  In the present embodiment, since the effective range 34 having the same distance value is set for all the divided regions 33, the imaging apparatus 1 does not have to divide the face region 32 into the divided regions 33. In this case, the effective range setting unit 6 regards the face region 32 as one region, and sets an effective range 34 based on the focus distance range for this one region. Therefore, the division process of the face area 32 can be omitted.

[Embodiment 5]
Embodiment 4 according to the present invention will be described below with reference to FIG. The members common to the above-described embodiments are denoted by the same reference numerals, and detailed description thereof is omitted.

  The face area extraction unit 4 according to the first embodiment uses color information (particularly skin color information) of each pixel in the captured image in order to detect a two-dimensional face area 31 from the captured image. On the other hand, the face area extraction unit 4 according to the present embodiment detects the face area 31 from the captured image by collating the feature pattern of the subject recorded in advance with the captured image.

  This detection method will be described below with reference to FIG. FIG. 7 is a diagram illustrating the face area 31 detection processing executed by the imaging apparatus 1 according to the fifth embodiment of the present invention.

  FIG. 7A shows a two-dimensional captured image obtained by the imaging unit 2. The face area extraction unit 4 performs a low resolution process on the captured image to generate a low resolution image shown in FIG. A face feature pattern shown in FIG. 7C is recorded in the memory of the imaging apparatus 1 in advance. This face feature pattern has a contour 41, a feature 42, a feature 43, and a feature 44. Features 42 and 43 correspond to a person's right eye and left eye, respectively, and feature 44 corresponds to a person's mouth.

  The face area extraction unit 4 collates the low-resolution image shown in FIG. 7B with the face feature pattern shown in FIG. An example of the collation here is pattern matching. Thereby, the matching positions of the features 42 to 44 in the low resolution image are obtained, and based on this, the corresponding positions of the contour 41 in the low resolution image are determined. The face area extraction unit 4 generates the captured image shown in FIG. 8D by synthesizing the contour 41 at the same position as the corresponding position of the contour 41 in the original captured image. The face area extraction unit 4 extracts the area within the contour 41 in the captured image shown in FIG. 8D as the two-dimensional face area 31 of the subject.

[Summary]
An imaging apparatus according to aspect 1 of the present invention captures a subject from a plurality of different viewpoints, and generates a plurality of images corresponding to each of the viewpoints, and performs a three-dimensional measurement process on the plurality of images. A three-dimensional measurement unit that generates a distance image, an extraction unit (face region extraction unit 4) that extracts a determination region in the distance image, and a division unit (region) that divides the determination region into a plurality of different division regions A dividing unit 5), a setting unit (effective range setting unit 6) for setting an effective range corresponding to the divided region for each divided region, and a plurality of pixels included in the divided region for each divided region. Among these, a determination unit (abnormal pixel determination unit 7) that determines that a pixel having a distance value outside the effective range set in the divided region is an abnormal pixel is provided.

  According to said structure, an imaging device sets the effective range according to each division area to each division area which comprises the determination area extracted in the distance image. Then, a pixel having a distance value out of the effective range set in the divided area among the plurality of pixels included in the divided area is determined as an abnormal pixel.

  In this way, since the imaging apparatus determines the presence or absence of abnormal pixels based on the effective range set in each divided region, unlike the prior art, it is not affected by the shape of the subject's three-dimensional model. Further, since the effective range corresponding to the divided area is set as the divided area, the presence or absence of abnormal pixels in the divided area can be determined with high accuracy regardless of the distance and orientation of the subject with respect to the imaging unit.

  As described above, the imaging apparatus can further increase the determination accuracy of abnormal pixels included in the distance image.

  In the imaging device according to aspect 2 of the present invention, in the aspect 1, the dividing unit divides the determination area into a plurality of different divided areas based on a pre-recorded area division pattern and the determination area. It is characterized by.

  According to said structure, a determination area | region can be appropriately divided | segmented into a some division area only by preparing an area division pattern previously.

  In the imaging device according to aspect 3 of the present invention, in the aspect 2, the division unit compares the outline of the area division pattern recorded in advance with the outline of the determination area, thereby making the determination areas different from each other. It is characterized by dividing into divided areas.

  According to the above configuration, the determination area can be divided into a plurality of divided areas according to the area division pattern regardless of the size of the determination area.

  The imaging device according to aspect 4 of the present invention is the imaging apparatus according to any one of the aspects 1 to 3, wherein the extraction unit extracts the determination region based on color information of each pixel included in at least one of the plurality of images. It is characterized by doing.

  According to said structure, a determination area | region can be extracted correctly from a distance image.

  In any one of the above-described aspects 1 to 3, the extraction unit according to the aspect 5 of the present invention compares the feature pattern of the subject recorded in advance with at least one of the plurality of images. The determination area is extracted.

  According to the above configuration, even when it is difficult to extract the determination area based on the color information due to the influence of the background color or the like, the determination area can be accurately extracted from the distance image.

  The imaging apparatus according to Aspect 6 of the present invention is the imaging device according to any one of Aspects 1 to 3, wherein, for each of the divided areas, a reference value for calculating a predetermined reference value using distance values of a plurality of pixels included in the divided area. A value calculating unit (effective range setting unit 6) is further provided, and the setting unit sets the effective range based on the reference value for each of the divided regions.

  According to the above configuration, an appropriate effective range according to the actual shape of the subject in each divided region can be set in each divided region.

  In the imaging device according to aspect 7 of the present invention, in the aspect 6, the reference value calculation unit calculates, for each divided area, an average distance value of distance values of a plurality of pixels included in the divided area as the reference value. And the setting unit sets the effective range based on the average distance value for each of the divided regions.

  According to the above configuration, an appropriate effective range according to the actual shape of the subject in each divided region can be set in each divided region.

  An imaging apparatus according to an eighth aspect of the present invention provides the imaging device according to any one of the first to fifth aspects, wherein for each of the divided regions, a distance value of a plurality of pixels included in the divided region is smoothed using a spatial filter. The setting unit is effective for each of the divided regions based on the maximum value and the minimum value of the plurality of distance values of the plurality of pixels included in the smoothed divided region. It is characterized by setting a range.

  According to the above configuration, an appropriate effective range according to the actual shape of the subject in each divided region can be set in each divided region.

  An imaging apparatus according to an aspect 9 of the present invention includes a focus distance range calculation unit that calculates a focus distance range in the imaging apparatus according to any one of the aspects 1 to 5, and the setting unit is provided for each of the divided regions. The effective range is set based on the focus distance range.

  According to said structure, even if the distance of a to-be-photographed part and an imaging part is unknown, an appropriate effective range can be set to each division area.

  The imaging method according to the tenth aspect of the present invention is to capture a subject from a plurality of different viewpoints to generate a plurality of images corresponding to each of the viewpoints, and to perform a three-dimensional measurement process on the plurality of images. For each of the divided regions, a three-dimensional measuring step for generating a distance image, an extracting step for extracting a determination region in the distance image, a dividing step for dividing the determination region into a plurality of different divided regions, and A setting step of setting an effective range according to the divided area, and a pixel having a distance value out of the effective range set in the divided area among the plurality of pixels included in the divided area for each of the divided areas. And a determination step of determining that the pixel is an abnormal pixel.

  According to said structure, the determination precision of the abnormal pixel contained in a distance image can be improved more.

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims. Embodiments obtained by appropriately combining technical means disclosed in different embodiments are also included in the technical scope of the present invention. Furthermore, a new technical feature can be formed by combining the technical means disclosed in each embodiment.

  The present invention can be used for an imaging apparatus and an imaging method using a stereo camera.

DESCRIPTION OF SYMBOLS 1 Imaging device 2 Imaging part 2a Right camera 2b Left camera 3 Three-dimensional measurement part 4 Face area extraction part (extraction part)
5 Area division part (division part)
6 Effective range setting unit (setting unit, average distance value calculating unit, reference value calculating unit, focus distance calculating unit)
7 Abnormal pixel determination unit (determination unit)
DESCRIPTION OF SYMBOLS 8 Interpolation part 9 Smoothing part 11 Parallelizing part 12 Pre filter part 13 Corresponding point search part 14 Distance image generation part 15 Calibration data

Claims (10)

An imaging unit that shoots a subject from a plurality of different viewpoints to generate a plurality of images corresponding to each of the viewpoints;
A three-dimensional measurement unit that generates a distance image by performing a three-dimensional measurement process on the plurality of images;
An extraction unit for extracting a determination region in the distance image;
A dividing unit that divides the determination area into a plurality of different divided areas;
For each of the divided areas, a setting unit that sets an effective range according to the divided area;
A determination unit that determines, for each of the divided regions, a pixel having a distance value out of the effective range set in the divided region among a plurality of pixels included in the divided region as an abnormal pixel; An imaging apparatus characterized by comprising:   The imaging apparatus according to claim 1, wherein the division unit divides the determination area into a plurality of different division areas based on a pre-recorded area division pattern and the determination area.   3. The division unit according to claim 2, wherein the division unit divides the determination region into a plurality of different division regions by collating an outline of a region division pattern recorded in advance with an outline of the determination region. Imaging device.   The imaging apparatus according to claim 1, wherein the extraction unit extracts the determination region based on color information of each pixel included in at least one of the plurality of images. .   The extraction section extracts the determination region by comparing a feature pattern of a subject recorded in advance with at least one of the plurality of images. The imaging device described in 1. Each of the divided regions further includes a reference value calculation unit that calculates a predetermined reference value using distance values of a plurality of pixels included in the divided region,
The imaging apparatus according to claim 1, wherein the setting unit sets the effective range based on the reference value for each of the divided regions. The reference value calculation unit calculates, for each of the divided regions, an average distance value of distance values of a plurality of pixels included in the divided region as the reference value,
The imaging apparatus according to claim 6, wherein the setting unit sets the effective range based on the average distance value for each of the divided regions. Each of the divided regions further includes a smoothing unit that smoothes the distance values of a plurality of pixels included in the divided region using a spatial filter,
The setting unit sets an effective range for each of the divided regions based on a maximum value and a minimum value among a plurality of the distance values of a plurality of pixels included in the smoothed divided region. The imaging device according to any one of claims 1 to 5. A focus distance range calculation unit for calculating a focus distance range in the imaging apparatus;
The imaging apparatus according to claim 1, wherein the setting unit sets the effective range for each of the divided regions based on the focus distance range. An imaging step of generating a plurality of images corresponding to each of the viewpoints by photographing the subject from a plurality of different viewpoints;
A three-dimensional measurement process for generating a distance image by performing a three-dimensional measurement process on the plurality of images;
An extraction step of extracting a determination region in the distance image;
A dividing step of dividing the determination area into a plurality of different divided areas;
For each of the divided areas, a setting step for setting an effective range according to the divided area;
Determining a pixel having a distance value out of the effective range set in the divided area among the plurality of pixels included in the divided area as an abnormal pixel for each of the divided areas. An imaging method characterized by the above.

Images