JP2629200B2 - Image processing device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION The present invention will be described in the following order.

A Industrial application field B Outline of the invention C Conventional technology D Problems to be solved by the invention E Means for solving the problems (FIGS. 1, 2 and 5)
(FIG. 7, FIG. 7, FIG. 20) F function (FIGS. 1, 2, 5, 5, 7, 20) G example (G1) Configuration of imaging section (FIGS. 1 to 5) (G2) Configuration of image data processing device (FIGS. 1, 5, and 6) (G3) Configuration of feature amount detection circuit (FIGS. 1, 6 to 13)
(G4) Configuration of corresponding point detection circuit (FIG. 1, FIGS. 14 to 19)
(G5) Configuration of depth map creation circuit (Fig. 1, Fig. 20 to Fig. 30)
(G6) Configuration of moving image analysis and recognition device (FIGS. 1 and 31) (G7) Operation of embodiment (FIGS. 1 to 31) (G8) Effect of embodiment (G9) Other embodiments Example H Effects of the Invention A Industrial Field of the Invention The present invention relates to an image processing device, and is suitably applied to, for example, a shape recognition device that recognizes the external shape of a measured object.

B. Summary of the Invention The present invention provides an image processing apparatus, which averages parallax data obtained based on a feature amount of a microscopic region of a captured image stereoscopically obtained in each microscopic region, and then sets the parallax data in a horizontal scanning direction. In addition, by performing processing so as to be continuously changed between the vertical scanning direction and the frame to obtain the examination information, it is possible to reliably obtain the depth information even in a natural moving image.

C Prior Art Conventionally, in this type of shape recognition device, depth information up to the imaging device is obtained for each minute area of a captured image obtained through the imaging device by stereoscopically viewing the object to be measured. Japanese Patent Laid-Open No. 60-199292, Japanese Patent Laid-Open No. 60-199292, and Japanese Patent Application Laid-Open Nos. 60-199292 and 60-199292 disclose a method of recognizing an outer shape of a measured object based on depth information.
JP-A-60-199293).

That is, when the measurement target is stereoscopically viewed using a plurality of imaging devices, parallax occurs between the imaging devices, and the magnitude of the parallax changes according to the depth from the imaging device to the measurement target.

Accordingly, if the magnitude of the parallax is detected, depth information from the imaging device to the object to be measured can be obtained.

Furthermore, if the magnitude of parallax is detected for each minute region of the captured image obtained from the imaging device instead of the measurement target, depth information can be obtained for each minute region.

Therefore, regarding the image information obtained from the imaging device, it is possible to determine whether the image information is the image information of the measured object or the background image information based on the depth information, whereby the image information of the measured object is extracted and the outer shape is determined. Can be recognized.

Therefore, according to this method, it is possible to apply even when the object to be measured is a stationary object and to apply only a part of the object to be measured, as compared with a case where the moving image portion is extracted and the external shape of the object to be measured is recognized. When the camera moves, when the entire imaging apparatus moves, when the illumination changes, and the like, the external shape of the measured object can be recognized. Further, even when the depth from the imaging device to the measurement target changes, the actual size of the measurement target can be recognized based on the depth information.

For this reason, in this type of shape recognition device, when depth information is obtained using an image processing device, for example, a contour portion is extracted based on image information obtained from a plurality of imaging devices, and a captured image of the imaging device is extracted. After obtaining corresponding points between the corresponding points (that is, points obtained by imaging the same minute area of the object to be measured or the background, hereinafter referred to as corresponding points), the amount of displacement between the corresponding points (that is, the magnitude of parallax) ) To obtain depth information of a minute area on a captured image.

D. Problems to be Solved by the Invention However, this type of image processing apparatus is still insufficient when depth information is obtained for a natural object such as a person or a car traveling on the street as a measurement target. There was a problem.

That is, in a captured image obtained by capturing such a natural object (hereinafter, referred to as a natural moving image), the luminance level of the measured object changes smoothly as a whole,
In some cases, it is difficult to extract a contour, and in practice, it becomes difficult to extract corresponding points from a natural moving image obtained through an imaging device, or conversely, a plurality of corresponding points are obtained. Atsuta.

Therefore, in this case, it is practically difficult to obtain correct depth information, and there has been a problem that it is difficult for a shape recognition device to accurately recognize an external shape.

The present invention has been made in consideration of the above points, and has as its object to propose an image processing apparatus capable of reliably obtaining depth information even from a natural moving image.

Means for Solving E Problem In order to solve such a problem, in the present invention, the optical axis
A plurality of imaging devices 31R, 31C, 31L arranged so that L _R , L _C , and L _L are parallel to each other on the same plane;
R, 31C, and 31L, the feature amounts representing the features of each of the minute regions S of the minute regions S of the captured images M _R1 , M _C1 , and M _L1
Feature detection circuits PRE _R , PRE _C , P for detecting D _FR , D _FC , D _FL
RE _L , 43R, 43C, 43L and a plurality of imaging devices 31R, 31C, 31L
The same feature amount D in the arrangement direction x of the plurality of imaging devices 31R, 31C, 31L between the captured images M _R1 , M _C1 , and M _L1 obtained from
_The small area S having _FR , D _FC , and D _FL is detected, and parallax data D _F0 , D _F1 , and D between the small areas S having the same feature amount are detected.
_{D F2, D F3, D F4} , the corresponding point detection circuit 47 which outputs a D _F5, parallax data D _F0, D _F1 outputted from the corresponding point detection circuit 47, D
_{F2, D F3, D F4,} D F5 to an averaging circuit 61 for averaging for each minute area S, the averaging circuit 61 captured image averaged parallax data D _FA is outputted from the M _R1, M _C1 , M an interpolation circuit 62 for interpolating process in the horizontal scanning direction of _L1, the interpolation circuit 62 captured image interpolation processing parallax data D _FM output from M _R1,
Smoothing circuits 63 and 6 for performing smoothing processing in the vertical scanning direction of M _C1 and M _L1
4, 65, and smoothing circuits 67, 68, 69 for performing smoothing processing on the smoothed parallax data D _FF output from the smoothing circuits 63, 64, 65 between frames of the captured images M _R1 , M _C1 , and M _L1. To be equipped.

For small area S of the F acting captured image _{M R1, M C1, M L1} , the feature amount D _FR, D _FC, parallax data is detected based on the D _FL D _F0,
After averaging D _F1 , D _F2 , D _F3 , D _F4 , and D _F5 for each minute area S, a plurality of corresponding points are obtained by performing interpolation processing and smoothing processing between the horizontal scanning direction and the vertical scanning direction and between frames. Even if is detected or the corresponding point is not detected, it is possible to obtain parallax data that continuously changes in each minute area S. Therefore, depth information can be reliably obtained based on the parallax data even when the measured object is a natural object.

G Example Hereinafter, an example of the present invention will be described in detail with reference to the drawings.

In FIG. 1, reference numeral 1 denotes a shape recognizing device as a whole, which is capable of recognizing the external shape of a walking natural person as a measurement object.

That is, in the image processing device 2, the image data D _AR , D _AC, and D _AL of the natural moving image captured by the imaging unit 3 are received by the image data processing device 4, and the natural moving image is divided into minute regions, and the depth information is obtained. Get.

Further, based on the depth information, the image data processing device 4
, A depth map corresponding to a natural moving image representing the depth of each minute region is obtained, and the moving image analysis and recognition device 5 can recognize the external shape of the measured object based on the depth map.

(G1) Configuration of Imaging Unit As shown in FIG. 2, the imaging unit 3 includes optical axes L _R , L _C, and L _L.
Are parallel to each other on the same plane, and the object to be measured is stereoscopically viewed by three imaging devices 31R, 31C and 31L arranged at equal intervals in the horizontal direction x.

That is, the sub-base 33C is fixed at the center position of the base 32, and the imaging device 31C is held on the sub-base 33C via the tilt correction mechanism 34C.

Tilt correction mechanism 34C is made an optical axis L _C of the imaging device 31C so as to rotate as the pivot center, the inclination correction mechanism 34C
By manipulating, it is configured so as to be able to correct the tilt in the horizontal direction x around the optical axis L _C of the imaging device 31C.

On the other hand, at positions equidistant to both sides of the sub-base 33C, the sub-base 33R is configured to be rotatable around the vertical axis of the imaging surface of the imaging devices 31R and 31L, respectively.
An elevation angle correction mechanism 35 is provided on the sub-bases 33R and 33L so as to be rotatable around the horizontal axes of the imaging surfaces of the imaging devices 31R and 31L, respectively.
R and 35L are installed.

Further, on the elevation angle correcting mechanisms 35R and 35L, an imaging device 31R is provided.
And 31L of the optical axis L _R and L _L imaging device through the tilt correction mechanism 34R and 34L has been made so as to rotate as the pivot center to
31R and 31L are held at the same height position as that of the imaging device 31C.

Accordingly, the sub-bases 33R and 33L and the elevation correction mechanism 35R are respectively provided.
And 35L, the imaging devices 31R and 31L
Can be adjusted so that the optical axes L _R and L _C of the image pickup device 31C are parallel to the same plane as the optical axis L _C of the imaging device 31C.
By operating R and 34L, the inclination of the horizontal axis of the imaging surface with respect to the imaging device 31C can be corrected.

Further, in the imaging devices 31R, 31C, and 31L, the position of the raster scanning is adjusted so that the imaging screen can be variably adjusted in a small range in the horizontal scanning direction and the vertical operation direction.

As shown in FIG. 3, when the imaging devices 31R, 31C, and 31L are arranged in this way, the apertures of the imaging devices 31R, 31C, and 31L are reduced.
From E _R , E _C, and E _L , the measured object P and the imaging surface M _R , M _C, and M _L
When the depth and the distance to are set as values F and f, respectively, and the horizontal distance from the optical axis L _C of the imaging device 31C grounded at the center to the object P is set as the value D, the imaging surface M _{C of the} imaging device 31C is set. in the following formula _{D / F = d C / f} ...... (1) to be measured from the center O _c of the value d _C only imaging plane M _C represented by the equation in a horizontal position moved in the horizontal scanning direction x target it can be obtained P image P _C of.

On the other hand, if the distances between the imaging devices 31R and 31C and 31C and 31L are set to values W _R and W _L , the imaging devices 31R and 31
For the 1L imaging planes M _R and M _L , the following expression (D−W _R ) / F = d _R / f (2) (D + W _L ) / F = d _L / f (3) The images P _R and P _L of the measurement object P are obtained at the horizontal position moved in the horizontal scanning direction x from the centers O _R and O _{L of the} imaging surfaces M _R and M _L by the values d _R and d _L represented by be able to.

Together by modifying the sub connexion (1) and (2) is the following relational expression _{d R -d C = -W R ·} f / F ...... (4) is obtained, (1) and (3 ), The following relational expression d _L −d _C = W _L · f / F (5) can be obtained.

Accordingly, when the depth F of the measured object P changes, the magnitude of the parallax of the imaging devices 31R and 31L on both sides with respect to the imaging device 31C installed at the center is inversely proportional to the depth F (that is, the expressions (4) and (5)). d _R -d _C and the value represented by d _L -d _C) is altered in expression, by detecting the size of the disparity, it is possible to detect the depth F to the object to be measured P.

However, in the case where the imaging apparatus is actually divided into minute regions and depth information of each minute region is obtained, occurrence of an error cannot be avoided.

In this embodiment, three imaging devices 31R, 31C and
By imaging the measurement target P using the 31L, the magnitude of parallax can be detected between the imaging devices 31R and 31C and the imaging devices 31C and 31L, and an error in the depth information can be detected. Can be prevented beforehand.

Accordance connexion as shown in FIGS. 4 and 5, on the optical axis L _C of center of the imaging device 31C the placed flat on the position of the depth F _G closer to the far almost unlimited example imaging unit 3 as the background G in case of imaging the object to be measured P ₂ spherical disposed at the position of depth F _2, in the captured image M _R1, M _C1 and M _L1, background G
Is near infinity, so the center G _{O of the} background G
And the image centers O _R1 , O _C1 and O of the image pickup devices M _R1 , M _C1 and M _L1
An image without parallax that matches _L1 can be obtained.

On the other hand, the measured object P ₂ has a shallow depth F ₂ and its center P ₂₀ is on the optical axis L _C of the central imaging device 31C.
In the captured image M _C1 of the central imaging device 31C, an image in which the image center O _C1 and the center P ₂₀ of the measured object P ₂ match is obtained, while the left and right imaging devices 31L and 31R obtain the same image. ,
Captured images M _L1 and M by depths d _L2 and d _R2 corresponding to depth F _2
R1 image center O _L1 and O _R1 position of the center P ₂₀ of the measurement object P ₂ is shifted from (i.e. resulting in the size of the disparity values d _L2 and d _R2) image is obtained.

Further, in this embodiment, the imaging device 31R, since it is arranged to 31C and 31L at regular intervals, the distance W _R and W _L, to obtain the following relational expression _{W L = W R ...... (6} ) can be, (4), it is possible to obtain a relational expression (5) based on the formula and (6), the following equation _{d L -d C = d C -d} R ...... (7).

Therefore, parallax of the same value can be obtained between the imaging devices 31R and 31C and the imaging devices 31C and 31L, and
R, upon 31C and image processing the captured image M _R1, M _C1 and M _L1 obtained from 31L, from the imaging device 31R and 31L on the captured image M _C1 obtained were placed on both sides from the center of the imaging device 31C The obtained captured images M _R1 and M _L1 can be image-processed in the same manner.

Accordingly, the overall configuration of the image processing apparatus 2 (FIG. 1) can be simplified accordingly.

That is, in FIG. 5, the captured image M _C1 obtained from the center of the imaging device 31C, the left and right image pickup devices 31L and
Be as large as d _L2 and d _R2 is equal parallax occurring in the captured image M _L1 and M _R1 obtained from 31R, the captured image M _L1 and M _R1 of the right and left with respect to the center of the captured image M _C1 like It is understood that image processing should be performed.

Further, in the imaging devices 31R, 31C and 31L, the optical axis
The inclination in the horizontal direction with respect to L _R , L _C and L _L and the imaging device 31C
Mechanical of parallel shift of the optical axis L _C imaging device 31R and 31L of the optical axis L _R and L _L respect, the optical axis L _R respectively, with respect to the horizontal axis and the vertical axis of the L _C and L _L and the imaging surface The adjustment work of the optical system of the imaging unit 3 can be simplified accordingly.

Further, by separately electrically adjusting the adjustment operation of the image center of the captured image, the above-described imaging device 31 can be used.
The optical axes L _R , L _C, and L _L of R, 31C, and 31L can be adjusted independently of the mechanical adjustment work, and thus the entire imaging unit can be adjusted with simple work.

(G2) Configuration of Image Data Processing Apparatus The image data processing apparatus 4 (FIG. 1) includes, as shown in FIG. 6, image data D output from three imaging devices 31R, 31C, and 31L (FIG. 2). _AR , D _AC and D _AL are respectively supplied to low-pass filter circuits 41R, 41C and 41L and a sampling circuit 42.
PRE _R , PRE _C and PRE composed of R, 42C and 42L
Give to _L.

Low pass filter circuit 41R, 41C and 41L is constituted by a digital filter circuit of each 2-dimensional image data D _AR, D _AC, and from the D _AL, low frequency components in a practically sufficient range for the recognition of the walking person Is extracted.

On the other hand, the sampling circuits 42R, 42C and 42L
Image data obtained in pixel units of the imaging devices 31R, 31C and 31L via the low-pass filter circuits 41R, 41C and 41L are sampled at predetermined pitches in the vertical scanning direction and the horizontal scanning direction and output.

As a result, the data amount of the image data D _LR , D _LC and D _LL output from the sampling circuits 42R, 42C and 42L can be remarkably reduced within a practically sufficient range for image recognition,
The data processing operation of the subsequent image data D _LR , D _LC and D _LL can be simplified accordingly.

Accordingly, by providing the low-pass filter circuits 41R, 41C and 41L and the sampling circuits 42R, 42C and 42L at the first stage of the image data processing device 4, the overall configuration of the image data processing device 4 can be simplified. it can.

Further, by providing the low-pass filter circuits 41R, 41C and 41L immediately before the sampling circuits 42R, 42C and 42L, the image data D _LR and D having the reduced sampling frequency via the sampling circuits 42R, 42C and 42L.
Signal components of higher frequency _LC and D _LL is able to prevent that remains was Zantsu, prevented the occurrence of an error in the data processing operations subsequent to effectively avoid the occurrence of moire due to so-called aliasing can do.

Thus, the captured images M _R1 , M _C1, and M _L1 (FIG. 5) are reduced in resolution, and the image data D _LR , D subdivided into minute regions by the sampling frequency pitches of the sampling circuits 42R, 42C, and 42L. _LC and _DLL can be obtained.

(G3) Configuration of feature amount detection circuit The feature amount detection circuits 43R, 43C, and 43L include image data D _LR , D obtained through the sampling circuits 42R, 42C, and 42L.
Based on _LC and _DLL , the direction in which the luminance level changes most in each of the micro regions of the captured images M _R1 , M _C1, and M _L1 is detected.

That is, as shown in FIG. 7, the image data D _LR, from D _LC and D _LL, respectively sampling circuits 42R, 42C and 42L
Are divided into three small areas S _{n, m} , S _{n-1, m} , S _{n-2, m} , S in the horizontal scanning direction and the vertical scanning direction.
_{n, m-1} , S _{n-1, m-1} , S _{n-2, m-1} , S _{n, m-2} , S _{n-1, m-2} , S
The image data of _{n-2 and m-2} are sequentially extracted, and the minute area S at the center is extracted.
_In which of the four diagonal directions A1 to A4 from _{n-1, m-1} the luminance level is most rapidly changed (hereinafter referred to as feature amount)
Is detected.

That feature amount detecting circuit 43R, 43C and 43L provide each image data D _LR, a D _LC and D _LL series-connected 1H delay circuit 44R and 45R, the 44C and 45C and 44L and 45L.

Further, the feature amount detection circuits 43R, 43C and 43L are four types of difference filter circuits M _1R , M _2R , M _3R and M _4R , M _1C , M _2C , M _3C , and M _4C and M, respectively, which are Robinson filter circuits
_1L, M _2L, M comprises a _3L and M _4L, respectively sampling circuit 42R, the image data D _LR output from 42C and 42L _(n), D
_{LC (n)} and DLL _(n) , 1H delay circuits 44R and 45R, 44C and 45
C and image data D _{LR (n-1)} output from 44L and 45L, D _{LR (n-2)} , D _{LL (n-1)} and D _{LC (n-2)} and D _{LL (n-1) ) And} D
_{LL (n-2)} is given.

Therefore, the differential filter circuits M _{1R to} M _4R , M _{1C to} M _4C and M
In _1L ~M _4L, respectively sampling circuits 42R, 42
Image data D _LR output from C and 42L, D _LC and D the image data _{D LR (n) ~D LR (} n of three lines in the vertical scanning direction of the captured image M _R1, M _C1 and M _L1 of _{LL -2)} , D _{LC (n) to} D _{LC (n-2)}
And D _{LL (n) to} D _{LL (n-2)} are input in parallel and the image data
By capturing D _{LR (n) to} D _{LL (n−2)} in three small areas in the horizontal scanning direction, the magnitude of the change in the luminance level in each of the directions of the arrows A1, A2, A3 and A4 is detected. .

That is, the difference filter circuits M _{1R to} M _4R , M _{1C to} M _4C and
As shown in FIGS. 8 to 11, M _{1L to} M _4L are 3 × 3 matrix-like direction difference masks M each assigned three weights in the vertical scanning direction and the horizontal scanning direction.
_M1 , _MM2 , _MM3, and _MM4 , and the direction difference mask M _M1 to
By sequentially weighting the image data D _{LR (n) to} D _{LL (n−2)} using M _M4 , the amount of change in the luminance level in the four directions corresponding to the direction difference masks M _M1 to M _M4 can be calculated. To detect.

That is, in the direction difference mask M _M1 , the amount of change in the luminance level in the direction indicated by the arrow A1 is obtained, so that a weighting value of 0 is assigned from the center position in a direction orthogonal to the arrow A1 and the arrow A1 A weighting value from value-2 to value2 is assigned to the direction.

On the other hand, in the direction difference mask M _M2 , the arrow A2
The amount of change in the luminance level in the direction indicated by is obtained,
A value 0 is assigned in a direction orthogonal to the arrow A2, and a weighting value from −2 to 2 is assigned in the direction of the arrow A2.

On the other hand, in the direction difference masks _MM3 and _MM4 ,
The amount of change in the luminance level in the direction indicated by arrows A3 and A4 is obtained, and the direction difference mask M _{M1 is used.}
And weighting the amount of in a direction opposite to that of the M _M2 from value of -2 to the value 2 is assigned.

The difference filter circuits M _{1R to} M _4R respectively apply three image data in the horizontal scanning direction and the vertical scanning direction to the direction difference mask.
After weighting using M _{M1 to} M _M4 , the sum is output as direction difference mask data.

Therefore, when the brightness levels of the minute regions S _{n−2, m−2 to} S _{n, m} are equal, the four types of direction difference data have the same value, for example, the brightness level gradually increases in the direction A1. Is increasing, the direction difference data obtained by using the direction difference mask M _{M1 in} which the weighting amount of the largest value is assigned to the direction A1 becomes the largest value compared to the other direction difference data. .

By comparing the values of the direction difference data obtained through the Supporting connexion each differential filter circuit M _1R ~M _4L, direction A1~A4
It can be detected in which direction the luminance level changes the most.

Comparator circuit 46R, 46C and 46L are each differential filter circuit M _1R ~M _4R, it receives the direction difference data outputted from the M _1C ~M _4C and M _1L ~M _4L, the largest value than the value The obtained direction is represented by feature amount data D _FR , D _FC representing the feature amount of the center minute region S _{n−1, m−1} among the minute regions S _{n−2, m−2 to} S _{n, m.}
And _DFL .

Further, at this time, in the case where the direction difference data having the largest value cannot be obtained in the direction difference data (for example, when the luminance level is equal between the minute regions S _{n−2, m−2 to} S _{n, m} , When the change in the brightness level is the largest in the directions indicated by B1 to B4 (FIG. 7), the feature amount of the minute area _{Sn-1, m-1} is output as no feature.

Therefore, since the background G is a plane as shown in FIG. 12 corresponding to FIG. 5, the luminance level is almost the same over almost the entire surface. , The feature amount of the background G portion is detected.

On the other hand, as shown in FIG. 13, the object P _{2 to} be measured is illuminated from the imaging unit 3 side (FIG. 4), and the object P _{2 is} measured.
Most luminance level is high, in the case where the subordinate connexion brightness level toward the periphery have been made to decrease the border horizontal axis and vertical axis passing through the center P ₂₀ of the measurement object P ₂ at the center P ₂₀ of the And the arrow A in the direction toward the center P ₂₀
An area where the luminance level changes most in the directions indicated by 1 to A4 is obtained.

Accordingly, when the characteristic amounts obtained via the comparison circuits 46R, 46C and 46L are represented by values 1, 2, 3 and 4 corresponding to the arrows A1, A2, A3 and A4, respectively, the region with no characteristic amount 0 Then, the circular shape is divided into four parts to obtain areas in which the feature values of the values 1, 2, 3, and 4 are detected, respectively.

In this embodiment, the corresponding point of each minute area is detected based on the feature value of the values 0 to 4 to obtain depth information.

In practice, the direction in which the change in the luminance level is the largest is added to the arrows A1 to A4 and the directions of the arrows B1 to B4 (FIG. 7)
In this case, the corresponding point may be erroneously detected in a natural moving image. Therefore, in this embodiment, four points are obliquely detected. 5 directions in total
By detecting two feature amounts, a corresponding point of a minute area is detected.

Thus, comparator circuit 46R, 46C and through 46L, the captured image M _R1, M _C1 and M _L1 was divided into small regions, the captured image M _R1 to respectively assign the five feature values including the featureless, Three feature distribution maps M _FR and M _FR1 corresponding to M _C1 and M _L1 ,
It can be obtained M _FC and M _FL.

Thus, if the corresponding point is detected using the direction in which the luminance level changes most as a feature amount, the signal component of a significantly lower frequency as compared with the case where the contour is extracted and this is used as the feature point And the configuration of the entire image processing apparatus 4 can be simplified accordingly.

Furthermore, since the feature can be reliably detected even when a natural object for which contour extraction is difficult to be measured is used, if corresponding points are obtained based on the feature, even in the case of a natural moving image, Depth information.

Further, a method of using a luminance level as a feature amount is also conceivable, but in such a case, it is necessary to correct variations in the entire signal level of image data obtained from the imaging device, and according to this embodiment, The feature amount can be detected with a simple configuration as a whole.

(G4) Configuration of Corresponding Point Detection Circuit As shown in FIG. 14, the corresponding point detection circuit 47 outputs the characteristic amount data D _FR , D _FC and D _FL output from the characteristic amount detection circuits 43R, 43C and 43L, respectively. The corresponding points are sequentially received, and the corresponding points are detected based on the received data, and the parallax data of the corresponding points is detected.

That is, three imaging devices 31R, 31C, and the captured image obtained through the M _R1, M _C1 and M _L1 (FIG. 5) is 31L, depth F _G is almost limitless from the imaging section 3 (FIG. 4) Since the image of the background G arranged at the large position is obtained at the same position, the three imaging devices 31R and 3 are used for an area of the background G where no parallax occurs (hereinafter, referred to as an area of zero parallax).
Image data D _AR , D _AC and D _AL (FIG. 6) can be obtained from 1C and 31L at the same timing.

Therefore, feature amount data D obtained from the central imaging device 31C
Against _FC, if so, respectively to compare the feature data D _FR and D _FL obtained at the same timing from the right and left imaging devices 31R and 31L, it is possible to detect a region of disparity 0.

On the other hand, in an area where the depth F is small, the parallax increases by that amount, so that the right and left imaging devices 31R
And in the captured image M _1R and M _1L of 31L, the display position of the small area of the depth on the captured image M _C1 obtained from the center of the imaging device 31C, shifted by parallax to the left and right positions The area with the small depth is displayed.

Therefore, in the area having the small depth, the image pickup device 31C at the center by an amount corresponding to the area being displayed shifted left and right.
It can be obtained respectively proceeds and delayed image data D _AR and D _AL from the right and left imaging devices 31R and 31L at the timing the image data D _AC obtained from.

Therefore, at the same timing, the feature data D _FR , D _FC and
By comparing D _FL, whereas it is possible to detect a region of disparity 0, the right and left imaging in delayed and advanced timing with respect to the feature amount data D _FC respectively obtained from the center of the imaging device 31C if to compare the device 31R and the feature data D _FR and D _FL obtained from 31L, it is possible to detect an area of a parallax corresponding to the delay and advance timing was.

Based on this measurement principle, the corresponding point detection circuit 47
The feature data D _FR , D _FC, and D _FL obtained from the two imaging devices 31R, 31C, and 31L are compared at the same timing, respectively, and the feature data D obtained from the central imaging device 31C are obtained.
Feature data D obtained from the right and left imaging devices 31R and 31L at a timing shifted by a predetermined amount with respect to the _{FC FR} and D
Compare _FL .

That corresponding point detecting circuit 47, comparator circuit via a delay circuit 48C of the feature amount data D _FC obtained from the center of the imaging device 31C is a delay time value dt ₀ 49R0,49R1,49R2,49R3,49R
4, 49R5 and 49L0, 49L1, 49L2, 49L3, 49L4, 49L5.

Each delay circuit 48R0,48R1,48R2,48R feature amount data D _FR derived from the right side of the image pickup device 31R contrast
Comparison circuit 49R0, 49R1, 49R via 3, 48R4, and 48R5
2, 49R3, 49R4 and 49R5, and the left imaging device 3
The delay circuit 48L feature amount data D _FL obtained from 1L
Comparison circuit 4 via 0, 48L1, 48L2, 48L3, 48L4 and 48L5
Give to 9L0, 49L1, 49L2, 49L3, 49L4 and 49L5.

The delay time of the delay circuit 48R0 and 48L0 are set to the same delay time dt ₀ and the delay circuit 48C, a delay circuit 48R contrast
1 is such that the delay time dt ₀ + Δdt is longer than the delay time dt ₀ by a predetermined time Δdt.

Further delay circuit 48R2 is twice the delay time difference .DELTA.DT delay circuits 48R0 and the delay time of 48R1 dt ₀ and dt ₀ + Δdt 2Δdt
The delay time dt ₀ + 2Δdt is made longer than that of the delay circuit 48R0, and then the delay times of the delay circuits 48R3, 48R and 48R5 are made longer by the values 3Δdt, 4Δdt and 5Δdt.

Delay circuit 48L1,48L2,48L3,48L the feature amount data D _FL obtained from the left side of the imaging device 31L contrary is input
4 and 48L5, the delay circuit 48R1,48R2,48R to the delay circuit 48R0 contrary to the delay circuit 48R1,48R2,48R3,48R4 and 48R5 feature data D _FR derived from the right side of the image pickup device 31R
3,48R4 and the delay time of _{48R5 dt 0 + Δdt, dt 0} + 2Δdt, d
Delay circuit due to extension of t ₀ + 3Δdt, dt ₀ + 4Δdt and dt ₀ + 5Δdt (Δdt, 2Δdt, 3Δdt, 4Δdt, 5Δdt)
Delay time for the delay time dt ₀ of 48LO dt ₀ -Δdt, dt ₀ -
2Δdt, dt ₀ −3Δdt, dt ₀ −4Δdt and dt ₀ −5Δdt are sequentially reduced.

On the other hand, the comparison circuits 49R0 to 49L5 output the characteristic amount data input via the delay circuits 48C and 48R0 to 48L5, respectively.
When the feature amounts of D _FR , D _FR and D _FL match, the parallax data D _A4R , D _AIR , whose logic level rises to logic “1”
D _A2R, D _A3R, D _A4R and D _{A5R and, D A0L, D AIL, D} A2L,
D _A3L, and outputs the D _A4L and D _A5L.

As a result, delay circuits 48R0, 48C and 4 having the same delay time
Through the 8L0, the three imaging devices 31R, 31 at the same timing
The feature amount data D _FR , D _FC, and D _FL obtained from C and 31L are obtained, and by comparing the feature amounts with the comparison circuits 49R0 and 49L0, an area with zero parallax can be detected.

That is, as shown in Figure 15, the feature distribution diagram M _FR and M
And _FC, by superimposing M _FC and M _FL (Figure 12) per comparing the feature amounts, consistent results were obtained, respectively feature values in a region other than the region of the circular shape of the values 1 to 4, thus Background The corresponding point can be detected in the minute area of G, and the parallax data of parallax 0 can be obtained for the area (FIGS. 15A and 15B).

Via the delay circuit 48R1 and 48L1 hand, to obtain a characteristic amount data D _FR and D _FL in delay and advanced timing than the feature amount data D _FC obtained from the delay circuit Δdt only delay circuit 48C Can be.

Accordingly, by obtaining the comparison result via the comparison circuits 49R1 and 49L1, a corresponding point of a region where a parallax having a size corresponding to the delay time Δdt has occurred (hereinafter, referred to as a region of parallax 1) is detected, and the corresponding point of the region is detected. Parallax data can be obtained.

Further delay circuits 48R2 and the feature at a timing shifted by the delay time 2Δdt through 48L2 data D _FR and D
_FL can be obtained, and corresponding points and parallax data of a region (hereinafter, referred to as a region of parallax 2) in which parallax having a size corresponding to the delay time 2Δdt has occurred can be detected via the comparison circuits 49R2 and 49L2. it can.

Thus, through the comparison circuits 49R0, 49R1, 49R2, 49R3, 49R4, and 49R5, from the imaging devices 31C and 31R disposed at the center and right in the parallax 0, the parallax 1, the parallax 2, the parallax 3, the parallax 4, and the parallax 5, respectively. corresponding point and parallax data D _A0R of the obtained feature amount data D _FC and _{D FR, D A1R, D A2R} , D
_A3R, it is possible to obtain the D _A4R and D _A5R.

Similarly, the comparison circuits 49L0, 49L1, 49L2, 49L3, 49L4, and 49
Disparity 0, disparity 1, disparity 2, disparity 3,
In the parallax 4 and the parallax 5, the imaging devices 31 on the center and the left side
C and the corresponding points of the feature amount data D _FC and D _FL obtained from 31L and parallax data _{D A0L, D A1L, D A2L} , D A3L, D A4L and
D _A5L can be obtained.

Here, in this embodiment, the area of the parallax 2 corresponding to the delay time 2Δdt is a spherical object P ₂ (FIG. 4).
Is adapted to substantially match the parallax caused Te depth F ₂ Niyotsu locations arranged.

Accordance connexion as shown in FIG. 16 corresponds to FIG. 12, in the feature data D _FR and D _FL is input to the comparison circuit 49R2 and 49L2, its feature distribution diagram M _FR, M _FC and M _FL feature data is input at a timing parallax delay time 2Δdt is shifted in the direction of reducing the (Figure 16 (a), (B) and (C), the object to be measured P ₂ of an image to be in the region of the parallax 2 position can be obtained feature amount distribution diagram M _FR and M _FL of timing coincides with the position of the obtained feature amount distribution diagram M _FC from the center of the imaging device 31C.

As shown in accordance connexion FIG. 17, it is possible to detect the corresponding points in the object to be measured P ₂ parts comprising a circular shape via a comparator circuit 49R2 and 49L2 (FIGS. 17 (A) and (B)) .

However, in practice, when the captured image is divided into minute regions and represented by five feature amounts, when five feature amounts are assigned at an equal ratio to the entire minute region, the parallax is different. However, a minute area having the same feature amount as one minute area can be obtained with a probability represented by the following equation Q _M = 1/5 (8).

As shown in FIG. 18 and FIG. 19 correspond to practice Figures 16 and FIG. 17, in a case of detecting the corresponding points for the region of the parallax 1, the object to be measured P ₂ parallax magnitude of from becoming at half the size, it is obtained via the right and left imaging devices 31R and 31L respectively center of the imaging device 31C feature amount data obtained from the D _FC (FIG. 18 (B)) some of the feature data D _FR and D _FL (FIG. 18 (a) and (C)) to be measured P ₂ when comparing, matching result is obtained, that the corresponding point is erroneously detected It can be seen (FIGS. 19 (A) and (B)).

Therefore, in this embodiment, the corresponding comparison circuit 49R0
And 49L0, 49R1 and 49L1, 49R2 and 49L2, 49R3 and 49L
3,49R4 and 49L4 and 49R5 and parallax data D obtained from 49L5 _A0R and D _A0L, D _A1R and D _A1L, D _A2R and D _A2L, D _A3R and D _A3L, the D _A4R and D _A4L and D _A5R and D _A5L By inputting them to the AND circuits 50, 51, 52, 53, 54 and 55 and outputting their logical product, erroneous detection of corresponding points is prevented beforehand.

That is, if AND is obtained via AND circuits 50, 51, 52, 53, 54 and 55, the delay circuit 48R0
Feature data D _FR , D _FC and D _FL obtained from ~ 48L5
The product is obtained, and the probability of erroneous detection of the corresponding point can be reduced to a value represented by the following equation Q _M2 = 1/5 ² (9) with respect to the equation (8).

As a result, it is represented by a feature amount distribution diagram as shown in FIGS. 15 (C), 17 (C) and 19 (C) via AND circuits 50, 51, 52, 53, 54 and 55, respectively. 0, parallax 1, parallax 2, parallax 3, parallax 4, parallax data D _F0 , D _F1 , D _F2 , comprising parallax 0, parallax 1, parallax
D _F3 , D _F4 and D _F5 can be obtained.

(G5) Configuration of Depth Map Creation Circuit As shown in FIG. 20, the depth map creation circuit 60 includes parallax data D _F0 , D _F1 , D _F2 , and D output from the corresponding point detection circuit 47.
_F3, based on D _F4 and D _F5, to create a depth map comprising assigning the depth information for each minute area of the captured image M _C1 obtained through the center of the imaging device 31C.

In practice, the corresponding points detected via the corresponding point detection circuit 47 include many erroneously detected corresponding points, and it is necessary to detect correct corresponding points from this to obtain depth information.

For example, FIG. 21 shows delay circuits 48R0, 48C and 48L0, respectively.
, Six feature areas S _n , S _{n + 1} , S _{n + 2} , with a parallax of 3 as shown by the feature amount data D _FR , D _FC , D _FL obtained through
In a measured object in which S _{n + 3} , S _{n + 4} and S _{n + 5} are continuous with the feature value of value 1, even if the feature values are compared at the same timing, a triple product is obtained in the AND circuits 50 to 52. it allows in the micro region S _n ~S _{n + 5,} the corresponding point detection error is prevented.

However, as shown in FIG. 22, when the feature quantities are compared with the timing shifted by one parallax (ie, the feature quantity data D obtained via the delay circuits 48R1, 48C and 48L1).
_{Based on FR} , D _FC , and D _FL , the parallax data D _F1 obtained from the AND circuit 51), a coincidence result is obtained in the minute regions S _{n + 2} and S _{n + 3} , and a corresponding point is erroneously detected ( Hereinafter, it is called a false corresponding point).

Subsequently, when the comparison is made at the timing of the disparity 2, the disparity data D _F2 obtained from the AND circuit 52 based on the feature data D _FR , D _FC and D _FL obtained via the delay circuits 48R2, 48C and 48L2. , Respectively, the minute areas S _{n + 1} , S
Match results are obtained for _{n + 2} , _{Sn + 3} and _{Sn + 4} , and false corresponding points are detected.

As shown in FIG. 23 with respect to this, when compared with the timing of the parallax 3 (consisting i.e. by the disparity data D _F3 output from the AND circuit 53), the feature amount distribution diagram M _FR, M _FC and M _FL
There it is possible to detect six correct corresponding points between the coupling connexion, minute area S _n ~S _{n + 5} completely overlap between the micro area S _n ~S _{n + 5.}

As shown with the symbol "○" corresponding points and false corresponding point in accordance connexion FIG. 24, by comparing the feature amount sequentially switching timing disparity, in addition to the correct corresponding point P _N, respectively, small Two and four false corresponding points _PNN are obtained in the regions _{Sn + 1} and _{Sn + 4} and _{Sn + 2} and _{Sn + 3} , respectively.

FIG. 17 (C) and FIG.
Also in FIG. 19 (C), there is an area where a matching result is obtained (ie, a partial area of the background G surrounding a circular measurement target) even though comparison is not performed at the correct parallax timing.

Further Figure 25, as shown in Figure 26 and Figure 27, the measured two micro regions of the feature 1 and 0 by the disparity 3 is provided with a small area S _n ~S _{n + 9} consecutive equidistantly In the target, when the feature amounts are compared at the same timing, the minute regions S _{n + 4} and S
_{While a} false corresponding point is obtained at _{n + 5} , when the feature amount is compared at the timing of parallax 1, after a state where the corresponding point is not detected occurs, the minute regions S _{n to} Sn _{+ 9} are obtained at the timing of parallax 3. A correct corresponding point is detected.

Accordingly, in such a case, as shown in FIG. 28 corresponding to FIG. 24, the correct corresponding point _PN is obtained at the timing of the parallax 3, while the minute areas S _{n + 4} and S _{n At +5} , a false corresponding point _PNN is detected separately at the timings of parallaxes 0 and 6, respectively.

Conversely, by arranging the imaging devices 31R, 31C, and 31L in the horizontal direction, a deep portion is obscured by the measured object having a small depth, so that the corresponding point cannot be detected. (I.e., a part of the background G where the logic level is logic "0" in the crescent shape in FIG. 15 (C)).

Such a phenomenon may also occur due to specular reflection generated on the surface of the object to be measured.

In this embodiment, in order to solve such a problem, in a micro area where a plurality of corresponding points are obtained, one piece of parallax data is obtained by averaging a plurality of pieces of parallax data obtained from the corresponding points. It has been made like that.

Further, after the parallax data is interpolated in the horizontal scanning direction of the captured image, and then smoothed in the vertical scanning direction and between frames, even if there is a minute area where a corresponding point cannot be obtained, the entire image is obtained. A depth map representing a distribution of depths close to a natural moving image in which the depth continuously changes is created.

For this reason, the depth map creation circuit 60 outputs the parallax data D _{F0 to} D _F0.
F5 is _supplied to an averaging circuit 61, which averages the _results .

That is, as shown in FIG. 29, one line of parallax data D _{F0 to} D _F5 composed of minute regions S _{1 to} S ₁₀ in the horizontal scanning direction of the imaging apparatus, a plurality of pieces of parallax data D _{F0 to} D _F5 . In the small areas S ₃ , S ₄ , S ₅ and S ₉ where the logic “1” is obtained (FIG. 29 (A)), the parallax data DF ₃ and D _{F of the} intermediate value are obtained.
_F5 is extracted (FIG. 29 (B)), and thus the disparity data D
_F0 ~ it is possible to obtain a parallax data D _FA obtained by averaging the D _F5.

Interpolation circuit 62 receives the averaged parallax data D _FA,
Using a technique of the linear interpolation in the small region where the parallax data D _F0 to D _F5 has failed obtained continuously generates parallax data so parallax changes with respect to the front and rear small area (in this case small area S ₁ and S ₇ for generating a parallax data D _FM in (Figure 29 (C))), thus to generate a parallax data D _FM varying continuously the depth in the horizontal scanning direction (Figure 29 (C) ) Output to 1H delay circuits 63 and 64 connected in series.

Smoothing circuit 65, along with receiving the interpolated disparity data D _FM outputted from the interpolation circuit 62 directly receives the parallax data D _DFH1 and D _DFH2 that the delay by one line through a 1H delay circuit 63 and 64 respectively Are smoothed and output.

Against slave connexion interpolator 62 parallax data D _FH that is adapted smoothly changes in the horizontal scanning direction over, parallax data was made as smoothly changes in the vertical scanning direction via the smoothing circuit 65 D _FF Can be obtained.

Thus, the 1H delay circuits 63 and 64 and the smoothing circuit 65 _convert the parallax data _DFH, which has been subjected to the interpolation processing as a whole, into the captured images M _R1 , M
A smoothing circuit for performing smoothing processing in the horizontal scanning direction of _C1 and _ML1 is configured.

Thus, in this embodiment, the parallax data D _FF
When creating a depth map based on the three-dimensional curved surface, an elastic plate is pressed from the imaging unit side to the natural object and the background to be measured from the imaging unit side, and the elastic plate is deformed and protruded toward the imaging unit side. A model (hereinafter referred to as an elastic plate model) is formed.

That is, as shown in FIG. 30, the depth map transformation circuit 67 has a flat elastic plate model H ₍₀₎ having a depth corresponding to, for example, a parallax of 4, and the elastic plate model H ₍₀₎ (30 th The depth data D _{F (0)} (ie, corresponding to the parallax data D _FF ) shown in FIG. 7A is output to the moving image analysis and recognition device 5 (FIG. 1) and compared via the one-frame delay circuit 68. Circuit 69
Give to.

The comparison circuit 69 includes the smoothing circuit 6 together with the depth data _{DF (0).}
Receiving the parallax data D _FF (FIG. 30 (B)) obtained through the step 5, it sequentially outputs the magnitude comparison results for the respective small areas S _{1 to} S ₁₀ to the depth map transformation circuit 67.

In this case, small areas S ₁ and S ₇ of the micro areas S ₁ to S ₁₀
Are obtained, whereas the minute regions S _{2 to} S ₆
In, the parallax data D obtained via the smoothing circuit 65
Large comparison it is the depth of _FF is obtained, minute regions S ₈ ~
In S _10, a small comparison depth conversely is obtained from this.

Depth map variations circuit 67, based on the comparison result, one unit for each micro area (in this case corresponds to the parallax 0.5, parallax 3.5 parallax 4 in a microscopic region S ₂ to S ₆
To, for minute area S ₈ to S ₁₀ is _{(deformed 0)} (Figure 30 (C) the parallax 4.5) depth data D _F disparity 4), the resulting depth data D _{F (1)} ( 30 (D) is transmitted.

Thus, elastic formed by deformation processing on the basis of elastic plate model H showing a cross section in the direction of the minute regions S ₁ to S ₁₀ Te convex ₍₀₎ (Figure 30 (A) and (B)) to the depth data D _FF Board model H
₍₁₎ (FIG. 30 (D)) can be obtained.

Subsequently, a comparison result of the depth data D _{F (0)} and the disparity data D _FF (FIG. 30 (E)) of the subsequent frame obtained through the smoothing circuit 65 is obtained through the comparison circuit 69 ( while matching result is obtained in this case small regions S ₁ and S _7, a minute area S ₂ to S in ₆ it is compared depth greater result of parallax data D _FF obtained via the smoothing circuit 65 is obtained is, in the micro region S ₈ to S _10, which small comparison depth conversely is obtained as), depth data D per minute region S ₂ to S ₆ and S ₈ to S ₁₀ on the basis of the comparison result _{F (1)}
Is updated one unit at a time (FIG. 30 (F)).

Thus, in the depth data D _{F (2)} , the parallax data D _FF
An elastic plate model H ₍₂₎ (FIG. 30 (G)) deformed to the depth represented by the following equation can be obtained.

Thus by deforming processing by micro depth based elastic body model H _(n) for each frame to the parallax data D _FF, obtained by smoothing processing parallax data D _FF between frames, for example between frames due to noise or the like Even in the case where the depth data fluctuates rapidly, it is possible to obtain an elastic plate model H _(n) whose depth changes gradually and continuously, which is close to a natural moving image.

Thus, by averaging parallax data for each minute area and then performing interpolation processing, continuous parallax data _{DFH in} the horizontal scanning direction is obtained, and then smoothing processing is performed to obtain continuous parallax data in the vertical scanning direction. D _FF is obtained.

Further, the elastic body model is repeatedly deformed for each frame with respect to the parallax data _DFF , so that time-series smoothly continuous depth data _{DF (n)} can be obtained.

Therefore, in recognizing the shape of the measured object using the elastic plate model that has been repeatedly deformed, it is necessary to obtain the depth of the stationary object and to determine that the object is a moving object by changing the depth data of each minute area. Can be detected.

When the elastic plate model is gradually deformed for each frame as described above, it is inevitable that a change in depth value is actually obtained with a delay with respect to a moving object. However, in the case of recognizing the external shape of a walking person as an object to be measured, even if the elastic plate model is repeatedly deformed between frames in this manner, the object to be measured is within a practically sufficient range. Can be recognized.

Thus, the depth map transformation circuit 67, one frame delay circuit
The 68 and the comparison circuit 69 constitute a smoothing circuit that smoothes the smoothed parallax data _DFF between frames.

Thus, in the image data processing device 4,
The captured images M _R1 , M _C1, and M _L1 are divided into minute regions, and the feature amount is sequentially detected for each minute region. Subsequently, the depth data D is sequentially subjected to comparison processing via delay circuits 48R0 to 48L5.
By obtaining _{F (n)} , image data can be sequentially processed in time series, and thus depth data can be obtained at an extremely high speed. Therefore, a depth map of a captured image can be obtained using a real-time technique, and it is not necessary to provide a storage unit for image data or the like, so that an image processing apparatus having a simple configuration can be obtained as a whole. .

(G6) Configuration of Moving Image Analysis / Recognition Apparatus The moving image analysis / recognition apparatus 5 (FIG. 1) is composed of an arithmetic processing unit, and the depth data D output from the image processing apparatus 2.
The outer shape of the measured object is recognized based on _{F (n)} .

That is, as shown in FIG. 31, after the image data is obtained from the imaging unit 3 from step SP1 to step SP2, the depth data _{DF (n)} is obtained from the image processing device 2 after moving to step SP3. Then, the process proceeds to step SP4, where the depth map is cut into predetermined depth regions.

As a result, since the depth map composed of the elastic plate model is cut, a plurality of regions (hereinafter, referred to as extraction regions) obtained by sequentially cutting the depth map at equal intervals are obtained, and the depth region is set to a predetermined value. Thus, the external shape of the measured object can be cut out from the depth map.

Subsequently, the image analysis / recognition apparatus 5 moves to step SP5 to perform labeling on the extraction area, and then moves to step SP6 to analyze each extraction area.

In other words, for each labeled extraction area,
From the contour of the external shape, for example, a horizontally elongated shape, a vertically elongated shape, and the like are analyzed.

The image analysis / recognition apparatus 5 then moves to step SP6, analyzes how the labeled extraction area is deformed between frames, and detects the motion of the object to be measured, which is the extraction area.

For example, when the extraction region having a vertically elongated shape is deformed so as to become larger as a whole, it can be seen that the measured object having the shape has moved from a distant place with a large depth to a near place with a small depth.

The image analysis / recognition device 5 then proceeds to step SP7 based on the detection result, and moves to the outer shape of the extraction area indicating that the object to be measured, for example, having a vertically elongated outer shape has moved from far to near, and its movement direction. After creating GKO information representing the object, the process proceeds to step SP8 to recognize the object having the external shape.

Here, for example, in the case of a measured object whose height calculated based on the depth data is about 1.8 (m) in the case of a vertically elongated moving object, it can be understood that the measured object is an adult human, For example, it can be seen that a measured object having a length of several tens of cm elongated in the horizontal direction is a small animal.

The image analysis and recognition device then proceeds to step SP9,
Based on the recognition result described above, for example, a message indicating that the adult person has moved from right to left of the captured image is output to the printer, and the process proceeds to step SP10 to end the processing procedure.

Thus, the depth information of the measured object is obtained based on the natural moving image, and the outer shape of the measured object can be recognized based on this.

(G7) Operation of Embodiment In the above configuration, the captured images M _R1 , M _C1, and M _L1 obtained via the imaging unit 3 are converted into low-pass filter circuits 41R, 41R.
C and 41L and the sampling circuit 42R, after being divided into minute regions through 42C and 42L, the magnitude of change in the luminance level of each direction A1~A4 is detected through a differential filter circuit M _1R ~M _4L .

Comparator circuit 46R, 46C and direction in which the largest luminance level changes of each minute region from the detection result of the direction difference filter circuit M _1R ~M _4L through 46L are detected as the feature amount, the feature amount data D _FR, D _FC And _DFL .

The feature data D _FR , D _FC and D _FL are supplied to the delay circuit 48R0.
To the comparison circuits 49R0 to 49L5 via the 4848L5, and the delay circuits 48R0 to 48L in the arrangement direction x of the imaging devices 31R, 31C and 31L.
By performing the comparison processing at a timing shifted by the delay time of 5, the minute regions having the same feature amount are respectively detected as the corresponding points, and the parallax data can be obtained.

A logical product is obtained from the detection results via the AND circuits 50 to 55, and thus the timings of the feature amount data D _FR , D _FC and D _FL obtained from the three imaging devices 31R, 31C and 31L are respectively shifted. Thus, the triple product parallax data D _{F0 to} D _F5 obtained by comparing the feature amounts is obtained.

The parallax data D _F0 to D _F5, after being processed averaged over the averaging circuit 61, are interpolated in the horizontal scanning direction through the interpolation circuit 62, thus parallax data D that is continuous with the horizontal scanning direction _FH is obtained.

The parallax data D _FH is subjected to smoothing processing via the smoothing circuit 65, so that the parallax data D
_The disparity data D _FF is converted into _FF , and the disparity data D _FF is input to the comparison circuit 69 and the depth map transformation circuit 67 to generate depth data _{DF (n) that} is continuous in time series. Thus, a depth map representing a two-dimensional arrangement of depth information corresponding to a captured image can be obtained as three-dimensional curved surface data composed of an elastic plate model.

The depth data _{DF (n)} is input to the moving image analysis and recognition device 5 and extracted for each depth region, and then subjected to a labeling process to create GKO information indicating the outer shape and its movement direction.

Further, by performing arithmetic processing based on the GKO information, a message describing the measured object and its movement is output.

(G8) Effects of the Embodiment According to the above configuration, the parallax data obtained based on the feature amount of the minute region of the captured image is averaged for each minute region, and then the horizontal scanning direction and the vertical scanning direction And between frames, interpolation processing and smoothing processing are performed so that the parallax data is continuously changed, so that even when a plurality of corresponding points are obtained or when the corresponding points are not detected, etc. Parallax data of each minute area can be detected, and thus depth information of a measured object in a natural moving image can be reliably detected.

(G9) Other Embodiments (G9-1) Other Embodiments of Imaging Unit In the above-described embodiment, three imaging devices are arranged at equal intervals so that the optical axes are parallel to each other on the same plane. Although the description has been given of the case in which the arrangement is performed, the arrangement interval is not limited to the equal interval, and can be widely applied. In this case, the feature amount data may be compared by shifting the timing according to the interval between the two imaging devices for comparing the feature amounts.

Further, in the above-described embodiment, the case where three imaging devices are used has been described. However, the number of imaging devices is not limited to three, and it is only necessary to provide a plurality of imaging devices for stereoscopic viewing.

Further, in the above-described embodiment, the case where three imaging devices are arranged in a horizontal direction has been described. However, the present invention is not limited to this, and it is essential that the optical axes are arranged even in parallel on the same plane. For example, they may be arranged in the vertical direction, for example.

(G9-2) Other Embodiments of Image Data Processing Apparatus (1) Preamble Circuit In the above-described embodiment, the sampling is performed by sampling the image data at a predetermined pitch via a low-pass filter circuit and a sampling circuit. The case where the captured image is divided into minute regions by the above pitch is described, but the present invention is not limited to this, and the number of pixels of the captured image obtained from the imaging unit is sufficiently small for practical use, or a subsequent feature amount detection circuit or the like is used. If the processing speed is practically sufficiently high, the low-pass filter circuit and the sampling circuit may be omitted, and the image data may be processed in units of pixels as minute regions.

(2) Feature Amount Detection Circuit Further, in the above-described embodiment, a case has been described in which a 3 × 3 directional difference mask is used to detect the magnitude of a change in luminance level in four oblique directions to detect a feature amount. However, the present invention is not limited to this. For example, a 4 × 4 directional difference mask may be used, or a differential value of the luminance level may be used.

Furthermore, in the above-described embodiment, the case where the magnitude of the change in the luminance level is detected in four oblique directions and the direction in which the luminance level changes most as a feature amount based on this is described. The direction of change is 4
The feature amount may be detected not only in the direction but also in six directions by adding the horizontal direction to the direction.

Further, in addition to the change direction of the luminance level, for example, the luminance level is quaternized and this value is used as a feature amount.
The magnitude of the gradient of the luminance level may be used, or the frequency component thereof may be used as a feature based on image data, or the feature may be represented by combining these.

The image data used at that time is not limited to the luminance level, and the feature amount may be detected using color data or the like.

(3) Corresponding Point Detection Circuit Further, in the above-described embodiment, the case where the feature amounts of the captured images obtained from the imaging devices arranged on the right and left sides are compared with respect to the captured image obtained from the central imaging device. As described above, the present invention is not limited to this. For example, it is also possible to compare the characteristic amounts of the captured images obtained from the imaging devices arranged at the center and the right side with reference to the captured image obtained from the left imaging device. good.

Further, in the above-described embodiment, the case has been described where the feature amount data obtained from the imaging devices arranged on the right and left sides are compared with the feature amount data obtained from the central imaging device via six delay circuits. The present invention is not limited to this, and the number of delay circuits and their delay times may be selected as needed.

Furthermore, in the above-described embodiment, a case has been described in which the comparison circuit and the AND circuit are used to obtain parallax data that is a triple product of feature amount data. However, the present invention is not limited to this, and the present invention is not limited to this. It is also possible to compare feature amount data obtained from the imaging apparatus and obtain parallax data as a triple product thereof.

Furthermore, in this case, when two or more imaging devices other than three are used, for example, the output of the comparison circuit may be simply output as parallax data, and the overall configuration may be simplified accordingly. Conversely, when four imaging devices are used, for example, a quadruple product of the feature amount data may be used as the parallax data. In this case, the corresponding point is detected with higher accuracy and the parallax is further improved. Data can be obtained.

(4) Depth map creation circuit In the above-described embodiment, after parallax data is averaged, interpolation calculation processing is performed in the horizontal scanning direction, smoothing processing is performed every three lines in the vertical scanning direction, and parallax data for one frame is obtained. Although the case where data is obtained has been described, the number of lines to be input at the time of smoothing processing is not limited to this, and various values may be applied within a practically sufficient range.

Furthermore, in the above-described embodiment, a case has been described where the depth map is deformed by comparing disparity data output in units of one frame with disparity data obtained through a smoothing circuit. The invention is not limited to this, and the point is that a time-series integration effect may be added to the parallax data and output.

(G9-3) Another Embodiment of Entire Image Processing Apparatus In the above-described embodiment, the present invention is applied to a shape recognition apparatus that recognizes the external shape of a measured object and describes and outputs the measured object and its movement direction. Has been described, but the present invention is not limited to this, and can be widely applied, for example, when detecting the number of individuals based on the external shape of the measured object.

Further, in the above-described embodiment, the case of detecting a walking person has been described. However, the present invention is not limited to this, and is limited to an image processing apparatus for a natural moving image obtained by capturing various natural objects, and further to a natural moving image. Instead, the present invention can be widely applied to an image processing apparatus for a captured image of a stationary natural object or a stationary or moving artificial object.

H Effects of the Invention As described above, according to the present invention, after averaging parallax data obtained based on the feature amount of each minute region of a captured image, the parallax data is displayed in the horizontal scanning direction, the vertical scanning direction, By making it change continuously between frames, the parallax data of each minute area can be detected with certainty, and depth information can be reliably obtained even when the brightness level changes smoothly like a natural moving image. Obtainable.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an embodiment of a shape recognition apparatus according to the present invention, FIG. 2 is a perspective view showing an image pickup unit, FIG. 3 is a schematic perspective view used to explain the measurement principle, and FIG. FIG. 5 is a perspective view for explaining stereoscopic vision, FIG. 5 is a schematic diagram showing a captured image obtained from an imaging device, FIG. 6 is a block diagram showing a part of an image data processing device, and FIG. , FIG. 9, FIG. 9, FIG. 10, and FIG. 11 are schematic diagrams showing a directional difference mask, and FIG. 13, FIG. 13 is a schematic front view showing a change direction of the luminance level of the object to be measured, FIG. 14 is a block diagram showing a corresponding point detecting circuit, and FIG. 15 is a description of comparison of feature amounts when parallax is 0. FIG. 16, FIG. 17, FIG. 18, FIG. 18 and FIG. 19 are schematic diagrams used to explain comparison of feature amounts in the case of parallax 2 and parallax 1, FIG. Is a block diagram showing the depth map creation circuit, FIG. 21, FIG. 22, FIG. 23, FIG. 24, FIG.
26, 27 and 28 are schematic diagrams for explaining the false corresponding points, FIG. 29 is a schematic diagram showing a procedure for creating depth data, FIG.
FIG. 30 is a schematic diagram showing a procedure for creating an elastic plate model, and FIG. 31 is a flowchart showing a processing procedure for the entire shape recognition device. DESCRIPTION OF SYMBOLS 1 ... Shape recognition apparatus, 2 ... Image processing apparatus, 3 ... Imaging part, 4 ... Image data processing apparatus, 5 ... Video analysis and recognition apparatus, 31R, 31C, 31L ... Imaging apparatus, 43R, 43C, 43L: Feature amount detection circuit, 47: Corresponding point detection circuit, 60: Depth map creation circuit.

Claims (1)

(57) [Claims] 1. A plurality of image pickup devices arranged so that their optical axes are parallel to each other on the same plane, and a characteristic region of each of the minute regions of a minute region of a captured image obtained from the image pickup device. A feature amount detection circuit for detecting the amount, between captured images obtained from the plurality of imaging devices,
A corresponding point detection circuit that detects the minute region having the same feature amount in the arrangement direction of the plurality of imaging devices and outputs parallax data between the minute regions having the same feature amount; An averaging circuit that averages the parallax data output from the circuit for each of the minute regions; and an interpolation circuit that interpolates the averaged parallax data output from the averaging circuit in the horizontal scanning direction of the captured image. And a smoothing circuit for smoothing the interpolated parallax data output from the interpolation circuit in the vertical scanning direction of the captured image, and converting the smoothed parallax data output from the smoothing circuit to a frame of the captured image. An image processing apparatus, comprising: a smoothing circuit for performing a smoothing process between the two.