JP2017167970A - Image processing apparatus, object recognition apparatus, device control system, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus, an object recognition apparatus, a device control system, an image processing method, and a program capable of accurately performing a tracking process on an object.SOLUTION: The image processing apparatus comprises: first matching means which acquires an outline of an object by connecting each pixel hit in distance information of the object when the object is searched from an upper side to a lower side of the object in a distance image corresponding to the current frame, and which specifies a candidate position of the object in a horizontal direction when the outline is found to correspond to an object to be searched by template matching using an outline template; and second matching means which determines a position of the object in the horizontal direction and a first position of the object in a vertical direction by performing template matching using an image template in a vertical direction from the candidate position in the distance image.SELECTED DRAWING: Figure 14

Description

  The present invention relates to an image processing device, an object recognition device, a device control system, an image processing method, and a program.

  Conventionally, in terms of safety of automobiles, body structures of automobiles have been developed from the viewpoint of how to protect pedestrians and protect passengers when pedestrians and automobiles collide. However, in recent years, with the development of information processing technology and image processing technology, technology for detecting people and cars at high speed has been developed. Automobiles that apply these technologies to automatically apply a brake before an automobile collides with an object to prevent the collision have already been developed. For automatic control of automobiles, it is necessary to accurately measure the distance to an object such as a person or another vehicle. For this purpose, distance measurement using millimeter wave radar and laser radar, distance measurement using a stereo camera, etc. are practical. It has become.

  Object recognition processing using a stereo camera can be broadly divided into clustering processing and tracking processing. The clustering process is a process of newly detecting an object using a luminance image captured in real time and a parallax image derived from a stereo camera. The tracking process is a process of following the object detected by the clustering process using information of a plurality of frames. In the tracking process, basically, a region similar to the object detected in the previous frame is detected from the current frame by template matching based on the parallax value or luminance value pattern on the two-dimensional image.

  As a technique for such tracking processing, the pedestrian recognition area that is recognized as having a pedestrian is identified and a pedestrian score indicating the accuracy of being a pedestrian is performed, and the pedestrian is present based on the pedestrian score. A technique for determining whether or not to accept the recognition result is proposed (see Patent Document 1).

  However, Patent Document 1 does not describe a process corresponding to a feature amount that varies from frame to frame, and there are various features for each individual such as a non-rigid pedestrian. Therefore, there is a problem that it is difficult to execute processing for accurately tracking (tracking) an object whose feature amount varies depending on the type of object.

  The present invention has been made in view of the above, and provides an image processing device, an object recognition device, a device control system, an image processing method, and a program that can accurately perform tracking processing on an object. Objective.

  In order to solve the above-described problems and achieve the object, the present invention goes to the distance information pixel of the object when searching from the upper side to the lower side of the object in the distance image corresponding to the current frame. First, the candidate position in the lateral direction of the object is specified when the contour of the object is obtained by connecting the corresponding pixels and the contour corresponds to the object to be detected by template matching using the contour template. Matching means and second matching means for performing template matching using an image template in the vertical direction from the candidate position in the distance image, and determining the horizontal position of the object and the first vertical position. It is characterized by having.

  According to the present invention, it is possible to accurately perform a tracking process on an object.

FIG. 1 is a diagram illustrating an example in which a device control system according to an embodiment is mounted on a vehicle. FIG. 2 is a diagram illustrating an example of an appearance of the object recognition apparatus according to the embodiment. FIG. 3 is a diagram illustrating an example of a hardware configuration of the object recognition apparatus according to the embodiment. FIG. 4 is a diagram illustrating an example of a functional block configuration of the object recognition apparatus according to the embodiment. FIG. 5 is a diagram illustrating an example of a functional block configuration of a disparity value calculation processing unit of the object recognition device according to the embodiment. FIG. 6 is a diagram for explaining the principle of deriving the distance from the imaging unit to the object. FIG. 7 is an explanatory diagram for obtaining a corresponding pixel in the comparison image corresponding to the reference pixel in the reference image. FIG. 8 is a diagram illustrating an example of a graph of a result of the block matching process. FIG. 9 is a diagram illustrating an example of a functional block configuration of the recognition processing unit of the object recognition apparatus according to the embodiment. FIG. 10 is a diagram illustrating an example of a V map generated from a parallax image. FIG. 11 is a diagram illustrating an example of a U map generated from a parallax image. FIG. 12 is a diagram illustrating an example of a real U map generated from the U map. FIG. 13 is a diagram illustrating processing for creating a detection frame. FIG. 14 is a diagram illustrating an example of a functional block configuration of the tracking processing unit of the recognition processing unit of the object recognition apparatus according to the embodiment. FIG. 15 is a flowchart illustrating an example of the block matching processing operation of the disparity value deriving unit according to the embodiment. FIG. 16 is a flowchart illustrating an example of the tracking processing operation of the tracking processing unit of the recognition processing unit according to the embodiment. FIG. 17 is a diagram for explaining the movement prediction operation. FIG. 18 is a flowchart illustrating an example of the branch processing operation of the determination unit of the tracking processing unit according to the embodiment. FIG. 19 is a flowchart illustrating an example of the operation of the pedestrian matching process of the matching unit of the tracking processing unit according to the embodiment. FIG. 20 is a diagram illustrating an operation of detecting a contour in the shape matching process in the pedestrian matching process. FIG. 21 is a diagram illustrating an example of a contour detected in the shape matching process. FIG. 22 is a diagram illustrating an example of a contour template detected from a parallax image corresponding to a previous frame. FIG. 23 is a diagram illustrating the operation of the shape matching process in the pedestrian matching process of the matching unit according to the embodiment. FIG. 24 is a diagram illustrating an example of an image template used in the image matching process in the pedestrian matching process. FIG. 25 is a diagram illustrating the operation of the image matching process in the pedestrian matching process of the matching unit according to the embodiment. FIG. 26 is a diagram illustrating the operation of the boundary determination process in the pedestrian matching process of the matching unit according to the embodiment. FIG. 27 is a diagram illustrating an operation of a frame correction process in the pedestrian matching process of the matching unit according to the embodiment.

  Hereinafter, embodiments of an image processing device, an object recognition device, a device control system, an image processing method, and a program according to the present invention will be described in detail with reference to FIGS. In addition, the present invention is not limited by the following embodiments, and constituent elements in the following embodiments are easily conceivable by those skilled in the art, substantially the same, and so-called equivalent ranges. Is included. Furthermore, various omissions, substitutions, changes, and combinations of the constituent elements can be made without departing from the scope of the following embodiments.

[Schematic configuration of vehicle with object recognition device]
FIG. 1 is a diagram illustrating an example in which a device control system according to an embodiment is mounted on a vehicle. An example in which the device control system 60 of this embodiment is mounted on a vehicle 70 will be described with reference to FIG.

  1A is a side view of a vehicle 70 on which the device control system 60 is mounted, and FIG. 1B is a front view of the vehicle 70.

  As shown in FIG. 1, a vehicle 70 that is an automobile is equipped with a device control system 60. The device control system 60 includes an object recognition device 1 installed in a passenger compartment that is a room space of the vehicle 70, a vehicle control device 6 (control device), a steering wheel 7, and a brake pedal 8.

  The object recognition device 1 has an imaging function for imaging the traveling direction of the vehicle 70 and is installed, for example, in the vicinity of a rearview mirror inside the front window of the vehicle 70. The object recognition apparatus 1 includes a main body 2, an imaging unit 10a fixed to the main body 2, and an imaging unit 10b. The imaging units 10a and 10b are fixed to the main body unit 2 so that a subject in the traveling direction of the vehicle 70 can be imaged.

  The vehicle control device 6 is an ECU (Electronic Control Unit) that performs various vehicle controls based on the recognition information received from the object recognition device 1. As an example of vehicle control, the vehicle control device 6 controls a steering system (an example of a control target) including a steering wheel 7 based on recognition information received from the object recognition device 1 to avoid an obstacle, Or the brake control etc. which decelerate and stop the vehicle 70 by controlling the brake pedal 8 (an example of a control object) are performed.

  As in the device control system 60 including the object recognition device 1 and the vehicle control device 6 described above, vehicle control such as steering control or brake control is executed, so that the driving safety of the vehicle 70 can be improved. it can.

  Note that, as described above, the object recognition device 1 captures the front of the vehicle 70, but is not limited thereto. That is, the object recognition device 1 may be installed so as to capture the rear or side of the vehicle 70. In this case, the object recognition apparatus 1 can detect the positions of the following vehicle and person behind the vehicle 70, or other vehicles and persons on the side. And the vehicle control apparatus 6 can detect the danger at the time of the lane change of the vehicle 70 or a lane merge, etc., and can perform the above-mentioned vehicle control. Further, the vehicle control device 6 determines that there is a risk of collision based on the recognition information about the obstacle behind the vehicle 70 output by the object recognition device 1 in the back operation when the vehicle 70 is parked. In this case, the vehicle control described above can be executed.

[Configuration of Object Recognition Device]
FIG. 2 is a diagram illustrating an example of an appearance of the object recognition apparatus according to the embodiment. As shown in FIG. 2, the object recognition device 1 includes the main body unit 2, the imaging unit 10a fixed to the main body unit 2, and the imaging unit 10b as described above. The imaging units 10 a and 10 b are constituted by a pair of cylindrical cameras arranged in parallel equiposition with respect to the main body unit 2. For convenience of explanation, the imaging unit 10a illustrated in FIG. 2 may be referred to as a right camera and the imaging unit 10b may be referred to as a left camera.

(Hardware configuration of object recognition device)
FIG. 3 is a diagram illustrating an example of a hardware configuration of the object recognition apparatus according to the embodiment. The hardware configuration of the object recognition apparatus 1 will be described with reference to FIG.

  As shown in FIG. 3, the object recognition device 1 includes a parallax value deriving unit 3 and a recognition processing unit 5 in the main body unit 2.

  The parallax value deriving unit 3 is a device that derives a parallax value dp indicating the parallax for the object from a plurality of images obtained by imaging the object, and outputs a parallax image indicating the parallax value dp in each pixel. Based on the parallax image output from the parallax value deriving unit 3, the recognition processing unit 5 performs object recognition processing or the like on an object such as a person or a car reflected in the captured image, and indicates information indicating the result of the object recognition processing This is a device that outputs the recognition information to the vehicle control device 6.

  As shown in FIG. 3, the parallax value deriving unit 3 includes an imaging unit 10a, an imaging unit 10b, a signal conversion unit 20a, a signal conversion unit 20b, and an image processing unit 30.

  The imaging unit 10a is a processing unit that images an object in front and generates an analog image signal. The imaging unit 10a includes an imaging lens 11a, a diaphragm 12a, and an image sensor 13a.

  The imaging lens 11a is an optical element that refracts incident light to form an image of an object on the image sensor 13a. The diaphragm 12a is a member that adjusts the amount of light input to the image sensor 13a by blocking part of the light that has passed through the imaging lens 11a. The image sensor 13a is a semiconductor element that converts light that has entered the imaging lens 11a and passed through the aperture 12a into an electrical analog image signal. The image sensor 13a is realized by, for example, a solid-state imaging device such as a charge coupled devices (CCD) or a complementary metal oxide semiconductor (CMOS).

  The imaging unit 10b is a processing unit that images an object in front and generates an analog image signal. The imaging unit 10b includes an imaging lens 11b, a diaphragm 12b, and an image sensor 13b. The functions of the imaging lens 11b, the diaphragm 12b, and the image sensor 13b are the same as the functions of the imaging lens 11a, the diaphragm 12a, and the image sensor 13a, respectively. Further, the imaging lens 11a and the imaging lens 11b are installed so that the respective lens surfaces are on the same plane so that the left and right cameras are imaged under the same conditions.

  The signal conversion unit 20a is a processing unit that converts the analog image signal generated by the imaging unit 10a into digital image data. The signal converter 20a includes a CDS (Correlated Double Sampling) 21a, an AGC (Auto Gain Control) 22a, an ADC (Analog Digital Converter) 23a, and a frame memory 24a.

  The CDS 21a removes noise from the analog image signal generated by the image sensor 13a by correlated double sampling, a horizontal differential filter, a vertical smoothing filter, or the like. The AGC 22a performs gain control for controlling the intensity of the analog image signal from which noise has been removed by the CDS 21a. The ADC 23a converts an analog image signal whose gain is controlled by the AGC 22a into digital image data. The frame memory 24a stores the image data converted by the ADC 23a.

  The signal conversion unit 20b is a processing unit that converts an analog image signal generated by the imaging unit 10b into digital image data. The signal conversion unit 20b includes a CDS 21b, an AGC 22b, an ADC 23b, and a frame memory 24b. The functions of the CDS 21b, AGC 22b, ADC 23b, and frame memory 24b are the same as the functions of the CDS 21a, AGC 22a, ADC 23a, and frame memory 24a, respectively.

  The image processing unit 30 is a device that performs image processing on the image data converted by the signal conversion unit 20a and the signal conversion unit 20b. The image processing unit 30 includes an FPGA (Field Programmable Gate Array) 31, a CPU (Central Processing Unit) 32, a ROM (Read Only Memory) 33, a RAM (Random Access Memory) 34, and an I / F (35). And a bus line 39.

  The FPGA 31 is an integrated circuit, and here performs a process of deriving a parallax value dp in an image based on image data. The CPU 32 controls each function of the parallax value deriving unit 3. The ROM 33 stores an image processing program that the CPU 32 executes in order to control each function of the parallax value deriving unit 3. The RAM 34 is used as a work area for the CPU 32. The I / F 35 is an interface for communicating with the I / F 55 in the recognition processing unit 5 via the communication line 4. As shown in FIG. 3, the bus line 39 is an address bus, a data bus, or the like that connects the FPGA 31, the CPU 32, the ROM 33, the RAM 34, and the I / F 35 so that they can communicate with each other.

  The image processing unit 30 includes the FPGA 31 as an integrated circuit for deriving the parallax value dp. However, the image processing unit 30 is not limited to this and may be an integrated circuit such as an ASIC (Application Specific Integrated Circuit). .

  As shown in FIG. 3, the recognition processing unit 5 includes an FPGA 51, a CPU 52, a ROM 53, a RAM 54, an I / F 55, a CAN (Controller Area Network) I / F 58, and a bus line 59. .

  The FPGA 51 is an integrated circuit, and here performs object recognition processing on an object based on the parallax image received from the image processing unit 30. The CPU 52 controls each function of the recognition processing unit 5. The ROM 53 stores an object recognition processing program for the CPU 52 to execute the object recognition processing of the recognition processing unit 5. The RAM 54 is used as a work area for the CPU 52. The I / F 55 is an interface for data communication with the I / F 35 of the image processing unit 30 via the communication line 4. The CAN I / F 58 is an interface for communicating with an external controller (for example, the vehicle control device 6 shown in FIG. 6). For example, the bus line 59 connected to the CAN of the automobile is as shown in FIG. An FPGA 51, a CPU 52, a ROM 53, a RAM 54, an I / F 55, and a CAN / F 58 are an address bus, a data bus, and the like that are connected so that they can communicate with each other.

  With this configuration, when a parallax image is transmitted from the I / F 35 of the image processing unit 30 to the recognition processing unit 5 via the communication line 4, the FPGA 51 is converted into a parallax image by a command of the CPU 52 in the recognition processing unit 5. Based on this, an object recognition process of an object such as a person and a car reflected in the captured image is executed.

  Each of the above-mentioned programs may be recorded and distributed on a computer-readable recording medium in a file that can be installed or executed. The recording medium is a CD-ROM (Compact Disc Read Only Memory) or an SD (Secure Digital) memory card.

  As shown in FIG. 3, the image processing unit 30 of the parallax value deriving unit 3 and the recognition processing unit 5 are separate devices, but the present invention is not limited to this. For example, the image processing unit 30 And the recognition processing unit 5 may be the same device to generate a parallax image and perform object recognition processing.

(Configuration and operation of functional block of object recognition device)
FIG. 4 is a diagram illustrating an example of a functional block configuration of the object recognition apparatus according to the embodiment. First, the configuration and operation of the functional blocks of the main part of the object recognition device 1 will be described with reference to FIG.

  As described above with reference to FIG. 3, as illustrated in FIG. 4, the object recognition apparatus 1 includes a parallax value deriving unit 3 and a recognition processing unit 5. Among these, the parallax value deriving unit 3 includes an image acquisition unit 100a (first imaging unit), an image acquisition unit 100b (second imaging unit), conversion units 200a and 200b, and a parallax value calculation processing unit 300 (generation unit). And).

  The image acquisition unit 100a is a functional unit that captures a front subject with the right camera, generates an analog image signal, and obtains a luminance image that is an image based on the image signal. The image acquisition unit 100a is realized by the imaging unit 10a illustrated in FIG.

  The image acquisition unit 100b is a functional unit that captures a front subject with the left camera, generates an analog image signal, and obtains a luminance image that is an image based on the image signal. The image acquisition unit 100b is realized by the imaging unit 10b illustrated in FIG.

  The conversion unit 200a is a functional unit that removes noise from the image data of the luminance image obtained by the image acquisition unit 100a, converts the image data into digital image data, and outputs the image data. The converter 200a is realized by the signal converter 20a shown in FIG.

  The conversion unit 200b is a functional unit that removes noise from the image data of the luminance image obtained by the image acquisition unit 100b, converts the image data into digital image data, and outputs the image data. The converter 200b is realized by the signal converter 20b shown in FIG.

  Here, of the image data of the two luminance images (hereinafter simply referred to as luminance images) output by the conversion units 200a and 200b, the luminance captured by the image acquisition unit 100a which is the right camera (imaging unit 10a). The image is the image data of the reference image Ia (hereinafter simply referred to as the reference image Ia) (first captured image), and the luminance image captured by the image acquisition unit 100b which is the left camera (imaging unit 10b) is a comparative image. It is assumed that the image data of Ib (hereinafter simply referred to as comparative image Ib) (second captured image). That is, the conversion units 200a and 200b output the reference image Ia and the comparison image Ib, respectively, based on the two luminance images output from the image acquisition units 100a and 100b.

  The disparity value calculation processing unit 300 derives a disparity value for each pixel of the reference image Ia based on the reference image Ia and the comparison image Ib received from the conversion units 200a and 200b, and the disparity value for each pixel of the reference image Ia It is a functional unit that generates a parallax image (an example of a distance image) that corresponds to a value. The parallax value calculation processing unit 300 outputs the generated parallax image to the recognition processing unit 5. Note that the image generated by the parallax value calculation processing unit 300 is not limited to a parallax image, and may be an image (distance image) having information indicating a distance to an object as a pixel value, similar to the parallax value.

  The recognition processing unit 5 recognizes (detects) an object based on the reference image Ia and the parallax image received from the parallax value deriving unit 3 and performs an object recognition process for tracking (tracking) the recognized object. It is.

<Configuration and Operation of Functional Block of Parallax Value Calculation Processing Unit>
FIG. 5 is a diagram illustrating an example of a functional block configuration of a disparity value calculation processing unit of the object recognition device according to the embodiment. FIG. 6 is a diagram for explaining the principle of deriving the distance from the imaging unit to the object. FIG. 7 is an explanatory diagram for obtaining a corresponding pixel in the comparison image corresponding to the reference pixel in the reference image. FIG. 8 is a diagram illustrating an example of a graph of a result of the block matching process.

  First, an outline of a distance measuring method using block matching processing will be described with reference to FIGS.

<< Principles of ranging >>
With reference to FIG. 6, the principle of deriving a parallax with respect to an object from the stereo camera by the stereo matching process and measuring the distance from the stereo camera to the object with the parallax value indicating the parallax will be described.

  The imaging system illustrated in FIG. 6 includes an imaging unit 10a and an imaging unit 10b that are arranged in parallel equiposition. The imaging units 10a and 10b respectively include imaging lenses 11a and 11b that refract incident light and form an image of an object on an image sensor that is a solid-state imaging device. The images captured by the imaging unit 10a and the imaging unit 10b are referred to as a reference image Ia and a comparative image Ib, respectively. In FIG. 6, the point S on the object E in the three-dimensional space is mapped to a position on a straight line parallel to a straight line connecting the imaging lens 11a and the imaging lens 11b in the reference image Ia and the comparison image Ib. Here, the point S mapped to each image is a point Sa (x, y) in the reference image Ia and a point Sb (X, y) in the comparison image Ib. At this time, the parallax value dp is expressed by the following (Equation 1) using the point Sa (x, y) at the coordinates on the reference image Ia and the point Sb (X, y) at the coordinates on the comparison image Ib. It is expressed in

  dp = X−x (Formula 1)

  In FIG. 6, the distance between the point Sa (x, y) in the reference image Ia and the intersection of the perpendicular line taken from the imaging lens 11a on the imaging surface is Δa, and the point Sb (X, y) in the comparative image Ib is The parallax value dp can also be expressed as dp = Δa + Δb, where Δb is the distance from the intersection of the perpendicular line taken from the imaging lens 11b on the imaging surface.

  Next, the distance Z between the imaging units 10a and 10b and the object E is derived by using the parallax value dp. Here, the distance Z is a distance from a straight line connecting the focal position of the imaging lens 11a and the focal position of the imaging lens 11b to the point S on the object E. As shown in FIG. 6, using the focal length f of the imaging lens 11a and the imaging lens 11b, the baseline length B that is the length between the imaging lens 11a and the imaging lens 11b, and the parallax value dp, According to 2), the distance Z can be calculated.

  Z = (B × f) / dp (Formula 2)

  From this (Equation 2), it can be seen that the larger the parallax value dp, the smaller the distance Z, and the smaller the parallax value dp, the larger the distance Z.

<< Block matching process >>
Next, a distance measurement method using block matching processing will be described with reference to FIGS.

  A method for calculating the cost value C (p, d) will be described with reference to FIGS. In the following description, C (p, d) represents C (x, y, d).

  7A is a conceptual diagram showing the reference pixel p and the reference region pb in the reference image Ia, and FIG. 7B corresponds to the reference pixel p shown in FIG. 7A. It is a conceptual diagram at the time of calculating the cost value C while sequentially shifting (shifting) the corresponding pixel candidates in the comparative image Ib. Here, the corresponding pixel indicates a pixel in the comparison image Ib that is most similar to the reference pixel p in the reference image Ia. The cost value C is an evaluation value (degree of coincidence) representing the similarity or dissimilarity of each pixel in the comparison image Ib with respect to the reference pixel p in the reference image Ia. The cost value C shown below is described as an evaluation value that represents a degree of dissimilarity indicating that the smaller the value is, the more similar the pixel in the comparative image Ib is to the reference pixel p.

  As shown in FIG. 7A, a candidate that is a candidate for a reference pixel p (x, y) in the reference image Ia and a corresponding pixel on the epipolar line EL in the comparison image Ib for the reference pixel p (x, y). Based on each luminance value (pixel value) of the pixel q (x + d, y), the cost value C (p, d) of the candidate pixel q (x + d, y) that is a candidate for the corresponding pixel with respect to the reference pixel p (x, y). ) Is calculated. d is a shift amount (shift amount) between the reference pixel p and the candidate pixel q, and the shift amount d is shifted in units of pixels. That is, the candidate pixel q (x + d, y) and the reference pixel p (x, y) are sequentially shifted by one pixel within a predetermined range (for example, 0 <d <25). ) Is calculated as a cost value C (p, d) which is a dissimilarity between the brightness value and the brightness value. In addition, in the present embodiment, block matching processing is performed as stereo matching processing for obtaining a corresponding pixel of the reference pixel p. In the block matching process, a reference region pb that is a predetermined region centered on the reference pixel p of the reference image Ia, and a candidate region qb (size is the same as the reference region pb) centered on the candidate pixel q of the comparison image Ib. Find dissimilarity of. The cost value C indicating the dissimilarity between the reference region pb and the candidate region qb is obtained by subtracting the average value of each block from the value of SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), or SSD. ZSSD (Zero-mean-Sum of Squared Difference) or the like is used. These evaluation values indicate dissimilarity because the values are smaller as the correlation is higher (the degree of similarity is higher).

  As described above, since the imaging units 10a and 10b are arranged in parallel equiposition, the reference image Ia and the comparison image Ib are also in parallel equivalence relations. Therefore, the corresponding pixel in the comparison image Ib corresponding to the reference pixel p in the reference image Ia exists on the epipolar line EL shown as a horizontal line in FIG. 7, and the corresponding pixel in the comparison image Ib is In order to obtain it, the pixel on the epipolar line EL of the comparison image Ib may be searched.

  The cost value C (p, d) calculated by such block matching processing is represented by, for example, a graph shown in FIG. 8 in relation to the shift amount d. In the example of FIG. 8, the cost value C is derived as the parallax value dp = 7 because the minimum value is obtained when the shift amount d = 7.

<< Specific Configuration and Operation of Functional Block of Parallax Value Calculation Processing Unit >>
A specific configuration and operation of the functional block of the parallax value calculation processing unit 300 will be described with reference to FIG.

  As illustrated in FIG. 5, the parallax value calculation processing unit 300 includes a cost calculation unit 301, a determination unit 302, and a first generation unit 303.

  The cost calculation unit 301 uses the reference pixel p (x, y) on the epipolar line EL in the comparison image Ib based on the luminance value of the reference pixel p (x, y) in the reference image Ia and the reference pixel p (x, y). Each candidate pixel q (x + d, y) is determined based on each luminance value of the candidate pixel q (x + d, y), which is a candidate for the corresponding pixel, specified by shifting from the pixel corresponding to the position y) by the shift amount d. This is a functional unit that calculates the cost value C (p, d) of y). Specifically, the cost calculation unit 301 performs, by block matching processing, a reference region pb that is a predetermined region centered on the reference pixel p of the reference image Ia and a candidate region qb centered on the candidate pixel q of the comparison image Ib. The dissimilarity with (the size is the same as the reference region pb) is calculated as the cost value C.

  The determination unit 302 determines the shift amount d corresponding to the minimum value of the cost value C calculated by the cost calculation unit 301 as the parallax value dp for the pixel of the reference image Ia for which the cost value C is calculated. It is a functional part.

  Based on the parallax value dp determined by the determination unit 302, the first generation unit 303 generates a parallax image that is an image obtained by replacing the pixel value of each pixel of the reference image Ia with the parallax value dp corresponding to the pixel. It is a functional part to do.

  The cost calculation unit 301, the determination unit 302, and the first generation unit 303 illustrated in FIG. 5 are each realized by the FPGA 31 illustrated in FIG. Note that part or all of the cost calculation unit 301, the determination unit 302, and the first generation unit 303 are realized by the CPU 32 executing a program stored in the ROM 33 instead of the FPGA 31 that is a hardware circuit. It may be a thing.

  Note that the cost calculation unit 301, the determination unit 302, and the first generation unit 303 of the parallax value calculation processing unit 300 illustrated in FIG. 5 conceptually illustrate functions, and are limited to such a configuration. is not. For example, a plurality of functional units illustrated as independent functional units in the parallax value calculation processing unit 300 illustrated in FIG. 5 may be configured as one functional unit. On the other hand, the parallax value calculation processing unit 300 illustrated in FIG. 5 may be configured such that the functions of one functional unit are divided into a plurality of functional units.

<Configuration and operation of functional block of recognition processing unit>
FIG. 9 is a diagram illustrating an example of a functional block configuration of the recognition processing unit of the object recognition apparatus according to the embodiment. FIG. 10 is a diagram illustrating an example of a V map generated from a parallax image. FIG. 11 is a diagram illustrating an example of a U map generated from a parallax image. FIG. 12 is a diagram illustrating an example of a real U map generated from the U map. FIG. 13 is a diagram illustrating processing for creating a detection frame. The configuration and operation of the functional block of the recognition processing unit 5 will be described with reference to FIGS.

  As illustrated in FIG. 9, the recognition processing unit 5 includes a second generation unit 500, a clustering processing unit 510 (detection unit), and a tracking processing unit 520.

  The second generation unit 500 receives the parallax image from the parallax value calculation processing unit 300 and the reference image Ia from the parallax value deriving unit 3, and receives the V-Disparity map, the U-Disparity map, and the Real U-Disparity. It is a functional unit that generates a map and the like. Specifically, the second generation unit 500 generates a V map VM that is a V-Disparity map illustrated in FIG. 10B in order to detect a road surface from the parallax image input from the parallax value calculation processing unit 300. . Here, the V-Disparity map is a two-dimensional histogram showing the frequency distribution of the parallax values dp with the vertical axis as the y-axis of the reference image Ia and the horizontal axis as the parallax value dp (or distance) of the parallax image. . In the reference image Ia shown in FIG. 10A, for example, a road surface 700, a utility pole 701, and a car 702 are reflected. The road surface 700 of the reference image Ia corresponds to the road surface portion 700a in the V map VM, the electric pole 701 corresponds to the electric pole portion 701a, and the vehicle 702 corresponds to the vehicle portion 702a.

  In addition, the second generation unit 500 linearly approximates the position estimated as the road surface from the generated V map VM. When the road surface is flat, it can be approximated by a single straight line. However, when the road surface changes in slope, it is necessary to divide the section of the V map VM and perform linear approximation with high accuracy. As the linear approximation, a Hough transform or a least square method which is a known technique can be used. In the V map VM, the electric pole portion 701a and the vehicle portion 702a, which are blocks located above the detected road surface portion 700a, correspond to the electric pole 701 and the vehicle 702, which are objects on the road surface, respectively. When a U-Disparity map is generated by the second generation unit 500 described later, only information above the road surface is used for noise removal.

  Further, the second generation unit 500 uses only information located above the road surface detected by the V map VM, that is, the left guard rail 711, the right guard rail 712, and the car 713 in the reference image Ia shown in FIG. And in order to recognize an object using the information on the parallax image corresponding to the car 714, the U map UM which is a U-Disparity map shown in FIG. 11B is generated. Here, the U map UM is a two-dimensional histogram showing the frequency distribution of the parallax values dp with the horizontal axis as the x-axis of the reference image Ia and the vertical axis as the parallax value dp (or distance) of the parallax image. In the U map UM, the left guard rail 711 of the reference image Ia shown in FIG. 11A corresponds to the left guard rail portion 711a, the right guard rail 712 corresponds to the right guard rail portion 712a, and the vehicle 713 corresponds to the vehicle portion 713a. Correspondingly, the vehicle 714 corresponds to the vehicle portion 714a.

  Further, the second generation unit 500 uses only information located above the road surface detected by the V map VM, that is, the left guard rail 711, the right guard rail 712, and the car 713 in the reference image Ia shown in FIG. And U map UM_H which is an example of the U-Disparity map shown in FIG.11 (c) is produced | generated using the information on the parallax image corresponding to the vehicle 714. FIG. Here, the U map UM_H, which is an example of the U-Disparity map, is an image in which the horizontal axis is the x axis of the reference image Ia, the vertical axis is the parallax value dp of the parallax image, and the pixel value is the height of the object. In the U map UM_H, the left guard rail 711 of the reference image Ia shown in FIG. 11A corresponds to the left guard rail portion 711b, the right guard rail 712 corresponds to the right guard rail portion 712b, and the vehicle 713 corresponds to the vehicle portion 713b. Correspondingly, the vehicle 714 corresponds to the vehicle portion 714b.

  Further, the second generation unit 500 generates a real U map RM that is a Real U-Disparity map shown in FIG. 12B in which the horizontal axis is converted into an actual distance from the generated U map UM shown in FIG. Is generated. Here, in the real U map RM, the horizontal axis is the actual distance in the direction from the imaging unit 10b (right camera) to the imaging unit 10a (left camera), and the vertical axis is the parallax value dp of the parallax image (or It is a two-dimensional histogram as a distance in the depth direction converted from the parallax value dp. The left guard rail portion 711a of the U map UM shown in FIG. 12A corresponds to the left guard rail portion 711c in the real U map RM, the right guard rail portion 712a corresponds to the right guard rail portion 712c, and the vehicle portion 713a Corresponding to the vehicle portion 713c, the vehicle portion 714a corresponds to the vehicle portion 714c. Specifically, in the U map UM, since the object is small in the distance (the disparity value dp is small), the second generation unit 500 has a small amount of disparity information and a small distance resolution. Since a large object is captured, the parallax information is large and the resolution of the distance is large. Therefore, the real U map RM is generated by thinning out pixels greatly. As will be described later, the clustering processing unit 510 can detect an object by extracting a block (object) of pixel values from the real U map RM. The second generation unit 500 is not limited to generating the real U map RM from the U map UM, and can also generate the real U map RM directly from the parallax image.

  Note that the image input from the parallax value derivation unit 3 to the second generation unit 500 is not limited to the reference image Ia, and may be the comparison image Ib.

  The clustering processing unit 510 is a functional unit that detects an object shown in the parallax image based on each map input from the second generation unit 500. The clustering processing unit 510 can specify the position and width (xmin, xmax) in the x-axis direction of the parallax image of the object and the reference image Ia from the generated U map UM or real U map RM. Further, the clustering processing unit 510 can identify the actual depth of the object from the height information (dmin, dmax) of the object in the generated U map UM or real U map RM. Further, the clustering processing unit 510 uses the generated V map VM to determine the position and height in the y-axis direction (ymin = “y corresponding to the maximum height from the road surface of the maximum parallax value) in the parallax image of the object and the reference image Ia. Coordinate ", ymax =" y coordinate indicating the height of the road surface obtained from the maximum parallax value "). Further, the clustering processing unit 510 determines the actual object from the width (xmin, xmax) in the x-axis direction and the height (ymin, ymax) in the y-axis direction and the corresponding parallax values dp of the object specified in the parallax image. The size in the x-axis direction and the y-axis direction can be specified. As described above, the clustering processing unit 510 uses the V map VM, the U map UM, and the real U map RM to specify the position of the object in the reference image Ia and the actual width, height, and depth. Can do. In addition, since the position of the object in the reference image Ia is specified, the clustering processing unit 510 can determine the position in the parallax image and can also specify the distance to the object.

  Then, as shown in FIG. 13A, the clustering processing unit 510 finally corresponds to the detection areas 721 to 724 of the objects specified (detected) on the real U map RM, as shown in FIG. Detection frames 721a to 724a on the reference image Ia or the parallax image Ip shown in b) are created.

  Further, the clustering processing unit 510 can identify what the object is using the following (Table 1) from the actual size (width, height, depth) specified for the object. For example, when the width of the object is 1300 [mm], the height is 1800 [mm], and the depth is 2000 [mm], the object can be specified as a “normal vehicle”. Note that information associating the width, height, and depth with the type of object (object type) as in (Table 1) may be stored in the RAM 54 or the like as a table.

  The clustering processing unit 510 generates information about the detected (recognized) object as recognition area information. Here, the recognition area information indicates information related to the object detected by the clustering processing unit 510. For example, the reference image Ia, V-Disparity map, U-Disparity map, and Real U-Disparity map of the detected object, etc. This includes information such as the position and size at, the type of the detected object, and a rejection flag, which will be described later.

  The second generation unit 500 and the clustering processing unit 510 of the recognition processing unit 5 illustrated in FIG. 9 are realized by the FPGA 51 illustrated in FIG. Part or all of the second generation unit 500 and the clustering processing unit 510 may be realized by the CPU 52 executing a program stored in the ROM 53 instead of the FPGA 51 which is a hardware circuit. .

  The tracking processing unit 520 is a functional unit that executes tracking processing that rejects an object or performs tracking processing based on recognition area information that is information about an object detected (recognized) by the clustering processing unit 510. is there. A specific configuration of the tracking processing unit 520 will be described with reference to FIG. Here, “rejection” indicates that the object is excluded from the target of subsequent processing (for example, control processing or the like in the vehicle control device 6).

  The “image processing apparatus” according to the present invention may be the tracking processing unit 520 or the recognition processing unit 5 including the tracking processing unit 520.

<< Configuration and operation of functional block of tracking processing section >>
FIG. 14 is a diagram illustrating an example of a functional block configuration of the tracking processing unit of the recognition processing unit of the object recognition apparatus according to the embodiment. The configuration and operation of the functional block of the tracking processing unit 520 of the recognition processing unit 5 will be described with reference to FIG.

  As shown in FIG. 14, the tracking processing unit 520 includes a movement prediction unit 600 (prediction unit), a matching unit 610, a check unit 620, a feature update unit 630 (update unit), and a state transition unit 640. Have.

  The movement prediction unit 600 uses the history of movement and operation state of the object newly detected by the clustering processing unit 510 and the vehicle information, and uses the current luminance image for each object that has been tracked (tracked). This is a functional unit that predicts a prediction region where there is a high probability that an object is present (or may simply be referred to as a “frame” hereinafter) (or a parallax image corresponding thereto). The movement prediction unit 600 uses movement information (for example, relative position history and relative speed history of the center of gravity, etc.) up to the previous frame (hereinafter sometimes simply referred to as “previous frame”) and vehicle information, and the xz plane. The motion of the object is predicted by (x: frame horizontal position, z: distance). Note that the movement prediction unit 600 may perform a process of enlarging the previously predicted prediction area in order to deal with an object having a motion that is greater than or equal to the prediction. Further, the above movement information may be included in the recognition area information for each detected object. In the following description, the recognition area information will be described as including the above movement information.

  The matching unit 610 performs template matching based on the similarity with the feature amount (template) obtained in the previous frame in the prediction region predicted by the movement prediction unit 600, and performs the current frame (hereinafter simply referred to as “current frame”). ) Is a functional unit that performs a matching process for obtaining the position of an object (in particular, a vehicle and a pedestrian). Here, the pedestrian refers to a person included in a captured image captured by an imaging unit or the like, and refers to all persons such as a walking person, a running person, and a stopped person. . The matching unit 610 includes a determination unit 611, a shape matching unit 612 (first matching unit), an image matching unit 613 (second matching unit), a boundary determination unit 614 (determination unit), a correction processing unit 615, Have

  The determination unit 611 determines whether the object is a vehicle or a pedestrian based on the recognition area information of the object. If the object is a vehicle, the vehicle for tracking the vehicle in the subsequent process If the user is a pedestrian, the function unit performs a branching process for executing the pedestrian matching process for tracking the pedestrian in the subsequent process.

  In the pedestrian matching process, the shape matching unit 612 detects a contour mainly including the head of the pedestrian in the parallax image, and uses the contour of the pedestrian detected in the parallax image corresponding to the previous frame as a template (contour template). ) Is a functional unit that performs shape matching processing for performing template matching.

  The image matching unit 613 performs a function of performing an image matching process that performs template matching using an image template based on a contour template detected from a parallax image corresponding to a previous frame in a luminance image that is a current frame in a pedestrian matching process. Part.

  In the pedestrian matching process, the boundary determination unit 614 detects a boundary with a pedestrian other than the pedestrian detected (position is determined) by the image matching unit 613 when the contours of a plurality of pedestrians are detected in the current frame. It is a functional part which performs the boundary determination process which determines.

  The correction processing unit 615 is a functional unit that performs frame correction processing on the frame (detection frame) of the detection area of the pedestrian detected by the image matching unit 613. That is, the image of the detection frame after the frame correction process is performed on the detection frame of the pedestrian by the correction processing unit 615 becomes the detection area of the pedestrian in the current frame (or the parallax image corresponding to the current frame). .

  The check unit 620 determines whether or not the size corresponds to the size of an object (for example, a pedestrian or a vehicle) targeted for tracking based on the size of the detection area of the object detected by the matching unit 610. It is.

  The feature update unit 630 updates the feature amount (contour template and image template) used in template matching of the shape matching unit 612 and the image matching unit 613 in the next frame from the image of the detection area of the object detected in the current frame. It is a functional part to do.

  The state transition unit 640 is a functional unit that transitions the state of the object based on the recognition area information of the object finally determined by the correction processing unit 615. For example, the state transition unit 640 detects an object that is not determined as an object to be tracked by the check unit 620 and an object that cannot be detected (tracked) by matching by the shape matching unit 612 and the image matching unit 613. The state of the object is transitioned by including a rejection flag indicating that the object is rejected in the recognition area information of the object. The state transition unit 640 outputs recognition area information reflecting the state of the transitioned object to the vehicle control device 6 (see FIG. 4) as recognition information.

  The movement prediction unit 600, the determination unit 611 of the matching unit 610, the shape matching unit 612, the image matching unit 613, the boundary determination unit 614 and the correction processing unit 615, the check unit 620, the feature update unit 630, and the state transition unit illustrated in FIG. 640 is realized by the FPGA 51 shown in FIG. Note that some or all of these functional units may be realized by the CPU 52 executing a program stored in the ROM 53 instead of the FPGA 51 that is a hardware circuit.

  Note that each function unit of the tracking processing unit 520 illustrated in FIG. 14 conceptually illustrates a function, and is not limited to such a configuration. For example, a plurality of functional units illustrated as independent functional units in the tracking processing unit 520 illustrated in FIG. 14 may be configured as one functional unit. On the other hand, the tracking processor 520 shown in FIG. 14 may be configured as a plurality of functional units by dividing a function of one functional unit into a plurality of functional units.

[Operation of object recognition device]
Next, a specific operation of the object recognition device 1 will be described with reference to FIGS.

(Block matching process of parallax value deriving unit)
FIG. 15 is a flowchart illustrating an example of the block matching processing operation of the disparity value deriving unit according to the embodiment. The flow of the operation of the block matching process of the parallax value deriving unit 3 of the object recognition device 1 will be described with reference to FIG.

<Step S1-1>
The image acquisition unit 100b of the parallax value deriving unit 3 images a front subject with the left camera (imaging unit 10b), generates an analog image signal, and obtains a luminance image that is an image based on the image signal. . As a result, an image signal to be subjected to subsequent image processing is obtained. And it transfers to step S2-1.

<Step S1-2>
The image acquisition unit 100a of the parallax value deriving unit 3 images a front subject with the right camera (imaging unit 10a), generates an analog image signal, and obtains a luminance image that is an image based on the image signal. . As a result, an image signal to be subjected to subsequent image processing is obtained. Then, the process proceeds to step S2-2.

<Step S2-1>
The conversion unit 200b of the parallax value deriving unit 3 removes noise from the analog image signal obtained by imaging by the imaging unit 10b and converts the analog image signal into digital image data. In this way, by converting the image data into digital image data, image processing for each pixel can be performed on an image based on the image data. And it transfers to step S3-1.

<Step S2-2>
The conversion unit 200a of the parallax value deriving unit 3 removes noise from the analog image signal obtained by imaging by the imaging unit 10a and converts the analog image signal into digital image data. In this way, by converting the image data into digital image data, image processing for each pixel can be performed on an image based on the image data. Then, the process proceeds to step S3-2.

<Step S3-1>
The conversion unit 200b outputs an image based on the digital image data converted in step S2-1 as the comparison image Ib in the block matching process. Thus, an image to be compared for obtaining a parallax value in the block matching process is obtained. Then, the process proceeds to step S4.

<Step S3-2>
The conversion unit 200a outputs an image based on the digital image data converted in step S2-2 as the reference image Ia in the block matching process. Thus, an image serving as a reference for obtaining a parallax value in the block matching process is obtained. Then, the process proceeds to step S4.

<Step S4>
The cost calculation unit 301 of the parallax value calculation processing unit 300 of the parallax value deriving unit 3 compares the luminance value of the reference pixel p (x, y) in the reference image Ia and the comparison image Ib based on the reference pixel p (x, y). On the epipolar line EL, the brightness value of the candidate pixel q (x + d, y) of the corresponding pixel specified by shifting by the shift amount d from the pixel corresponding to the position of the reference pixel p (x, y). Based on this, the cost value C (p, d) of each candidate pixel q (x + d, y) is calculated. Specifically, the cost calculation unit 301 performs, by block matching processing, a reference region pb that is a predetermined region centered on the reference pixel p of the reference image Ia and a candidate region qb centered on the candidate pixel q of the comparison image Ib. The dissimilarity with (the size is the same as the reference region pb) is calculated as the cost value C. Then, the process proceeds to step S5.

<Step S5>
The determination unit 302 of the parallax value calculation processing unit 300 of the parallax value deriving unit 3 uses the shift amount d corresponding to the minimum value of the cost value C calculated by the cost calculation unit 301 as a target for calculating the cost value C. The parallax value dp for the pixel of the reference image Ia is determined. Then, the first generation unit 303 of the parallax value calculation processing unit 300 of the parallax value deriving unit 3 sets the luminance value of each pixel of the reference image Ia to that pixel based on the parallax value dp determined by the determination unit 302. A parallax image that is an image represented by the corresponding parallax value dp is generated. The first generation unit 303 outputs the generated parallax image to the recognition processing unit 5.

  The stereo matching process described above has been described by taking the block matching process as an example, but is not limited thereto, and may be a process using an SGM (Semi-Global Matching) method.

(Tracking process of the tracking processor of the recognition processor)
FIG. 16 is a flowchart illustrating an example of the tracking processing operation of the tracking processing unit of the recognition processing unit according to the embodiment. FIG. 17 is a diagram for explaining the movement prediction operation. The flow of the tracking processing operation of the tracking processing unit 520 of the recognition processing unit 5 will be described with reference to FIGS.

<Step S11>
The movement prediction unit 600 of the tracking processing unit 520 uses the recognition area information including the history of the movement and motion state of the object newly detected by the clustering processing unit 510 in the previous stage and the vehicle information so far ( For each object that has been tracked), as shown in FIG. 17, a prediction region 800 having a high probability that an object exists on the current frame (reference image Ia) (or a corresponding parallax image) is predicted. Then, the process proceeds to step S12.

<Step S12>
The matching unit 610 of the tracking processing unit 520 performs template matching based on the similarity to the feature amount (template) obtained in the previous frame in the prediction region 800, and the position of an object (particularly, a vehicle and a pedestrian) in the current frame. Ask for. Details of the matching processing by the matching unit 610 will be described later with reference to FIGS. Then, the process proceeds to step S13.

<Step S13>
Whether or not the check unit 620 of the tracking processing unit 520 corresponds to the size of an object (for example, a pedestrian or a vehicle) targeted for tracking based on the size of the detection area of the object detected by the matching unit 610. Determine whether. Then, the process proceeds to step S14.

<Step S14>
The feature update unit 630 of the tracking processing unit 520 uses, from the image of the detection area of the object detected in the current frame, the feature amount (contour template and profile template) used in the template matching of the shape matching unit 612 and the image matching unit 613 in the next frame. Update the image template. Then, the process proceeds to step S15.

<Step S15>
The state transition unit 640 of the tracking processing unit 520 changes the state of the object based on the recognition area information of the object finally determined by the correction processing unit 615. The state transition unit 640 outputs recognition area information reflecting the state of the transitioned object to the vehicle control device 6 as recognition information.

  The tracking process by the tracking processing unit 520 is performed by the processes in steps S11 to S15 described above. Note that the processing in steps S11 to S15 is executed for each detection region of an object newly detected by the clustering processing unit 510.

(Branch processing in tracking processing)
FIG. 18 is a flowchart illustrating an example of the branch processing operation of the determination unit of the tracking processing unit according to the embodiment. The flow of the branch processing operation of the matching unit 610 of the tracking processing unit 520 will be described with reference to FIG.

<Step S121>
The determination unit 611 of the matching unit 610 determines the type of the object (tracked object) to be tracked based on the recognition area information of the object. When the object is a vehicle (step S121: vehicle), the process proceeds to step S122. When the object is a pedestrian (step S121: pedestrian), the process proceeds to step S123. When the object to be tracked is limited to a vehicle and a pedestrian, the determination unit 611 determines that the object is not a vehicle or a pedestrian as a result of determining the type of the object. A rejection flag indicating that the object is to be rejected may be included.

<Step S122>
If the determination unit 611 determines that the object is a vehicle, the determination unit 611 executes a vehicle matching process for tracking the vehicle. Then, the branch process ends.

<Step S123>
If the determination unit 611 determines that the object is a pedestrian, the determination unit 611 executes a pedestrian matching process for tracking the pedestrian. Details of the pedestrian matching process will be described later with reference to FIG. Then, the branch process ends.

  The branch processing by the matching unit 610 of the tracking processing unit 520 is performed by the processing of steps S121 to S123 described above.

(Pedestrian matching process in tracking process)
FIG. 19 is a flowchart illustrating an example of the operation of the pedestrian matching process of the matching unit of the tracking processing unit according to the embodiment. FIG. 20 is a diagram illustrating an operation of detecting a contour in the shape matching process in the pedestrian matching process. FIG. 21 is a diagram illustrating an example of a contour detected in the shape matching process. FIG. 22 is a diagram illustrating an example of a contour template detected from a parallax image corresponding to a previous frame. FIG. 23 is a diagram illustrating the operation of the shape matching process in the pedestrian matching process of the matching unit according to the embodiment. FIG. 24 is a diagram illustrating an example of an image template used in the image matching process in the pedestrian matching process. FIG. 25 is a diagram illustrating the operation of the image matching process in the pedestrian matching process of the matching unit according to the embodiment. FIG. 26 is a diagram illustrating the operation of the boundary determination process in the pedestrian matching process of the matching unit according to the embodiment. FIG. 27 is a diagram illustrating an operation of a frame correction process in the pedestrian matching process of the matching unit according to the embodiment. The flow of the operation of the matching process for pedestrians of the matching unit 610 of the tracking processing unit 520 will be described with reference to FIGS.

<Step S1231>
The shape matching unit 612 of the matching unit 610 detects a contour mainly including the head of the pedestrian in the parallax image, and uses the contour of the pedestrian detected from the parallax image corresponding to the previous frame as a template (contour template). A shape matching process for matching is performed. Specifically, as shown in FIG. 20, the shape matching unit 612 first moves downward from the upper end at each x coordinate in the prediction region 801 predicted by the movement prediction unit 600 in the parallax image corresponding to the current frame. The position where the pedestrian's parallax value is reached (Y coordinate) is specified. Next, the shape matching unit 612 acquires a contour 901 that is a contour from the head of the pedestrian to the vicinity of the shoulder in the prediction region 801 as shown in FIG. 21 by connecting the specified positions. However, it is not determined at this point whether the acquired contour is the contour of a pedestrian or the contour of an object other than a pedestrian at this point, but here, the pedestrian shown in FIG. 20 will be described as an example. And In the example of FIG. 21, an example in which the contours of two pedestrians are extracted is shown. Next, the shape matching unit 612 generates a template (contour template 902 shown in FIG. 22) of the pedestrian contour detected in the prediction region 802 of the parallax image corresponding to the previous frame, and the horizontal direction (X direction) in the prediction region 801. ) And template matching for the contour 901 is performed. For example, SAD, SSD, or ZSSD may be used as the cost value indicating the similarity used for template matching. In the example shown in FIG. 23, the case where SAD is used is shown.

  In the prediction region 801, the shape matching unit 612 calculates the SAD while shifting the contour template 902 in the horizontal direction, and specifies the position in the X direction where the SAD value is minimized (that is, the position with high similarity). In the example of FIG. 23, the shape matching unit 612 identifies candidate positions P1 and P2 that are positions in the X direction where the SAD value is minimized, that is, positions in the X direction that are candidates for the presence of the pedestrian to be tracked. To do. Here, the shape matching unit 612 determines the position determined as the position in the X direction of the pedestrian to be tracked, for example, the X of the pixel existing on the uppermost side of the contour template 902 located when the SAD becomes the smallest. What is necessary is just to set it as the position of a direction. The shape matching unit 612 sends information on the identified candidate positions P1 and P2 to the image matching unit 613. Then, control goes to a step S1232.

<Step S1232>
The image matching unit 613 of the matching unit 610 performs image matching processing for performing template matching on the luminance image (reference image Ia) that is the current frame using an image template based on the contour template detected in the parallax image corresponding to the previous frame. Do. Specifically, as shown in FIG. 24A, a contour is formed at a position on the luminance image (reference image Ia) corresponding to the position of the contour template 902 detected in the parallax image corresponding to the previous frame. An image template 902a composed of N pixels in the downward direction from each pixel is created in advance by the process of the feature update unit 630 for the previous frame. First, as shown in FIG. 25, the image matching unit 613 performs the X-direction specified by the shape matching unit 612 in the prediction region 811 on the luminance image (reference image Ia) corresponding to the prediction region 801 on the parallax image. Template matching is performed while shifting the image template 902a downward from the upper end of the prediction region 811 at each of the candidate positions P1 and P2, which are candidate positions. For example, SAD, SSD, or ZSSD may be used as the cost value indicating the similarity used for template matching. In the example shown in FIG. 25, the case where SAD is used is shown.

  In the prediction area 811, the image matching unit 613 calculates SAD while shifting the image template 902a in the Y direction, and specifies the position in the Y direction where the SAD value is the smallest (that is, the position where the similarity is the highest). In the example of FIG. 25, since template matching based on the image template 902a is performed at both candidate positions P1 and P2, the image matching unit 613 selects a smaller candidate position from among the SADs having the smallest value at each candidate position as a pedestrian. Is determined to be in the X direction. Further, the image matching unit 613 determines the position in the Y direction where the SAD is the smallest at the candidate position corresponding to the determined position of the pedestrian in the X direction as the position of the pedestrian in the Y direction (first position). To do. Here, the position determined as the position in the Y direction of the pedestrian to be tracked by the image matching unit 613 is, for example, Y of the pixel existing on the uppermost side of the image template 902a positioned when the SAD becomes the smallest. What is necessary is just to set it as the position of a direction. As a result, the position of the parallax image (or luminance image) of the pedestrian to be tracked is detected.

  Here, for example, the detection frame indicating the detection area of the pedestrian whose position is detected in the current frame is assumed to have the same area of the detection area, that is, the area in the detection frame, and walking in the previous frame, for example. The position of the detection frame in the current frame may be determined such that the relative position of the detection frame surrounding the pedestrian with respect to the position of the person is the same. Note that the position and size of the detection frame determined in this way are corrected by frame correction processing by the correction processing unit 615 described later.

  Note that the image matching processing by the image matching unit 613 is limited to a method of obtaining SAD while shifting the image template 902a downward from the upper end at the candidate position (for example, candidate positions P1 and P2 shown in FIG. 25) as described above. Is not to be done. A rectangular template 902b shown in FIG. 24B is obtained by converting each N pixel extending in the Y direction in each X coordinate in the image template 902a shown in FIG. 24A created by the feature update unit 630 into the Y direction (height direction). ) To make it a rectangular shape. The image matching unit 613 lowers the N pixels of each X coordinate constituting the rectangular template 902b toward the contour 901 at the candidate positions P1 and P2 specified by the shape matching unit 612, and sets each column of N pixels to The uppermost pixel in the column is arranged so as to overlap each pixel of the contour 901. In this case, the image matching unit 613 calculates SAD or the like using each pixel value of the rectangular template 902b and each pixel value of a portion where each column of N pixels overlaps in the prediction region 811. When a value such as is less than a predetermined threshold, the position determined by the contour portion in the contour 901 corresponding to the calculated SAD or the like is detected as the position of the parallax image (or luminance image) of the pedestrian to be tracked. Also good. Thereby, since it is not necessary to calculate SAD or the like at each position while shifting the image template 902a in the Y direction, the processing speed for detecting the position of the pedestrian to be tracked can be improved.

  The image matching unit 613 sends the determined position of the pedestrian and information on the position and size of the detection frame to the boundary determining unit 614. Then, control goes to a step S1233.

<Step S1233>
The boundary determination unit 614 of the matching unit 610 determines a boundary with a pedestrian other than the pedestrian detected (position is determined) by the image matching unit 613 when the contours of a plurality of pedestrians are detected in the current frame. Perform boundary determination processing. Specifically, the boundary determination unit 614 first lowers the reference line BL extending in the X direction from the top to the bottom in the prediction region 801 in the parallax image for the current frame, as shown in FIG. . The boundary determining unit 614 then lowers the reference line BL after the parallax value cluster (isolated region) of the object (pedestrian) whose position has been determined by the image matching unit 613 begins to overlap the reference line BL. The region from the position where the reference line BL starts to overlap to the current position of the reference line BL is a region where parallax values are continuous in the Y direction, and the length of the region in the Y direction is a predetermined length (for example, , 20 [cm]) or more.

  At the same time, when the contours of a plurality of pedestrians are extracted, such as the contour 901 shown in FIG. 21, the boundary determination unit 614 and the pedestrian to be tracked whose position is determined by the image matching unit 613 For other pedestrians, the parallax in the Y direction is determined for the region from the position where the isolated area of the other pedestrian begins to overlap the reference line BL to the current position of the reference line BL, as in the above-described process. It is determined whether the value is a continuous region and the length of the region in the Y direction is a predetermined length (for example, 20 [cm]). In this case, for another pedestrian, the region from the position where the reference line BL begins to overlap to the current position of the reference line BL is a region where parallax values continue in the Y direction, and the length of the region in the Y direction If no region having a length equal to or longer than a predetermined length (for example, 20 [cm]) is not detected, the boundary determination unit 614 does not perform the subsequent boundary determination processing.

  Then, when the pedestrian whose position is determined by the image matching unit 613 is on the right side of another pedestrian, the boundary determination unit 614 overlaps the reference line BL for each isolated region of the pedestrian and another pedestrian. The areas from the start position to the current position of the reference line BL are areas where parallax values are continuous in the Y direction, and the length of the area in the Y direction is a predetermined length (for example, 20 [cm ] The pedestrian in the X direction is the midpoint between the left end of the pedestrian area at the position in the Y direction (second position) of the reference line BL and the right end of another pedestrian area at the time point And the boundary position with another pedestrian. On the other hand, when the pedestrian whose position is determined by the image matching unit 613 is on the left side of another pedestrian, the boundary determination unit 614 overlaps the reference line BL for each isolated region of the pedestrian and another pedestrian. The areas from the start position to the current position of the reference line BL are areas where parallax values are continuous in the Y direction, and the length of the area in the Y direction is a predetermined length (for example, 20 [cm ]) The middle point between the right end of the pedestrian region at the position in the Y direction (second position) of the reference line BL and the left end of another pedestrian at the time of the above becomes different from the pedestrian in the X direction. Boundary position with pedestrians. For example, in FIG. 26, when the pedestrian on the right side in the drawing is a pedestrian whose position is determined by the image matching unit 613, the boundary determination unit 614 determines the isolated area of each pedestrian and another pedestrian. The areas from the position where the reference line BL begins to overlap to the current position of the reference line BL are areas where parallax values are continuous in the Y direction, and the length of the area in the Y direction is a predetermined length. (For example, 20 [cm]) or more, the left end of the pedestrian region at the position in the Y direction of the reference line BL is detected as the detection position P3, and the right end of another pedestrian region is detected as the detection position P4. To do. Then, the boundary determination unit 614 detects the midpoint between the detection position P3 and the detection position P4 as the boundary position Pb.

  The boundary determination unit 614 sends information about the position of the pedestrian, the position and size of the detection frame of the pedestrian, and the boundary position to the correction processing unit 615. Then, control goes to a step S1234.

<Step S1234>
The correction processing unit 615 of the matching unit 610 performs frame correction processing on the frame (detection frame) of the detection area of the pedestrian whose position is detected in the current frame. Specifically, first, as shown in FIG. 27, the correction processing unit 615 performs the frequency of pixels including a parallax value in the X direction for an image in the pedestrian detection frame 820 on the parallax image corresponding to the current frame. And a histogram 911 indicating the frequency of pixels including a parallax value in the Y direction. Then, as illustrated in FIG. 27, the correction processing unit 615 sets the positions in the X direction that exceed the threshold Th in the histogram 910 as the positions of the left end and the right end of the corrected detection frame 821, and sets the threshold Th in the histogram 911. The positions in the Y direction that exceed the values are the positions of the upper end and the lower end of the corrected detection frame 821, respectively. The threshold value Th may be a value of 10 to 20% with respect to the maximum value of the histogram, for example. In this case, in FIG. 27, the threshold values in the X direction and the Y direction are set to the threshold value Th, but they need not be the same threshold value. In this way, the image of the detection frame 821 that has been subjected to the frame correction processing by the correction processing unit 615 becomes a detection region that is finally detected by the pedestrian matching processing by the matching unit 610. Then, the correction processing unit 615 includes information (position, size, etc.) of the detected detection area of the pedestrian in the recognition area information of the pedestrian. And the matching process for pedestrians is complete | finished.

  The matching process for pedestrians of the matching unit 610 is performed by the processes in steps S1231 to S1234 described above. In addition, after the pedestrian matching process of the matching unit 610 is finished, as described above, the feature update unit 630 determines that the position of the pedestrian is determined by the image matching unit 613 in the prediction region 801 corresponding to the current frame. The outline of the pedestrian determined by the above is updated instead of the currently stored outline template 902 as the outline template used in the parallax image corresponding to the next frame. Further, the feature update unit 630 creates an image template for the pedestrian determined by the position of the pedestrian determined by the image matching unit 613 in the prediction area 811 of the current frame, and stores the currently stored image. It updates instead of the template 902a. In this case, for example, the feature update unit 630 uses, as an image template to be used in the next frame, an image composed of N pixels downward from the position of each pixel of the confirmed pedestrian outline in the current frame. That's fine.

  As described above, in the pedestrian matching process in the tracking process of the object recognition apparatus 1 according to the present embodiment, the contour from the pedestrian's head to the vicinity of the shoulder is detected by the shape matching process, and Template matching is used to identify pedestrian position candidates in the X direction, and image matching processing is used for image matching in the Y direction at the candidate positions in the X direction using image templates from the head to the vicinity of the shoulder. Thus, by specifying the position of the pedestrian in the Y direction, the position of the pedestrian is finally specified. In this way, since the position of the pedestrian is detected using the contour from the head of the pedestrian to be tracked and the vicinity of the shoulder, the posture of the pedestrian's limbs or the like changes or is different. Even if there is another pedestrian in the vicinity such as wearing clothes, the pedestrian can be detected individually with high accuracy. In addition, the pedestrian position candidates in the X direction can be specified in advance by shape matching processing, and template matching can be performed using the image template in the Y direction at the candidate positions in the X direction. The processing speed can be improved.

  In addition, a parallax value histogram or contour is determined by a boundary determination process for determining a position that is assumed to be the head of a pedestrian parallax value lump (isolated region) and determining a boundary with another pedestrian. The boundary with another pedestrian can be specified with higher accuracy than the process of finding a local valley using the shape. As a result, since the positions of a plurality of pedestrians are approaching, it is possible to detect individual pedestrians even in a state where they are easily detected as one object.

  In addition, the shape matching process, the image matching process, and the boundary determination process are not performed on the entire image (luminance image or parallax image), but on the prediction region in which the pedestrian is predicted to exist by the movement prediction unit 600. Therefore, the processing speed can be improved as compared with the case of processing the entire image.

  In the above-described embodiment, the cost value C is an evaluation value that represents dissimilarity, but may be an evaluation value that represents similarity. In this case, the shift amount d at which the cost value C, which is the similarity, is maximized (extreme value) is the parallax value dp.

  Moreover, although the above-mentioned embodiment demonstrated the object recognition apparatus mounted in the motor vehicle as the vehicle 70, it is not limited to this. For example, it may be mounted on a vehicle such as a motorcycle, bicycle, wheelchair, or agricultural cultivator as an example of another vehicle. In addition to a vehicle as an example of a moving body, a moving body such as a robot may be used.

  In the above-described embodiment, when at least one of the functional units of the parallax value deriving unit 3 and the recognition processing unit 5 of the object recognition device 1 is realized by executing a program, the program is stored in a ROM or the like in advance. Provided embedded. The program executed by the object recognition apparatus 1 according to the above-described embodiment is a file in an installable format or an executable format, and is a CD-ROM, a flexible disk (FD), a CD-R (Compact Disk Recordable). It may be configured to be recorded on a computer-readable recording medium such as a DVD (Digital Versatile Disc). Further, the program executed by the object recognition apparatus 1 of the above-described embodiment may be configured to be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. The program executed by the object recognition apparatus 1 according to the above-described embodiment may be configured to be provided or distributed via a network such as the Internet. The program executed by the object recognition apparatus 1 according to the above-described embodiment has a module configuration including at least one of the above-described functional units. As actual hardware, the CPU 52 (CPU 32) is described above. By reading and executing the program from the ROM 53 (ROM 33), the above-described functional units are loaded onto the main storage device (RAM 54 (RAM 34) or the like) and generated.

DESCRIPTION OF SYMBOLS 1 Object recognition apparatus 2 Main body part 3 Parallax value derivation part 4 Communication line 5 Recognition processing part 6 Vehicle control apparatus 7 Steering wheel 8 Brake pedal 10a, 10b Imaging part 11a, 11b Imaging lens 12a, 12b Aperture 13a, 13b Image sensor 20a, 20b Signal converter 21a, 21b CDS
22a, 22b AGC
23a, 23b ADC
24a, 24b Frame memory 30 Image processing unit 31 FPGA
32 CPU
33 ROM
34 RAM
35 I / F
39 Bus line 51 FPGA
52 CPU
53 ROM
54 RAM
55 I / F
58 CANI / F
59 bus line 60 device control system 70 vehicle 100a, 100b image acquisition unit 200a, 200b conversion unit 300 parallax value calculation processing unit 301 cost calculation unit 302 determination unit 303 first generation unit 500 second generation unit 510 clustering processing unit 520 tracking processing Unit 600 movement prediction unit 610 matching unit 611 determination unit 612 shape matching unit 613 image matching unit 614 boundary determination unit 615 correction processing unit 620 check unit 630 feature update unit 640 state transition unit 700 road surface 700a road surface unit 701 utility pole 701a power pole unit 702 vehicle 702a Car part 711 Left guard rail 711a to 711c Left guard rail part 712 Right guard rail 712a to 712c Right guard rail part 713 Car 713a to 713c Car part 714 Car 714a -714c Vehicle part 721-724 Detection area 721a-724a Detection frame 800-802 Prediction area 811 Prediction area 820, 821 Detection frame 901 Outline 902 Outline template 902a Image template 902b Rectangular template 910, 911 Histogram B Base line length BL Base line C Cost Value d shift amount dp parallax value E object EL epipolar line f focal length Ia reference image Ib comparison image Ip parallax image p reference pixel P1, P2 candidate position P3, P4 detection position pb reference area Pb boundary position q candidate pixel qb candidate area RM Real U Map S, Sa, Sb Point Th Threshold UM, UM_H U Map VM V Map Z Distance

JP 2014-146267 A

Claims (10)

In the distance image corresponding to the current frame, when searching from the upper side to the lower side of the object, obtain the contour of the object by connecting the pixels that hit the pixel of the distance information of the object, First matching means for specifying a candidate position in the lateral direction of the object when the contour corresponds to an object to be detected by template matching using a contour template with respect to the contour;
Second matching means for performing template matching using an image template in a vertical direction from the candidate position in the distance image, and determining a horizontal position of the object and a first position in the vertical direction;
An image processing apparatus.   The image processing apparatus according to claim 1, wherein the first matching unit performs template matching on the contour using the contour template including at least a part of a head of a person.   When a plurality of the contours corresponding to the object to be detected are specified by the first matching means, and when searching from the upper side to the lower side of each object in the distance image, A second position in the vertical direction when the length of the distance information pixels in the direction reaches a predetermined length is specified, and a horizontal boundary of each object is determined based on the second position. The image processing apparatus according to claim 1, further comprising a determination unit that determines the value.   The determination means determines, as the boundary, a midpoint of each position in the lateral direction of an end portion closer to another object in the pixel row of the distance information in the lateral direction of the second position of each object. Item 4. The image processing apparatus according to Item 3.   In the distance image corresponding to the previous frame of the current frame, the contour of the object whose position is determined by the second matching means is updated as the contour template, and corresponds to the contour in the previous frame 5. The image processing apparatus according to claim 1, further comprising an update unit configured to update an image configured by a predetermined number of pixel groups in a downward direction from the position of each pixel as the image template. A prediction unit for obtaining a prediction region in which the object exists in a distance image corresponding to the current frame;
The first matching processing means performs template matching using the contour template in the prediction region,
The image processing apparatus according to claim 1, wherein the second matching processing unit performs template matching using the image template in an area of the current frame corresponding to the prediction area. First imaging means for obtaining a first captured image by imaging a subject;
A second imaging unit that is arranged at a position different from the position of the first imaging unit and obtains a second captured image by imaging the subject;
Generating means for generating the distance image based on distance information obtained for the subject from the first captured image and the second captured image;
Detection means for newly detecting an object based on the first captured image or the second captured image and the distance image;
The image processing apparatus according to any one of claims 1 to 6,
An object recognition device comprising: The object recognition device according to claim 7;
A control device that controls a control object based on information on the object detected by the object recognition device;
Equipment control system equipped with. In the distance image corresponding to the current frame, when searching from the upper side to the lower side of the object, obtaining the contour of the object by connecting the pixels that have reached the pixel of the distance information of the object Steps,
A first matching step of specifying a candidate position in the lateral direction of the object when the contour corresponds to an object to be detected by template matching using a contour template with respect to the contour;
A second matching step of performing template matching using an image template in a vertical direction from the candidate position in the distance image, and determining a horizontal position of the object and a first position in the vertical direction;
An image processing method. Computer
In the distance image corresponding to the current frame, when searching from the upper side to the lower side of the object, obtain the contour of the object by connecting the pixels that hit the pixel of the distance information of the object, First matching means for specifying a candidate position in the lateral direction of the object when the contour corresponds to an object to be detected by template matching using a contour template with respect to the contour;
Second matching means for performing template matching using an image template in a vertical direction from the candidate position in the distance image, and determining a horizontal position of the object and a first position in the vertical direction;
Program to make it function.

Images