JP6569416B2 - Image processing apparatus, object recognition apparatus, device control system, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing device, an object recognition device, a device control system, an image processing method, and an image processing program.

  In recent years, with the development of information processing technology and image processing technology, in-vehicle recognition technology for detecting (recognizing) a person or a car at high speed using parallax has been developed. For example, when the detection target is a person or a car, image data that is not targeted for detection may be rejected from the captured image data to improve detection accuracy and reduce image data.

  Further, in Patent Document 1, an identification value indicating a degree of identification as a detection target based on a model of the detection target created in advance from each of the images output in time series by the imaging unit is a predetermined value. An object detection apparatus is disclosed that acquires the above regions, collates whether the acquired regions correspond to a sequence of regions stored in a storage unit, and determines whether the sequence of regions is a detection target. ing.

  However, when the parallax is calculated on the basis of the luminance image, the parallax may not be accurately calculated due to the problem of being affected by the illumination condition or the texture of the subject, or the block matching performance of the luminance image. Therefore, when image data that is not subject to detection is rejected from captured image data, the accuracy of rejection may not be sufficiently improved. In other words, when image data is rejected by a predetermined method, even if it is visually similar, if the image information changes due to lighting conditions as described above, an object that can be rejected and an object that cannot be rejected are generated due to the difference. There was a possibility.

  The present invention has been made in view of the above, and an image processing device, an object recognition device, a device control system, and an image that can accurately reject non-target image data from captured image data. It is an object to provide a processing method and an image processing program.

  In order to solve the above-described problems and achieve the object, the present invention occurs between a reference image in which one or more objects are imaged and a comparative image imaged from an imaging position different from the reference image. First determination for determining, based on a frequency distribution of parallax, that part of the reference image data indicating the reference image is installation object image data indicating an installation object that is installed in a predetermined direction. A first rejection unit that rejects the installation object image data from the reference image data, and a part of the reference image data that the first rejection unit rejects the installation object image data. Based on the data, a second determination unit that determines that the image is continuous object image data indicating an object continuous to the installation object, and the reference image data in which the first rejection unit rejects the installation object image data, The continuous image data And having a second rejection unit for rejecting the data, the.

  According to the present invention, there is an effect that image data that is not subject to detection can be accurately rejected from captured image data.

FIG. 1 is a diagram showing the position of a stereo camera mounted on a vehicle which is a device control system according to an embodiment of the present invention. FIG. 2 is a diagram illustrating a stereo camera included in the vehicle and a configuration around the stereo camera. FIG. 3 is a diagram illustrating a configuration example of an object recognition apparatus including a stereo camera. FIG. 4 is a functional block diagram showing an outline of the device control function based on the image of the vehicle. FIG. 5 is a diagram schematically illustrating a reference image captured by the camera unit. FIG. 6 is a diagram showing a frequency U map corresponding to the reference image shown in FIG. FIG. 7 is a diagram showing a result of converting the frequency U map shown in FIG. 6 into a real frequency U map. FIG. 8 is a block diagram illustrating a configuration example of the rejection unit. FIG. 9 is a diagram illustrating an example of the frequency distribution of the parallax of the preceding vehicle in the overhead view map. FIG. 10 is a diagram illustrating a process of detecting the parallax frequency distribution of the preceding vehicle traveling in the left lane, the corner of the preceding vehicle, and the like. FIG. 11 is a diagram illustrating a block a in which the viewing angle is maximized and a block b in which the viewing angle is minimized. FIG. 12 is a diagram illustrating a first selection method for points A and B having a high parallax frequency. FIG. 13 is a diagram illustrating a second method for selecting points A and B having a high parallax frequency. FIG. 14 is a diagram illustrating a third method for selecting points A and B having a high parallax frequency. FIG. 15 is a diagram illustrating a process of detecting a parallax frequency distribution of a preceding vehicle traveling in the same lane as the own vehicle, a corner of the preceding vehicle, and the like. FIG. 16 is a diagram illustrating a state where there is a side wall on the left side of the traveling vehicle. FIG. 17 is a diagram illustrating a case where the passing vehicle traveling on the right side of the traveling vehicle and the far side wall object are newly detected objects. FIG. 18 is a diagram illustrating a case where there are one and two side walls. FIG. 19 is a diagram illustrating two surfaces formed by a vehicle traveling ahead. FIG. 20 is a flowchart showing the operation of the first calculation unit. FIG. 21 is a flowchart illustrating a first operation example of the rejection unit. FIG. 22 is a diagram illustrating a misrecognized object appearing in the side wall object. FIG. 23 is a diagram illustrating an example of a case where a surface that has not been rejected is determined to be a part of the side wall object. FIG. 24 is a flowchart illustrating a second operation example of the rejection unit. FIG. 25 is a diagram illustrating an example of a case where the remaining object is determined as an erroneously detected object. FIG. 26 is a diagram illustrating conditions for determining a remaining object as an erroneously detected object. FIG. 27 is a diagram schematically illustrating a third operation example of the rejection unit. FIG. 28 is a diagram schematically illustrating a fourth operation example of the rejection unit. FIG. 29 is a flowchart showing a fourth operation example of the rejection unit. FIG. 30 is a diagram schematically illustrating a fifth operation example of the rejection unit.

  A device control system according to an embodiment will be described below with reference to the accompanying drawings. FIG. 1 is a diagram showing the position of a stereo camera 2 mounted on a vehicle 1 that is an example of a device control system according to an embodiment of the present invention. For example, the vehicle 1 is an automobile in which a stereo camera 2 that captures a predetermined imaging range on the front side in the traveling direction is provided at the front. As will be described later with reference to FIG. 3, the stereo camera 2 includes two image sensors 22 and is an imaging unit that captures two images of the left visual field and the right visual field.

  FIG. 2 is a diagram illustrating a configuration example of the stereo camera 2 included in the vehicle 1 and its surroundings. The stereo camera 2 outputs, for example, two captured images to a vehicle ECU (Engine Control Unit) 3. The vehicle ECU 3 is installed in the vehicle 1 and comprehensively performs the engine of the vehicle 1 and other electrical control and processing of the vehicle 1.

  FIG. 3 is a diagram illustrating a configuration example of the object recognition device 4 including the stereo camera 2. The object recognition device 4 includes, for example, a stereo camera 2 and an image processing device 30. In the stereo camera 2, a camera unit 2 a for the left eye and a camera unit 2 b for the right eye are assembled in parallel (horizontal) to shoot a moving image (or still image) in the imaging target area.

  Each of the camera units 2a and 2b includes a lens 21, an image sensor 22, and a sensor controller 23. The image sensor 22 is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor. The sensor controller 23 performs, for example, exposure control of the image sensor 22, image readout control, communication with an external circuit, image data transmission control, and the like.

  The image processing device 30 is provided, for example, in the vehicle ECU 3 shown in FIG. The image processing apparatus 30 includes, for example, a data bus line 300, a serial bus line 302, a CPU (Central Processing Unit) 304, an FPGA (Field-Programmable Gate Array) 306, a ROM (Read Only Memory) 308, and a RAM (Random Access Memory) 310. Serial IF (Interface) 312 and data IF (Interface) 314.

  The stereo camera 2 described above is connected to the image processing apparatus 30 via the data bus line 300 and the serial bus line 302. The CPU 304 controls the overall operation of the image processing apparatus 30 and executes image processing and image recognition processing. Luminance image data of captured images captured by the image sensors 22 of the camera units 2 a and 2 b are written into the RAM 310 of the image processing device 30 via the data bus line 300. Sensor exposure value change control data, image readout parameter change control data, various setting data, and the like from the CPU 304 or FPGA 306 are transmitted and received via the serial bus line 302.

  The FPGA 306 generates real-time parallax images by performing parallax calculation based on, for example, gamma correction, distortion correction (parallelization of left and right images), and block matching, which is a process that requires real-time processing for image data stored in the RAM 310. , Write again to the RAM 310. The CPU 304 performs control of each sensor controller 23 of the stereo camera 2 and overall control of the image processing apparatus 30. The ROM 308 stores a (three-dimensional) object recognition program for executing situation recognition, prediction, object recognition, and the like.

  The object recognition program is an example of an image processing program. The CPU 304 acquires, for example, CAN (Controller Area Network) information of the host vehicle as parameters (vehicle speed, acceleration, steering angle, yaw rate, etc.) via the data IF 314. Then, the CPU 304 executes various processes such as situation recognition using the luminance image data and the parallax image data stored in the RAM 310 in accordance with the object recognition program stored in the ROM 308, for example, a detection target such as a preceding vehicle. Is detected (recognized). The CPU 304 also performs image data rejection processing.

  Detection data (recognition data) of a detection target (recognition target) is output via a serial IF 312 to, for example, an automatic brake system or an automatic speed control system provided with a control function in the vehicle ECU 3. The automatic brake system performs brake control of the vehicle 1 using recognition data to be recognized. In addition, the automatic speed control system performs speed control of the vehicle 1 using recognition data to be recognized.

  Next, device control executed by the vehicle 1 using the stereo camera 2 will be described. FIG. 4 is a functional block diagram showing an outline of the device control function based on the image of the vehicle 1. In FIG. 4, a device control function performed by the vehicle 1 using the stereo camera 2 is shown.

  The vehicle 1 includes an imaging unit (stereo camera) 2, a parallax calculation unit 40, a clustering unit 42, a rejection unit 44, a tracking unit 46, and an output control unit 32 in order to execute device control using images. Here, the parallax calculation unit 40, the clustering unit 42, the rejection unit 44, and the tracking unit 46 are realized by the image processing apparatus 30 executing the object recognition program. The output control unit 32 is provided, for example, in the vehicle ECU 3 and controls a mechanism such as an engine and a brake provided in the vehicle 1. Note that the parallax calculation unit 40, the clustering unit 42, the rejection unit 44, the tracking unit 46, and the output control unit 32 are not limited to being configured by software, and some or all are configured by hardware. May be.

  Here, a case where the vehicle 1 processes one frame image of a moving image (two images of the left visual field and the right visual field) will be described as an example. First, the imaging unit (stereo camera) 2 acquires frame information. The input frames are two images captured by the stereo camera 2 using the two image sensors 22. One of the two images is a reference image and the other is a comparative image. The comparative image is taken from a different imaging position from the reference image. The camera unit 2a captures a reference image schematically illustrated in FIG. 5, for example. In the reference image, the preceding vehicle 500, the oncoming vehicle 502, the left guard rail 504a, and the right guard rail 504b are photographed.

  The parallax calculation unit 40 (FIG. 4) calculates the parallax for each pixel by block matching of the local region using the reference image and the comparison image.

  The clustering unit 42 counts the frequency of the parallax calculated by the parallax calculation unit 40, and distributes pixels having pixel values according to the frequency of the parallax on the two-dimensional orthogonal coordinate system of the luminance image (frequency U map). Create For example, the clustering unit 42 creates a frequency U map using the parallax of an image having a height from a road surface of 30 cm to 2 m taken in the reference image and the comparison image.

  FIG. 6 is a frequency U map corresponding to the reference image shown in FIG. In the frequency U map, the horizontal axis is the same as the horizontal direction (X direction) of the image, and the vertical axis is the depth direction. In the frequency U map shown in FIG. 6, pixels indicating parallax are densely arranged at positions corresponding to the preceding vehicle 500, the oncoming vehicle 502, the left guard rail 504 a, and the right guard rail 504 b captured in the reference image of FIG. 5. Area (dense area).

  Further, the clustering unit 42 converts the frequency U map into a real frequency U map. The distance in the x direction on the frequency U map is higher on the frequency U map as the distance from the stereo camera 2 increases (towards the upper side of the frequency U map) even if the distance in the x direction in the real space is the same. As it approaches the vanishing point, it becomes shorter. FIG. 7 is a diagram showing a result of converting the frequency U map shown in FIG. 6 into a real frequency U map. The real frequency U map is obtained by converting the distance in the X direction in the frequency U map into the position in the x direction on the real space regardless of the distance from the stereo camera 2 to the photographed object. In other words, the real frequency U map is a map that represents the distribution of the parallax frequency when the region captured in the reference image is viewed from directly above in real space. The vertical axis of the real frequency U map indicates the distance from the stereo camera 2.

  In the real frequency U map shown in FIG. 7, the left guard rail 504a and the right guard rail 504b are dense areas extending in the vertical direction. In addition, the preceding vehicle 500 and the oncoming vehicle 502 are dense regions having shapes corresponding to the back surface and one side surface of the vehicle. Thus, in the real frequency U map, the dense area indicating the object has a shape close to the shape of the object when the area in front of the vehicle 1 in the real space is viewed from directly above. The real frequency U map may be created for the entire frame (parallax image) or may be created for a partial region. Note that the real frequency U map may be referred to as an overhead map.

  Then, the clustering unit 42 detects a dense area on the overhead map as a newly detected object. At this time, the clustering unit 42 assigns the individual information such as the object type (person or car) predicted from the coordinates and size on the luminance image to the newly detected object. In the bird's-eye view map shown in FIG. 7, the preceding vehicle 500, the oncoming vehicle 502, the left guard rail 504 a, and the right guard rail 504 b captured in the reference image in FIG. 5 are close to the actual shape viewed from directly above. It has become.

  The rejection unit 44 (FIG. 4) uses the frame information (luminance image, parallax image), the bird's-eye view map, and the individual information of the newly detected object to determine whether the newly detected object is a detection target (recognition target) object. It is determined whether the object is a detection target. Here, for example, humans and automobiles are set as detection targets, and other side walls are set as non-detection targets. Note that the detection target object is treated as an object (tracking target) to be detected continuously. The rejection unit 44 determines that the image data indicating the non-detection target is the image data to be rejected (rejection determination). The rejection unit 44 assigns a rejection determination result to the image data (individual information) that is the target of rejection.

  Further, the rejection unit 44 will be described in detail. FIG. 8 is a block diagram illustrating a configuration example of the rejection unit 44. The rejection unit 44 may include, for example, a network bus 440, an acquisition unit 442, a first processing unit 444, and a second processing unit 446, and may include a storage unit 448. The memory | storage part 448 memorize | stores the result which each part which comprises the rejection part 44 processed, for example. Further, an output unit 449 may be provided. The output unit 449 outputs the rejection result in the rejection unit 44 to a subsequent processing unit (for example, the tracking unit 46). And the rejection part 44 may perform the process with respect to a view frame using the non-detection object rejected before the view frame. Note that the acquisition unit 442, the first processing unit 444, and the second processing unit 446 may be partially or entirely configured by hardware or software.

  The network bus 440 is a bus that transmits data. For example, the network bus 440 transmits data for each newly detected object.

  The acquisition unit 442 acquires individual information of a newly detected object detected by the clustering unit 42, a frame (luminance image, parallax image), an overhead map, and the like. More specifically, the acquisition unit 442 acquires information necessary for the rejection process. The acquisition unit 442 may directly read information acquired by a device such as the stereo camera 2 or store the information in a recording medium such as a CD, DVD, HDD, or a network storage in advance, and store these information when necessary. It may be read. Further, the acquisition unit 442 may acquire still images or sequentially acquire moving image data captured by the stereo camera 2.

  Information acquired by the acquisition unit 442 may be configured in any combination. For example, the acquisition unit 442 may acquire the overhead map generated from the frame (parallax image) captured by the stereo camera 2 and the newly detected object information generated as a clustering result. The newly detected object information is obtained by arbitrarily combining object types (pedestrians or people), rejection information, tracking information, and the like determined from the position and size on the luminance image obtained as a clustering result for each object. Further, the acquisition unit 442 may acquire a cropped parallax image, or may acquire information indicating a pixel at a corresponding location in the parallax image based on coordinate information included in the object information.

  The first processing unit 444 includes, for example, a first calculation unit 450, a target surface determination unit 451, a first determination unit 452, and a first rejection unit 453. Note that the first processing unit 444 may have a configuration that does not include the target surface determination unit 451.

  The first calculation unit 450 calculates the surface of the newly detected object and further calculates the calculated angle of the surface. Note that the angle of the surface of the newly detected object is, for example, an angle on an overhead map with respect to straight lines passing through the two image sensors 22 of the stereo camera 2.

  Here, a specific example in which the first calculation unit 450 calculates the surface of the newly detected object and further calculates the calculated angle of the surface will be described. FIG. 9 is a diagram illustrating an example of the frequency distribution of the parallax of the preceding vehicle in the overhead view map. If the resolution of the bird's-eye view map is, for example, 128 mm, the parallax frequency distribution of the preceding vehicle has a shape in which a plurality of 128 mm × 128 mm blocks are arranged. Note that if the resolution of the overhead view map is increased (for example, 64 mm), the accuracy increases, but the density of the blocks becomes sparse and it becomes difficult to detect the object. Further, when the resolution of the overhead view map is lowered (for example, 256 mm), the density of the blocks becomes dense. However, since the resolution is low, it is difficult to detect the exact position of the object.

  In the parallax frequency distribution of the preceding vehicle, the blocks extend in a diagonal direction at a position far from the stereo camera 2 in the Z-axis direction and the X-axis direction shown in FIG. This is due to parallax errors (dispersion) that occur according to conditions at the time of shooting. When only the side surface of the preceding vehicle is shown in the reference image and the comparison image from which the bird's-eye view map is created, there are few blocks extending in the diagonal direction. In addition, in the reference image and comparison image from which the bird's-eye view map is created, if an object that is located farther than the preceding vehicle is reflected in the background, the error due to the parallax of the background Thus, the number of blocks extending in the diagonal direction increases.

  Next, a process of detecting a corner of the preceding vehicle (a boundary between the side surface and the rear surface of the preceding vehicle) and the like using the parallax frequency distribution of the preceding vehicle will be described. FIG. 10 is a diagram illustrating a process of detecting the parallax frequency distribution of the preceding vehicle traveling in the left lane, the corner of the preceding vehicle, and the like.

  First, the first calculation unit 450 performs a parallax in each of a region near the position where the viewing angle is minimum and a region near the position where the viewing angle is maximum with respect to the block dense area indicating the parallax of the detected preceding vehicle. The points A and B with high frequency are calculated. Note that the viewing angle is, for example, an angle with respect to a straight line extending in the Z direction from the center point of each of the two image sensors 22 (the center point of the imaging position of each of the reference image and the comparison image).

  More specifically, as shown in FIG. 11, the first calculation unit 450 first determines a region near the position where the viewing angle is minimum and a region near the position where the viewing angle is maximum. The block a having the largest viewing angle and the block b having the smallest viewing angle are searched and selected.

  Next, as shown in FIG. 12, the first calculation unit 450 sets a circumscribed rectangle 600 that includes each block as a first selection method, and includes a triangular area 602 that includes the block a and a block b. The triangular area 604 to be set is set. Here, a triangular area 602 is set at the lower left of the circumscribed rectangle 600, and a triangular area 604 is set at the upper right. Then, the first calculation unit 450 selects a block having the highest frequency in the triangular area 602 as the point A, and selects a block having the highest frequency in the triangular area 604 as the point B.

  Further, as shown in FIG. 13, the first calculation unit 450 sets the second selection method to a predetermined rectangular range 606 (for example, 5 × 5 or rectangular) centered on the selected block a and block b, respectively. Point A and point B with the highest frequency are selected from the included block set.

  In addition, as shown in FIG. 14, the first calculation unit 450 uses a circle 608 (for example, a radius of 3.5 or an ellipse with a predetermined radius) centered on the selected block a and block b as shown in FIG. ), The points A and B with the highest frequency are selected from the block set included in ().

  As described above, the first calculation unit 450 calculates the points A and B having a high parallax frequency by using any one of the first selection method to the third selection method.

  Next, as shown in FIG. 10, the first calculation unit 450 sets a point D on the circumference having the points A and B as end points of the diameter on the overhead view map. And the 1st calculation part 450 calculates the average value of the frequency of the point (block) on the bird's-eye view map through which line segment AD and line segment BD pass as a frequency average value.

  The first calculation unit 450 may calculate the frequency average value as follows, for example. First, the first calculation unit 450 sets the center of each block on the overhead view map as an integer coordinate value of the X coordinate and the Z coordinate. Then, the value is changed one by one from the X coordinate value of the point A to the X coordinate value of the point D, and each X coordinate value is quantized by rounding off the decimal point of the Z coordinate of the line segment AD. . Then, the first calculation unit 450 sums up the frequency values of each block of the quantized Z coordinate in each X coordinate. That is, the total frequency value for the line segment AD is calculated.

  Similarly, the first calculation unit 450 also calculates the total value of the frequencies for the line segment BD. Then, the first calculation unit 450 divides the total value of the frequency of the line segment AD and the line segment BD by the number of each block in which the value of its own frequency is included in the total value. In the case of the example shown in FIG. 10, the frequency values corresponding to the blocks with ◯ are totaled and divided by 11, which is the number of blocks with ◯. In addition, the first calculation unit 450 calculates the total value by integrating the frequency value for each block on the overhead view map through which the line segment AD and the line segment DB pass, and the length of the line segment AD and the line segment DB. The frequency average value may be calculated by dividing by. Further, it is possible to use the total value of the frequencies without using the frequency average value. In short, any method can be used as long as it can be determined that two line segments are arranged on the parallax distribution.

  Further, the first calculation unit 450 moves the point D from the point A to the point B, and repeats the calculation of the frequency average value. For example, the first calculation unit 450 moves the point D so that the angle from the center of the circle changes by a predetermined angle (for example, 2 °) on the circumference having the points A and B as the end points of the diameter. And the 1st calculation part 450 outputs the point D where a frequency average value becomes maximum between arc AB as Dmax. In the case of the example shown in FIG. 10, the angles of the line segment ADmax and the line segment BDmax with respect to the X-axis direction are 0 degrees and 90 degrees, respectively.

  FIG. 15 is a diagram illustrating a process of detecting a parallax frequency distribution of a preceding vehicle traveling in the same lane (front) as the own vehicle, a corner of the preceding vehicle, and the like. When the point Dmax at which the frequency average value is maximized is calculated with respect to the parallax frequency distribution of the preceding vehicle traveling in the same lane (front) as the own vehicle, the position of the point Dmax is the position of the point A or the point B. Become. In the case of the example shown in FIG. 15, the angles of the line segment ADmax and the line segment BDmax with respect to the X-axis direction are both 0 degrees.

  Then, the first calculation unit 450 calculates the angle of the plane corresponding to the line segment ADmax, the angle of the plane corresponding to the line segment BDmax, and the (X, Y) coordinates of the vertical sides of each plane. That is, the first calculation unit 450 calculates the angle of the surface of the object imaged in the reference image with respect to the straight lines passing through the imaging positions of the reference image and the comparison image based on the parallax frequency distribution.

  When the newly detected object has a plurality of surfaces, the target surface determination unit 451 (FIG. 8) determines a surface that is a target of the rejection process according to the number and shape (size) of the plurality of surfaces (FIG. 18, 19 later).

  The first determination unit 452 determines whether the newly detected object is a detection target or a non-detection target depending on the angle of the surface calculated by the first calculation unit 450 or the presence / absence of the surface determined by the target surface determination unit 451. Determine. For example, the first determination unit 452 determines that a newly detected object having a surface angle of approximately 90 degrees (a predetermined range) is a non-detection target. Then, the first determination unit 452 sets image data indicating a non-detection target as a rejection target.

  FIG. 16 is a diagram illustrating a state where there is a side wall on the left side of the traveling vehicle. In the state illustrated in FIG. 16, the first determination unit 452 determines whether the newly detected object is a detection target or a non-detection target according to the angle of the surface calculated by the first calculation unit 450. For example, when the angle of the surface calculated by the first calculation unit 450 is within a predetermined range, the first determination unit 452 sets the newly detected object that is the processing target as the rejection target. For example, the side wall object may have an angle of 90 degrees or more with respect to a straight line passing through the two image sensors 22, and the angle exceeds 90 degrees at the center part (frame part) of the side wall object. . That is, the first determination unit 452 rejects the non-detection target without erroneously rejecting the detection target by setting the range of the angle of the surface that cannot be held by the vehicle as a predetermined value as a determination reference. The target can be determined.

  When the first determination unit 452 makes a determination on an overtaking vehicle, the overtaking vehicle often has a surface angle similar to that of the side wall object, so the surface angle serving as a reference for determination is set to 90 degrees or more. If it does, there is a possibility that the number of non-detectable objects that can be rejected will decrease. Therefore, the first determination unit 452 may limit the determination target using the position on the image and the actual distance from the own vehicle. Thereby, the 1st determination part 452 can set the range (for example, 70 degree | times or more) of a narrow viewing angle as a determination reference | standard with respect to the area | region which does not include the overtaking vehicle.

  FIG. 17 is a diagram illustrating a case where the passing vehicle traveling on the right side of the traveling vehicle and the far side wall object are newly detected objects. As shown in FIG. 17, the overtaking vehicle often exists at a short distance from the own vehicle (the center in FIG. 17). Therefore, the first determination unit 452 may measure the actual distance to each object centering on the own vehicle, and may set a new detection target (a frame part of the side wall object) whose distance is a predetermined value or more as a rejection processing target. Further, as a method of using the position on the image, there is a method in which the image is divided into several parts in the horizontal direction and the object included in the regions at both ends is determined. As described above, the first determination unit 452 determines that a part of the reference image data is the installation object image data based on the angle of the surface of the object having one surface calculated by the first calculation unit 450. To be rejected. That is, the first determination unit 452 determines that a part of the reference image data indicating the reference image is the installation object image data indicating the installation object that is continuously installed in the predetermined direction, and is to be rejected. It is said.

  Next, a process performed by the target surface determination unit 451 when the newly detected object has a plurality of surfaces will be described. FIG. 18 is a diagram illustrating a case where the side wall object has one surface (a) and two cases (b). Originally, there should be one surface of the side wall object, but when the surface and angle are calculated using the overhead view map, the length of the bottom of the surface as shown in FIG. There is a possibility that a surface that is greatly different from the other is calculated.

  It is necessary to clarify whether the newly detected object having two surfaces is a side wall object or a vehicle. Accordingly, the target surface determination unit 451 performs processing for confirming that the vehicle is not a vehicle and processing for determining which of the two surfaces is to be calculated for a newly detected object having two surfaces.

  The target surface determination unit 451 calculates the ratio of the bases of the two surfaces with respect to the newly detected object having two surfaces, and calculates whether the newly detected object is a vehicle and the angle according to the calculated ratio. Determine the surface to be used.

  FIG. 19 is a diagram illustrating two surfaces formed by a vehicle traveling ahead. As shown in FIG. 19, when the newly detected object is a vehicle, two surfaces are calculated in the same manner as the side wall object, but the difference between the lengths of the bottoms of the two surfaces does not increase. Therefore, when two surfaces are calculated, the target surface determination unit 451 calculates the ratio of the bottom or top of the two surfaces, and if the ratio is equal to or greater than a predetermined value, the newly detected object is a side wall object. And the surface for calculating the angle is determined. On the other hand, when the ratio of the bases of the two surfaces is less than the predetermined value, the target surface determination unit 451 assumes that the newly detected object is a vehicle and does not perform the process for rejection.

  Note that the target plane determination unit 451 may calculate the ratio of the bottom (or top) on a two-dimensional image (luminance image, parallax image), or may calculate and use the actual distance from the overhead map. Good. Furthermore, the target surface determination unit 451 may use the area ratio of two surfaces instead of the ratio of sides. Moreover, even if three or more surfaces are calculated for the newly detected object, the target surface determination unit 451 similarly performs the above-described processing on any two surfaces and determines a surface to be used for the rejection determination. May be.

  In addition, the first determination unit 452 may have the function of the target surface determination unit 451. That is, the first determination unit 452 installs the object based on the ratio of the lengths of the plurality of surfaces indicated by the image data of the object to the image data indicating the object including the plurality of surfaces calculated by the first calculation unit 450. It may be configured to determine whether it is image data. Of course, an area ratio may be used instead of the length ratio described above. That is, the first determination unit 452 is based on the ratio of the area of each of the plurality of surfaces indicated by the image data of the object to the image data indicating the object including a plurality of surfaces calculated by the first calculation unit 450. It may be configured to determine whether it is data.

  The first rejection unit 453 (FIG. 8) rejects image data (installed object image data) indicating the non-detection target that is the rejection target by the first determination unit 452 from the image data of the reference image.

  In this way, the method in which the first determination unit 452 determines the image data of the object to be rejected according to the angle of the surface of the newly detected object is referred to as a surface angle rejection method. Also, a newly detected object whose image data has been rejected by the surface angle rejection method will be referred to as a reject object. Also, a newly detected object whose image data has not been rejected by the surface angle rejection method will be referred to as a remaining object. Further, a newly detected object that is a non-detection target for which image data is to be rejected by the surface angle rejection method and for which image data has not been rejected is referred to as an erroneously detected (falsely recognized) object.

  The second processing unit 446 includes a second calculation unit 454, a second determination unit 455, and a second rejection unit 456, and can be operated by arbitrarily combining the operations described below.

  The second calculation unit 454 performs at least one of the following operations. For example, the second calculation unit 454 calculates a predetermined feature amount of the reject object and a predetermined feature amount of the remaining object whose surface angle is within a predetermined range with respect to the surface angle of the reject object. To do. Further, the second calculation unit 454 calculates an approximate straight line that approximates the reject object on the overhead map. The second calculation unit 454 calculates the distance from the rejected object to the remaining object in the frame of interest and the distance from the rejected object to the remaining object in the frame before the frame of interest.

  The second determination unit 455 performs at least one of the following operations. For example, the second determination unit 455 compares the feature amounts of the rejected object and the remaining object calculated by the second calculation unit 454, and if the difference between the feature amounts is equal to or less than a predetermined value, the remaining object is not detected. It is determined that it is a target. That is, the second determination unit 455 determines that a part of the reference image data having a feature amount having a similarity greater than or equal to a predetermined value with respect to the feature amount included in the installation object image data is continuous object image data.

  Further, the second determination unit 455 determines that the remaining object existing on the approximate straight line calculated by the second calculation unit 454 is a false detection object and determines that it is a non-detection target. In addition, the second determination unit 455 is a continuous object image data that is a part of the reference image data indicating an object on an approximate straight line obtained by approximating the installation object image data so as to indicate the direction in which the installation object is continuous. May be determined.

  The second determination unit 455 determines that the remaining object is a non-detection target when an approximate straight line is sandwiched between the remaining object and an intermediate point between the imaging positions of the reference image and the comparison image. That is, when a part of the reference image data indicating the object puts an approximate line between the reference image data and the intermediate point of the imaging position of each of the comparison images, the second determination unit 455 determines the reference image data indicating the object. A part is determined to be rejection target image data.

  Further, the second determination unit 455 determines that the difference between the distance from the rejected object to the remaining object in the frame of interest and the distance from the rejected object (installed object) to the remaining object in the frame before the frame of interest is equal to or less than a predetermined value. The remaining object is determined as a non-detection target (continuous object image data). That is, the second determination unit 455 is a continuous object image that indicates a part of the reference image data in which the first rejection unit 453 rejects the installation object image data, based on the installation object image data. Judged as data.

  In addition, the second determination unit 455 performs the first determination based on the frequency distribution of the parallax generated between the frame of the reference image captured at the time before the reference image and the comparison image are captured and the frame of the comparison image. The installation object image data determined by the unit 452 may be regarded as installation object image data determined by the first determination unit 452 based on a frequency distribution of parallax generated between the reference image and the comparison image. That is, the second determination unit 455 may be configured to use past installation object image data for determination.

  Here, a supplementary description will be given of the feature amount of the image and the recognition process (detection process). Various image recognition methods have been proposed so far, but any method may be used as long as the similarity between two images can be quantified. For example, there is a method of performing template matching on an erroneously detected object that is erroneously determined to be a detection target using a reject object as a template.

  In addition, Bag-of-Visual-Words method that uses a vector called visual words obtained by quantizing local features such as SIFT (Scale-Invariant Feature Transform), and Deep Learning with multilayered neural nets are used. There is also a method.

  In order to quantify the relationship between two objects, a distance measure may be used in addition to the similarity. In the case of similarity, it is considered that two objects are similar when the score is high. On the other hand, when using a distance scale, it is considered that two objects are similar when the score is small. Therefore, finally, when the feature amount is calculated from the two objects, and the similarity or distance is calculated using them, the two objects are similar when this score is included in the predetermined range. Is determined.

  The second rejection unit 456 rejects the image data of the newly detected object determined by the second determination unit 455 as a non-detection target from the image data of the reference image. That is, the second rejection unit 456 rejects the continuous object image data (rejection target image data) determined by the second determination unit 455 from the reference image data in which the first rejection unit 453 rejects the installation object image data.

  The tracking unit 46 (FIG. 4) determines whether or not it is a tracking target when a newly detected object appears continuously in a plurality of frames. When the newly detected object is a tracking target, the tracking unit 46 adds the information to the individual information.

  The output control unit 32 controls a mechanism such as an engine and a brake of the vehicle 1 according to the result determined by the tracking unit 46.

  Next, operation | movement of the principal part which comprises the rejection part 44 is demonstrated using a flowchart. FIG. 20 is a flowchart showing the operation of the first calculation unit 450. As illustrated in FIG. 20, the first calculation unit 450 creates an overhead map using the reference image and the comparison image (S100).

  Next, the first calculation unit 450 calculates points A and B having a high parallax frequency in each of the region near the position where the viewing angle is minimum and the region near the position where the viewing angle is maximum on the overhead view map. . That is, the first calculation unit 450 sets both end points (A, B) shown in FIG. 10 and the like (S102), and sequentially sets points (D) on the circumference (S104).

  Next, the first calculation unit 450 calculates the frequency average value of points through which the line segment AD and line segment BD pass (S106), and determines the point (D) having the maximum frequency average value as Dmax (S108). ). This point Dmax is the closest corner position of the preceding vehicle.

  When the first calculation unit 450 uses a front car or a side wall object as a new detection object, one of the two surfaces may be detected to be extremely small. The first calculation unit 450 deletes the small surface (S110).

  The first calculation unit 450 calculates the angles of the AD and BD surfaces that have not been deleted (S112), and calculates the maximum and minimum of the X and Y coordinates of the left and right sides of each plane in the corresponding image frame ( S114).

  FIG. 21 is a flowchart illustrating a first operation example of the rejection unit 44. First, the rejection unit 44 reads an overhead map and newly detected object information (S200). Next, the rejection unit 44 uses the overhead map to determine rejection for each newly detected object by the surface angle rejection method (S202). Next, the rejection unit 44 detects image feature amounts of the rejection object and the remaining object, and measures the similarity (S204). At this time, the number and order of the newly detected objects processed by the rejection unit 44 may be arbitrarily set. Then, rejection unit 44 performs rejection determination based on whether or not the similarity is within a predetermined range (S206). The processing of S202 and S204 is performed for each rejected object, but the determination process of other rejected objects is performed for the remaining objects that are determined to be close by a certain rejected object and are rejected. You don't have to.

  Next, a second operation example of the rejection unit 44 will be described. FIG. 22 is a diagram illustrating a misrecognized object appearing in the side wall object. FIG. 22A shows a misrecognized object that appears in the side wall object on the luminance image. FIG.22 (b) has shown the misrecognized object which appears in a side wall thing on a bird's-eye view map. Here, it is assumed that three newly detected objects (surface A, surface B, and surface C) are calculated for the side wall object. Among these, surface A and surface C were rejected as non-detection targets, and surface B was not rejected because the surface angle was not accurately calculated due to the influence of parallax dispersion on the overhead view map. To do.

  At this time, the second determination unit 455 uses the rejected surface C (or surface A) to determine whether or not an erroneously recognized object exists on the approximate straight line that approximates the surface C on the overhead view map. . As shown in FIG. 22, the surface B exists on an approximate straight line that approximates the surface C, and thus is determined to be a part of the same side wall object (see FIG. 23).

  FIG. 24 is a flowchart illustrating a second operation example of the rejection unit 44. In addition, the same code | symbol is attached | subjected to the process substantially the same as the process mentioned above in each operation example of the rejection part 44. FIG. The second determination unit 455 determines the presence / absence of a remaining object that exists on an approximate line that approximates a rejected object on the overhead map (S300).

  For example, as illustrated in FIG. 25, the second determination unit 455 determines which of the plurality of remaining objects 702 exists on the approximate line 700. Here, the determination of whether or not the approximate straight line 700 exists may be determined to be included if the approximate straight line 700 partially overlaps the remaining object 702 (see FIG. 26A). It may be determined that the object 702 is included if it overlaps to a position that is a predetermined distance away from the center of the object 702 (see FIG. 26B). Under the latter condition, it is determined that the remaining object 702 does not exist on the approximate line 700 in the overlapping method shown in FIG.

  Then, the second rejection unit 456 determines that the object existing on the approximate line is an erroneously detected object and rejects it (S302).

  FIG. 27 is a diagram schematically illustrating a third operation example of the rejection unit 44. In FIG. 27, the second determination unit 455 determines that the surface C and the surface D positioned outside the approximate straight line described above are non-detection targets when viewed from the midpoint (center) of the two image sensors 22. For example, the second determination unit 455 determines that an object existing outside the roadway (that is, outside the side wall object) is a non-detection target when recognition of an object existing outside the roadway is unnecessary.

  As described above, the rejection unit 44 assumes that the side wall object is a boundary between the roadway and the sidewalk, and rejects the object existing in the area that is outside the side wall object, thereby erroneously rejecting the object existing on the roadway. And unnecessary objects can be rejected.

  Next, a fourth operation example of the rejection unit 44 will be described. Since the side wall object is a stationary object, it does not move between frames of the moving image. Therefore, the distance between the same objects detected in each of the two frames is unchanged. On the other hand, the distance between the moving vehicle and the non-moving sidewall object varies between frames. In the fourth operation example of the rejection unit 44, by utilizing this feature, among the rejection object that is a part of the side wall object rejected by the surface angle rejection method and the false detection object that is a part of the side wall object that is not rejected. The object whose distance between them does not change is regarded as a part of the side wall and is rejected. The distance here may be a distance on the image or an actual distance.

  FIG. 28 is a diagram schematically illustrating a fourth operation example of the rejection unit 44. FIG. 28 shows an example when the positional relationship between the rejected object 800 and the detection targets (non-rejected objects) 802 and 804 and the positional relationship between the rejected object 800 and the erroneously detected object 806 are viewed between adjacent frames. Has been.

  In the upper diagram of FIG. 28, when the image changes from frame n-1 to frame n, the reject object (side wall object) 800 moves, and the detection targets 802 and 804 do not move. The vehicles that are the detection targets 802 and 804 are traveling at the same speed as the own vehicle. When the image changes from frame n-1 to frame n, the distances from the rejected object 800 to the detection targets 802 and 804 change.

  On the other hand, in the lower diagram of FIG. 28, since the reject object (side wall object) 800 and the erroneously detected object (side wall object) 806 do not move, the distance between them does not change. In many cases, the same object is detected in successive frames. By storing the positional relationship between the reject object 800 and the false detection object 806 (can be a reject object) in the previous frame, the false detection object can be rejected.

  Note that if the CAN data of the vehicle 1 is used, it can be determined whether the vehicle is moving forward or backward, and therefore the appearance position of the pair in the previous frame can be estimated from the movement vector. Therefore, when referring to the information of the previous frame, it is possible to refer to only the pair existing at the estimated position as a processing target. This contributes to speeding up of processing. Further, the rejection unit 44 may limit the pair to be processed to an object that has been rejected and an object that exists on an approximate line that approximates the object. Thereby, the risk that the object which is not contained on a side wall object will be accidentally rejected can be avoided. Moreover, although the above assumes the case where the host vehicle is traveling forward, the rejection process can be performed in the same manner also in the reverse direction, that is, when traveling backward.

  FIG. 29 is a flowchart illustrating a fourth operation example of the rejection unit 44. The second determination unit 455 stores the newly detected object information indicating the newly detected object pair of the frame of interest in the storage unit 448 and acquires the newly detected object information indicating the newly detected object pair of the previous frame from the storage unit 448. (S400). In the fourth operation example of the rejection unit 44, at least two pieces of newly detected object information are used. Therefore, when the number of objects detected in the frame of interest is 1 or less, all subsequent processes are skipped.

  At this time, as information to be stored, a pair is generated so that at least one rejected object is included, and newly detected object information indicating the corresponding pair is stored. This is because if the newly detected object information of the detection target pair is stored in error, it will be rejected if the detection target pair appears at the position estimated from the movement vector in the next frame. This is because there is a risk that it will.

  The rejection unit 44 may always store only one frame of information when storing newly detected object information, or may store several frames of information. When the information of one frame is stored and no pair is detected in the next frame, the information referred to in the second frame is the information two frames before.

  The second rejection unit 456 performs rejection determination using the distance between the frame of interest and the newly detected object pair in the previous frame (S402).

  FIG. 30 is a diagram schematically illustrating a fifth operation example of the rejection unit 44. The rejection unit 44 may be configured to use not only the information of the frame of interest but also one or more previous frame information when performing the operations shown in the first operation example to the fourth operation example. .

  For example, assume that there is a frame of interest (n) as shown in FIG. In FIG. 30A, a remaining object 808 exists in the frame (n). In FIG. 30B, a reject object 810 and a remaining object 808 exist in the frame (n−1).

  In FIG. 30A, the remaining object 808 present in the frame (n) is rejected because the reject object 810 is not present in the same frame even though it is a part of the side wall object (false detection object). I can't. Therefore, the rejection unit 44 uses one or more previous frame information.

  In FIG. 30B, since the reject object 810 exists in the frame (n−1), it is possible to reject the remaining object 808 that is a false detection object in the frame (n) using this object. It is.

  In the example shown in FIG. 30, the previous frame information is stored, but two or more previous frame information may be stored. This is because there is a risk that the reject object 810 will not be detected accidentally due to a change in lighting conditions or the like in the previous frame. The risk that the reject object 810 is not detected can be avoided by using two or more previous frame information. In the present embodiment, the rejection unit 44 is configured to have a function of performing rejection determination, but is not limited thereto, and has a function of outputting to a subsequent processing unit (for example, the tracking unit 46). You may do it.

1 Vehicle 2 Stereo camera (imaging unit)
2a, 2b Camera unit 3 Vehicle ECU
4 Object Recognition Device 21 Lens 22 Image Sensor 23 Sensor Controller 30 Image Processing Device 32 Output Control Unit (Control Unit)
40 Parallax calculation unit 42 Clustering unit 44 Rejection unit 46 Tracking unit 304 CPU
308 ROM
310 RAM
442 Acquisition unit 444 First processing unit 446 Second processing unit 448 Storage unit 450 First calculation unit 451 Target surface determination unit 452 First determination unit 453 First rejection unit 454 Second calculation unit 455 Second determination unit 456 Second rejection Part 700 Approximate line 702 Remaining object

JP 2005-316991 A

Claims (15)

One reference image data indicating the reference image based on a frequency distribution of parallax generated between a reference image in which one or more objects are captured and a comparative image captured from an imaging position different from the reference image. A first determination unit that determines that the unit is installation image data indicating an installation installed so as to be continuous in a predetermined direction;
A first rejection unit that rejects the installation object image data from the reference image data;
A part of the reference image data in which the first rejection unit rejects the installation object image data is determined to be continuous object image data indicating an object continuous to the installation object based on the installation object image data. A second determination unit to perform,
A second rejection unit that rejects the continuous object image data from the reference image data in which the first rejection unit rejects the installation object image data;
An image processing apparatus comprising: A first calculating unit that calculates an angle of a surface included in an object imaged in the reference image with respect to straight lines passing through the imaging positions of the reference image and the comparative image based on the frequency distribution of the parallax;
The first determination unit includes:
2. The apparatus according to claim 1, wherein a part of the reference image data is determined to be the installation object image data based on an angle of a surface of an object having one surface calculated by the first calculation unit. The image processing apparatus described. The first determination unit includes:
It is determined whether the image data indicating the object having a plurality of surfaces calculated by the first calculation unit is the installation object image data based on a predetermined criterion for each of the plurality of surfaces indicated by the image data of the object. The image processing apparatus according to claim 2, wherein: The first determination unit includes:
Whether the image data is the installation object image data based on the ratio of the lengths of the plurality of surfaces indicated by the image data of the object with respect to the image data indicating the object including a plurality of surfaces calculated by the first calculation unit. The image processing apparatus according to claim 3, wherein the determination is performed. The first determination unit includes:
It is determined whether or not the image data is the installation object image data based on a ratio of the areas of the plurality of surfaces indicated by the image data of the object with respect to the image data indicating the object having a plurality of surfaces calculated by the first calculation unit. The image processing apparatus according to claim 3, wherein: The second determination unit includes
The part of the reference image data having a feature amount having a similarity in a predetermined range with respect to the feature amount of the installation object image data is determined as the continuous object image data. The image processing apparatus according to claim 5. The second determination unit includes
Determining that a part of the reference image data indicating an object on an approximate straight line obtained by approximating the installation object image data so as to indicate a direction in which the installation object is continuous is the continuous object image data. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The second determination unit includes
When a part of the reference image data indicating an object sandwiches the approximate straight line between the reference image and the intermediate point of the respective imaging positions of the comparison image, a part of the reference image data indicating the object is rejected Determine that it is target image data,
The second rejection unit is
The image processing apparatus according to claim 7, wherein the first rejection unit rejects the rejection target image data from the reference image data in which the installation object image data is rejected. The second determination unit includes
The installation determined by the first determination unit based on a frequency distribution of parallax generated between a frame of the reference image captured at a time before the reference image and the comparison image are captured and a frame of the comparison image A part of the reference image data indicating the object when the distance from the object indicated by a part of the reference image data is a predetermined value or less with respect to the installation object indicated by the image data even if time passes. The image processing apparatus according to claim 1, wherein the image processing device is determined to be the continuous image data. The second determination unit includes
The installation determined by the first determination unit based on a frequency distribution of parallax generated between a frame of the reference image captured at a time before the reference image and the comparison image are captured and a frame of the comparison image The image data is regarded as installation image data determined by the first determination unit based on a frequency distribution of parallax generated between the reference image and the comparison image. The image processing apparatus according to item. The image processing apparatus according to claim 1, further comprising an output unit that outputs a rejection result by the second rejection unit. A stereo camera that captures the reference image and the comparison image;
The image processing apparatus according to any one of claims 1 to 11,
An object recognition apparatus comprising: The image processing apparatus according to claim 1, wherein the second rejection unit includes a control unit that controls an apparatus based on the reference image data in which the continuous image data is rejected. A characteristic device control system. One reference image data indicating the reference image based on a frequency distribution of parallax generated between a reference image in which one or more objects are captured and a comparative image captured from an imaging position different from the reference image. A step of determining that the image data is an installation object image indicating an installation object installed in a predetermined direction.
Rejecting the installation object image data from the reference image data;
Determining a part of the reference image data in which the installation object image data is rejected as continuous object image data indicating an object continuous to the installation object based on the installation object image data;
A step of rejecting the continuous object image data from the reference image data in which the installation object image data is rejected;
An image processing method comprising: One reference image data indicating the reference image based on a frequency distribution of parallax generated between a reference image in which one or more objects are captured and a comparative image captured from an imaging position different from the reference image. Determining that the portion is installation image data indicating an installation installed so as to be continuous in a predetermined direction;
Rejecting the installation object image data from the reference image data;
Determining a part of the reference image data in which the installation object image data is rejected as continuous object image data indicating an object continuous to the installation object based on the installation object image data;
A step of rejecting the continuous object image data from the reference image data in which the installation object image data is rejected;
An image processing program for causing a computer to execute.

Images