JP2018092603A - Information processing device, imaging device, apparatus control system, movable body, information processing method, and information processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to detect objects accurately at a high speed on the basis of two-dimensional distribution information, such as a parallax image indicating positions in the transverse direction and positions in the depth direction of objects.SOLUTION: In an image processing device, a creation part 44 creates two-dimensional distribution information indicating positions in the transverse direction and positions in the depth direction of objects on the basis of information in which positions in the transverse direction, the positions in the vertical direction, and the positions in the depth direction of the objects are associated with each other. A separation part 46 detects the lengths of rows along the depth direction in the objects in the two-dimensional distribution information, and performs separation processing on the objects along the depth direction at the boundary between the rows with predetermined lengths.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an information processing device, an imaging device, a device control system, a moving body, an information processing method, and an information processing program.

  2. Description of the Related Art Today, automobile body structures and the like have been developed from the viewpoint of automobile safety, such as how to protect pedestrians and protect passengers in the event of a collision with a pedestrian or automobile. In recent years, with the development of information processing technology and image processing technology, technology for detecting humans, automobiles and the like at high speed is known. There is also known an automobile provided with a collision prevention system that uses these techniques to automatically brake before a collision to prevent the collision. In the case of this collision prevention system, a distance to a person or another vehicle is measured using a millimeter wave radar device, a laser radar device, a stereo camera device, or the like, and brake control is performed based on the measured result. Thus, the brake can be automatically applied according to the distance from the person or other vehicle.

  Patent Document 1 (Japanese Patent Laid-Open No. 6-266828) discloses a vehicle exterior monitoring device that detects a continuous three-dimensional object serving as a road boundary such as a guardrail, planting, and pylon train as a side wall. This vehicle exterior monitoring apparatus captures images of objects within an installation range outside a vehicle with a stereo optical system. The stereo image processing apparatus calculates a distance distribution over the entire captured image of the stereo optical system 10. The road / side wall detection device calculates the three-dimensional position of each part of the subject corresponding to this distance distribution. The road / side wall detection device detects the shape and the side wall of the road using the calculated three-dimensional position information.

  Here, the object detection process such as a person and an automobile is performed by measuring the distance to an object such as a person or another vehicle as described above. For this reason, it is preferable that each object is first separated and detected accurately.

  However, if an object is to be detected at high speed, the analysis process in the object detection process must be rough. For this reason, in the case of the conventional object detection technique including the technique of Patent Document 1, for example, when the wall and the vehicle are located close to each other, there is a problem that the wall and the vehicle are erroneously detected as the same object. In addition, when it appears that a number of preceding vehicles are vertically connected and running or stopped, there is a problem of erroneously detecting the vehicle row as a wall. In addition, although the accuracy of object recognition can be improved by performing a thorough analysis process, in this case, the analysis time becomes long, which hinders the speeding up of the object detection process. There is such a trade-off relationship between the detection processing time and the detection accuracy.

  The present invention has been made in view of the above-described problems, and provides an information processing device, an imaging device, a device control system, a moving body, an information processing method, and an information processing program capable of detecting an object at high speed and with high accuracy. The purpose is to do.

  In order to solve the above-described problems and achieve the object, the present invention is based on information in which a vertical position, a horizontal position, and a depth direction position of an object are associated with each other. A generating unit for generating two-dimensional distribution information indicating the length of each column along the depth direction in an object on the two-dimensional distribution information, and in the depth direction with a column of a predetermined length as a boundary And a separation unit that separates the object.

  According to the present invention, it is possible to detect an object at high speed and with high accuracy.

FIG. 1 is a diagram illustrating a position of a stereo camera provided in a vehicle which is a device control system according to an embodiment. FIG. 2 is a diagram illustrating a stereo camera included in the vehicle and a configuration around the stereo camera. FIG. 3 is a diagram illustrating a configuration example of an imaging apparatus including a stereo camera. FIG. 4 is a functional block diagram illustrating functions of the device control system according to the embodiment. FIG. 5 is a diagram for explaining separation processing in the column direction for an object. FIG. 6 is a flowchart illustrating a flow of separation processing in the column direction for objects in the device control system according to the first embodiment. FIG. 7 is a schematic diagram for explaining the separation processing in the column direction for the object in the device control system according to the first embodiment. FIG. 8 is a flowchart illustrating a flow of separation processing in the column direction for objects in the device control system according to the second embodiment. FIG. 9 is a schematic diagram for explaining the separation processing in the column direction for the object in the device control system according to the second embodiment. FIG. 10 is a diagram for explaining row-direction separation processing for objects in the device control system according to the third embodiment. FIG. 11 is a schematic diagram for explaining row-direction separation processing for objects in the device control system according to the third embodiment. FIG. 12 is a flowchart illustrating a flow of separation processing in the row direction for an object in the device control system according to the third embodiment. FIG. 13 is a diagram for explaining row-direction separation processing for an object in the device control system according to the fourth embodiment. FIG. 14 is a flowchart illustrating a flow of separation processing in the row direction for an object in the device control system according to the fourth embodiment. FIG. 15 is a diagram for explaining row-direction separation processing for an object in the device control system according to the fourth embodiment. FIG. 16 is a schematic diagram for explaining row-wise separation processing for an object in the device control system according to the fourth embodiment. FIG. 17 is a diagram illustrating objects separated in the row direction in the device control system according to the fourth embodiment. FIG. 18 is a flowchart illustrating a flow of separation processing in the row direction for an object in the device control system according to the fifth embodiment. FIG. 19 is a flowchart illustrating a flow of separation processing in the row direction for an object in the device control system according to the sixth embodiment. FIG. 20 is a functional block diagram illustrating functions of the device control system according to the embodiment.

  Hereinafter, an apparatus control system according to an embodiment will be described with reference to the drawings.

(First embodiment)
(System configuration)
As shown in FIG. 1, the device control system of the first embodiment includes a stereo camera 2 that is provided on the windshield of the vehicle 1 and captures a predetermined imaging range on the front side in the traveling direction. As will be described later with reference to FIG. 3, the stereo camera 2 includes two image sensors 22 and is an imaging unit that captures two images of the left visual field and the right visual field.

  FIG. 2 is a diagram illustrating a configuration example of the stereo camera 2 included in the vehicle 1 which is an example of the moving body and the vicinity thereof. The stereo camera 2 outputs, for example, two captured images to a vehicle ECU (Engine Control Unit) 3. The vehicle ECU 3 is installed in the vehicle 1 and controls the vehicle 1 such as engine control, braking control, travel lane keep assist, steering assist, and the like. Hereinafter, a vehicle which is an example of a moving body will be described, but the device control system of the present embodiment can be applied to a ship, an aircraft, a robot, and the like.

(Configuration of imaging device)
FIG. 3 is a diagram illustrating a configuration example of the imaging device 4 including the stereo camera 2. The imaging device 4 includes, for example, a stereo camera 2 and an image processing device 30. In the stereo camera 2, a camera unit 2 a for the left eye and a camera unit 2 b for the right eye are assembled in parallel (horizontal) to shoot a moving image (or still image) in the imaging target area.

  Each of the camera units 2a and 2b includes a lens 21, an image sensor 22, and a sensor controller 23. The image sensor 22 is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor. The sensor controller 23 performs, for example, exposure control of the image sensor 22, image readout control, communication with an external circuit, image data transmission control, and the like.

  The image processing device 30 is provided, for example, in the vehicle ECU 3 shown in FIG. The image processing apparatus 30 includes, for example, a data bus line 300, a serial bus line 302, a CPU (Central Processing Unit) 304, an FPGA (Field-Programmable Gate Array) 306, a ROM (Read Only Memory) 308, and a RAM (Random Access Memory) 310. Serial IF (Interface) 312 and data IF (Interface) 314.

  The stereo camera 2 described above is connected to the image processing apparatus 30 via the data bus line 300 and the serial bus line 302. The CPU 304 controls the overall operation of the image processing apparatus 30 and executes image processing and image recognition processing. Luminance image data of captured images captured by the image sensors 22 of the camera units 2 a and 2 b are written into the RAM 310 of the image processing device 30 via the data bus line 300. Sensor exposure value change control data, image readout parameter change control data, various setting data, and the like from the CPU 304 or FPGA 306 are transmitted and received via the serial bus line 302.

  The FPGA 306 generates real-time parallax images by performing parallax calculation based on, for example, gamma correction, distortion correction (parallelization of left and right images), and block matching, which are processes that require real-time processing for image data stored in the RAM 310. , Write again to the RAM 310. The CPU 304 performs control of each sensor controller 23 of the stereo camera 2 and overall control of the image processing apparatus 30. The CPU 304 acquires, for example, CAN (Controller Area Network) information of the host vehicle as parameters (vehicle speed, acceleration, steering angle, yaw rate, etc.) via the data IF 314. In the following description, the term “image”, such as “parallax image”, does not necessarily require display such as a display, but includes a mere collection of information.

  The detection data (recognition data) of the detection target (recognition target) is supplied to the vehicle ECU 3 via the serial IF 312 and used in, for example, an automatic brake system or a travel assist system provided as a control function of the vehicle ECU 3. The automatic brake system performs brake control of the vehicle 1. The travel assist system performs travel lane keep assist, steering assist, and the like of the vehicle 1.

(Function of image processing device)
As shown in FIG. 4, the image processing apparatus 30 includes a parallelizing unit 42, a parallax image generating unit 43, a U map generating unit 44 (an example of a generating unit), a wall detecting unit 45, and a separating unit 46 (an example of a separating unit). Has each function. The parallelizing unit 42 and the parallax image generating unit 43 are realized by hardware by, for example, programmable logic blocks of the FPGA 306. In this example, the parallelizing unit 42 and the parallax image generating unit 43 are realized by hardware, but may be realized by software by a three-dimensional object recognition program or the like described below.

  The functions of the U map generation unit 44, the wall detection unit 45, and the separation unit 46 are realized by the CPU 304 executing a three-dimensional object recognition program stored in a storage unit such as the RAM 310 or the ROM 308, for example.

  In this example, the description will be made on the assumption that the U map generation unit 44 to the separation unit 46 are realized by software. However, part or all of the U map generation unit 44 to the separation unit 46 may be integrated with an IC (Integrated Circuit). It may be realized by hardware such as.

  The three-dimensional object recognition program may be provided as a file in an installable format or an executable format and recorded on a recording medium readable by a computer device such as a CD-ROM or a flexible disk (FD). Further, the program may be provided by being recorded on a recording medium readable by a computer device such as a CD-R, a DVD (Digital Versatile Disk), a Blu-ray Disc (registered trademark), or a semiconductor memory. The three-dimensional object recognition program may be provided by being installed via a network such as the Internet. The three-dimensional object recognition program may be provided by being incorporated in advance in a ROM or the like in the device.

(Operation of image processing device)
The stereo camera 41 generates luminance image data by color luminance conversion processing in each of the camera units 2 a and 2 b and supplies the luminance image data to the parallelizing unit 42 of the FPGA 306. The parallelizing unit 42 performs parallelization processing on the luminance image data supplied from the stereo camera 41, thereby generating a parallel luminance image in which the distortion of the optical system of each camera unit 2a, 2b is corrected.

  The parallax image generation unit 43 generates a parallax image having a parallax value for each pixel, which is an example of a distance image having distance information for each pixel. That is, the parallax image is an example of information in which the vertical position, the horizontal position, and the depth direction position of the object are associated with each other. Based on the parallax image generated by the parallax image generation unit 43, the U map generation unit 44 is a U map that is a two-dimensional histogram in which the horizontal axis represents the actual distance, the vertical axis represents the parallax value d, and the frequency is represented on the z axis. Is generated. That is, the U map is a bird's-eye view map (bird's-eye view image, bird's-eye view image) generated based on the parallax image, and is two-dimensional distribution information in which the horizontal position and the depth direction position of the object are associated with each other. For convenience of explanation, the term “map” is used, but instead of actually generating a map, information having such characteristics is generated and stored as two-dimensional distribution information such as RAM 310 or ROM 308. It should be understood that it is stored in the department.

  The parallax image generation unit 43 calculates the height of the point (x, y, d) of the parallax image whose height from the road surface is within a predetermined height range (for example, 20 cm to 3 m) based on the value of (d). It may be calculated to determine whether or not to vote. The position of the road surface and the height from the road surface can be calculated by a known method using a parallax image. The lateral position of the voting position is calculated using the value of (x, d) of the point (x, y, d) of the parallax image. The distance between the center of the parallax image and the x pixel is converted into an actual distance, and the lateral position of the vote is determined. For example, it is assumed that a wall w1 is provided along the right end of the road illustrated in FIG. 5A, and a parallax image corresponding to a captured image in which the vehicle c1 is traveling along the wall w1 is supplied. In this case, as shown in FIG. 5 (b), the U map generation unit 44 includes a straight bar-like wall w1 extending in the depth direction from the front, and a rectangular vehicle c1 in contact with the wall w1. Generate a U map with the back voted.

  For example, the wall detection unit 45 detects the wall w1 voted on the U map shown in FIG. However, in the case of the example shown in FIG. 5B, since voting is performed in a state where the wall w1 and the vehicle c1 are in contact with each other, there is a possibility that the wall w1 and the vehicle c1 are erroneously detected as a single wall. Therefore, the separation unit 46 performs separation processing in the column direction to separate the objects detected as a single wall by the wall detection unit 45 into the wall w1 and the vehicle c1.

(Column direction separation processing)
FIG. 6 shows a flow of processing for separating an object (wall detection result) detected as a lump of walls into a wall w1 and a vehicle c1. The separation unit 46 starts the process via step S1, and repeatedly executes the processes of steps S2 to S7 until the processes for all the wall detection results are completed.

  Step S2: The height (length in the depth direction) is detected for each column of the image (object) that is the wall detection result.

  Step S3: The column height (length in the depth direction) detected in step S2 is the height of the detected column (depth direction) in the object whose height (length in the depth direction) is currently detected. It is determined whether or not the maximum height (the length in the depth direction) is included. If step S3 is No, the process proceeds to step S5.

  Step S4: When it is determined that the detected column height (length in the depth direction) is the maximum (step S3: Yes), the value of the maximum column height (depth direction length) for the current object, Then, the position information of the column having the value of the maximum height (length in the depth direction) is updated.

  Step S5: It is determined whether or not the detection (search) of the height (length in the depth direction) of all the columns for the current object has been completed. If the search for all the columns of the current object has not been completed (step S5: No), the process returns to step S2, and the search is performed for the column in the current object that has not been searched.

  Step S6: When the search for all the columns of the current object is completed, the position information of the column of the maximum height (length in the depth direction) of the current object is used as the wall position information indicating the wall position. It memorize | stores in memory | storage parts, such as RAM310 or ROM308.

  Step S7: When the search for all objects is completed, the process of the flowchart of FIG. 6 is terminated (step S7: Yes). If the search for all objects is not completed (step S7: No), step S1 is performed. The process is returned to the above, and the above-described processes are repeatedly executed on other objects.

  FIG. 7 is a schematic diagram showing such separation processing of objects in the column direction. That is, as shown in FIG. 7A, the separation unit 46 detects the height (length in the depth direction) of each column of objects (steps S2 to S5). Next, the separation unit 46 stores the position information of the longest column among the columns constituting the object in the storage unit as wall position information.

  The separating unit 46 separates the wall w1 and the vehicle c1 by separating a group of objects with the column indicated by the wall position information as a boundary. As a result, even if a U map having a low resolution is used, the objects can be accurately separated by a simple separation process in which the objects are separated from each other at the maximum length of the columns constituting the object. Further, since the objects can be separated by simple arithmetic processing, the object separation processing can be performed at high speed. That is, the object separation process can be performed at high speed with a simple information processing operation.

(Second Embodiment)
Next, a device control system according to the second embodiment will be described. When the object is separated using the maximum length column as a boundary as in the first embodiment, an object that is difficult to identify as an object, such as the partial object on the left side after separation shown in FIG. May remain. That is, in the example of FIG. 7C, the left partial object after separation has a shape that is difficult to detect as a vehicle.

  In the device control system according to the second embodiment, in order to prevent such inconvenience, the separation unit 46 performs processing according to the flowchart of FIG. 8 so that the objects can be separated with higher accuracy. In the flowchart of FIG. 8, steps S1 to S5 and S7 are the same processes as steps S1 to S5 and S7 of the flowchart of FIG. That is, the difference between the flowchart of FIG. 6 and the flowchart of FIG. 8 is that steps S11 and S12 are interposed between steps S2 and S3, and steps S13 are interposed between steps S5 and S7. That is. The first embodiment described above is different from the second embodiment described below only in this respect. For this reason, only the difference between the two will be described below, and redundant description will be omitted.

  In the case of the second embodiment, the separation unit 46 detects the height (length in the depth direction) of each column of the wall detection result (object) as shown in FIG. 9A in step S2. Then, the process proceeds to step S11, and it is determined whether or not the height (length in the depth direction) of the currently searched column is a height (length in the depth direction) equal to or greater than a threshold value. As an example, the threshold value is set to a value (slightly longer value) such as “10 m” or the like that exceeds the length of a vehicle that can be considered in common sense.

  When the separation unit 46 detects a column having a height (depth in the depth direction) that is equal to or higher than the threshold (step S11: Yes), the separation unit 46 proceeds to step S12, and proceeds to step S12. The position information is stored in the storage unit such as the RAM 310 as wall position information. In the case of the example shown in FIG. 9B, the shaded columns indicate the columns having a height (length in the depth direction) equal to or greater than the threshold value. The separation unit stores the position information of the shaded row in the storage unit.

  When the separation unit 46 finishes detecting a column having a height (length in the depth direction) that is equal to or greater than the threshold among the columns that form the object in this manner, in step S3 and step S4, the separation unit 46 has a height that is equal to or greater than the threshold ( Among the columns in the depth direction), the column having the maximum height (the length in the depth direction) is detected. In step S13, the separation unit 46 is adjacent to the column having the maximum height (length in the depth direction) and the column having the maximum height (length in the depth direction), and is equal to or greater than the above-described threshold value. Are stored in the storage unit as wall position information, respectively.

  In the case of the example of FIG. 9C, the column CL6 is a column having the maximum height (length in the depth direction). Of the columns adjacent to the maximum column CL6, a column having a height (length in the depth direction) equal to or greater than the threshold is the column CL5. The column CL7 is a column that is less than the threshold value. In this case, the separation unit 46 stores the positions of the rows CL5 and CL6 in the storage unit as wall position information. As a result, when the objects are separated with the column CL5 and the column CL6 as the boundary, as shown in the column CL1 to the column CL4 in FIG. 9C, an object that can be specified as a vehicle (object) can be left. In addition to making it possible to separate the objects with high accuracy, the same effects as in the first embodiment described above can be obtained.

(Third embodiment)
(Row direction separation processing)
Next, a device control system according to the third embodiment will be described. The first and second embodiments described above are examples in which objects are separated along the row direction. In contrast, the device control system according to the third embodiment is an example in which objects are separated along the column direction. The above-described embodiments are different from the third embodiment described below only in this respect. For this reason, only the difference will be described below, and a duplicate description will be omitted.

  As an example, as shown in FIG. 10 (a), when the vehicle c1 to the vehicle c3 preceding the left lane are connected and appear to be connected, on the U map, as shown in FIG. 10 (b). The bar-like (straight-line) object corresponding to the side surface of each vehicle c1 to c3 in the connected state appears. Further, a rectangular object corresponding to the rear surface of the rearmost vehicle c1 appears in a form connected to the right front portion of the rod-shaped object. That is, when the vehicles c1 to c3 preceding the left lane appear to be connected, an “L-shaped” object appears on the U map as shown in FIG.

  When such a U map is obtained, the separation unit 46 of the device control system according to the third embodiment causes the line width of each line of the object that is the wall detection result to have a difference that exceeds a predetermined value. The object is separated along the row direction with the border as the boundary. Specifically, as illustrated in FIGS. 11A to 11C, the separation unit 46 sequentially detects the width of each row that configures the object from the front of the object. The separating unit 46 compares the width of each row with the first threshold value every time it is detected. As an example, for example, “1.5 m” is set as the first threshold value, which is a value that is slightly narrower than the width of the rear surface of one vehicle. When the number of row widths that are equal to or greater than the first threshold every time the width of each row is detected is equal to one vehicle, the separation unit 46 is finally equal to or greater than the first threshold. The object is separated on the line where is detected.

  As a result, as shown in FIG. 11 (d), the “front vehicle c 1” and the “vehicle c 2 and vehicle c 3 corresponding to the preceding vehicle with respect to the vehicle c 1” can be separated. Further, the control of the vehicle provided with the device control system is often performed with reference to the traveling state of the vehicle c1 in front. For this reason, the separation unit 46 discards the objects corresponding to the vehicle c2 and the vehicle c3 other than the vehicle c1 in the foreground without being used for vehicle control as a wall detection result. In addition, you may use the object corresponding to the vehicle c2 and the vehicle c3 as a wall detection result.

  The flowchart of FIG. 12 shows the flow of the row direction separation processing of the separation unit 46 in the third embodiment. The separation unit 46 starts the process via step S21. In step S33, each of the steps S22 to S33 is performed until it is determined that the row direction separation process has been completed for all the wall detection results (objects). Repeat the process for each object.

  In step S22, as described with reference to FIGS. 11A to 11C, the separation unit 46 detects the width of each row. In step S23, it is determined whether or not the separation unit 46 is equal to or greater than a first threshold that is set slightly narrower than the width of the rear surface of one vehicle. In step S24, the number of row widths that are equal to or greater than the first threshold is counted. In step S25, it is determined whether or not the number of row widths that are equal to or greater than the first threshold is equal to or greater than a predetermined number (a predetermined count number or more) corresponding to the width of the rear portion of one vehicle, for example. In step S26, since the number of rows having a width equal to or greater than the first threshold is equal to or greater than the predetermined count, the separation unit 46 stores information indicating that the object corresponds to a vehicle in the storage unit described above. (Set the flag to ON).

  In step S27, the separation unit 46 determines whether or not the flag is on. If the flag is not on (step S27: No), the separation unit 46 determines whether or not the processing for all the rows of the object currently being searched has been completed in step S32. If it is determined that the processing for all rows has been completed, the separation unit 46 proceeds to step S33, and determines whether or not the row direction separation processing has been completed for all objects (wall detection results). When it is determined that the row direction separation process for all objects (wall detection results) has been completed (step S33: Yes), the separation unit 46 ends the process of the flowchart of FIG. 12 and performs the row direction separation process for all objects. If it is determined that the process has not ended (step S33: No), the process returns to step S22 via step S21, and detection of the width of the next object row is started.

  In step S28, since the flag is on, the separation unit 46 determines whether or not the row width is less than the second threshold value. The second threshold is set to be slightly larger than the width of the rear surface of one vehicle, for example, greater than the width of the object formed corresponding to the vehicle c2 and the vehicle c3 traveling in front of the foremost vehicle c1. The width is set to a value narrower than the first threshold value. In step S29, the separation unit 46 counts the number of rows having a width less than the second threshold. In step S30, when the count value is equal to or greater than the count value counted when a predetermined number of vehicles are continuously connected (step S30: Yes), the object is displayed at the current row position. The position information of the object stored in the storage unit is corrected so as to be separated.

  The separation unit 46 excludes the wall position information of the objects corresponding to the vehicle c2 and the vehicle c3 after separation from the storage unit. Thereby, driving assistance such as brake control and travel position control of the vehicle 1 is performed based on the movement of the vehicle c1. However, since the vehicle c2 and the vehicle c3 are located farther (the depth side) than the vehicle c1, there is no influence as much as the difference.

  As shown in FIG. 11 (d), the device control system according to the third embodiment includes a “frontmost vehicle c1” and a “vehicle c2 and a vehicle c3 traveling in front of the vehicle c1” along the row direction. Can be separated. For this reason, accurate object recognition processing can be performed based on the separated objects, and in addition to improving the accuracy of control in the device control system, the same effects as those of the above-described embodiments can be obtained. Can be obtained.

(Fourth embodiment)
(Another embodiment of row direction separation processing)
Next, a device control system according to the fourth embodiment will be described. As shown in FIG. 13A, when the wall w1 is provided along the left end portion of the road, and the vehicle c1 is traveling at a position on the back side of the road (a position on the far side), FIG. A U map in which the object shown in b) appears is obtained. That is, as shown in FIG. 13B, an object in a shape in which the object of the rod-shaped wall w1 and the rectangular object corresponding to the back surface of the vehicle c1 are connected on the back side appears on the U map. . The device control system according to the fourth embodiment obtains an object on the wall w1 and an object on the vehicle c1 by separating such objects along the row direction. The third embodiment described above is different from the fourth embodiment described below only in this respect. For this reason, only the difference will be described below, and a duplicate description will be omitted.

  The flowchart of FIG. 14 shows the flow of the row direction separation process of the separation unit 46 in the fourth embodiment. The separation unit 46 starts processing through step S41, and in step S48, each of the steps S42 to S48 is performed until it is determined that the row direction separation processing has been completed for all wall detection results (objects). Repeat the process for each object.

  First, in step S42, the separation unit 46 detects the widths of all lines of the wall detection result (object). Specifically, in the case of the captured image shown in FIG. 13A, a U map in which an object as shown in FIG. 15 appears is obtained. The separation unit 46 detects the widths of the first row L1 to the fourteenth row L14 of the object shown in FIG.

  Next, in step S43, the separation unit 46 takes a line unit section in the rectangle of the wall detection result, and slides the section upward from the lower end of the rectangle. That is, as shown in FIG. 16A, the separation unit 46 is a one-dimensional form in which the width value of the 14th row L14 is stacked in order from the width value of the first row L1 of the object. A histogram is generated, and for example, a first interval including five values and a second interval including five values that are continuous to the first interval are set. In the case of the example of FIG. 16A, the section from the first line L1 to the fifth line L5 is the first section, and the sixth line L6 to the tenth line L10 continuous to the first section. This section becomes the second section. As will be described below, the separation unit 46 averages the width values of the first and second sections while sliding the first and second sections on the histogram at the same time for each row width value. Calculate the value.

  That is, in step S44, the separation unit 46 calculates the average value (lower average value) of the width values of the first section from the first row L1 to the fifth row L5 shown in FIG. The average value (upper average value) of the width values of the second section of the sixth line L6 to the tenth line L10 is calculated. In step S45, the separation unit 46 determines whether “(average value of first section × threshold)> average value of second section” is satisfied. For example, a value “2” is used as the threshold value.

  In the case of the example of FIG. 16A, the separation unit 46 is “2” which is an average value of 2, 3, 2, 3, 3 which is a value of each width from the first row L1 to the fifth row L5. .6 ”and“ 3.0 ”, which is the average value of the respective widths from the sixth line L6 to the tenth line L10, 3, 2, 2, 2, 6 is calculated. Then, the separation unit 46 multiplies “2.6” that is the average value in the first section by “2” that is the threshold value, and this multiplication value, that is, the weighted average value “5.2”. Is compared with “3.0”, which is the average value of the second section.

  In the case of the comparison process in this example, the lower n multiplication value “5.2” is larger than the average value “3.0” of the upper second section (step S45: No). The separation unit 46 proceeds to step S47, and determines whether or not the first and second sections have been slid to a position corresponding to the uppermost end of the histogram. When it is determined that the first and second sections have been slid to the position corresponding to the uppermost end of the histogram (step S47: Yes), the separation unit 46 proceeds to step S48, and all the wall detection results It is determined whether or not (object) has been subjected to the row direction separation processing described below. When it is determined that the row direction separation process described below has been performed on all wall detection results (objects) (step S48: Yes), the row direction separation process shown in the flowchart of FIG. 14 is terminated. .

  On the other hand, if it is determined that the row direction separation processing described below has not been performed on all wall detection results (objects) (step S48: No), the separation unit 46 passes through step S41. The processing is returned to step S42, and the row direction separation processing described below is performed on the next wall detection result (object).

  That is, as shown in FIG. 16A to FIG. 16E, the separation unit 46 slides the first section and the second section on the histogram by one width value at a time. An average value weighted with the threshold value of the section and an unweighted average value of the second section are calculated and compared. Accordingly, in the example of FIGS. 16A to 16E, “3 <(2.6 × 2) (FIG. 16A) according to the slide positions of the first section and the second section. )) ”→“ 3.4 <(2.8 × 2) (FIG. 16B) ”→“ 4.0 <(2.6 × 2) (FIG. 16C) ”→“ 4.6 <(2.6 × 2) (FIG. 16 (d)) ”→“ 5.2> (2.4 × 2) (FIG. 16 (e)) ”can be obtained.

  In step S45, the separation unit 46, when the average value not weighted in the second section becomes larger than the average value weighted by the threshold value in the first section, as shown in FIG. (Step S45: Yes) In step S46, as shown in FIG. 17, the objects are separated along the row direction at the boundary between the first section and the second section. That is, in the example of FIG. 16E, the position of the ninth value from the bottom of the histogram is the boundary between the first section and the second section. Therefore, as shown in FIG. 17, the separation unit 46 separates the object along the row direction at the position of the ninth row L9. Thus, since the separation in the row direction is performed using an index for separating an object having a large width with respect to the wall, such as a vehicle, the object corresponding to the vehicle c1 located on the back side (the tenth row L10 to the 14th line L14) and the object corresponding to the wall w1 (the 1st line L1 to the 9th line L9) can be separated.

  Since the device control system according to the fourth embodiment can accurately separate the coupled objects along the row direction, accurate object recognition processing can be performed based on the separated objects. The same effects as those of the above-described embodiments can be obtained, for example, the accuracy of control in the device control system can be improved.

(Fifth embodiment)
(First modification of row direction separation processing)
Next, a device control system according to the fifth embodiment will be described. In the case of the above-described fourth embodiment, the average value calculation process, the comparison process, and the like are performed after first calculating the width of each row. On the other hand, the fifth embodiment is an example in which the calculation of the average value of each section, the separation process, and the like are performed at the timing when the setting of each section described above becomes possible. The fourth embodiment described above is different from the fifth embodiment described below only in this respect. For this reason, only the difference will be described below, and a duplicate description will be omitted.

  The flowchart of FIG. 18 shows the flow of the row direction separation process of the separation unit 46 in the fifth embodiment. The separation unit 46 starts processing through step S51, and in step S59, each of the steps S52 to S59 is performed until it is determined that the row direction separation processing has been completed for all wall detection results (objects). Repeat the process for each object.

  First, in step S52, the separation unit 46 performs scanning while sequentially moving from the lower row to the upper row of the wall detection result (object), and in step S53, the width of the current row is calculated. That is, the row width is calculated sequentially for each moving operation. In step S54, the separation unit 46 determines whether or not the width calculation has been completed a predetermined number of times or more. That is, in step S54, the separation unit 46 determines whether or not calculation of, for example, 10 width values that can set the first section and the second section is completed. In addition, since the first section and the second section are examples including five width values as described above, it is determined whether or not the calculation of the width values for ten sections has been completed. However, if each section includes three width values, it is determined whether or not the calculation of the widths for six sections has been completed. More specifically, in step S54, the separation unit 46 determines whether or not the calculation of the width value has been completed for the number of times that the first section and the second section can be set.

  When the width calculation is less than the predetermined number of times (step S54: No), the separation unit 46 proceeds to step S58, and determines whether or not scanning of all rows of the current object has been completed. If the scanning of all the rows of the current object has been completed (step S58: Yes), the separation unit 46 proceeds to step S59 to determine whether or not the row direction separation processing for all the objects has been completed. To do. When the row direction separation process for all objects has been completed (step S59: Yes), the separation unit 46 ends the process of the flowchart of FIG. If the row direction separation process for all the objects has not been completed (step S59: No), the separation unit 46 returns the process to step S52 via step S51, and will be described below for other objects. Execute row direction separation processing.

  On the other hand, when the scanning of all the rows of the object has not been completed (step S58: No), the separation unit 46 returns the process to step S52, and executes the scanning of the other rows of the object.

  Next, when it is determined in step S54 that the measurement of the predetermined number of times or more has been completed (step S54: Yes), the separating unit 46 determines the above-described histogram (see FIGS. 16A to 16E). The first section and the second section described above are set in the downward direction from the current row, and the average value of each section is calculated in step S56. In step S57, the separation unit 46 determines whether or not “average value of second section> (average value of first section × 2)”. When “average value of the second section> (average value of the first section × 2)” (step S57: Yes), the separation unit 46 determines whether the first section and the second section are in step S60. The objects are separated at the boundary (see FIGS. 16E and 17).

  On the other hand, when “average value of second section> (average value of first section × 2)” is not satisfied (step S57: No), the separation unit 46 proceeds to step S58 and performs the above-described process. As described above, it is determined whether or not scanning of all the rows in the object is completed. If not, the process returns to step S52. In the fifth embodiment, since the search for the line, the measurement of the width, and the check of the change can be simultaneously performed from the lower side of the wall detection result (object), the object separation process can be quickly performed. In addition to contributing to high-speed device control, the same effects as those of the above-described embodiments can be obtained.

(Sixth embodiment)
(Second modification of row direction separation processing)
Next, a device control system according to the sixth embodiment will be described. In the sixth embodiment, the separating unit 46 searches the U map for objects that are connected in order from the front to the depth direction, so that a wall-like object continuous in the depth direction is obtained. It is an example which isolate | separates other objects, such as a detected wall and a vehicle, while detecting. It should be noted that only the above points are different between the above-described embodiments and the sixth embodiment. For this reason, only the difference will be described below, and a duplicate description will be omitted.

  The flowchart of FIG. 19 shows the flow of the row direction separation processing of the separation unit 46 in the sixth embodiment. The separation unit 46 starts processing through step S51, and in step S59, each of the steps S52 to S59 is performed until it is determined that the row direction separation processing has been completed for all wall detection results (objects). Repeat the process for each object.

  First, in step S52, the separation unit 46 searches the first column of the U map from below (nearest distance area) (step S61). Next, the separation unit 46 determines whether the three pixels in the row above the currently searched pixel (the pixels in the row above the currently searched pixel and the pixels adjacent to the left and right of this row) are displayed. It is determined whether or not there is a pixel having a frequency value (step S62). If there is no pixel having a frequency value among the above three pixels (step S62: No), the separation unit 46 proceeds to step S75, and whether or not the search for the last data in the last column is completed. Is determined.

  When the separation unit 46 determines that the search for the last data in the last column is completed (step S75: Yes), the process of the flowchart of FIG. On the other hand, if it is determined that the search for the last data in the last column has not been completed, the separation unit 46 returns the process to step S61 to search for the next column. That is, when the separation unit 46 does not detect a pixel having a frequency value in the three pixels in the row above the pixel currently being searched, for example, the labeling number (label 1) of the current search is “1”. The first search with () is terminated, and the second search with a labeling number of “2” is entered.

  Next, when a pixel having a frequency value exists in the above three pixels (step S62: Yes), the separation unit 46 updates the current search position to the position of the pixel having a frequency value (step S63). ). Then, the separation unit 46 determines whether or not a labeling number (labeling number) has already been assigned to the pixel at the current search position (step S64).

  If it is determined that a labeling number has already been assigned, the separation unit 46 terminates the process of the flowchart of FIG. 19 via the above-described step S75, or performs a process to search for the next column. Return to S61.

  On the other hand, when it is determined that the labeling number is not attached (step S64: No), the separation unit 46 increments the labeling number by one (step S65), and the pixel at the current search position is A labeling process for assigning an incremented labeling number is performed (step S66).

  Next, in step S67, the separation unit 46 calculates the width of the current row (see FIGS. 11A to 11C).

  Next, the separation unit 46 selects a pixel at a position above the pixel at the current search position, a pixel at the position above the pixel at the current search position (pixels to which a labeling number is assigned) at the current search position. It is determined whether or not there is a pixel having a frequency value among pixels adjacent to the left and right of (step S68). That is, in step S68, the separation unit 46 determines whether or not there is a pixel having a frequency value among the above-described three pixels.

  When it is determined that there is no pixel having the frequency value among the above three pixels (step S68: No), the separation unit 46 ends the process of the flowchart of FIG. 19 through the above step S75. Alternatively, the process returns to step S61 to search for the next column.

  On the other hand, if it is determined that there is a pixel having a frequency value among the above three pixels (step S68: Yes), the separation unit 46 updates the position of the pixel with the highest frequency as the current search position. (Step S69).

  Next, the separation unit 46 determines whether or not a different ID has already been assigned to the updated pixel at the current search position (step S70). If it is determined that a different ID has already been assigned (step S70: Yes), the separation unit 46 ends the process of the flowchart of FIG. 19 or searches for the next column via step S75 described above. To do so, the process returns to step S61.

  On the other hand, when it is determined that a different ID is not already assigned to the updated pixel at the current search position (step S70: No), the separation unit 46 labels the pixel at the current search position with a labeling number. Is given (step S71).

  Next, in step S72, the separation unit 46 calculates the size of the current row width (see FIGS. 11A to 11C). In step S73, the separation unit 46 determines whether or not to perform any of the separation processes described in the first to fifth embodiments using the width information calculated so far. When performing the separation process (step S73: Yes), the separation unit 46 advances the process to step S76, and executes the separation process along any one of the row directions described in the third to fifth embodiments. To do. When any separation process is executed, the separation unit 46 terminates the process of the flowchart of FIG. 19 via step S75, or returns the process to step S61 to search for the next column.

  On the other hand, when not performing separation processing (Step S73: No), separation part 46 advances processing to Step S74. In step S74, the separation unit 46 determines whether or not the search for the last data (pixel) in the currently searched row has been completed. If it is determined that the search for the last pixel in the current row has not been completed (step S74: No), the separation unit 46 proceeds to step S68 to search for the next pixel in the current row. return. On the other hand, if it is determined that the search for the last pixel in the current row has ended (step S74: Yes), the separation unit 46 ends the process of the flowchart of FIG. 19 through step S75 described above. Alternatively, the process returns to step S61 to search for the next column.

  In the case of such a sixth embodiment, wall detection and separation processing of an object such as a wall and a vehicle can be performed at the same time, which contributes to high-speed device control. Similar effects can be obtained.

(Seventh embodiment)
Next, a device control system according to the seventh embodiment will be described. In each embodiment described above, the U map generation unit 44 generates a U map based on the parallax image from the parallax image generation unit. On the other hand, the device control system according to the seventh embodiment generates a U map reflecting the height of the road surface, and separates the objects on the U map as described in the above embodiments. This is an example that enables separation of objects with higher accuracy. It should be noted that only the above points are different between the above-described embodiments and the seventh embodiment. For this reason, only the difference will be described below, and a duplicate description will be omitted.

  In the case of the device control system according to the seventh embodiment, the CPU 304 executes the three-dimensional object recognition program, so that the parallax interpolation unit as illustrated in FIG. 20 is performed together with the parallelization unit 42 to the separation unit 46 illustrated in FIG. 51, the V map generating unit 52, the road surface shape detecting unit 53, and the road surface height calculating unit 54 are realized.

  Although the parallax interpolation unit 51 to the road surface height calculation unit 54 are realized by software, part or all of them may be realized by hardware such as an IC (Integrated Circuit). It is as follows.

  The parallax interpolation unit 51 performs parallax image interpolation processing. The V map generation unit 52 generates a V map based on each pixel value in which the voting area is limited. The road surface shape detection unit 53 and the road surface height calculation unit 54 calculate the height of the road surface before the generation of the V map corresponding to the parallax image is completed.

  Specifically, for example, when a parallax image in which a power pole is present on the left side of a vehicle traveling on a flat road surface extending in the center of the screen is obtained, the V map generation unit 52 performs parallax among the pixels of the captured image. The frequency (frequency) counted up by one is assigned to the pixel having the value D and the y coordinate value. Then, the V map generation unit 52 votes as a right-downward straight line by voting each pixel on a two-dimensional histogram with the X axis as the parallax value D, the Y axis as the y coordinate value, and the Z axis as the frequency. On the road surface, a V map in which vehicles and utility poles are voted is created. By specifying a linear pixel group on the V map, a pixel group corresponding to the road surface can be specified. In such a V map, the parallax value of the portion below the road surface is not detected.

  Next, the road surface shape detection unit 53 detects a road surface that is a reference object that serves as a reference for the height of each object. The road surface shape detection unit 53 linearly approximates a position estimated as a road surface on the V map. When the road surface is flat, it is approximated by a single straight line. Further, in the case of a road surface where the slope changes midway, the V map is divided into a plurality of sections to perform linear approximation. Thereby, even in the case of a road surface whose slope changes midway, linear approximation can be performed with high accuracy.

  Next, the road surface height calculation unit 54 calculates the road surface height and stores it in a storage unit such as the RAM 310 (forms a table). Since a linear expression representing a road surface is obtained from the V map, if the parallax d is determined, the corresponding y coordinate y0 is determined. This coordinate y0 is the height of the road surface. The road surface height calculation unit 54 generates a road surface height list in which coordinates y0 indicating the height of the road surface are associated with a necessary range of parallax. When the parallax is d and the y coordinate is y ′, y′−y0 indicates the height from the road surface when the parallax is d. The road surface height calculation unit 54 calculates the height H of the coordinates (d, y ′) from the road surface using an arithmetic expression “H = (z × (y′−y0)) / f”. In this arithmetic expression, “z” is a distance calculated from the parallax d (z = BF / (d−offset)), and “f” is the focal length of the stereo camera 2 in units of (y′−y0). The value converted to the same unit. Here, BF is a value obtained by multiplying the base line length B of the imaging unit 2 and the focal length f, and offset is a parallax when an object at infinity is photographed.

  As described above, the U map generation unit 44 generates a U map that is a two-dimensional histogram in which the horizontal axis is x of the parallax image, the vertical axis is the parallax d of the parallax image, and the frequency is the Z axis. At this time, a point (x, y, d) of the parallax image whose height from the road surface is in a predetermined height range such as 20 cm to 3 m is voted based on the value of (x, d). The U map generation unit 44 generates a U map by voting pixels in a predetermined range of a parallax image at a point (x, y, d) of a certain parallax image. That is, the sky is shown in the upper 1/6 region of the parallax image, and in most cases, an object that needs to be recognized is not shown. For this reason, the U map generation unit 44 generates a U map of the frequency by voting pixels in a 5/6 area (area other than the upper 1/6 area) of the ground side of the parallax image, for example.

  Using the U map that reflects the height from the road surface and the like, the separation process can be performed with higher accuracy by performing the separation process described above, and the same effects as those of the above-described embodiments can be obtained. Can be obtained.

  Each above-mentioned embodiment is shown as an example and is not intending limiting the range of the present invention. Each of the novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. For example, a distance value (distance value) and a parallax value can be handled equivalently. For this reason, in the description of the above-described embodiment, a parallax image is used as an example of a distance image, but the present invention is not limited to this. For example, a distance image may be generated by integrating distance information generated using a detection device such as a millimeter wave radar or a laser radar with a parallax image generated using a stereo camera. Further, the detection accuracy may be further enhanced by using a stereo camera and a detection device such as a millimeter wave radar or a laser radar in combination with the detection result of the object by the stereo camera described above.

  Such embodiments and modifications of the embodiments are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

1 vehicle 2 stereo camera (imaging part)
2a Camera unit 2b Camera unit 3 Vehicle ECU
4 imaging device 30 image processing device 42 collimating unit 43 parallax image generating unit 44 U map generating unit 45 wall detecting unit 46 separating unit 51 parallax interpolating unit 52 V map generating unit 53 road surface shape detecting unit 54 road surface height calculating unit

JP-A-6-266828

Claims (13)

A generating unit that generates two-dimensional distribution information indicating the horizontal position and the depth direction position of the object based on information in which the vertical position, the horizontal position, and the depth direction position of the object are associated;
Separation in which the length of each column along the depth direction in the object on the two-dimensional distribution information is detected, and the object is separated along the depth direction with a column of a predetermined length as a boundary And an information processing apparatus. The information processing apparatus according to claim 1, wherein the separation unit separates the object along the depth direction with a longest column as a boundary. The separating unit recognizes the longest column and the column adjacent to the longest column as a lump object among the columns having a length equal to or greater than a predetermined threshold, and the recognized lump object as a boundary, The information processing apparatus according to claim 1, wherein the object is separated along the depth direction. The separation unit detects a width that is the size of each row of the object along a row direction that is two-dimensionally orthogonal to the depth direction, and there is a difference of a predetermined amount or more in the width size. The information processing apparatus according to any one of claims 1 to 3, wherein the object is separated along the row direction with an existing row as a boundary. The separation unit detects a width that is the size of each row of the object along a row direction that is two-dimensionally orthogonal to the depth direction, and determines one or more values of the width. Including a first section and a second section that are continuous along the column direction, such that the second section is on the back side, and each of the sections is one along the column direction. Alternatively, the average value of the width values of the second section and the average value of the width values of the first section are calculated while moving each of the plurality of width values, and the first weight is calculated. 2 when the average value of the two sections is larger than the average value of the first section with the predetermined weighting, in the row direction at the boundary of the sections at the movement position of the sections. The object according to any one of claims 1 to 3, wherein the object is separated along the line. The information processing apparatus described in 1. When the width of each row of the object is detected by an amount that can set each section, the separating section sets each section, calculates an average value of each section, and separates the objects The information processing apparatus according to claim 5, wherein: The separation unit performs a search when performing a labeling process of performing a search for the object a plurality of times, detecting an information unit having a value equal to or greater than a predetermined threshold for each search, and attaching a label corresponding to the number of searches. If an information unit having a value equal to or greater than the threshold is selected from a plurality of information units close to the search direction with respect to a certain information unit, the number of searches is handled when the selected information unit is not labeled. A labeling process for attaching a label to be detected, a value in the row direction of the object is detected, and a number of the width values that enable separation along the row direction of the object are detected in the row direction. The information processing apparatus according to any one of claims 4 to 6, wherein the object is separated along. The information processing apparatus according to any one of claims 1 to 7, wherein the generation unit generates the two-dimensional distribution information that reflects a height of a road surface. An imaging unit that generates imaging information for generating two-dimensional distribution information indicating a lateral position and a depth direction position of the object;
An imaging device comprising: the information processing device according to any one of claims 1 to 8. An imaging device according to claim 9,
A control unit that controls a predetermined device based on the two-dimensional distribution information of the object separated by the information processing apparatus. It is controlled by the said control part of the apparatus control system of Claim 10, The moving body characterized by the above-mentioned. A generating step for generating two-dimensional distribution information indicating a horizontal position and a depth direction position of the object based on information in which the vertical position, the horizontal position, and the depth direction position of the object are associated;
The separation unit detects the length of each column along the depth direction in the object on the two-dimensional distribution information, and the object is detected along the depth direction with a column having a predetermined length as a boundary. An information processing method comprising: a separation step for performing separation processing. Computer
A generating unit that generates two-dimensional distribution information indicating the horizontal position and the depth direction position of the object based on information in which the vertical position, the horizontal position, and the depth direction position of the object are associated;
Separation in which the length of each column along the depth direction in the object on the two-dimensional distribution information is detected, and the object is separated along the depth direction with a column of a predetermined length as a boundary Information processing program characterized by functioning as a part.

Images