JP7179687B2 - Obstacle detector - Google Patents

Description

This embodiment relates to an obstacle detection device.

In recent years, for example, a stereo camera is mounted on a moving body such as a vehicle, and an obstacle is detected based on the images output from the camera at regular intervals, and the vehicle is controlled to avoid contact with the obstacle. A system (safe driving system) that assists driving by performing such automatic control is being put into practical use.

Accurate detection of obstacles is important when controlling a moving object using such a safe driving system.

JP 2013-239905 A

An object of the present embodiment is to provide an obstacle detection device that can accurately detect an obstacle.

An obstacle detection device according to the present embodiment is an obstacle detection device that detects obstacles existing outside a moving body based on a captured image obtained by a sensor provided on the moving body. a three-dimensional point detection unit for detecting the feature points and estimating the three-dimensional coordinates, which are the positions of the three-dimensional points, for each of the feature points; A 3D point cloud storage unit and a noise point removal unit that detects and removes the 3D points whose 3D coordinates are erroneously estimated from the 3D point cloud as noise points.
The noise point removal unit extracts one of the three-dimensional points as a noise candidate point, and when projecting the three-dimensional point group onto the captured image, the projection position of the noise candidate point is the center of the captured image. a three-dimensional point to be determined, which is one or more of the three-dimensional points projected in a predetermined area where the is compared with the average distance of , and it is determined whether or not the noise candidate point is a noise point.

The figure which shows an example of the mobile body 100 which mounts the obstacle detection apparatus concerning this embodiment. 1 is a block diagram showing an example of a configuration of a moving body 100 including an obstacle detection device according to this embodiment; FIG. 1 is a block diagram showing an example of an image processing device 10 mounted on a mobile body 100; FIG. 2 is a schematic block diagram showing an example of a configuration related to an obstacle detection device in the image processing device 10; FIG. Explanatory diagram of three-dimensional coordinate calculation using image template matching and triangulation. Explanatory diagram of three-dimensional coordinate calculation using image template matching and triangulation. FIG. 4 is a diagram for explaining an example of a mechanism of generating noise points; FIG. 4 is a diagram for explaining an example of a mechanism of generating noise points; 4 is a flowchart for explaining a procedure for removing noise points from a three-dimensional point cloud according to the first embodiment; FIG. 4 is a diagram for explaining an example of a noise point removal method; FIG. 4 is a diagram for explaining an example of a noise point removal method; 9 is a flowchart for explaining the procedure for removing noise points from a three-dimensional point cloud according to the second embodiment; FIG. 10 is a diagram for explaining a summation calculation procedure of values in an evaluation region based on an integral image;

Embodiments will be described below with reference to the drawings.
(First embodiment)
FIG. 1 is a diagram showing an example of a moving object 100 equipped with an obstacle detection device according to this embodiment.

The moving body 100 includes an image processing device 10, an output section 100A, a sensor 100B, a power control section 100G, and a power section 100H.

A moving object 100 is a movable object. The mobile object 100 is, for example, a vehicle (motorcycle, automobile, bicycle), cart, robot, ship, flying object (airplane, unmanned aerial vehicle (UAV: Unmanned Aerial Vehicle), etc.). The mobile object 100 is, for example, a mobile object that travels through human driving operation, or a mobile object that can automatically travel (autonomously travel) without human driving operation. A mobile body capable of autonomous travel is, for example, an autonomous vehicle. The moving body 100 of the present embodiment will be described as an example of a vehicle that can travel autonomously and that can also be operated by a person.

The output unit 100A outputs various information. For example, the output unit 100A outputs output information from various processes.

The output unit 100A has, for example, a communication function for transmitting output information, a display function for displaying output information, a sound output function for outputting sound indicating output information, and the like. For example, the output section 100A includes a communication section 100D, a display 100E, and a speaker 100F.

The communication unit 100D communicates with an external device. The communication unit 100D is a VICS (registered trademark) communication circuit or a dynamic map communication circuit. The communication unit 100D transmits output information to an external device. The communication unit 100D also receives road information and the like from an external device. Road information includes traffic signals, signs, surrounding buildings, road width of each lane, lane center lines, and the like. The road information may be stored in a memory 10b such as RAM and ROM provided in the image processing apparatus 10, or may be stored in a memory separately provided in the moving body. The specific configuration of the memory 10b will be described in detail in the description of the image processing apparatus 10 in FIG. 3, which will be described later.

The display 100E displays output information. The display 100E is, for example, a known LCD (Liquid Crystal Display), projection device, light, or the like. The speaker 100F outputs sound indicating the output information.

The sensor 100B is a sensor that acquires the traveling environment of the moving body 100. FIG. The driving environment is, for example, observation information of the mobile object 100 and peripheral information of the mobile object 100 . The sensor 100B is, for example, an external sensor or an internal sensor.

An internal sensor is a sensor that observes observation information. The observation information includes, for example, the acceleration of the mobile object 100, the speed of the mobile object 100, and the angular velocity of the mobile object 100 (yaw axis angular velocity).

The internal sensor is, for example, an inertial measurement unit (IMU: Inertial Measurement Unit), an acceleration sensor, a speed sensor, a rotary encoder, a yaw rate sensor, or the like. The IMU observes observation information including triaxial acceleration and triaxial angular velocity of the moving body 100 .

The external sensor observes surrounding information of the moving object 100 . The external sensor may be mounted on the moving body 100 or may be mounted outside the moving body 100 (for example, another moving body or an external device).

Peripheral information is information indicating the surrounding situation of the moving object 100 . The periphery of the moving body 100 is an area within a predetermined range from the moving body 100 . This range is the observable range of the external sensor. This range may be set in advance.

Peripheral information is, for example, a photographed image and distance information of the periphery of the mobile object 100 . In addition, the peripheral information may include position information of the moving object 100 . A photographed image is photographed image data obtained by photographing (hereinafter, simply referred to as a photographed image in some cases). The distance information is information indicating the distance from the moving body 100 to the object. A target is a point in the external world observable by an external sensor. The position information may be relative position or absolute position.

External sensors include, for example, a photographing device (camera) that obtains a photographed image by photographing, a distance sensor (millimeter wave radar, laser sensor, distance image sensor), a position sensor (GNSS (Global Navigation Satellite System), GPS (Global Positioning System) , wireless communication device), etc.

The captured image is digital image data that defines the pixel value for each pixel, a depth map (distance image) that defines the distance from the sensor 100B for each pixel, or the like. The laser sensor is, for example, a two-dimensional LIDAR (Laser Imaging Detection and Ranging) sensor installed parallel to a horizontal plane, or a three-dimensional LIDAR sensor.

The power control section 100G controls the power section 100H. The power unit 100H is a driving device mounted on the moving body 100 . The power section 100H is, for example, an engine, a motor, wheels, and the like.

The power section 100H is driven under the control of the power control section 100G. Since the moving object 100 of the present embodiment can travel autonomously, the power control unit 100G can monitor the surrounding situation based on output information generated by the image processing device 10, information obtained from the sensor 100B, and the like. Acceleration amount, brake amount, steering angle, etc. are controlled. That is, when an obstacle detected in front of the moving body 100 may collide with the moving body 100, the power unit 100H is controlled to avoid contact between the moving body 100 and the obstacle.

Next, the electrical configuration of the moving body 100 is described in detail. FIG. 2 is a block diagram showing an example of the configuration of the moving body 100 including the obstacle detection device according to this embodiment.

The moving body 100 includes an image processing device 10, an output section 100A, a sensor 100B, a power control section 100G, and a power section 100H. Output unit 100A includes communication unit 100D, display 100E, and speaker 100F, as described above. The obstacle detection device 1 of this embodiment includes an image processing device 10 .

Image processing device 10, output unit 100A, sensor 100B, and power control unit 100G are connected via bus 100I. The power section 100H is connected to the power control section 100G.

At least one of the output unit 100A (communication unit 100D, display 100E, speaker 100F), sensor 100B, and power control unit 100G may be connected to the image processing apparatus 10 by wire or wirelessly. At least one of the output unit 100A (communication unit 100D, display 100E, speaker 100F), sensor 100B, and power control unit 100G may be connected to the image processing device 10 via a network.

FIG. 3 is a block diagram showing an example of the image processing device 10 mounted on the mobile body 100. As shown in FIG. The image processing device 10 includes an I/F 10c, a memory 10b, and a processor 10a.

The I/F 10c is connected to a network (N/W) or the like with other systems. Also, the I/F 10c controls transmission and reception of information with the communication unit 100D. Through the I/F 10c, information on a recognized target such as a person and information on the distance to the recognized target are output.

The memory 10b stores various data. The memory 10b is, for example, a RAM (Random Access Memory), a semiconductor memory device such as a flash memory, a hard disk, an optical disk, or the like. Note that the memory 10 b may be provided outside the image processing apparatus 10 . The ROM holds programs executed by the processor 10a and necessary data. The RAM functions as a work area for the processor 10a. Also, the memory 10b may be provided outside the moving body 100 . For example, the memory 10b may be arranged in a server device installed on the cloud.

Also, the memory 10b may be a storage medium. Specifically, the storage medium may store or temporarily store programs and various information downloaded via a LAN (Local Area Network), the Internet, or the like. Also, the memory 10b may be composed of a plurality of storage media.

Each processing function of the processor 10a is stored in the memory 10b in the form of a computer-executable program. The processor 10a is a processor that implements a functional unit corresponding to each program by reading and executing the program from the memory 10b.

Note that the processing circuit 10e may be configured by combining a plurality of independent processors for realizing each function. In this case, each processor implements each function by executing a program. Further, each processing function may be configured as a program, and one processing circuit 10e may execute each program, or an image processing accelerator 10d may be provided as a dedicated circuit, and a specific function may be implemented in an independent program execution circuit. good too.

The processor 10a implements its functions by reading and executing programs stored in the memory 10b. Instead of storing the program in the memory 10b, the processor may be configured with an FPGA or the like, and the program may be configured to be directly embedded in the circuit. In this case, the processor implements its functions by reading and executing the program embedded in the circuit.

FIG. 4 is a schematic block diagram showing an example of the configuration of the image processing apparatus 10. As shown in FIG. FIG. 4 shows only the configuration related to the obstacle detection device. The image processing device 10 is composed of an image processing section 11 and an obstacle identification section 12 .

The image processing unit 11 estimates the orientation of the camera included in the sensor 100B in a three-dimensional space (hereinafter referred to as camera orientation). Also, obstacles in front of the camera are extracted as a three-dimensional point group. The image processing unit 11 includes an image processing control unit 111, a camera posture estimation unit 112, a 3D point group detection unit 113, a 3D point group storage unit 114, and a noise point removal unit 115. be.

The obstacle identification unit 12 identifies obstacles in front of the camera based on the camera orientation estimated by the image processing unit 11 and the extracted three-dimensional point group.

The image processing control unit 111 controls operations of other components of the image processing unit 11 so as to estimate the camera pose and extract the 3D point cloud.

The camera posture estimation unit 112 acquires captured images in time series from the camera included in the sensor 100B. Time-series captured images are input at regular time intervals. In the following description, the times at which captured images are input are represented by numbers such as 0, 1, 2, . . . , t−1, t, t+1, . Also, in the following description, a photographed image obtained from a camera is referred to as an input image, and an input image at time t is referred to as frame t. The camera posture estimation unit 112 analyzes the acquired time-series captured images and estimates the camera posture using a technique such as SfM (Structure from Motion).

A three-dimensional point group detection unit 113 as a three-dimensional point detection unit detects feature points from an input image and calculates three-dimensional coordinates of positions where the feature points actually exist. In the following description, a point where a feature point is estimated to actually exist will be referred to as a three-dimensional point, and the position of the three-dimensional point will be referred to as a three-dimensional coordinate. A method of calculating three-dimensional coordinates from feature points in the three-dimensional point group detection unit 113 will be described. First, the three-dimensional point group detection unit 113 uses, for example, a Sobel filter or the like to detect boundaries (edges) of an object or the like existing in the input image (frame t) and set them as feature points. Note that feature point detection may be performed using other techniques or algorithms such as GFTT (Good Features To Track).

Next, the 3D point cloud detection unit 113 associates the feature points detected in the current frame (frame t) with the feature points detected in the past frame (eg, frame t−1), Identify point pairs. That is, a corresponding point pair is a pair of feature points detected for the same portion of the same structure existing in both the past frame and the current frame. Corresponding point pairs can be identified, for example, by image template matching.

Specifically, for the feature points detected in the current frame and the previous frame, a small area centered on the feature point is set. Then, the degree of similarity between the small area set in the current frame and the small area set in the past frame is calculated. If the similarity is equal to or higher than the set threshold, the feature points in the small area of the current frame and the feature points in the small area of the past frame are taken as corresponding point pairs. Similarity calculations are ZNCC (Zero-mean Normalization Cross-Correction, zero-mean normalization correlation), NCC (Normalization Cross-Correction, normalization correlation), SAD (Sum of Squared Difference), SSD (Sum of Absolute Difference) , etc., using a generally known similarity calculation. Note that the identification of corresponding point pairs is not limited to the image template matching described above. For example, corresponding point pairs may be identified using feature point matching techniques such as ORB features (Oriented FAST and Rotated BRIEF).

Finally, for each identified pair of corresponding points, the feature points Estimate the 3D coordinates from A general three-dimensional coordinate estimation method such as triangulation is used to estimate the coordinates.

A method for estimating three-dimensional coordinates from feature points in the three-dimensional point group detection unit 113 will be specifically described with reference to FIGS. 5 and 6 are explanatory diagrams of three-dimensional coordinate calculation using image template matching and triangulation. FIG. 5 is an input image acquired by the three-dimensional point cloud detection unit 113, and shows a frame t, which is a current input image, and a frame t-1, which is a past input image, arranged in chronological order from left to right. ing. In the following drawings, the Y direction is the direction orthogonal to the road surface on which the moving body 100 travels, the Z direction is the direction parallel to the road surface in which the moving body 100 travels (depth direction), and the Z direction is parallel to the road surface. The direction orthogonal to the traveling direction of the body 100 is defined as the X direction.

First, the three-dimensional point group detection unit 113 detects feature points PFt in a frame t, which is the current frame. FIG. 5 shows an example in which edges (corners) of a box-shaped obstacle placed on the road are detected as feature points PFt. Next, the feature points PFt of the frame t are associated with the feature points already detected in the frame t−1, which is the past frame, to identify corresponding point pairs. In the example shown in FIG. 5, image template matching identifies a feature point PFt of frame t and a feature point PFt-1 of frame t-1 as a pair of corresponding points.

Finally, the three-dimensional point group detection unit 113 uses triangulation or the like to estimate the three-dimensional point of the feature point PFt of the frame t and calculate the three-dimensional coordinates. A specific method will be described with reference to FIG. First, a straight line is drawn connecting the optical center of the lens of the camera 100B(t) when the frame t is captured and the feature point PFt on the frame t. Next, draw a straight line connecting the optical center of the lens of the camera 100B(t-1) when the frame t-1 was shot and the feature point PFt-1 on the frame t-1. Then, the intersection of the two straight lines is estimated to be the real point (three-dimensional point Ptd) of the edge (corner) of the obstacle detected as the feature point PFt.

The position coordinates of the three-dimensional point Ptd are: (i) the camera postures of the frames t-1 and t, (ii) the coordinates of the feature point PFt-1 in the frame t-1, and (iii) the feature point PFt in the frame t. and (iv) the focal length of the camera (the distance from the optical center of the camera lens to the frame).

The 3D point cloud storage unit 114 stores the 3D points calculated by the 3D point cloud detection unit 113 in a buffer or the like. In general, multiple feature points are detected for one frame. Then, for each feature point for which a corresponding point pair has been specified, a three-dimensional point is estimated and three-dimensional coordinates are calculated. In the following description, a set of three-dimensional points will be referred to as a three-dimensional point group. That is, the 3D point cloud accumulation unit 114 acquires and accumulates 3D point cloud information (three-dimensional coordinates) from the 3D point cloud detection unit 113 .

The 3D point cloud accumulation unit 114 accumulates information on 3D point clouds calculated in past frames. Each time a new frame is input to the 3D point cloud detection unit 113 and the 3D coordinates of the feature points are calculated for that frame (current frame), the 3D point cloud storage unit 114 transfers the 3D point cloud information is output. The 3D point cloud storage unit 114 stores information on the input 3D point cloud in addition to information on the 3D point cloud that has already been stored. When the number of 3D points in one frame is large and the number of frames to be input is large, the number of 3D points to be input to the 3D point cloud accumulation unit 114 becomes enormous, and it is difficult to store all the information. Then, a huge amount of buffer is required. Since the capacity of a buffer is generally limited, it is preferable to use a ring buffer or the like to replace data to be stored in a FIFO (first in first out) manner.

The noise point removal unit 115 extracts and removes noise points from the three-dimensional points accumulated in the three-dimensional point cloud accumulation unit 114 . Three-dimensional points included in a three-dimensional point group detected in a local portion of the same structure will have similar distances (coordinates in the Z direction, depth) from the camera if they are detected correctly. Therefore, if the 3D point group includes a 3D point whose position is clearly different from the Z coordinate of another 3D point, the 3D coordinate of the 3D point is calculated correctly. Most likely not. Therefore, by extracting such three-dimensional points as noise points and removing them from the three-dimensional point group, it is possible to improve detection accuracy in subsequent obstacle detection.

Here, the noise point generation mechanism will be described with reference to FIGS. 7 and 8 are diagrams for explaining an example of the mechanism of noise point generation. One of the main reasons why three-dimensional coordinates are not calculated correctly is erroneous identification of corresponding point pairs. When a repetitive pattern exists in an image, a plurality of feature points with similar degrees of similarity exist on the repetitive pattern. An example of a repeating pattern is a guiding fluid (a road marking in which white lines are drawn on the road at regular intervals, also called a zebra zone) as shown in FIG. In frame t-1, which is a past input image, it is assumed that feature point PFt-1 is detected in the fifth white line portion from the Y direction among the white lines of the conducting fluid. In frame t, which is the current input image, a feature point PFt′ is detected in the white line portion of the leading fluid that is the sixth white line from the Y direction, and a feature point PFt is detected in the white line portion that is the fifth from the Y direction. shall be detected.

When identifying a corresponding point pair by image template matching, a minute area centered on each feature point is cut out to calculate the degree of similarity. Correctly, the feature point PFt-1 of the frame t-1 and the feature point PFt of the frame t are the corresponding point pair. (Both feature points are located on the fifth white line portion from the Y direction of the white line of the guiding fluid.) However, both the feature point PFt′ and the feature point PFt are present on the white line of the guiding fluid. there is For this reason, the minute area of the feature point PFt' and the minute area of the feature point PFt are very similar, and the feature point PFt-1 of the frame t-1 and the feature point PFt' of the frame t form a corresponding point pair. May be misidentified.

When a correct three-dimensional point is estimated using the feature points PFt as corresponding point pairs, the estimated three-dimensional point Ptd is on the road surface (Y=0) and is at the depth from the camera 100B(t), as shown in FIG. is calculated to be the position of Z=d1. On the other hand, if an incorrect three-dimensional point is estimated using an incorrect feature point PFt' as a pair of corresponding points, the estimated three-dimensional point Ptd' is located at a height Y=h2 from the road surface and from the camera 100B(t). is calculated to be the position of Z=d2.

In this way, if a 3D point is estimated using an erroneous pair of corresponding points, 3D coordinates different from those of a correctly estimated 3D point will be calculated. In particular, the Z-axis, which is the depth direction, tends to result in a large difference between correct and incorrect 3D point positions.

The noise point removal unit 115 removes noise points from the three-dimensional points accumulated in the three-dimensional point cloud accumulation unit 114 based on the properties of the noise points described above. That is, when the distance of a certain 3D point from the camera 100B in the depth direction is significantly different from other 3D point groups present in the neighboring area on the image, the 3D point is presumed to be a noise point and removed.

FIG. 9 is a flowchart for explaining the procedure for removing noise points from the three-dimensional point cloud according to the first embodiment. First, the 3D point cloud stored in the 3D point cloud storage unit 114 is acquired and projected onto the current frame (frame t). Then, the coordinates (image coordinates) on the frame are calculated for each three-dimensional point (S1). For example, when T 3D point clouds are accumulated in the 3D point cloud accumulation unit 114, T image coordinates are calculated in S1.

Next, a three-dimensional point is set from among the T three-dimensional points to determine whether it is a noise point or not (S2). For example, a serial number from 1 to T is given to each of T three-dimensional points. In S2, the three-dimensional point assigned number 1 is set as the point to be first subjected to noise point determination. In the procedure of FIG. 9, noise point determination target points as noise candidate points are specified by substituting and holding the number of the three-dimensional point for which noise point determination is to be performed in the variable n.

Subsequently, for the three-dimensional points that are noise point determination target points, an evaluation region Re including the three-dimensional points is set (S3). 10 and 11 are diagrams for explaining an example of the noise point removal method. FIG. 10 shows an image obtained by projecting the 3D point cloud stored in the 3D point cloud storage unit 114 onto the current frame. FIG. 11 is a diagram for explaining three-dimensional coordinates of three-dimensional points included in the evaluation area Re. In FIGS. 10 and 11, the three-dimensional point Ptd indicated by the hatched circle indicates the correctly estimated three-dimensional point, and the three-dimensional point Ptd′ indicated by the white circle including the cross mark. indicates an erroneously estimated 3D point.

First, as shown in FIG. 11, it is assumed that a 3D point group PG consisting of 15 3D points is accumulated in the 3D point group accumulation unit 114 . Projecting these 15 3D point clouds PG onto the current frame yields the image shown in FIG. For example, in S2, it is assumed that the three-dimensional point P1 is set as the noise point determination target point. In this case, in S3, a rectangular area having a predetermined size centered on the point P1 is set as the evaluation area Re.

Then, three-dimensional points included in the evaluation region Re are extracted (S4). In the example shown in FIG. 10, four 3D points including 3D point P1 are extracted. The pixel coordinates calculated in S1 are used to determine whether or not the three-dimensional point projected onto the current frame is included in the evaluation area Re.

Subsequently, the average distance of the three-dimensional points extracted in S4 (the average value of the distances in the depth direction from the camera to each three-dimensional point) is calculated (S5). That is, the average value in the Z direction is calculated for the three-dimensional coordinates of the three-dimensional points extracted in S4. For example, in the case of the example shown in FIG. 11, the three-dimensional points included in the evaluation region Re are four points, Ptd1′, Ptd2, Ptd3, and Ptd4. Therefore, the average value of the Z coordinates of these four points is calculated.

Then, the difference between the distance in the depth direction from the camera of the noise point determination target point and the average distance calculated in S4 is calculated. Further, the calculated difference is divided by the average distance to calculate a noise evaluation value (S6). For example, the calculation of the noise evaluation value in S6 is performed using the following equation (1).

Noise evaluation value = | distance of determination target point - average distance | / average distance (1) In equation (1), the noise evaluation value is the average distance of the distance of the determination target point (distance in the depth direction from the camera). The closer to , the closer to zero. Conversely, the farther the distance of the determination target point (the distance in the depth direction from the camera) from the average distance, the greater the value.

Next, the noise evaluation value calculated in S6 is compared with a preset threshold value. If the noise evaluation value is greater than the threshold (S7, YES), the determination target point is determined as a noise point and removed from the three-dimensional point group (S8). When S8 ends, the process proceeds to S9. On the other hand, if the noise evaluation value is equal to or less than the threshold (S7, NO), the determination target point is determined not to be a noise point (it is a correctly estimated three-dimensional point), so S8 is skipped and the process proceeds to S9. In S7, since the noise evaluation value is obtained by dividing the distance difference by the average distance, it is possible to perform noise judgment for all the judgment target points using a single threshold regardless of the magnitude of the average distance.

In S9, it is checked whether noise point determination has been performed for all three-dimensional points acquired in S1. If an undetermined three-dimensional point exists (S9, NO), the next point to be determined is set (S10), and a series of noise point determination and removal procedures from S3 to S8 are repeatedly executed. If there is no undetermined 3D point, that is, if noise point determination has been performed for all 3D points (S9, YES), a series of procedures for removing noise points from the 3D point group is terminated.

For example, if a serial number from 1 to T is given to each of T three-dimensional points, and the points to be determined are sequentially numbered from 1, in S9, the current point to be determined, which is the value of the variable n, is If T (n==T), it can be considered that all 3D points have been determined. In S9, if n<T, a series of noise point determination and removal procedures from S3 to S8 are repeatedly executed with the next numbered three-dimensional point as a determination target point (n=n+1).

The noise point removal unit 115 outputs the three-dimensional point cloud for which noise point removal has been completed to the obstacle identification unit 12 .

As described above, according to the present embodiment, it is determined whether or not the 3D coordinates of the 3D points included in the 3D point group estimated from the input image are correctly calculated based on the depth positions. 3D points whose 3D coordinates are estimated to be incorrectly calculated are removed from the 3D point group as noise points, and then the 3D point group is input to the obstacle identification unit 12 . Therefore, since the obstacle identification unit 12 can identify an obstacle using highly accurate three-dimensional points from which noise points have been removed, there is a possibility of erroneously detecting an obstacle in an empty space. is reduced, and obstacles can be detected with high accuracy.

Note that the number of feature points detected from an input image varies greatly depending on the image. In the example shown in FIGS. 10 and 11, the number of feature points is 15, but in reality there may be cases where the number of feature points is detected on the order of hundreds to thousands.

In the above example, the input image is a photographed image captured by a camera, but a depth map (distance image) obtained by a distance sensor such as a laser sensor or a position sensor such as GPS may be used as the input image.
(Second embodiment)
In the above-described first embodiment, when extracting the 3D points included in the evaluation area from the 3D point group, it is determined whether or not the pixel coordinates of each 3D point exist within the evaluation area. there is In this case, the number of determination processes per frame is about the square of the number of three-dimensional points. Therefore, as the number of three-dimensional points increases, the number of times of determination processing increases quadratically, and the processing time becomes longer, which may make real-time processing difficult.

On the other hand, the present embodiment differs in that an integral image is used to determine whether or not a three-dimensional point is included in the evaluation region, thereby speeding up the processing. The configuration of the obstacle detection device according to this embodiment is the same as that of the first obstacle detection device shown in FIGS. 1 to 4, so the description thereof is omitted.

FIG. 12 is a flowchart for explaining procedures for removing noise points from a three-dimensional point cloud according to the second embodiment. First, the 3D point cloud stored in the 3D point cloud storage unit 114 is acquired and projected onto the current frame (frame t). Then, the coordinates (image coordinates) on the frame are calculated for each three-dimensional point (S11).

Next, a distance image and a point cloud number image are created (S12). The distance image is an image in which the pixel value of the image coordinates of the three-dimensional point calculated in S11 is the value of the three-dimensional coordinates of the three-dimensional point in the Z direction. For example, if the image coordinates obtained by projecting a 3D point onto the current frame are (X, Y)=(100, 100) and the distance in the depth direction of the 3D point is 25, then the created distance The pixel value of (X, Y)=(100, 100) of the image is 25. Note that pixel values at coordinates where no 3D point exists are assumed to be 0 in the frame onto which the 3D point is projected. When a plurality of three-dimensional points are projected onto one pixel, the pixel value of that pixel in the range image is the sum of the Z-direction values of the plurality of three-dimensional points projected onto that pixel.

The point cloud number image is an image whose pixel value is the number of three-dimensional points projected on each coordinate in the frame on which the three-dimensional points are projected in S11. For example, the image coordinates obtained by projecting four 3D points onto the current frame are (X, Y) = (100, 100), (100, 100), (100, 101), (99, 100). , the pixel value of (X, Y) = (100, 100) is 2, the pixel value of (X, Y) = (100, 101), (99, 100) is 1, and Become. In a frame on which a 3D point is projected, the pixel value of coordinates where no 3D point exists is set to 0.

Next, an integral image of the distance image created in S12 (hereinafter referred to as a distance integral image) and an integral image of the point cloud number image (hereinafter referred to as a point cloud number integral image) are created (S13). An integral image is an image obtained by integrating the pixel values of an image, and is an image created to quickly calculate the layer sum of pixel values within a rectangular area arbitrarily set in the original image. The pixel value of each pixel in the integral image is the pixel value of the pixel value within a rectangular area in which the pixel of the same coordinates in the original image is the lower right corner and the origin ((X, Y)=(0, 0) point) is the upper left corner. summation. In S13, an integral image is created for each of the distance image and the point cloud number image.

Next, from among the three-dimensional points acquired in S11, a three-dimensional point is set first for determining whether or not it is a noise point (S14). For example, a serial number from 1 to T is given to each of T three-dimensional points. In S14, the three-dimensional point assigned number 1 is set as the point for which noise point determination is performed first. In the procedure of FIG. 12, a noise point determination target point is specified by substituting and holding the number of a three-dimensional point whose noise point is to be determined in the variable n.

Next, for the three-dimensional points that are the noise point determination target points, an evaluation region Re including the three-dimensional points is set (S15), and the average distance of the three-dimensional point group included in the evaluation region Re (from the camera to each cubic An average value of distances in the depth direction to the original point) is calculated (S16). The calculation of the average distance in S16 is performed using the integral image created in S13.

13A and 13B are diagrams for explaining the summation calculation procedure of the values in the evaluation area based on the integral image. As shown in FIG. 13, the sum of pixel values included in the evaluation region Re is calculated using the following equation (2).

SRe=SRa-SRb-SRc+SRd Equation (2) where SRx represents the sum of pixel values in the rectangular region Rx.
SRa, SRb, SRc, and SRd can be obtained as pixel values at the lower right corners of rectangular regions Ra, Rb, Rc, and Rd in the integral image, respectively. Therefore, equation (2) can be calculated by acquiring pixel values at four pixel positions from the integral image and performing sum-difference operation three times.

The average distance can be calculated by dividing the sum of the distances of the three-dimensional points included in the evaluation region Re by the number of three-dimensional points included in the evaluation region Re. Therefore, the average distance is calculated by dividing the sum of the pixel values included in the evaluation region Re in the distance integral image created in S12 by the sum of the pixel values included in the evaluation region Re in the point cloud number integral image created in S12. be able to.

Then, the difference between the distance in the depth direction from the camera of the noise point determination target point and the average distance calculated in S16 is calculated. Further, the calculated difference is divided by the average distance to calculate a noise evaluation value (S17). Calculation of the noise evaluation value in S17 is the same as in S6 of the first embodiment described using FIG. Then, the noise evaluation value calculated in S17 is compared with a preset threshold value, and if the noise evaluation value is greater than the threshold value (S18, YES), the determination target point is determined as a noise point, and the three-dimensional point group (S19). When S19 ends, the process proceeds to S20. On the other hand, if the noise evaluation value is less than or equal to the threshold value (S18, NO), it is determined that the determination target point is not a noise point (it is a correctly estimated three-dimensional point), so S19 is skipped and the process proceeds to S20.

In S20, it is confirmed whether noise point determination has been performed for all three-dimensional points acquired in S11. If an undetermined three-dimensional point exists (S20, NO), the next point to be determined is set (S21), and a series of noise point determination and removal procedures from S15 to S19 are repeatedly executed. If there are no undetermined 3D points, that is, if noise point determination has been performed for all 3D points (S20, YES), a series of procedures for removing noise points from the 3D point group is terminated.

In the procedure for removing noise points in the first embodiment described above, each time the determination target point is set to the next three-dimensional point, it is determined whether or not the pixel coordinates of each three-dimensional point exist within the evaluation area. In order to do so, it was necessary to perform calculations equivalent to the square of three-dimensional points. Then, it was necessary to calculate the noise evaluation value using the three-dimensional points existing within the evaluation area. On the other hand, in this embodiment, even if the three-dimensional point of the determination target point changes, the noise evaluation value can be calculated only by performing the four arithmetic operations seven times using the distance image created in S13. Therefore, it is possible to reduce the amount of calculation and the processing cost, so that the speed of calculation can be increased.

The noise point removal unit 115 may be configured to switch between the noise point removal procedure according to the first embodiment and the noise removal procedure according to the second embodiment according to the number of three-dimensional points. .

While several embodiments of the invention have been described, these embodiments have been presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.

Reference Signs List 1 obstacle detection device 10 image processing device 10a processor 10b memory 10d image processing accelerator 10e processing circuit 11 image processing unit 12 obstacle identification unit 100 moving object 100A output unit 100B sensor (camera )
100D...communication section 100E...display 100F...speaker 100G...power control section 100H...power section 100I...bus 111...image processing control section 112...camera attitude estimation section 113...three-dimensional point group detection section 114...three-dimensional point group accumulation section 115... Noise point removing unit

Claims (4)

An obstacle detection device that detects an obstacle existing outside the moving object based on a photographed image obtained by a sensor provided on the moving object,
a three-dimensional point detection unit that detects a plurality of feature points from the captured image and estimates three-dimensional coordinates, which are the positions of the three-dimensional points, for each of the feature points;
a 3D point cloud accumulation unit for accumulating a 3D point cloud composed of the plurality of detected 3D points;
a noise point removal unit that detects and removes, as noise points, the three-dimensional points whose three-dimensional coordinates are erroneously estimated from the three-dimensional point group;
has
The noise point removal unit extracts one of the three-dimensional points as a noise candidate point, and when projecting the three-dimensional point group onto the captured image, the projection position of the noise candidate point is the center of the captured image. a three-dimensional point to be determined, which is one or more of the three-dimensional points projected in a predetermined area where the , and determines whether or not the noise candidate point is a noise point . The noise point removal unit divides the difference between the distance of the candidate noise point from the sensor and the average distance of the three-dimensional point to be determined from the sensor by the average distance so that the evaluation value is greater than a predetermined threshold. 2. The obstacle detection device according to claim 1 , wherein said noise candidate point is determined to be a noise point when it is large. 2. The obstacle detection device according to claim 1 , further comprising an obstacle identification unit that identifies the obstacle using the three-dimensional point group from which the noise points have been removed. The noise point removal unit is configured to, in a projection image obtained by projecting the three-dimensional point group onto the captured image, a first integral image for the distance from the sensor of the three-dimensional point group, and A second integral image is created, and an average distance of the three-dimensional point to be determined from the sensor is calculated using the first integral image and the second integral image. The obstacle detection device according to .

Images