JP2021189917A - Object detection system, object detection method, and object detection program - Google Patents

Abstract

To provide an object detection system configured to allow a user to define an obstacle and detect an object, such as an obstacle, with high accuracy, while enabling high-speed processing.SOLUTION: An object detection system 10 includes: image generation means 30 which generates a three-dimensional image 31 on the basis of imaging data obtained by imaging means 20; perspective data generation means 40 which generates perspective data 41 by perspective processing on the basis of the three-dimensional image generated by the image generation means; and analysis means 50 which detects depth where an object exists in each subdivided part by horizontal subdivision and depth subdivision based on the perspective data, and combines detection results to detect the object. The perspective data generation means extracts cross sections, as slice images, in a depth direction of the three-dimensional image, to generate perspective data by reducing the dimensions of the slice images. The analysis means extracts cross sections, as slice data, in a height direction on the basis of the perspective data, to detect a direction and a distance of the object.SELECTED DRAWING: Figure 1

Description

The present invention converts, for example, a three-dimensional original image captured by a camera or the like into low-dimensional bird's-eye view data by bird's-eye view processing, and detects objects such as people and obstacles in the original image based on the bird's-eye view data. It relates to an object detection system, an object detection method, and an object detection program for detecting an object in a three-dimensional image.

For example, in the automatic driving of a traveling vehicle such as a car, an object such as a person, an obstacle, or a roadside edge is detected from the front view of the traveling vehicle, and a driveable area is confirmed to avoid a collision with the object. , It is necessary to control the drive of the traveling vehicle. Conventionally, when detecting such an object in front of the traveling direction, the object detection is performed as follows.

First, a stereo camera attached to the front of the traveling vehicle captures the front of the traveling vehicle, and a three-dimensional image is generated from a pair of left and right images of the stereo camera. Subsequently, by performing bird's-eye view processing on this three-dimensional image, a bird's-eye view image viewed from above regarding the imaging range of the three-dimensional image is generated. At that time, the bird's-eye view processing, that is, the image conversion processing from the three-dimensional image to the bird's-eye view image is performed by using the neural network. Here, as the neural network, a so-called convolutional neural network is used, and learning is performed by deep learning to obtain a desired bird's-eye view image. Next, based on the bird's-eye view image obtained in this way, an object is detected in the area of the bird's-eye view image by image processing. This object detection process also uses a neural network in the same manner, and can be learned by deep learning to perform desired object detection.

On the other hand, for example, Non-Patent Document 1 discloses a method of detecting an obstacle from a single color image by using a convolutional neural network.

D. Levi, N. Garnett and E. Fetaya, “StixelNet: A Deep Convolutional Network for Obstacle Detection and Road Segmentation”, http://www.bmva/2015/papers/paper109.pdf C. Godard, O. M. Aodha, M Firman and G. Brostow, “Digging Into Self-Supervised Monocular Depth Estimate”, https://arxiv.org/abs/1806.01260 J. Castorena, U.S. Kamilov and P. T. Boufounos, “AUTOCALIBRATION OF LIDAR AND OPTICAL CAMERAS VIA EDGE ALIGNMENT”, https://www.merl.com/publications/docs/TR2016-009.pdf

However, image processing conversion from a three-dimensional image directly to a bird's-eye view image requires a huge amount of processing data and takes a long time to process. When trying to process continuously, the process may not be in time. Further, as the distance from the stereo camera to the object increases, the detection accuracy of the object decreases significantly. On the other hand, in the obstacle detection method of Non-Patent Document 1, since a two-dimensional image by a monocular camera is used, it is not assumed that the obstacle is detected directly from the three-dimensional image.

In view of the above points, it is the first aspect of the present invention to provide an object detection system that can be processed quickly, can define an obstacle by a user, and can detect an object such as an obstacle with higher accuracy. The first object is to provide an object detection method, and the third object is to provide an object detection program.

The first object of the present invention is an image generation means that generates a three-dimensional image based on the image pickup data acquired by the image pickup means, and a bird's-eye view data by a bird's-eye view process based on the three-dimensional image generated by the image generation means. An analysis means for detecting an object by detecting the depth at which an object exists for each divided portion by horizontal division and depth division based on the bird's-eye view data and combining the detection results. The bird's-eye view data generation means takes out a plurality of cross sections as slice images in the depth direction of the three-dimensional image, lowers each slice image to generate bird's-eye view data, and the analysis means. It is achieved by an object detection system that detects the direction and distance of an object by extracting a plurality of cross sections in the height direction as slice data based on the bird's-eye view data.

Preferably, the bird's-eye view data generation means is composed of an autoencoder composed of a convolutional neural network, and the autoencoder reduces the dimension of each slice image to generate bird's-eye view data.
The neural network is preferably a multi-layer neural network consisting of an input layer, at least one intermediate layer and an output layer, and each slice image input to the input layer during training is low-dimensional in any intermediate layer. After converting to intermediate data, the output layer decodes it into reconstructed data of the same dimension as the slice image, and learns by deep learning so that the reconstructed data can reproduce the object in the slice image, and after learning, from the intermediate layer. The intermediate data is output to the analysis means as bird's-eye view data.
The bird's-eye view data generation means preferably slices each slice image in the horizontal direction to generate a slice piece, and lowers the dimension of the slice piece to generate the bird's-eye view data.
The bird's-eye view data is preferably a feature vector mapped to a non-sparse feature space with each slice image or slice piece as a vector.
The analysis means is preferably composed of a convolutional neural network and is learned by deep learning.

The second purpose is the image generation stage of generating a three-dimensional image based on the captured data, and the bird's-eye view data generation stage of generating the bird's-eye view data by the bird's-eye view processing based on the three-dimensional image generated in the image generation stage. It includes an analysis stage in which the depth at which an object exists is detected for each divided portion by horizontal division and depth division based on the bird's-eye view data, and the object is detected by combining the detection results, and the bird's-eye view data is generated. At the stage, multiple cross sections in the depth direction of the three-dimensional image are taken out as slice images, and each slice image is made low-dimensional to generate bird's-eye view data. This is achieved by an object detection method that detects an object by extracting cross sections at a plurality of locations as slice data.

The third purpose is an image generation procedure for generating a three-dimensional image based on captured data, a bird's-eye view data generation procedure for generating a bird's-eye view data by a bird's-eye view process based on the three-dimensional image generated by the image generation procedure, and a procedure for generating a bird's-eye view data. To detect the depth at which an object exists for each division in horizontal division and depth division based on bird's-eye view data, and to have the computer execute the processing of the analysis procedure to detect the object by combining the detection results. It is an object detection program, and in the bird's-eye view data generation procedure, cross sections of multiple points in the depth direction of the three-dimensional image are taken out as slice images, each slice image is made low-dimensional, and bird's-eye view data is generated, and in the analysis procedure. This is achieved by having a computer perform detection of an object by extracting multiple cross sections in the height direction as slice data based on bird's-eye view data.

In this way, according to the present invention, an object detection system, an object detection method, and an object detection program that can be processed quickly with a simple configuration and can detect an object such as an obstacle with higher accuracy. Can be provided.

It is a block diagram which shows the structure of one Embodiment of the object detection system by this invention. In the object detection system of FIG. 1, (A) is an image pickup screen, (B) is a three-dimensional image, and (C) is a diagram showing a depth slice. It is a schematic diagram which shows the operation principle of the autoencoder which constitutes the 3D image generation means in the object detection system of FIG. It is explanatory drawing explaining the operation at the time of (A) learning and (B) operation in the autoencoder of FIG. It is a flowchart which shows the operation of the object detection system of FIG. It is explanatory drawing explaining the operation of the 3D image generation means in the object detection system of FIG. It is a schematic diagram which shows the slice piece in the object detection system of FIG. In the object detection system of FIG. 1, (A) is a schematic diagram of object detection, and (B) is an explanatory diagram showing an actual object detection state. In the object detection system of FIG. 1, (A) is a depth slice, (B) is a low-dimensional feature vector, (C) is a set of feature vectors for each depth slice, and (D) is a combination of feature vectors. It is explanatory drawing which shows the tensor. It is a schematic diagram which sequentially shows the procedure of creating the bird's-eye view data in the object detection system of FIG. It is a schematic diagram which sequentially shows the analysis procedure of the neural network of the analysis means in the object detection system of FIG. It is a schematic diagram which shows the partial structure of the series of vectors obtained by the analysis procedure of FIG. In an experimental example at a construction site using the object detection system of FIG. 1, (A) is an image pickup screen, and (B) is a schematic diagram showing object detection. In an experimental example in an urban environment using the object detection system of FIG. 1, (A) is an image pickup screen, and (B) is a schematic diagram showing object detection.

Hereinafter, the present invention will be described in detail based on the embodiments shown in the drawings.
1 is a block diagram of an embodiment of an object detection system 10 according to the present invention, FIG. 2A is an image pickup screen, FIG. 2B is a three-dimensional image 31, and FIG. 2C is a depth slice 31a. 3 is a schematic diagram showing the operating principle of the auto encoder 70 constituting the three-dimensional image generation means 30 in the object detection system 10 of FIG. 1.
In FIG. 1, the object detection system 10 includes an image pickup means 20, a three-dimensional image generation means 30, a bird's-eye view data generation means 40, an analysis means 50, these image pickup means 20, a three-dimensional image generation means 30, and a bird's-eye view data. It is composed of a control unit 60 that controls the generation means 40 and the analysis means 50 by a program.

Here, the control unit 60 is composed of a computer, and by executing the object detection program according to the present invention installed in advance, the image pickup means 20, the three-dimensional image generation means 30, the bird's-eye view data generation means 40, and the analysis means 50 described above are executed. The object detection system 10 and the object detection method according to the present invention are realized by controlling the above.

The image pickup means 20 is arranged so as to image the front of, for example, an automobile or the like, and can sequentially image the front field as the automobile or the like travels, and has a known configuration of a monocular camera, a stereo camera, and a rider. It is either. The image pickup means 20 is controlled by the control unit 60 to perform image pickup at predetermined time intervals, and sequentially outputs an image pickup signal 21. The lidar is a sensor that is also called a laser radar, and is a sensor that performs laser image detection and ranging (Laser Imaging Detection and Ranging), and is also referred to as LIDAR. A three-dimensional rider can be used as the rider.

As shown in FIG. 2A, the three-dimensional image generation means 30 detects the depth (distance) to the object shown in the image pickup screen 21a based on each image pickup signal 21 from the image pickup means 20. As shown in FIG. 2B, the three-dimensional image 31 is generated. The three-dimensional image generation means 30 generates a three-dimensional image 31 from each image pickup signal 21 controlled by the control unit 60 and sequentially input.

When the image pickup means 20 is a monocular camera, the color image pickup screen 21a by the image pickup signal 21 is a two-dimensional image representing a plane as shown in FIG. 2A, but is disclosed in, for example, Non-Patent Document 2. It is possible to generate distance information (point cloud) at each point of the two-dimensional image by using a known method.
Then, the three-dimensional image generation means 30 projects the color intensity of the image onto the corresponding points in order to fuse the above-mentioned two-dimensional image pickup screen and the point cloud, so that the scene (the scene represented by one camera image) can be used. A three-dimensional image 31 is generated by an algorithm of "painting" each point of the point cloud with the corresponding color. The three-dimensional image generation means 30 requires a color imaging screen of a monocular camera and distance information (point cloud) of each point on the imaging screen as inputs.

Further, when the image pickup means 20 is a stereo camera, the color image pickup signal 21 includes a pair of image pickup screens from a pair of left and right cameras, so that one camera is based on the misalignment of the subject on each image pickup screen. It is possible to generate the image pickup screen of the above and the distance information (pseudo point cloud) of each point on the image pickup screen.
Here, the three-dimensional image 31 extends in the horizontal direction H, the vertical direction V, and the depth direction D as shown in FIG. 2 (B). Further, when the image pickup means 20 is a three-dimensional lidar, the three-dimensional image 31 may be acquired by, for example, a known method reported in Non-Patent Document 3. For example, a two-dimensional black-and-white image having the number of pixels (800 × 600) and depth information acquired by a three-dimensional lidar can be combined to acquire a three-dimensional image 31 sliced in the depth direction, which will be described later.

The bird's-eye view data generation means 40 performs bird's-eye view processing based on each three-dimensional image 31 from the three-dimensional image generation means 30, and generates low-dimensional bird's-eye view data 41. Here, the bird's-eye view data generation means 40 includes an autoencoder 42 composed of a convolutional neural network, and lower-dimensional bird's-eye view data 41 is obtained from each three-dimensional image 31 controlled by the control unit 60 and sequentially input. Generate.

Specifically, the bird's-eye view data generation means 40 first slices the three-dimensional image 31 at equal intervals in the depth direction as shown in FIG. 2B to obtain a plurality of depth slices 31a. As a result, each point in the point cloud is divided by distance. Here, the depth slice 31a has the same number of pixels (960 × 1280) with respect to the number of pixels (for example, 960 × 1280) in the original camera image.

Subsequently, the bird's-eye view data generation means 40 further slices each depth slice 31a in the horizontal direction with respect to each of the obtained depth slices 31a, as shown in FIG. 2C, to form a plurality of vertically long slice pieces 31b. obtain. The slice piece 31b has the number of pixels (960 × 16) by being divided into, for example, 80 pieces with respect to the depth slice 31a having the number of pixels (960 × 1280). Then, the bird's-eye view data generation means 40 inputs the slice piece 31b to the input layer of the convolutional neural network of the autoencoder described later. At that time, as shown in FIG. 10A, the slice piece 31b is further divided into a plurality of pieces in the height direction and processed. As a result, the detection accuracy of the object is improved also in the height direction.

Here, the autoencoder 70 is generally as shown schematically in FIG. The autoencoder 70 is, for example, a multi-layer neural network including an input layer 71, an intermediate layer 72, 73, 74 and an output layer 75. In FIG. 3, for convenience, each layer 71 to 75 is shown in six dimensions, four dimensions, two dimensions, four dimensions, and six dimensions, respectively. The autoencoder 70 is reduced to two dimensions by two-step encoding of the input data, then reconstructed into six dimensions by two-step decoding, and output from the output layer 75.
The auto encoder 70 compares the input data input to the input layer 71 with the reconstructed data output from the output layer 75 with respect to a large number of sample data so that the reconstructed data has the same characteristics as the input data. In addition, it is learned by deep learning. Specifically, assuming that the input data is I, in the intermediate layer 72, the function f (I) = h maps to the four-dimensional space, and in the intermediate layer 73, the function g (h) = e maps to the two-dimensional space. Then, this e becomes two-dimensional data. On the other hand, the intermediate layer 74 and the output layer 75 are mapped to the four-dimensional space and the six-dimensional space by the functions j and k, respectively, and the output layer 75 outputs the six-dimensional reconstructed data Ir. All of these mappings are non-linear. Each of the above-mentioned functions f, g, j, and k is an unknown function, and a large number of sample data are input to the input layer 71 to minimize ^{the difference (I-Ir) 2 between the input data I and the reconstructed data Ir.} By doing so, each function f to k is determined by learning by so-called deep learning.

In the object detection system 10, the bird's-eye view data is obtained by extracting the encoded data from the intermediate layers 72 to 74 by using only the encoded portion of the autoencoder 70 that has been sufficiently learned by deep learning. I'm trying to get 41.

Therefore, in the object detection system 10 shown in FIG. 4A, the autoencoder 42 encodes each slice piece 31b, which is input data, at the encoder portion 42a to generate a low-dimensional bird's-eye view data 41, and further, a decoder. The bird's-eye view data 41 is decoded in the portion 42b to generate the reconstructed data 43, and the reconstructed data is learned by deep learning so as to have the same characteristics with respect to each slice piece 31b which is the input data and the object.
After learning by deep learning as described above, in actual operation, the autoencoder 42 uses only the encoder portion 42a as shown in FIG. 4 (B), and is related to the above-mentioned one depth slice 31a. The slice pieces 31b are encoded to generate a low-dimensional bird's-eye view data 41. Each slice piece 31b is encoded in a low dimension by the autoencoder 42, and is mapped to a low-dimensional non-sparse feature space (hereinafter referred to as a non-sparse feature space) as a feature vector representing the existence of an object. At that time, since each slide piece 31b is divided in the horizontal direction, spatial information is retained with respect to the horizontal direction when mapped to the non-sparse feature space, and the original imaging screen with high accuracy is obtained by a series of feature vectors. The existence of the object in.
In this way, the autoencoder 42 generates bird's-eye view data 41 composed of a series of feature vectors for each depth slice 31a for one scene. In the bird's-eye view data 41, each depth slice 31a represents a depth position, and each feature vector represents a horizontal position. By combining these series of feature vectors, a tensor as the bird's-eye view data 41 is formed.

In the case of a black-and-white image having 1280 × 960 pixels, the matrix I _1280,960 in which each element is in the matrix ∈ [0,255] is handled. By mixing this matrix I with the depth information and _{adding the distances di and j of} each pixel in the image to each pixel, the tensor of [1280,960,2] is obtained. The depth interval _{n d} (in this _case, n d = 64) ranging from 0m to the maximum depth to define, dividing the tensor _{n d} pieces of the matrix. For each divided matrix ^ai , all elements in the matrix within the range defined by the i-th depth interval are captured. For example, if the first depth interval is 0m to 1m, then one by capturing all the intensity information of the elements in the tensor whose depth is within this range (0m to 1m) and filling the remaining space with zeros. th matrix a ⁰ in is generated. In this manner, n _d number of sparse matrix is obtained.

Each of these sparse matrices ^ai represents a depth. Additionally, a sparse matrix ^{a i,} when divided into _{n w} columns of width w, the matrix ^{a i} of size (1280,960) is divided by w = 16, 80 of the size (16,960) It becomes a matrix of pieces. These new smaller matrices bi ^{, j} represent horizontal positions. For example, matrix b ^0,0 represents the leftmost information in an image in the range 0m to 1m. Then, each matrix bi ^{, j} is acquired and encoded as described above using a pre-learned autoencoder. Then, from the equation b'i ^{, j} = g (f (bi ^{, j} )), we get what is in the smaller latent space used to encode into the vector of 500.

It should be noted here that, for the sake of clarity, only the two functions f and g are shown, but this number can increase depending on the number of hidden layer selections in the neural network, ie. It may be three or more functions. i a is the depth, these vectors b ^'i where j represents horizontal position ^{from j,} defined as the concatenation operator ∩ linking the matrix, perform the calculation of the following formula (1), further connected to the depth Then, c ^j number (in this case, 80), each representing the horizontal position matrix is obtained.

The c ^j number each matrix is input to the neural network learned by deep learning, classification is performed. This neural network learns at what depth interval an object is present or is not at all. That is, this neural network is trained by deep learning so that ^{the function h (c j} ) = μ ^{j is expressed as μ j} ∈ [0, n _d + 1] (integer). Then, when μ ^j is converted into a vector v ^j _{of length n d} + 1 with zeros at all points other than the ^{μ j} th element, the following formula (2) is obtained from the output of the neural network for each j. ) Generates the matrix M.

The matrix M of this magnitude (n _w , _nd ) has 1 for each column: horizontal position and for each row: distance at which the object is present. Such a matrix M becomes the final output of the neural network.

The analysis means 50 detects an object by extracting cross sections at a plurality of locations in the height direction as slice data based on the bird's-eye view data 41 from the bird's-eye view data generation means 40. The analysis means 50 is composed of a convolutional neural network, is controlled by the control unit 60, and detects an object on the imaging screen from the bird's-eye view data 41 sequentially input. Here, the convolutional neural network is widely used for image recognition by machine learning, and it is possible to perform image recognition with high accuracy.

Specifically, the analysis means 50 classifies where the object exists at all the depth positions for the bird's-eye view data 41. The analysis means 50 sweeps from one side (for example, the left side) to the other side (for example, the right side) in the horizontal direction with respect to the feature vector corresponding to the 80 slice pieces 31b described above, and an object for each horizontal position. Determines the depth position where is present.

Here, the criteria for determining the existence of an object by the analysis means 50 are learned in advance by deep learning, and are appropriately set according to the type of the object. As a result, the user of the object detection system 10 can define an obstacle according to the type of the object, and the object detection system 10 learns to recognize the type of the object as an obstacle. Specifically, in an urban environment including roads on which automobiles and the like travel, objects to be detected are vehicles, people, sidewalks, etc., and in construction zones where construction vehicles and workers enter and exit, objects to be detected are construction. Vehicles and workers. Judgment criteria for objects to be detected are determined in response to such various zone environments. For example, in an urban environment, objects that are closest to each slice piece 31b are positioned and marked by distance, and the distances of the marked objects are classified into levels _{from 0 to nd + 1.} This class is the target class for learning neural networks.

The object detection system 10 of the present embodiment is configured as described above, and operates as follows according to the flowchart of FIG.
That is, in step ST1, imaging is performed by a monocular camera as the imaging means 20, and in step ST2, depth evaluation for the monocular camera is performed, and as shown in step ST3, the color intensity of the monochrome image is obtained and the step. Depth is obtained at ST4. When the imaging means 20 is a stereo camera, color imaging is performed in step ST1a, depth evaluation is performed in step ST2a, and when the imaging means 20 is LIDAR, imaging is performed in step ST1b. Is done.

Subsequently, in step ST5, the color intensity value is projected onto the corresponding three-dimensional point. Then, in step ST6, the three-dimensional image 31 is sliced in the depth direction, and in step ST7, a two-dimensional depth slice 31a is obtained.
Next, in step ST8, each depth slice 31a is sliced to a predetermined width indicating the horizontal direction, and in step ST9, a slice piece 31b is obtained. After that, in step ST10, each slice piece 31b is input to the autoencoder to perform non-linear encoding to reduce the dimension. As a result, the feature vector is obtained in step ST11. Then, in step ST12, the feature vector is connected to the depth to form a two-dimensional matrix representing the horizontal direction. As a result, the feature matrix is obtained in step ST13. Finally, in step ST14, the feature matrix is input to the neural network and each feature matrix is classified. As a result, each class classified in each horizontal direction in step ST15 corresponds to a depth level indicating the depth at which the object exists.

Further, the autoencoder 42 of the bird's-eye view data generation means 40 operates as shown in the flowchart of FIG. 6 during learning and operation.
First, in step ST21, when the two-dimensional slice piece 31b is input to the first layer of the neural network in the autoencoder 42, it becomes hidden layer feature data 1 by nonlinear encoding in step ST22, and then in step ST23. When the data is input to the second layer and becomes the hidden layer feature data 2 by nonlinear encoding in step ST24, and is similarly nonlinearly encoded sequentially and input to the nth layer of the neural network in step ST26, it is input to the nth layer of the neural network in step ST27. It becomes an encoded feature vector.

The feature vector is subsequently input to the (n + 1) layer in step ST28, becomes hidden layer feature data (n + 1) by non-linear encoding in step ST29, is similarly non-linearly encoded, and in step ST30 of the neural network. When input to the second layer, it becomes a two-dimensional slice piece reconstructed by nonlinear encoding in step ST31.
Then, as shown in FIG. 7, a large number of sample data are repeatedly input so that the error between the slice piece 31b which is the input data and the reconstructed two-dimensional slice piece which is the reconstructed data is minimized by deep learning. The autoencoder is learned. Here, when the autoencoder is operating, the feature vector in step ST27 is analyzed by the analysis means 50 as shown in step ST32, and the object is detected. It should be noted that such learning by deep learning of the autoencoder is performed using, for example, sample data of several thousand or more.

In this way, in the object detection system 10, the image pickup screen shown in FIG. 2 (A) is closest to the horizontal position as schematically shown in the plan view in FIG. 8 (A) by the analysis means 50. The distance to the object will be detected. As for this detection result, when viewed from the image pickup position of the image pickup means 20, the position of the object is actually grasped with respect to the fan-shaped region as shown in FIG. 8 (B).

Next, an example of object detection using an actual image pickup screen will be described below.
A plurality of depth slices 31a are generated for one three-dimensional image 31, and each depth slice 31a (number of pixels 960 × 1280) has a plurality of slice pieces 31b (pixels) in the horizontal direction as shown in FIG. 9 (A). It is divided into the number 960 × 16). By encoding each of the slice pieces 31b, as shown in FIG. 9B, the same number of feature vectors as the slice pieces 31b can be obtained. Then, when all the depth slices 31a by the three-dimensional image 31 are encoded, as shown in FIG. 9C, a set of 80 feature vectors is obtained for each depth slice 31a. Finally, by extracting the feature vectors corresponding to each horizontal position from each depth slice 31a and combining them, a tensor composed of a series of feature vectors is obtained as shown in FIG. 9 (D).

The object detection system 10 operates as described above, but by combining the vectors output from the neural network by the analysis means 50, the distance to the nearest object is grasped for all horizontal positions, and the closest object in the scene is obtained. A specific example of detecting the position of an object will be described.

Since the bird's-eye view data 41 is a matrix containing one feature vector for every slice piece 31b for one scene, the analysis means 50 has a bird's-eye view in order to detect the depth layer in which the closest object exists. The data 41 is input to the neural network learned by deep learning and classified.

FIG. 10 sequentially shows the procedure for creating the bird's-eye view data 41 in the object detection system 10 of FIG. 1, and FIG. 11 sequentially shows the analysis procedure of the neural network of the analysis means 50 in the object detection system 10 of FIG. 12 shows a partial configuration of a series of vectors obtained by the analysis procedure of FIG.
As shown at the left end of FIG. 10A, in the above matrix, feature vectors arranged in the horizontal direction H (corresponding to a set of slice pieces 31b) are aligned along the depth direction D for each depth slice 31a. ing. Then, the analysis means 50 takes out a feature vector aligned in the depth direction D at each horizontal position as shown in FIG. 10B from each vector constituting this matrix, and as shown in FIG. 10C. By combining these with, a tensor consisting of a series of feature vectors is generated as shown in FIG. 10 (D).

Then, as shown in FIG. 11, the analysis means 50 processes this tensor to detect an object by using, for example, a five-layer neural network (see Non-Patent Document 1), for example, a convolutional neural network, preferably a perceptron. do.
In FIG. 11, assuming that the number of pixels of the image pickup screen is width w = 24, height h = 370, and minimum height h _min = 140, the first layer of the neural network receives an input image of 24 × 370 × 3. At each pixel position (stride 1), it is folded by 64 filters having a size of 11 × 5 × 3. The second layer uses 200 kernels with a size of 5x3x64. The maximum pooling layer calculates the maximum value beyond the separation region of size 8x4 for the first layer and size 4x3 for the second layer. That is, there is no overlap between the pooling regions. The fully connected hidden layers (third and fourth layers) have neurons of size 1024 and 2048, and the output layer (fifth layer) has 50 neurons.

Here, the vector output from the output layer (right end in FIG. 11) is everything except one box (the right end in FIG. 11, which is filled in black), which is the position estimated by the neural network to be the closest object. The element of is a vector V of 0. Each element of this vector V is separated in meters, and represents the distance to the object detected by the position of the black-painted portion described above. And such processing is repeated for all the matrices in the scene.

As shown in FIG. 12, the analysis means 50 grasps the distance to the nearest object for all horizontal positions by combining the vectors output from these neural networks, and determines the position of the closest object in the scene. Can be detected. As a result, the analysis means 50 selects whether or not an object exists in the scene based on the bird's-eye view data 41, and estimates the distance to the object.

As described above, according to the object detection system 10 of the present invention, the bird's-eye view data generation means 40 is a three-dimensional image with respect to the three-dimensional image 31 generated by the image generation means based on the image pickup data from the image pickup means 20. By converting each slice image of 31 into low-dimensional bird's-eye view data, the amount of data is reduced by the amount of the reduced dimension, so that the analysis means 50 can detect the object more quickly. Therefore, for example, when an object is detected in front of a three-dimensional image 31 that captures the front view of an automobile, it is possible to avoid an object such as an obstacle by sequentially detecting an object in the front view as the automobile travels. can. Further, since the object is detected based on the cross section of the three-dimensional image 31 in the depth direction, the accuracy of the detected object in the depth direction is improved, and the distance to the object can be grasped more accurately. Become.

The bird's-eye view data generation means 40 is composed of an autoencoder 42 composed of a convolutional neural network, and the autoencoder 42 reduces the dimension of each slice image to generate the bird's-eye view data 41. The convolutional neural network is a multi-layer neural network consisting of an input layer 71, at least one intermediate layer 72 to 74, and an output layer 75, and each slice image input to the input layer 71 at the time of training is selected as one of the intermediate layers. After converting to low-dimensional intermediate data in layers 72 to 74, it is decoded into reconstructed data of the same dimension as the sliced image in the output layer 75, and learned by deep learning so that the reconstructed data can reproduce the object in the sliced image. After learning, the intermediate data from the intermediate layers 72 to 74 is output to the analysis means 50 as bird's-eye view data 41.

According to the configuration of the bird's-eye view data generation means 40, the bird's-eye view data 41 can be generated more accurately by fully learning the neural network by using the neural network, so that the object can be detected more accurately. It is done with high accuracy.

The bird's-eye view data generation means 40 further slices each slice image in the horizontal direction to generate a slice piece 31b, and lowers the slice piece 31b to generate the bird's-eye view data 41. By dividing each slice image in the horizontal direction, it is possible to control the object in the horizontal direction to some extent in the subsequent lower dimension, so that it is possible to detect the object with higher accuracy in the horizontal direction and each of them. By sequentially and continuously processing the slice pieces 31b, it becomes possible to more quickly convert one three-dimensional image 31 into the bird's-eye view data 41.

The bird's-eye view data 41 is a feature vector mapped to a feature space that is not sparse (hereinafter referred to as a non-sparse feature space) with each slice image or slice piece 31b as a vector. Since the bird's-eye view data 41 is not a visible bird's-eye view image but a bird's-eye view data 41 composed of feature vectors, the conversion process to the bird's-eye view data 41 is further shortened, and the bird's-eye view data is generated in a short time.

The analysis means 50 is composed of a convolutional neural network and learns by deep learning. With sufficient learning of deep learning, the three-dimensional image 31 is converted into the bird's-eye view data 41 with higher accuracy, and the object can be detected with higher accuracy based on the bird's-eye view data 41. Hereinafter, the description will be described in more detail with reference to Examples.

The image pickup means 20 and the control unit 60 of the object detection system 10 used a computer having the following configuration.
Imaging means: Stereo camera (manufactured by ZMP Inc., model number: Robovision 2)
Control unit:
CPU: Intel (registered trademark), model number: Core (registered trademark) i7-8700
RAM (random access memory): 32GB
Storage device: 1TB
GPU: NVIDIA (registered trademark), model number: GeForce (registered trademark) RTX2070,
RAM: 8GB

FIG. 13 shows an experimental example of object detection at a construction site. As shown in FIG. 13 (A), two workers A and B are visible on the image pickup screen 21a, but the other area is the "operable area" of the construction vehicle. The number of input pixels of the color image pickup signal 21 by the stereo camera is 1280 × 960, but it was downscaled to 640 × 480 by the program. The number of frames per second for imaging was 12.5 fps. When the object was detected by the object detection system 10 on the image pickup screen 21a, the detection result shown in FIG. 13B was obtained. The number of pixels of the reconstructed two-dimensional slice piece (see step ST31 in FIG. 6) shown in FIG. 13 (B) is 80 × 60, and the calculation time for obtaining the output image in FIG. 13 (B) is 8 ms. there were.
In this detection result, the x-axis represents the horizontal position and the y-axis represents the depth, and if no object is detected, the background remains black, but the object, in this case two workers A and B, When it is detected, the part farthest from the shortest distance in the horizontal direction becomes a slightly white display, indicating that an object exists. In FIG. 13B, it can be confirmed that the two workers A and B are clearly detected and the depth is positioned according to the distance.

FIG. 14 shows an experimental example of object detection in an urban environment acquired under the same conditions as in FIG. 13 (A). Pedestrians, except for roads without obstacles for driving on urban roads. The purpose was to detect all obstacles including sidewalks, trees, vehicles, etc.
As shown in FIG. 14A, the image pickup screen 21a shows an image of the front image taken from a vehicle traveling on the road, and the front vehicle C, the sidewalk D at the left end, and the road boundary fence E on the right side are visible. .. The number of input pixels of the color image pickup signal 21 by the stereo camera is 1280 × 960, but it was downscaled to 640 × 480 by the program. The number of frames per second for imaging was 12.5 fps. When the object was detected by the object detection system 10 on the image pickup screen 21a, the detection result shown in FIG. 14B was obtained. The number of pixels of the reconstructed two-dimensional slice piece (see step ST31 in FIG. 6) shown in FIG. 14 (B) is 80 × 60, and the calculation time for obtaining the output image in FIG. 14 (B) is 8 ms. there were.
In FIG. 14B, it can be seen that the vehicle C in front, the sidewalk D at the left end, and the road boundary fence E are detected, respectively. In this case, a good IoU accuracy of about 88% was obtained as an Intersection over Union (called IoU accuracy), which is an evaluation index in object detection, by the image pickup screen 21a imaged from a moving vehicle at 12.5 fps. In addition, in order to confirm the actual distance and position to the object, as shown in FIG. 8B, a simple projection onto the three-dimensional space is required.

The present invention can be implemented in various forms without departing from the spirit of the present invention. For example, in the above-described embodiment, the image pickup means 20 uses a stereo camera, but it is also possible to obtain a three-dimensional image 31 by using, for example, a rider for forward monitoring used in an automatically driving vehicle. Further, a stereoscopic image 31 may be obtained by combining a captured image of the monocular camera and a point cloud by a conventionally known method using a monocular camera.

10: Object detection system, 20: Imaging means, 21: Imaging signal,
21a: Imaging screen, 30: 3D image generation means, 31: 3D image, 31a: Depth slice, 31b: Slice piece, 40: Bird's-eye view data generation means, 41: Bird's-eye view data, 42: Autoencoder, 42a: Encoder part ,
42b: Decoder part, 50: Analysis means, 60: Control unit, 70: Autoencoder, 71: Input layer, 72 to 74: Intermediate layer, 75: Output layer

Claims (8)

An image pickup means, an image generation means for generating a three-dimensional image based on the image pickup data acquired by the image pickup means, and a bird's-eye view data are generated by a bird's-eye view process based on the three-dimensional image generated by the image generation means. A means for generating bird's-eye view data and an analysis means for detecting an object by detecting the depth at which an object exists for each divided portion in horizontal division and depth division based on the bird's-eye view data and combining the detection results. , Including,
The bird's-eye view data generation means takes out a plurality of cross sections as slice images in the depth direction of the three-dimensional image, lowers each slice image to generate bird's-eye view data, and generates bird's-eye view data.
An object detection system in which the analysis means detects an object by extracting cross sections at a plurality of points in the height direction as slice data based on the bird's-eye view data. The bird's-eye view data generation means is composed of an autoencoder composed of a convolutional neural network.
The object detection system according to claim 1, wherein the autoencoder reduces the dimension of each slice image to generate the bird's-eye view data. The convolutional neural network is a multi-layer neural network consisting of an input layer, at least one intermediate layer, and an output layer, and each slice image input to the input layer during training is low in any of the intermediate layers. After conversion to dimensional intermediate data, the output layer decodes it into reconstructed data of the same dimension as the slice image, and the reconstructed data is learned by deep learning so that the object in the slice image can be reproduced, and after learning. Is the object detection system according to claim 2, wherein the intermediate data is output from the intermediate layer as bird's-eye view data to the analysis means. The method according to any one of claims 1 to 3, wherein the bird's-eye view data generation means further slices each slice image in the horizontal direction to generate a slice piece, and the slice piece is made low-dimensional to generate the bird's-eye view data. Object detection system. The object detection system according to any one of claims 1 to 4, wherein the bird's-eye view data is a feature vector mapped to a non-sparse feature space with each slice image or slice piece as a vector. The object detection system according to any one of claims 1 to 5, wherein the analysis means is composed of a convolutional neural network and is learned by deep learning. Based on the image generation stage that generates a three-dimensional image based on the captured data, the bird's-eye view data generation stage that generates the bird's-eye view data by the bird's-eye view processing based on the three-dimensional image generated in the image generation stage, and the bird's-eye view data. It includes an analysis stage in which the depth at which an object exists is detected for each divided portion in horizontal division and depth division, and the object is detected by combining the detection results.
At the bird's-eye view data generation stage, cross sections of a plurality of points in the depth direction of the three-dimensional image are taken out as slice images, and each slice image is made low-dimensional to generate bird's-eye view data.
An object detection method for detecting an object by extracting cross sections at a plurality of points in the height direction as slice data based on the bird's-eye view data at the analysis stage. Based on the image generation procedure that generates a three-dimensional image based on the captured data, the bird's-eye view data generation procedure that generates the bird's-eye view data by the bird's-eye view processing based on the three-dimensional image generated by the image generation procedure, and the bird's-eye view data. It is an object detection program for detecting the depth at which an object exists for each divided part in horizontal division and depth division, and letting a computer execute the processing of the analysis procedure for detecting the object by combining the detection results. hand,
In the bird's-eye view data generation procedure, cross sections of a plurality of points in the depth direction of the three-dimensional image are taken out as slice images, and each slice image is made low-dimensional to generate bird's-eye view data.
An object detection program that causes a computer to detect the direction and distance of an object by extracting a plurality of cross sections in the height direction as slice data based on the bird's-eye view data in the analysis procedure.

Images