JP2021026683A - Distance estimation apparatus - Google Patents

Abstract

To provide a distance estimation apparatus capable of improving the accuracy of estimating a distance to an object shown in images.SOLUTION: A distance estimation apparatus includes: a storage part 22 for storing, for each of plural categories of objects that are detection targets, a reference size and size distribution information that represents a degree of variation in size of the object of the category in an actual space; an object detection part 31 for detecting an object zone in which an object is represented by a series of images obtained in time series, and calculating, for each category, a degree of confidence in that the object represented in the object zone is an object of the category; a distance distribution calculation part 32 for calculating, for each image, a distribution of distances to the object represented in the image, on the basis of a size of the object zone in the image and size distribution information of each of the plural categories of objects and the confidence degree; and a distance estimation part 33 for estimating a distance to the object at a time point of obtaining each image, through prediction processing that uses a distribution of the estimated distance for each image as an observation error model for a distance to the object.SELECTED DRAWING: Figure 3

Description

The present invention relates to a distance estimation device that estimates the distance to an object represented in an image.

Research is being conducted on techniques for detecting objects represented by sensor information, such as images obtained by cameras. Such an object detection technique is used, for example, in automatic driving control of a vehicle to detect an object around the vehicle from an image obtained by a camera provided in the vehicle. In recent years, a technique for improving detection accuracy has been proposed by using a machine learning method such as a so-called deep neural network (hereinafter, simply referred to as DNN) to detect an object (for example, Non-Patent Documents 1 to 1). See 4).

In addition, in automatic driving control of a vehicle, in order to set a route on which the vehicle travels so that the vehicle does not collide with surrounding objects, it is required to accurately grasp the relative positional relationship between the vehicle and surrounding objects. Be done. However, the image obtained by one camera does not contain information about the distance to the object represented in the image. Therefore, a technique for estimating the inter-vehicle distance based on the horizontal or vertical edge width of the vehicle in front shown in the image and the estimated horizontal or vertical width has been proposed (for example, Patent Document 1). reference).

JP-A-2002-327635

Wei Liu et al., "SSD: Single Shot MultiBox Detector", ECCV2016, 2016 Shaoqing Ren et al., "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", NIPS, 2015 Alex Kendall et al., "Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics", CVPR2018, 2018 Anna Petrovskaya et al., "Model Based Vehicle Tracking for Autonomous Driving in Urban Environments", Robotics: Science and Systems 2008, 2008

However, the greater the error in the actual width of the object to be estimated in real space, the greater the error in the estimated distance. For example, when the object detected from the image is a vehicle, in the above technique, the width of the reference vehicle in the real space is assumed, and the inter-vehicle distance is estimated using the assumed width. However, some vehicles, such as buses or trucks, have a relatively large width, while others, such as small passenger cars, have a relatively small width. Therefore, depending on the type of vehicle detected, the difference between the assumed width of the vehicle and the width of the actual vehicle becomes large, and as a result, the error of the estimated inter-vehicle distance may also become large.

Therefore, an object of the present invention is to provide a distance estimation device capable of improving the estimation accuracy of the distance to the object represented by the image.

According to one embodiment, a distance estimation device is provided. This distance estimation device stores size distribution information indicating the reference size of an object of that type in the real space and the degree of variation in the size of the object of that type in the real space for each of a plurality of types of objects to be detected. For each of the storage unit and a series of images obtained in time series, the object area in which the object is represented is detected from the image, and for each of the plurality of types of objects, the object represented in the object area is the type. For each of the object detection unit that calculates the certainty of the object and the image in which the object is detected in the series of images, the size of the object area on the image and the size distribution of each of the plurality of types of the object. A distance distribution calculation unit that obtains the distribution of the estimated distance to the object represented in the image based on information and certainty, and the distribution of the estimated distance for each of the images in the series of images in which the object was detected. It has a distance estimation unit that estimates the distance to the object at each acquisition time of the image in which the object is detected by the prediction processing using the above as an observation error model of the distance to the object.

The distance estimation device according to the present invention has the effect of improving the estimation accuracy of the distance to the object shown in the image.

It is a schematic block diagram of the vehicle control system in which the distance estimation device is mounted. It is a hardware block diagram of the electronic control apparatus which is one Embodiment of a distance estimation apparatus. It is a functional block diagram of the processor of the electronic control device concerning the vehicle control processing including the distance estimation processing. It is a figure which shows an example of the structure of the DNN used as a classifier. It is a figure which shows an example of the management table. It is an operation flowchart of the vehicle control processing including the distance estimation processing.

Hereinafter, the distance estimation device will be described with reference to the drawings. This distance estimation device estimates the distance to an object being tracked for an object detected and tracked from a series of images obtained in time series. At that time, this distance estimation device obtains the certainty of each type of object obtained by inputting the image into the classifier for each image obtained by the imaging unit during the tracking period, and the object on the image. Based on the size and the size distribution information that represents the reference size and the degree of variation in size as seen from the imaging unit in real space for each type of object, the statistical representative value of the estimated value of the distance to the object and the degree of variation Find the distribution of the estimated distance including. Then, this distance estimation device estimates the distance to the object at the time of acquiring each image during tracking by performing prediction processing using the distribution as a model of observation error.

In the following, an example in which the distance estimation device is applied to the vehicle control system will be described. In this example, the distance estimation device detects and detects an object to be detected around the vehicle by executing a distance estimation process on an image obtained by a camera mounted on the vehicle. The distance to the detected object is estimated, and the detection result of the object and the estimated value of the distance to the detected object are used for the operation control of the vehicle. In the following embodiment, the distance from the camera mounted on the vehicle to the object is defined as the distance from the vehicle to the object.

FIG. 1 is a schematic configuration diagram of a vehicle control system in which a distance estimation device is mounted. Further, FIG. 2 is a hardware configuration diagram of an electronic control device which is an example of a distance estimation device. In the present embodiment, the vehicle control system 1 mounted on the vehicle 10 and controlling the vehicle 10 includes a camera 2 for photographing the surroundings of the vehicle 10 and an electronic control device (ECU) which is an example of a distance estimation device. Has 3 and. The camera 2 and the ECU 3 are communicably connected to each other via an in-vehicle network 4 that conforms to a standard such as a controller area network.

The camera 2 is an example of an imaging unit, and is a two-dimensional detector composed of an array of photoelectric conversion elements having sensitivity to visible light, such as a CCD or C-MOS, and an object to be photographed on the two-dimensional detector. It has an imaging optical system that forms an image of a region. Then, the camera 2 is attached to, for example, the interior of the vehicle 10 so as to face the front of the vehicle 10. Then, the camera 2 photographs the front region of the vehicle 10 at predetermined imaging cycles (for example, 1/30 second to 1/10 second), and generates an image in which the front region is captured. The image obtained by the camera 2 may be a color image or a gray image. The vehicle 10 may be provided with a plurality of cameras having different shooting directions or focal lengths.

Each time the camera 2 generates an image, the camera 2 outputs the generated image and the shooting time (that is, the image acquisition time) to the ECU 3 via the in-vehicle network 4.

The ECU 3 controls the vehicle 10. In the present embodiment, the ECU 3 controls the vehicle 10 so as to automatically drive the vehicle 10 based on an object detected from a series of time-series images obtained by the camera 2. Therefore, the ECU 3 has a communication interface 21, a memory 22, and a processor 23.

The communication interface 21 is an example of a communication unit, and has an interface circuit for connecting the ECU 3 to the in-vehicle network 4. That is, the communication interface 21 is connected to the camera 2 via the in-vehicle network 4. Then, each time the communication interface 21 receives an image from the camera 2, the received image is passed to the processor 23.

The memory 22 is an example of a storage unit, and includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. Note that the memory 22 may have a dedicated memory circuit for each arithmetic unit when the processor 23 has a plurality of arithmetic units as described later. The memory 22 is an internal parameter that represents information about the camera 2, such as various data used in the distance estimation process executed by the processor 23 of the ECU 3, such as the focal length of the camera 2, and a classifier used in the distance estimation process. Various parameters for specifying the above, and a management table which is an example of size distribution information are stored. Further, the memory 22 stores the image received from the camera 2 together with the shooting time for a certain period of time. Furthermore, the memory 22 is generated during the distance estimation process, such as information representing the object area in which the object is represented and a list of detected objects in each image obtained during the period in which the detected object is being tracked. Various data to be stored is stored for a certain period of time. Furthermore, the memory 22 may store information used for traveling control of the vehicle 10, such as map information.

The processor 23 is an example of a control unit, and includes one or a plurality of CPUs (Central Processing Units) and peripheral circuits thereof. The processor 23 may further include other arithmetic circuits such as a logical operation unit, a numerical operation unit, or a graphic processing unit. Then, each time the processor 23 receives an image from the camera 2 while the vehicle 10 is traveling, the processor 23 executes a vehicle control process including a distance estimation process for the received image. Then, the processor 23 controls the vehicle 10 so as to automatically drive the vehicle 10 based on the detected estimated distance to an object around the vehicle 10.

FIG. 3 is a functional block diagram of the processor 23 of the ECU 3 regarding vehicle control processing including distance estimation processing. The processor 23 includes an object detection unit 31, a distance distribution calculation unit 32, a distance estimation unit 33, a driving planning unit 34, and a vehicle control unit 35. Each of these parts of the processor 23 is, for example, a functional module realized by a computer program running on the processor 23. Alternatively, each of these parts included in the processor 23 may be a dedicated arithmetic circuit provided in the processor 23. Further, among these units of the processor 23, the object detection unit 31, the distance distribution calculation unit 32, and the distance estimation unit 33 execute the distance estimation process. When a plurality of cameras are provided in the vehicle 10, the processor 23 may execute the distance estimation process for each camera based on the image obtained by the cameras.

Each time the object detection unit 31 receives an image from the camera 2, the object detection unit 31 inputs the latest received image into the classifier, and for each of the plurality of areas on the image, for each type of object to be detected. , Find the certainty that represents the certainty that an object of that kind is represented in that area. Then, the object detection unit 31 determines that the object of that type is represented in the region where the certainty of any type is equal to or higher than the certainty threshold.

In the present embodiment, the object detection unit 31 uses a DNN that has been learned in advance as a discriminator so as to obtain the certainty for each type of the object to be detected for each of the plurality of regions on the image. The DNN used by the object detection unit 31 is, for example, a Single Shot MultiBox Detector (SSD) described in Non-Patent Document 1 or a convolutional network having the same structure as the Faster R-CNN described in Non-Patent Document 2. It can be a tional neural network (CNN).

FIG. 4 is a diagram showing an example of the configuration of a DNN used as a discriminator. The DNN 400 has a main trunk unit 401 provided on the input side where an image is input, a position detection unit 402 and a type estimation unit 403 provided on the output side of the main trunk unit 401. The position detection unit 402 outputs an circumscribed rectangle of the region in which the object to be detected is represented on the image, based on the output from the main trunk unit 401. The type estimation unit 403 calculates the certainty level for each type of the object represented in the region detected by the position detection unit 402 based on the output from the main trunk unit 401. The position detection unit 402 and the type estimation unit 403 may be integrally formed.

The main trunk unit 401 can be, for example, a CNN having a plurality of layers connected in series from the input side to the output side. The plurality of layers include two or more convolutional layers. Further, the plurality of layers included in the main trunk portion 401 may include a pooling layer provided for each one or a plurality of convolution layers. Furthermore, the plurality of layers included in the main trunk portion 401 may include one or more fully connected layers. For example, the main executive 401 may have the same configuration as the base layer of the SSD. In this case, the main executive 401 performs two convolution layers → Max Pooling (that is, outputs the maximum value out of n × n inputs) in order from the input side, as in the case of VGG-16. (Hereinafter, simply referred to as pooling layer) → 2 layers of convolution layer → pooling layer → 3 layers of convolution layer → pooling layer → 3 layers of convolution layer → pooling layer → 3 layers of convolution layer → pooling layer → full bond of 3 layers It may be composed of layers. Alternatively, the cadre 401 may be configured according to other CNN architectures such as VGG-19, AlexNet or Network-In-Network.

When an image is input, the main executive 401 outputs a feature map calculated from the image by executing operations on the image in each layer.

The feature map output from the main executive unit 401 is input to the position detection unit 402 and the type estimation unit 403, respectively. The position detection unit 402 and the type estimation unit 403 can each be, for example, a CNN having a plurality of layers connected in series from the input side to the output side. For each of the position detection unit 402 and the type estimation unit 403, the plurality of layers included in the CNN include two or more convolution layers. Further, for each of the position detection unit 402 and the type estimation unit 403, the plurality of layers included in the CNN may include a pooling layer provided for each one or a plurality of convolution layers. The convolutional layer and pooling layer of the CNN may be shared with respect to the position detection unit 402 and the type estimation unit 403. Further, for each of the position detection unit 402 and the type estimation unit 403, the plurality of layers may include one or more fully connected layers. In this case, it is preferable that the fully connected layer is provided on the output side of each convolutional layer. Further, the output from each convolutional layer may be directly input to the fully connected layer. Further, the output layer of the type estimation unit 403 may be a softmax layer that calculates the certainty of each type of the object to be detected according to the softmax function, or each type of the object to be detected according to the sigmoid function. It may be a sigmoid layer for calculating the certainty of.

The position detection unit 402 and the type estimation unit 403 output, for example, the certainty of each type of the object to be detected for each region of various positions, various sizes, and various aspect ratios on the image. Be learned. Therefore, when the image is input, the classifier 400 outputs the certainty of each type of the object to be detected for each region of various positions, various sizes, and various aspect ratios on the image. ..

The image (teacher image) included in the teacher data used for learning the classifier 400 includes, for example, the type of object to be detected (for example, a car, a person, a road sign, a traffic light, another object on the road, etc.). ) And the circumscribing rectangle of the object, which represents the area in which the object to be detected is represented, are tagged.

The classifier 400 is trained using a large number of teacher images as described above according to a learning method such as an error back propagation method. By using the classifier 400 learned in this way, the processor 23 can accurately detect the object to be detected from the image.

The object detection unit 31 compares the certainty of each of the types of objects calculated for each of the plurality of regions for which the certainty has been calculated with the certainty threshold set for the type. Then, the object detection unit 31 represents an object in which the object to be detected is represented in a region in which the certainty of any of the object types is equal to or higher than the certainty threshold applied to the type among the plurality of regions. Detect as an area. Further, the object detection unit 31 estimates the type of the object corresponding to the certainty level equal to or higher than the certainty degree threshold value in the object area as the type of the object represented in the object area.

Alternatively, the object detection unit 31 may use a classifier other than the DNN as the classifier. For example, the object detection unit 31 inputs a feature amount (for example, Histograms of Oriented Gradients, HOG) calculated from a window set on the image as a classifier, and inputs the type of object to be detected in the window. For each, a pre-trained support vector machine (SVM) may be used to output the certainty that the object of that type is represented. The object detection unit 31 calculates a feature amount from the window while variously changing the position, size, and aspect ratio of the window set on the image, and inputs the calculated feature amount to the SVM to obtain the window. Find the certainty of each type. Then, when the certainty of any kind is equal to or higher than the certainty threshold value, the object detection unit 31 sets the window as the object area in which the object of the kind is represented.

According to the modification, even if the certainty of any type is less than the certainty threshold, the object detection unit 31 of each type belonging to the class for any of the classes including two or more types of objects. A region in which the sum of certainty is equal to or greater than the certainty threshold for the class may be detected as an object region in which an object of the class is represented. The class is a higher category of the object type and is set in advance. The class may also be set to include a combination of object types to which Non-Maximum Suppression is applied. For example, if the class "vehicle" includes a type "ordinary passenger car" and a type "large vehicle", and the certainty threshold is set to 0.7 for each type and class. To do. Then, it is assumed that the confidence levels calculated for "ordinary passenger cars" and "large vehicles" are 0, 5, 0.3 for a certain area. In this case, since the certainty of both the "ordinary passenger car" and the "large vehicle" is less than the certainty threshold, the types of objects represented in the area are "ordinary passenger car" and "large vehicle". Neither is judged. However, the sum (0.8) of the certainty (0.5) calculated for the "ordinary passenger car" and the certainty (0.3) calculated for the "large vehicle" is equal to or higher than the certainty threshold. Therefore, it is determined that an object of the class "vehicle" is represented in that area.

As a result, the object detection unit 31 can detect the object represented in the region even when it is difficult to distinguish the object represented in the region from any of a plurality of types belonging to the predetermined class.

In this case, for each type of object, a class table representing the class to which the type belongs is stored in the memory 22 in advance. Then, the object detection unit 31 may refer to the class table and specify the type of the object belonging to the class for each class.

The object detection unit 31 includes information representing the position and range of each object area on the image (for example, the coordinates of the upper left end and the lower right end of the object area), the type of the object estimated for the object area, and the type thereof. The certainty calculated for each type of the class to which the belongs is registered in the detected object list, which is a list of detected objects. Then, the object detection unit 31 stores the detected object list in the memory 22.

The distance distribution calculation unit 32 calculates an estimated distance distribution representing the degree of variation in the estimated distance of the object represented in the object region for each of the object regions detected from the image. In the present embodiment, the distance distribution calculation unit 32 is represented in the object region for each of a plurality of types of objects to be detected based on the size of the object region, the size distribution information, and the internal parameters of the camera 2. The normal distribution representing the distribution of the estimated distance when the object is that type of object is obtained, and the estimated distance distribution is obtained by weighting the obtained normal distribution of each type with the certainty of that type. Since the distance distribution calculation unit 32 may execute the same processing for each object region, the processing for one object region will be described below.

As an estimated distance distribution, the distance distribution calculation unit 32 is obtained by referring to the management table for each type of class to which the type of the object represented in the object area belongs, and is representative of the type obtained in advance in the real space. A mixed normal distribution is calculated using the degree of variation in size and size and the certainty calculated for the type. Further, the distance distribution calculation unit 32 approximates the obtained mixed normal distribution with one normal distribution so that the estimated distance distribution can be applied to the prediction filter described later.

FIG. 5 is a diagram showing an example of a management table. In the management table 500, row by row, from left to right, the type of object, the typical horizontal size of the type in real space as seen from the camera 2, the standard deviation of the horizontal size, and the type. The typical vertical size in real space, the standard deviation of the vertical size, and the weighting coefficient of the horizontal size are shown. Hereinafter, the horizontal size of the object in the real space seen from the camera 2 is referred to as an actual width, and a typical actual width is referred to as a reference actual width. Similarly, the vertical size of an object in the real space seen from the camera 2 is called the actual height, and the representative actual height is called the reference actual height. The standard deviation of the actual width represents the degree of variation in the horizontal size of the type of object. Similarly, the standard deviation of the actual height represents the degree of variation in the vertical size of that type of object. The width weighting coefficient Dw represents the reliability of the distance to the object estimated based on the actual width, and is set to a value of 0 or more and 1 or less, for example. The actual height weighting coefficient for the same object type is represented by, for example, (1-Dw). Therefore, the higher the reliability of the distance to the object estimated based on the actual width is relative to the reliability of the distance to the object estimated based on the actual height, the higher the weighting coefficient Dw of the actual width is. Set to a large value. This is due to the following reasons. Depending on the type of object, the degree of variation in either the actual width or the actual height may be lower than the degree of variation in the other. For example, when the type of the object is "pedestrian", the actual width greatly differs depending on the direction of the pedestrian as seen from the camera 2, but the actual height does not change much. Therefore, the degree of variation in actual height is smaller than the degree of variation in actual width. Therefore, for "pedestrians", the distance estimated based on the actual height is more reliable than the distance estimated based on the actual width. Therefore, by setting such a weighting coefficient, the distribution of the estimated distance can be expressed more accurately.

ここで、パラメータhorizontal_fov_in_radは、カメラ２の水平方向画角（単位：ラジアン）であり、パラメータvertical_fov_in_radは、カメラ２の垂直方向画角（単位：ラジアン）である。また、パラメータhnは、画像の水平方向画素数であり、パラメータvnは、画像の垂直方向画素数である。さらに、パラメータwidthは、物体領域の水平方向サイズ（単位：画素数）であり、パラメータheightは、物体領域の垂直方向サイズ（単位：画素数）である。さらに、パラメータw_X、σ_w,Xは、それぞれ、管理テーブルに表された、物体の種類Xについての基準実幅及び実幅の標準偏差である。さらに、パラメータh_X、σ_h,Xは、それぞれ、管理テーブルに表された、物体の種類Xについての基準実高及び実高の標準偏差である。 The distance distribution calculation unit 32 estimates the distance from the vehicle 10 to the object of that type (hereinafter, width estimation) according to the following equation for each of the class types to which the type of the object determined to be represented in the object area belongs. Calculate the mean distance_w of (called the distance) and the standard deviation σ _distancew of the estimated width distance. Similarly, the distance distribution calculation unit 32 determines the estimated distance from the vehicle 10 to the object of that type (hereinafter, for each of the class types to which the type of the object determined to be represented in the object area belongs, according to the following equation. , Called the height estimation distance) Find the mean distance_h and the standard deviation σ _distance h of the height estimation distance.

Here, the parameter horizontal_fov_in_rad is the horizontal angle of view (unit: radian) of the camera 2, and the parameter vertical_fov_in_rad is the vertical angle of view (unit: radian) of the camera 2. Further, the parameter hn is the number of pixels in the horizontal direction of the image, and the parameter vn is the number of pixels in the vertical direction of the image. Further, the parameter width is the horizontal size of the object area (unit: number of pixels), and the parameter height is the vertical size of the object area (unit: number of pixels). Further, the parameters w _X , σ _{w, and X} are the reference actual width and the standard deviation of the actual width for the object type X, respectively, represented in the management table. Further, the parameters h _X , σ _{h, and X} are the reference actual height and the standard deviation of the actual height for the object type X, respectively, represented in the management table.

ここで、Mは、物体領域に表された物体の種類が属するクラスに含まれる種類の数である。また、係数aj(j=1,2,..,M)は、物体領域に表された物体の種類が属するクラスに含まれる種類jについて求められた確信度である。さらに、Dwjは、管理テーブルに表された、種類jについての基準実幅に対する重み係数である。さらに、Nwjは、種類jについての正規分布Nwであり、Nhjは、種類jについての正規分布Nhである。 As a result, for each type of object, the normal distribution N (distance_w, σ _distancew ² _{) defined by the mean value distance_w of the estimated width distance and the square of the standard deviation σ distancew} of the estimated width distance (hereinafter, simply referred to as Nw). ) And the normal distribution N (distance_h, σ _distanceh ² ) (hereinafter, simply referred to as Nh) defined by the mean value distance_h of the estimated height distance and _{the square of the standard deviation σ distanceh of the estimated height distance.} Be done. Therefore, the distance distribution calculation unit 32 can obtain a mixed normal distribution GMM representing the distribution of the estimated distance to the object represented in the object region according to the following equation.

Here, M is the number of types included in the class to which the type of the object represented in the object area belongs. The coefficient aj (j = 1,2, .., M) is the degree of certainty obtained for the type j included in the class to which the type of the object represented in the object area belongs. Furthermore, Dwj is a weighting factor for the reference actual width for type j represented in the management table. Further, Nwj is the normal distribution Nw for the type j, and Nhj is the normal distribution Nh for the type j.

In this way, the certainty of each type of object that may be represented in the object area is weighted by the distribution of the estimated value of the distance obtained from the reference value of the actual width or height and the degree of variation for each type. By doing so, the distance distribution calculation unit 32 can accurately evaluate the degree of variation in the distance to the object represented in the object region. Further, the distance distribution calculation unit 32 weights the distribution of the estimated distance based on the actual width and the distribution of the estimated distance based on the actual height for each type of the object by the reliability of the actual width and the actual width. , The degree of variation in the distance to the object represented in the object area can be evaluated more accurately.

ここで、関数STDw()は、幅推定距離とその標準偏差の関係を表す関数であり、実際に画像に表されている物体の幅に対する物体領域の幅のバラツキ度合いに応じて予め求められる。同様に、関数STDh()は、高さ推定距離とその標準偏差の関係を表す関数であり、実際に画像に表されている物体の高さに対する物体領域の高さのバラツキ度合いに応じて予め求められる。関数STDw()及び関数STDh()は、例えば、メモリ２２に記憶される。なお、（３）式に従って幅推定距離の標準偏差σ_distancew、及び、高さ推定距離の標準偏差σ_distancehが算出される場合には、管理テーブルにおいて、基準実幅の標準偏差及び基準実高の標準偏差は省略されてもよい。 If the degree of variation in the width of the object region with respect to the actual width of the object on the image cannot be ignored, the distance distribution calculation unit 32 considers the degree of variation and sets the standard deviation σ _{distance w} of the width estimation distance. You may ask. Similarly, when the degree of variation in the height of the object region with respect to the actual height of the object on the image cannot be ignored, the distance distribution calculation unit 32 considers the degree of variation and standardizes the estimated height distance. The deviation σ _{distance h} may be obtained. In this case, the distance distribution calculation unit 32, for example, according to the following equation, the standard deviation sigma _Distancew width estimated distance, and may calculate the standard deviation sigma _DistanceH height estimated distance.

Here, the function STDw () is a function representing the relationship between the estimated width distance and its standard deviation, and is obtained in advance according to the degree of variation in the width of the object region with respect to the width of the object actually represented in the image. Similarly, the function STDh () is a function that expresses the relationship between the estimated height distance and its standard deviation, and is in advance according to the degree of variation in the height of the object region with respect to the height of the object actually shown in the image. Desired. The function STDw () and the function STDh () are stored in, for example, the memory 22. _{If the standard deviation σ distancew of} the estimated width distance and the standard deviation σ distanceh of the estimated height _distance are calculated according to Eq. (3), the standard deviation of the standard actual width and the standard actual height of the reference actual width are calculated in the management table. The standard deviation may be omitted.

The distance distribution calculation unit 32 calculates the mean value and standard deviation of the one normal distribution by approximating the mixed normal distribution representing the distribution of the estimated distance to the object represented in the object region with one normal distribution. .. Then, the distance distribution calculation unit 32 _{stores in the memory 22 the mean value d final} and the standard deviation σ _final of one normal distribution calculated for each object area as the distribution of the estimated distance in association with the object area. ..

The distance estimation unit 33 obtains an estimated distance from the vehicle 10 to the object represented in the object region for each of the object regions detected from the image. Since the distance estimation unit 33 may execute the same processing for each object region, the processing for one object region will be described below.

In the present embodiment, the distance estimation unit 33 sets the object area of interest detected from the latest image by associating the object represented in the object area with the object detected from the past image. Track the represented object. Then, the distance estimation unit 33 determines the distribution of the estimated distance obtained for the object region in which the object is represented in the individual images obtained during the period in which the object is being tracked (hereinafter, simply referred to as the tracking period). , Kalman Filter, which is used as a model of the observation error of the estimated distance, and according to the prediction processing by the prediction filter, the distance to the object at the time of each image acquisition during the tracking period is estimated by the minimum mean square error (MMSE). To do.

The distance estimation unit 33 applies tracking processing based on an optical flow, such as the Lucas-Kanade method, to the object region of interest in the latest image and the object region in the past image, thereby applying the tracking process to the object region. Track the represented object retroactively. Therefore, the distance estimation unit 33 extracts a plurality of feature points from the object region of interest by applying a filter for feature point extraction such as SIFT or Harris operator to the object region of interest, for example. Then, the distance estimation unit 33 may calculate the optical flow by specifying the corresponding points in the object region in the past image for each of the plurality of feature points according to the applied tracking method. Alternatively, the distance estimation unit 33 applies another tracking method applied to tracking a moving object detected from the image to the object region of interest in the latest image and the object region in the past image. , The object represented in the object area may be tracked.

When the object represented in the object area of interest in the latest image is associated with the object detected from the past image, the distance estimation unit 33 acquires each image during the tracking period according to the method of the applied prediction filter. Estimate the minimum mean square error of the distance to the object at time. For example, when the Kalman Filter is applied as a prediction filter, the distance estimation unit 33 performs a measurement update that updates the observed value of the estimated distance for each image during the tracking period, and the next time (in this example, the next image). A process update that updates the predicted value of the estimated distance at the time of acquisition) is executed using the distribution of the estimated distance obtained for the object region in which the object is represented and Process noise. For example, the distance estimation unit 33 uses a linear Kalman Filter or an Extended Kalman Filter to covariance the observed residuals, with _{the mean value d final} of the estimated distance distribution as the observed value and the standard deviation σ _{final of the estimated distance distribution as the observation error.} Perform a measurement update by finding the Innovation covariance) and Kalman Gain. Further, the distance estimation unit 33 may perform a process update by using a linear Kalman Filter or an Extended Kalman Filter. Further, the Process noise represents the uncertainty of the behavior of the object or the behavior of the vehicle 10 itself. For example, it is represented by a normal distribution and is stored in advance in the memory 22 together with the model matrix of the Process noise.

According to the modification, the distance estimation unit 33 may execute the measurement update by obtaining the covariance of the observation residual and the Kalman Gain by the Unscented Kalman Filter. In this case, the mixed normal distribution itself representing the distribution of the estimated distance obtained for the object region in which the object is represented for each image during the tracking period may be used as a model of the observation error of the estimated distance. _{In this case, the distance estimation unit 33 determines a plurality of sigma points around the mean value d final} based on the mixed normal distribution representing the distribution of the estimated distance, and the observation residual is based on the plurality of sigma points. The covariance and Kalman Gain may be calculated. Further, in this case, the distance estimation unit 33 may perform a process update by the Unscented Kalman Filter.

Further, in the prediction process using the Kalman Filter, the state vector representing the position of the object being tracked at the time of acquiring each image during the tracking period is, for example, the estimated distance from the vehicle 10 to the object and the time of the estimated distance. It is represented by a two-dimensional vector whose elements are the relative velocities that are the differential values. Alternatively, the state vector is represented by the camera coordinate system, which is a two-dimensional Cartesian coordinate system set on a plane parallel to the road surface with respect to the camera 2, and the position of the object (that is, the object with respect to the vehicle 10). It may be represented by a four-dimensional vector whose elements are relative position coordinates) (x, y) and its time derivative (vx, vy). Further, the state vector is the estimated distance d from the vehicle 10 to the object and the azimuth angle φ from the camera 2 to the object (that is, the angle in the direction from the camera 2 to the object with respect to the optical axis direction of the camera 2). And, it may be represented by a four-dimensional vector having each of the time derivative value of the estimated distance and the time derivative value of the azimuth as elements. The position of the object region on the image, for example, the position of the center of gravity of the object region represents the azimuth angle φ from the camera 2 to the object. Therefore, when the azimuth angle φ is included as an element of the state vector, the distance estimation unit 33 can obtain the azimuth angle φ based on the position of the center of gravity of the object region. Further, when a position in the camera coordinate system is included as an element of the state vector, the distance estimation unit 33 can calculate the position from the estimated distance d and the azimuth angle φ.

As described above, the distance estimation unit 33 accurately estimates the distance to the object represented in the object area at the time of acquiring each image by repeating the measurement update and the process update for each image during the tracking period. it can.

The distance estimation unit 33 notifies the operation planning unit 34 of the state vector at the time of acquiring each image during the tracking period for the object represented in the object area for each object area of the latest image.

The operation planning unit 34 refers to the detected object list obtained for each image and the state vector for the object being tracked so that the object existing around the vehicle 10 and the vehicle 10 do not collide with each other. Generate one or more planned travel routes (trajectory). The planned travel route is represented as, for example, a set of target positions of the vehicle 10 at each time from the current time to a predetermined time ahead. For example, the driving planning unit 34 executes the viewpoint conversion process using information such as the mounting position of the camera 2 on the vehicle 10, so that the coordinates in the image of the object area in the object detection list can be changed to the coordinates on the bird's-eye view image ( Convert to bird's-eye coordinates). At that time, the driving planning unit 34 may specify the position of the object on the bird's-eye view image by referring to the state vector and using the estimated distance from the vehicle 10 to the object represented in the object region. .. Then, the operation planning unit 34 tracks the objects registered in the detected object list by executing the tracking process using the Kalman Filter or the Particle filter for a series of bird's-eye view coordinates, and obtains the tracking result. The predicted trajectory of the object up to a predetermined time ahead is estimated from the trajectory. Based on the predicted trajectory of each object being tracked and the position, speed, and posture of the vehicle 10, the driving planning unit 34 determines the distance between each of the objects being tracked up to a predetermined time and the vehicle 10 for any object. The planned travel route of the vehicle 10 is generated so that the predicted value of is equal to or greater than a predetermined distance. The operation planning unit 34 determines the position, speed, and posture of the vehicle 10 based on, for example, the current position information representing the current position of the vehicle 10 obtained from a GPS receiver (not shown) mounted on the vehicle 10. Can be estimated. Alternatively, the localization processing unit (not shown) detects the left and right lane marking lines of the vehicle 10 from the image each time an image is obtained by the camera 2, and stores the detected lane marking lines and the memory 22. The position, speed, and attitude of the vehicle 10 may be estimated by matching with the existing map information. Further, the driving planning unit 34 may confirm the number of lanes in which the vehicle 10 can travel by referring to, for example, the current position information of the vehicle 10 and the map information stored in the memory 22. Then, when there are a plurality of lanes in which the vehicle 10 can travel, the driving planning unit 34 may generate a planned travel route so as to change the lane in which the vehicle 10 travels.
The operation planning unit 34 may generate a plurality of planned travel routes. In this case, the driving planning unit 34 may select the route that minimizes the sum of the absolute values of the accelerations of the vehicle 10 from the plurality of planned traveling routes.

The operation planning unit 34 notifies the vehicle control unit 35 of the generated planned travel route.

The vehicle control unit 35 controls each part of the vehicle 10 so that the vehicle 10 travels along the notified travel schedule route. For example, the vehicle control unit 35 obtains the acceleration of the vehicle 10 according to the notified planned travel route and the current vehicle speed of the vehicle 10 measured by the vehicle speed sensor (not shown), and accelerates the accelerator so as to be the acceleration. Set the opening or brake amount. Then, the vehicle control unit 35 obtains the fuel injection amount according to the set accelerator opening degree, and outputs a control signal corresponding to the fuel injection amount to the fuel injection device of the engine of the vehicle 10. Alternatively, the vehicle control unit 35 outputs a control signal according to the set brake amount to the brake of the vehicle 10.

Further, when the vehicle 10 changes the course of the vehicle 10 in order to travel along the planned travel route, the vehicle control unit 35 obtains the steering angle of the vehicle 10 according to the planned travel route, and responds to the steering angle. The control signal is output to an actuator (not shown) that controls the steering wheels of the vehicle 10.

FIG. 6 is an operation flowchart of the vehicle control process including the distance estimation process executed by the processor 23. Each time an image is received from the camera 2, the processor 23 executes the vehicle control process according to the operation flowchart shown in FIG. In the operation flowchart shown below, the processes of steps S101 to S105 correspond to the distance estimation process.

The object detection unit 31 of the processor 23 inputs the latest image obtained from the camera 2 into the classifier, and is convinced of various areas on the image for each type of object to be detected represented in the area. Find the degree (step S101). Then, the object detection unit 31 is the type of the object region in the latest image in which the certainty of any of the object types is equal to or higher than the certainty threshold for the type among the plurality of regions for which the certainty is calculated. It is determined that the object of (step S102) has been detected. As described above, the object detection unit 31 determines that an object of that class has been detected in an object region where the sum of the certainty of each type belonging to that class is equal to or greater than the certainty threshold. You may. Then, the object detection unit 31 registers the detected object in the detection object list.

The distance distribution calculation unit 32 of the processor 23 calculates the estimated distance distribution representing the degree of variation in the estimated distance from the vehicle 10 to the object represented in the object region for each of the object regions on the latest image, and determines the size of the object region. , Calculation based on the internal parameters of the camera 2, the size distribution information for each type of object, and the certainty (step S103).

For each of the object areas on the latest image, the distance estimation unit 33 of the processor 23 traces the object represented in the object area retroactively, and during the tracking period and the tracking period in which the object is tracked. The corresponding object region in each image is specified (step S104). Then, the distance estimation unit 33 determines the distance from the vehicle 10 to the object at the time of acquiring each image during the tracking period of the object represented by the object region for each of the object regions on the latest image. The minimum mean squared error is estimated by the prediction process using the estimated distance distribution for the corresponding object region in each image of (step S105) as a model of the observation error.

The operation planning unit 34 of the processor 23 makes the predicted locus of the object estimated based on the estimated distance to the object at the time of each image acquisition during the tracking period and a predetermined distance or more for each object being tracked. As described above, the planned travel route of the vehicle 10 is generated (step S106). Then, the vehicle control unit 35 of the processor 23 controls the vehicle 10 so that the vehicle 10 travels along the planned travel route (step S107). Then, the processor 23 ends the vehicle control process.

As described above, this distance estimation device estimates the distance to the object being tracked for the object detected and tracked using the classifier from a series of images obtained in time series. .. At that time, this distance estimation device determines the certainty of each type of object obtained by inputting the image into the classifier for each image obtained during the tracking period, and the size of the object on the image. Based on the size distribution information for each type of object, the distribution of the estimated distance including the statistical representative value of the estimated value of the distance to the object and the degree of variation is obtained. Then, this distance estimation device estimates the minimum mean square error of the distance to the object at the time of acquiring each image during tracking by performing prediction processing using the distribution as a model of the observation error. Therefore, this distance estimation device can improve the estimation accuracy of the distance to the object shown in the image.

According to the modified example, the distance distribution calculation unit 32 is normal for obtaining an estimated distance distribution for an object represented in the object region, even for a type having a predetermined similarity relationship with the type estimated as the object. It may be one of the types for obtaining the distribution. A predetermined similarity relationship is set in advance, for example, and a similarity reference table representing the similarity relationship is stored in the memory 22 in advance. The distance distribution calculation unit 32 refers to the similarity reference table and identifies the type of the object represented in the object region and the type having a similarity relationship with the type whose certainty is equal to or higher than the certainty threshold. Then, the distance distribution calculation unit 32 may obtain a normal distribution for the types of objects for which similar relationships are set, as in the above embodiment. In this way, by using the normal distribution for the types of objects with similar relationships to calculate the estimated distance distribution, the types of objects shown in the image are erroneously recognized as other types with similar relationships. However, since the estimated distance distribution is obtained in consideration of the original type of size distribution information, the estimated distance distribution can more accurately represent the distribution of the estimated distance to the object.

Similarities are set, for example, between types of objects that have common appearance features. For example, "two-wheeled vehicle", which is one of the types of objects whose certainty is calculated by the object detection unit 31, and "pedestrian", which is another type of object, are both common to humans. Includes appearance features. Therefore, the two types "two-wheeled vehicle" and "pedestrian" are set to have a similar relationship with each other.

Alternatively, by executing clustering processing on the feature amount extracted from the area in which the object is represented for each type of the object to be detected, a similar relationship can be established between the types determined to belong to the same class. It may be set to exist. In this case, the feature amount used for determining the similarity is, for example, a Haar-like feature, a Histograms of Oriented Gradients (HOG), or a feature calculated by a classifier used by the object detection unit 31 to calculate the certainty. It can be a feature amount that can be used to determine the type of object, such as a map.

The distance estimation device according to the above embodiment or modification may be mounted on a device other than the in-vehicle device. For example, the distance estimation device according to the above embodiment or modification detects an object from an image generated by a surveillance camera installed so as to capture a predetermined area outdoors or indoors at predetermined intervals, and the detected object. It may be configured to estimate the distance to.

Further, the computer program that realizes the functions of each part of the processor 23 of the distance estimation device according to the above embodiment or modification is a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. It may be provided in a recorded form.

As described above, those skilled in the art can make various changes within the scope of the present invention according to the embodiment.

1 Vehicle control system 2 Camera 3 Electronic control device (distance estimation device)
4 In-vehicle network 21 Communication interface 22 Memory 23 Processor 31 Object detection unit 32 Distance distribution calculation unit 33 Distance estimation unit 34 Driving planning unit 35 Vehicle control unit

Claims (1)

For each of a plurality of types of objects to be detected, a storage unit that stores size distribution information indicating the reference size of the object of the type in the real space and the degree of variation in the size of the object of the type in the real space.
For each of the series of images obtained in time series, the object region in which the object is represented is detected from the image, and for each of the plurality of types, the object represented in the object region is the object of the type. An object detection unit that calculates the certainty of
For each of the images in which the object is detected in the series of images, the size of the object region on the image, the size distribution information of each of the plurality of types, and the certainty of the object are used. A distance distribution calculation unit that obtains the distribution of the estimated distance to the object shown in the image,
Of the series of images, each of the images in which the object was detected by the prediction process using the distribution of the estimated distance for each of the images in which the object was detected as an observation error model of the distance to the object. A distance estimation unit that estimates the distance to the object at the acquisition time,
Distance estimator with.

Images