JP2021009487A - Information processing device and on-vehicle system - Google Patents

Abstract

To provide a technique capable of appropriately calculating a distance to an object.SOLUTION: A first calculation unit 33 calculates the number of parallax points belonging to a plurality of respective sections demarcated based on parallax values, with the parallax points in an object area 43 as objects according to a parallax image 41 acquired using a stereo camera 11, on which the periphery of a vehicle 3 is photographed. A second calculation unit 34 calculates a distance to an object shown in the object area 43 based on the number of the parallax points belonging to the plurality of respective sections calculated by the first calculation unit 33. One or more object sections among the plurality of sections is/are compared with one or more sections relating to the parallax values each smaller than an object section, and a range of each of the parallax values, which demarcates a section, is large.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a technique for recognizing an object around a vehicle.

Conventionally, there is known a technique of detecting a distance to an object around a vehicle based on an image taken by a stereo camera mounted on the vehicle. For example, Patent Document 1 below proposes a technique for improving the stability of distance measurement on the distant side by relatively improving the resolution at a long distance.

Japanese Unexamined Patent Publication No. 2011-191905

In a parallax image acquired from a stereo camera or the like, the distance is obtained from the parallax value of the parallax point. When measuring the distance of an object around the vehicle, a histogram classified by the parallax value can be formed, and the distance of the object can be calculated based on the number of votes in each bin. However, as a result of detailed examination by the inventor, the distance width corresponding to one bin of the histogram classified by the parallax becomes smaller as the distance from the vehicle is relatively small, so that 1 in the short-distance bin. A problem has been found in which the number of votes for one bin is reduced and the distance of an object may not be calculated properly.

One aspect of the present disclosure is to provide a technique capable of appropriately calculating the distance to an object.

One aspect of the present disclosure is an information processing device (13) used in a vehicle (3) equipped with an image pickup device (11). The information processing device includes a first calculation unit (33) and a second calculation unit (34). The first calculation unit targets a plurality of sections defined based on the parallax value for the parallax points in the predetermined region (43) in the parallax image (41) acquired by using the imaging device and captured around the vehicle. Calculate the number of parallax points belonging to each. The second calculation unit calculates the distance to the object shown in the predetermined area based on the number of disparity points belonging to each of the plurality of sections calculated by the first calculation unit. The target section of one or more of the plurality of sections is configured so that the range of the parallax value that defines the section is larger than that of the one or more sections that relate to the parallax value smaller than the target section.

With such a configuration, in the target section at a relatively short distance, the range of the parallax value that defines the section is larger than that at the section at a relatively long distance. Therefore, it is possible to prevent the number of disparity points belonging to each section included in the target section from becoming too small, and it is possible to appropriately calculate the distance to the object.

Another aspect of the present disclosure is an in-vehicle system (1) mounted on a vehicle (3) and includes an imaging device (11) and an information processing device (13) described above. The image pickup device is a device for acquiring a parallax image (41) in which the periphery of the vehicle is photographed.

According to such a configuration, it is possible to realize the calculation of the distance to an appropriate object by providing the information processing device of one aspect of the present disclosure described above.

It is a block diagram which shows the structure of a driving support system. It is a functional block diagram of the information processing apparatus of embodiment. It is a flowchart of an object detection process. It is a figure which shows the example of the parallax image. FIG. 5A is a diagram showing an example of a parallax histogram based on a parallax image, and FIG. 5B is a diagram showing a state in which a part is switched to a distance histogram based on the same parallax image. It is a graph which shows the relationship between the parallax value and the distance. FIG. 7A is a diagram showing an example of a case where the rear glass of the preceding vehicle has a reflection, and FIG. 7B is a diagram showing an example of a case where the inside of the preceding vehicle can be visually recognized. It is a figure explaining the switching method of a histogram. It is a figure explaining the voting number determination method of the modification, and is the figure which shows the position which switches the number of bins to be referred. It is a figure explaining the voting number determination method of the modification, and is the figure which shows the change of the number of bins to be referred.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.
[1. Embodiment]
[1-1. Constitution]
The in-vehicle system 1 shown in FIG. 1 is a system mounted on a vehicle 3 and includes a stereo camera 11, a driving support device 12, and an information processing device 13.

The stereo camera 11 includes a right camera 11a and a left camera 11b. The right camera 11a and the left camera 11b include image sensors such as a CCD sensor and a CMOS sensor that are synchronized with each other. The right camera 11a and the left camera 11b are attached to the inside of the windshield of the vehicle 3, for example, at the same height from the road surface at predetermined intervals in the vehicle width direction. The right camera 11a and the left camera 11b repeatedly photograph the periphery of the vehicle 3 at a predetermined cycle (for example, at intervals of 0.1 seconds). The stereo camera 11 corresponds to an imaging device. If a parallax image can be acquired, a device other than a stereo camera may be used.

The driving support device 12 is a device including a microcomputer provided with a CPU, ROM, RAM, I / O, etc. (not shown). As an example, the driving support device 12 is configured to be capable of controlling so-called adaptive cruise control (hereinafter referred to as ACC) that accelerates / decelerates so as to maintain an inter-vehicle distance from the preceding vehicle. The driving support device 12 executes ACC using the object information output by the information processing device 13.

The information processing device 13 includes a microcomputer having a CPU 21 and, for example, a semiconductor memory such as RAM or ROM (hereinafter, memory 22). Each function of the information processing device 13 is realized by the CPU 21 executing a program stored in the memory 22.

As shown in FIG. 2, the information processing device 13 includes a first calculation unit 33 and a second calculation unit 34. Further, the information processing device 13 may include a parallax detection unit 31, an object recognition / tracking unit 32, an information acquisition unit 35, a type determination unit 36, and an output unit 37.

The parallax detection unit 31 detects a parallax point indicating an object projected on the parallax image from the parallax image taken around the vehicle 3 acquired by using the stereo camera 11. The parallax image includes a large number of parallax points centered on the edge portion on the image. That is, the parallax point is detected at a position in the parallax image where there is a high probability that an object exists. The disparity point can be obtained by a known method. In the present embodiment, the parallax detection unit 31 converts the acquired analog data of the left and right images into digital data having a predetermined luminance gradation for each pixel each time the images are taken by the right camera 11a and the left camera 11b. To do. Then, the parallax detection unit 31 performs stereo matching of the left and right digital images, calculates the parallax, and generates the parallax image.

The object recognition / tracking unit 32 detects an object from the parallax image, tracks the same object in the parallax images acquired at different timings, and further determines the type of the object. The object recognition / tracking unit 32 executes these processes by using a known method.

The first calculation unit 33 calculates the number of parallax points belonging to each of a plurality of sections defined based on the parallax value, targeting the parallax points in a predetermined object region in the parallax image. In the present embodiment, a plurality of bins of the parallax histogram or the distance histogram described later correspond to the plurality of sections described above. Further, the target section, which is one or more of the above-mentioned plurality of sections, has a larger range of parallax values for defining the sections than one or more sections having a parallax value smaller than the target section. .. In the present embodiment, the target section is a section to which a parallax point of a predetermined parallax value or more belongs among a plurality of sections. Further, in the present embodiment, among the plurality of bins in the histogram, the bin corresponding to a short distance whose distance is equal to or less than a predetermined value corresponds to the target section.

The second calculation unit 34 calculates the distance to the object shown in the object region of the parallax image based on the number of parallax points belonging to each bin of the histogram calculated by the first calculation unit 33. In the present embodiment, the distance corresponding to the parallax value related to the bin, which is the mode value, is calculated as the distance of the object. Of course, if it is calculated based on the number of parallax points belonging to each bin, the distance may be calculated by a method other than the mode. The object area is an area in which an object exists in the parallax image. The method of setting the object area and the specific mode are not particularly limited, and various methods and shapes can be used for setting.

The information acquisition unit 35 acquires object information which is information indicating the behavior of the preceding vehicle. The object information includes information on the position or movement of the preceding vehicle shown in the parallax image, but the specific content thereof is not particularly limited, and various information that can be used for driving support by the driving support device 12 is used as object information. May be used as. Examples of the object information include the distance to the object, the relative velocity of the object with respect to the traveling direction of the vehicle 3, the position and relative velocity of the object in the direction orthogonal to the traveling direction, the width of the object, the type of the object, and the like. The object type is specified by the type determination unit 36. The output unit 37 outputs the object information acquired by the information acquisition unit 35 to the driving support device 12.

[1-2. processing]
Next, the object detection process executed by the information processing apparatus 13 will be described with reference to the flowchart of FIG. This process is periodically executed every time an image is acquired by, for example, the right camera 11a and the left camera 11b.

First, in S1, the parallax detection unit 31 calculates the parallax. Here, the parallax detection unit 31 generates a parallax image including a parallax point based on the left and right captured images taken by the right camera 11a and the left camera 11b.

In S2, the object recognition / tracking unit 32 detects an object from the parallax image. At this time, an object area is set for each of the detected objects. As shown in FIG. 4, the parallax image 41 includes a large number of parallax points 42. The object area 43 is set in the area where the object exists in the parallax image 41. In the present embodiment, the target object of the object region 43 is the preceding vehicle 101.

In S3, the object recognition / tracking unit 32 tracks the object. Here, the object detected in the parallax image (hereinafter, also referred to as the previous image) one frame before the parallax image acquired this time (hereinafter, also referred to as the current image) is tracked and specified in the current image. .. The specific method of object tracking is not particularly limited. For example, the likelihood of the object detected in the previous image and the object detected in the current image may be calculated, and the same object may be determined as the object whose likelihood is equal to or higher than a predetermined threshold value. When there are a plurality of objects in the image this time, the one with the highest likelihood may be determined to be the same object.

In S4, the first calculation unit 33 creates a parallax histogram for the parallax point 42 in the object region 43 as shown in FIG. 5A. Each bin of the parallax histogram is defined by a parallax value, and the parallax histogram shows the number of parallax points 42 in the object region 43 belonging to each bin. The range of parallax values corresponding to each bin is set to a constant width based on the parallax value. However, in FIG. 5A, since the horizontal axis is converted into the distance from the stereo camera 11 to the object, the shorter the distance, the narrower the width of the bin even if the parallax value width is the same. Hereinafter, when simply described as a distance, it means this distance. As confirmedly shown in FIG. 6, the parallax value and the distance are in an inversely proportional relationship. The larger the parallax value, the smaller the distance from the stereo camera 11 to the object. Further, even if the width of the parallax value is the same, the width of the corresponding distance becomes larger as the distance is longer (that is, the parallax value is smaller). Therefore, even if the bins are defined at equal intervals based on the parallax value, the bins are not evenly spaced when the horizontal axis is the distance as shown in FIG. 5A, and the closer the distance is, the closer the bin boundaries are.

Here, when the parallax value corresponding to the mode bin is calculated as the distance to the object based on the number of parallax points belonging to each bin obtained by the parallax histogram as shown in FIG. 5A, it is as follows. Problems may occur.

In the parallax histogram shown in FIG. 5A, the parallax points are concentrated at two points, a short distance and a long distance. For example, as shown in FIG. 7A, when the rear glass 102 reflects an object behind and far away, or when the object inside the vehicle interior is projected through the rear glass 102 as shown in FIG. 7B. Parallax points are also detected for these objects, and the parallax histogram is voted. These are voted for bins that correspond to relatively long distances. On the other hand, the disparity point related to the back surface of the preceding vehicle 101 is voted for the bin corresponding to the relatively short distance. Since the resolution of the parallax point becomes higher as the distance becomes shorter, the parallax value changes greatly with a slight change in distance. Therefore, the bins to be voted are dispersed, and the number of disparity points belonging to one bin tends to decrease as the distance increases. On the contrary, the longer the distance, the larger the number of discrimination points tends to be.

The distance desired to be acquired by driving support is often the distance to the back surface of the preceding vehicle 101, which is the distance corresponding to a short-distance bin. Even if the shorter the distance, the more discriminant points are detected, the longer the distance, the more discriminant points are included in one bin, and as shown in FIG. 5A, the long-distance bin tends to be the mode. There is a risk of

Therefore, in the following S5, as shown in FIG. 5B, when the parallax value is equal to or more than the predetermined threshold value (that is, when the distance is shorter than the predetermined distance) and the value is less than the predetermined threshold value, the predetermined parallax value is used as the threshold value. Switch the histogram between case and. Specifically, a distance histogram is generated for a parallax point whose parallax value is equal to or greater than a predetermined threshold value.

In the distance histogram, the range of each bin is set by the distance at regular intervals. That is, when the range of each bin is viewed on the basis of the parallax value, it is set to be equal when converted into the range of the distance, and the range of the parallax value becomes wider as the bin on the shorter distance side.

Each bin of the distance histogram is configured to have a larger range of parallax values that define the bin than each bin of the parallax histogram. As a result, the number of votes for the disparity point increases in the short-distance bin (that is, the disparity points belonging to the bin increase), so that the short-distance bin can be easily calculated as the mode.

As shown in FIG. 8, when the bin spacing of the distance histogram is A [m], the parallax histogram has the largest spacing among the bins whose distance-converted bin spacing is A [m] or less. However, the distance histogram may be switched to the boundary. This allows the bin spacing to switch smoothly.

In the present embodiment, each bin to which a parallax point having a parallax value equal to or higher than a predetermined threshold belongs (that is, each bin of the distance histogram in FIG. 5B) corresponds to "one or more target sections among a plurality of sections". .. Further, each bin to which a parallax point having a parallax value less than a predetermined threshold belongs (that is, each bin of the parallax histogram in FIG. 5B) corresponds to "one or more sections having a parallax value smaller than the target section". .. Further, the plurality of bins related to the distance histogram in FIG. 5B correspond to "two or more sections".

In S6, the second calculation unit 34 calculates the mode value of the created histogram of FIG. 5B as the object distance.
In S7, the information acquisition unit 35 acquires object information including the position and velocity of the object whose distance is set in S6 and which is tracked in S3. The speed of the object can be calculated from, for example, the speed of the vehicle 3 and the relative speed of the vehicle 3 and the object.

In S8, the type determination unit 36 performs template matching and the like to determine the type of the detected object. After that, this process ends.
The various information acquired by this process is output to the driving support device 12 by the output unit 37.

[1-3. effect]
According to the embodiment described in detail above, the following effects are obtained.
(1a) In the information processing apparatus 13, the range of the parallax value that defines the bin corresponding to the relatively short distance becomes larger than the range of the parallax value that defines the bin corresponding to the relatively long distance. It is configured as follows. Therefore, it is possible to prevent the number of parallax values belonging to the bin corresponding to a long distance from being output as an erroneous distance due to a large range of bins based on the distance.

(1b) The information processing apparatus 13 changes the range of the parallax value related to the bin to which the parallax point equal to or more than the predetermined parallax value belongs to a range different from the bin of the parallax histogram created in the range of the constant parallax value. .. As a result, it is possible to prevent voting for excessively finely classified bins for short-distance disparity points, and it is possible to appropriately measure the distance to an object.

(1c) The information processing device 13 votes using a distance histogram for parallax points equal to or higher than a predetermined parallax value. Therefore, for bins for which short-distance disparity points are voted, the width can be set so that the distance intervals are equal when converted to the actual distance. As a result, it is possible to suppress the bias of the ease of voting of the bin due to the distance from the stereo camera 11 and appropriately measure the distance to the object.

[2. Other embodiments]
Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments, and can be implemented in various modifications.

(2a) In the above embodiment, the distance histogram is applied to the bin to which the parallax point whose parallax value is equal to or higher than a predetermined threshold value belongs, and the parallax histogram is applied to the other bins to obtain the mode. did. In addition, a configuration in which the bin of the histogram is used as an interval for obtaining the mode is illustrated.

However, one or more of the plurality of sections should be configured such that the range of the parallax value defining the section is larger than that of the one or more sections having a parallax value smaller than that section. For example, the specific configuration is not particularly limited.

For example, the distance histogram may not be applied in the range of all the parallax values among the parallax values equal to or higher than the predetermined threshold value, and the distance histogram may be applied only in the range of a part of the parallax values.

Also, for example, each interval for obtaining the mode may not correspond to one bin of the histogram. As shown in FIG. 9, when the parallax value is equal to or more than a predetermined threshold value, the voting value is calculated by referring to a plurality of bins, and when the parallax value is less than a predetermined threshold value, one bin is referred to. The voting value may be calculated. Of course, the number of bins to be referred to is not limited to these, and it is sufficient that the closer the distance is, the more bins are referenced. For example, as shown in FIG. 10, the closer the distance is, the more bins may be referred to in stages. With such a configuration, the distance can be calculated appropriately without generating a distance histogram.

In the configuration described above, the bin of the parallax histogram corresponds to "a plurality of minute sections having the same parallax value range". The section to which the parallax point belongs is composed of one or a plurality of continuous bins, and the section to which the parallax point having a parallax value exceeding a predetermined threshold value belongs includes a plurality of minute sections as compared with other sections. As described above, in the above-mentioned example, the interval for obtaining the mode includes one or more minute intervals.

Of course, the section may be set in a mode other than these. For example, the range of each of a plurality of sections defined based on the parallax value does not use a bin whose range is set so that the parallax value or the distance is constant, and the parallax value range is larger as the distance is closer. It may be set to an arbitrary range such as. Further, the configuration in which the predetermined threshold value of the parallax value for switching the setting method of the parallax value range is one in the above embodiment is illustrated. However, for example, the parallax value range setting method may be switched by dividing into three or more regions with two or more threshold values.

(2b) In the above embodiment, a configuration in which a histogram is created based on a discrimination point belonging to the object region 43 is illustrated. That is, the configuration in which the object region 43 is a predetermined region including a discrepancy point to be measured for measuring the distance of the object is illustrated. However, the predetermined region is not limited to the object region 43 described above, and various regions in the parallax image can be adopted. For example, the entire parallax image, a preset range in the parallax image, for example, a portion including a central portion or a vanishing point of the parallax image, a portion corresponding to the traveling direction of the vehicle 3, and the like may be set as a predetermined region.

(2c) The information processing apparatus 13 and its method described in the present disclosure are provided by configuring a processor and memory programmed to perform one or more functions embodied by a computer program. It may be realized by a dedicated computer. Alternatively, the information processing device 13 and its method described in the present disclosure may be realized by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the information processing device 13 and its method described in the present disclosure include a processor and memory programmed to perform one or more functions and a processor composed of one or more hardware logic circuits. It may be realized by one or more dedicated computers configured by a combination. The computer program may also be stored on a computer-readable non-transitional tangible recording medium as an instruction executed by the computer. The method for realizing the functions of each part included in the information processing apparatus 13 does not necessarily include software, and all the functions may be realized by using one or a plurality of hardware.

(2d) A plurality of functions possessed by one component in the above embodiment may be realized by a plurality of components, or one function possessed by one component may be realized by a plurality of components. .. Further, a plurality of functions possessed by the plurality of components may be realized by one component, or one function realized by the plurality of components may be realized by one component. Further, a part of the configuration of the above embodiment may be omitted. In addition, at least a part of the configuration of the above embodiment may be added or replaced with the configuration of the other above embodiment.

(2e) In addition to the above-mentioned information processing device 13, a system having the information processing device 13 as a component, a program for operating a computer as the information processing device 13, a non-transitional semiconductor memory in which this program is recorded, and the like. The present disclosure can also be realized in various forms such as an actual recording medium and a distance measurement determination method.

3 ... Vehicle, 11 ... Stereo camera, 13 ... Information processing device, 33 ... First calculation unit, 34 ... Second calculation unit, 41 ... Parallax image, 43 ... Object area

Claims (6)

An information processing device (13) used in a vehicle (3) equipped with an image pickup device (11).
The parallax point in a predetermined region (43) in the parallax image (41) acquired by using the image pickup device and photographed around the vehicle belongs to each of a plurality of sections defined based on the parallax value. The first calculation unit (33) that calculates the number of parallax points, and
A second calculation unit (34) for calculating the distance to the object shown in the predetermined area based on the number of the difference points belonging to each of the plurality of sections calculated by the first calculation unit is provided.
The target section, which is one or more of the plurality of sections, has a larger range of the parallax value that defines the section as compared with one or more sections having a parallax value smaller than the target section. Information processing device that is configured in. The information processing device according to claim 1.
The target section is an information processing apparatus to which the parallax point having a predetermined parallax value or more belongs among the plurality of sections. The information processing device according to claim 1 or 2.
The target section is two or more sections,
An information processing device in which the ranges related to the two or more sections are set to be equal when the ranges are converted into distance ranges. The information processing device according to claim 3.
An information processing device in which the range related to one or more sections related to a parallax value smaller than the target section is set to have a constant parallax value. The information processing device according to claim 1 or 2.
Each of the plurality of sections includes one or more minute sections having the same parallax value range.
The information processing apparatus including the target section includes a large number of the minute sections as compared with one or more sections having a parallax value smaller than the target section. An in-vehicle system (1) used by being mounted on a vehicle (3).
An image pickup device (11) for acquiring a parallax image (41) taken around the vehicle, and
An in-vehicle system including the information processing device (13) according to any one of claims 1 to 4.