JP2021086265A - Image processing device - Google Patents

Abstract

To provide an image processing device capable of appropriately performing image processing.SOLUTION: An image processing device according to the embodiment of the present disclosure comprises: an image area setting unit that sets a first image area including an image of a first object and a second image area including an image of a second object in a stereo image; and a combine processing unit that is capable of performing combine processing for combining the first image area and the second image area, and calculates a first actual size that is a vertical actual size of the first object and a second actual size that is the vertical actual size of the second object based on the stereo image and a distance image generated based on the stereo image and avoids the combine processing when a first condition including that the first actual size and the second actual size are smaller than a first threshold.SELECTED DRAWING: Figure 10

Description

The present disclosure relates to an image processing apparatus that performs image processing on a captured image.

For example, an image pickup device mounted on a vehicle and imaging the front of the vehicle may image fallout such as snow particles and rain particles. For example, Patent Document 1 discloses a technique for removing noise such as snow and rain included in a captured image.

International Publication No. 2008-11549

It is desired that the image processing apparatus appropriately perform image processing even when fallout such as snow particles and rain particles is imaged.

It is desirable to provide an image processing apparatus capable of appropriately performing image processing.

The image processing apparatus according to the embodiment of the present disclosure includes an image area setting unit and a coupling processing unit. The image area setting unit is configured to set a first image area including an image of a first object and a second image area including an image of a second object in a stereo image. The combining processing unit can perform a combining process for combining the first image region and the second image region, and the first object is based on the stereo image and the distance image generated based on the stereo image. The first actual size, which is the actual size in the vertical direction of, and the second actual size, which is the actual size in the vertical direction of the second object, are calculated, and the first actual size and the second actual size are the first. It is configured to avoid the join process when the first condition including being smaller than the threshold value of is satisfied.

According to the image processing apparatus according to the embodiment of the present disclosure, the first actual size which is the vertical actual size of the first object and the second actual size which is the vertical actual size of the second object. The actual size is calculated, and when the first condition including that the first actual size and the second actual size are smaller than the first threshold value is satisfied, the join processing is avoided. Image processing can be performed.

It is a block diagram which shows one structural example of the image processing apparatus which concerns on one Embodiment of this disclosure. It is an image diagram which shows an example of a stereo image. It is an image diagram which shows an example of an image area. It is an image diagram which shows an example of the image at the time of snowfall. It is another image diagram which shows an example of the image at the time of snowfall. It is a flowchart which shows one operation example of the coupling processing part shown in FIG. It is a flowchart which shows an example of the simple determination process shown in FIG. It is explanatory drawing which shows an example of the process shown in FIG. 7. It is another explanatory diagram showing an example of the process shown in FIG. 7. It is a flowchart which shows an example of the determination process shown in FIG. It is explanatory drawing which shows an example of the process shown in FIG. It is a flowchart which shows an example of another determination process shown in FIG. It is explanatory drawing which shows an example of the process shown in FIG. It is another explanatory diagram showing an example of the process shown in FIG. It is a flowchart which shows one operation example of the coupling processing part which concerns on the modification.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

<Embodiment>
[Configuration example]
FIG. 1 shows a configuration example of an image processing device (image processing device 1) according to an embodiment. The image processing device 1 has a stereo camera 11 and a processing unit 20. The image processing device 1 is mounted on a vehicle 10 such as an automobile.

The stereo camera 11 is configured to generate a set of images (left image PL and right image PR) having parallax with each other by photographing the front of the vehicle 10. The stereo camera 11 has a left camera 11L and a right camera 11R. Each of the left camera 11L and the right camera 11R includes a lens and an image sensor. In this example, the left camera 11L and the right camera 11R are arranged in the vehicle of the vehicle 10 near the upper part of the windshield of the vehicle 10 by a predetermined distance in the width direction of the vehicle 10. The left camera 11L and the right camera 11R perform an imaging operation in synchronization with each other. The left camera 11L generates the left image PL, and the right camera 11R generates the right image PR. The left image PL and the right image PR constitute a stereo image PIC. The stereo camera 11 is adapted to generate a series of stereo image PICs by performing an imaging operation at a predetermined frame rate (for example, 60 [fps]).

FIG. 2 shows an example of a stereo image PIC, FIG. 2 (A) shows an example of a left image PL, and FIG. 2 (B) shows an example of a right image PR. In this example, another vehicle (preceding vehicle 90) is traveling in front of the vehicle 10 on the traveling road on which the vehicle 10 is traveling. The left camera 11L generates a left image PL by imaging the preceding vehicle 90, and the right camera 11R generates a right image PR by imaging the preceding vehicle 90. The stereo camera 11 is adapted to generate a stereo image PIC including such a left image PL and a right image PR.

Based on the stereo image PIC supplied from the stereo camera 11, the processing unit 20 creates an image area for images of various objects such as vehicles and obstacles in front of the vehicle 10 included in the left image PL and the right image PR. By setting R, it is configured to recognize an object. Then, the processing unit 20 outputs information about the recognition result. In the vehicle 10, for example, based on the information about the object recognized by the processing unit 20, the traveling control of the vehicle 10 can be performed, or the information about the recognized object can be displayed on the console monitor. ing. The processing unit 20 is composed of, for example, a CPU (Central Processing Unit) for executing a program, a RAM (Random Access Memory) for temporarily storing processing data, a ROM (Read Only Memory) for storing the program, and the like. The processing unit 20 includes a distance image generation unit 21, an image area setting unit 22, and a combination processing unit 23.

The distance image generation unit 21 is configured to generate a distance image PZ by performing predetermined image processing including stereo matching processing and filtering processing based on the left image PL and the right image PR. The distance image PZ has a plurality of pixel values. Each of the plurality of pixel values corresponds to the distance value Zmeas to the point corresponding to each pixel in the three-dimensional real space. One pixel in the distance image PZ corresponds to, for example, a plurality of (for example, 16) pixels in the left image PL and a plurality of (for example, 16) pixels in the right image PR.

The image area setting unit 22 is configured to set the image area R in the left image PL and the right image PR based on the distance image PZ. Specifically, the image area setting unit 22 sets the image area R so that, for example, in the distance image PZ, a plurality of pixels having continuous distance values Zmeas belong to one image area R.

FIG. 3 shows an example of the image area R. In this example, the right image PR in the stereo image PIC includes an image of the preceding vehicle 90 traveling in front of the vehicle 10 which is the own vehicle. In FIG. 3, the right image PR is taken as an example, but the same applies to the left image PL. The preceding vehicle 90 is traveling in front of the vehicle 10 along the traveling path, and the right image PR includes an image of the back surface of the preceding vehicle 90. Since the distance value Zmeas in the distance image PZ is continuous on the back surface of the preceding vehicle 90, the image area setting unit 22 sets the image area R (image area R1) on the back surface of the preceding vehicle 90.

When a plurality of image areas R are set, the combination processing unit 23 is configured to perform an area combination process J for combining the plurality of image areas R. Specifically, for example, when the preceding vehicle 91 changes lanes from the lane next to the lane in which the vehicle 10 travels to the lane in which the vehicle 10 travels in front of the vehicle 10, for example, the right image PR and the left image. Each of the PLs may include a back image and a side image of the preceding vehicle 91. In this case, the image area setting unit 22 sets the image area R (image area R11) on the back portion of the preceding vehicle 91, and sets another image area R (image area R12) on the side surface portion of the preceding vehicle 91. May be set. In such a case, the combination processing unit 23 combines the two image regions R11 and R12. In this way, the coupling processing unit 23 sets one image area R for the preceding vehicle 91.

Further, the coupling processing unit 23 avoids the region coupling processing J when a predetermined determination condition is satisfied. Specifically, when a plurality of image areas R are set for fallout such as snow particles and raindrops, the combination processing unit 23 avoids combining the plurality of image areas R. Perform processing.

FIGS. 4 and 5 show an example of an image at the time of snowfall. In this example, the right image PR includes images of a plurality of snow grain loci 100 in addition to the preceding vehicle 90. The stereo camera 11 generates a stereo image PIC by performing an exposure operation for an exposure time of a predetermined length. Since the snow particles descend and the vehicle 10 travels forward, the position of the snow particles as seen from the vehicle 10 moves within the exposure time. As a result, the snow particles are imaged as the locus 100. The image area setting unit 22 may set the image area R for such a locus 100. If the combination processing unit 23 combines the image areas R of a plurality of loci 100 as shown in FIG. 5, the combined image area R (image area R2) becomes a large area, so that the vehicle Based on the recognition result in the image area R2, the 10 may perform, for example, travel control of the vehicle 10 or display information on the console monitor. Therefore, the coupling processing unit 23 performs a determination process (simple determination process J1 and determination processes J2A, J2B, which will be described later), and when a predetermined determination condition is satisfied, the object in each image area R is a snow grain, a raindrop, or the like. Determined to be fallout. Then, when the object in each image area R is a fallout, the combination processing unit 23 avoids the combination of the plurality of image areas R. As a result, the image area R of the fallout remains a small area, so that the vehicle 10 performs, for example, travel control of the vehicle 10 or displays information on the console monitor based on the recognition results in these image areas R. It is possible to reduce the risk of this.

The processing unit 20 recognizes various objects such as a vehicle and an obstacle in front of the vehicle 10 based on the image area R set in this way. Then, the processing unit 20 outputs information about the recognition result.

Here, the image area setting unit 22 corresponds to a specific example of the “image area setting unit” in the present disclosure. The binding processing unit 23 corresponds to a specific example of the “bonding processing unit” in the present disclosure. The stereo image PIC corresponds to a specific example of the "stereo image" in the present disclosure. The distance image PZ corresponds to a specific example of the "distance image" in the present disclosure. The distance value Zmeas corresponds to a specific example of the "distance value" in the present disclosure. The vehicle 10 corresponds to a specific example of the "vehicle" in the present disclosure.

[Operation and action]
Subsequently, the operation and operation of the image processing device 1 of the present embodiment will be described.

(Overview of overall operation)
First, an outline of the overall operation of the image processing device 1 will be described with reference to FIG. The stereo camera 11 generates a stereo image PIC including a left image PL and a right image PR by photographing the front of the vehicle 10. Based on the stereo image PIC supplied from the stereo camera 11, the processing unit 20 creates an image area for images of various objects such as vehicles and obstacles in front of the vehicle 10 included in the left image PL and the right image PR. By setting R, the object is recognized.

(Detailed operation)
In the processing unit 20, the distance image generation unit 21 generates a distance image PZ based on the stereo image PIC each time the stereo image PIC is supplied from the stereo camera 11, and the image area setting unit 22 generates this distance image. The image area R is set based on the PZ. Then, when a plurality of image areas R are set, the combination processing unit 23 performs the area combination process J for combining the plurality of image areas R. Further, the coupling processing unit 23 avoids the region coupling processing J when a predetermined determination condition is satisfied. The processing in the joining processing unit 23 will be described in detail below.

FIG. 6 shows an example of processing in the coupling processing unit 23. The join processing unit 23 determines whether to avoid the area join process J by performing the simple determination process J1 and the determination processes J2A and J2B. This process will be described in detail below.

First, the coupling processing unit 23 performs the simple determination processing J1 (step S101). This simple determination process J1 is performed as a pre-stage of the more detailed determination processes J2A and J2B described later.

FIG. 7 shows an example of the simple determination process J1. In this simple determination process J1, the combination processing unit 23 determines whether or not each of the three determination conditions shown in steps S201 to S203 is satisfied.

First, the combination processing unit 23 determines whether or not the horizontal image width of each of the plurality of image regions R is smaller than the threshold value TA (step S201).

FIG. 8 shows an example of the process of step S201. In this example, three image regions R21 to R23 are set by the image region setting unit 22 for the three loci 121 to 123. The combining processing unit 23 calculates the image width W21 of the image area R21, calculates the image width W22 of the image area R22, and calculates the image width W23 of the image area R23 based on the stereo image PIC and the distance image PZ. Then, in this example, the coupling processing unit 23 determines whether or not each of these image widths W21 to W23 is smaller than the threshold value TA. That is, when the object in the image region R is a fallout such as a snow grain or a raindrop, the image width in the horizontal direction is expected to be small because the snow grain or the raindrop is small. Therefore, the coupling processing unit 23 can determine whether or not the object is a fallout by determining whether or not the image width in the horizontal direction is smaller than the threshold value TA. The image width and the threshold value TA can be expressed using, for example, the number of pixels.

Next, the combination processing unit 23 determines whether or not the distance value Zmeas of the object has continuity in the plurality of image regions R (step S202).

FIG. 9 shows an example of the process of step S202. In FIG. 9, among the plurality of pixels included in the distance image PZ, the pixels related to the three loci 121 to 123 are illustrated as distance points A. For example, when the distance values Zmeas at all the distance points A related to these three loci 121 to 123 are substantially the same, the combination processing unit 23 has continuity in the distance values Zmeas in the plurality of image regions R. Is determined. In this example, the coupling processing unit 23 determines whether or not the distance values Zmeas of these loci 121 to 123 are not continuous based on the distance image PZ. That is, when the object in the image region R is a fallout such as a snow grain or a raindrop, the distance value Zmeas may differ between the trajectories, so that it is expected that the distance value Zmeas is not continuous. Therefore, the coupling processing unit 23 can determine whether or not the object is a fallout by determining whether or not the distance value Zmeas of the object is not continuous.

Next, the coupling processing unit 23 determines whether or not the total accuracy of the plurality of distance values Zmeas of the object is smaller than the threshold value TB in each of the plurality of image regions R (step S203).

For example, in the example of FIG. 9, the coupling processing unit 23 determines whether or not the total accuracy of the distance values Zmeas at the plurality of distance points A of the locus 121 is smaller than the threshold value TB. Specifically, first, the coupling processing unit 23 calculates the accuracy of the distance value Zmeas at the plurality of distance points A related to the locus 121. For example, the combination processing unit 23 evaluates the clarity of the image edge at the distance point A based on the gradient of the pixel values at the plurality of pixels corresponding to each distance point A in the left image PL and the right image PR. , The accuracy of the distance value Zmeas at the distance point A can be calculated based on the clarity of the image edge. That is, the clearer the image edge in the left image PL or the right image PR, the more accurately the distance image generation unit 21 can obtain the distance value Zmeas. Therefore, the combination processing unit 23 is based on the clarity of the image edge. Therefore, the accuracy of the distance value Zmeas can be calculated. Then, the coupling processing unit 23 calculates the total accuracy of the distance values Zmeas at the plurality of distance points A calculated in this way, and determines whether or not the total value is smaller than the threshold value TB. That is, when the object in the image region R is a descent such as a snow grain or a raindrop, the number of distance points A is small and the accuracy of the distance value Zmeas is low, so that the accuracy of the plurality of distance values Zmeas is high. The total value is expected to be small. Therefore, the coupling processing unit 23 can determine whether or not the object is a fallout by determining whether or not the total accuracy of the plurality of distance values Zmeas of the object is smaller than the threshold value TB. it can.

Similarly, the coupling processing unit 23 determines whether or not the total accuracy of the distance values Zmeas at the plurality of distance points A of the locus 122 is smaller than the threshold value TB, and determines whether the total value of the accuracy of the distance values Zmeas is smaller than the threshold TB, and the distances at the plurality of distance points A of the locus 123 Determine if the total accuracy of the value Zmeas is less than the threshold TB.

This completes the simple determination process J1. The processes of steps S201 to S203 are not limited to this order, and may be replaced.

Next, as shown in FIG. 6, the combination processing unit 23 confirms whether or not all the determination conditions in the simple determination process J1 are satisfied (step S102). That is, in the combining processing unit 23, the horizontal image width of each of the plurality of image areas R is smaller than the threshold value TA (step S201), and the distance value Zmeas of the objects in the plurality of image areas R is not continuous. Whether or not the three determination conditions of that (step S202) and that the total accuracy of the plurality of distance values Zmeas of the object is smaller than the threshold value TB (step S203) are satisfied in each of the plurality of image areas R. To confirm. If at least one of these three determination conditions is not satisfied (“N” in step S102), the process proceeds to step S108.

When all of these three determination conditions are satisfied in step S102 (“Y” in step S102), the combination processing unit 23 performs determination processing J2A (step S103).

FIG. 10 shows an example of the determination process J2A. In this determination process J2A, the combination processing unit 23 determines whether or not each of the three determination conditions shown in steps S211 to S213 is satisfied.

First, the coupling processing unit 23 determines whether or not the actual size of the object in the vertical direction is smaller than the threshold value TC in each of the plurality of image regions R (step S211).

FIG. 11 shows an example of the process of step S211. In FIG. 11, the locus 121 is drawn as an example, but the same applies to the loci 122 and 123. Since the snow particles descend and the vehicle 10 travels forward, the locus 121 is extended so as to be closer to the vehicle 10 as it is closer to the road surface 110 of the travel path. The coupling processing unit 23 calculates the actual size S of such a locus 121 in the vertical direction at various positions based on the stereo image PIC and the distance image PZ. Then, the coupling processing unit 23 selects the maximum value of the actual size S, and determines whether or not this maximum value is smaller than the threshold value TC. That is, when the object in the image region R is a fallout such as a snow grain or a raindrop, the actual size S of the object in the vertical direction is expected to be small. Therefore, the coupling processing unit 23 can determine whether or not the object is a fallout by determining whether or not the actual size S in the vertical direction is smaller than the threshold value TC. The actual size S and the threshold value TC are values in units of millimeters, for example.

Next, the coupling processing unit 23 determines whether or not the number of distance points A of the object is smaller than the threshold value TD in each of the plurality of image regions R (step S212).

For example, in the example of FIG. 9, the coupling processing unit 23 calculates the number of a plurality of distance points A of the locus 121 based on the stereo image PIC and the distance image PZ. Then, the coupling processing unit 23 determines whether or not the number of the distance points A is smaller than the threshold value TD. That is, when the object in the image region R is a fallout such as a snow grain or a rain drop, it is expected that the number of distance points A is small. Therefore, the coupling processing unit 23 can determine whether or not the object is a fallout by determining whether or not the number of distance points A of the object is smaller than the threshold value TD.

Similarly, the coupling processing unit 23 determines whether the number of the plurality of distance points A of the locus 122 is smaller than the threshold value TD, and whether the number of the plurality of distance points A of the locus 123 is smaller than the threshold value TD. Judge whether or not.

Next, the coupling processing unit 23 determines whether or not the total accuracy of the plurality of distance values Zmeas of the object is smaller than the threshold value TB in each of the plurality of image regions R (step S213). The process of step S213 is the same as the process of step S203 in the simple determination process J1 shown in FIG. 7.

This completes the determination process J2A. The processes of steps S211 to S213 are not limited to this order, and may be replaced.

Next, as shown in FIG. 6, the coupling processing unit 23 confirms whether or not all the determination conditions in the determination process J2A are satisfied (step S104). That is, the combination processing unit 23 has the actual size of the object in the vertical direction smaller than the threshold value TC in each of the plurality of image areas R (step S211), and the distance points of the objects in each of the plurality of image areas R. The number of A is smaller than the threshold value TD (step S212), and the total accuracy of the plurality of distance values Zmeas of the object is smaller than the threshold value TB in each of the plurality of image areas R (step S213). ) Is confirmed whether or not the three judgment conditions are satisfied. When all of these three determination conditions are satisfied (“Y” in step S104), the process proceeds to step S107.

If at least one of these three determination conditions is not satisfied in step S104 (“N” in step S104), the combination processing unit 23 performs determination processing J2B (step S105).

FIG. 12 shows an example of the determination process J2B. In this determination process J2B, the combination processing unit 23 determines whether or not each of the four determination conditions shown in steps S221 to S224 is satisfied.

First, the combination processing unit 23 determines whether or not the horizontal image width of each of the plurality of image regions R is smaller than the threshold value TA (step S221). The process of step S221 is the same as the process of step S201 in the simple determination process J1 shown in FIG. 7.

Next, the coupling processing unit 23 determines whether or not the total accuracy of the plurality of distance values Zmeas of the object is smaller than the threshold value TB in each of the plurality of image regions R (step S222). The process of step S222 is the same as the process of step S203 in the simple determination process J1 shown in FIG. 7.

Next, the coupling processing unit 23 determines whether or not the height value indicating the height of the object from the road surface is higher than the threshold value TE in each of the plurality of image regions R (step S223).

FIG. 13 shows an example of the process of step S223. In FIG. 13, the locus 121 is drawn as an example, but the same applies to the loci 122 and 123. The locus 121 is extended so as to be closer to the vehicle 10 as it is closer to the road surface 110 of the traveling road. Based on the stereo image PIC and the distance image PZ, the coupling processing unit 23 calculates a height value H indicating the height of the locus 121 from the road surface 110 in the portion of the locus 121 closest to the vehicle 10. To do. Then, the coupling processing unit 23 determines whether or not the height value H is larger than the threshold value TE. That is, when the object in the image region R is a fallout such as a snow grain or a raindrop, the height value H is expected to be large because the object is floating. Therefore, the coupling processing unit 23 can determine whether or not the object is a fallout by determining whether or not the height value H is larger than the threshold value TE. The height value H and the threshold value TE are values in units of millimeters, for example.

Next, the coupling processing unit 23 determines whether or not the distance value indicating the horizontal separation distance between the plurality of image regions R is larger than the threshold value TF (step S224).

FIG. 14 shows an example of the process of step S224. In this example, two trajectories 121 and 122 are shown. The coupling processing unit 23 calculates a distance value D indicating a separation distance between the locus 121 and the locus 122 based on the stereo image PIC and the distance image PZ. Then, in this example, the coupling processing unit 23 determines whether or not the distance value D is larger than the threshold value TF. That is, when the object in the image region R is a fallout such as a snow grain or a raindrop, the plurality of fallouts are separate fallouts, so that the distance value D is expected to be large. Therefore, the coupling processing unit 23 can determine whether or not the object is a fallout by determining whether or not the distance value D is larger than the threshold value TF. The distance value D and the threshold value TF are values in units of millimeters, for example.

Similarly, the coupling processing unit 23 determines whether or not the distance value indicating the horizontal separation distance between the locus 122 and the locus 123 is larger than the threshold value TF.

The threshold value TF may be a fixed value or a variable value. When the threshold value TF is set to a variable value, the coupling processing unit 23 can set the threshold value TF based on the accuracy of a plurality of distance values Zmeas of the object, for example. Specifically, when the coupling processing unit 23 calculates the threshold value TF for the separation distance between the locus 121 and the locus 122, the accuracy of the distance value Zmeas at the plurality of distance points A related to the locus 121 The sum of the total value and the total accuracy of the distance value Zmeas at the plurality of distance points A related to the locus 122 is calculated. Then, the coupling processing unit 23 can set the threshold value TF to a smaller value as the sum value is smaller, and can set the threshold value TF to a larger value as the sum value is larger. As a result, for example, when the object is a fallout such as a snow grain or a raindrop, the number of distance points A is small and the accuracy of the distance value Zmeas is low, so that the threshold value TF becomes a small value. .. As a result, the distance value D tends to be larger than the threshold value TF, so that it is easy to determine that the object is a falling object. Further, for example, when the object is a rear portion and a side portion of the preceding vehicle 91, the number of distance points A is large and the accuracy of the distance value Zmeas is high, so that the threshold value TF becomes a large value. .. As a result, the distance value D tends to be smaller than the threshold value TF, so that it is easy to determine that the object is not a falling object.

This completes the determination process J2B. The processes of steps S221 to S224 are not limited to this order, and may be replaced.

Next, as shown in FIG. 6, the combination processing unit 23 confirms whether or not all the determination conditions in the determination process J2B are satisfied (step S106). That is, in the combining processing unit 23, the horizontal image width of each of the plurality of image areas R is smaller than the threshold value TA (step S221), and in each of the plurality of image areas R, a plurality of distance values Zmeas of the object. The total value of the accuracy of is smaller than the threshold value TB (step S222), and the height value indicating the height of the object from the road surface in each of the plurality of image areas R is higher than the threshold value TE (step). It is confirmed whether or not the four determination conditions of S223) and the distance value indicating the horizontal separation distance between the plurality of image regions R being larger than the threshold value TF (step S224) are satisfied. If all of these four determination conditions are satisfied (“Y” in step S106), the process proceeds to step S107, and if at least one of these four determination conditions is not satisfied (“N” in step S106). ), The process proceeds to step S108.

In step S104, when all three determination conditions in the determination process J2A are satisfied (“Y” in step S104), or in step S106, when all four determination conditions in the determination process J2B are satisfied (“Y” in step S106). In "), the coupling processing unit 23 determines that each object in the plurality of image regions R is a fallout such as snow particles or rain particles (step S107). This is the end of this flow. That is, in this case, the combination processing unit 23 does not perform the area combination process J for combining these plurality of image areas R.

Further, in step S102, when at least one of the three determination conditions in the simple determination process J1 is not satisfied (“N” in step S102), or in step S106, among the four determination conditions in the determination process J2B. When at least one is not satisfied (“N” in step S106), the coupling processing unit 23 determines that each object in the plurality of image regions R is not a fallout such as snow particles or rain particles (“N”). Step S108). Then, the combination processing unit 23 performs the area combination process J for combining these a plurality of image areas R (step S109). This is the end of this flow.

Here, the three determination conditions in the determination process J2A correspond to a specific example of the "first condition" in the present disclosure. The four determination conditions in the determination process J2B correspond to a specific example of the "second condition" in the present disclosure. The three determination conditions in the simple determination process J1 correspond to a specific example of the "third condition" in the present disclosure. The threshold TC corresponds to a specific example of the "first threshold" in the present disclosure. The threshold TD corresponds to a specific example of the "second threshold" in the present disclosure. The threshold TB corresponds to a specific example of the "third threshold" in the present disclosure. The threshold TF corresponds to a specific example of the "fourth threshold" in the present disclosure. The threshold value TE corresponds to a specific example of the "fifth threshold value" in the present disclosure. The threshold TA corresponds to a specific example of the "sixth threshold" in the present disclosure.

As described above, in the image processing apparatus 1, in the determination process J2A, when a plurality of determination conditions including the fact that the actual size of the object in the vertical direction in the vertical direction is smaller than the threshold value TC are satisfied in each of the plurality of image regions R. , The area join process J is avoided. That is, when the object in the image region R is a descending object such as a snow grain or a raindrop, the actual size of the object in the vertical direction is expected to be small. Is now less than the threshold TC. As a result, in the image processing device 1, when a plurality of determination conditions in the determination process J2A are satisfied, a plurality of image regions R for fallout such as snow particles and rain particles can be prevented from being combined, which is appropriate. Image processing can be performed. As a result, it is possible to reduce the possibility that the vehicle 10 performs an inappropriate operation such as controlling the running of the vehicle 10 or displaying information on the console monitor based on these fallouts.

Further, in the image processing device 1, when the determination process J2B satisfies a plurality of determination conditions including the fact that the distance value indicating the horizontal separation distance between the plurality of image regions R is larger than the threshold value TF, the region The join process J is avoided. That is, when the object in the image region R is a fallout such as a snow grain or a raindrop, the plurality of fallouts are separate fallouts, so that the distance value D is expected to be large. In the processing device 1, it is determined whether or not the distance value is larger than the threshold value TF. As a result, in the image processing device 1, when a plurality of determination conditions in the determination process J2B are satisfied, a plurality of image regions R for fallout such as snow particles and rain particles can be prevented from being combined, which is appropriate. Image processing can be performed.

Further, in the image processing device 1, the simple determination process J1 is performed before the determination processes J2A and J2B. The judgment condition of step S201 in the simple judgment process J1 is the same as the judgment condition of step S221 in the judgment process J2B, and the judgment condition of step S203 in the simple judgment process J1 is the judgment condition and judgment process of step S213 in the judgment process J2A. It is the same as the determination condition of step S222 in J2B. As a result, when the image processing apparatus 1 does not satisfy these determination conditions, the simple determination process J1 can make a determination without performing the determination processes J2A and J2B, so that the processing amount in the processing unit 20 can be determined. Can be reduced.

Further, in the image processing device 1, in the simple determination process J1, the determination processes J2A and J2B are performed when a plurality of determination conditions including the fact that the distance values Zmeas of the objects are not continuous in the plurality of image areas R are satisfied. I made it. That is, when the object in the image region R is a descent such as a snow grain or a raindrop, the distance value Zmeas is different between the loci, and it is expected that the distance value Zmeas is not continuous. Therefore, image processing is performed. In the device 1, it is determined whether or not the distance value Zmeas of the object has continuity. As a result, the image processing device 1 performs more detailed determination processes J2A and J2B when a plurality of determination conditions in the simple determination process J1 are satisfied, and according to the determination result, fallout such as snow particles and raindrops. Since it is possible to prevent the plurality of image regions R of the above from being combined, image processing can be appropriately performed.

[effect]
As described above, in the present embodiment, in the determination process J2A, when a plurality of determination conditions including the fact that the actual size of the object in the vertical direction in the vertical direction is smaller than the threshold value TC are satisfied in each of the plurality of image regions, the region Since the combination processing J is avoided, the image processing can be appropriately performed.

In the present embodiment, in the determination process J2B, when a plurality of determination conditions including that the distance value indicating the horizontal separation distance between the plurality of image regions is larger than the threshold value TF are satisfied, the region combination process J Since the above is avoided, image processing can be performed appropriately.

In the present embodiment, in the simple determination process J1, the determination processes J2A and J2B are performed when a plurality of determination conditions including the fact that the distance values of the objects are not continuous in the plurality of image regions are satisfied. Image processing can be performed appropriately.

Although the present technology has been described above with reference to the embodiments, the present technology is not limited to these embodiments and can be modified in various ways.

For example, in the above embodiment, as shown in FIG. 6, the coupling processing unit 23 performs the determination processing J2A (step S103) and does not satisfy at least one of the three determination conditions of the determination processing J2A. In the case (“N” in step S104), the determination process J2B is performed (step S105), but the present invention is not limited to this. Instead, for example, when the coupling processing unit 23 performs the determination processing J2B (step S113) as shown in FIG. 15 and does not satisfy at least one of the four determination conditions of the determination processing J2B. The determination process J2A may be performed (step S115) in (“N” in step S114). Then, when all three determination conditions of the determination process J2A are satisfied (“Y” in step S116), the coupling processing unit 23 determines that the object is a fallout (step S107), and determines that the object is a fallout (step S107). If at least one of the three determination conditions is not satisfied (“N” in step S116), it may be determined that the object is not fallout (step S108).

Further, for example, in the above embodiment, three determination conditions are provided in the simple determination process J1, but the present invention is not limited to this, and instead, for example, two or less determination conditions may be provided. Alternatively, four or more determination conditions may be provided. Similarly, in the above-described embodiment, the determination process J2A is provided with three determination conditions, but the present invention is not limited to this, and instead, for example, two or less determination conditions may be provided. Four or more determination conditions may be provided. Similarly, in the above embodiment, four determination conditions are provided in the determination process J2B, but the present invention is not limited to this, and instead, for example, four or less determination conditions may be provided. Five or more determination conditions may be provided.

Further, for example, in the above embodiment, the stereo camera 11 images the front of the vehicle 10, but the present invention is not limited to this, and for example, the side or the rear of the vehicle 10 may be imaged. ..

It should be noted that the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

1 ... image processing device, 10 ... vehicle, 11 ... stereo camera, 11L ... left camera, 11R ... right camera, 20 ... processing unit, 21 ... distance image generation unit, 22 ... image area setting unit, 23 ... combination processing unit, 100, 121-123 ... Trajectory, A ... Distance point, J ... Area combination processing, J1 ... Simple judgment processing, J2A, J2B ... Judgment processing, PIC ... Stereo image, PL ... Left image, PR ... Right image, PZ ... Distance Image, R, R1, R2, R21 to R23 ... Image area, TA to TF ... Threshold, Zmeas ... Distance value.

Claims (11)

In a stereo image, an image area setting unit that sets a first image area including an image of a first object and a second image area including an image of a second object, and an image area setting unit.
It is possible to perform a combining process for combining the first image region and the second image region, and based on the stereo image and the distance image generated based on the stereo image, the first object. The first actual size, which is the actual size in the vertical direction, and the second actual size, which is the actual size in the vertical direction of the second object, are calculated, and the first actual size and the second actual size are calculated. An image processing apparatus including a coupling processing unit that avoids the coupling processing when the first condition including that is smaller than the first threshold value is satisfied. The distance image includes a distance value at each pixel.
The combination processing unit calculates the number of pixels related to the first object in the first image region in the distance image as a first distance point, and also calculates the number of pixels in the second image region. The number of the pixels related to the second object is calculated as the second distance point, and the number is calculated.
The image processing apparatus according to claim 1, wherein the first condition includes the first distance points and the second distance points smaller than the second threshold value. The distance image includes a distance value at each pixel.
Based on the stereo image and the distance image, the combination processing unit calculates a first total value which is a total value of the accuracy of a plurality of the distance values in the first image region in the distance image. At the same time, a second total value, which is the total value of the accuracy of the plurality of distance values in the second image region, is calculated.
The image processing apparatus according to claim 1 or 2, wherein the first condition includes that the first total value and the second total value are smaller than the third threshold value. The coupling processing unit calculates a distance value indicating a horizontal separation distance between the first object and the second object, and includes that the distance value is larger than a fourth threshold value. The image processing apparatus according to claim 1, wherein the coupling process is avoided when the condition of 2 is satisfied. The distance image includes a distance value at each pixel.
The combination processing unit is based on the accuracy of the plurality of distance values in the first image region in the distance image and the accuracy of the plurality of distance values in the second image region. The image processing apparatus according to claim 4, wherein the threshold value is set. The imaging unit that generates the stereo image is mounted on the vehicle and
Based on the stereo image and the distance image, the coupling processing unit has a first height value indicating the height of the first object from the road surface of the traveling road on which the vehicle travels, and a first height value from the road surface. A second height value indicating the height of the second object is calculated.
The image processing apparatus according to claim 4 or 5, wherein the second condition includes that the first height value and the second height value are larger than the fifth threshold value. The distance image includes a distance value at each pixel.
Based on the stereo image and the distance image, the combination processing unit calculates a first total value which is a total value of the accuracy of a plurality of the distance values in the first image region in the distance image. At the same time, a second total value, which is the total value of the accuracy of the plurality of distance values in the second image region, is calculated.
The image processing according to any one of claims 4 to 6, wherein the second condition includes that the first total value and the second total value are smaller than the third threshold value. apparatus. The second condition is any one of claims 4 to 7, which includes that the width of the first image area and the width of the second image area are smaller than the sixth threshold value. The image processing apparatus described. The distance image includes a distance value at each pixel.
The coupling processing unit has continuity in the plurality of distance values in the image of the first object included in the first image region and the image of the second object included in the second image region. The image processing apparatus according to any one of claims 4 to 8, which determines whether or not the first condition and the second condition are satisfied when the third condition including the absence is satisfied. The image processing apparatus according to any one of claims 1 to 9, wherein each of the first object and the second object is a fallout. The image processing device according to claim 10, wherein the fallout is snow.

Images