JP6983334B2 - Image recognition device - Google Patents

Description

The present invention relates to an image recognition device.

In recent years, there has been an increasing demand for improved performance of image recognition devices required for driving support and automatic driving. For example, in the collision safety function for pedestrians, performance improvement is required, such as the addition of a collision safety test for night pedestrians in an automobile assessment. In order to achieve this, high recognition performance for three-dimensional objects such as pedestrians is required.

Patent Document 1 describes, in a situation where a moving three-dimensional object and another three-dimensional object overlap each other, a pedestrian or the like existing inside the region by tracking the feature points inside a predetermined region containing the three-dimensional object. A recognition device for detecting a moving three-dimensional object has been proposed.

Japanese Unexamined Patent Publication No. 2017-142760

However, the conventional device has a problem that the recognition performance is deteriorated when the entire three-dimensional object cannot be detected well.

Image recognition apparatus according to the first aspect of the present invention, the detection area of the three-dimensional object that has been set on the captured image by the imaging unit, and enlarged or reduced on the image based on the detection characteristics information of said three-dimensional object This recognizes the three-dimensional object area setting unit that sets the three-dimensional object area on the image and the recognition process that specifies the type of the three-dimensional object for the three-dimensional object area set by the three-dimensional object area setting unit. It is equipped with a processing unit.
Image recognition device according to the second aspect of the present invention, or a larger detection area of the three-dimensional object that has been set on the captured image by the imaging unit on the first on the basis of the characteristic information image of the three-dimensional object By reducing the size, the third characteristic information of the three-dimensional object is based on the three-dimensional object area setting unit that sets the three-dimensional object region on the image and the three-dimensional object region obtained by the three-dimensional object region setting unit as a reference size. Based on the recognition magnification setting unit that defines recognition regions of a plurality of sizes, and the recognition regions defined by the recognition magnification setting unit, from the recognition region based on the third characteristic information of the three-dimensional object. It also includes a scanning area setting unit that sets a plurality of wide scanning areas, and a recognition processing unit that performs recognition processing for specifying the type of the three-dimensional object using the scanning area set by the scanning area setting unit. ..
The image recognition device according to the third aspect of the present invention has the detection area of the three-dimensional object set on the image captured by the imaging unit based on the detection characteristic information of the three-dimensional object. A recognition process that specifies the type of the three-dimensional object for the three-dimensional object area set by the three-dimensional object area setting unit and the three-dimensional object area setting unit that sets the three-dimensional object area by enlarging or reducing the A unit and a recognition magnification setting unit that defines recognition areas of a plurality of sizes based on the recognition characteristic information of the three-dimensional object, with the three-dimensional object area obtained by the three-dimensional object area setting unit as a reference size, are provided. The recognition processing unit performs the recognition processing on each of the recognition areas of a plurality of sizes defined by the recognition magnification setting unit.

According to the present invention, it is possible to provide an image recognition device that accurately detects a three-dimensional object and has improved recognition performance.

It is a block diagram which shows the whole structure of an image recognition apparatus. It is a flowchart which shows the operation of an image recognition apparatus. It is a figure which shows the 3D object area set on the image by the detection process. It is a flowchart which shows the detail of the recognition process. It is a figure explaining the principle of a three-dimensional object area setting process. It is a figure explaining the principle of the recognition magnification setting process. It is a figure explaining the normalization in the recognition magnification setting process. It is a figure explaining the principle of a scanning area setting process. It is a figure explaining the principle of the scan recognition processing for every magnification. It is a figure explaining the principle of the optimum magnification setting process. It is a figure explaining the principle of a detailed recognition position determination process. It is a figure explaining the principle of a detailed recognition process. It is a block diagram which shows the whole structure of the image recognition apparatus which concerns on the modification.

FIG. 1 is a block diagram showing an overall configuration of an image recognition device 100 according to the present embodiment. The image recognition device 100 includes a left camera 101 and a right camera 102 mounted on the vehicle and arranged on the left and right in front of the vehicle. The cameras 101 and 102 constitute a stereo camera, and image a three-dimensional object such as a pedestrian, a vehicle, a signal, a sign, a white line, a tail lamp of a car, and a headlight. The image recognition device 100 recognizes the environment outside the vehicle based on the image information in front of the vehicle captured by the cameras 101 and 102. Then, the vehicle (own vehicle) controls the brake, steering, and the like based on the recognition result by the image recognition device 100.

The image recognition device 100 captures the images captured by the cameras 101 and 102 from the image input interface 103. The image information taken in from the image input interface 103 is sent to the image processing unit 104 via the internal bus 109. Then, it is processed by the arithmetic processing unit 105, and the result in the process of processing, the image information of the final result, and the like are stored in the storage unit 106.

The image processing unit 104 compares the first image obtained from the image sensor of the left camera 101 with the second image obtained from the image sensor of the right camera 102, and makes the image sensor for each image. The resulting device-specific deviation is corrected, image correction such as noise interpolation is performed, and this is stored in the storage unit 106. Further, the points corresponding to each other between the first image and the second image are calculated, the parallax information is obtained, and this is stored in the storage unit 106 as the distance information corresponding to each pixel on the image. do. The image processing unit 104 is connected to the arithmetic processing unit 105, the CAN interface 107, and the control processing unit 108 via the internal bus 109.

The arithmetic processing unit 105 recognizes a three-dimensional object in order to grasp the environment around the vehicle by using the image information and the distance information (parallax information) stored in the storage unit 106. A part of the recognition result of the three-dimensional object and the intermediate processing result is stored in the storage unit 106. The arithmetic processing unit 105 recognizes a three-dimensional object with respect to the captured image, and then calculates the vehicle control using the recognition result. The vehicle control policy obtained as a result of the vehicle control calculation and a part of the recognition result are transmitted to the vehicle-mounted network CAN 110 via the CAN interface 107, whereby the vehicle is controlled.

The control processing unit 108 monitors whether each processing unit has caused an abnormal operation, whether an error has occurred during data transfer, and the like, and prevents the abnormal operation. The image processing unit 104, the arithmetic processing unit 105, and the control processing unit 108 may be configured by a single computer unit or a plurality of computer units.

FIG. 2 is a flowchart showing the operation of the image recognition device 100.
An image is captured by the left camera 101 and the right camera 102 provided in the image recognition device 100, and each of the captured image information 203 and 204 is an image such as a correction for absorbing the peculiar habit of the image sensor. Process 205 is performed. The processing result of the image processing 205 is stored in the image buffer 206. The image buffer 206 is provided in the storage unit 106 of FIG.

Next, the parallax processing 207 is performed. Specifically, the two corrected images are collated with each other, and the parallax information of the images obtained by the left camera 101 and the right camera 102 is obtained. By the parallax of the left and right images, a certain point of interest on the image of the three-dimensional object is obtained as the distance to the three-dimensional object by the principle of triangulation. The image processing 205 and the parallax processing 207 are performed by the image processing unit 104 of FIG. 1, and the finally obtained image information and the parallax information are stored in the storage unit 106.

Then, in the next detection process 208, a three-dimensional object in the three-dimensional space is detected by using the parallax information obtained by the parallax process 207 for the parallax or the distance of each pixel of the left and right images. FIG. 3 is a diagram showing a detection area of a three-dimensional object set on the image by the detection process 208. FIG. 3 shows a pedestrian detection area 301 and a vehicle detection area 302 detected by the cameras 101 and 102 on the image as a result of the detection process 208. These detection regions 301 and 302 may be rectangular as shown in FIG. 3, or may be irregular regions obtained from parallax or distance. It is generally treated as a rectangle in order to facilitate the handling by a computer in the subsequent processing. In the present embodiment, the area will be treated as a rectangle, and a pedestrian will be mainly used as an example of a three-dimensional object.

Next, in the recognition process 209, a recognition process for specifying the type of the three-dimensional object is performed for the detection area set on the image by the detection process 208. The three-dimensional object to be recognized by the recognition process 209 is, for example, a pedestrian, a vehicle, a signal, a sign, a white line, a tail lamp of a car, a headlight, or the like, and the type of any of these is specified. In order for the recognition process 209 to stably recognize a three-dimensional object, it is necessary that the detection area on the image and the area of the target to be recognized match. However, in the cameras 101 and 102, it may not be possible to completely match the areas on the image to be recognized due to the brightness of the external environment and the variation in the imaging performance between the cameras. This is the same even when a radar such as a millimeter wave is combined with an image sensor such as a camera. The details of the recognition process 209 that solves this problem will be described later.

Next, in the vehicle control process 210, for example, a warning is issued to the occupant in consideration of the recognition result of the three-dimensional object and the state of the own vehicle (speed, steering angle, etc.), and the braking and steering angle adjustment of the own vehicle are performed. And so on. Alternatively, avoidance control for the recognized three-dimensional object is determined, and the result is output as automatic control information via the CAN interface 107. The recognition process 209 and the vehicle control process 210 are performed by the arithmetic processing unit 105 of FIG.

The program shown in the flowchart of FIG. 2 and the flowchart of FIG. 4 described later can be executed by a computer equipped with a CPU, a memory, and the like. All processing or some processing may be realized by a hard logic circuit. Further, this program can be previously stored and provided in the storage medium of the image recognition device 100. Alternatively, the program can be stored and provided in an independent storage medium, or the program can be recorded and stored in the storage medium of the image recognition device 100 by a network line. It may be supplied as a computer-readable computer program product in various forms such as a data signal (carrier wave).

FIG. 4 is a flowchart showing the details of the recognition process 209. As shown in FIG. 4, this flowchart shows a three-dimensional object area setting process 401, a recognition magnification setting process 402, a scanning area setting process 403, a magnification-by-scanning recognition process 404, an optimum magnification setting process 406, and a detailed recognition position determination process 407. The detailed recognition process 408 is performed. Hereinafter, each process will be described in order. In addition, these processes will be described on the premise of a stereo camera.

[Three-dimensional object area setting process]
In the three-dimensional object area setting process 401, the three-dimensional object detection area 301 obtained by the detection process 208 is enlarged or reduced based on the three-dimensional object detection characteristic information to set the three-dimensional object area 501.

FIG. 5 is a diagram illustrating the principle of the three-dimensional object area setting process 401. FIG. 5 shows an example in which the pedestrian detection area 301 is enlarged based on the detection characteristic information of the three-dimensional object to set the three-dimensional object area 501. The detection characteristic information includes, for example, (1) distinctiveness of a three-dimensional object, (2) distance from the three-dimensional object, (3) size of the three-dimensional object, (4) assumed size of the three-dimensional object, and (5) brightness of the external environment. These include (6) the direction of the headlight, (7) the height of the road surface on which the three-dimensional object exists, and (8) the sensor resolution. The detection characteristic information will be described below.

(1) As for the distinctiveness of a three-dimensional object, for example, in cameras 101 and 102, it may be difficult to obtain a three-dimensional object depending on the combination with the background area. This includes the clothes of pedestrians of the same color as the road surface and the crown of pedestrians at night. It is also conceivable that the cameras 101 and 102 are partially blurred due to the influence of raindrops and the like, and the three-dimensional object region is missing. In such a case, the detection area is expanded. In addition, there are cases where a person and a non-human area in a three-dimensional space are combined to form a three-dimensional object. This is because it is difficult to separate before identification, such as the connection between a person and a three-dimensional object such as a utility pole or fence on the shoulder of a road. In such a case, the detection area is reduced based on the color, brightness, and edge of the image. Further, when another type of sensor such as a radar sensor or the cameras 101 and 102 have different mounting heights in the horizontal direction, hiding occurs mainly in the upper direction, and the three-dimensional object appears small. If it has the processing characteristics of such a configuration, the detection area is expanded.

(2) As for the distance to the three-dimensional object, if it is far, the three-dimensional object area is greatly expanded, and if it is close, the three-dimensional object area is reduced. This magnification may be determined by the sensor resolution including the cameras 101 and 102. This is because the farther the object is, the larger the size that occupies the three-dimensional space per pixel becomes, and the error is added.

(3) As for the size of the three-dimensional object, if the three-dimensional object is small, the three-dimensional object area is expanded, and if the three-dimensional object is large, the three-dimensional object area is reduced.

(4) The assumed size of a three-dimensional object is, for example, assuming that the three-dimensional object is a pedestrian, and the three-dimensional object that is too small assuming that it is a pedestrian expands the three-dimensional object area and is too large assuming that it is a pedestrian. Reduces the three-dimensional object area. The extent to which the target is targeted may be determined in consideration of (5) the brightness of the external environment and (6) the direction of the headlights. For example, in a bright environment in the daytime, the three-dimensional object area is reduced, and in a dark environment at night, the three-dimensional object area is expanded. Further, depending on the direction of the headlight, for example, if the direction of the headlight is a low beam, the light hits the feet, so that the three-dimensional object region is expanded in the height direction. If the direction of the headlight is high beam, the whole body is exposed to light, so the three-dimensional object area is reduced. Further, the three-dimensional object region may be expanded or reduced depending on the distance to the three-dimensional object and (7) the height of the road surface at the position where the three-dimensional object exists. For example, when the height of the road surface is low, if the high beam is used, the light does not hit the feet, so the three-dimensional object area is expanded downward.

(8) If the sensor is a camera 101 or 102, the size per pixel changes depending on the distance, so the sensor resolution expands or contracts the three-dimensional object area by combining with the size of the three-dimensional object and the target distance. do. For example, when a three-dimensional object is in the vicinity, the resolution of the three-dimensional space per pixel is high, so that the three-dimensional object area is expanded. When the three-dimensional object is far away, the resolution of the three-dimensional space per pixel is low, so the three-dimensional object area is reduced. Further, depending on the characteristics of the area acquired as the three-dimensional object area, the three-dimensional object area is reduced. The characteristic of the region acquired as the three-dimensional object region is, for example, the case where the parallax region or the sensor response region where the three-dimensional object region is obtained is set to be larger, and in such a case, the three-dimensional object region is reduced.

An example of setting the three-dimensional object region 501 based on the detection characteristic information of the three-dimensional object will be described. For example, in a pedestrian, the limb region where the change on the image is large is likely to be chipped and become small. Also, at night, if you have black hair, your head will be mixed with the background and it will be difficult to detect. In such a case, the three-dimensional object region 501 on the image is changed based on the size in the three-dimensional space per pixel. For example, in the bright daytime, the expansion of the region considered to be the head is 0 cm, the expansion of the foot region is 10 cm, and in the nighttime, the expansion of the head region is 10 cm, and the expansion of the foot region is 10 cm. Further, if the low beam of the vehicle is within reach, the expansion of the head area is 10 cm, and the expansion of the foot area is 0 cm. Similarly, the width is also enlarged or reduced as appropriate. Further, the correction may be performed based on the amount of change in the width over time. Further, the recognition area may be reduced depending on the content of the recognition process in the subsequent stage. For example, in the case of a pedestrian, recognition is performed only by the upper body. The size to be enlarged or reduced may be determined by a predetermined ratio or the size on the image, but by setting the size on the three-dimensional space as a reference, it is possible to exclude a size that cannot be recognized.
Further, the size of the three-dimensional object region 501 due to this enlargement / reduction may be the same as that of the detection region 301 due to the relationship between the distance in the three-dimensional space and the pixels.

Although the detection characteristic information has been described above, in the three-dimensional object area setting process 401, the three-dimensional object area 501 is set more accurately by combining a plurality of these detection characteristic information. For example, by combining the distance, the brightness of the outside environment, the direction of the light, the height of the road surface, the sensor resolution, and the like, the three-dimensional object region 501 that further reduces the influence of day and night is set. Further, the number of pixels and the size described here are examples, and are not limited to this range.

[Recognition magnification setting process]
Next, the recognition magnification setting process 402 shown in FIG. 4 will be described.
Assuming that the three-dimensional object region 501 set in the three-dimensional object region setting process 401 is the reference size of the recognition process 209, this reference size is not always the optimum recognition area for recognition. Therefore, in the recognition magnification setting process 402, recognition areas of a plurality of sizes are determined by using the recognition characteristic information of the three-dimensional object. At this time, since the optimum recognition area is unknown, the recognition area is expanded or reduced based on the reference size to determine recognition areas of a plurality of sizes. The recognition characteristic information includes, for example, (1) the distance to the three-dimensional object, (2) the size of the three-dimensional object, (3) the limit size of the three-dimensional object, and (4) the sensor resolution. These recognition characteristic information will be described below.

(1) The distance to the three-dimensional object is an index for determining the amount of enlargement or reduction when the recognition area is expanded or reduced. For example, when the three-dimensional object is far away, the size of the three-dimensional object per pixel becomes large. In this case, since the input of the classifier 901, which will be described later, to perform the recognition process is an image, when the three-dimensional object is far away, the number of pixels to be enlarged or reduced is smaller than when the three-dimensional object is near. Therefore, based on the reference size, the recognition area is expanded or reduced according to the distance to the three-dimensional object, and the recognition areas of a plurality of sizes are defined.

(2) The size of a three-dimensional object is an index for determining the amount of enlargement or reduction when the recognition area is enlarged or reduced. For example, when the three-dimensional object is large, the number of pixels when enlarging or reducing the image is smaller than when the three-dimensional object is small. Further, when the recognition areas of a plurality of sizes are determined based on the sizes in the real space, the number of pixels when enlarging or reducing the image is smaller when the area is far away than when the area is near the area. If the recognition area is not set for each sub-pixel, it may be the same as the standard size.

(3) The limit size of a three-dimensional object is a limit size assumed when the three-dimensional object is a recognition target. For example, when the three-dimensional object is a pedestrian, if the height of the three-dimensional object is large, such as exceeding 2.5 meters, the area is set in the reduction direction. On the contrary, if the height is small, such as less than 0.8 meters, the area is set in the expansion direction. If it is in the middle, set the area on both sides. The upper and lower limits of the area to be set may be determined from the three-dimensional object to be recognized, the limitation of the recognition process, and the like.

(4) As for the sensor resolution, for example, if the sensors are cameras 101 and 102, the size per pixel changes according to the distance. Therefore, the range of enlargement or reduction can be determined based on the sensor resolution. For example, in a distant place where the size in the three-dimensional space per pixel exceeds 20 cm, the range of enlargement or reduction is defined as a small range of 1 pixel and 2 pixels. On the contrary, at a short distance where the size in the three-dimensional space per pixel is less than 1 cm, it is enlarged or reduced in a large range such as 10 pixels or 20 pixels.

The size on the image may be obtained by back calculation from the size of a three-dimensional object in the three-dimensional space. Further, regarding the detection characteristic information that is not considered in the setting of the three-dimensional object area 501 in the three-dimensional object area setting process 401, the variation of the recognition area may be set by using the detection characteristic information in the recognition magnification setting process 402. In that case, which condition of the detection characteristic information the recognition area is expanded or contracted is the same as the content described in the three-dimensional object area setting process 401. Further, the number of pixels and the size described here are examples, and are not limited to this range.

FIG. 6 is a diagram illustrating the principle of the recognition magnification setting process 402. The recognition magnification setting process 402 defines the recognition areas 601 and 602 in which the three-dimensional object area 501 is set as the reference size recognition area and is enlarged or reduced. The recognition area 501 is a reference size, the recognition area 601 is a recognition area having a reduced reference size and a small recognition magnification, and the recognition area 602 is a recognition area having an enlarged reference size and a large recognition magnification. In the example of FIG. 6, two types of recognition areas enlarged or reduced with respect to the reference size are shown, but this number may have many variations if there is a margin in the recognition processing time. Further, only enlargement or reduction may be set by setting the detection process 208 or the three-dimensional object area setting process 401. The amount of enlargement or reduction of the recognition area is set based on the recognition characteristic information. In this case, similar to the three-dimensional object area setting process 401, it may be the same as the recognition area of the reference size depending on the resolution of the image.

FIG. 7 is a diagram illustrating normalization in the recognition magnification setting process 402.
As shown in FIG. 7, the area for normalizing the recognition area (501, 601 and 602) when the recognition process in the subsequent stage is performed is defined. The recognition area indicates a range in which the recognition process is performed in the recognition process described later. The recognition area 501 is a reference size, the recognition area 601 is a recognition area having a reduced reference size and a small recognition magnification, and the recognition area 602 is a recognition area having an enlarged reference size and a large recognition magnification.

In the recognition process, it is necessary to match the number of dimensions of the input information. There is no guarantee that the recognition area 501 of the reference size will capture the target object neatly, and how it should be captured depends on the characteristics of the recognition process mounted on the device. Therefore, the area to be normalized is set in advance. In the example of FIG. 7, the recognition area 501 contains the head and the foot, whereas the recognition area 601 having a small recognition magnification has the crown and the limbs protruding, and the recognition area 602 having a large recognition magnification has the head on the contrary. There are margins on the top and feet. When these recognition areas are normalized to the same size, as shown in FIG. 7, the normalized recognition areas are 701, 702, and 703, and the same processing can be performed in the recognition processing described later. However, this normalization process is not always performed by the recognition magnification setting process 402. It may be carried out as a part of the processing of the scan recognition process 404 for each magnification described later and the detailed recognition process 408 described later.

[Scanning area setting process]
Next, the scanning area setting process 403 of FIG. 4 will be described. The scanning area setting process 403 sets a scanning area larger than the recognition area for each recognition area based on the arrangement characteristic information of the three-dimensional object. The scanning area is set as an area on the image, and in the recognition process, the inside of the set scanning area is scanned by the recognition area. That is, the recognition area indicates a range in which the recognition process is performed in the recognition process described later, and the scanning area is a range in which the recognition area is moved within the range of the scanning area. As a result, the recognition process is performed while moving the recognition area within the range of the scanning area. The arrangement characteristic information that determines the size of the scanning region is, for example, (1) the perspective position of the three-dimensional object, (2) the height of the road surface on which the three-dimensional object exists, and the like. The arrangement characteristic information will be described below.

(1) The perspective position of a three-dimensional object is an index when setting the scanning area. For example, when a three-dimensional object is nearby, the scanning area on the image is largely defined. If the three-dimensional object is far away, the scanning area is set small. This is because the sensor resolution is high when the sensor is near, and the scanning amount in the three-dimensional space when scanning one pixel is about several mm, whereas when it is far away, it exceeds 10 cm. The scanning area is also determined by the characteristics such as the amount of deviation of detection generated by the detection of a three-dimensional object. For example, when the recognition process that exerts the best performance when the horizontal position center of the three-dimensional object is used, the deviation amount and dispersion between the horizontal position center of the three-dimensional object and the horizontal position center of the actual recognition target are taken into consideration in the scanning area. It may be set so that the center of the horizontal position of the recognition target fits.

(2) The height of the road surface where the three-dimensional object exists is an index when setting the scanning area. For example, when the road surface is raised and a three-dimensional object (pedestrian, etc.) is at a higher position than the own vehicle, the hiding on the head side increases and the height becomes smaller than the actual height. In addition, when a three-dimensional object (pedestrian, etc.) is in a low position, it is possible that the feet may be cut off or hidden by a bumper depending on the angle of view. The scanning area is enlarged or reduced according to such a state.

Further, regarding the detection characteristic information and the recognition characteristic information that are not considered in the setting of the three-dimensional object area and the recognition area in the three-dimensional object area setting process 401 and the recognition magnification setting process 402, the scanning area is used in the scanning area setting process 403. May be determined. In this case, whether the scanning area is expanded or contracted according to which condition is the same as in the three-dimensional object area setting process 401 and the recognition magnification setting process 402. Further, the number of pixels and the size described here are examples, and are not limited to this range.

FIG. 8 is a diagram illustrating the principle of the scanning area setting process 403. The scanning area setting process 403 defines scanning areas 801, 802, and 803 for each recognition area 501, 601, 602, respectively. The scanning areas 801 and 802, 803 are the same as or larger than the recognition areas 501, 601 and 602. However, since the inside of the scanning areas 801, 802, and 803 is scanned in the recognition areas 501, 601 and 602, the scanning amount is not always large. The scanning regions 801, 802, and 803 define regions on the image from the arrangement characteristic information. At this time, depending on the resolution of the image, the recognition area and the scanning area may be the same on the image. The scanning area is individually determined for each recognition area, but if there is a margin in the processing time, one having the largest scanning area may be adopted. Further, if the processing time is not sufficient, one small scanning area may be applied to each recognition area.

[Scanning recognition process for each magnification]
Next, the scan recognition process 404 for each magnification shown in FIG. 4 will be described. In the magnification-by-magnification scanning recognition process 404, the images corresponding to the scanning areas 801, 802, and 803 and the disparity area (distance area) are scanned in the recognition areas 501, 601 and 602, and the recognition process is performed for each scanning position of each size. It is determined whether the scanning position of the target is a three-dimensional object.

Here, if the performance of the recognition process is sufficient, the vehicle control process 210 may be performed using the result of the scan recognition process 404 for each magnification as shown by the broken line 405 of FIG. The scan recognition process 404 for each magnification may have a plurality of results depending on the magnification, scanning position, etc., but this is narrowed down by a process such as selecting the one with the best recognition result.

FIG. 9 is a diagram illustrating the principle of the scan recognition process 404 for each magnification. The response position 902 of the result recognized by the classifier 901 that performs the recognition process is obtained while scanning the inside of each of the scanning areas 801, 802, and 803 in the recognition areas 501, 601 and 602. The response position 902 is indicated by x in FIG. The larger the number of response positions 902, the better the recognition process. As an example of the result of recognizing the inside of the scanning areas 801, 802, and 803 by the classifier 901, the scanning area 801'is the largest as shown by the response position 902 of the scanning areas 801', 802', 803'.

The classifier 901 may use machine learning or heuristic threshold determination. If this determination result is sufficient, recognition may be completed using this result as shown by the broken line 405 of FIG. In that case, for example, the one with the best recognition processing is adopted.

In the magnification per scan recognition process 404, when the performance of the recognition process is insufficient due to reduction of the calculation cost of the recognition process or the like, detailed processing may be performed using the result of the magnification per scan recognition process 404. .. In the present embodiment, a case where the optimum magnification setting process 406, the detailed recognition position determination process 407, and the detailed recognition process 408 are provided as detailed processes will be described.

[Optimal magnification setting process]
The optimum magnification setting process 406 shown in FIG. 4 selects the optimum recognition area for the detailed recognition process from the recognition areas of a plurality of sizes created by the recognition magnification setting process 402. As the selection method, for example, the number of responses determined to be the recognition target and its reliability in the recognition processing result obtained by scanning, the number determined to be the non-recognition target and its reliability, the distribution of the recognition result, and the like are used. The amount and reliability of are compared between recognition areas of multiple sizes, and the optimum recognition area is used. The optimum magnification setting process 406 may be omitted if there is not sufficient grace in the processing time.

FIG. 10 is a diagram illustrating the principle of the optimum magnification setting process 406. From the recognition results of multiple magnifications, select the optimum magnification with the best response. As described above, the optimum magnification is selected by using the number of responses in the scanning area of the recognition process and its reliability. In the example of FIG. 10, the scanning region 801'with the largest number of responses is selected, but this scanning region 801'corresponds to the reference size scanning region 801 and the reference size scanning region 801 recognizes the reference size. It corresponds to the area 501.

[Detailed recognition position determination process]
The detailed recognition position determination process 407 shown in FIG. 4 determines a representative position for performing detailed recognition for the optimum magnification obtained by the optimum magnification setting process 406. For detailed recognition, for example, the position where the reliability of the recognition process obtained by the scan recognition process 404 for each magnification is maximum is selected. Alternatively, the position may be determined by using a clustering means such as the mean shift method (Mean Shift method). When the optimum magnification setting process 406 is not performed, the detailed recognition position determination process 407 may be performed for each magnification.

FIG. 11 is a diagram illustrating the principle of the detailed recognition position determination process 407. From one or more response positions obtained from the scan recognition process 404 for each magnification, the representative position 111 to perform the detailed recognition process 408 is determined. If there are multiple reaction points, a clustering technique such as the Mean Shift method is used. The area centered on the determined representative position 111 is the detailed identification area.

[Detailed recognition processing]
The detailed recognition process 408 shown in FIG. 4 performs detailed recognition on the representative position 111 determined by the detailed recognition position determination process 407, and calculates the type and reliability of the target. Alternatively, detailed recognition is performed using a recognition area of the optimum size selected based on the response position by the scan recognition process 404 for each magnification, and the type and reliability of the target are calculated. The detailed recognition process 408 uses a classifier 120 having a classification performance equal to or higher than that of the recognition process used in the magnification-based scanning recognition process 404.

FIG. 12 is a diagram illustrating the principle of the detailed recognition process 408. The representative position 111 obtained by the detailed recognition position determination process 407 is subjected to detailed recognition processing using the classifier 120 to determine the type of the three-dimensional object. The types of three-dimensional objects are, for example, pedestrians, vehicles, traffic lights, signs, white lines, tail lamps and headlights of vehicles.

Examples of the recognition process used in the magnification-by-magnification scan recognition process 404 and the detailed recognition process 408 include the following techniques. A technology that uses template matching that compares a recognition area with a template that has the appearance of a recognition target prepared in advance. Technology that uses a classifier that combines features such as brightness images, HOG, and Haar-Like with machine learning methods such as support vector machines, Ada-Boost, and Deep Learning. Further, the edge shape or the like may be recognized by an artificially determined threshold value determination. The scan recognition process 404 for each magnification and the detailed recognition process 408 include image processing such as resizing, smoothing, edge extraction, normalization, isolated point removal, gradient extraction, color conversion, and histogram creation necessary for performing these processes. ..

(Modification example)
In this embodiment, the image recognition device 100 using a stereo camera has been described. However, it may be realized by using an image recognition device 100'that does not use a stereo camera.
FIG. 13 is a diagram showing a processing operation in the image recognition device 100'. The same parts as those of the image recognition device 100 shown in FIG. 2 are designated by the same reference numerals, and the description thereof will be omitted.

The image recognition device 100'includes an optical camera 1301 and a radar sensor 1302. As a result, a three-dimensional object is detected. An image is captured by the optical camera 1301, and the captured image information is subjected to image processing 205 such as correction for absorbing the peculiar habit of the image sensor. The processing result of the image processing 205 is stored in the image buffer 206. Further, the radar sensor 1302 can obtain the distance to the three-dimensional object. The detection process 1303 detects a three-dimensional object in a three-dimensional space based on the distance to the three-dimensional object. The recognition process 209 performs a recognition process for specifying the type of the three-dimensional object for the detection area set by the detection process 1303.

The detection process 1303, which inputs the distance from the radar sensor 1302 to the output stereoscopic object, needs to perform the detection process in consideration of the sensor characteristics of the radar sensor 1302 used for the distance measurement, but after determining the detection area. The processing can be performed in the same manner as the configuration by the stereo camera described in the image recognition device 100. Further, the image recognition device 100'does not require a plurality of images in the image processing 205.

According to the embodiment described above, the following effects can be obtained.
(1) The image recognition devices 100 and 100'are for the three-dimensional object detection area 301 set on the image captured by the cameras 101 and 102, based on the three-dimensional object detection characteristic information. Performs recognition processing for specifying the type of three-dimensional object for the three-dimensional object area setting processing 401 that sets the three-dimensional object area 501 by enlarging or reducing 301 and the three-dimensional object area 501 set by the three-dimensional object area setting process 401. The recognition process 209 and the like are provided. The detection characteristic information includes, for example, the distinctiveness of a three-dimensional object, the distance to the three-dimensional object, the size of the three-dimensional object, the assumed size of the three-dimensional object, the brightness of the external environment, the direction of the headlight, and the height of the road surface on which the three-dimensional object exists. It is at least one of the sensor resolutions of the image pickup unit. This makes it possible to provide an image recognition device that accurately detects a three-dimensional object and has improved recognition performance.

(2) The image recognition devices 100 and 100'are based on the first characteristic information of the three-dimensional object with respect to the detection area 301 of the three-dimensional object set on the images captured by the cameras 101 and 102. The second characteristic information of the three-dimensional object is based on the three-dimensional object area setting process 401 for setting the three-dimensional object area 501 by enlarging or reducing the detection area 301 and the three-dimensional object area 501 obtained by the three-dimensional object area setting process 401 as a reference size. Based on the above, the recognition magnification setting process 402 that defines the recognition areas 601 and 602 of a plurality of sizes and the plurality of recognition areas 601 and 602 defined by the recognition magnification setting process 402 are used as the third characteristic information of the three-dimensional object. Recognition that performs recognition processing using the scanning area setting process 403 for setting a plurality of scanning areas 802 and 803 wider than the recognition areas 601 and 602 and the scanning areas 802 and 803 set in the scanning area setting process 403. Processing 209 and. The first characteristic information to the third characteristic information include, for example, the distinctiveness of a three-dimensional object, the distance from the three-dimensional object, the size of the three-dimensional object, the assumed size of the three-dimensional object, the brightness of the external environment, the direction of the headlight, and the like. It is at least one of the height of the road surface on which the three-dimensional object exists, the sensor resolution of the image pickup unit, the limit size of the three-dimensional object, the perspective position of the three-dimensional object, and the road surface height on which the three-dimensional object exists. This makes it possible to provide an image recognition device that accurately detects a three-dimensional object and has improved recognition performance.

The present invention is not limited to the above-described embodiment, and other embodiments considered within the scope of the technical idea of the present invention are also included within the scope of the present invention as long as the features of the present invention are not impaired. .. Further, the configuration may be a combination of the above-described embodiment and the modified example.

100, 100'Image recognition device, 101, 102 camera, 103 image input interface, 104 image processing unit, 105 arithmetic processing unit, 106 storage unit, 107 CAN interface, 108 control processing unit, 109 internal bus, 110 in-vehicle network CAN

Claims (21)

The detection region of the three-dimensional object that has been set on the captured image by the imaging unit, by expanding or shrinking on the image based on the detection characteristics information of said three-dimensional object, sets the three-dimensional object area on the image Three-dimensional object area setting part and
A recognition processing unit that performs recognition processing for specifying the type of the three-dimensional object with respect to the three-dimensional object region set by the three-dimensional object region setting unit.
Image recognition device. In the image recognition device according to claim 1,
The detection characteristic information includes the distinctiveness of the three-dimensional object, the distance from the three-dimensional object, the size of the three-dimensional object, the assumed size of the three-dimensional object, the brightness of the external environment, the direction of the headlight, and the three-dimensional object. An image recognition device that is at least one of the height of the road surface and the sensor resolution of the image pickup unit. In the image recognition apparatus according to claim 1 or 2.
A recognition magnification setting unit for defining recognition areas of a plurality of sizes based on the recognition characteristic information of the three-dimensional object is provided with the three-dimensional object area obtained by the three-dimensional object area setting unit as a reference size.
The recognition processing unit is an image recognition device that performs the recognition processing on each of the recognition areas of a plurality of sizes defined by the recognition magnification setting unit. In the image recognition device according to claim 3,
The recognition characteristic information is an image recognition device having at least one of a distance from the three-dimensional object, a size of the three-dimensional object, a limit size of the three-dimensional object, and a sensor resolution of the imaging unit. In the image recognition device according to claim 3,
A scanning area setting unit for setting a plurality of scanning areas wider than the recognition area based on the arrangement characteristic information of the three-dimensional object is provided for the plurality of recognition areas defined by the recognition magnification setting unit.
The recognition processing unit is an image recognition device that performs the recognition processing using the scanning area set by the scanning area setting unit. In the image recognition device according to claim 5,
The arrangement characteristic information is an image recognition device that is at least one of the perspective position of the three-dimensional object and the road surface height on which the three-dimensional object exists. In the image recognition device according to claim 5,
An image recognition device including a magnification-by-magnification scanning recognition processing unit that scans a plurality of the scanning regions by the plurality of recognition regions to obtain a response position of a recognition result. In the image recognition device according to claim 7,
An image recognition device including an optimum magnification setting unit that selects the recognition area of the optimum size based on the response position by the magnification-by-magnification scanning recognition processing unit. In the image recognition device according to claim 8,
An image recognition device including a detailed recognition position determination processing unit that determines a representative position for performing the recognition processing based on the response position of the scanning region corresponding to the recognition region selected by the optimum magnification setting unit. In the image recognition device according to claim 9,
An image recognition device including a detailed recognition processing unit that performs the recognition process using the recognition area or the representative position of the optimum size and specifies the type of the three-dimensional object to be recognized. The detection region of the three-dimensional object that has been set on the captured image by the imaging unit, by expanding or shrinking on the image based on the first characteristic information of the solid object, the solid object region on the image The three-dimensional object area setting unit to be set and
A recognition magnification setting unit that defines recognition areas of a plurality of sizes based on the second characteristic information of the three-dimensional object, using the three-dimensional object area obtained by the three-dimensional object area setting unit as a reference size.
A scanning area setting unit that sets a plurality of scanning areas wider than the recognition area based on the third characteristic information of the three-dimensional object with respect to the plurality of recognition areas defined by the recognition magnification setting unit.
A recognition processing unit that performs recognition processing for specifying the type of the three-dimensional object using the scanning area set by the scanning area setting unit, and a recognition processing unit.
Image recognition device. In the image recognition device according to claim 11,
The first characteristic information, the second characteristic information, and the third characteristic information are the distinctiveness of the three-dimensional object, the distance from the three-dimensional object, the size of the three-dimensional object, and the three-dimensional object, respectively. The assumed size, the brightness of the external environment, the direction of the headlight, the height of the road surface on which the three-dimensional object exists, the sensor resolution of the image pickup unit, the limit size of the three-dimensional object, the perspective position of the three-dimensional object, and the three-dimensional object. An image recognition device that is at least one of the existing road surface heights. A solid that sets the three-dimensional object area by enlarging or reducing the detection area of the three-dimensional object based on the detection characteristic information of the three-dimensional object with respect to the detection area of the three-dimensional object set on the image captured by the image pickup unit. Object area setting part and
A recognition processing unit that performs recognition processing for specifying the type of the three-dimensional object with respect to the three-dimensional object region set by the three-dimensional object region setting unit.
A recognition magnification setting unit that defines recognition areas of a plurality of sizes based on the recognition characteristic information of the three-dimensional object is provided with the three-dimensional object area obtained by the three-dimensional object area setting unit as a reference size .
The recognition processing unit is an image recognition device that performs the recognition processing on each of the recognition areas of a plurality of sizes defined by the recognition magnification setting unit. In the image recognition device according to claim 13,
The detection characteristic information includes the distinctiveness of the three-dimensional object, the distance from the three-dimensional object, the size of the three-dimensional object, the assumed size of the three-dimensional object, the brightness of the external environment, the direction of the headlight, and the three-dimensional object. An image recognition device that is at least one of the height of the road surface and the sensor resolution of the image pickup unit. In the image recognition device according to claim 13,
The recognition characteristic information is an image recognition device having at least one of a distance from the three-dimensional object, a size of the three-dimensional object, a limit size of the three-dimensional object, and a sensor resolution of the imaging unit. In the image recognition device according to claim 13,
A scanning area setting unit for setting a plurality of scanning areas wider than the recognition area based on the arrangement characteristic information of the three-dimensional object is provided for the plurality of recognition areas defined by the recognition magnification setting unit.
The recognition processing unit is an image recognition device that performs the recognition processing using the scanning area set by the scanning area setting unit. In the image recognition apparatus according to claim 16,
The arrangement characteristic information is an image recognition device that is at least one of the perspective position of the three-dimensional object and the road surface height on which the three-dimensional object exists. In the image recognition apparatus according to claim 16,
An image recognition device including a magnification-by-magnification scanning recognition processing unit that scans a plurality of the scanning regions by the plurality of recognition regions to obtain a response position of a recognition result. In the image recognition device according to claim 18,
An image recognition device including an optimum magnification setting unit that selects the recognition area of the optimum size based on the response position by the magnification-by-magnification scanning recognition processing unit. In the image recognition device according to claim 19,
An image recognition device including a detailed recognition position determination processing unit that determines a representative position for performing the recognition processing based on the response position of the scanning region corresponding to the recognition region selected by the optimum magnification setting unit. In the image recognition device according to claim 20,
An image recognition device including a detailed recognition processing unit that performs the recognition process using the recognition area or the representative position of the optimum size and specifies the type of the three-dimensional object to be recognized.

Images