JPWO2020071133A1 - Sign recognition device - Google Patents

Abstract

The purpose is to obtain a sign recognition device that suppresses erroneous sign recognition of an object such as a speed compliance sticker that closely resembles a road sign as a road sign. A solid that recognizes at least one imaging unit 101, 102, a sign estimation unit 111 that estimates a label candidate 603 from the images obtained by the imaging units 101, 102, and a three-dimensional object 602 included in the imaging range of the imaging units 101, 102. A sign determination unit that determines whether or not the sign candidate 603 is a road sign from the information regarding the estimation of the sign candidate 603 by the object recognition unit 112 and the sign estimation unit 111 and the information regarding the recognition of the three-dimensional object 602 by the three-dimensional object recognition unit 112. A sign recognition device 1 comprising 113.

Description

The present invention relates to a sign recognition device that recognizes a road sign.

With the improvement of social recognition for safe driving and autonomous driving, the demand for object recognition and distance measurement in in-vehicle camera devices is increasing.

Since the in-vehicle camera device simultaneously measures visual information by images and distance information to the object, various objects (people, cars, three-dimensional objects, white lines, road surfaces, signs, etc.) around the automobile are measured in detail. It is a device that can be grasped and contributes to improving safety during driving assistance.

There is a sign as one of the objects of object recognition in the in-vehicle camera device. Generally, the sign recognition function is used for accelerating or decelerating a vehicle that automatically drives in cooperation with map information. In addition, EuroNCAP (2016-2020 update), which is an evaluation index for advanced driver assistance systems, also has evaluation items related to Speed Assistance Systems (SAS), and their importance is increasing.

A device that recognizes a sign is described in Patent Document 1.

Japanese Unexamined Patent Publication No. 2017-26430

The problem of the road sign recognition function of the in-vehicle camera device is to output the recognition result of the road sign for the target road sign under various driving conditions such as urban areas and highways, and for objects that are not road signs. The purpose is to improve the accuracy of not outputting the recognition result of the road sign.

However, in order to identify the sign from the appearance by image processing, the device described in Patent Document 1 is attached to a signboard very similar to a road sign installed in an urban area or a highway, or behind a large vehicle such as a truck. There is a possibility that an object that looks very similar to the target road sign, such as a speed compliance sticker, may be mistakenly recognized as a road sign.

The present invention has been made to solve the above-mentioned problems, and obtains a sign recognition device that suppresses erroneous sign recognition of an object such as a speed compliance sticker that closely resembles a road sign as a road sign. With the goal.

In order to achieve the above object, the present invention comprises at least one imaging unit, a labeling estimation unit that estimates a label candidate from an image obtained by the imaging unit, and a predetermined three-dimensional object included in the imaging range of the imaging unit. It is determined whether or not the sign candidate is a road sign from the information regarding the estimation of the sign candidate by the sign estimation unit and the information regarding the recognition of the three-dimensional object by the three-dimensional object recognition unit. It is a sign recognition device including a sign determination unit.

According to the sign recognition device according to the present invention, it is possible to suppress the output of an erroneous sign recognition result for an object that closely resembles a target sign that appears in various traveling conditions such as an urban area or a highway.

It is a block diagram which shows the schematic structure of the sign recognition apparatus which concerns on embodiment of this invention. It is a processing flow diagram by a camera device. It is a flowchart which shows the processing content of the sign determination of FIG. It is a figure of an example of a road sign candidate provided behind a bus. Enlarged image of sign candidates Enlarged image of sign candidates It is a figure of the road sign candidate provided on the road. It is a figure of another example of a road sign candidate provided behind a bus. It is a top view of a road sign candidate provided behind a bus.

-System configuration-
FIG. 1 is a block diagram showing a schematic configuration of a sign recognition device according to an embodiment of the present invention. As shown in FIG. 1, the sign recognition device 1 according to the present embodiment is a device mounted on a vehicle and recognizes the outside environment of the vehicle based on image processing of a region to be photographed in front of the vehicle. The sign recognition device 1 has a function of recognizing, for example, white lines of roads, pedestrians, vehicles, other three-dimensional objects, traffic lights, road signs, lighting lamps, etc. by image processing, and determining the authenticity of road signs. In addition, a function of adjusting the brake and steering of the own vehicle equipped with the sign recognition device 1 may be provided, but this function can be omitted.

The sign recognition device 1 is a road sign recognition system mounted on a vehicle, and includes a camera device 100, a CAN (Controller Area Network) 110, an output device 114, and the like.

The camera device 100 includes a left camera 101, a right camera 102, an image input interface 103, an image processing unit 104, an arithmetic processing unit 105, a storage unit 106, a CAN interface 107, a control processing unit 108, and a bus 109.

The image input interface 103, the image processing unit 104, the arithmetic processing unit 105, the storage unit 106, the CAN interface 107, and the control processing unit 108 can be configured by a single computer unit or a plurality of computer units. In this embodiment, a case in which a single computer unit is used is illustrated, and the image processing unit 104, the control processing unit 108, the storage unit 106, the arithmetic processing 105, the image input interface 103, and the CAN interface 107 are cameras. They are connected to each other via the internal bus 109 of the device 100.

Further, the camera device 100 is connected to another computer unit or output device 114 via the CAN 110 described above. Hereinafter, each part will be described in sequence.

The left camera 101 and the right camera 102 are arranged on the left and right to acquire image information. The left camera 101 and the right camera 102 are mounted on the vehicle with a set interval on the left and right so that a predetermined area is within the field of view.

Further, the image input interface 103 controls the imaging of the left camera 101 and the right camera 102, and captures the captured images. The image data taken in through the image input interface 103 is sent to, for example, the image processing unit 104 and the arithmetic processing unit 105 via the bus 109.

The image processing unit 104 compares the first image obtained from the image sensor of the left camera 101 with the second image obtained from the image sensor of the right camera 102, and causes each image to be caused by the image sensor. The device-specific deviation correction and the image correction such as noise interpolation are performed, and this is stored in the storage unit 106. Furthermore, the points corresponding to each other are calculated between the first image and the second image, and the parallax information (the difference in the direction seen when the same point is viewed from two different points, that is, between the two directions) is calculated. (Information on the angle of) is calculated and stored in the storage unit 106.

The arithmetic processing unit 105 includes a sign estimation unit 111, a three-dimensional object recognition unit 112, and a sign determination unit 113.

The sign estimation unit 111 estimates a sign candidate from an image captured by the left camera 101 and the right camera 102 and stored in the storage unit 106. Here, the image used as the basis for estimating the marker candidate may be either an image taken by the left camera 101 or an image taken by the right camera 102, or an image taken by both cameras may be used. Regarding the estimation method, for example, the shape recognized by image processing is collated with, for example, a database of road signs stored in the storage unit 106, and a road sign candidate having a degree of matching with any of the road signs is considered as a candidate for the road sign. presume.

The three-dimensional object recognition unit 112 recognizes a three-dimensional object included in the imaging range of the left camera 101 and the right camera 102 from the parallax information calculated by the image processing unit 104. Specifically, a three-dimensional object is recognized by stereo image processing from the images obtained by the left camera 101 and the right camera 102. The three-dimensional object recognition unit 112 recognizes a predetermined three-dimensional object necessary for perceiving the environment around the vehicle by using the image and the parallax information (distance information for each point on the image) stored in the storage unit 106. .. The three-dimensional object to be recognized is a person, a car, or other obstacle, such as a traffic light, a road sign, a tail lamp of a car, and a headlight. A database of these target three-dimensional objects (for example, reference data for identification) is stored in, for example, a storage unit 106, and the three-dimensional object recognition unit 112 has a cut-out image of each subject and its parallax information, etc. for the input captured image. Recognize three-dimensional objects based on the database.

The sign determination unit 113 determines whether or not the sign candidate estimated by the sign estimation unit 111 is a road sign from the information regarding the estimation of the sign by the sign estimation unit 111 and the information regarding the recognition of the three-dimensional object by the three-dimensional object recognition unit 112. do. The recognition result of the road sign by the calculation processing unit 105 and the intermediate calculation result (for example, the calculation result of the sign estimation unit 111 and the three-dimensional object recognition unit 112) are appropriately stored in the storage unit 106. This determination algorithm will be described later.

The storage unit 106 is, for example, a memory, and stores information and the like obtained by the image processing unit 104 and the arithmetic processing unit 105.

The CAN interface 107 is an input / output unit for the CAN 110, and the calculation information of the camera device 100 is output to the CAN 110 via the CAN interface 107 and output to the control system of its own vehicle via the CAN 110. Specifically, the CAN interface 107 outputs a sign recognition result to the output device 114 when the sign determination unit 113 determines that the sign candidate is a road sign via the CAN 110.

The control processing unit 108 has a role of monitoring the operation of the camera device 100 as described above to see if each processing unit has caused an abnormal operation or if an error has occurred during data transfer to prevent the abnormal operation. Fulfill.

The road sign determination result and the like calculated by the camera device 100 as described above are transmitted to the output device 114 and other in-vehicle computer units (for example, a unit that executes vehicle control) via the CAN 110. The output device 114 is a device mounted in the driver's cab that outputs a display or a sound of, for example, a monitor, a lamp, a buzzer, a speaker, etc., and visually or audibly displays a road sign based on the information input from the camera device 100. Notify the occupants. The method of notification may be such that only the presence or absence of a road sign is notified, or the type of road sign may also be notified. Further, at least one of the visual output mode and the auditory output mode can be selected. When the brake or steering of the vehicle is controlled by another computer unit, the road sign information calculated by the camera device 100 is added to the basic information of the control by the other computer unit.

-Processing flow-
Next, the processing flow by the camera device 100 will be described with reference to FIG. FIG. 2 is a processing flow diagram by the camera device 100. The in-vehicle camera device 200 shown in FIG. 2 is a schematic representation of the camera device 100 of FIG. First, imaging 201 by the left camera 101 and imaging 202 by the right camera 102 are performed. Then, for each of the image data 203 captured by the left camera 101 and the image data 204 captured by the right camera 102, image processing such as correction for absorbing the peculiar habit of the image sensor and front imaging are performed. Stereo image processing 205 such as calculating the measurement distance (parallax information) for an object is performed. The stereo image processing 205 is executed by the image processing unit 104.

Next, the stereoscopic object detection 206, which is an image process such as a predetermined cropping of a stereo image or a gradation image, is performed. The result of the three-dimensional object detection 206 and the like are stored in the storage unit 106 of FIG. After that, based on the measurement distance result 207 obtained by the stereo image processing 205 and the processing result of the three-dimensional object detection 206, the object recognition 208 process, which is an image process (calculation process for object recognition), is performed. The processing of the three-dimensional object detection 206 and the object recognition 208 is performed by the three-dimensional object recognition unit 112 of FIG. As mentioned above, the three-dimensional objects recognized here are people, cars and other obstacles such as traffic lights, road signs, car tail lamps and headlights. After that, the object recognition result 216 (including the recognized three-dimensional object information and other parallax information not recognized as the three-dimensional object) obtained by the processing of the three-dimensional object detection 206 and the object recognition 208 is stored in the storage unit 106. Preservation process 215 is performed. In the present embodiment, the object recognition result 216 of the three-dimensional object recognition unit 112 is stored in the storage unit 106, and at the same time, is output to another computer unit or the sign determination unit 113 via the CAN 110.

In the camera device 100, in parallel with the above object recognition process, the sign recognition process 209 is performed by inputting a monocular image. The sign recognition process 209 includes a sign detection 210 that extracts a circular object from an input image, a sign identification 211 that identifies the type of the circular object, a sign tracking 212 that associates images with each other, and a comprehensive determination in a plurality of frames. It is composed of four processes of the sign determination 213.

That is, in the present embodiment, a circular shape is assumed as the outer shape of the road sign to be determined, and in the processing of the sign detection 210, for example, a shape whose degree of coincidence with the perfect circle is estimated to be a certain level or more by image processing is a sign candidate. Is detected as. In the subsequent processing of the sign identification 211, the sign candidates detected in the process of the sign detection 210 are collated with, for example, a sign database (recognition dictionary) stored in the storage unit 106, and the type of the sign candidate (which road sign is the sign candidate). Is it a candidate?) Is identified. In the processing of the sign tracking 212, sign candidates that can be displaced due to the traveling of a vehicle or the like in the camera field of view are tracked by associating between consecutive frames (camera images) by image processing. Each process up to the sign detection 210, the sign identification 211, and the sign tracking 212 up to this point is executed by, for example, the sign estimation unit 111.

In the process of the sign determination 213, for example, it is determined from the images of a plurality of frames whether the sign candidate is a real road sign. The processing of the sign determination 213 is executed by the sign determination unit 113, and is characterized in that it is executed based on the object recognition result 216 obtained by the processing of the object recognition 208 by the three-dimensional object recognition unit 112. In the present embodiment, it is determined whether or not the road sign is a true road sign from the inclusion relationship and the positional relationship between the parallax information of the three-dimensional object and other objects recognized by the three-dimensional object recognition unit 112 and the sign candidate estimated by the sign estimation unit 111. .. The details of this sign determination process will be described later.

Finally, a storage process 214 for storing the sign recognition result obtained in the above sign recognition process 209 in the storage unit 106 is performed. The sign recognition result by the sign recognition process 209 is stored in the storage unit 106, and at the same time, is output to another computer unit or output device 114 via the CAN 110. In the present embodiment, for an object very similar to the target sign (that is, an object presumed not to be a true road sign), the sign recognition result is not output to the output device 114, and it is presumed to be a true road sign. The sign recognition result is output to the output device 114 and notified to the occupant only for the object.

FIG. 3 is a flowchart showing the processing content of the sign determination of FIG. Hereinafter, the processing flow of the label determination 213 of the present invention will be described with reference to FIG.

(Step 501)
By inputting the result of the sign tracking 212 in FIG. 2, the identification result of the sign candidate in a plurality of images is obtained. After that, the process proceeds to the next step 502.

(Step 502)
The final judgment result is output in consideration of the identification result of each acquired image. As a determination method, a majority vote of the identification result of each image may be acquired, or the determination result may be acquired in consideration of the reliability obtained at the same time as the identification result. It should be noted that the slower the traveling speed of the own vehicle, the more image identification results can be obtained, and the less the image blur, which is an advantageous situation. After that, the process proceeds to the next step 503.

(Step 503)
As the first confirmation, the degree of overlap between the sign candidate and the three-dimensional object recognized by the three-dimensional object recognition unit 112 is checked. Check the inclusion relationship between the sign candidate and the three-dimensional object, and if all the sign candidates are included in the three-dimensional object, judge that it is unlikely that it is a road sign installed on the road, and do not output it as a sign result. And. After that, the process ends.

On the other hand, if the sign candidate is not included in the three-dimensional object, it is determined that the sign is a road sign installed on the road, and the process proceeds to step 504.

(Step 504)
This step is a process for detecting a part of a sign candidate that can be recognized as a road sign in step 503 but is not actually a road sign (that is, the judgment omission in step 503), and the sign candidate is outside the shield (that is, the sign candidate is outside the shield (that is, the judgment omission in step 503). For example, it is determined whether or not it is on the left and right outside of the image. The shield means parallax information that cannot be collated with the database related to the three-dimensional object by the three-dimensional object recognition unit 112 and is not recognized as a predetermined three-dimensional object. One typical example of parallax information treated as a shield is one of a subject that is actually a predetermined three-dimensional object and the parallax information is input but the entire image is not shown in the captured image and cannot be recognized as a predetermined three-dimensional object. It is a part (a side surface 802 of the bus 801 in FIG. 6 and the like) in which the whole image is reflected in the photographed image. If the shield is between the vehicle (camera) and the actual road sign, the entire road sign is not shown and the sign candidate is likely not a true road sign. In order to identify such an event, it is determined whether the sign candidate is located outside the shield in the left-right direction in the image (whether there is a shield existing inside the sign candidate in the left-right direction). .. When the sign candidate is located outside the shield, it is judged that it is unlikely that the sign candidate is a road sign installed on the road, as in the examples described later in FIGS. 6 and 7. , It is not output as a labeling result. That is, when a shield is located between the sign candidate and the own vehicle, it is determined that the sign candidate is not a road sign. After that, the process ends.

On the other hand, when the sign candidate is not located outside the shield (when there is no shield inside the sign candidate in the left-right direction), it is determined that the sign is a road sign installed on the road, and the process is performed. , Step 505.

(Step 505)
The sign determination 213 outputs a sign determination result indicating that the sign candidate is a road sign. After that, the process ends.

− Example 1-
FIG. 4A is a diagram of an example of a road sign candidate provided behind the bus. As shown in FIG. 4A, the bus 601 includes a sign candidate 603 and a sign candidate 604 on the rear surface. The rear surface of the bus 601 is recognized as a three-dimensional object 602 by the camera device 100.

On the rear surface of large vehicles such as trucks and buses in Europe, it is shown that this vehicle travels at 80 kph indicated by sign candidate 603 on general roads and at 100 kph indicated by sign candidate 604 on highways. , A speed compliance sticker is affixed.

As shown in the enlarged images of the sign candidate 603 and the sign candidate 604 of FIGS. 4B and 4C, since these objects closely resemble the speed limit sign, they are output as the sign recognition result in the processing of the conventional camera device. It ends up.

On the other hand, in the sign recognition device 1 of the present invention, since the sign candidate 603 and the sign candidate 604 are included in the three-dimensional object 602, it is unlikely that the sign recognition device 1 is a road sign installed on the road in step 503 of FIG. It is judged that the sign recognition result is suppressed from being output. In the example of FIG. 4A, it is thus identified that the sign candidate 603 and the sign candidate 604 are not road signs, and false alarms to the occupants are avoided.

− Example 2-
FIG. 5 is a diagram of road sign candidates provided on the road. As shown in FIG. 5, the labeling candidate 701 is not included in the three-dimensional object 702. Further, the label candidate 701 is not included in the three-dimensional object 703. In the case of this example, since the labeling candidate 701 is not included in the three-dimensional object 702 or the three-dimensional object 703, the procedure shifts from step 503 to step 504 in FIG. In step 504, it is determined whether the sign candidate 701 is outside the screen from the obstruction, but since the obstruction is not detected, the determination is not satisfied and the procedure moves to step 505, and the sign candidate 701 is on the road. It is output to the output device 114 as a true road sign installed above.

− Example 3-
FIG. 6 is a diagram of another example of a road sign candidate provided behind a bus. In the example of FIG. 6, the speed compliance sticker attached to the rear surface 804 of the bus 801 is recognized as the sign candidate 803. When a large vehicle such as bus 801 is traveling in the adjacent lane, or when a large vehicle is parked on the shoulder of the road, the position of the rear surface 804 of the bus 801 deviates from the imaging range of the camera device 100, and the rear surface The entire 804 is not displayed and the bus 801 is not recognized as a bus. In the example of FIG. 6, the sign candidate 803 is included in the rear surface 804 of the bus 801. However, since the rear surface 804 is not recognized as a predetermined three-dimensional object, the sign candidate 803, which is a speed compliance sticker, is a road in the determination of step 503 of FIG. It can be judged as a sign.

However, the entire side surface 802 of the bus 801 is reflected, and it is not recognized as a predetermined three-dimensional object and is classified as a shield. In the example of FIG. 6, since the sign candidate 803 is located outside the side surface 802 which is a shield, the determination of step 504 is satisfied in FIG. 3, and the sign candidate 803 is erroneously output to the output device 114 as a road sign. Can be avoided.

FIG. 7 is a top view of a road sign candidate provided behind the bus (a model diagram showing the example of FIG. 6 from a bird's-eye view). As shown in FIG. 7, looking at the positional relationship between the side surface 802 of the bus 801 and the sign candidate 803, if the sign candidate 803 is a road sign 805 installed on the road, this side surface 802 is used. The road sign 805 should not appear in the captured image because it is shielded by the bus 801 that it has. By confirming the positional relationship between the sign candidate 803 and the side surface 802 in this way, it is possible to prevent an erroneous sign recognition result from being output as shown in step 504 of FIG.

-Modification example-
Although an example in which the camera device 100 is provided with the left camera 101 and the right camera 102 is shown in the present embodiment, the present invention is not limited to this, and either the left camera 101 or the right camera 102 is omitted to form a stereoscopic structure. It is also possible to configure the object recognition to be executed by monocular processing. The camera also described the flow having the procedure of steps 503 and 504 in the example of FIG. 3, but if the procedure of step 503 can obtain sufficient accuracy for determining the authenticity of the road sign, the procedure of step 504 is described. May be omitted.

1 sign recognition device, 100 camera device, 101 left camera, 102 right camera, 103 image input interface, 104 image processing unit, 105 arithmetic processing unit, 106 storage unit, 107 CAN interface, 108 control processing unit, 109 bus, 110 CAN , 111 sign estimation unit, 112 three-dimensional object recognition unit, 113 sign determination unit, 114 output device, 200 camera device, 201 imaging, 202 imaging, 203 image data, 204 image data, 205 stereo image processing, 206 three-dimensional object detection, 207 Measurement distance result, 208 object recognition, 209 sign recognition processing, 210 sign detection, 211 sign identification, 212 sign tracking, 213 sign judgment, 214 storage processing, 215 storage processing, 601 bus, 602 three-dimensional object, 603 sign candidate, 604 sign Candidate, 701 sign candidate, 702 three-dimensional object, 703 three-dimensional object, 801 bus, 802 three-dimensional object, 803 sign candidate, 804 rear surface, 805 road sign

Claims (7)

With at least one imaging unit
A marker estimation unit that estimates a label candidate from an image obtained by the imaging unit, and a label estimation unit.
A three-dimensional object recognition unit that recognizes a predetermined three-dimensional object included in the imaging range of the imaging unit,
It is provided with a sign determination unit for determining whether or not the sign candidate is a road sign from the information regarding the estimation of the sign candidate by the sign estimation unit and the information regarding the recognition of the three-dimensional object by the three-dimensional object recognition unit. A sign recognition device characterized by. In the sign recognition device according to claim 1,
The sign estimation unit is a sign recognition device characterized in that a sign candidate is estimated by monocular image processing from an image obtained by a single imaging unit. In the sign recognition device according to claim 1,
The three-dimensional object recognition unit is a marker recognition device characterized by recognizing the three-dimensional object by stereo image processing from images obtained by the plurality of image pickup units. In the sign recognition device according to claim 3,
The sign determination unit
When the inclusion relationship between the three-dimensional object recognized by the three-dimensional object recognition unit and the sign candidate acquired by the sign estimation unit is determined and the sign candidate is included in the three-dimensional object, the sign candidate is a road. A sign recognition device characterized in that it is determined that it is not a sign. In the sign recognition device according to claim 3,
The sign determination unit
The positional relationship between the shield, which is parallax information not recognized as the three-dimensional object by the stereo image processing, and the sign candidate acquired by the sign estimation unit is determined, and the sign candidate is located outside the shield. When the sign is used, the sign recognition device is characterized in that the sign candidate is determined not to be a road sign. In the sign recognition device according to claim 1,
A sign recognition device further comprising an output device that outputs a sign recognition result by the sign determination unit. In the sign recognition device according to claim 6,
The output device is a sign recognition device characterized by being a monitor or a buzzer.

Images