WO2019177019A1

WO2019177019A1 - Traffic signal recognizing device, traffic signal recognition method, and program

Info

Publication number: WO2019177019A1
Application number: PCT/JP2019/010260
Authority: WO
Inventors: 良介後藤; サヒムコルコス; 村田　久治; 本村　秀人
Original assignee: パナソニックＩｐマネジメント株式会社
Priority date: 2018-03-14
Filing date: 2019-03-13
Publication date: 2019-09-19

Abstract

Provided is a traffic signal recognizing device with which it is possible to ease the restriction on imaging a traffic signal. A traffic signal recognizing device (100) is provided with a processor and a memory. The processor uses the memory to extract a region in which a traffic signal having N lights is shown from an image that is acquired by a sensor and is subject to recognition, the region being extracted as a traffic signal region, and calculates, from the traffic signal region, the level of similarity of a plurality of images in which lights different from each other are shown.

Description

Signal recognition device, signal recognition method and program

The present disclosure relates to a traffic signal recognition device that recognizes traffic signals.

Conventionally, a traffic signal recognition device for recognizing a traffic signal has been proposed (for example, see Patent Document 1). This traffic light recognition device acquires an image generated by shooting a traffic light, and recognizes the light color of the traffic light based on the image.

JP 2009-244946 A

However, the signal recognition device of Patent Document 1 has a problem that an image of a signal generated by photographing under severe restrictions is necessary in order to properly recognize the signal.

Therefore, the present disclosure provides a traffic light recognition device that can relax restrictions on shooting of a traffic light.

A traffic signal recognition apparatus according to an aspect of the present disclosure includes a processor and a memory, and the processor uses the memory to recognize N images (N is an integer of 3 or more) from recognition target images acquired by a sensor. An area in which a traffic light having a lamp part is projected is extracted as a traffic light area, and a similarity between a plurality of images in which different lamp parts are projected is calculated from the traffic light area.

Note that these comprehensive or specific modes may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, and the system, method, integrated circuit, and computer program. And any combination of recording media.

The traffic signal recognition device of the present disclosure can ease restrictions on traffic signal shooting.

FIG. 1 is a block diagram illustrating a configuration of a traffic signal recognition apparatus according to an embodiment. FIG. 2 is a diagram for explaining processing of the region extraction unit, the region division unit, and the similarity calculation unit in the embodiment. FIG. 3 is a diagram for explaining processing of the light color recognition unit in the embodiment. FIG. 4 is a flowchart showing the processing operation of the traffic signal recognition apparatus in the embodiment. FIG. 5 is a block diagram illustrating a configuration of the traffic signal recognition apparatus according to the first modification of the embodiment. FIG. 6 is a flowchart illustrating the processing operation of the traffic signal recognition apparatus according to the first modification of the embodiment. FIG. 7 is a block diagram illustrating a configuration of a traffic signal recognition apparatus according to Modification 2 of the embodiment. FIG. 8 is a diagram for explaining the recognition of the light color by the traffic light recognition apparatus according to the second modification of the embodiment. FIG. 9 is a flowchart illustrating the processing operation of the traffic signal recognition apparatus according to the second modification of the embodiment. FIG. 10 is a block diagram illustrating a configuration of a traffic signal recognition apparatus according to the third modification of the embodiment. FIG. 11 is a diagram for explaining the recognition of the light color by the traffic light recognition apparatus according to the third modification of the embodiment. FIG. 12 is a flowchart illustrating the processing operation of the traffic signal recognition apparatus according to the third modification of the embodiment. FIG. 13 is a diagram for explaining recognition of the traveling direction by the traffic light recognition apparatus according to the fourth modification of the embodiment. FIG. 14 is a diagram illustrating an example of a method of calculating the similarity of the first lamp unit region in the embodiment. FIG. 15 is a diagram illustrating an example of a method for calculating the similarity of each lamp area according to the fifth modification of the embodiment. FIG. 16 is a diagram illustrating another example of a method for calculating the similarity of each lamp unit region according to the fifth modification of the embodiment. FIG. 17 is a diagram illustrating an example of a method of calculating the similarity of the four lamp unit regions (N = 4) according to the fifth modification of the embodiment.

(Knowledge that became the basis of this disclosure)
The present inventor has found that the following problems occur with respect to the traffic signal recognition device of Patent Document 1 described in the “Background Art” section.

The traffic signal recognition device of Patent Document 1 acquires a color feature image generated by shooting at a predetermined shutter speed. Then, the traffic signal recognizing device extracts, from the color feature image, a circular area that shows the same color as the lighting color of the traffic light as a color feature candidate area. Further, the traffic signal recognition apparatus acquires a shape feature image generated by photographing at a shutter speed determined based on the average luminance around the color feature candidate region. Then, the traffic signal recognition device extracts a region that matches the shape of a predetermined traffic signal as a shape feature candidate region from the periphery of the color feature candidate region in the shape feature image. The shape feature candidate area extracted in this way is recognized as a traffic light area, and the color of the color feature candidate area is recognized as a lighting color of the traffic light, that is, a light color.

That is, in this traffic light recognition device, in order to recognize the light color of the traffic light, a color feature image generated by photographing at a predetermined shutter speed is necessary. The predetermined shutter speed is set so that the lighting color of the traffic light displayed in the color feature image appears clearly. That is, the predetermined shutter speed needs to be adjusted to a shutter speed suitable for the situation or environment so that color saturation does not occur in the color feature image.

As described above, in the traffic signal recognition device of Patent Document 1 described above, in order to properly recognize the light color of the traffic signal, an image of the traffic signal generated by photographing under severe restrictions is necessary.

Also, in order to recognize the traffic light area, it is necessary to readjust the shutter speed based on the average luminance around the color feature candidate area so that the shape of the traffic light clearly appears in the shape feature image. Furthermore, in order to search for a region that matches the shape of a predetermined traffic signal from the periphery of the color feature candidate region, a template that indicates the shape of the traffic signal is required. Since this search is performed based on the position of the color feature candidate area, a template is required for each position of the lighting color in the traffic light. Therefore, multiple types of templates are necessary.

In order to solve such a problem, a traffic light recognition apparatus according to an aspect of the present disclosure includes a processor and a memory, and the processor uses the memory to recognize from a recognition target image acquired by a sensor, An area in which a traffic light having N lamps (N is an integer of 3 or more) is displayed as a traffic light area, and a plurality of similarities of images in which different light parts are projected from the traffic light area. Is calculated. For example, the processor may generate N lamp part areas as the plurality of images, and calculate the similarity of each of the N lamp part areas.

Thus, a lamp part that is not similar to any other lamp part among the N lamp parts of the traffic light can be recognized as a lit lamp part. As a result, it is not necessary to recognize the color of each lamp area from the image of that area, and the restrictions on photographing of the traffic light can be relaxed.

Further, the processor is similar to any other lamp section area among the N lamp section areas based on the similarity calculated for each of the N lamp section areas. It may be determined whether or not there is a dissimilar region that is not a lamp part region.

Thereby, among the N lamp parts of the traffic light, a lamp part that is not similar to any other lamp part can be more appropriately recognized as a lit lamp part. As a result, it is not necessary to recognize the color of each lamp area from the image of that area, and the restrictions on shooting of the traffic light can be relaxed appropriately.

For example, when the processor further determines that the dissimilar region exists, the processor recognizes a color preliminarily associated with the position of the dissimilar region in the traffic light region as a light color in the recognition target image. May be. Specifically, in the recognition of the lamp color, the processor refers to the association information indicating the color associated with each position of the N lamp unit regions in the traffic light region, and the association information The color associated with the position of the dissimilar region may be recognized as the lamp color in the recognition target image.

Thereby, when a lamp part area that is not similar to any of the N lamp part areas exists as a dissimilar area, the lamp color is recognized based on the position of the dissimilar area. For example, if one of the N lamp sections of the traffic light is turned on and the other N-1 lamp sections are turned off, the one lamp section that is lit is displayed. The region is determined as a dissimilar region. If the dissimilar area is at the left end of the traffic light area, blue is recognized as the lamp color. As described above, since the lamp color is recognized based on the similarity between the N lamp section areas, that is, the difference in the feature amount of the N lamp section areas, the lamp color can be appropriately recognized.

For example, when color saturation occurs, even if the blue light part of the traffic light is lit, in the light part region corresponding to that light part of the recognition target image, the light part is not blue but is a color close to white or yellow. It may be reflected in. However, in the traffic light recognition apparatus according to the above aspect, since the lamp color is recognized based on the difference in the feature amount of the N lamp area even if color saturation occurs in the recognition target image, such recognition is performed. It is possible to appropriately recognize the light color even for the target image.

Therefore, since color saturation may occur in the recognition target image, it is possible to eliminate the need to adjust the camera to a predetermined shutter speed so that color saturation does not occur when shooting a traffic light with the camera. As a result, the shutter speed of the camera can be fixed. In addition, it is possible to appropriately recognize the light color even with respect to the recognition target image generated by photographing at night or rainy weather where color saturation is likely to occur.

Furthermore, since the light color of the light part of the traffic light does not have to be correctly displayed in the recognition target image, a camera with no high accuracy can be used for shooting the traffic light. For example, since environmental conditions are severe in a vehicle, it is difficult to use a high-definition and wide dynamic range camera attached to the vehicle. However, the traffic light recognition apparatus according to the aspect described above can recognize the light color even from an image generated by photographing with a camera that is not highly accurate. Therefore, it is possible to appropriately recognize the light color even with respect to an image generated by photographing with a non-accurate camera attached to a vehicle that can withstand severe environmental conditions.

Furthermore, since pattern matching is not performed, the shape of each of the N lamp units included in the traffic light may be circular or quadrangular, or any shape. In addition, the color of the lamp portion of the traffic light is not limited to blue, yellow and red, and any color can be appropriately recognized. In addition, since a template for pattern matching is not required, the memory capacity for holding the template can be reduced.

Further, in the calculation of the similarity, the processor calculates, for each lamp area in the recognition target image, a correlation coefficient between the lamp area and each of at least two other lamp areas. The similarity of the lamp area may be calculated by selecting the largest correlation coefficient as the similarity.

Thus, for example, if one of the N lamp sections of the traffic light is turned on and the other N-1 lamp sections are turned off, the one lamp section that is lit is displayed. A small value is calculated as the similarity of the lamp part area. On the other hand, a large value is calculated as the similarity of the lamp area where each of the other N-1 lamp sections is projected. Therefore, it is possible to more appropriately determine whether or not there is a dissimilar region and which lamp unit region is a dissimilar region by using the similarity of these N lamp unit regions. Can do.

In addition, in determining whether or not the dissimilar area exists, the processor determines that the lamp part area having the smallest similarity among the similarities of the N lamp part areas is the dissimilar area. May exist.

Thus, it is possible to determine that there is a dissimilar region in any recognition target image and recognize the light color in the recognition target image.

Further, in determining whether or not the dissimilar area exists, the processor determines the smallest similarity when the smallest similarity among the similarities of the N lamp part areas is equal to or less than a threshold value. You may determine with the lamp part area | region which has a similarity degree existing as said dissimilar area | region.

Thus, it is determined that the dissimilar region exists when the smallest similarity is equal to or smaller than the threshold, and it is determined that the dissimilar region does not exist when the smallest similarity is larger than the threshold. Therefore, it is possible to recognize the lamp color until there is no significant difference between the similarities of the N lamp section areas, and as a result, it is possible to suppress recognition of an inappropriate lamp color.

Further, when the processor determines that the dissimilar region does not exist in the recognition of the lamp color, the processor refers to the history information indicating the lamp color recognized for the image acquired in the past, and The lamp color indicated by the history information may be recognized as the lamp color in the recognition target image.

For example, there is a timing when all the lights of the traffic light are not lit due to flicker. There is no dissimilar region in the recognition target image generated by shooting at such timing. However, among a series of images generated by shooting at a constant frame rate, an image generated at a point before the recognition target image, that is, a past image may not be affected by flicker. high. That is, there is a high possibility that the past image is projected in a state in which the lamp portion that has been turned off by flicker when the recognition target image is captured is turned on. Therefore, in the traffic signal recognition device according to the above aspect, when there is no dissimilar region in the recognition target image, the lamp color recognized for the past image is recognized as the lamp color in the recognition target image. . Accordingly, it is possible to appropriately prevent the recognition of the light color due to flicker.

Further, the processor refers to history information indicating a light color recognized for each of a plurality of images acquired before the recognition target image, and is recognized for the recognition target image. Among the lamp colors and lamp colors in each of the plurality of images indicated by the history information, the most lamp colors are specified as lamp colors, and the lamp colors recognized for the recognition target image are You may update to the said multiple lamp color.

This refers to the lamp color recognized for each of the plurality of images acquired before the recognition target image. The plurality of images and recognition target images are, for example, a series of images generated by shooting at a constant frame rate. Then, the lamp color recognized for the recognition target image is updated to the multiple lamp colors that are the most lamp colors among the lamp colors and the lamp colors of the plurality of images. Therefore, for example, due to factors other than flicker such as noise, even if it is suddenly determined that there is no dissimilar area in the recognition target image, or an incorrect light color is recognized for the recognition target image, The error can be easily corrected.

Further, in determining whether or not the dissimilar region exists, the processor, for each of a plurality of images acquired before the recognition target image, for each position in the traffic signal region in the image, The history information indicating the similarity of the lamp area at the position is referred to, and the average of the similarity of the lamp area at the position in the recognition target image and the plurality of images is calculated for each position in the traffic light area. The lamp area at the position corresponding to the average similarity that is the smallest of the average similarities calculated for each position among the N lamp areas in the recognition target image. May be determined to exist as the dissimilar region.

Thereby, the similarity calculated for each of the plurality of images acquired before the recognition target image is referred to. The plurality of images and recognition target images are, for example, a series of images generated by shooting at a constant frame rate. Then, for each position in the traffic signal area, the average similarity between the recognition target image and the lamp area at the position in the plurality of images is calculated as the average similarity. For example, the average similarity of the lamp area at the left end in the traffic light area, the average similarity of the lamp area in the center of the traffic light area, and the average similarity of the lamp area at the right end in the traffic light area Calculated. Here, for example, when N = 3, if the average similarity of the lamp area at the left end is the smallest, it is determined that the lamp area at the left end in the traffic light area in the recognition target image exists as a dissimilar area. Therefore, for example, due to factors other than flicker such as noise, even if it is suddenly determined that there is no dissimilar area in the recognition target image, or an incorrect light color is recognized for the recognition target image, The error can be corrected. Further, since the dissimilar region is determined based on the average similarity using the past similarity, the dissimilar region can be determined more accurately than using the light color recognized in the past. .

Further, when the traffic light further has M (M is an integer equal to or greater than 1) direction indicator lamps, each of which indicates a traveling direction of the vehicle, the processor may In the generation, by dividing the traffic light area, the N lamp part areas and the M lamp part areas in which the direction indicating lamp parts are respectively projected are generated. Based on the recognized lamp color and the respective feature values of the M lamp area, indicated by at least one direction indicating lamp section that is lit among the M direction indicating lamp sections. The traveling direction may be recognized. Alternatively, when the traffic light further has M (M is an integer of 1 or more) direction indicator lamps, each of which indicates a traveling direction of the vehicle, the processor In the generation, by dividing the traffic light area, the N lamp section areas and M lamp section areas each displaying a direction indicating lamp section are generated, and the M lamp sections are further generated. The traveling direction indicated by at least one direction indicator lamp unit that is lit among the M direction indicator lamp units may be recognized based on the feature amount of each region.

This makes it possible to appropriately recognize the traveling direction of the directional indicator lamp portion that is lit out of the M direction indicator lamp portions that indicate the traveling direction by, for example, an arrow. For example, by inputting each feature amount of M lamp areas into a model such as a neural network generated by machine learning, the direction indication of lighting is indicated from among the M direction indicating lamp sections. It is possible to properly recognize the lamp unit. That is, it is possible to appropriately recognize the traveling direction of the directional indicator lamp unit that is lit. Furthermore, when the direction indicator lamp part which may be turned on for every lamp color of the traffic light is determined in advance, the direction of lighting from the M direction indicator lamp parts according to the recognized lamp color. It is possible to narrow down the candidates for the indicator lamp part. In this case, it is possible to appropriately recognize the lit direction indicator lamp unit from the narrowed candidates. As described above, in the traffic signal recognition device according to the above aspect, the lighting direction and the direction indication lamp unit that may be lit may be turned on regardless of whether the lighting direction and the direction indicating lamp unit that may be turned on are determined in advance. It is possible to appropriately recognize the traveling direction of the lamp unit.

In the calculation of the similarity, the processor may calculate, for each of the N lamp areas, an image of the lamp area and a group of at least one other lamp area other than the lamp area. You may calculate the similarity of the said lamp | ramp part area | region by comparing with an image. For example, when comparing the image of the lamp unit area and the image of the group for each of the N lamp unit areas, the processor, for each block included in the lamp unit area, The similarity of the lamp area may be calculated by searching the block most similar to the group. For each of the N lamp areas, the processor compares the image of the block with the image of the block for each block included in the group when comparing the image of the lamp area with the image of the group. You may calculate the similarity of the said lamp part area | region by searching for a similar block from the said lamp part area | region.

Thus, for example, when a group is composed of a plurality of lamp sections, when calculating the similarity of the lamp sections, it is time and effort to individually compare the lamp section with each of the other lamp sections. Can be omitted. That is, by comparing the lamp area with a group of (N−1) lamp areas other than the lamp area, the similarity of the lamp area can be easily calculated.

Hereinafter, embodiments will be specifically described with reference to the drawings.

It should be noted that each of the embodiments described below shows a comprehensive or specific example. Numerical values, shapes, materials, components, arrangement positions and connection forms of components, steps, order of steps, and the like shown in the following embodiments are merely examples, and are not intended to limit the present disclosure. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the highest concept are described as optional constituent elements.

Each figure is a schematic diagram and is not necessarily shown strictly. Moreover, in each figure, the same code | symbol is attached | subjected about the same structural member.

(Embodiment)
FIG. 1 is a block diagram illustrating a configuration of a traffic signal recognition apparatus according to an embodiment.

For example, the vehicle 10 includes a camera 210, a map storage unit 220, a position detection system 230, a vehicle control unit 240, and a traffic signal recognition device 100.

The camera 210 captures, for example, the front of the vehicle 10 and outputs an image generated by the capturing (hereinafter referred to as a captured image) to the traffic signal recognition apparatus 100. Specifically, the camera 210 captures images at a constant frame rate, and outputs an image as a captured image to the traffic signal recognition apparatus 100 each time an image is generated by the capture. In the present embodiment, the camera 210 is configured as a sensor that acquires a captured image (that is, a recognition target image described later). Note that the shutter speed used for shooting may be fixed or variable.

The map storage unit 220 stores at least map information indicating a map around the vehicle 10. Such map information is wirelessly transmitted from a server via a network such as the Internet and stored in the map storage unit 220, for example.

The position detection system 230 detects the position of the vehicle 10 and outputs position information indicating the position to the traffic signal recognition device 100. Specifically, the position detection system 230 is configured as a GPS (Global Positioning System) receiver.

The vehicle control unit 240 includes, for example, one or more ECUs (Electronic Control Units). Such a vehicle control part 240 acquires the light color information output from the traffic signal recognition apparatus 100, and controls driving | running | working of the vehicle 10 based on the light color of the traffic light shown by the light color information. That is, the vehicle control unit 240 controls the vehicle 10 so that the vehicle 10 automatically travels on the road where the traffic light is arranged.

The traffic signal recognition device 100 acquires a captured image from the camera 210 and recognizes the traffic signal displayed in the captured image. For example, the traffic light recognition device 100 recognizes the light color of the traffic light and outputs the light color information indicating the light color to the vehicle control unit 240. Note that the captured image used for the current traffic signal recognition is also referred to as a recognition target image.

The traffic light recognition apparatus 100 according to the present embodiment includes an area extraction unit 101, an area division unit 102, a similarity calculation unit 103, and a light color recognition unit 104.

The area extraction unit 101 acquires a captured image from the camera 210 and extracts an area where the traffic signal is displayed as a traffic signal area from the captured image. Here, the traffic light has N lamp units (N is an integer of 3 or more). For example, N is three, and the traffic light has a blue lamp unit, a yellow lamp unit, and a red lamp unit. When the captured image is used for recognition of the current lighting color, the captured image is treated as a recognition target image.

Here, the region extraction unit 101 uses, for example, the map information stored in the map storage unit 220 and the position of the vehicle 10 detected by the position detection system 230 when extracting the traffic signal region from the captured image.

Specifically, the map information indicates the position and form of each traffic signal in the three-dimensional space. That is, the map information not only shows the position of the road and the building, but also shows the position and form of the traffic signal arranged on the road. The position of the traffic light is indicated by a three-dimensional coordinate system. That is, the position of the light box of the traffic light is indicated by longitude, latitude, and height. The height may be a height from a road or an altitude. The form of the traffic signal includes the shape and size of the traffic signal when the traffic signal is viewed from the front.

The region extraction unit 101 identifies the position of the vehicle 10 on the map by mapping the position indicated by the position information output from the position detection system 230 to the map indicated by the map information. Furthermore, the area extraction unit 101 displays the traffic signal in the captured image based on the position of the vehicle 10 and the traffic signal on the map, the form of the traffic signal, the mounting position of the camera 210 on the vehicle 10 and the shooting direction. The area is detected geometrically. Then, the area extraction unit 101 extracts the detected area from the captured image as a traffic light area.

The area dividing unit 102 divides the traffic signal area to generate N lamp part areas each displaying the lamp part. The area dividing unit 102 may equally divide the signal area into N parts when dividing the signal area into N lamp part areas. The area dividing unit 102 may perform division using map information. That is, the map information may indicate not only the position of the traffic signal but also the number (N) of the lamp units included in the traffic signal and the arrangement of the N lamp units. For example, when N = 3, the map information indicates the number of lamp units “3” and indicates that three lamp units are arranged at equal intervals in the horizontal direction. The area dividing unit 102 divides the traffic signal area according to the number and arrangement of the lamp units. In this embodiment, the area dividing unit 102 divides the traffic light area in order to generate N lamp part areas, but generates N lamp part areas without dividing the traffic light area. Also good. For example, the area dividing unit 102 extracts, for each lamp unit, an area in which the lamp unit is projected from the entire signal area. In this extraction, at least two lamp section areas out of the N lamp section areas may include the same image such as a background. In addition, an image such as a background that is not included in any of the N lamp sections may be in the traffic signal area. As described above, in the present embodiment, if N lamp parts areas in which different lamp parts are projected are generated from the traffic light area, what kind of image processing is performed on the traffic light area. Also good.

The similarity calculation unit 103 acquires the N lamp unit areas described above from the region dividing unit 102, and calculates the similarity of each of the N lamp unit regions.

The lamp color recognition unit 104 acquires the similarity calculated for each of the N lamp unit regions from the similarity calculation unit 103. Then, the lamp color recognition unit 104 recognizes the color associated with the position of the dissimilar area specified based on the similarity as the lamp color in the captured image. In addition, a dissimilar area | region is a lamp part area | region which is not similar to any other lamp part area | regions among N lamp part area | regions. Specifically, the lamp color recognition unit 104 recognizes the color associated with the position of the dissimilar region in the association information as the lamp color in the captured image by referring to association information described later. To do. The association information may be included in the map information. That is, the map information may indicate the colors associated with the positions of the lamp units in addition to the number (N) of the lamp units included in the traffic light and the arrangement of the N lamp units. . Further, the lamp color recognition unit 104 outputs information indicating the recognized lamp color to the vehicle control unit 240 as lamp color information.

FIG. 2 is a diagram for explaining the processing of the region extracting unit 101, the region dividing unit 102, and the similarity calculating unit 103.

The region extraction unit 101 acquires a captured image Pa as a recognition target image from the camera 210 (step S1). Then, the area extraction unit 101 extracts the traffic light area Pb from the captured image Pa (step S2). Here, in the traffic light area Pb, a traffic light having three lamp parts is displayed as N lamp parts. That is, N = 3. The three lamp parts are composed of a blue lamp part, a yellow lamp part, and a red lamp part.

Next, the area dividing unit 102 divides the traffic signal area Pb into three lamp part areas Pb1, Pb2, and Pb3 (step S3). For example, in the lamp area Pb1, only the blue lamp section is projected, only the yellow lamp section is projected in the lamp section area Pb2, and only the red lamp section is projected in the lamp section area Pb3. Yes. The lamp section area Pb1 is an area located at the left end of the traffic signal area Pb, and is also referred to as a first lamp section area Pb1. The lamp part area Pb2 is an area located at the center of the traffic light area Pb, and is also referred to as a second lamp part area Pb2. The lamp part area Pb3 is an area located at the right end of the traffic signal area Pb, and is also referred to as a third lamp part area Pb3.

Next, the similarity calculation unit 103 calculates the respective similarities k1, k2, and k3 of the three lamp region Pb1, Pb2, and Pb3 (steps S4 to S6).

Specifically, when calculating the similarity k1 of the first lamp part area Pb1 (step S4), the similarity calculation unit 103 firstly determines between the first lamp part area Pb1 and the second lamp part area Pb2. A correlation coefficient k12 is calculated. That is, the similarity calculation unit 103 compares the feature quantity of the image of the first lamp area Pb1 with the feature quantity of the image of the second lamp area Pb2, and sets the correlation between these feature quantities as the correlation coefficient k12. calculate. The correlation coefficient indicates a larger value as the feature amounts of the images in these regions are similar.

For example, the similarity calculation unit 103 divides the first lamp area Pb1 into blocks composed of n × n pixels, and for each block of the first lamp area Pb1, the block most similar to the image of the block is determined. Search from the second lamp area Pb2. Specifically, the similarity calculation unit 103 represents the feature amount of the block in the first lamp unit region Pb1 as a vector. Then, the similarity calculation unit 103 finds a block having the feature amount of the vector having the shortest distance from the vector from the second lamp unit region Pb2. For example, a vector representing a feature amount of a block is composed of an array of pixel values (specifically, luminance values) of n × n pixels constituting the block. Thereby, the shortest distance between vectors is obtained for each block of the first lamp area Pb1. The similarity calculation unit 103 calculates the correlation coefficient k12 from the shortest inter-vector distance obtained for each block of the first lamp unit region Pb1. For example, the correlation coefficient k12 is the reciprocal of the value obtained by averaging the shortest vector distances obtained for each block of the first lamp area Pb1.

Note that the calculation method of the correlation coefficient is not limited to this, and any method may be used. For example, in the above-described example, the shortest inter-vector distance is calculated for each block of the first lamp area Pb1, but the shortest inter-vector distance may be calculated in finer units. For example, the shortest inter-vector distance for the image in the processing target window may be calculated while shifting the processing target window one pixel at a time in the first lamp area Pb1. For example, the processing target window has a size that surrounds n × n pixels.

The similarity calculation unit 103 calculates the correlation coefficient k13 between the first lamp unit region Pb1 and the third lamp unit region Pb3 in the same manner as described above. And the similarity calculation part 103 selects the largest correlation coefficient of the correlation coefficient k12 and the correlation coefficient k13 as the similarity k1 of the 1st lamp part area | region Pb1. Thereby, the similarity k1 of the first lamp part region Pb1 is calculated.

Similarly, when the similarity calculation unit 103 calculates the similarity k2 of the second lamp unit region Pb2 (step S5), first, the phase relationship between the second lamp unit region Pb2 and the first lamp unit region Pb1. The number k21 is calculated. Further, the similarity calculation unit 103 calculates a correlation coefficient k23 between the second lamp part region Pb2 and the third lamp part region Pb3. And the similarity calculation part 103 selects the largest correlation coefficient of the correlation coefficient k21 and the correlation coefficient k23 as the similarity k2 of the 2nd lamp part area | region Pb2. Thereby, the similarity k2 of the second lamp area Pb2 is calculated.

Similarly, when the similarity calculation unit 103 calculates the similarity k3 of the third lamp unit region Pb3 (step S6), first, the phase relationship between the third lamp unit region Pb3 and the first lamp unit region Pb1. The number k31 is calculated. Further, the similarity calculation unit 103 calculates a correlation coefficient k32 between the third lamp unit region Pb3 and the second lamp unit region Pb2. And the similarity calculation part 103 selects the largest correlation coefficient of the correlation coefficient k31 and the correlation coefficient k32 as the similarity k3 of the 3rd lamp part area | region Pb3. Thereby, the similarity k3 of the 3rd lamp part area | region Pb3 is calculated.

As described above, the similarity calculation unit 103 according to the present embodiment calculates a correlation coefficient between each lamp unit area and each of at least two other lamp unit areas for each lamp unit area in the recognition target image. Among them, the largest correlation coefficient is selected as the similarity. As a result, the similarity of the lamp area is calculated.

For example, if one of the three lamps of a traffic light is lit and the other two lamps are not lit, the lamp unit area in which one of the lit lamps is projected is displayed. A small value is calculated as the similarity. On the other hand, a large value is calculated as the similarity of the lamp area where each of the other two lamp sections is projected. Therefore, by using the similarity of these three lamp part areas, a lamp part area that is not similar to any other lamp part area can be easily found out of the three lamp part areas. it can.

FIG. 3 is a diagram for explaining processing of the light color recognition unit 104.

The lamp color recognition unit 104 acquires the similarities k1 to k3 calculated for each of the first lamp part region Pb1, the second lamp part region Pb2, and the third lamp part region Pb3. Then, the lamp color recognition unit 104 recognizes the color of the lit lamp part among the three lamp parts of the traffic light as the lamp color in the captured image Pa based on the similarity k1 to k3. .

Specifically, the lamp color recognizing unit 104 selects any of the three lamp part areas Pb1 to Pb3 based on the similarities k1 to k3 calculated for the three lamp part areas Pb1 to Pb3. It is determined whether or not there is a dissimilar region that is a lamp region that is not similar to the lamp region.

For example, as shown in FIG. 3, in the lamp color recognition unit 104, the similarity k1 of the similarities k1 to k3 is the smallest, so the first lamp region Pb1 having the similarity k1 is a dissimilar region. Is determined. That is, the lamp color recognition unit 104 determines that the first lamp part region Pb1 having the smallest similarity k1 among the similarities k1 to k3 of the three lamp part regions Pb1 to Pb3 exists as a dissimilar region. judge. As a result, it is possible to determine that there is a dissimilar region in any captured image Pa and recognize the light color in the captured image Pa. Alternatively, the lamp color recognizing unit 104 determines the smallest similarity k1 when the smallest similarity k1 among the similarities k1 to k3 of the three lamp part regions Pb1 to Pb3 is equal to or less than the threshold Th. It determines with the 1st lamp part area | region Pb1 to have exists as a dissimilar area | region. Thereby, it is determined that the dissimilar region exists when the smallest similarity is equal to or less than the threshold, and it is determined that the dissimilar region does not exist when the smallest similarity is larger than the threshold. Therefore, it is possible to recognize the lamp color until there is no significant difference between the similarities of the N lamp section areas, and as a result, it is possible to suppress recognition of an inappropriate lamp color.

Further, when it is determined that a dissimilar region exists, the light color recognition unit 104 recognizes the light color based on the position of the dissimilar region in the traffic signal region Pb. For example, at this time, the light color recognition unit 104 refers to the association information. That is, the lamp color recognition unit 104 refers to the association information indicating the colors associated with the respective positions of the three lamp unit areas Pb1 to Pb3 in the traffic signal area Pb. Then, the lamp color recognition unit 104 recognizes the color associated with the position of the dissimilar region as the lamp color in the captured image Pa in the association information. For example, as shown in FIG. 3, the association information indicates the blue color associated with the position of the first lamp section area Pb1 in the traffic light area Pb, that is, the left end. Similarly, the association information indicates the position of the second lamp area Pb2, that is, yellow associated with the center, and the position of the third lamp area Pb3, that is, red associated with the right end. In such association information, the lamp color recognition unit 104 recognizes, for example, the blue color associated with the position of the first lamp unit region Pb1 determined as the dissimilar region as the lamp color in the captured image Pa. .

FIG. 4 is a flowchart showing the processing operation of the traffic signal recognition apparatus 100 in the present embodiment.

The area extraction unit 101 of the traffic signal recognition apparatus 100 acquires a captured image Pa as a recognition target image from the camera 210 (step S101). Then, the area extracting unit 101 extracts an area in which a traffic light having N (for example, N = 3) lamps is displayed as a traffic light area Pb from the captured image Pa that is the recognition target image (step S102). .

Next, the area dividing unit 102 divides the traffic signal area Pb to generate three lamp part areas Pb1 to Pb3 in which the lamp parts are respectively projected (step S103).

Next, the similarity calculation unit 103 calculates the respective similarities k1 to k3 of the three lamp unit regions Pb1 to Pb3 (step S104).

Then, the lamp color recognition unit 104 lights up of the three lamp units of the traffic light based on the similarities k1 to k3 calculated for the three lamp unit areas Pb1 to Pb3. The color of the existing lamp part is recognized as the lamp color in the captured image Pa.

That is, in this lamp color recognition, the lamp color recognizing unit 104 is not similar to any of the other lamp section areas among the three lamp section areas Pb1 to Pb3 based on the similarities k1 to k3. It is determined whether or not there is a dissimilar region that is a lamp region (step S105). If the lamp color recognition unit 104 determines that the dissimilar area exists (Yes in step S105), the lamp color recognition unit 104 recognizes the lamp color based on the position of the dissimilar area in the traffic light area Pb (step S106).

As described above, in the traffic signal recognition apparatus 100 according to the present embodiment, the region extraction unit 101 acquires a captured image Pa as a recognition target image. Then, the area extraction unit 101 extracts an area where a traffic light having N lamp parts is displayed as a traffic light area from the recognition target image. The area dividing unit 102 divides the traffic signal area to generate N lamp part areas each displaying the lamp part. The similarity calculation unit 103 calculates the similarity of each of the N lamp unit regions. The lamp color recognizing unit 104 is not similar to any other lamp unit region among the N lamp unit regions based on the similarity calculated for each of the N lamp unit regions. It is determined whether or not a dissimilar area that is a lamp area exists. Here, if the lamp color recognition unit 104 determines that a dissimilar area exists, the color of the lit lamp part among the N lamp parts is determined based on the position of the dissimilar area in the traffic light area. Is recognized as a lamp color in the recognition target image. For example, the lamp color recognition unit 104 refers to the association information indicating the colors associated with the positions of the N lamp unit areas in the traffic signal area Pb. And the lamp color recognition part 104 recognizes the color matched with the position of a dissimilar area | region as the lamp color in a recognition target image in the matching information. Note that the association information indicating the color associated with each position of the N lamp section regions may be included in the map information as described above, or may not be included in the map information. The association information may be stored in advance in a memory, or may be stored in a cloud server that can communicate with the traffic signal recognition apparatus 100. The light color recognition unit 104 may read the association information from the memory, or may dynamically read the association information from the cloud server. Further, the light color recognition unit 104 may inquire the cloud server about the color associated with the position of the dissimilar region and specify the color.

As a result, if one of the N lamp sections of the traffic light is turned on and the other N-1 lamp sections are turned off, the lamp in which the one lamp section that is lit is displayed. The partial area is determined as a dissimilar area. If the dissimilar area is at the left end of the traffic light area, blue is recognized as the lamp color. As described above, since the lamp color is recognized based on the similarity between the N lamp section areas, that is, the difference in the feature amount of the N lamp section areas, the lamp color can be appropriately recognized.

For example, when color saturation occurs, even if the blue light part of the traffic light is lit, in the light part region corresponding to that light part of the recognition target image, the light part is not blue but is a color close to white or yellow. It may be reflected in. However, in the traffic signal recognition apparatus 100 according to the present embodiment, even if color saturation occurs in the recognition target image, the lamp color is recognized based on the difference in the feature amounts of the N lamp part regions. It is possible to appropriately recognize the light color for the recognition target image.

Accordingly, since color saturation may occur in the recognition target image, it is possible to eliminate the necessity of adjusting the camera 210 to a predetermined shutter speed so that color saturation does not occur when the traffic light is captured by the camera 210. . As a result, the shutter speed of the camera 210 can be fixed. In addition, it is possible to appropriately recognize the light color even with respect to the recognition target image generated by photographing at night or rainy weather where color saturation is likely to occur.

Furthermore, since the lighting color of the light part of the traffic light does not have to be correctly projected on the recognition target image, the camera 210 that is not highly accurate can be used for shooting the traffic light. For example, since environmental conditions are severe in a vehicle, it is difficult to use a high-definition and wide dynamic range camera attached to the vehicle. However, the traffic light recognition apparatus 100 according to the present embodiment can recognize the light color even from a photographed image generated by photographing with the camera 210 that is not highly accurate. Therefore, it is possible to appropriately recognize the light color even for a captured image that is attached to the vehicle and is generated by the camera 210 that is not highly accurate and can withstand severe environmental conditions.

(Modification 1)
Here, there may be a case where an unlit traffic light is displayed on the captured image Pa due to the flicker of the traffic light. In other words, the lighting of the traffic light is periodically performed. For example, the lamp unit blinks at a frequency of 100 Hz or 120 Hz. Therefore, at the timing when the captured image Pa is generated by capturing with the camera 210, there is a case where none of the lamps of the traffic light is lit. In the captured image Pa generated at such a timing, any lamp part of the traffic light is projected darkly.

Therefore, the traffic light recognition device according to the present modification uses the light color recognized for the past captured image when a non-light traffic signal is displayed in the captured image Pa that is the recognition target image. The lamp color in the recognition target image is recognized.

FIG. 5 is a block diagram showing the configuration of the traffic signal recognition apparatus according to this modification.

The traffic signal recognizing device 100a according to the present modification includes the components of the traffic signal recognizing device 100 in the above embodiment, and also includes a history holding unit 105 and a flicker processing unit 106. That is, the traffic signal recognition apparatus 100a includes an area extraction unit 101, an area division unit 102, a similarity calculation unit 103, a light color recognition unit 104, a history holding unit 105, and a flicker processing unit 106.

If the lamp color recognizing unit 104 according to the present modification determines that there is a dissimilar area in the traffic signal area Pb, it outputs the lamp color information to the history holding unit 105 and the flicker processing unit 106. That is, if the traffic light displayed in the captured image Pa that is the recognition target image is not lightless, the light color information is output. On the other hand, if it is determined that there is no dissimilar area in the traffic light area Pb, the light color recognition unit 104 outputs lightless information indicating that the traffic light is lightless to the flicker processing unit 106. That is, when the traffic light shown in the captured image Pa is unlit, unlit information is output.

The history holding unit 105 is a recording medium for holding the lamp color information output from the lamp color recognition unit 104 as history information. Specifically, the history holding unit 105 includes a hard disk or a memory. Note that the memory may be nonvolatile or volatile. The memory may be ROM (Read Only Memory) or RAM (Random Access Memory).

Such a history holding unit 105, when acquiring the lamp color information from the lamp color recognition unit 104, holds the lamp color information as history information. Here, when the lamp color recognition unit 104 stores the lamp color information in the history holding unit 105, the old lamp color information that has already been stored may be deleted. The lamp color recognition unit 104 may store only the latest lamp color information in the history holding unit 105.

When the flicker processing unit 106 acquires the light color information from the light color recognition unit 104, the flicker processing unit 106 outputs the light color information to the vehicle control unit 240. On the other hand, when the flicker processing unit 106 acquires the no-light information from the light color recognition unit 104, the flicker processing unit 106 reads the history information held in the history holding unit 105. Then, the flicker processing unit 106 outputs the history information as lighting color information for the captured image Pa that is the recognition target image.

FIG. 6 is a flowchart showing the processing operation of the traffic signal recognition apparatus 100a according to this modification.

The traffic light recognition apparatus 100a according to the present modification executes the processes of steps S101 to S106, similar to the processing operation of the flowchart shown in FIG.

When the lamp color recognition unit 104 of the traffic signal recognition apparatus 100a recognizes the lamp color in step S106, the lamp color information indicating the lamp color is stored in the history holding unit 105 as history information (step S107). This history information is used when it is determined that there is no dissimilar region in the recognition of the light color for a future captured image.

Then, when the flicker processing unit 106 acquires the lamp color information from the lamp color recognition unit 104, the flicker processing unit 106 outputs the lamp color information to the outside of the traffic light recognition device 100a (step S108). For example, the flicker processing unit 106 outputs the light color information to the vehicle control unit 240.

If it is determined in step S105 that there is no dissimilar region (No in step S105), the light color recognition unit 104 outputs no-light information to the flicker processing unit 106. For example, when a no-light signal is displayed in the captured image Pa that is the recognition target image, the smallest similarity is greater than the threshold Th, and therefore it is determined in step S105 that there is no dissimilar region. The In such a case, the light color recognition unit 104 outputs no-light information to the flicker processing unit 106. When the flicker processing unit 106 acquires the no-light information, the flicker processing unit 106 reads the history information from the history holding unit 105 (step S109). The flicker processing unit 106 outputs the read history information as lighting color information for the recognition target image (step S110).

As described above, in the traffic signal recognition apparatus 100a according to the present modification, the flicker processing unit 106 applies to the captured image acquired in the past when the light color recognition unit 104 determines that there is no dissimilar region. The history information indicating the recognized light color is referred to. The flicker processing unit 106 recognizes the lamp color indicated by the history information as the lamp color in the recognition target image.

Thus, when there is no dissimilar region in the recognition target image, the lamp color recognized for the past photographed image is recognized as the lamp color in the recognition target image. Accordingly, it is possible to appropriately prevent the recognition of the light color due to flicker.

The past photographed image is, for example, a photographed image (that is, a frame) immediately before the recognition target image generated by photographing at a constant frame rate by the camera 210, for example. Therefore, there is a high possibility that the lamp unit that has been turned off by flicker at the timing of capturing the recognition target image is lit at the timing of capturing the previous captured image. Therefore, by referring to the history information as described above, it is possible to appropriately recognize the light color even if flicker occurs.

(Modification 2)
The cause that makes it difficult to recognize the light color in the recognition target image is not limited to the flicker of the traffic light. For example, a captured image in which it is difficult to recognize the light color may be generated due to noise in the camera 210 or a shooting environment.

Therefore, the traffic light recognition apparatus according to the present modification recognizes the light color in the recognition target image using the processing results for a plurality of past captured images.

FIG. 7 is a block diagram showing the configuration of the traffic signal recognition apparatus according to this modification.

The traffic signal recognizing device 100b according to the present modification includes the components of the traffic signal recognizing device 100 in the above embodiment, and also includes a history holding unit 105 and a time series processing unit 107. That is, the traffic signal recognition apparatus 100a includes an area extraction unit 101, an area division unit 102, a similarity calculation unit 103, a light color recognition unit 104, a history holding unit 105, and a time series processing unit 107.

When the lamp color recognition unit 104 according to the present modification determines that there is a dissimilar area in the traffic signal area Pb, the lamp color information is output to the history holding unit 105 as in the first modification. On the other hand, if it is determined that there is no dissimilar area in the traffic signal area Pb, the lamp color recognition unit 104 recognizes the lamp color as black, for example, and outputs lamp color information indicating the black color to the history holding unit 105.

The history holding unit 105, when acquiring the lamp color information from the lamp color recognition unit 104, holds the lamp color information as a part of the history information. For example, the history holding unit 105 has a storage capacity for holding L pieces of light color information (L is an integer of 3 or more). Therefore, when the lamp color information is stored in the history holding unit 105 when the L lamp color information is already stored, the lamp color recognition unit 104 is the oldest among the lamp color information. Delete the light color information. Thereby, free capacity is secured in the history holding unit 105. Then, the lamp color recognition unit 104 stores the latest lamp color information in the history holding unit 105 in which the free space is secured. Thereby, the history holding unit 105 always holds the latest L pieces of lamp color information as history information.

The time series processing unit 107 reads out the history information stored in the history holding unit 105 every time the lighting color information is stored in the history holding unit 105. The history information includes the latest L pieces of lamp color information. The time series processing unit 107 updates the already recognized lamp color for the recognition target image based on the lamp color indicated by each of the L lamp color information. In other words, the lamp color recognition unit 104 performs a lamp color temporary determination on the recognition target image, and the time-series processing unit 107 detects the lamp color recognized for the past (L-1) photographed images. Is used to make a final determination of the light color for the recognition target image.

FIG. 8 is a diagram for explaining the recognition of the light color by the traffic signal recognition apparatus 100b according to the present modification.

For example, when L = 4, the history holding unit 105 holds, as history information, lamp color information indicating the lamp color recognized for each of the past four captured images. These four captured images are composed of the (n-4) th frame, the (n-3) th frame, the (n-2) th frame, and the (n-1) th frame. For example, the lighting color information of the (n−1) th frame included in the history information indicates black as the lighting color, and the other three lighting color information indicates blue as the lighting color. Each frame is a captured image generated by shooting at a constant frame rate by the camera 210.

Here, the lamp color recognition unit 104 recognizes the lamp color with respect to the nth frame which is the latest captured image and is the recognition target image. For example, the lamp color recognition unit 104 recognizes a blue lamp color. The lamp color recognition unit 104 stores lamp color information indicating the lamp color recognized for the nth frame in the history holding unit 105. At this time, the light color recognition unit 104 first deletes the light color information of the (n−4) th frame from the history information stored in the history holding unit 105. Next, the lamp color recognition unit 104 stores lamp color information indicating the lamp color recognized for the nth frame in the history holding unit 105 as new lamp color information. As a result, the history information of the history holding unit 105 includes the light color information of the nth frame instead of the light color information of the (n-4) th frame.

Next, the time series processing unit 107 reads out the history information stored in the history holding unit 105. Then, the time-series processing unit 107 updates the lamp color (for example, blue) recognized for the nth frame based on the lamp color indicated by each of the latest four lamp color information included in the history information. To do. That is, the time-series processing unit 107 uses the lamp colors obtained by provisional determination for each of the (n-3) th frame, the (n-2) th frame, the (n-1) th frame, and the nth frame. The final determination of the light color for the nth frame is performed.

Specifically, the time-series processing unit 107 performs a majority decision of the lamp color indicated by each of the four lamp color information, and determines the lamp color recognized for the nth frame as the lamp color determined by the majority vote. Update to For example, the time-series processing unit 107 recognizes the lamp colors recognized for each of the (n-3) th frame, the (n-2) th frame, the (n-1) th frame, and the nth frame, that is, blue Blue, black, and blue majority vote. The time-series processing unit 107 specifies the lamp color determined by the majority decision, that is, blue, as the majority lamp color. Then, the time series processing unit 107 updates the lamp colors recognized for the nth frame to the multiple lamp colors.

FIG. 9 is a flowchart showing a processing operation by the traffic signal recognition apparatus 100b according to the present modification.

First, the traffic signal recognition apparatus 100b according to the present modification executes steps S101 to S106 in the same manner as the processing operation of the flowchart shown in FIG.

Next, when the lamp color recognition unit 104 of the traffic light recognition apparatus 100b recognizes the lamp color in step S106, the lamp color information indicating the lamp color is stored in the history holding unit 105 as part of the history information (step S122). . If the lamp color recognition unit 104 determines that there is no dissimilar area in step S105 (No in step S105), for example, the lamp color recognition unit 104 recognizes the lamp color for the recognition target image as black (step S121). Then, the lamp color recognition unit 104 stores the lamp color information indicating the black color in the history holding unit 105 as part of the history information (step S122).

Next, the time series processing unit 107 reads the history information stored in the history holding unit 105 (step S123). Then, the time-series processing unit 107 updates the lamp color in the recognition target image recognized in step S106 or S121 by the majority of the four lamp colors indicated by the history information (step S124).

In the above example, the history holding unit 105 holds history information including four pieces of light color information. However, the number of lamp color information included in the history information is not limited to four, and may be three or four or more.

In the above-described example, the time series processing unit 107 reads the history information from the history holding unit 105 to acquire the light color information of the recognition target image included in the history information. However, the time series processing unit 107 may directly acquire the light color information of the recognition target image from the light color recognition unit 104. In this case, the history information read from the history holding unit 105 by the time-series processing unit 107 does not include the light color information of the recognition target image, but includes only the light color information of the past captured image.

As described above, in the traffic signal recognition apparatus 100b according to the present modification, the time-series processing unit 107 includes history information indicating the lamp color recognized for each of the plurality of captured images acquired before the recognition target image. Refer to Then, the time-series processing unit 107 calculates the largest number of lamp colors among the lamp colors recognized for the recognition target image and the lamp colors in each of the plurality of captured images indicated by the history information. Specify as color. The time-series processing unit 107 updates the lamp colors recognized for the recognition target image to the multiple lamp colors.

As a result, for example, even if it is suddenly determined that there is no dissimilar area in the recognition target image due to factors other than flicker such as noise, or an incorrect light color is recognized for the recognition target image. The error can be easily corrected.

(Modification 3)
As in the second modification, the traffic signal recognition apparatus according to the second modification recognizes the light color in the recognition target image using the processing results for a plurality of past captured images. However, the traffic light recognition apparatus according to the present modification uses not the lamp color recognition result but the similarity calculation result as the processing result for a plurality of past captured images.

FIG. 10 is a block diagram showing a configuration of a traffic signal recognition apparatus according to this modification.

The traffic light recognition device 100c according to this modification includes a light color recognition unit 104c instead of the light color recognition unit 104 among the components of the traffic light recognition device 100 in the above embodiment. Further, the traffic signal recognition apparatus 100 c includes a history holding unit 105. That is, the traffic signal recognition apparatus 100a includes an area extraction unit 101, an area division unit 102, a similarity calculation unit 103, a light color recognition unit 104c, and a history holding unit 105.

The similarity calculation unit 103 according to the present modification calculates the similarity of each of the N lamp unit regions in the traffic signal region Pb. Then, for each position in the traffic light area Pb, the similarity calculation unit 103 outputs similarity information indicating the similarity of the lamp area at the position to the history holding unit 105. That is, the similarity calculation unit 103 stores the similarity information in the history holding unit 105. For example, when N = 3, the similarity information includes the similarity of the first lamp area Pb1 at the left end of the traffic light area Pb, the similarity of the second lamp area Pb2 at the center of the traffic light area Pb, and the traffic light area. The similarity of the 3rd lamp part area | region Pb3 in the right end of Pb is shown.

The history holding unit 105 holds the similarity information output from the similarity calculation unit 103 as part of the history information. For example, the history holding unit 105 has a storage capacity for holding L pieces of similarity information (L is an integer of 2 or more). Therefore, when the similarity calculation unit 103 stores the similarity information in the history holding unit 105, if the L old similarity information is already stored, the oldest similarity among the similarity information is stored. Delete information. Thereby, free capacity is secured in the history holding unit 105. Then, the similarity calculation unit 103 stores the latest similarity information in the history holding unit 105 in which the free space is secured. Thereby, the history holding unit 105 always holds the latest L pieces of similarity information as history information.

The lighting color recognition unit 104c according to the present modification reads the history information stored in the history holding unit 105 every time the similarity information is stored in the history holding unit 105. The history information includes L pieces of recent similarity information. The light color recognition unit 104c calculates the average of the similarities indicated by the recent L pieces of similarity information as the average similarity. For example, when N = 3, the similarity information indicates the similarity of the lamp area at each of the left end, the center, and the right end of the traffic light area Pb as described above. Therefore, the lamp color recognition unit 104c calculates the average of the similarity of the lamp region at the left end indicated by the L pieces of similarity information as the average similarity of the lamp region. Similarly, the lamp color recognition unit 104c calculates the average similarity of the lamp area at the center as the average similarity of the lamp area, and calculates the average similarity of the lamp area at the right end. Calculated as the average similarity of partial areas. Then, the lamp color recognizing unit 104c determines that the smallest average similarity among the calculated three average similarity is, for example, the average similarity of the lamp region at the left end, in the traffic light region Pb of the recognition target image. The lamp area at the left end is specified as a dissimilar area.

FIG. 11 is a diagram for explaining the recognition of the light color by the traffic signal recognition device 100c according to this modification.

For example, when L = 4, the history holding unit 105 holds similarity information indicating similarity calculated for each of the past four captured images as history information. These four captured images are composed of the (n-4) th frame, the (n-3) th frame, the (n-2) th frame, and the (n-1) th frame. Note that these frames are captured images generated by capturing at a constant frame rate by the camera 210.

Here, when N = 3, the similarity calculation unit 103 calculates the similarity of each of the three lamp unit regions with respect to the nth frame which is the latest captured image and is the recognition target image. Then, the similarity calculation unit 103 stores similarity information indicating the similarity calculated for the n-th frame in the history holding unit 105. At this time, the similarity calculation unit 103 first deletes the similarity information of the (n−4) th frame from the history information stored in the history holding unit 105. Next, the similarity calculation unit 103 stores similarity information indicating the similarity calculated for the n-th frame in the history holding unit 105 as new similarity information. As a result, the history information of the history holding unit 105 includes the similarity information of the nth frame instead of the similarity information of the (n−4) th frame.

Next, the light color recognition unit 104 c reads the history information stored in the history holding unit 105. The history information includes similarity information of the (n-3) th frame, similarity information of the (n-2) th frame, similarity information of the (n-1) th frame, and similarity of the nth frame. Degree information is included. For example, the similarity information of the (n-3) th frame indicates 30/101/99 as the similarity of the lamp area at each of the left end, the center, and the right end in the traffic light area Pb. The similarity information of the (n−2) th frame indicates 70/111/105 as the similarity of the lamp area at each of the left end, the center, and the right end in the traffic light area Pb. The similarity information of the (n−1) th frame indicates 107/110/114 as the similarity of the lamp area at each of the left end, the center, and the right end in the traffic light area Pb. The similarity information of the nth frame indicates 21/112/105 as the similarity of the lamp area at each of the left end, the center, and the right end in the traffic light area Pb.

The lamp color recognition unit 104c calculates the average of the similarity of the lamp area at the left end in the traffic signal area Pb indicated by the similarity information of each of the four frames. That is, the lamp color recognition unit 104c calculates the average similarity of the leftmost lamp unit region by (30 + 70 + 107 + 21) / 4. Similarly, the lamp color recognition unit 104c calculates the average similarity of the central lamp unit area by (101 + 111 + 110 + 112) / 4, and calculates the average similarity of the rightmost lamp unit region by (99 + 105 + 114 + 105) / 4.

Thereby, the lamp color recognition unit 104c calculates 57/108/106 as the average similarity of the lamp area at each of the left end, the center, and the left end in the traffic light area Pb.

Then, the lamp color recognition unit 104c identifies the position corresponding to the smallest average similarity “57” among these average similarities, that is, the lamp region at the left end, as a dissimilar region. That is, the lamp color recognizing unit 104c determines that the leftmost lamp unit region among the three lamp unit regions in the recognition target image exists as a dissimilar region.

Next, similarly to the lamp color recognition unit 104 in the above embodiment, the lamp color recognition unit 104c recognizes the lamp color in the recognition target image based on the position of the dissimilar region. For example, if the lamp color recognition unit 104c determines that the leftmost lamp unit region exists as a dissimilar region as described above, it recognizes blue as the lamp color in the recognition target image.

FIG. 12 is a flowchart showing a processing operation by the traffic signal recognition apparatus 100c according to the present modification.

First, the traffic signal recognition apparatus 100c according to the present modification executes the processing of steps S101 to S104, similarly to the processing operation of the flowchart shown in FIG.

Next, the similarity calculation unit 103 of the traffic signal recognition apparatus 100c stores the similarity information indicating the similarity of each lamp unit area calculated in step S104 in the history holding unit 105 (step S104a).

Next, the light color recognition unit 104c reads the history information stored in the history holding unit 105 (step S104b). That is, the latest L pieces of similarity information (for example, L = 4) included in the history information are read out. Then, when N = 3, the lamp color recognition unit 104c calculates the average similarity of the lamp unit regions at the left end, the center, and the left end in the traffic signal region Pb (step S104c).

Next, based on the average similarity calculated for each of the three lamp section areas, the lamp color recognition section 104c determines which of the three lamp sections that the traffic light has. The color is recognized as the lamp color in the recognition target image.

That is, the lamp color recognition unit 104c has a dissimilar region that is a lamp region that is not similar to any other lamp unit region among the three lamp unit regions based on the average similarity. It is determined whether or not (step S105). If the lamp color recognition unit 104 determines that the dissimilar area exists (Yes in step S105), the lamp color recognition unit 104 recognizes the lamp color based on the position of the dissimilar area in the traffic light area Pb (step S106).

In the above example, the lamp color recognition unit 104c calculates the average of the similarity of the lamp area in each frame, but may calculate the weighted average as the average similarity. For example, a frame (captured image) that is closer in time to the recognition target image is multiplied by a greater weight to the similarity of the lamp area in that frame. This makes it possible to calculate a more appropriate average similarity for the recognition target image.

In the above example, the light color recognition unit 104c reads the history information from the history holding unit 105, thereby acquiring similarity information of the recognition target image included in the history information. However, the light color recognition unit 104c may directly acquire the similarity information of the recognition target image from the similarity calculation unit 103. In this case, the history information read from the history holding unit 105 by the lamp color recognition unit 104c does not include the similarity information of the recognition target image, and includes only the similarity information of the past captured images.

As described above, in the traffic light recognition device 100c according to the present modification, the lamp color recognition unit 104c has history information indicating similarity calculated for each of a plurality of captured images acquired before the recognition target image. Refer to This history information indicates, for each of a plurality of photographed images, the degree of similarity of the lamp section region at the position for each position in the traffic light region Pb in the photographed image. Then, for each position in the traffic light area, the lamp color recognition unit 104c calculates the average similarity of the lamp area at the position in the recognition target image and the plurality of images as the average similarity. Further, the lamp color recognizing unit 104c has a lamp unit region at a position corresponding to the smallest average similarity among the average similarity calculated for each of the N lamp unit regions in the recognition target image. It is determined that it exists as a dissimilar region.

As a result, for example, even if it is suddenly determined that there is no dissimilar area in the recognition target image due to factors other than flicker such as noise, or an incorrect light color is recognized for the recognition target image. The error can be corrected. Further, since the dissimilar region is determined based on the average similarity using the past similarity, the dissimilar region can be determined more accurately than using the light color recognized in the past. .

(Modification 4)
In the above embodiment and Modifications 1 to 3, the traffic light recognition device recognizes the light color of the traffic light. However, the traffic signal recognizing device may recognize not only the lamp color but also the traveling direction indicated by the lit arrow light portion for the traffic light having the arrow light portion. The lamp part indicated by the arrow is hereinafter referred to as a direction indicator lamp part, and the lamp part indicating a color such as blue, yellow, or blue is hereinafter also referred to as a color lamp part.

FIG. 13 is a diagram for explaining the recognition of the traveling direction by the traffic signal recognition device according to the present modification.

For example, as shown in FIG. 13, the traffic signal includes not only a blue color lamp unit, a yellow color lamp unit, and a red color lamp unit, but also a direction indicator lamp unit that indicates the right direction as the traveling direction.

In such a case, the area extraction unit 101 of the traffic light recognition device extracts an area in which the three color lamp units and one direction indicator lamp unit are projected from the captured image Pa as the traffic signal area Pb. Then, the area dividing unit 102 divides the traffic light area Pb to thereby divide the lamp part areas Pb1, Pb2, and Pb3 in which each of the three color lamp parts is displayed, and the lamp in which the direction indicator lamp part is displayed. A partial area Pb4 is generated.

The traffic signal recognizing device recognizes the light color of the traffic signal based on the lamp area Pb1, Pb2 and Pb3 as in the above embodiment and the first to third modifications. Then, the traffic light recognition device determines that the direction indicator lamp unit is not lit if the lamp color is a predetermined color. For example, when the lamp color is blue or yellow, the traffic light recognition device determines that the direction indicator lamp unit is not lit. On the other hand, when the lamp color is red, the traffic light recognition device determines whether or not the direction indicator lamp unit is lit using the lamp unit region Pb4. For example, the traffic light recognition apparatus inputs the feature amount of the lamp part region Pb4 to a model such as a neural network generated by machine learning. The traffic signal recognizing device determines whether or not the direction indicator lamp unit is lit according to the output from the model.

Alternatively, the traffic light recognition device calculates a correlation coefficient between a region other than the dissimilar region (hereinafter referred to as a similar region) of the lamp unit regions Pb1, Pb2, and Pb3 and the lamp unit region Pb4. It may be determined whether the number of relationships is greater than or equal to a threshold value. That is, since the similar area is darkly displayed in the captured image, if the correlation coefficient is equal to or greater than the threshold value, the lamp area Pb4 is also dark and the direction indicator lamp section is likely to be turned off. If the correlation coefficient is less than the threshold, the lamp area Pb4 is bright and the direction indicator lamp section is likely to be lit. Therefore, the traffic light recognition device determines that the direction indicator lamp unit is turned off when the correlation coefficient is equal to or greater than the threshold value. Conversely, the traffic light recognition device determines that the direction indicator lamp unit is lit when the correlation coefficient is less than the threshold value.

When the traffic light recognition device determines that the direction indicator lamp unit is lit as described above, it refers to the association information. Similar to the association information shown in FIG. 3, this association information indicates the colors associated with the positions of the lamp areas Pb1, Pb2, and Pb3 and is associated with the position of the lamp area Pb4. Indicates the direction of travel. In the association information, the traffic signal recognition device recognizes the traveling direction associated with the position of the lamp unit region Pb4, for example, the right direction, as the traveling direction of the lit direction indicator lamp unit.

Here, a traffic signal having a plurality of direction indicator lamps is also arranged on the road. In this traffic light, only one of the direction indicator lamps is turned on, or two or more direction indicator lamps are turned on simultaneously.

Even for such a traffic light, the traffic light recognition device recognizes the light color of the traffic light based on the respective lamp area of the N lamp sections, as in the above embodiment and the first to third modifications. . Then, the traffic signal recognizing device narrows down the direction indicator lamp units that are lit from M (M is an integer of 1 or more) direction indicator lamp units based on the lamp color. For example, if the lamp color is blue, the traffic light recognition device narrows down the direction indicator lamp unit indicating the straight direction as a candidate, and if the lamp color is red, the traffic signal recognition device detects the direction indicator lamp unit indicating the right direction. Narrow down as candidates. Then, similarly to the above, the traffic signal recognizing device determines whether or not the candidate direction indicator lamp unit is lit using a model or a correlation coefficient.

As described above, in the present modification, when the traffic light has M (M is an integer of 1 or more) direction indicator lamps, each of which indicates a traveling direction of the vehicle, the traffic signal recognition device is turned on. The traveling direction indicated by the direction indicator lamp unit is recognized. That is, the area dividing unit 102 divides the traffic light area Pb, thereby generating N lamp part areas and M lamp part areas in which the direction indicating lamp parts are respectively projected. The traffic light recognition device is further turned on in the M direction indicator lamp units based on the lamp color recognized for the recognition target image and the feature quantities of the M lamp unit regions. The traveling direction indicated by the at least one direction indicator lamp unit is recognized. Alternatively, the traffic light recognition device first recognizes a traveling direction indicated by at least one direction indication lamp unit that is lit among the M direction indication lamp units, and based on the recognition result, the color lamp unit You may recognize the light color. Note that the above-described feature amount may be a vector in the above embodiment or a correlation coefficient.

Thereby, for example, among the M number of direction indicator lamps indicating the traveling direction by an arrow or the like, it is possible to appropriately recognize the lit direction indicator lamp part. For example, in the case where a direction indicator lamp unit that may be lit is predetermined for each lamp color of the traffic light, the direction in which the M direction indicator lamp units are lit up according to the recognized lamp color. It is possible to narrow down the candidates for the indicator lamp part. Further, for example, by inputting each feature amount of the M lamp area into a model such as a neural network generated by machine learning, a directional indicator lamp that is lit is selected from the narrowed candidates. Can be recognized properly.

(Modification 5)
In the above-described embodiment and the modifications thereof, when calculating the similarity of each of the N lamp areas, the lamp area and each of the remaining (N−1) lamp areas are individually Compare to. On the other hand, in this modification, the similarity of the lamp area is calculated by comparing the lamp area with a group of the remaining (N−1) lamp areas.

FIG. 14 is a diagram illustrating an example of a method of calculating the similarity of the first lamp part region Pb1 in the above embodiment.

In the above-described embodiment, the similarity calculation unit 103 divides the first lamp unit region Pb1 into a plurality of blocks each composed of n × n pixels as described above. And the similarity calculation part 103 searches the block most similar to the image of the block from 2nd lamp part area | region Pb2 for every block of 1st lamp part area | region Pb1. A block is also called a patch.

For example, as shown in FIG. 14A, the similarity calculation unit 103 searches the second lamp part area Pb2 for a block most similar to the image of the block B11 at the upper left corner of the first lamp part area Pb1. Then, the similarity calculation unit 103 calculates the inter-vector distance (for example, “5”) between the block B11 and the most similar block searched from the second lamp unit region Pb2. The larger the distance between the vectors, the more the distance between the vectors indicates that the two blocks are not similar. The smaller the distance between the vectors, the more similar the distance between the two blocks is. Indicates. Therefore, it can be said that the distance between vectors is dissimilarity.

Next, as shown in FIG. 14B, the similarity calculation unit 103 performs the same processing as that of the block B11 on the block B12 next to the block B11. That is, the similarity calculation unit 103 searches the second lamp part area Pb2 for a block most similar to the image of the second block B12 from the upper left end of the first lamp part area Pb1 to the right. Then, the similarity calculation unit 103 calculates the inter-vector distance between the block B12 and the most similar block searched from the second lamp unit region Pb2.

The similarity calculation unit 103 repeatedly executes the processes shown in FIGS. 14A and 14B, so that each block included in the first lamp area Pb1 is shown in FIG. 14C. On the other hand, a distance between vectors (that is, dissimilarity) is calculated.

The similarity calculation unit 103 calculates the average or sum of the dissimilarities of these blocks, and the reciprocal of the calculation result is used as the correlation coefficient k12 of the first lamp part region Pb1 with respect to the second lamp part region Pb2. calculate. Similarly, the similarity calculation unit 103 calculates a correlation coefficient k13 of the first lamp part region Pb1 with respect to the third lamp part region Pb3, and calculates the largest correlation coefficient among the correlation coefficient k12 and the correlation coefficient k13. , Calculated as the similarity k1 of the first lamp area Pb1.

Thus, in the above embodiment, in the calculation of the similarity of the first lamp section area Pb1, the second lamp section area Pb1 is compared with the second lamp section area Pb2 by comparing the first lamp section area Pb1 and the second lamp section area Pb2. A correlation coefficient k12 for the lamp area Pb2 is calculated as the first similarity. Further, by comparing the first lamp part area Pb1 and the third lamp part area Pb3, the correlation coefficient k13 of the first lamp part area Pb1 with respect to the third lamp part area Pb3 is calculated as the second similarity. Then, the largest similarity between the first similarity and the second similarity is calculated as the similarity of the first lamp area Pb1. Therefore, in the calculation of the similarity of the first lamp part area Pb1, the first lamp part area Pb1, and each of the second lamp part area Pb2 and the third lamp part area Pb3 are individually compared. The similarity between the second lamp part area Pb2 and the third lamp part area Pb3 is also calculated in the same manner as the similarity between the first lamp part area Pb1.

In this modification, the similarity calculation unit 103 calculates the similarity of the first lamp part area Pb1, and each of the first lamp part area Pb1, the second lamp part area Pb2, and the third lamp part area Pb3. Are not compared individually. That is, the similarity calculation unit 103 compares the first lamp part area Pb1 with the group including the second lamp part area Pb2 and the third lamp part area Pb3, thereby determining the similarity of the first lamp part area Pb1. calculate.

FIG. 15 is a diagram illustrating an example of a method of calculating the similarity of each lamp area according to the present modification.

As shown in FIG. 15A, the similarity calculating unit 103 calculates the similarity of the first lamp unit area Pb1, as shown in FIG. 15A, the first lamp unit area Pb1, the second lamp unit area Pb2, and the third lamp unit. The first group G1 composed of the region Pb3 is compared. That is, the similarity calculation unit 103 divides the first lamp unit area Pb1 into a plurality of blocks, and searches the first group G1 for the block most similar to the image of the block for each block of the first lamp unit area Pb1. To do. Then, similar to the example shown in FIG. 14, the similarity calculation unit 103 obtains the inter-vector distance of each block included in the first lamp region Pb1 by the search, and based on the inter-vector distance, The correlation coefficient of one lamp area Pb1 is calculated as the similarity.

As shown in FIG. 15B, the similarity calculation unit 103 calculates the similarity of the second lamp unit area Pb2, as shown in FIG. 15B, the second lamp unit area Pb2, the first lamp unit area Pb1, and the third lamp unit. The second group G2 composed of the region Pb3 is compared. That is, the similarity calculation unit 103 divides the second lamp unit region Pb2 into a plurality of blocks, and searches the second group G2 for the block most similar to the image of the block for each block of the second lamp unit region Pb2. To do. Similar to the example shown in FIG. 14, the similarity calculation unit 103 obtains the inter-vector distance of each block included in the second lamp part region Pb2 by the search, and based on the inter-vector distance, The correlation coefficient of the two lamp area Pb2 is calculated as the similarity.

As shown in FIG. 15C, the similarity calculation unit 103 calculates the similarity of the third lamp unit area Pb3, as shown in FIG. 15C, the third lamp unit area Pb3, the first lamp unit area Pb1, and the second lamp unit. The third group G3 composed of the region Pb2 is compared. That is, the similarity calculation unit 103 divides the third lamp unit region Pb3 into a plurality of blocks, and searches for the block most similar to the image of the block from the third group G3 for each block of the third lamp unit region Pb3. To do. Then, similar to the example shown in FIG. 14, the similarity calculation unit 103 obtains the inter-vector distance of each block included in the third lamp part region Pb2 by the search, and based on the inter-vector distance, The correlation coefficient of the three lamp area Pb3 is calculated as the similarity.

Thus, the respective similarities of the first lamp area Pb1, the second lamp area Pb2, and the third lamp area Pb3 are calculated using the groups. For example, if the lamp part projected in the first lamp part area Pb1 is turned on and the other lamp parts are turned off, a low similarity is calculated only for the first lamp part area Pb1, and the second lamp part is calculated. A high degree of similarity is calculated for each of the partial area Pb2 and the third lamp part area Pb3. Therefore, even with the calculation method shown in FIG. 15, it is possible to calculate an appropriate degree of similarity for each lamp area, as in the above embodiment.

FIG. 16 is a diagram showing another example of a method for calculating the similarity of each lamp section according to the present modification.

In the example shown in FIG. 15, the lamp section area that is the target of similarity calculation is divided into a plurality of blocks, but conversely, the group may be divided into a plurality of blocks.

For example, the similarity calculation unit 103 calculates the similarity of the third lamp part region Pb3 as shown in FIG. 16A by using a third lamp part region Pb1 and a second lamp part region Pb2. The group G3 is compared with the third lamp part region Pb3. That is, the similarity calculation unit 103 divides the third group G3 into a plurality of blocks, and searches the third lamp unit region Pb3 for a block most similar to the image of each block of the third group G3. Next, similar to the example shown in FIG. 14, the similarity calculation unit 103 obtains the inter-vector distance of each block included in the third group G3 by the search. And the similarity calculation part 103 calculates the correlation coefficient of the 3rd group G3 as a similarity of 3rd lamp part area | region Pb3 based on each vector distance of several blocks of the 3rd group G3. .

Further, in the calculation of the similarity of the second lamp section area Pb2, the similarity calculation section 103 calculates the second lamp composed of the third lamp section area Pb3 and the first lamp section area Pb1, as shown in FIG. The group G2 is compared with the second lamp area Pb2. That is, the similarity calculation unit 103 divides the second group G2 into a plurality of blocks, and searches the second lamp unit region Pb2 for a block most similar to the image of each block of the second group G2. Next, similar to the example shown in FIG. 14, the similarity calculation unit 103 obtains the inter-vector distance of each block included in the second group G2 by the search. And the similarity calculation part 103 calculates the correlation coefficient of the 2nd group G2 as a similarity of 2nd lamp part area | region Pb2 based on each vector distance of the some block of 2nd group G2. .

Further, the similarity calculation unit 103 calculates the similarity of the first lamp part area Pb1, as shown in FIG. 16C, the first lamp part area Pb3 and the second lamp part area Pb2. The group G1 is compared with the first lamp area Pb1. That is, the similarity calculation unit 103 divides the first group G1 into a plurality of blocks, and searches the first lamp unit region Pb1 for a block most similar to the image of the block for each block of the first group G1. Next, similar to the example shown in FIG. 14, the similarity calculation unit 103 obtains the inter-vector distance of each block included in the first group G1 by the search. And the similarity calculation part 103 calculates the correlation coefficient of the 1st group G1 as a similarity of 1st lamp | ramp part area | region Pb1 based on each inter-vector distance of the some block of 1st group G1. .

In this way, in the example shown in FIG. 16 as well, as in the example shown in FIG. 15, the respective similarities of the first lamp part area Pb1, the second lamp part area Pb2, and the third lamp part area Pb3 are calculated using groups. Is done. For example, if the lamp part projected in the first lamp part area Pb1 is turned on and the other lamp parts are turned off, a low similarity is calculated only for the first lamp part area Pb1, and the second lamp part is calculated. A high degree of similarity is calculated for each of the partial area Pb2 and the third lamp part area Pb3. Therefore, even in the calculation method shown in FIG. 16, an appropriate similarity can be calculated for each lamp area, as in the above embodiment.

FIG. 17 is a diagram illustrating an example of a method for calculating the similarity of the four lamp areas (N = 4) according to the present modification.

For example, the camera 210 images a traffic light having four lamp parts (that is, N = 4). As a result, the traffic light area of the recognition target image obtained by the photographing has four lamp part areas, that is, the first lamp part area Pb1, the second lamp part area Pb2, the third lamp part area Pb3, and the fourth lamp part. Divided into regions Pb4.

Therefore, the similarity calculation unit 103 calculates the similarity of the fourth lamp part area Pb4, as shown in FIG. 17A, the first lamp part area Pb1, the second lamp part area Pb2, and the third lamp part. The fourth group Gp4 including the partial area Pb3 is compared with the fourth lamp part area Pb4. That is, the similarity calculation unit 103 divides the fourth group Gp4 into a plurality of blocks, and searches the fourth lamp unit region Pb4 for a block most similar to the image of the block for each block of the fourth group Gp4. Next, similarly to the example shown in FIG. 14, the similarity calculation unit 103 obtains the inter-vector distance of each block included in the fourth group Gp4 by the search. And the similarity calculation part 103 calculates the correlation coefficient of the 4th group Gp4 as a similarity of 4th lamp part area | region Pb4 based on each vector distance of 4th group Gp4.

Further, the similarity calculation unit 103 calculates the similarity of the first lamp area Pb1, as shown in FIG. 17B, the second lamp area Pb2, the third lamp area Pb3, and the fourth lamp. The first group Gp1 including the partial area Pb4 is compared with the first lamp part area Pb1. That is, the similarity calculation unit 103 divides the first group Gp1 into a plurality of blocks, and searches the first lamp unit region Pb1 for a block most similar to the image of the block for each block of the first group Gp1. Next, similar to the example shown in FIG. 14, the similarity calculation unit 103 obtains an inter-vector distance of each block included in the first group Gp1 by the search. And the similarity calculation part 103 calculates the correlation coefficient of the 1st group Gp1 as a similarity of 1st lamp part area | region Pb1 based on each vector distance of 1st group Gp1.

Further, the similarity calculation unit 103 calculates the similarity of the second lamp part area Pb2, as shown in FIG. 17C, the third lamp part area Pb3, the fourth lamp part area Pb4, and the first lamp part. The second group Gp2 including the partial area Pb1 is compared with the second lamp area Pb2. That is, the similarity calculation unit 103 divides the second group Gp2 into a plurality of blocks, and searches the second lamp unit region Pb2 for a block most similar to the image of the block for each block of the second group Gp2. Next, similar to the example shown in FIG. 14, the similarity calculation unit 103 obtains the inter-vector distance of each block included in the second group Gp2 by the search. And the similarity calculation part 103 calculates the correlation coefficient of the 2nd group Gp2 as a similarity of 2nd lamp part area | region Pb2 based on each vector distance of 2nd group Gp2.

Further, in calculating the similarity of the third lamp part area Pb3, the similarity calculation unit 103 calculates the fourth lamp part area Pb4, the first lamp part area Pb1, and the second lamp as shown in FIG. The third group Gp3 composed of the partial area Pb2 is compared with the third lamp part area Pb3. That is, the similarity calculation unit 103 divides the third group Gp3 into a plurality of blocks, and searches the third lamp unit region Pb3 for a block most similar to the image of the block for each block of the third group Gp3. Next, similar to the example shown in FIG. 14, the similarity calculation unit 103 obtains the inter-vector distance of each block included in the third group Gp3 by the search. And the similarity calculation part 103 calculates the correlation coefficient of the 3rd group Gp3 as a similarity of 3rd lamp part area | region Pb3 based on each vector distance of 3rd group Gp3.

Thus, in the example shown in FIG. 17 as well, as in the examples shown in FIGS. 15 and 16, the similarities of the first lamp part area Pb1, the second lamp part area Pb2, and the third lamp part area Pb3 are grouped. Is used to calculate. For example, if the lamp part projected in the first lamp part area Pb1 is turned on and the other lamp parts are turned off, a low similarity is calculated only for the first lamp part area Pb1, and the second lamp part is calculated. A high degree of similarity is calculated for each of the partial area Pb2 and the third lamp part area Pb3. Therefore, even with the calculation method shown in FIG. 17, it is possible to calculate an appropriate degree of similarity for each lamp area, as in the above embodiment.

That is, in the present modification, as shown in FIGS. 15 to 16, the similarity calculation unit 103, for each of the N lamp unit regions, includes an image of the lamp unit region and at least one other than the lamp unit region. The similarity of the lamp area is calculated by comparing an image of a group of two other lamp areas. For example, as shown in FIG. 15, the similarity calculation unit 103 includes, for each of the N lamp unit regions, an image of the lamp unit region and the group image included in the lamp unit region. For each block, the similarity of the lamp area is calculated by searching the group for the block most similar to the image of the block. Alternatively, as shown in FIG. 16 and FIG. 17, when the similarity calculation unit 103 compares the image of the lamp unit region with the image of the group for each of the N lamp unit regions, For each included block, the similarity of the lamp area is calculated by searching the lamp area for the block most similar to the image of the block.

Thus, for example, even when a group is composed of a plurality of lamp areas, when calculating the similarity of the lamp areas, the lamp area and each of the other lamp areas are individually set. It is possible to save time and effort for comparison. That is, by comparing the lamp area with a group of (N−1) lamp areas other than the lamp area, the similarity of the lamp area can be easily calculated. Further, in the calculation of the degree of similarity, image regions having different sizes (for example, a lamp region and a group) can be appropriately compared.

(Other variations)
In the above-described embodiment and each modification, each component included in the traffic signal recognition apparatus is configured by dedicated hardware or can be realized by executing a software program suitable for each component. Good. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software that realizes the traffic signal recognition apparatus in the embodiment and each modification causes the computer to execute each step included in the flowcharts shown in FIGS. 4, 6, 9, and 12.

As described above, the traffic signal recognizing device according to one or a plurality of aspects has been described based on the embodiment and each modification, but the present disclosure is not limited to this embodiment. As long as it does not deviate from the gist of the present disclosure, various modifications conceived by those skilled in the art are applied to the present embodiment or the modified examples, and forms constructed by combining the components in the embodiments and the modified examples are also disclosed in the present disclosure It may be included in the range.

For example, in the above embodiment and each modification, the area extraction unit 101 geometrically uses the position of the vehicle 10, the position and form of the traffic light, and the map information to geometrically determine the traffic light area Pb in the captured image Pa. To detect. However, the area extraction unit 101 may detect the traffic signal area Pb using another detection method instead of such a detection method. For example, the area extraction unit 101 may detect the traffic light area Pb by image recognition by machine learning, or may detect the traffic light area Pb by pattern matching.

Further, in the above embodiment and each modification, the traffic signal recognition device recognizes the lighting color of the traffic signal having three lamp parts having different colors. That is, the traffic light recognition device recognizes the light color when N = 3. However, the number (N) of the lamp units may be any number as long as it is three or more. Further, the colors of the N lamp parts may be colors other than blue, yellow, and red, and two or more of the N lamp parts may be the same color.

Moreover, the traffic signal recognition apparatus shown in FIGS. 1, 5, 7, and 10 may be configured by a processor and a memory. That is, the constituent elements other than the history holding unit 105 among the constituent elements of the traffic signal recognition apparatus are realized by the processor. The processor implements components other than the history holding unit 105 by executing a program stored in the memory. The memory may be configured as the history holding unit 105. That is, the memory may store a program for controlling the processor or may store history information.

The present disclosure can be used for a traffic signal recognition device that is mounted on, for example, an automatic traveling vehicle and recognizes a traffic signal on a traveling route of the vehicle.

DESCRIPTION OF SYMBOLS 10

Vehicle

100, 100a, 100b, 100c Traffic signal recognition apparatus 101 Area extraction part 102 Area division part 103

Similarity calculation part

104, 104c Light color recognition part 105 History holding part 106 Flicker processing part 107 Time series processing part 210 Camera 220 Map storage 230 Position detection system 240 Vehicle control unit

Claims

A processor;
With memory,
The processor uses the memory,
From the recognition target image acquired by the sensor, an area where a traffic light having N (N is an integer of 3 or more) lamps is displayed is extracted as a traffic light area.
From the traffic signal area, to calculate the similarity of a plurality of images in which different lamp parts are projected,
Traffic light recognition device.
The processor is
N lamp areas are generated as the plurality of images,
Calculating the similarity of each of the N lamp areas;
The traffic light recognition apparatus according to claim 1.
The processor is
Based on the similarity calculated for each of the N lamp part areas, the lamp part area is not similar to any other lamp part area among the N lamp part areas. Determine whether there is a dissimilar region,
The traffic light recognition apparatus according to claim 2.
The processor further includes:
If it is determined that the dissimilar region exists, a color associated with the position of the dissimilar region in the traffic light region in advance is recognized as a lamp color in the recognition target image.
The traffic light recognition apparatus according to claim 3.
The processor is
In the recognition of the light color,
With reference to the association information indicating the color associated with each position of the N lamp section areas in the traffic light area,
In the association information, a color associated with the position of the dissimilar region is recognized as the lamp color in the recognition target image.
The traffic light recognition apparatus according to claim 4.
The processor is
In calculating the similarity,
For each lamp area in the recognition target image, by selecting the largest correlation coefficient as the similarity among the correlation coefficients between the lamp area and each of at least two other lamp areas. Calculating the similarity of the lamp area,
The traffic signal recognizing device according to any one of claims 3 to 5.
The processor is
In determining whether the dissimilar region exists,
The traffic light recognition according to any one of claims 3 to 6, wherein it is determined that a lamp unit region having the smallest similarity among the similarities of the N lamp unit regions exists as the dissimilar region. apparatus.
The processor is
In determining whether the dissimilar region exists,
The lamp unit region having the smallest similarity is determined to be present as the dissimilar region when the smallest similarity among the similarities of the N lamp unit regions is equal to or less than a threshold value. 7. The traffic light recognition apparatus according to any one of 3 to 6.
The processor is
In the recognition of the light color,
When determining that the dissimilar region does not exist,
Refer to the history information indicating the light color recognized for images acquired in the past,
The traffic light recognition apparatus according to claim 4, wherein a lamp color indicated by the history information is recognized as a lamp color in the recognition target image.
The processor further includes:
With reference to history information indicating the lamp color recognized for each of a plurality of images acquired before the recognition target image,
Among the lamp colors recognized for the recognition target image and the lamp colors in each of the plurality of images indicated by the history information, the most lamp colors are specified as the lamp colors,
The traffic light recognition apparatus according to claim 4, wherein a lamp color recognized for the recognition target image is updated to the multiple lamp colors.
The processor is
In determining whether the dissimilar region exists,
For each of a plurality of images acquired before the recognition target image, for each position in the traffic light area in the image, refer to history information indicating the similarity of the lamp area at the position,
For each position in the traffic signal area, the average similarity of the lamp area at the position in the recognition target image and the plurality of images is calculated as an average similarity,
Among the N lamp areas in the recognition target image, there is a lamp area at the position corresponding to the smallest average similarity among the average similarities calculated for each position as the dissimilar area. The traffic signal recognizing device according to any one of claims 3 to 6.
In the case where the traffic light further has M (M is an integer of 1 or more) direction indicator lamps, which are lamp parts each indicating the traveling direction of the vehicle,
The processor is
In the generation of the N lamp areas,
By dividing the traffic light area, the N lamp part areas and M lamp part areas each displaying the direction indicator lamp part are generated,
Further, at least one of the M direction indicator lamp units that is lit based on the lamp color recognized for the recognition target image and the feature amount of each of the M lamp unit regions. The traffic signal recognizing device according to claim 4, wherein the traveling direction indicated by the two direction indicator lamp units is recognized.
In the case where the traffic light further has M (M is an integer of 1 or more) direction indicator lamps, which are lamp parts each indicating the traveling direction of the vehicle,
The processor is
In the generation of the N lamp areas,
By dividing the traffic light area, the N lamp part areas and M lamp part areas each displaying the direction indicator lamp part are generated,
Furthermore, the traveling direction indicated by at least one direction indicator lamp unit that is lit among the M direction indicator lamp units is recognized based on the feature amount of each of the M lamp unit regions. The traffic light recognition apparatus according to any one of 3 to 11.
The processor is
In calculating the similarity,
For each of the N lamp areas, the image of the lamp area is compared with an image of a group of at least one other lamp area other than the lamp area. Calculate similarity,
The traffic light recognition device according to any one of claims 3 to 13.
The processor is
For each of the N lamp areas, when comparing the image of the lamp area and the image of the group, for each block included in the lamp area, the block most similar to the image of the block is selected. By searching from the group, the similarity of the lamp area is calculated.
The traffic light recognition apparatus according to claim 14.
The processor is
For each of the N lamp areas, when comparing the image of the lamp area with the image of the group, for each block included in the group, the block most similar to the image of the block is selected. The traffic light recognition apparatus according to claim 14, wherein the similarity of the lamp section area is calculated by searching from the section area.
From the recognition target image acquired by the sensor, an area where a traffic light having N (N is an integer of 3 or more) lamps is displayed is extracted as a traffic light area.
From the traffic light area, N different lamp areas are displayed in which different lamp sections are projected,
Calculating the similarity of each of the N lamp areas;
Based on the similarity calculated for each of the N lamp part areas, the lamp part area is not similar to any other lamp part area among the N lamp part areas. Determine whether there is a dissimilar region,
Signal recognition method.
From the recognition target image acquired by the sensor, an area where a traffic light having N (N is an integer of 3 or more) lamps is displayed is extracted as a traffic light area.
From the traffic light area, N different lamp areas are displayed in which different lamp sections are projected,
Calculating the similarity of each of the N lamp areas;
Based on the similarity calculated for each of the N lamp part areas, the lamp part area is not similar to any other lamp part area among the N lamp part areas. Determine whether there is a dissimilar region,
A program that causes a computer to execute.