JP5407920B2 - Lighting color identification device and program - Google Patents

Description

  The present invention relates to a lighting color identification device and a program, and more particularly to a lighting color identification device and a program for identifying a lighting color of a signal lamp from a captured image.

  Conventionally, a traffic signal detection device capable of accurately detecting a traffic signal ahead of the host vehicle is known (Patent Document 1). In this traffic light detection device, the red light emission part of the traffic light at the intersection cannot be extracted based on the basic color information and the like, and when the vehicle stops at the intersection, it is received from the camera ECU until the host vehicle starts. Image recognition data is stored in the data recording unit in time series. Then, when the host vehicle starts, the position where the red region disappears is detected from each image recognition data from the predetermined time stored in the data recording unit until the start, and the disappeared red region is used as the red light emitting unit of the traffic light. I have identified.

JP 2009-129290 A

  Since red and yellow are originally the same color and are difficult to discriminate due to the influence of the brightness and directivity of the lamp and the brightness of the surroundings, yellow and red of the signal lamp are often mistakenly identified.

  However, the technique described in Patent Document 1 has a problem that the lighting color cannot be accurately identified because misrecognition of the lighting color of the signal lamp is not considered.

  The present invention has been made to solve the above problems, and an object of the present invention is to provide a lighting color identification device that can accurately identify the lighting color of a signal lamp represented by a captured image.

  In order to achieve the above object, a lighting color identification device according to the present invention is based on color information of each lighting color of a predetermined signal lamp from a captured image captured by an imaging device that captures the periphery of the host vehicle. The candidate area extracting means for repeatedly extracting the signal lamp candidate area and tracking the corresponding candidate area for the signal lamp and the change in the position of the tracked candidate area for the signal lamp are determined. When it is determined that the lighting color has been switched by the switching determination means and the determination means, the change in position and color of the candidate area of the signal lamp when the lighting color is switched is the signal lamp by the switching of the predetermined lighting color. A change rule determining means for determining whether or not the change is in accordance with a rule relating to a change in position and lighting color; and When it is determined that the change in the position and color of the candidate area is not a change according to the rule, a lighting color identification that identifies the lighting color when changed according to the rule as the lighting color of the candidate area of the signal light Means.

  The program according to the present invention repeats the signal lamp candidate area based on color information of each lighting color of a predetermined signal lamp from a captured image captured by an imaging apparatus that images the periphery of the host vehicle. Candidate area extracting means for extracting and tracking the corresponding candidate area of the signal lamp, switching determination means for determining switching of the lighting color from a change in the position of the tracked candidate area of the signal lamp, and lighting by the determination means When it is determined that the color has been switched, the position and color change of the candidate area of the signal lamp when the lighting color is switched are in accordance with the rules regarding the change of the position of the signal lamp and the lighting color due to the switching of the lighting color. The change rule determination means for determining whether or not the change has occurred, and the position of the candidate area of the signal lamp by the change rule determination means If the change in color is determined not to be a change in accordance with the rule, the change in the color according to the rule is made to function as a lighting color identification means for identifying the lighting color in the candidate area of the signal light It is a program for.

  According to the present invention, the candidate area of the signal lamp is determined based on the color information of each lighting color of the predetermined signal lamp from the captured image captured by the imaging apparatus that images the periphery of the host vehicle by the candidate area extraction unit. It extracts repeatedly and tracks the corresponding signal lamp candidate area.

  Then, the switching determination means determines the switching of the lighting color from the change in the position of the tracked candidate area of the signal light. When it is determined by the change rule determination means that the lighting color has been switched by the determination means, the position of the candidate area of the signal lamp and the color change when the lighting color is switched are the position of the signal lamp by the switching of the predetermined lighting color And whether or not the change is in accordance with the rules regarding the change in lighting color.

  And, when the change of the position and color of the candidate area of the signal light is determined not to change according to the rule by the change rule determination means by the lighting color identification means, the lighting color when changed according to the rule, It identifies as a lighting color of the candidate area of a signal lamp.

  As described above, when it is determined that the lighting color has been switched for the candidate area of the signal lamp extracted from the captured image, the lighting color of the signal lamp is identified according to the rules regarding the position of the signal lamp and the change in the lighting color due to the switching of the lighting color. By doing so, the lighting color of the signal lamp represented by the captured image can be accurately identified.

  In the lighting color identification device according to the present invention, when the change rule determining means determines that the position and color change of the signal lamp candidate area is not a change in accordance with the rule, the color information of the signal lamp candidate area is changed to the rule. Storage means for storing in the color information storage means as the color information of the lighting color when changing along the line, and the candidate area extraction means is the color information of the lighting color stored in the color information storage means from the captured image Can be used to repeatedly extract signal lamp candidate areas and track corresponding signal lamp candidate areas. This makes it possible to accurately track the signal lamp candidate area.

  In the present invention, the color information of the signal lamp candidate area stored in the color information storage means is erased from the color information storage means when the signal lamp candidate area corresponding to the signal lamp candidate area is no longer extracted from the captured image. Can be done.

  The lighting color identification device according to the present invention includes a color information storage unit that stores color information representing a histogram indicating the frequency of each color component for each lighting color, and a color information storage unit that stores the color information stored in the color information storage unit. The frequency is summed up for each lighting color, and when the total value of the frequency of any color component is equal to or greater than a predetermined value for the histogram of any lighting color, the frequency of the color component of the lighting color histogram Color information updating means for updating the frequency of the color component indicated by the histogram of the lighting color stored in the color information storage means based on the total value of the color information storage means, and the candidate area extracting means Based on the color information of each lighting color stored in the information storage means, it is possible to repeatedly extract signal lamp candidate areas and track the corresponding signal lamp candidate areas. This makes it possible to accurately extract signal lamp candidate regions from the captured image.

  The switching determination means according to the present invention including the color information storage means described above includes the change in the position of the tracked signal lamp candidate area, the color information of the lighting color stored in the color information storage means, and the tracked signal lamp candidate. Based on the distance from the color information of the area, it is possible to determine the switching of the lighting color.

  In the lighting color identification device according to the present invention, when the change rule determining unit determines that the change in the position and color of the candidate area of the signal lamp is not a change according to the rule, the lighting color when the change occurs according to the rule. A distance determination unit that calculates a distance between the color information of the signal light and the color information of the candidate area of the signal lamp, and determines that the candidate area of the signal lamp does not represent the signal lamp if the calculated distance is equal to or greater than a predetermined value. The lighting color identifying means further includes a case where the change rule determining means determines that the change in the position and color of the candidate area of the signal light is not a change according to the rule, and is calculated by the distance determining means. When the distance is less than the predetermined value, the lighting color when changing according to the rule can be identified as the lighting color of the candidate area of the signal lamp.

  The lighting color identifying unit according to the present invention can identify the previously identified lighting color as the lighting color of the candidate area of the signal lamp when the determining unit determines that the lighting color has not been switched.

  The candidate area extracting means according to the present invention extracts the signal lamp candidate area from the captured image and detects the lighting color of the signal lamp candidate area, and the lighting color identifying means determines the signal lamp candidate area by the change rule determining means. When it is determined that the change in position and color is a change according to the rule, the lighting color detected by the candidate area extraction unit can be identified as the lighting color of the candidate area of the signal light.

  As described above, according to the lighting color identification device of the present invention, when it is determined that the lighting color has been switched for the candidate area of the signal lamp extracted from the captured image, the position and lighting of the signal lamp due to the switching of the lighting color. By identifying the lighting color of the signal lamp in accordance with the rule regarding the color change, an effect that the lighting color of the signal lamp represented by the captured image can be accurately identified is obtained.

It is a block diagram which shows the structure of the vehicle signal lamp identification device which concerns on the 1st Embodiment of this invention. It is a graph which shows distribution of the a ^* value and b ^* value in a blue signal, a yellow signal, and a red signal. It is a graph showing the color distribution data which is a histogram of a ^* value and b ^* value. (A) It is an image figure which shows switching of a horizontal signal lamp, (B) It is an image figure which shows switching of a vertical signal lamp. It is a flowchart which shows the content of the lighting color identification process routine in the vehicle-mounted signal light identification device which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the structure of the vehicle signal lamp identification device which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows the content of the signal information update process routine in the vehicle signal lamp identification device which concerns on the 2nd Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the present embodiment, a case where the present invention is applied to an in-vehicle signal lamp identification device that is mounted on a vehicle and that identifies and displays a lighting color of a signal lamp from a captured front image will be described as an example.

  As shown in FIG. 1, an in-vehicle signal light identification device 10 according to a first embodiment includes an imaging device 12 including a CCD camera that captures an image ahead of the host vehicle, and a display device that displays a car navigation screen and the like. 14 and a computer 18 that controls the display of the display device 14 while identifying the lighting color of the front signal lamp based on the captured front image.

  The imaging device 12 monitors the front of the vehicle and captures a front image. In order to identify the lighting color of the signal lamp, a color camera is used as the imaging device 12. The signal lamp is a self-luminous object and has high luminance in the image. If the dynamic range of the camera is insufficient, pixel values are saturated and correct color information cannot be acquired (color information is lost). Therefore, it is preferable to use a camera equipped with an image sensor with a high dynamic range. If the dynamic range cannot be ensured, the lens aperture and exposure time may be adjusted appropriately so that color information is not lost in the signal lamp imaging region.

  The display device 14 includes a head-up display and a navigation screen.

  The computer 18 includes a CPU, a ROM that stores a program for a lighting color identification processing routine that will be described later, a RAM that stores data, and a bus that connects these. If the computer 18 is described in terms of functional blocks divided for each function realization means determined based on hardware and software, the computer 18 inputs a front image output from the imaging device 12 as shown in FIG. Based on the color distribution data of the signal information storage unit 22 from the image input unit 20, the signal information storage unit 22 that stores the color distribution data of each lighting color of the signal lamp in advance, and the front image that is the output of the image input unit 20. The candidate area extracting unit 24 for extracting the signal lamp candidate areas, the tracking section 26 for tracking the signal lamp candidate areas extracted in time series in association with each other, and switching the lighting color for the tracked signal lamp candidate areas. Based on the comparison result between the lighting determination rule and the switching determination unit 28 that compares with the lighting rule of the change in the candidate area of the signal lamp due to the switching, The lighting color identifying unit 30 for identifying the lighting color of the candidate region and displaying the identification result on the display device 14 and the color distribution storage unit 32 for storing the color distribution data of the candidate region of the signal light are provided. . The candidate area extraction unit 24 and the tracking unit 26 are examples of candidate area extraction means, and the switching determination unit 28 is an example of switching determination means and change rule determination means.

  The image input unit 20 includes, for example, an A / D converter and an image memory that stores image data for one screen.

  The color distribution data of the signal lamp stored in the signal information storage unit 22 is learned using an image captured in advance by the same imaging device 12. For example, a plurality of sample images in which red, yellow, and blue signal lights are lit are prepared, each pixel is converted into an appropriate color space, and color distribution data of each signal color is obtained. The color distribution data may be expressed by the center position and variance, or may be expressed as a histogram in which the axis of the color space is divided into several bins (color components) and the number of pixels voted for each bin is aggregated. Also good.

The color space to be used is not limited, and L ^* a ^* b ^* , HSV, or the like may be used, or RGB values obtained from a color camera may be used as they are. FIG. 2 shows an example of the color distribution of the red signal, the yellow signal, and the blue signal. This is a two-dimensional plane of a ^* -b ^* by extracting pixels included in red, yellow, and blue signal lights from daytime sample images and converting the values of each pixel to L ^* a ^* b ^* color space. It is the figure projected on the top. The blue signal is clearly separated from the other two lighting colors, but it can be seen that the yellow signal and the red signal have overlapping portions and are difficult to separate. Of these, FIG. 3 shows data obtained by converting the color distribution of the red signal into a histogram expression. The range of −60 to 80 of a ^* is divided into 14 bins, the range of −60 to 100 of b ^* is divided into 16 bins, and the number of pixels voted for the color area corresponding to each bin is expressed as a frequency. .

  The luminance distribution is also investigated in advance from a sample image obtained by imaging a signal lamp. However, the luminance may be calculated using a color space different from the color space used for learning the color distribution data.

  The color distribution data and the luminance distribution data are stored in advance in the signal information storage unit 22 for each lighting color.

  The candidate area extraction unit 24 extracts color information and luminance information for each pixel (or several small areas) of the acquired forward image in the same procedure as that for learning, and the extracted color information and luminance for each pixel. The information is compared with the color distribution data and luminance distribution data of each lighting color stored in the signal information storage unit 22, and it is determined whether or not it is a candidate for a signal lamp.

  The candidate area extraction unit 24 extracts an area where pixels determined to be signal lamp candidates are concentrated as a signal lamp candidate, and determines whether or not the shape of the extracted signal lamp candidate is circular. There are various methods for calculating the degree of circularity. Here, the circularity is calculated by the following equation (1). For example, a value that falls between 0.8 and 1.2 is determined to be circular.

  Where S is the area of the candidate region, a is the width of the candidate region, and b is the height of the candidate region.

  Further, the candidate area extraction unit 24 extracts the signal lamp candidates determined to be circular in shape as signal lamp candidate areas. In addition, the candidate area extraction unit 24 determines the candidate area of the signal lamp depending on which color distribution data and luminance distribution data of the lighting color the color information and luminance information of each pixel in the area extracted as the candidate area of the signal lamp. The lighting color of is detected.

  The candidate area extraction unit 24 repeatedly performs the extraction of the signal lamp candidate areas for each of the continuously obtained front images.

  The tracking unit 26 associates the candidate areas of the signal light extracted continuously according to the color, brightness, and shape of the candidate areas, and performs a tracking process on the candidate areas of the signal lights. At this time, by verifying the time continuity, erroneous extraction of signal lamp candidate regions is reduced. In addition, the collation range to be associated is dynamically set based on the size of the candidate area so that the candidate area of the signal lamp can be associated even if the lighting color of the signal lamp changes between the input front images. Set to.

  The switching determination unit 28 determines whether or not the lighting color has been switched for the signal lamp candidate area based on the position change of the candidate area being tracked. For example, when the position of the candidate area of the signal light being tracked changes discontinuously, it is determined that the lighting color has been switched. At this time, it may be determined whether or not the position of the candidate area of the signal light being tracked has changed discontinuously using the motion information of the host vehicle obtained by a sensor or the like. Further, it may be determined whether or not the lighting color has been switched in consideration of whether or not the color information of the candidate area of the signal light being tracked has changed.

  Further, when it is determined that the lighting color has been switched, the switching determination unit 28 determines whether or not the color change in the signal lamp candidate area is a color change that the signal lamp can take. That is, when a color change occurs, the switching determination unit 28 determines whether the color change in the signal lamp candidate area being tracked contradicts the signal light lighting rule.

  As shown in FIG. 4 (A) or FIG. 4 (B), there are differences between the vertical type and the horizontal type, but since the arrangement and lighting order of the lamps are determined in advance, the rules regarding the change in the position of the signal lamp to light (Right change → Change right → Change left → Change right or change up → Change up → Change down → Change up) and lighting color rules (red → yellow → blue → red ..) is predetermined as a lighting rule for signal lights. For example, after the blue signal, the lighting of the yellow signal is limited, and the lamp position moves to the right or up.

  Here, it is assumed that the candidate area extraction unit 24 detects that the yellow signal and the red signal are erroneous and detects that the signal is a red signal. In such a case, it is determined that the change is not in line with the lighting rule (there is a contradiction with the lighting rule). If the detection result in the candidate area extraction unit 24 is also a yellow signal, it is determined that the change is in accordance with the lighting rule (no conflict with the lighting rule).

  In addition, the lighting color identification unit 30 performs final color identification. If there is no color change in the signal lamp candidate area being tracked, it is determined that the lighting color has not been switched, and the lighting color identifying unit 30 determines the previously identified lighting color as the lighting color of the developing signal lamp. Continue to hold as an identification result.

When the switching determination unit 28 determines that the color change in the candidate area of the signal light is not a change in accordance with the lighting rule (the lighting rule and the detection result are inconsistent), the lighting color identifying unit 30 In order to verify the difference, the distance in the color space is calculated. For example, when the L ^* a ^* b ^* color space is used, the following equation (2) is used to determine the color information of the candidate area in the detected image and the color information predicted from the lighting rule (learning Data).

  Here, the subscript s indicates color information calculated from the current front image, and r indicates color information obtained from the color distribution data stored in the signal information storage unit 22.

  If the calculated value r is smaller than the threshold value, the two colors are close to each other, so it is determined that the lighting color is detected by mistake when extracting the candidate area of the signal light, and the lighting color estimated from the lighting rule (changed according to the lighting rule) The current lighting color) is identified as the lighting color of the current signal lamp candidate area. On the other hand, when the calculated value r is larger than the threshold value, it is determined that the color change is impossible as a signal light, and it is determined that the candidate area of the signal light does not represent the signal light, and detection / tracking is interrupted.

Here, the above threshold value is the a ^* -b ^* space shown in FIG. 2, and allows the distance between the red signal data and the yellow signal data, and the distance from the other color (especially blue) data. Is set in advance within a range that can be eliminated.

  In addition, when the lighting color identification unit 30 determines that there is an error in the detection result of the signal light color from the front image, the color distribution storage unit 32 temporarily stores the color distribution data of the current signal light candidate region. To store. The stored color distribution is used for the next extraction process by the candidate area extraction unit 24, and is also used as reference data for determining the switching timing to the next lighting color in the switching determination unit 28. Further, the color distribution data stored in the color distribution storage unit 32 is stored until the candidate area of the signal lamp being tracked is out of the frame and is not extracted from the front image. When the frame is out, the color distribution data is erased from the color distribution storage unit 32.

  Further, the lighting color identification unit 30 determines that the color change in the signal lamp candidate area is a change in accordance with the lighting rule (no contradiction between the lighting rule and the detection result) by the switching determination unit 28. The lighting color detected by the candidate area extraction unit 24 is identified as the current lighting color of the signal lamp.

  Further, the lighting color identification unit 30 causes the display device 14 to display the result of identifying the lighting color of the candidate area for the signal lamp.

  Next, the operation of the in-vehicle signal light identification device 10 according to the first embodiment will be described. When the front of the host vehicle is imaged by the imaging device 12 while the host vehicle equipped with the on-vehicle signal lamp identification device 10 is traveling on the road, the computer 18 turns on the lighting color identification processing routine shown in FIG. Is repeatedly executed.

  First, in step 100, a front image captured by the imaging device 12 is acquired, and in step 102, pixel color information and luminance information and a signal information storage unit 22 for each pixel of the front image acquired in step 100. Are compared with the color distribution data and luminance distribution data of each lighting color stored in the above, to extract a signal lamp candidate area and detect a lighting color of the signal lamp candidate area. Here, when color distribution data of at least one lighting color is stored in the color distribution storage unit 32, instead of the color distribution data of the lighting color stored in the signal information storage unit 22, Using the color distribution data of the lighting color stored in the color distribution storage unit 32, signal lamp candidate regions are extracted.

  In step 104, the signal lamp candidate area is traced by associating it with the signal lamp candidate area extracted in step 102 and the previously extracted signal lamp candidate area. In the next step 106, it is determined whether or not the previously extracted signal lamp candidate area has been out of frame. If there is no signal lamp candidate area associated with the previously extracted signal lamp candidate area in step 104 and the signal lamp candidate area cannot be traced, it is determined that the frame has been out, and in step 108 the color distribution is determined. All the color distribution data stored in the storage unit 32 are deleted.

  In step 110, whether the lighting color has been switched by comparing the position and color information of the candidate area of the signal light extracted last time with the position and color information of the candidate area of the signal light extracted in step 102. Determine. If it is determined that the lighting color has not been switched, the lighting color identification result identified in the previous process is maintained in step 112 to obtain the current lighting color identification result, and the process proceeds to step 124.

  On the other hand, if it is determined in step 110 that the lighting color has been switched, in step 113, the position and color change between the previously extracted signal lamp candidate area and the current signal lamp candidate area are inconsistent with the lighting rule. It is determined whether or not. If the change in the position and color of the candidate area of the signal lamp is a change in accordance with the lighting rule and there is no contradiction, the lighting color detected in step 102 is changed to the lighting color of the current signal lamp in step 114. And the process proceeds to step 124.

On the other hand, if it is determined in step 113 that the change in the position and color of the candidate area of the signal light is not a change in accordance with the lighting rule but is inconsistent, in step 116, the above equation (2) is applied. The average color information (a ^* value, b ^* value) obtained from the color distribution data of the signal lamp candidate area, and the lighting color when changing according to the lighting rule stored in the signal information storage unit 22 The distance in the L ^* A ^* B ^* color space with the average color information (a ^* value, b ^* value) obtained from the color distribution data is calculated, and the calculated distance is equal to or less than a predetermined threshold value. It is determined whether or not. If the calculated distance is greater than the threshold value, it is determined that the color change is not possible as a signal light, and it is determined in step 118 that the candidate area of the signal light is not a signal light, and the process proceeds to step 124. In this case, the tracking process for the candidate area of the signal light is stopped.

  On the other hand, if the calculated distance in step 116 is equal to or smaller than the threshold value, it is determined that there is an error in the detection result of the lighting color, and in step 120, the color distribution data of the candidate area of the signal light is converted into the color distribution. The data is temporarily stored in the storage unit 32. Then, in step 122, the lighting color when changed in accordance with the lighting rule is identified as the current lighting color of the signal lamp, and the process proceeds to step 124.

  In step 124, the identification result of step 112, 114 or 122 is displayed on the display device 14, and the lighting color identification processing routine is terminated.

  By repeatedly executing the above-described lighting color identification processing routine, the signal light represented in the front image is tracked, and the identification result of the lighting color is sequentially displayed.

  As described above, according to the on-vehicle signal lamp identification device according to the first embodiment, when it is determined that the lighting color has been switched for the candidate area of the signal lamp extracted from the captured front image, the lighting is performed. By identifying the lighting color of the signal lamp according to the lighting rules regarding the position of the signal lamp and the change of the lighting color due to the color switching, the lighting color of the signal lamp represented by the front image can be accurately identified. In particular, misrecognition of a red signal and a yellow signal can be reduced.

  The red and yellow signal lights are distributed at close positions in the color space. When the color distributions of the red signal and the yellow signal are examined from an image actually taken with a color camera, the color distributions of the two overlap. In addition to the colors that are close to each other, it is considered that there are individual differences in the brightness of the lamp due to the influence of lighting conditions, dirt, and deterioration over time. For this reason, when identifying the color of a signal light based on color information, yellow and red may be mistaken. However, since the order of the color of the lamps to be turned on is determined, in this embodiment, the detection result of whether the current lighting color is red or yellow is determined by determining the switching of the signal lights. It is possible to correct.

  If the color is determined to be red even though it is yellow, the color distribution is temporarily stored as yellow. When the time change of the color distribution is small, it is recognized as a “yellow signal”. It is possible to recognize that the lighting color continues together with the determination that the signal has not been switched from the tracking result. Thus, the robustness in identifying the lighting color can be further improved.

  In addition, the signal light can be stably recognized by an autonomous sensor such as an imaging device without using a map.

  In the above embodiment, the data format stored in the color distribution storage unit 32 is not limited, but the data format of the signal information storage unit 22 is used for data integration in the second embodiment to be described later. It is good to keep together.

  Further, the above equation (2) may be extended so as to calculate the distance including the luminance information as shown in the following equation (3).

  In addition, the switching determination unit 28 and the lighting color identification unit 30 have been described separately. However, since there are portions where the processing contents are common, the switching determination unit 28 and the lighting color identification unit 30 are configured to be integrated. Is also possible. For example, the switching determination unit 28 may be configured to use the above equation (2) when considering changes in color information.

  Further, the case where the color distribution data is stored in the color distribution storage unit 32 when it is determined that the lighting color has been switched has been described. However, the present invention is not limited to this, and it is determined that the lighting color has been switched next. Even if the color distribution data in the signal lamp candidate region is different from the color distribution data in the signal information storage unit 22, the color distribution data stored in the color distribution storage unit 32 may be updated.

  Next, a second embodiment will be described. In addition, about the part which becomes the structure similar to 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  In the second embodiment, the color distribution data temporarily stored is accumulated and reflected in the color distribution data of each lighting color stored in the signal information storage unit. And mainly different.

  As shown in FIG. 6, the computer 218 of the in-vehicle signal light identification device 210 according to the second embodiment includes an image input unit 20, a signal information storage unit 22, a candidate area extraction unit 24, a tracking unit 26, and the like. The switching determination unit 28, the lighting color identification unit 30, the color distribution storage unit 32, the color distribution storage unit 234 that stores color distribution data, and the color distribution data stored in the color distribution storage unit 32 are converted into color distributions. And a signal information updating unit 236 that updates the color distribution data of each lighting color of the signal lamp stored in the signal information storage unit 22 based on the accumulated color distribution data. .

  After the color distribution data is stored in the color distribution storage unit 32, when the signal lamp having the color distribution data is switched to the next lighting color or when it is out of the frame and is not extracted from the front image, the color distribution The data is output from the color distribution storage unit 32 to the signal information update unit 236.

  The signal information update unit 236 converts the color distribution data output from the color distribution storage unit 32 into a prescribed format, and sequentially adds it to the color distribution storage unit 234 for each lighting color. The data format of the color distribution may be arbitrary, but here it is assumed that the data is converted into the same format as the signal information storage unit 22 (histogram format shown in FIG. 3). The color distribution data for the same lighting color is added for each bin, and the color distribution data is stored in the color distribution storage unit 234 so that the color distribution data of the same lighting color input in the past and the current input data are integrated. Accumulate. Here, it is desirable to perform normalization processing so that the overall frequency becomes 1 when integrating the data.

  After integration, if the frequency of at least one bin in the histogram represented by the color distribution data for a certain lighting color is equal to or higher than a specified value, the signal color corresponding to that bin is detected with a high probability in that lighting color. The frequency of the corresponding bin in the histogram represented by the color distribution data for the lighting color in the signal information storage unit 22 is added and updated.

  Here, the addition rate w is determined according to the following equation (4), and the frequency is added according to the addition rate w.

  Here, n is the number of data accumulated in the bin to be added, f is a total value of frequencies represented by the data accumulated in the bin to be added, and r is stored in the signal information storage unit 22. The distance between the color distribution data of the lighting color and the bin to be added (distance to the nearest bin among the bins having a frequency in the color distribution data). In the above equation (4), the value of w increases as the number of data n increases, and the value of w increases as the total frequency f increases, and the distance r to the color distribution data increases. The addition rate w is determined so that the value of w becomes smaller.

  Next, the operation of the in-vehicle signal light identification device 210 according to the second embodiment will be described. When the host vehicle equipped with the in-vehicle signal light identification device 210 is traveling on a road, when the front of the host vehicle is imaged by the imaging device 12, the computer 218 is similar to the first embodiment. The lighting color identification processing routine is repeatedly executed. In step 108, all the color distribution data stored in the color distribution storage unit 32 are deleted, and all the color distribution data to be deleted are output from the color distribution storage unit 32.

  The computer 218 executes a signal information update processing routine shown in FIG.

  First, in step 250, it is determined whether or not the color distribution data temporarily stored is output from the color distribution storage unit 32. If the color distribution data is not output from the color distribution storage unit 32, the process returns to step 250. In step 108, the color distribution data stored in the color distribution storage unit 32 is deleted, and When the color distribution data is output from the color distribution storage unit 32, the color distribution data is acquired and the process proceeds to step 252.

  In step 252, the data format conversion is performed on the color distribution data output from the color distribution storage unit 32 in step 250 so that the color distribution data in the signal information storage unit 22 has the data format. In step 254, the color distribution data whose data format is converted in step 252 is integrated into the color distribution data corresponding to the lighting color identified in the color distribution data stored in the color distribution storage unit 234. . That is, the frequency of each bin of the color distribution data converted in step 252 to the color distribution data corresponding to the lighting color identified in the color distribution data stored in the color distribution storage unit 234, Addition is performed for each bin and normalization processing is performed.

  In step 256, it is determined whether or not there is a bin having a frequency equal to or higher than a predetermined value in the color distribution data corresponding to the lighting color accumulated in step 256. If any bin has a frequency less than the predetermined value, the process returns to step 250. On the other hand, if the frequency of at least one bin is equal to or greater than a predetermined value in step 256, the addition rate w is set for the bin whose frequency is equal to or greater than the predetermined value in step 258 according to the above equation (4). calculate.

  Then, in step 260, using the addition rate w calculated in step 258, the total color frequency value in the bin of the accumulated color distribution data is stored in the signal information storage unit 22. The color distribution data of the lighting color stored in the signal information storage unit 22 is updated by adding to the bin of the color distribution data.

  In the next step 262, in the color distribution data of the lighting color stored in the color distribution storage unit 234, the bin frequency in which the color distribution data is updated is reset, and the process returns to step 250.

  As described above, according to the in-vehicle signal light identification device according to the second embodiment, the color distribution data in the signal information storage unit 22 is updated according to the actual color distribution of the signal light obtained from the front image. Therefore, the robustness in the signal lamp extraction process can be improved.

  In the above embodiment, the timing of outputting the color distribution data from the color distribution storage unit 32 to the signal information update unit 236 is determined when the signal light having the target color distribution is switched to the next lighting color or out of frame. However, the present invention is not limited to this, and color distribution data may be output from the color distribution storage unit 32 to the signal information update unit 236 for each frame. Further, the number of data stored in the color distribution storage unit 234 (number of observations) may be limited, or may be stored as needed without limitation.

  Further, every time data is accumulated a predetermined number of times, the color distribution accumulation unit 234 may reset bins whose total frequency is lower than a predetermined value.

  In the first embodiment and the second embodiment described above, the case where the identification result of the lighting color of the signal lamp is displayed on the display device is described as an example. However, the present invention is not limited to this. The identification result of the lighting color may be output to a driving support system that performs driving intervention control or the like.

  Further, the candidate area extraction unit 24 may further extract a candidate area of the signal light in consideration of a three-dimensional position such as the height of the candidate area.

  In addition, the candidate area extracting unit 24 also extracts the entire area of the traffic signal based on the shape (pattern), and the switching determination unit 28 changes the relative position between the entire area of the traffic signal and the lit lamp (high luminance area). It may be determined whether the lighting color is switched. Further, the error of the recognized lighting color may be determined by considering the relative position of the lighting lamp with respect to the entire area of the traffic light.

  It is also possible to provide the program of the present invention by storing it in a storage medium.

DESCRIPTION OF SYMBOLS 10,210 Vehicle signal light identification device 12 Imaging device 18, 218 Computer 22 Signal information storage unit 24 Candidate area extraction unit 26 Tracking unit 28 Switching determination unit 30 Lighting color identification unit 32 Color distribution storage unit 234 Color distribution storage unit 236 Signal information Update department

Claims (9)

Based on color information of each lighting color of a predetermined signal lamp from a captured image captured by an imaging device that captures the periphery of the host vehicle, the signal lamp candidate area is repeatedly extracted and the corresponding signal lamp candidate Candidate area extraction means for tracking the area;
From a change in the position of the tracked candidate area of the signal light, switching determination means for determining switching of the lighting color,
When it is determined that the lighting color has been switched by the determining means, the position and color change of the candidate area of the signal lamp when the lighting color is switched are the position of the signal lamp and the lighting color by the predetermined lighting color switching. Change rule determining means for determining whether or not the change is in accordance with a change rule;
When the change rule determining means determines that the change in the position and color of the candidate area of the signal light is not a change in accordance with the rule, the lighting color when the change in the rule is changed is determined as the candidate for the signal light. A lighting color identification means for identifying the lighting color of the area;
Lighting color identification device including When the change rule determining means determines that the change in the position and color of the signal lamp candidate area is not a change in accordance with the rule, the color information of the signal lamp candidate area has been changed in accordance with the rule. Storage control means for storing in the color information storage means as the color information of the lighting color at the time,
The candidate area extraction unit repeatedly extracts the signal lamp candidate area from the captured image using the color information of the lighting color stored in the color information storage unit, and tracks the corresponding signal lamp candidate area. The lighting color identification device according to claim 1.   The color information of the signal lamp candidate area stored in the color information storage means is erased from the color information storage means when the signal lamp candidate area corresponding to the signal lamp candidate area is no longer extracted from the captured image. The lighting color identification device according to claim 2. Color information storage means for storing the color information representing a histogram indicating the frequency of each color component for each lighting color;
The frequency of each color component indicated by the lighting color histogram stored in the color information storage means is totaled for each lighting color, and for any lighting color histogram, the total value of the frequencies of any of the color components is When the predetermined value or more is reached, the frequency of the color component indicated by the lighting color histogram stored in the color information storage means is updated based on the total frequency of the color components of the lighting color histogram. Color information updating means for performing,
The candidate area extracting unit repeatedly extracts a signal lamp candidate area from the captured image based on the color information of each lighting color stored in the color information storage unit, and tracks the corresponding signal lamp candidate area. The lighting color identification device according to claim 2 or 3.   The switching determination means includes a change in position of the tracked candidate area of the signal light, and color information of the lighting color stored in the color information storage means and color information of the tracked candidate area of the signal light. The lighting color identification device according to any one of claims 2 to 4, wherein switching of the lighting color is determined based on the distance. When the change rule determining means determines that the position and color change of the candidate area of the signal light is not a change according to the rule, the color information of the lighting color when changed according to the rule; and A distance determination unit that calculates a distance from the color information of the candidate area of the signal lamp, and determines that the candidate area of the signal lamp does not represent the signal lamp if the calculated distance is equal to or greater than a predetermined value;
The lighting color identifying means is a case where the change rule determining means determines that the change in the position and color of the candidate area of the signal light is not a change according to the rule, and the distance determining means When the calculated distance is less than the predetermined value, a lighting color when changing according to the rule is identified as a lighting color of the candidate area of the signal lamp. The lighting color identification device according to item.   The lighting color identifying unit identifies the previously identified lighting color as the lighting color of the candidate area of the signal light when the determining unit determines that the lighting color has not been switched. The lighting color identification device according to any one of the preceding claims. The candidate area extracting means extracts the signal lamp candidate area from the captured image and detects the lighting color of the signal lamp candidate area,
The lighting color identifying means is detected by the candidate area extracting means when the change rule determining means determines that the change in the position and color of the candidate area of the signal light is a change according to the rule. The lighting color identification device according to any one of claims 1 to 7, wherein a lighting color is identified as a lighting color of a candidate area of the signal lamp. Computer
Based on color information of each lighting color of a predetermined signal lamp from a captured image captured by an imaging device that captures the periphery of the host vehicle, the signal lamp candidate area is repeatedly extracted and the corresponding signal lamp candidate Candidate area extraction means for tracking the area;
From a change in the position of the tracked candidate area of the signal light, a switching determination means for determining switching of the lighting color,
When it is determined that the lighting color has been switched by the determining means, the position and color change of the candidate area of the signal lamp when the lighting color is switched are the position of the signal lamp and the lighting color by the predetermined lighting color switching. A change rule determining unit that determines whether or not the change is in accordance with a change rule; and the change rule determining unit determines that the change in position and color of the candidate area of the signal light is not a change in accordance with the rule. If so, a program for causing a lighting color when changing according to the rule to function as a lighting color identification means for identifying the lighting color of the candidate area of the signal light.

Images