JP2012173879A - Traffic signal detection apparatus and program therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately detect a traffic light while preventing a failure of detection, even if an inexpensive color camera having less dynamic range and less sensitivity is used.SOLUTION: A signal light candidate extraction section 26 extracts candidate areas of a signal light from a frame image captured by an imaging apparatus 12 based on color distribution and shapes. A signal light likelihood calculation section 28 calculates a likelihood indicating a signal light likelihood of each of the candidate areas of the signal light. A traffic light candidate extraction section 32 extracts candidate areas of a traffic light based on a shape feature. A traffic light tracking section 34 predicts a detection position in a present frame using a tracking model based on a detection result up to a previous frame. If candidate areas of the traffic light corresponding to the predicted position exist, the signal light is tracked to be detected by using a less-likelihood area in the candidate areas of the signal light corresponding to the candidate areas of the traffic light. If no candidate area of the traffic light corresponding to the predicted position exists, the signal light is tracked to be detected by using a high-likelihood area in the candidate areas of the signal light corresponding to the predicted position.

Description

  The present invention relates to a traffic signal detection apparatus and program, and more particularly, to a traffic signal detection apparatus and program for detecting a traffic signal from a captured image.

  Conventionally, in order to extract a high-luminance first object including a light emitter of a traffic light by excluding objects such as signboards with low luminance, the first extraction means obtains luminance from an image in front of the vehicle captured by a television camera. A first object having a threshold value of 1 or more is extracted, a re-extraction area setting unit sets a re-extraction area having a relatively small predetermined size around the first object, and a second extraction unit has a low luminance such as a signboard. A traffic light recognition device that determines whether or not a second object is a light emitter of a traffic light by extracting a second object having a luminance of a second threshold value that is less than the first threshold value in a re-extraction area where no object exists. It has been proposed (see, for example, Patent Document 1).

  In addition, the circular ground height calculated by the height calculation unit is higher than a predetermined reference height for the installation of traffic lights, and the distance from the host vehicle to the circular shape calculated by the distance calculation unit is the intersection distance. When the distance calculated by the calculation unit is greater than the distance obtained by adding 1/2 of the width of the intersection road to the distance from the own vehicle to the intersection, the traffic light in the red light state is detected in front of the traveling direction of the own vehicle. There has been proposed a traffic light recognition device that determines that a signal has been received (see, for example, Patent Document 2).

  In addition, a front-facing camera, a vehicle speed sensor, a video RAM to which image signals captured by the camera are input, and image data from the video RAM are read to extract a high-luminance area, and the size and color of this area Based on the window, detect the edge in the window and compare with the pre-stored reference traffic signal edge, and recognize the traffic signal as long as the edge detected in the window matches the reference traffic signal edge A signal recognition device has been proposed (see, for example, Patent Document 3).

JP 2009-43067 A JP 2007-257303 A JP-A 63-78299

  However, on general roads, there are many bulb-type traffic signals that are not picked up due to high brightness due to dirt on the lamp surface of the traffic light, a decrease in light emission intensity due to deterioration over time, and inclination of the lamp surface. In addition, if it is a self-luminous signboard, night window light, street light, etc., as shown in FIGS. 10A and 10B, a signboard made of a material with high reflectivity even during the daytime, a signal light The luminance value is similar to. For this reason, the technique described in Patent Document 1 has a problem that even if the threshold value is lowered, a signal lamp, a signboard, and the like may be erroneously detected.

  In addition, the technology described in Patent Document 2 has a problem that although there are exceptions depending on surrounding structures, the installation height of the traffic light is stipulated by laws and regulations, and there are many cases where it cannot be applied. . In addition, when a traffic signal ahead of about 100 m is set as a detection target, it is often difficult to correctly estimate the height of the traffic signal based on a vehicle pitch fluctuation or a road surface gradient. In addition, in the case where the detection target is a small signal light far away, it is difficult to calculate an accurate distance, and as shown in FIG. It may be detected as an existing red signal.

  In addition, in the technique described in Patent Document 3, when an inexpensive color camera is used, it is difficult to simultaneously capture the color information of the signal lamp and the edge information of the entire traffic light because the sensitivity and dynamic range are low. There is a problem.

  The present invention has been made to solve the above-described problems. Even when an inexpensive color camera having a low dynamic range or high sensitivity is used, non-detection can be prevented and a traffic signal can be accurately detected. An object of the present invention is to provide a traffic light detection device and a program that can be used.

  In order to achieve the above object, the traffic light detection device according to the present invention is configured such that color information in a frame image captured for each frame by an imaging unit that captures the periphery of the host vehicle has at least one lighting color of a predetermined signal lamp. A signal lamp candidate extracting means for extracting a circular or elliptical area indicating a signal lamp candidate area; and a likelihood calculating means for calculating a likelihood indicating the signal lamp likelihood of the signal lamp candidate area extracted by the signal lamp candidate extracting means; A signal candidate extraction unit that extracts a region indicating a predetermined shape feature from the frame image as a signal candidate region, and a peripheral region that includes the signal candidate region extracted by the signal candidate extraction unit. The signal lamp candidate area having a likelihood equal to or greater than the first threshold, and the area extracted other than the peripheral area including the signal candidate area Degree is based on the first candidate region of a second threshold value or more signal lights is higher than the threshold value, is configured to include a detecting means for detecting the position and color of the traffic signal, the.

  According to the traffic light detection apparatus of the present invention, the imaging means images the periphery of the host vehicle for each frame. The signal lamp candidate extracting means extracts a circular or elliptical area in which the color information in the frame image captured by the imaging means indicates at least one lighting color of a predetermined signal lamp as a signal lamp candidate area, and likelihood calculating means Calculates the likelihood indicating the signal lamp likelihood of the candidate area of the signal lamp extracted by the signal lamp candidate extraction means. The traffic signal candidate extracting means extracts a region indicating a predetermined shape feature from the frame image as a traffic signal candidate region.

  And the detection means is a signal lamp candidate area whose likelihood extracted in the peripheral area including the traffic signal candidate area extracted by the traffic signal candidate extraction means, and a peripheral area other than the peripheral area including the traffic signal candidate area. The position and color of the traffic light are detected based on the signal lamp candidate area whose likelihood extracted in the area is equal to or higher than the second threshold higher than the first threshold.

  As described above, even if the signal lamp candidate area has a low likelihood of indicating the likelihood of a signal lamp being undetected, if it corresponds to the candidate area of the traffic light, the dynamic range is used for detection of the signal lamp. Even when an inexpensive color camera with low sensitivity is used, it is possible to prevent undetection and detect a traffic signal with high accuracy.

  Further, the likelihood calculating means is based on at least one of the degree of coincidence between at least one lighting color of the predetermined signal lamp and the color information of the candidate area of the signal lamp, and the ellipticity of the candidate lamp area. Thus, the likelihood can be calculated.

  In addition, the detection means includes a corresponding signal candidate region in the signal tracking region in the current frame image predicted from the signal tracking result in the previous frame image, and a periphery including the signal candidate region. The tracking result can be updated and the tracking can be continued in at least one of the cases where the signal lamp candidate region exists in the region or the peripheral region including the tracking region. Thereby, stable tracking can be performed until the tracking target deviates from the frame image.

  In addition, the traffic light detection apparatus of the present invention controls the imaging unit so that a captured image is captured with a first exposure amount at which color information is not lost and a second exposure amount brighter than the first exposure amount. The signal lamp candidate extracting means extracts the signal lamp candidate area from the captured image captured at the first exposure amount, and the signal candidate detection means When the surrounding environment of the host vehicle is brighter than a predetermined value, the candidate area of the traffic light is extracted from the captured image captured at the first exposure amount. When the surrounding environment of the host vehicle is darker than the predetermined value The candidate area of the traffic light can be extracted from the captured image captured with the second exposure amount. Thereby, both the color information of a signal lamp and the shape information of a traffic light can be detected appropriately.

  Further, the traffic light detection device of the present invention is based on the ellipticity of the high-luminance area of the captured image captured at the second exposure amount corresponding to the signal lamp candidate area extracted by the signal lamp candidate extraction unit. Verification means for verifying the extracted candidate area of the signal light can be included. Thereby, both the color information of a signal lamp and the shape information of a signal lamp can be detected appropriately.

  The traffic light detection program of the present invention is a circle in which color information in a frame image captured for each frame by an imaging unit that captures the periphery of the host vehicle indicates at least one lighting color of a predetermined signal lamp. Alternatively, a signal lamp candidate extracting means for extracting an elliptical area as a signal lamp candidate area, a likelihood calculating means for calculating a likelihood indicating the signal lamp likelihood of the signal lamp candidate area extracted by the signal lamp candidate extracting means, from the frame image, A traffic signal candidate extraction unit that extracts a region indicating a predetermined shape feature as a traffic signal candidate region, and the likelihood extracted in a peripheral region including the signal candidate region extracted by the traffic signal candidate extraction unit is first. Extracted in a region other than the peripheral region including the signal lamp candidate region that is equal to or greater than the threshold value and the signal device candidate region Degree is based on the first candidate region of a second threshold value or more signal lights is higher than the threshold value, a program for functioning as a detecting means for detecting the position and color of the traffic signal.

  The storage medium for storing the program of the present invention is not particularly limited, and may be a hard disk or a ROM. Further, it may be a CD-ROM, a DVD disk, a magneto-optical disk or an IC card. Furthermore, the program may be downloaded from a server or the like connected to the network.

  As described above, according to the traffic signal detection apparatus and program of the present invention, even if it is a signal lamp candidate area having a low likelihood of indicating the likelihood of a signal lamp being undetected, it can correspond to the candidate area of the traffic light. In this case, since it is used for detection of signal lights, even when using an inexpensive color camera that does not have a high dynamic range or sensitivity, it is possible to prevent undetection and detect a traffic signal with high accuracy. It is done.

It is a block diagram which shows the structure of the vehicle signal detector which concerns on 1st Embodiment. (A) It is a figure which shows an example of the dark lamp which may become undetected, and (b) a signal lamp with uneven brightness. It is a figure for demonstrating an example of the process of the production | generation of a tracking model, update, and a tracking end. It is a flowchart which shows the content of the signal lamp detection process routine in the vehicle signal detector which concerns on 1st Embodiment. It is a flowchart which shows the content of the candidate extraction process routine in the vehicle signal detector which concerns on 1st Embodiment. It is a flowchart which shows the content of the tracking process routine in the vehicle signal detector which concerns on 1st Embodiment. It is a block diagram which shows the structure of the vehicle signal detector which concerns on 2nd Embodiment. It is a figure for demonstrating the difference in the appearance of a high-intensity area | region in a bright image and a dark image. It is a flowchart which shows the content of the candidate extraction process routine in the vehicle-mounted signal detection apparatus which concerns on 2nd Embodiment. It is a figure for demonstrating the example of a misdetection.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the present embodiment, a case where the present invention is applied to an in-vehicle signal detection device that detects and tracks a signal from a frame image that is mounted on a vehicle and continuously captured will be described as an example.

  As shown in FIG. 1, an in-vehicle signal detector 10 according to a first embodiment includes an imaging device 12 including a CCD camera that captures an image ahead of the host vehicle, and a display device that displays a car navigation screen and the like. 18 and a computer 20 that detects the traffic light ahead and controls the display of the display device 18 based on the captured frame image.

  The imaging device 12 images the front of the host vehicle and generates an image signal of the image (not shown), and an A / D conversion unit (not shown) that A / D converts the image signal generated by the imaging unit. And an image memory (not shown) for temporarily storing the A / D converted image signal. The imaging device 12 continuously captures frame images for each frame. Further, as the imaging device 12, a color camera is used to identify the lighting color of the signal lamp.

  The display device 18 includes a head-up display and a navigation screen.

  The computer 20 is configured to include a CPU, a ROM for storing a signal detection routine to be described later and various information, a RAM for storing data, and a bus for connecting these. If the computer 20 is described in terms of functional blocks divided for each function realization means determined based on hardware and software, as shown in FIG. 1, the computer 20 uses a traffic light in a frame image output from the imaging device 12. A search range setting unit 22 that sets a search range to search, a signal color distribution storage unit 24 that stores color distribution data of each lighting color of the signal lamp in advance, and a color of the signal color distribution storage unit 24 from the search range of the frame image Based on the distribution data, a signal lamp candidate extraction unit 26 that extracts a signal lamp candidate area, a signal lamp likelihood calculation unit 28 that calculates a likelihood indicating the signal lamp likelihood of the extracted signal lamp candidate area, and a shape characteristic of the traffic light Based on the traffic signal shape storage unit 30 stored in advance and the frame image search range, based on the traffic signal shape characteristics of the traffic signal shape storage unit 30, A traffic signal candidate extraction unit 32 that extracts a candidate region of a traffic light, and a traffic signal tracking unit 34 that detects a traffic signal by tracking and updating a tracking model by associating the candidate region and the candidate region of a traffic light extracted in time series. And can be represented by a configuration including The traffic light tracking unit 34 is an example of the detection means of the present invention.

  The search range setting unit 22 acquires a frame image captured for each frame by the imaging device 12, estimates a region where a traffic signal exists on the frame image based on the mounting posture of the imaging device 12, and the region Is set as a search range for searching for traffic lights. Note that the vanishing point may be obtained by recognizing a lane mark or the like from the frame image, and the search range may be set based on the vanishing point. Further, when there is a surplus in calculation resources, the entire frame image may be set as the search range.

The color distribution data of the signal lamp stored in the signal color distribution storage unit 24 is learned using an image captured in advance by the same imaging device 12. For example, a plurality of sample images in which red, yellow, and blue signal lights are lit are prepared, each pixel is converted into an appropriate color space, and color distribution data of each signal color is obtained. The color distribution data may be expressed by the center position and variance, or by dividing the color space axis into several bins (color components) and expressing it as a histogram totaling the number of pixels voted for each bin. May be. The color space to be used is not limited, and L ^* a ^* b ^* or HSV may be used, or RGB values obtained from a color camera may be used as they are. The luminance distribution is also investigated in advance from a sample image obtained by imaging a signal lamp. Note that the luminance may be calculated using a color space different from the color space used for learning the color distribution data. The color distribution data and the luminance distribution data are stored in advance in the signal color distribution storage unit 24 for each lighting color.

  The signal light candidate extraction unit 26 extracts color information and luminance information for each pixel (or several small regions) within the search range set by the search range setting unit 22 in the same procedure as that for learning. The extracted color information and luminance information are compared with the color distribution data and luminance distribution data of each lighting color stored in the signal color distribution storage unit 24 to determine whether it is a candidate for a signal lamp.

  Further, the signal lamp candidate extraction unit 26 extracts an area where pixels determined to be signal lamp candidates are concentrated, and determines whether or not the shape of the extracted area is an ellipse. There are various methods for calculating the ellipticity. Here, the ellipticity E is calculated by the following equation (1). For example, an area where this value E is between 0.8 and 1.2 is calculated by the signal lamp. Extract as a candidate area.

          E = S / πab (1)

  Here, S indicates the area of the extracted region, a indicates the width of the extracted region, and b indicates the height of the extracted region.

  Further, the signal lamp candidate extraction unit 26 selects the lighting color of the candidate area of the signal lamp depending on which lighting color distribution data and luminance distribution data the color information and luminance information of each pixel in the candidate area of the signal lamp match. To do. Note that the present invention is not limited to the case of detecting signal lights of all three colors of red, yellow, and blue. At least one signal light color to be detected is determined and the color distribution data of the signal color is stored in the signal color distribution storage unit 24. May be stored, and the signal lamp candidate extraction unit 26 may extract only the signal lamp candidate area of the signal color.

  The signal lamp likelihood calculating unit 28 calculates a numerical value (likelihood) indicating the likelihood of a signal lamp for each of the signal lamp candidate areas extracted by the signal lamp candidate extracting unit 26. The likelihood is calculated based on information on the shape, color distribution, and luminance of the candidate area of the signal light.

For example, the likelihood P _E regarding the shape of the candidate area of the signal lamp is calculated by the following equation (2).

P _E = 1− | 1-E | = 1− | 1−S / πab | (2)

In addition, a likelihood indicating how much the color distribution of the candidate area of the signal lamp matches the learned color distribution is also calculated. Likelihood P _C relating to the color distribution, for example, when the color distribution data learning is represented as a histogram is calculated by the following equation (3).

  Expression (3) is an expression for calculating a similarity measure between histograms called a histogram intersection, and takes a value of 0 to 1. Here, H is a normalization histogram of candidate areas of signal lights, M is a normalization histogram of signal color c learned in advance, and I is the number of bins of the histogram. In this case, the likelihood is calculated by the expression (3) for the three types of signal colors, and the signal color indicating the highest likelihood is selected as the lighting color of the signal lamp candidate region. When the learned color distribution data is expressed by the center of gravity and the variance, the likelihood relating to the color distribution may be calculated using the Mahalanobis distance or the like.

When the likelihood P _B related to the luminance is also used, the likelihood P _B related to the luminance is calculated in the same procedure as the equation (3).

The final likelihood P _S is comprehensively calculated from the likelihood P _E related to the shape, the likelihood P _C related to the color distribution, and the likelihood P _B related to the luminance. For example, they may be integrated in the form of products of respective likelihoods as shown in equation (4), or may be integrated in the form of sums. Further, any one of them may be selected as a representative value as shown in equation (5).

P _S = P _E · P _C · P _B (4)
P _S = max {P _E , P _C , P _B } (5)

  The traffic signal shape features stored in the traffic signal shape storage unit 30 are learned using images captured in advance by the same imaging device 12. For example, a traffic signal shape storage unit 30 uses a discriminator constructed by extracting a HOG (Histogram-Of-Gradient) feature from an image and learning using a technique such as AdaBoost or SVM (Support vector machine) as a traffic signal shape feature. Is stored in advance.

  The traffic signal candidate extraction unit 32 extracts images in the window while scanning windows of various sizes within the search range set by the search range setting unit 22, and the same procedure as that at the time of learning from the extracted window image Then, the feature is extracted, and using the discriminator as the shape feature stored in the traffic signal shape storage unit 30, it is evaluated whether or not the window image is an image showing the traffic signal. Then, an area corresponding to the window image having a high evaluation value is extracted as a candidate area for the traffic signal.

  The traffic light tracking unit 34 generates and updates a tracking model based on the signal light candidate region extracted by the signal light candidate extraction unit 26 and the traffic signal candidate region extracted by the traffic light candidate extraction unit 32. The tracking model has the position, size, and color of the signal lamp in the frame image as state quantities. The position in the three-dimensional space may be used as the state quantity. The state quantity of the tracking model includes the position, size, and color of the signal lamp indicated by the candidate area of the signal lamp extracted from the current frame, and the position and size of the traffic light indicated by the candidate area of the traffic light (hereinafter referred to as “observed values”). Also, it is sequentially updated by using an observed value corresponding to the previous frame or a value estimated from this observed value.

More specifically, the detection position in the current frame is predicted based on the movement of the host vehicle between the previous frame and the current frame with respect to the tracking model generated up to one frame before. Next, signal candidate regions extracted around the predicted prediction position are detected as signal candidate regions corresponding to the tracking model. When a candidate area of a traffic light corresponding to the predicted position is detected, the signal lamp candidate area extracted in the peripheral area including the candidate area of the traffic signal, and the likelihood calculated by the signal lamp likelihood calculating unit 28 is used. A signal lamp candidate area having a degree equal to or greater than the threshold Pth _L is detected as a signal lamp candidate area corresponding to the tracking model. If no candidate area of the traffic light corresponding to the predicted position is detected, a candidate area of the signal lamp corresponding to the tracking model is selected from the candidate area of the signal lamp having the likelihood equal to or greater than the threshold Pth _{H in} the peripheral area of the predicted position. To detect. The threshold value Pth _H is set to a value higher than the threshold value Pth _L. By processing in this way, a signal lamp candidate area having a high likelihood is detected as a signal lamp even when the correspondence with the candidate area of the traffic signal is not achieved, and when the signal lamp is dark as shown in FIG. (A)) or a signal lamp candidate area having a low likelihood as in the case of a signal lamp that does not appear circular due to uneven brightness ((b) in the figure) If there is a correspondence with the candidate area of the traffic signal that can be matched with the tracking model, it can be picked up. Thereby, undetected can be prevented, maintaining the detection accuracy of a signal lamp.

  Note that the matching range is dynamically determined based on the state quantity of the tracking model and the lighting rules of the signal lamp so that the tracking model can be associated with the candidate area of the signal lamp even if the color of the signal lamp changes between frames. You should set it.

  Next, as shown in Table 1 below, the tracking model is updated in accordance with the detection status of the candidate area of the traffic light and the candidate area of the signal light around the predicted position.

  When a signal lamp candidate area corresponding to the tracking model is detected, the state quantity of the tracking model is updated using the observed value. When only the candidate area of the traffic light corresponding to the tracking model is detected, the position and size of the traffic light are estimated based on the color information of the tracking model and the position and size of the candidate area of the traffic light. For example, if the color information of the tracking model is “blue”, the candidate area of the traffic light is divided into three equal parts in the horizontal direction, and the center position of the left area is estimated as the center of gravity (position) of the traffic light. The vertical width is estimated as the diameter (size) of the signal lamp. Then, the tracking model is updated using the estimated position and size of the signal lamp. Color information is not updated. If neither the signal candidate area nor the signal lamp candidate area corresponding to the tracking model is detected, the position and size of the signal lamp are estimated based on the predicted position of the tracking model in the same manner as described above. Update. However, if neither the candidate area of the traffic light nor the candidate area of the signal lamp corresponding to the specified number of times or more is detected, the tracking model is deleted.

  When at least one of the signal lamp candidate area and the signal lamp candidate area is extracted by the signal lamp candidate extraction section 26 and the traffic signal candidate extraction section 32, but there is no corresponding tracking model, it is shown in Table 2 below. As described above, a new tracking model is generated based on the observation value of the current frame.

  If both the candidate area for traffic lights and the candidate area for signal lights are extracted and there is a corresponding relationship between them (if there is no contradiction in the relative positional relationship between the two), a new tracking model is created based on these observations. Is generated. The tracking model is given “determined” information indicating that it is a tracking model generated by extracting both the candidate area of the traffic light and the candidate area of the signal light. When only the signal lamp candidate area is extracted and the corresponding signal candidate area is not extracted, a new tracking model is generated based on the observation value of the signal lamp candidate area. In addition, information of “undecided” indicating that the tracking model is generated by extracting only the candidate area of the signal lamp is given to the tracking model. When only the candidate area of the traffic light is extracted and the candidate area of the corresponding signal lamp is not extracted, the position and size of the traffic light are estimated based on the position and size of the candidate area of the traffic light. Further, for example, the candidate area of the traffic light is divided into three equal parts in the horizontal direction, and the color information of each area is converted into the color distribution data of the signal lamp corresponding to each area (the left area is the color distribution data of the signal lamp “blue”, the central area Is evaluated using the color distribution data of the signal lamp “yellow” and the right area is the color distribution data of the signal lamp “red”, and the color of the area having the highest evaluation value is estimated as the color of the signal lamp. Then, a new tracking model is generated based on the estimated value. In addition, information of “undecided” indicating that the tracking model is generated by extracting only the candidate area of the traffic light is given to the tracking model. The tracking model to which the “indeterminate” information is assigned is determined by the “determined” information when a correspondence with an indeterminate candidate (a signal lamp candidate area or a traffic light candidate area) can be obtained in the processing after the next frame. Change to

The signal lamp candidate area corresponding to the traffic signal candidate area is searched from the signal lamp candidate area having a likelihood equal to or greater than the threshold value Pth _L , and has no correspondence with the signal lamp candidate area. When a new tracking model is created, a signal lamp candidate region having a likelihood equal to or higher than a threshold value Pth _H is used.

  Finally, the traffic light tracking unit 34 determines the position, the size of the signal light, based on the state quantity of the tracking model to which the “confirmation” information is given among the updated tracking model and the newly generated tracking model. The color information is displayed on the display device 18 as a detection result.

  With reference to FIG. 3, an example of the tracking model generation, update, and tracking end process by the traffic light tracking unit 34 will be described.

  First, (1) in a scene at t = k−4, when a candidate area of a traffic light and a candidate area of a signal light that have a corresponding relationship are extracted together, a new tracking model (“ Confirm ") is generated.

  Next, (2) when only the signal lamp candidate area corresponding to the tracking model generated in (1) above is detected in the scene of t = k−3, the observation value of the signal lamp candidate area is used. The tracking model is updated.

  Next, in the case of (3) t = k−2 where only the traffic signal candidate area corresponding to the tracking model updated in (2) above is detected, it is estimated from the observation value of the traffic signal candidate area The tracking model is updated using the estimated values.

  Next, (4) in the scene of t = k−1, when neither the candidate area of the traffic light nor the candidate area of the signal light corresponding to the tracking model updated in (3) is detected, the tracking model The tracking model is updated using the estimated value estimated from the predicted position.

  Next, (5) In the scene of t = k, if neither the candidate area of the traffic light nor the candidate area of the signal light corresponding to the tracking model updated in (4) has been detected a predetermined number of times, the tracking is finished. To do.

  Next, the operation of the vehicle signal detector 10 according to the first embodiment will be described. If the front of the host vehicle is imaged by the imaging device 12 when the host vehicle equipped with the on-vehicle signal detector 10 is traveling on a road, the signal detection processing routine shown in FIG. Repeatedly executed.

  First, in step 100, candidate extraction processing for extracting a candidate area for a traffic light and a candidate area for a signal light from a frame image is executed.

  Here, the candidate extraction processing routine will be described with reference to FIG.

  In step 1000, a frame image captured for each frame by the imaging device 12 is acquired. Next, in step 1002, a traffic light is displayed on the frame image acquired in step 1000 based on the mounting orientation of the imaging device 12 and the like. An existing area is estimated, and the area is set as a search range for searching for traffic lights.

  Next, in step 1004, color information and luminance information are extracted for each pixel (or several small areas) within the search range set in step 1002 in the same procedure as that for learning. The extracted color information and luminance information are compared with the color distribution data and luminance distribution data of each lighting color stored in the signal color distribution storage unit 24 to determine whether or not it is a candidate for a signal lamp. Also, an area where pixels determined to be signal lamp candidates concentrate is extracted, and whether or not the shape of the extracted area is an ellipse is calculated using, for example, the equation (1). Determine based on. A region having an ellipticity greater than or equal to a predetermined value is extracted as a candidate region for a signal lamp.

  Next, in step 1006, for each of the signal lamp candidate areas extracted in step 1004, the likelihood indicating the likelihood of signal lamp is calculated according to, for example, formulas (2) to (5).

  Next, in step 1008, images in the window are extracted while scanning windows of various sizes within the search range set in step 1002, and features are extracted from the extracted window image in the same procedure as in learning. Is extracted, and the discriminator as the shape feature stored in the traffic signal shape storage unit 30 is used to evaluate whether or not the window image is an image showing the traffic signal. Then, an area corresponding to the window image having a high evaluation value is extracted as a candidate area for the traffic signal.

  Next, the process proceeds to step 102 of the traffic light detection process (FIG. 4), and the traffic light candidate area and the traffic light candidate area extracted in time series are associated with each other to update the tracking model and detect the traffic light. Perform tracking processing.

  Here, the tracking processing routine will be described with reference to FIG.

  In step 1020, a detection position in the current frame is predicted based on the movement of the host vehicle between the previous frame and the current frame with respect to the tracking model generated up to one frame before.

  Next, in step 1022, it is determined whether or not there is a corresponding signal candidate region around the predicted position predicted in step 1020. If it exists, the signal candidate area is detected, and the process proceeds to step 1024. If not, the process proceeds to step 1032.

In step 1024, the threshold value of the likelihood of the signal lamp candidate region is set to the threshold value Pth _L.

Next, in step 1026, there is a signal lamp candidate area extracted in the peripheral area including the candidate area of the traffic light detected in step 1022 and having a likelihood of a signal lamp having a likelihood equal to or greater than the threshold value Pth _L. It is determined whether or not. If it exists, the signal lamp candidate area is detected, and the process proceeds to step 1028. If not, the process proceeds to step 1030.

  In step 1028, the tracking model is updated using the observation value of the signal lamp candidate area detected in step 1026. On the other hand, in step 1030, the position and size of the signal light are estimated based on the color information of the tracking model and the position and size of the candidate area of the traffic light detected in step 1022, and the estimated value is used for tracking. Update the model. Color information is not updated.

On the other hand, if it is determined in step 1022 that there is no corresponding signal candidate area and the process proceeds to step 1032, the likelihood threshold of the signal lamp candidate area is set to the threshold Pth _H.

Next, in step 1034, whether or not there is a signal lamp candidate area extracted in the peripheral area of the predicted position predicted in step 1020 and having a likelihood greater than or equal to the threshold value Pth _H. Determine. If it exists, the signal lamp candidate area is detected, and the process proceeds to step 1028. If not, the process proceeds to step 1036.

  In step 1036, the position and size of the signal light are estimated based on the color information of the tracking model and the predicted position predicted in step 1022, and the tracking model is updated using the estimated value. Color information is not updated.

  Next, in step 1038, the tracking model in which neither the candidate area of the traffic light nor the candidate area of the signal lamp corresponding to the specified number of times or more is deleted.

  Next, in step 1040, a tracking model is newly created based on at least one of the signal lamp candidate area and the traffic light candidate area for which no corresponding tracking model exists. At this time, if the signal lamp candidate area and the traffic light candidate area corresponding to each other are extracted, a tracking model with “confirmed” information is generated, and the signal lamp candidate area and the traffic light candidate area are generated. If any of the above is not extracted, a tracking model to which “indeterminate” information is added is generated.

  Next, among the tracking model updated in step 1028, 1030, or 1036 in step 1042 and the tracking model newly generated in step 1040, the state quantity of the tracking model to which “confirmed” information is given. Based on the above, the information on the position, size, and color of the signal lamp is displayed on the display device 18 as a detection result, and the process is terminated after returning.

  As described above, according to the traffic light detection apparatus of the first embodiment, the signal lamp candidate area and the traffic signal candidate area are extracted, and when there is a traffic signal candidate area corresponding to the tracking model, the signal lamp If there is no candidate area for the traffic light corresponding to the tracking model using information on the candidate area of the signal lamp with low likelihood indicating the likelihood, the position of the signal lamp using the information on the candidate area of the signal lamp with high likelihood Therefore, even when an inexpensive imaging device that does not have high dynamic range or sensitivity is used to detect the size and color, it is possible to prevent undetection and detect a traffic signal with high accuracy.

  Further, by using the two features of the signal light and the shape information of the traffic light together, it is possible to detect the traffic light with high accuracy without estimating the installation height of the traffic light.

  In addition, since the tracking model is updated when any one of the signal light candidate area and the traffic light candidate area is detected, the tracking target can be stably tracked until the tracking target is out of frame.

  Next, a second embodiment will be described. In addition, about the part which becomes the structure similar to 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  The second embodiment is different from the first embodiment in that a plurality of frame images having different exposure amounts are captured and images suitable for extraction of a candidate area for a signal lamp and a candidate area for a traffic light are used. Is different.

  As shown in FIG. 7, the computer 220 of the vehicle signal detector 210 according to the second embodiment has an imaging control unit 36 that controls the imaging device 12 so that a plurality of frame images with different exposure amounts are captured. The candidate area of the extracted signal lamp is verified using the search range setting section 22, the signal color distribution storage section 24, the signal lamp candidate extraction section 226 for extracting the candidate area of the signal lamp from the dark image, and the bright image. Based on the recognition results of the signal light candidate verification unit 38, the signal light likelihood calculation unit 28, the environmental illuminance recognition unit 40 that recognizes the environmental illuminance around the host vehicle, the traffic light shape storage unit 30, and the environmental illuminance recognition unit 40, It can be represented by a configuration including a traffic signal candidate extraction unit 232 that extracts a candidate region of a traffic signal from either a bright image or a dark image, and a traffic signal tracking unit 34.

  The imaging control unit 36 appropriately switches the exposure amount when imaging the front of the host vehicle with the imaging device 12, and controls the imaging device 12 so that a plurality of frame images with different exposure amounts are captured for each frame. To do. The switching of the exposure amount may be controlled so as to be repeatedly imaged with a plurality of preset exposure amounts, or the exposure amount may be made variable according to the brightness of the acquired frame image. When using a preset exposure amount, the exposure amount setting may be changed between daytime and nighttime. Further, the exposure amount may be controlled according to the luminance of the signal lamp being detected and tracked. However, since the signal lamp is a self-luminous object and has high luminance in the image, if the dynamic range of the imaging device 12 is insufficient, the pixel value is saturated and correct color information may not be acquired (color information may be lost). Therefore, at least one exposure amount that does not lose color information is included. Further, since it may be difficult to recognize shape information from an image in which the exposure amount is suppressed so that color information is not lost, an exposure amount larger than the exposure amount that does not lose color information is included. Hereinafter, a frame image captured with an exposure amount that does not lose color information is referred to as a “dark image”, and a frame image captured with an exposure amount greater than the dark image is referred to as a “bright image”.

  The signal lamp candidate extraction unit 226 uses the search range setting unit 22 to extract signal lamp candidate regions from the search range set in the dark image. By extracting a signal lamp candidate area from the dark image, the color information of the area can be detected with high accuracy. The method for extracting the candidate area of the signal lamp is the same as that of the signal lamp candidate extraction unit 26 of the first embodiment.

  The signal lamp candidate verification unit 38 verifies the shape of each candidate area of the signal lamp extracted by the signal lamp candidate extraction unit 226 using a bright image. Even if the candidate area of the signal lamp extracted from the dark image is actually a signal lamp, it may not necessarily be extracted as an area having a high ellipticity due to the dirt of the lamp, aging deterioration, apparent angle, or the like. On the other hand, there is a case where only the vicinity of the center of the high luminance region is extracted in a circular shape in the dark image because the exposure is narrowed down although it is not actually circular at night. For example, as shown in FIG. 8, on a bright image, a streetlight that is not circular (FIG. 8A) and a window light (FIG. 8B) are imaged with an exposure amount that can acquire color information of a signal light at night. On the dark image, only the central portion with high luminance is extracted in a circular shape (FIGS. (C) and (d)), and may be erroneously detected as a signal lamp.

  Therefore, the signal lamp candidate verification unit 38 extracts a high-luminance area of the bright image corresponding to the candidate area of the signal lamp extracted from the dark image, and calculates the ellipticity of the high-luminance area using, for example, equation (1). To do. If the ellipticity calculated by the signal lamp candidate extraction unit 226 is improved, the signal lamp candidate area is left as a signal lamp candidate area. If the ellipticity deteriorates, the signal lamp candidate area is excluded from the signal lamp candidate area. In addition, when the ellipticity is improved, the likelihood is preferably calculated by the signal lamp likelihood calculation unit 28 at the subsequent stage using the improved ellipticity.

  The ambient illuminance recognition unit 40 recognizes the brightness of the surrounding environment of the host vehicle and selects whether to input a bright image or a dark image to the subsequent traffic signal candidate extraction unit 232. The brightness of the surrounding environment may be determined from the luminance value of the entire acquired frame image (bright image or dark image), or a illuminometer, a sensor for detecting the headlight lighting state, etc. are provided and detected by this sensor. You may determine from a value. When the surrounding environment is bright (for example, in the daytime), a dark image is input to the traffic signal candidate extraction unit 232, and when it is dark (for example, at night), the bright image is input to the traffic signal candidate extraction unit 232. .

  The traffic signal candidate extraction unit 232 extracts a candidate region of the traffic signal from the bright image or the dark image input by the environment illuminance recognition unit 40. In this way, by using an image appropriately selected based on the surrounding ambient illuminance, the texture of the building and the background is also clarified, and the shape characteristics of the traffic light can be accurately determined. The method for extracting the candidate area of the traffic signal is the same as that of the candidate traffic signal extracting unit 32 of the first embodiment.

  Next, the operation of the vehicle signal detector 210 according to the second embodiment will be described. When the own vehicle equipped with the on-vehicle signal detector 210 is traveling on the road, the imaging device 12 controls the exposure amount by the imaging control unit 36 and images the front of the own vehicle. At 220, the traffic light detection processing routine shown in FIG. 4 is repeatedly executed. Note that the candidate detection process routine of the second embodiment will be described with reference to FIG. 9 because only the candidate extraction process is different from the traffic light detection process routine of the first embodiment. Further, the same processing as the candidate extraction processing of the first embodiment is denoted by the same reference numeral, and detailed description thereof is omitted.

  In step 2000, a bright image and a dark image captured for each frame by the imaging device 12 are acquired. Next, in step 1002, a search range is set for the bright image and the dark image acquired in step 2000. Next, in step 2004, candidate areas for signal lights are extracted from the dark image.

  Next, in step 2005, a high-intensity region of the bright image corresponding to the signal lamp candidate region extracted from the dark image is extracted, and the ellipticity of the high-intensity region is calculated using, for example, equation (1). If the ellipticity calculated when extracting the signal lamp candidate area in step 2004 is improved, it is left as a signal lamp candidate area. If the ellipticity deteriorates, the signal lamp candidate area is deteriorated. Exclude from

  Next, in step 1006, the likelihood indicating the likelihood of a signal lamp is calculated for each candidate area of the signal lamp verified in step 2005.

  Next, in step 2007, it is determined whether or not the environment around the host vehicle is daytime by determining whether the sum or average of the luminance values of the entire bright image or dark image acquired is equal to or greater than a predetermined threshold value. In the case of daytime, the process proceeds to step 2008 to extract a candidate area of the traffic light from the dark image. On the other hand, if the environment around the host vehicle is nighttime, the process proceeds to step 2009 to extract a candidate area of the traffic light from the bright image and return.

  As described above, according to the on-vehicle signal detector of the second embodiment, a dark image captured with an exposure amount that does not lose color information and a bright image captured with an exposure amount larger than the dark image are obtained. Therefore, even when an inexpensive imaging device that does not have a high dynamic range or sensitivity is used, it is possible to accurately detect both the color information of the signal lamp and the shape characteristics of the traffic light.

DESCRIPTION OF SYMBOLS 10,210 Vehicle signal detector 12 Imaging device 18 Display device 20 Computer 22 Search range setting part 24 Signal color distribution memory | storage part 26, 226 Signal lamp candidate extraction part 28 Signal lamp likelihood calculation part 30 Signal apparatus shape memory | storage part 32, 232 Signal candidate Extraction unit 34 Traffic light tracking unit 36 Imaging control unit 38 Signal lamp candidate verification unit 40 Environmental illuminance recognition unit

Claims (7)

A signal lamp that extracts a circular or elliptical area in which color information in a frame image captured for each frame by an imaging unit that captures the periphery of the host vehicle indicates at least one lighting color of a predetermined signal lamp as a candidate area of the signal lamp Candidate extraction means;
Likelihood calculating means for calculating likelihood indicating the signal lamp likelihood of the candidate area of the signal lamp extracted by the signal lamp candidate extracting means;
From the frame image, a traffic signal candidate extracting means for extracting a region indicating a predetermined shape feature as a traffic signal candidate region;
Extracted in a signal lamp candidate area having a likelihood equal to or greater than a first threshold extracted in a peripheral area including a traffic signal candidate area extracted by the traffic signal candidate extraction means, and in an area other than the peripheral area including the signal candidate area Detection means for detecting a position and a color of a traffic light based on a candidate area of a signal lamp having a likelihood that is greater than or equal to a second threshold higher than the first threshold;
A traffic light detection device.   The likelihood calculating means is based on at least one of the degree of coincidence between at least one lighting color of the predetermined signal lamp and the color information of the candidate area of the signal lamp, and the ellipticity of the signal lamp candidate area, The traffic light detection apparatus according to claim 1, wherein the likelihood is calculated.   The detection means includes a corresponding signal candidate region in a signal tracking region in a current frame image predicted from a signal tracking result in a previous frame image, and a peripheral region including the signal candidate region or 3. The traffic light detection apparatus according to claim 1, wherein tracking is continued by updating the tracking result in at least one of the cases where the candidate area for the signal lamp exists in a peripheral area including the tracking area. Imaging control means for controlling the imaging means so that a captured image is captured with a first exposure amount at which color information is not lost and a second exposure amount brighter than the first exposure amount,
The signal lamp candidate extraction means extracts a candidate area of the signal lamp from a captured image captured with the first exposure amount,
When the surrounding environment of the host vehicle is brighter than a predetermined value, the traffic signal candidate detecting unit extracts a candidate area of the traffic signal from a captured image captured at the first exposure amount, and the surrounding environment of the host vehicle The traffic light detection device according to any one of claims 1 to 3, wherein when the signal is darker than a predetermined value, a candidate area of the traffic light is extracted from a captured image captured with the second exposure amount.   The extracted candidate area of the signal lamp is verified based on the ellipticity of the high brightness area of the captured image captured at the second exposure amount corresponding to the candidate area of the signal lamp extracted by the signal lamp candidate extracting means. 5. The traffic light detection apparatus according to claim 4, further comprising verification means for performing the verification. Computer
A signal lamp that extracts a circular or elliptical area in which color information in a frame image captured for each frame by an imaging unit that captures the periphery of the host vehicle indicates at least one lighting color of a predetermined signal lamp as a candidate area of the signal lamp Candidate extraction means,
Likelihood calculating means for calculating a likelihood indicating the likelihood of signal light in the candidate area of the signal light extracted by the signal light candidate extracting means;
A signal candidate extraction unit that extracts a region indicating a predetermined shape feature as a candidate region of a traffic signal from the frame image, and a peripheral region that includes the candidate region of the traffic signal extracted by the signal candidate extraction unit. Signal lamp candidate areas having a likelihood equal to or higher than the first threshold, and signal lights having a likelihood equal to or higher than the second threshold extracted in areas other than the peripheral area including the signal candidate area A traffic light detection program for functioning as detection means for detecting the position and color of a traffic light based on a candidate area.   A signal detection program for causing a computer to function as each means constituting the signal detection device according to any one of claims 1 to 5.