JP6825299B2 - Information processing equipment, information processing methods and programs - Google Patents

Description

The present invention relates to information processing devices, information processing methods and programs.

A system that assists the driver by analyzing a captured image taken by a photographing device such as an in-vehicle camera and detecting a signal of a traffic light (typically a lit signal) included in the captured image is known. ing. Further, in order to recognize the signal of the traffic light, a device for detecting the signal color of the traffic light is known.

For example, Patent Document 1 discloses a method in which the gain of a camera is adjusted twice to acquire two captured images, and the signal color is detected using these two captured images. In Patent Document 1, the gain for the second shooting is adjusted by using the shot image obtained in the first shooting.

However, in the conventional technique, a plurality of captured images having different gains are required. In addition, the detection accuracy may be significantly reduced due to an error in gain adjustment. Further, in the state before detecting the region of the captured image in which the traffic light is reflected, the position of the signal in the captured image is unknown, so that it is difficult to set the gain that can accurately detect the signal color.

Further, even if the camera gain is optimally set, if an object having the same color and shape as the signal of the traffic light appears, there is a problem that the object is erroneously recognized as a signal of the traffic light. For example, if a part of the red signboard is hidden by a tree and the visible part is shaped like a circle indicating the color of the red traffic light, the red signboard is recognized as an area indicating the red traffic light. With conventional technology, false recognition cannot be deleted.

That is, with the conventional technique, there is a problem that it is difficult to recognize the signal of the traffic light reflected in the captured image with high accuracy.

The present invention has been made in view of the above, and an object of the present invention is to provide an information processing device, an information processing method, and a program capable of recognizing a signal of a traffic light reflected in a captured image with high accuracy. To do.

In order to solve the above-mentioned problems and achieve the object, the present invention recognizes a signal region indicating a signal of a signal among an acquisition unit that acquires a captured image and the captured image acquired by the acquisition unit. Based on the signal area recognition unit, the target signal identification information predetermined for identifying the target signal indicating the signal of the signal to be detected, and the signal area recognized by the signal area recognition unit. The target signal identification information includes a determination unit for determining whether or not the signal region is a region indicating the target signal, and the target signal identification information is different from the target signal region in the captured image and the target signal. It is an information processing device that is an evaluation function that calculates the separation surface from the false recognition area indicating the signal area from a feature quantity vector consisting of at least a plurality of color signals, the distance from the end of the captured image, and the width of the signal area. ..

According to the present invention, the signal of the traffic light reflected in the captured image can be recognized with high accuracy.

FIG. 1 is an explanatory diagram of an example of an information processing device. FIG. 2 is a diagram showing an example of a function of the information processing apparatus of the embodiment. FIG. 3 is a diagram showing a signal region showing a green light of a traffic light arranged in the horizontal direction. FIG. 4 is a diagram showing a signal region showing a red light of a traffic light arranged in the horizontal direction. FIG. 5 is a diagram showing an example of the recognition result of the signal area recognition unit. FIG. 6 is a diagram showing an example of the result of recognizing a plurality of traffic lights. FIG. 7 is a diagram showing an example of the distribution of the two-dimensional feature amount vector in any one time zone. FIG. 8 is a diagram showing a detailed configuration example of the time zone determination unit. FIG. 9 is a diagram showing an example of sample points indicated by the average luminance (Iav) and the number of low luminance blocks (Nblk) in the reference captured image. FIG. 10 is a diagram showing a pixel example of a red signal extracted by threshold processing in the (U, V) space of pixel values. FIG. 11 is a diagram showing an example of an image after the signal pixel region is expanded. FIG. 12 is a diagram showing a circular region of a red light extracted by the Hough transform. FIG. 13 is a diagram showing a signal region determined by the signal region determination unit. FIG. 14 is a flowchart showing an operation example of the recognition processing unit. FIG. 15 is a diagram showing an example of the hardware configuration of the photographing apparatus. FIG. 16 is a block diagram showing a hardware configuration example of the information processing device.

Hereinafter, embodiments of the information processing apparatus, information processing method, and program according to the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is an explanatory diagram of an example of the information processing device 10 of the present embodiment. The information processing device 10 is mounted on a mobile body, for example. A moving body is an object that can move its position in real space by running independently or towing. The moving body is, for example, a vehicle, an airplane, a train, a bogie, or the like. In the present embodiment, the case where the moving body is the vehicle 20 will be described as an example. That is, in the present embodiment, a mode in which the information processing device 10 is mounted on the vehicle 20 will be described as an example.

The vehicle 20 is equipped with a photographing device 12. The photographing device 12 obtains a photographed image of the periphery of the vehicle 20. The photographing device 12 is, for example, a known video camera, a digital camera, or the like. In the present embodiment, the photographing device 12 continuously photographs the periphery of the vehicle 20 to capture a plurality of captured images (that is, a plurality of frames). The photographing device 12 may be configured integrally with the information processing device 10 or may be configured as a separate body from the information processing device 10.

Further, the photographing device 12 is not limited to the form mounted on the vehicle 20. The photographing device 12 may be installed at a position where the traffic light 30 can be photographed, and may be fixed to the ground. When the detection result detected by the information processing device 10 of the present embodiment is used for driving support of the driver of the vehicle 20, the photographing device 12 is preferably mounted on the vehicle 20.

In the present embodiment, the photographing device 12 is equipped with an automatic gain control (AGC) function. Therefore, the photographing device 12 automatically adjusts the sensitivity and obtains a photographed image automatically adjusted so that the brightness of the entire screen of the photographed image is optimized.

The information processing device 10 analyzes the captured image. Then, the information processing device 10 detects the traffic light 30 included in the captured image.

Next, an example of the function of the information processing apparatus 10 will be described. FIG. 2 is a block diagram showing an example of the functions of the information processing device 10. For convenience of explanation, the functions related to the present embodiment are mainly illustrated here, but the functions of the information processing apparatus 10 are not limited to these.

The information processing device 10 includes an interface unit 14, a recognition processing unit 16, and a storage unit 18. The interface unit 14 and the storage unit 18 are electrically connected to the recognition processing unit 16.

The interface unit 14 receives a photographed image from the photographing device 12. The photographing device 12 continuously photographs the periphery of the vehicle 20 over time, and outputs each of the captured images (color images in this example) obtained by the photographing to the interface unit 14 in the order of photographing. The interface unit 14 sequentially receives captured images from the photographing device 12, and sequentially outputs them to the recognition processing unit 16 in the order of acceptance.

The storage unit 18 stores various data. In the present embodiment, the storage unit 18 has a time zone recognition dictionary DB 18A and a target signal identification dictionary DB 18B, the time zone recognition dictionary DB 18A holds a time zone recognition dictionary, and the target signal identification dictionary DB 18B identifies the target signal. Keep a dictionary. Details of these will be described later.

The recognition processing unit 16 analyzes the captured image and recognizes (detects) the signal of the traffic light reflected in the captured image. As shown in FIG. 2, the recognition processing unit 16 includes an image acquisition unit 101, a signal area recognition unit 102, a determination unit 103, a time zone determination unit 104, a position information acquisition unit 105, and a selection unit 106. The output unit 107 and the like are included. The signal area recognition unit 102 includes a candidate area recognition unit 111, a signal shape recognition unit 112, and a signal area determination unit 113.

Some or all of the image acquisition unit 101, signal area recognition unit 102, determination unit 103, time zone determination unit 104, position information acquisition unit 105, selection unit 106, and output unit 107 may be, for example, a CPU (Central Processing Unit) or the like. The program may be executed by the processing unit of the above, that is, it may be realized by software, it may be realized by hardware such as an IC (Integrated Circuit), or it may be realized by using software and hardware together. Good.

The image acquisition unit 101 is an example of the “acquisition unit” and acquires a captured image. In this example, the image acquisition unit 101 acquires captured images (moving images) sequentially input from the interface unit 14. That is, the image acquisition unit 101 acquires a photographed image taken by the photographing device 12 mounted on the vehicle.

The signal area recognition unit 102 recognizes a signal area indicating a signal of a traffic light among the captured images acquired by the image acquisition unit 101. In this example, the signal area recognition unit 102 includes a candidate area recognition unit 111, a signal shape recognition unit 112, and a signal area determination unit 113. Although a specific recognition method of the signal region will be described later, the signal region determination unit 113 determines the signal region based on the recognition result of the candidate region recognition unit 111 and the recognition result of the signal shape recognition unit 112. FIG. 3 is a diagram showing a signal region showing a green signal of a traffic light arranged in the horizontal direction, and FIG. 4 is a diagram showing a signal region showing a red signal of a traffic light arranged in the horizontal direction. The shape of the traffic light differs depending on the country or region. 3 and 4 are diagrams showing an example of a traffic light in Japan.

FIG. 5 is a diagram showing an example of the recognition result of the signal area recognition unit 102. The origin of the image is the point O on the upper left of the captured image (input image). The horizontal axis in the left-right direction is x, and the vertical axis in the downward direction is y. Here, the recognition result of the signal area is represented by a rectangular area surrounding the lit signal. The specific recognition method of the signal region will be described later. Further, here, the coordinates P (X, Y) on the upper left of the recognized rectangular area are output as the recognition result of the signal area. Y indicates the distance from the top of the screen to the point P on the upper left of the recognized rectangular area. This Y is one feature amount for discriminating the signal of the traffic signal to be detected. X indicates the distance from the leftmost of the screen to the point P on the upper left of the recognized rectangular area. This X is also a feature amount for discriminating the signal of the traffic signal to be detected. Further, here, the width of the recognized rectangular area is W, and the height is H. For convenience of explanation, the rectangular area is a square with W = H.

FIG. 6 is a diagram showing an example of the result of recognizing a plurality of traffic lights. The recognition result A is a signal region indicating a red light of the target traffic light. The recognition result B is a signal region indicating a green signal of a traffic light existing in a distant place, and is not a region of a target signal (a signal of a traffic light to be detected). The recognition result C is a signal region indicating a green signal of a road in the vertical direction, and is not a region of a target signal. The recognition results D to F indicate the result of erroneous recognition (not in the region of the target signal). The recognition result of the traffic light is described in the form of (Cr, Cy, Cg) representing each of the red signal, the yellow signal, and the green signal. Each element of (Cr, Cy, Cg) is a value of 1 or 0. The numerical value "1" means that a signal of the corresponding color has been detected. (1,0,0) means that a red light was detected, (0,1,0) means that a yellow light was detected, and (0,0,1) means that a green light was detected. Represents.

The explanation will be continued by returning to FIG. The determination unit 103 is based on predetermined target signal identification information for identifying a target signal indicating a signal of a signal to be detected (target signal) and a signal area recognized by the signal area recognition unit 102. Therefore, it is determined whether or not the signal region is a region indicating a target signal. The target signal identification information is created in advance by using a machine learning method using SVM (Support Vector Machine). More specifically, the target signal identification information is an evaluation function indicating a separation surface between a region of the target signal in the captured image and a false recognition region indicating a region of a signal different from the target signal.

Let the feature amount (Cr, Cy, Cg, W, X, Y) of the recognition result by the signal area recognition unit 102 be a six-dimensional feature amount vector. Actually, the target signal is discriminated by the 6-dimensional feature vector (vector data), but for convenience of explanation, an example of the 2-dimensional feature vector will be described. The two-dimensional feature amount vector representing the recognition result by the signal area recognition unit 102 describes the target signal (hereinafter, “red signal” is taken as an example, but signals of other colors can be considered in the same manner). When creating the target signal identification information for identifying whether to show or not, separately from the captured image used for detecting the target signal, the target area is photographed in advance for each of a plurality of time zones before the detection process. An image (referred to as a signal image) obtained by photographing the target traffic light by the device 12 is collected. Then, a feature amount vector (two-dimensional feature amount vector) including the width W of the signal region of the red signal reflected in the signal image and the distance Y from the upper end of the screen is obtained for each of a plurality of time zones. FIG. 7 is a diagram showing an example of the distribution of the two-dimensional feature amount vector in any one time zone. In the example of FIG. 7, the feature amount vector of the signal area in which the width (size) W of the signal area is relatively large and the distance Y from the upper edge of the screen is relatively small is the feature amount data representing the area of the target signal. Is a set of white rectangles on the upper left. On the other hand, the feature amount vector of the signal region in which the width W of the signal region is relatively small and the distance Y from the upper edge of the screen is relatively large is a region of a signal different from the target signal (for example, a signal existing in a distant place). It is feature data representing (misrecognition area), and is a set of black circles in the lower right.

Here, the determination unit 103 arranges a separation surface that separates a set of feature amount data (correct answer data) in the target signal area and a set of feature amount data in the erroneous recognition area so that the margin d is maximized. To do. Then, the determination unit 103 can pre-calculate (calculate by learning) the evaluation function indicating this separation surface (straight line L in the example of FIG. 7) as the target signal identification dictionary 18B. In this example, the subject that calculates the evaluation function is the determination unit 103, but the present invention is not limited to this, and the subject that calculates the evaluation function may have a function other than the determination unit 103.

The evaluation function is represented by a linear discriminant function such as the following equation (1), but is not limited to this and may be another non-linear discriminant function. A, B, and C in the following equation (1) are the coefficients of the evaluation function. Here, this evaluation function corresponds to the "target signal identification information".

The determination unit 103 substitutes the feature amount data (corresponding to the signal area recognized by the signal area recognition unit 102) indicating the recognition result by the signal area recognition unit 102 into the evaluation function, and evaluates the evaluation function f (W, Y). Calculate the value of. If the calculated value of the evaluation function f (W, Y) is equal to or greater than a predetermined threshold value, the determination unit 103 indicates the recognition result by the signal area recognition unit 102, and the feature amount data indicating the recognition result represents the area of the target signal. Judge. On the other hand, if the calculated value of the evaluation function f (W, Y) is less than the threshold value, the determination unit 103 does not represent the area of the target signal in the feature amount data indicating the recognition result by the signal area recognition unit 102. Judge.

In the above, the feature amount data indicating the recognition result to be judged whether or not the answer is correct has been described by taking the case of a two-dimensional feature amount vector as an example, but the number of dimensions can be changed arbitrarily. is there. For example, in the case of 6 dimensions, the evaluation function f (Cr, Cy, Cg, W, X, Y) indicating the separation surface that separates the set of correct answer data and the set of feature data of the misrecognition region is the following equation. It is represented by (2).

In this case, the feature amount data (Cr, Cy, Cg, W, X, Y) showing the recognition result by the signal area recognition unit 102 is 6-dimensional vector data, and the determination unit 103 uses this feature amount data (Cr, By substituting Cy, Cg, W, X, Y) into the above evaluation function, the value of the evaluation function f (Cr, Cy, Cg, W, X, Y) is calculated. Then, if the value of the calculated evaluation function f (Cr, Cy, Cg, W, X, Y) is equal to or higher than a predetermined threshold value, the determination unit 103 determines the feature amount data indicating the recognition result by the signal area recognition unit 102. Is determined to represent the region of the target signal. On the other hand, if the value of the calculated evaluation function f (Cr, Cy, Cg, W, X, Y) is less than the threshold value, the determination unit 103 targets the feature amount data indicating the recognition result by the signal area recognition unit 102. It is judged that it does not represent the area of the signal.

Here, an evaluation function is created by pre-learning for each combination of the time zone and the country or region, and stored in the target signal identification dictionary DB18B. The determination unit 103 is based on the evaluation function selected by the selection unit 106, which will be described later, and the feature amount data indicating the recognition result by the signal area recognition unit 102, and the feature amount data showing the recognition result by the signal area recognition unit 102. Determines whether or not indicates the region of the target signal.

The output unit 107 outputs a signal area corresponding to the feature amount data (feature amount data indicating the recognition result by the signal area recognition unit 102) determined to be the area indicating the target signal as the recognition result.

The time zone determination unit 104 determines the time zone in which the captured image acquired by the image acquisition unit 101 was captured. The time zone determination unit 104 determines the shooting time zone of the captured image by analyzing the captured image.

The shooting time zone is a day (24 hours) divided into a plurality of time zones different from each other in the shooting environment. Different shooting environments mean different light intensities. For example, the shooting time zone is daytime or nighttime. The shooting time zone may be daytime, nighttime, or evening. Further, the shooting time zone may be any time zone in which the day is divided into a plurality of time zones having different shooting environments, and is not limited to day, night, evening, and the like. The shooting time zone is not limited to two or three types, and may be four or more types. In addition, the shooting time zone may be appropriately set according to the shooting environment of the shot image (for example, season, country, region, northern hemisphere / southern hemisphere).

In the present embodiment, as an example, a case where the shooting time zone indicates daytime or nighttime will be described. The shooting time zone indicates daytime means that the light intensity of the shooting environment is equal to or higher than the threshold value. The shooting time zone is night is a time zone in which the light intensity of the shooting environment is less than the threshold value. An arbitrary value may be set in advance for the threshold value of this light intensity. For example, the light intensity in the region indicating the signal 30A of the traffic light 30 included in the captured image P begins to saturate at the threshold value of the light intensity by the auto gain control function of the photographing device 12 that captured the captured image P to be processed ( Alternatively, the light intensity (which is desaturated) may be determined.

The time zone determination unit 104 determines the shooting time zone of the captured image by using the brightness of the captured image acquired by the image acquisition unit 101. More specifically, as shown in FIG. 8, the time zone determination unit 104 includes a first calculation unit 116H, a second calculation unit 116I, and a third calculation unit 116J.

The first calculation unit 116H calculates the average brightness of the captured image. The first calculation unit 116H calculates the average value of the brightness values of each of the pixels included in the captured image. As a result, the first calculation unit 116H calculates the average brightness of the captured image. When the shooting time is daytime, the average brightness of the shot image is high. On the other hand, when the shooting time zone is night, the average brightness of the shot image is lower than that of daytime.

The second calculation unit 116I divides the captured image into a plurality of blocks. For example, the second calculation unit 116I divides the captured image into m × n blocks. Note that m and n are integers of 1 or more, and at least one of m and n is an integer of 2 or more.

Then, the second calculation unit 116I calculates the average brightness for each of the blocks. For example, the second calculation unit 116I calculates the average value of the brightness values of the pixels included in the block for each block. As a result, the second calculation unit 116I calculates the average brightness for each of the blocks included in the captured image P.

Further, the second calculation unit 116I calculates the number of blocks whose average brightness in the captured image is equal to or less than the threshold value as the number of low-luminance blocks in the captured image. An arbitrary value may be predetermined for the threshold value of the average luminance used for calculating the number of low-luminance blocks.

Then, the time zone determination unit 104 determines the photographing time zone based on the feature amount including the average brightness of the photographed image and the number of low-luminance blocks of the photographed image.

Specifically, the time zone determination unit 104 uses, for example, the average brightness of the captured image and the number of low-luminance blocks of the captured image as feature quantities at the time of time zone determination.

In the present embodiment, the time zone determination unit 104 creates in advance the time zone recognition dictionary 18A used for determining the shooting time zone. That is, the time zone determination unit 104 creates the time zone recognition dictionary 18A in advance before the time zone determination process of the captured image.

In the present embodiment, the time zone determination unit 104 creates a time zone recognition dictionary 18A in advance by using a machine learning method using an SVM (Support Vector Machine).

Specifically, the time zone determination unit 104 uses a plurality of captured images captured in advance for each imaging time zone as reference captured images.

The reference captured image is a captured image previously captured by the imaging device 12 prior to the detection process, in addition to the captured image used for detecting the region showing the signal of the traffic light. First, the time zone determination unit 104 registers a group of sample points indicated by the average luminance and the number of low-luminance blocks in the reference captured image in a two-dimensional space determined by the average luminance and the number of low-luminance blocks. FIG. 9 is a diagram showing an example of sample points indicated by the average luminance (Iav) and the number of low luminance blocks (Nblk) in the reference captured image.

In FIG. 9, the sample point group 40B is a group of sample points indicated by the average brightness and the number of low-luminance blocks in the reference shooting image taken during the shooting time zone indicating night. Further, the sample point group 40A is a group of sample points indicated by the average luminance and the number of low-luminance blocks in the reference photographing image photographed in the photographing time zone indicating daytime.

The time zone determination unit 104 sets a separation surface (here, a straight line La) that separates the daytime sample point group 40A and the nighttime sample point group 40B, and each of the daytime sample point group 40A and the nighttime sample point group 40B. The distance d (sometimes referred to as a margin) between the boundary lines (straight line La1 and straight line La2) is maximized. Then, the time zone determination unit 104 calculates an evaluation function indicating this separation surface (straight line La in FIG. 9). The following equation (3) is an equation showing an evaluation function showing this straight line La.

In the above equation (3), f (Iav, Nblk) is an evaluation function when the average brightness of the captured image and the number of low-luminance blocks of the captured image are used as feature quantities at the time of time zone determination. In formula (3), A, B, and C are coefficients of the evaluation function. Here, the evaluation function f (Iav, Nblk) or the coefficients A, B, and C correspond to the time zone recognition dictionary 18A.

The time zone determination unit 104 creates an evaluation function represented by the equation (3) in advance and stores it in the storage unit 18 in advance as the time zone recognition dictionary 18A.

When determining the shooting time zone of the captured image, the time zone determining unit 104 uses the time zone recognition dictionary 18A (evaluation function shown in the equation (3)) stored in the storage unit 18 to capture the captured image. Determine the band.

Specifically, the time zone determination unit 104 calculates the average brightness (Iav) and the number of low-luminance blocks (Nblk) of the captured image calculated by the first calculation unit 116H and the second calculation unit 116I in the equation (3). By substituting into, the value of the evaluation function f (Iav, Nblk) is calculated.

Then, when the value of the calculated evaluation function f (Iav, Nblk) is equal to or greater than a predetermined threshold value, the time zone determination unit 104 determines that the imaging time zone of the captured image indicates daytime. On the other hand, when the value of the calculated evaluation function f (Iav, Nblk) is less than the threshold value, the time zone determination unit 104 determines that the imaging time zone of the captured image indicates night. This threshold value may be set in advance according to the light intensity of the shooting environment in the shooting time zone to be discriminated.

The time zone determination unit 104 is based on a feature amount including the average brightness of the captured image, the number of low-luminance blocks of the captured image, and the dispersion value of the average brightness of each block in the captured image. The band may be discriminated.

In this case, the third calculation unit 116J of the time zone determination unit 104 calculates the dispersion value of the average brightness for each block in the captured image. The third calculation unit 116J calculates the dispersion value of the average brightness for each block divided by the second calculation unit 116I. For example, the time zone determination unit 104 calculates the variance value (σ) using the following equation (4).

In the formula (4), σ indicates the dispersion value of the average brightness for each block in the captured image. In the formula (4), N indicates the number of blocks (that is, the value of m × n) included in the captured image. Ii indicates the average brightness of the i-th block. Iav indicates the overall average brightness of the captured image.

In this case, the time zone determination unit 104 uses a time zone recognition dictionary that uses the average brightness of the captured image, the number of low-luminance blocks of the captured image, and the dispersion value of the average brightness of each block in the captured image as feature quantities. 18A may be created in advance.

That is, the time zone determination unit 104 uses a machine learning method using SVM (Support Vector Machine), as in the case of using the average brightness of the captured image and the number of low-luminance blocks of the captured image as feature quantities. Therefore, the time zone recognition dictionary 18A may be created in advance. In this case, the time zone determination unit 104 determines a group of sample points indicated by the average brightness, the number of low-luminance blocks, and the variance value in the reference captured image in a three-dimensional space determined by the average brightness, the number of low-luminance blocks, and the variance value. You can register at.

Then, the time zone determination unit 104 arranges the separation surface for separating the daytime sample point cloud and the nighttime sample point cloud so as to maximize the margin in the same manner as described above. Then, the time zone determination unit 104 may calculate the evaluation function indicating this separation surface as the time zone recognition dictionary 18A. The following equation (5) is an evaluation function (time zone recognition dictionary 18A) when the average brightness of the captured image P, the number of low-luminance blocks of the captured image P, and the variance value are used as feature quantities at the time of time zone discrimination. is there.

In equation (5), f (Iav, Nblk, σ) is an evaluation function (time) when the average brightness of the captured image, the number of low-luminance blocks of the captured image, and the variance value are used as feature quantities at the time of time zone discrimination. The band recognition dictionary 18A). In formula (5), A, B, C, and D are coefficients of the evaluation function.

As described above, the time zone determination unit 104 may, for example, create the evaluation function represented by the equation (5) in advance and store it in the storage unit 18 in advance as the time zone recognition dictionary 18A.

In this case, the time zone determination unit 104 uses the time zone recognition dictionary 18A (evaluation function shown in the equation (5)) stored in the storage unit 18 to determine the shooting time zone of the captured image. The shooting time zone of P is determined.

Specifically, the time zone determination unit 104 has an average brightness (Iav) and low brightness of the captured image calculated by each of the first calculation unit 116H, the second calculation unit 116I, and the third calculation unit 116J. The number of blocks (Nblk) and the variance value (σ) are obtained. Then, the time zone determination unit 104 substitutes the average luminance (Iav), the number of low-luminance blocks (Nblk), and the variance value (σ) into the equation (5), thereby substituting the evaluation function f (Iav, Nblk). , Σ) is calculated.

Then, when the value of the calculated evaluation function f (Iav, Nblk, σ) is equal to or greater than a predetermined threshold value, the time zone determination unit 104 determines that the imaging time zone of the captured image indicates daytime. On the other hand, when the value of the calculated evaluation function f (Iav, Nblk, σ) is less than the threshold value, the time zone determination unit 104 determines that the imaging time zone of the captured image indicates night. This threshold value may be set in advance.

As described above, the time zone determination unit 104 determines the time zone in which the captured image acquired by the image acquisition unit 101 was captured, and outputs the determined shooting time zone to the selection unit 106.

Returning to FIG. 2, the description of the function of the recognition processing unit 16 will be continued. The position information acquisition unit 105 acquires the position information of the own device. For example, the position information acquisition unit 105 can acquire position information from an external device such as a GPS device. The position information acquisition unit 105 outputs the acquired position information to the selection unit 106.

The selection unit 106 selects the target signal identification information corresponding to the time zone determined by the time zone determination unit 104. Further, the selection unit 106 selects the target signal identification information corresponding to the position information (country or region can be specified) acquired by the position information acquisition unit 105. More specifically, the selection unit 106 corresponds to a combination of the time zone determined by the time zone determination unit 104 and the position information (country or region can be specified) acquired by the position information acquisition unit 105. Select signal identification information.

Next, a method of recognizing the signal region by the signal region recognition unit 102 will be described. The signal region recognition unit 102 extracts a signal pixel region including a signal pixel indicating the signal color of the traffic light from the captured image acquired by the image acquisition unit 101, and previously expands the extracted signal pixel region into an expanded expansion region. When a standard region having a predetermined shape is included, the region including the standard region is recognized as a signal region. More specifically, when the expansion region includes a circular standard region, the signal region recognition unit 102 recognizes the region including the standard region as a signal region.

The signal area recognition unit 102 includes a candidate area recognition unit 111, a signal shape recognition unit 112, and a signal area determination unit 113. The candidate area recognition unit 111 extracts (recognizes) a signal pixel area including a signal pixel indicating the signal color of the traffic light from the captured image image acquired by the image acquisition unit 101 as a candidate area. The signal shape recognition unit 112 recognizes whether or not the expansion region in which the candidate region recognized by the candidate region recognition unit 111 is expanded includes a fixed region having a predetermined shape. The signal area determination unit 113 determines the signal area based on the recognition result by the candidate area recognition unit 111 and the recognition result by the signal shape recognition unit 112. The specific contents will be described below.

When the captured image acquired by the image acquisition unit 101 is image data in the (R, G, B) color space, the candidate area recognition unit 111 uses the captured image (input) in the RGB color space acquired by the image acquisition unit 101. Image) is changed to image data in the (Y, U, V) color space, and signal pixels are extracted. Here, the conversion between the color space of (R, G, B) and the color space of (Y, U, V) is performed by the following equation (6).

In this example, a signal recognition dictionary indicating the range of color values of the signal of the traffic light is obtained in advance by learning by collecting captured images (referred to as "signal image samples") obtained by photographing the signal of the traffic light in advance by the photographing device 12. Create it. For example, a separation surface that separates the (U, V) value distribution showing a red signal and the (U, V) value distribution other than the red signal is arranged so that the margin (distance) between the two is maximized. Then, the evaluation function indicating the separation surface can be calculated as a red signal recognition dictionary for recognizing the red signal. The following equation (7) is an equation representing the evaluation function. In the equation (7), f (U, V) is an evaluation function indicating a red light recognition dictionary, and a, b, and c are coefficients of this evaluation function. A green signal recognition dictionary and a yellow signal recognition dictionary can be created in the same manner.

The candidate area recognition unit 111 sets the (U, V) value of each pixel constituting the captured image in the (Y, U, V) color space to each signal color (any of red, yellow, and blue). Is assigned to the evaluation function f (U, V) corresponding to, and the value of the evaluation function f (U, V) is obtained. If the value of the calculated evaluation function f (U, V) is equal to or greater than a predetermined threshold value, it is determined that the pixel is a signal pixel of the corresponding signal color. For example, a signal pixel may be extracted by setting a minimum value threshold value and a maximum value threshold value. Thresholds for the minimum and maximum values of U and the minimum and maximum values of V, respectively, of red, blue, and yellow may be set. Further, the threshold value is set so as not to overlap with the (U, V) values of pixels other than the signal pixel.

FIG. 10 is a diagram showing a pixel example of a red signal extracted by threshold processing in the (U, V) space of pixel values. Since the signal pixel region including the signal pixels (pixels showing the red signal) in the image of FIG. 10 includes saturated pixels and noise pixels, the extracted signal pixel region is a part of the actual region showing the red signal. There is only. Therefore, the candidate area recognition unit 111 performs expansion processing on the extracted signal pixels. In the expansion process, an N × N pixel block is added to one pixel from the original signal image. For example, when N = 7, a 7 × 7 pixel block is added to one pixel to be expanded from the original signal image. FIG. 11 is a diagram showing an example of an image after the signal pixel region of FIG. 10 has been expanded. Hereinafter, the signal pixel region after the expansion processing is referred to as an “expansion region”.

The signal shape recognition unit 112 performs signal shape recognition processing on the expansion region. Shape recognition is performed for red, blue, and yellow signals by circular extraction. Here, the shape of the expanded region is recognized by extracting a circle. The circle extraction process can perform circle extraction by Hough transform. When a circle is extracted by the signal shape recognition unit 112, the signal area determination unit 113 obtains a rectangle circumscribing the extracted circle area. Then, the signal area determination unit 113 determines this rectangular area as a signal area, and calculates the feature amount (feature amount data indicating the recognition result) corresponding to the signal area. The circle drawn by the thick line shown in FIG. 12 indicates the circle region of the red signal extracted by the Hough transform. The rectangular region shown in FIG. 13 indicates a signal region determined by the signal region determination unit 113. The rectangular region shown in FIG. 5 is a diagram showing a candidate region (signal region showing a red light) recognized from the captured image of FIG.

FIG. 14 is a flowchart showing an operation example of the recognition processing unit 16 of the present embodiment. Since the specific contents of each step are as described above, detailed description thereof will be omitted as appropriate. As shown in FIG. 14, first, the image acquisition unit 101 acquires a captured image (step S1). Next, the signal area recognition unit 102 recognizes the signal area indicating the signal of the traffic light among the captured images acquired in step S1 (step S2). Further, in parallel with the above processing, the time zone determination unit 104 determines the time zone in which the captured image acquired in step S1 was captured (step S3).

Next, the selection unit 106 selects an evaluation function corresponding to the combination of the time zone determined in step S3 and the country or region specified by the position information acquired by the position information acquisition unit 105 (step S4). ). Next, the determination unit 103 determines whether or not the signal region is a region indicating the target signal based on the signal region (feature amount data) recognized in step S2 and the evaluation function selected in step S4. Performs a determination process to determine whether or not (step S5). The output unit 107 outputs the signal area determined to be the area indicating the target signal by the determination process in step S5 as the recognition result (step S6).

As described above, in the present embodiment, the target signal identification information predetermined for identifying the target signal indicating the signal of the signal to be detected and the signal region recognized by the signal region recognition unit 102. Based on, it is determined whether or not the signal region is a region indicating a target signal. According to this embodiment, the signal of the traffic light reflected in the captured image can be recognized with high accuracy.

Next, an example of the hardware configuration of the photographing device 12 will be described. FIG. 15 is a diagram showing an example of the hardware configuration of the photographing device 12.

The photographing device 12 includes a photographing optical system 2010, a mechanical shutter 2020, a motor driver 2030, a CCD (Charge Coupled Device) 2040, a CDS (Correlated Double Sample) circuit 2050, an A / D converter 2060, and a timing signal generator. 2070, image processing circuit 2080, LCD (Liquid Crystal Device) 2090, CPU (Central Processing Unit) 2100, RAM (Random Access Memory) 2110, ROM (Read Only Memory) 2120, SRAM (Syndy) It includes an extension circuit 2140, a memory 2150, an operation unit 2160, and an output I / F 2170.

The image processing circuit 2080, CPU 2100, RAM 2110, ROM 2120, SDRAM 2130, compression / decompression circuit 2140, memory 2150, operation unit 2160, and output I / F 2170 are connected via a bus 2200.

The photographing optical system 2010 collects the reflected light of the subject. By opening the mechanical shutter 2020 for a predetermined time, the light focused by the photographing optical system 2010 is incident on the CCD 2040. The motor driver 2030 drives the photographing optical system 2010 and the mechanical shutter 2020.

The CCD 2040 forms an image of the light incident through the mechanical shutter 2020 as an image of the subject, and inputs analog image data showing the image of the subject to the CDS circuit 2050.

When the CDS circuit 2050 receives analog image data from the CCD 2040, the CDS circuit 2050 removes the noise component of the image data. Further, the CDS circuit 2050 performs analog processing such as correlated double sampling and gain control. Then, the CDS circuit 2050 outputs the processed analog image data to the A / D converter 2060.

When the A / D converter 2060 receives the analog image data from the CDS circuit 2050, the A / D converter 2060 converts the analog image data into digital image data. The A / D converter 2060 inputs digital image data to the image processing circuit 2080. The timing signal generator 2070 transmits a timing signal to the CCD 2040, the CDS circuit 2050, and the A / D converter 2060 in response to the control signal from the CPU 2100, thereby transmitting the timing signal to the CCD 2040, the CDS circuit 2050, and the A / D converter 2060. Controls the operation timing of.

When the image processing circuit 2080 receives the digital image data from the A / D converter 2060, the image processing circuit 2080 uses the SDRAM 2130 to perform image processing of the digital image data. The image processing includes, for example, CrCb conversion processing, white balance control processing, contrast correction processing, edge enhancement processing, color conversion processing, and the like.

The image processing circuit 2080 outputs the image data subjected to the above-mentioned image processing to the LCD 2090 or the compression / decompression circuit 2140. The LCD 2090 is a liquid crystal display that displays image data received from the image processing circuit 2080.

When the compression / decompression circuit 2140 receives the image data from the image processing circuit 2080, the compression / decompression circuit 2140 compresses the image data. The compression / decompression circuit 2140 stores the compressed image data in the memory 2150. Further, when the compression / decompression circuit 2140 receives the image data from the memory 2150, the compression / decompression circuit 2140 decompresses the image data. The compression / decompression circuit 2140 temporarily stores the decompressed image data in the SDRAM 2130. Memory 2150 stores compressed image data.

The output I / F 2170 outputs the image data processed by the image processing circuit 2080 to the information processing apparatus 10 as a captured image P.

At least a part of the functional unit included in the interface unit 14 and the recognition processing unit 16 described with reference to FIG. 2 may be mounted on the photographing device 12 as a signal processing board (signal processing circuit).

Next, the hardware configuration of the information processing apparatus 10 according to the above embodiment will be described. FIG. 16 is a block diagram showing a hardware configuration example of the information processing device 10 of the above embodiment.

In the information processing device 10 of the above embodiment, the output unit 80, the I / F unit 82, the input unit 94, the CPU 86, the ROM (Read Only Memory) 88, the RAM (Random Access Memory) 90, the HDD 92, and the like are mutually connected by the bus 96. It is connected to and has a hardware configuration using a normal computer.

The CPU 86 is an arithmetic unit that controls the processing executed by the information processing device 10 of the above embodiment. The RAM 90 stores data required for various processes by the CPU 86. The ROM 88 stores a program or the like that realizes various processes by the CPU 86. The HDD 92 stores the data stored in the storage unit 18 described above. The I / F unit 82 is an interface for transmitting and receiving data to and from other devices.

A program for executing the various processes executed by the information processing apparatus 10 of the above embodiment is provided by being incorporated in ROM 88 or the like in advance.

The program executed by the information processing device 10 of the above-described embodiment is a file in a format that can be installed or executed in these devices, and is a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital). It may be configured to be recorded and provided on a computer-readable recording medium such as a Versail Disc).

Further, the program executed by the information processing apparatus 10 of the above embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading the program via the network. Further, the program for executing each of the above processes in the information processing apparatus 10 of the above embodiment may be configured to be provided or distributed via a network such as the Internet.

Each of the above-described parts of the program for executing the various processes executed by the information processing device 10 of the above-described embodiment is generated on the main storage device.

The various information stored in the HDD 92 may be stored in an external device. In this case, the external device and the CPU 86 may be connected to each other via a network or the like.

10 Information processing device 14 Interface unit 16 Recognition processing unit 18 Storage unit 101 Image acquisition unit 102 Signal area recognition unit 103 Judgment unit 104 Time zone discrimination unit 105 Position information acquisition unit 106 Selection unit 107 Output unit 111 Candidate area recognition unit 112 Signal shape Recognition unit 113 Signal area determination unit

JP-A-2009-2449446

Claims (8)

The acquisition unit that acquires the captured image and
Among the captured images acquired by the acquisition unit, a signal area recognition unit that recognizes a signal area indicating a signal of a traffic light and a signal area recognition unit.
Based on the target signal identification information predetermined for identifying the target signal indicating the signal of the signal to be detected and the signal region recognized by the signal region recognition unit, the signal region is the said. It is equipped with a judgment unit that determines whether or not the area indicates the target signal.
The target signal identification information is obtained from at least a plurality of color signals and edges of the captured image on a separation surface between a region of the target signal in the captured image and a false recognition region indicating a region of a signal different from the target signal. An information processing device that is an evaluation function calculated from a feature quantity vector consisting of the distance of and the width of the signal area . The target signal identification information is created in advance by using a machine learning method using SVM (Support Vector Machine).
The information processing device according to claim 1. The target signal identification information is predetermined for each of a plurality of time zones.
A time zone determination unit that determines the time zone in which the captured image was taken, and
A selection unit for selecting the target signal identification information corresponding to the time zone determined by the time zone determination unit is further provided.
The information processing device according to any one of claims 1 and 2 . The target signal identification information is predetermined for each country or region.
The location information acquisition unit that acquires location information and
A selection unit for selecting the target signal identification information corresponding to the position information acquired by the position information acquisition unit is further provided.
The information processing device according to any one of claims 1 to 3 . The signal area recognition unit
When a signal pixel region including a signal pixel indicating the signal color of a traffic light is extracted from the captured image, and the expanded region obtained by expanding the extracted signal pixel region includes a fixed region having a predetermined shape, the fixed shape is used. Recognizing a region including a region as the signal region,
The information processing device according to any one of claims 1 to 4 . When the expansion region includes the circular standard region, the signal region recognition unit recognizes the region including the standard region as the signal region.
The information processing device according to claim 5 . The acquisition step to acquire the captured image and
Among the captured images acquired by the acquisition step, a signal region recognition step for recognizing a signal region indicating a signal of a traffic light and a signal region recognition step.
A separation surface is defined in advance to identify a target signal indicating a signal of a signal to be detected, and a separation surface between a target signal region in the captured image and a false recognition region indicating a signal region different from the target signal. , At least a plurality of color signals, target signal identification information which is an evaluation function calculated from a feature quantity vector consisting of a distance from an edge of a captured image and a width of a signal region, and the signal region recognized by the signal region recognition step. A determination step for determining whether or not the signal region is a region indicating the target signal based on the above.
Information processing method. On the computer
The acquisition step to acquire the captured image and
Among the captured images acquired by the acquisition step, a signal region recognition step for recognizing a signal region indicating a signal of a traffic light and a signal region recognition step.
A separation surface is defined in advance to identify a target signal indicating a signal of a signal to be detected, and a separation surface between a target signal region in the captured image and a false recognition region indicating a signal region different from the target signal. , At least a plurality of color signals, target signal identification information which is an evaluation function calculated from a feature quantity vector consisting of a distance from an edge of a captured image and a width of a signal region, and the signal region recognized by the signal region recognition step. A program for executing a determination step of determining whether or not the signal region is a region indicating the target signal based on the above.

Images