JP2018072884A - Information processing device, information processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing device, an information processing method and a program that can recognize a signal apparatus projected into an imaged image with a high degree of accuracy.SOLUTION: An information processing device according to the present invention comprises an acquisition unit, a signal area recognition unit and a determination unit. The acquisition unit acquires imaged images. The signal area recognition unit recognizes a signal area indicating a signal of a signal apparatus of the imaged images acquired by the acquisition unit. The determination unit determines whether or not the signal area is an area indicating an object signal on the basis of object signal identification information predetermined for identifying the object signal indicating a signal of the signal apparatus being a detection object and the signal area recognized by the signal area recognition unit.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an information processing apparatus, an information processing method, and a program.

  A system for assisting a driver by analyzing a photographed image taken by a photographing device such as an in-vehicle camera and detecting a signal of a traffic light (typically a lit signal) included in the photographed image is known. ing. There is also known an apparatus for detecting the signal color of a traffic light in order to recognize the signal of the traffic light.

  For example, Patent Document 1 discloses a method of adjusting two camera gains to acquire two captured images and detecting a signal color using these two captured images. In Patent Document 1, the gain for the second shooting is adjusted using the shot image acquired in the first shooting.

  However, in the conventional technique, a plurality of photographed images having different gains are required. In addition, detection accuracy may be significantly reduced due to an error in gain adjustment. Furthermore, in the state before detecting the area in which the traffic signal is reflected in the photographed image, the position of the signal in the photographed image is unknown, so it is difficult to set the gain that can accurately detect the signal color.

  Even if the camera gain is set optimally, if an object having the same color and shape as the signal of the traffic light appears, there is a problem that the object is erroneously recognized as a signal of the traffic light. For example, when a part of a red signboard is hidden by a tree and the shape of a visible part becomes a circle indicating the color of a red signal, the red signboard is recognized as an area indicating a red signal. With the conventional technology, it is impossible to delete the erroneous recognition.

  That is, the conventional technique has a problem that it is difficult to recognize the signal of the traffic signal reflected in the captured image with high accuracy.

  The present invention has been made in view of the above, and an object of the present invention is to provide an information processing apparatus, an information processing method, and a program capable of accurately recognizing a signal of a traffic signal reflected in a captured image. To do.

  In order to solve the above-described problems and achieve the object, the present invention recognizes a signal area indicating a signal of a traffic light among an acquisition unit that acquires a captured image and the captured image acquired by the acquisition unit. Based on the signal region recognition unit, the target signal identification information predetermined for identifying the target signal indicating the signal of the traffic signal to be detected, and the signal region recognized by the signal region recognition unit, And a determination unit that determines whether or not the signal area is an area indicating the target signal.

  According to the present invention, it is possible to recognize a signal of a traffic signal reflected in a captured image with high accuracy.

FIG. 1 is an explanatory diagram of an example of an information processing apparatus. FIG. 2 is a diagram illustrating an example of functions of the information processing apparatus according to the embodiment. FIG. 3 is a diagram showing a signal area indicating a green signal of a traffic light arranged in the horizontal direction. FIG. 4 is a diagram showing a signal area indicating a red signal of a traffic light arranged in the horizontal direction. FIG. 5 is a diagram illustrating an example of a recognition result of the signal region recognition unit. FIG. 6 is a diagram illustrating an example of a result of recognizing a plurality of traffic lights. FIG. 7 is a diagram illustrating an example of the distribution of a two-dimensional feature vector in any one time zone. FIG. 8 is a diagram illustrating a detailed configuration example of the time zone determination unit. FIG. 9 is a diagram illustrating an example of sample points indicated by the average luminance (Iav) and the number of low luminance blocks (Nblk) in the reference photographed image. FIG. 10 is a diagram illustrating a pixel example of a red signal extracted by threshold processing in a (U, V) space of pixel values. FIG. 11 is a diagram illustrating an example of an image after the signal pixel region is expanded. FIG. 12 is a diagram illustrating a circular region of a red signal extracted by the Hough transform. FIG. 13 is a diagram illustrating signal regions determined by the signal region determination unit. FIG. 14 is a flowchart illustrating an operation example of the recognition processing unit. FIG. 15 is a diagram illustrating an example of a hardware configuration of the photographing apparatus. FIG. 16 is a block diagram illustrating a hardware configuration example of the information processing apparatus.

  Hereinafter, embodiments of an information processing apparatus, an information processing method, and a program according to the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 is an explanatory diagram of an example of the information processing apparatus 10 according to the present embodiment. The information processing apparatus 10 is mounted on a moving body, for example. A moving body is an object that can move in real space by self-running or towing. The moving body is, for example, a vehicle, an airplane, a train, a carriage, or the like. In the present embodiment, a case where the moving body is the vehicle 20 will be described as an example. That is, in the present embodiment, an example in which the information processing apparatus 10 is mounted on the vehicle 20 will be described.

  A photographing device 12 is mounted on the vehicle 20. The photographing device 12 obtains a photographed image obtained by photographing the periphery of the vehicle 20. The imaging device 12 is, for example, a known video camera or digital camera. In the present embodiment, the imaging device 12 captures a plurality of captured images (that is, a plurality of frames) by continuously capturing the periphery of the vehicle 20. The imaging device 12 may be configured integrally with the information processing device 10 or may be configured separately from the information processing device 10.

  Moreover, the imaging device 12 is not limited to the form mounted on the vehicle 20. The imaging device 12 only needs to be installed at a position where the traffic light 30 can be imaged, and may be fixed to the ground. When the detection result detected by the information processing apparatus 10 according to the present embodiment is used for driving assistance of the driver of the vehicle 20, the photographing apparatus 12 is preferably mounted on the vehicle 20.

  In the present embodiment, the photographing apparatus 12 has an auto gain control (AGC) function. For this reason, the imaging device 12 automatically adjusts the sensitivity and obtains a captured image that is automatically adjusted so that the brightness of the entire screen of the captured image is optimal.

  The information processing apparatus 10 analyzes the captured image. Then, the information processing apparatus 10 detects the traffic light 30 included in the captured image.

  Next, an example of functions that the information processing apparatus 10 has will be described. FIG. 2 is a block diagram illustrating an example of functions that the information processing apparatus 10 has. For convenience of explanation, functions related to the present embodiment are mainly exemplified here, but the functions of the information processing apparatus 10 are not limited to these.

  The information processing apparatus 10 includes an interface unit 14, a recognition processing unit 16, and a storage unit 18. The interface unit 14 and the storage unit 18 are electrically connected to the recognition processing unit 16.

  The interface unit 14 receives a photographed image from the photographing apparatus 12. The imaging device 12 continuously captures the periphery of the vehicle 20 over time, and outputs each of the captured images (color images in this example) obtained by capturing to the interface unit 14 sequentially in the order of shooting. The interface unit 14 sequentially receives captured images from the imaging device 12 and sequentially outputs them to the recognition processing unit 16 in the order of reception.

  The storage unit 18 stores various data. In the present embodiment, the storage unit 18 includes a time zone recognition dictionary DB 18A and a target signal identification dictionary DB 18B. The time zone recognition dictionary DB 18A holds a time zone recognition dictionary, and the target signal identification dictionary DB 18B is a target signal identification. Keep a dictionary. Details of these will be described later.

  The recognition processing unit 16 analyzes the captured image and recognizes (detects) the signal of the traffic light reflected in the captured image. As shown in FIG. 2, the recognition processing unit 16 includes an image acquisition unit 101, a signal region recognition unit 102, a determination unit 103, a time zone determination unit 104, a position information acquisition unit 105, a selection unit 106, And an output unit 107. The signal region recognition unit 102 includes a candidate region recognition unit 111, a signal shape recognition unit 112, and a signal region determination unit 113.

  Some or all of the image acquisition unit 101, the signal region recognition unit 102, the determination unit 103, the time zone determination unit 104, the position information acquisition unit 105, the selection unit 106, and the output unit 107 are, for example, a CPU (Central Processing Unit) or the like. The processing apparatus may execute a program, that is, may be realized by software, may be realized by hardware such as an IC (Integrated Circuit), or may be realized by using software and hardware in combination. Good.

  The image acquisition unit 101 is an example of an “acquisition unit”, and acquires a captured image. In this example, the image acquisition unit 101 acquires captured images (moving images) sequentially input from the interface unit 14. That is, the image acquisition unit 101 acquires a captured image captured by the imaging device 12 mounted on the vehicle.

  The signal area recognizing unit 102 recognizes a signal area indicating a signal from the traffic light among the captured images acquired by the image acquiring unit 101. In this example, the signal area recognition unit 102 includes a candidate area recognition unit 111, a signal shape recognition unit 112, and a signal region determination unit 113. Although a specific method for recognizing the signal region will be described later, the signal region determination unit 113 determines the signal region based on the recognition result of the candidate region recognition unit 111 and the recognition result of the signal shape recognition unit 112. FIG. 3 is a diagram showing a signal area indicating a blue signal of a traffic light arranged in the horizontal direction, and FIG. 4 is a diagram showing a signal area showing a red signal of the traffic light arranged in the horizontal direction. The shape of the traffic light varies depending on the country and region. 3 and 4 are diagrams showing examples of traffic lights in Japan.

  FIG. 5 is a diagram illustrating an example of a recognition result of the signal area recognition unit 102. The origin of the image is a point O at the upper left of the captured image (input image). The horizontal axis in the horizontal direction is x, and the vertical axis in the downward direction is y. Here, the recognition result of the signal area is represented by a rectangular area surrounding the signal to be lit. A specific method for recognizing the signal area will be described later. Here, the upper left coordinate P (X, Y) of the recognized rectangular area is output as the recognition result of the signal area. Y indicates the distance from the top of the screen to the upper left point P of the recognized rectangular area. This Y is one feature value for discriminating the signal of the traffic signal to be detected. X represents the distance from the leftmost part of the screen to the upper left point P of the recognized rectangular area. This X is also one feature amount for discriminating the signal of the traffic signal to be detected. Here, the width of the recognized rectangular area is W and the height is H. For convenience of explanation, the rectangular area is a square of W = H.

  FIG. 6 is a diagram illustrating an example of a result of recognizing a plurality of traffic lights. The recognition result A is a signal area indicating a red signal of the target traffic light. The recognition result B is a signal region indicating a blue signal of a traffic light that exists far away, and is not a region of a target signal (signal of a traffic signal to be detected). The recognition result C is a signal area indicating a green signal of a road in the vertical direction, and is not a target signal area. Recognition results D to F indicate misrecognition results (not target signal areas). The recognition result of the traffic light is described in the form of (Cr, Cy, Cg) representing each of the red signal, the yellow signal, and the blue signal. Each element of (Cr, Cy, Cg) has a value of 1 or 0. The numerical value “1” means that the corresponding color signal is detected. (1, 0, 0) indicates that a red signal has been detected, (0, 1, 0) indicates that a yellow signal has been detected, and (0, 0, 1) indicates that a blue signal has been detected. Represents.

  Returning to FIG. 2, the description will be continued. The determination unit 103 is based on target signal identification information determined in advance for identifying a target signal indicating a signal of a signal device (target signal device) to be detected and a signal region recognized by the signal region recognition unit 102. Thus, it is determined whether or not the signal area is an area indicating the target signal. The target signal identification information is created in advance using a machine learning method using SVM (Support Vector Machine). More specifically, the target signal identification information is an evaluation function that indicates a separation plane between a target signal area and a misrecognition area indicating a signal area different from the target signal in the captured image.

  The feature amount (Cr, Cy, Cg, W, X, Y) of the recognition result by the signal region recognition unit 102 is set as a six-dimensional feature amount vector. Actually, the target signal is discriminated based on a six-dimensional feature vector (vector data). For convenience of explanation, an example of a two-dimensional feature vector will be described. A two-dimensional feature vector representing a recognition result by the signal area recognition unit 102 is a target signal (hereinafter described by taking “red signal” as an example, but signals of other colors can be considered in the same manner). When creating target signal identification information for identifying whether or not to show, separately from the captured image used for detection of the target signal, before the detection process, the image is captured in advance for each of a plurality of time zones in the target area. An image (referred to as a signal image) obtained by photographing the target traffic signal with the device 12 is collected. Then, a feature quantity vector (two-dimensional feature quantity vector) including the width W of the signal area of the red signal reflected in the signal image and the distance Y from the upper end of the screen is obtained for each of a plurality of time zones. FIG. 7 is a diagram illustrating an example of the distribution of a two-dimensional feature vector in any one time zone. In the example of FIG. 7, the feature quantity vector of the signal area where the width (size) W of the signal area is relatively large and the distance Y from the upper end of the screen is relatively small is feature quantity data representing the area of the target signal. It is a set of white rectangles in the upper left. On the other hand, the feature quantity vector of the signal area in which the width W of the signal area is relatively small and the distance Y from the upper end of the screen is relatively large is an area of a signal different from the target signal (for example, a signal existing far away). This is feature amount data representing (misrecognition region), and is a set of black circles on the lower right.

  Here, the determination unit 103 arranges the separation plane that separates the set of feature amount data (correct data) in the target signal region and the set of feature amount data in the erroneous recognition region so that the margin d is maximized. To do. Then, the determination unit 103 can previously calculate (calculate by learning) an evaluation function indicating this separation plane (the straight line L in the example of FIG. 7) as the target signal identification dictionary 18B. In this example, it has been described that the subject that calculates the evaluation function is the determination unit 103, but the present invention is not limited thereto, and the subject that calculates the evaluation function may be a function other than the determination unit 103.

The evaluation function is represented by a linear discriminant function such as the following formula (1), but is not limited to this, and may be another non-linear discriminant function. A, B, and C in the following formula (1) are coefficients of the evaluation function. Here, this evaluation function corresponds to “target signal identification information”.

  The determination unit 103 substitutes feature amount data (corresponding to the signal region recognized by the signal region recognition unit 102) indicating the recognition result by the signal region recognition unit 102 into the evaluation function, and evaluates the evaluation function f (W, Y). Is calculated. If the value of the calculated evaluation function f (W, Y) is equal to or greater than a predetermined threshold value, the determination unit 103 indicates that the feature amount data indicating the recognition result by the signal region recognition unit 102 represents the region of the target signal. Judge. On the other hand, if the value of the calculated evaluation function f (W, Y) is less than the threshold value, the determination unit 103 does not represent the feature amount data indicating the recognition result by the signal region recognition unit 102 as the region of the target signal. Judge.

In the above description, the case has been described in which the feature amount data indicating the recognition result that is the target of determining whether the answer is correct is a two-dimensional feature amount vector. However, the number of dimensions can be arbitrarily changed. is there. For example, in the case of 6 dimensions, an evaluation function f (Cr, Cy, Cg, W, X, Y) indicating a separation surface that separates a set of correct answer data and a set of feature amount data in a misrecognized region is expressed by the following equation: It is represented by (2).

  In this case, the feature amount data (Cr, Cy, Cg, W, X, Y) indicating the recognition result by the signal region recognition unit 102 is 6-dimensional vector data, and the determination unit 103 uses the feature amount data (Cr, Cy, Cg, W, X, Y) is substituted into the evaluation function, and the value of the evaluation function f (Cr, Cy, Cg, W, X, Y) is calculated. If the value of the calculated evaluation function f (Cr, Cy, Cg, W, X, Y) is equal to or greater than a predetermined threshold value, the determination unit 103 indicates feature amount data indicating a recognition result by the signal region recognition unit 102. Is determined to represent the region of the target signal. On the other hand, if the value of the calculated evaluation function f (Cr, Cy, Cg, W, X, Y) is less than the threshold value, the determination unit 103 uses the feature amount data indicating the recognition result by the signal region recognition unit 102 as the target It is determined that it does not represent a signal area.

  Here, an evaluation function is created by prior learning for each combination of time zone and country or region, and stored in the target signal identification dictionary DB 18B. Based on the evaluation function selected by the selection unit 106 (to be described later) and the feature amount data indicating the recognition result by the signal region recognition unit 102, the determination unit 103 indicates feature amount data indicating the recognition result by the signal region recognition unit 102. Determines whether it indicates the region of the target signal.

  The output unit 107 outputs, as a recognition result, a signal region corresponding to feature amount data (feature amount data indicating a recognition result by the signal region recognition unit 102) determined to be a region indicating the target signal.

  The time zone discriminating unit 104 discriminates the time zone when the captured image acquired by the image acquisition unit 101 was captured. The time zone discriminating unit 104 discriminates the shooting time zone of the captured image by analyzing the captured image.

  The shooting time zone is obtained by dividing one day (24 hours) into a plurality of time zones having different shooting environments. Different shooting environments mean different light intensities. For example, the shooting time zone is daytime or night. The shooting time zone may be daytime, nighttime, evening. The shooting time zone may be any time zone obtained by dividing one day into a plurality of time zones having different shooting environments, and is not limited to daytime, night, evening, or the like. Also, the shooting time period is not limited to two or three types, and may be four or more types. The shooting time zone may be set as appropriate according to the shooting environment of the shot image (for example, the season, country, region, northern hemisphere / southern hemisphere).

  In the present embodiment, as an example, a case where the shooting time zone indicates day or night will be described. The shooting time zone indicates daytime is a time zone in which the light intensity of the shooting environment is equal to or greater than a threshold value. The shooting time zone is night is a time zone in which the light intensity of the shooting environment is less than the threshold value. An arbitrary value may be set in advance as the threshold value of the light intensity. For example, the light intensity threshold starts to saturate in the region indicating the signal 30 </ b> A of the traffic light 30 included in the captured image P by the auto gain control function of the imaging device 12 that captured the captured image P to be processed ( Alternatively, the light intensity may be determined.

  The time zone discrimination unit 104 discriminates the shooting time zone of the shot image using the luminance of the shot image acquired by the image acquisition unit 101. Specifically, as shown in FIG. 8, the time zone determination unit 104 includes a first calculation unit 116H, a second calculation unit 116I, and a third calculation unit 116J.

  The first calculation unit 116H calculates the average luminance of the captured image. The first calculation unit 116H calculates an average value of the luminance values of the pixels included in the captured image. Thereby, the first calculation unit 116H calculates the average luminance of the captured image. When the shooting time zone is daytime, the average brightness of the shot image is high. On the other hand, when the shooting time zone is night, the average brightness of the shot image is lower than that of the daytime.

  The second calculation unit 116I divides the captured image into a plurality of blocks. For example, the second calculation unit 116I divides the captured image into m × n blocks. Note that m and n are integers of 1 or more, and at least one of m and n is an integer of 2 or more.

  Then, the second calculation unit 116I calculates the average luminance for each of the blocks. For example, the second calculation unit 116I calculates the average value of the luminance values of the pixels included in the block for each block. Thereby, the second calculation unit 116I calculates the average luminance for each of the blocks included in the captured image P.

  Furthermore, the second calculation unit 116I calculates the number of blocks whose average luminance in the captured image is equal to or less than the threshold as the number of low luminance blocks in the captured image. An arbitrary value may be set in advance as the threshold value of the average luminance used for calculating the number of low luminance blocks.

  Then, the time zone determination unit 104 determines the shooting time zone based on the feature amount including the average luminance of the captured image and the number of low luminance blocks of the captured image.

  Specifically, the time zone determination unit 104 uses, for example, the average luminance of the captured image and the number of low luminance blocks of the captured image as the feature amount at the time zone determination.

  In the present embodiment, the time zone determination unit 104 creates a time zone recognition dictionary 18A that is used in advance to determine the shooting time zone. That is, the time zone discrimination unit 104 creates the time zone recognition dictionary 18A in advance before the time zone discrimination process for the captured image.

  In the present embodiment, the time zone determination unit 104 creates a time zone recognition dictionary 18A in advance using a machine learning method using SVM (Support Vector Machine).

  Specifically, the time zone determination unit 104 uses a plurality of taken images taken in advance for each shooting time zone as the reference taken images.

  The reference photographed image is a photographed image that has been photographed in advance by the photographing device 12 prior to this detection process, separately from the photographed image used for detecting the area indicating the signal of the traffic light. First, the time zone determination unit 104 registers a group of sample points indicated by the average luminance and the number of low luminance blocks in the reference photographed image in a two-dimensional space determined by the average luminance and the number of low luminance blocks. FIG. 9 is a diagram illustrating an example of sample points indicated by the average luminance (Iav) and the number of low luminance blocks (Nblk) in the reference photographed image.

  In FIG. 9, a sample point group 40B is a group of sample points indicated by the average luminance and the number of low luminance blocks in the reference photographed image taken in the photographing time zone indicating night. The sample point group 40A is a group of sample points indicated by the average luminance and the number of low luminance blocks in the reference photographed image taken in the photographing time zone indicating the daytime.

  The time zone discriminating unit 104 uses a separation plane (in this case, a straight line La) to separate the day sample point group 40A and the night sample point group 40B from each of the day sample point group 40A and the night sample point group 40B. Are arranged such that a distance d (sometimes referred to as a margin) between the boundary lines (straight line La1 and straight line La2) is maximized. Then, the time zone determination unit 104 calculates an evaluation function indicating this separation plane (the straight line La in FIG. 9). The following expression (3) is an expression indicating an evaluation function indicating the straight line La.

  In the above equation (3), f (Iav, Nblk) is an evaluation function when the average luminance of the photographed image and the number of low-luminance blocks of the photographed image are used as the feature quantities at the time zone discrimination. In formula (3), A, B, and C are coefficients of the evaluation function. Here, the evaluation function f (Iav, Nblk) or the coefficients A, B, and C correspond to the time zone recognition dictionary 18A.

  The time zone discriminating unit 104 creates an evaluation function shown in Expression (3) in advance and stores it in the storage unit 18 in advance as a time zone recognition dictionary 18A.

  The time zone discriminating unit 104 uses the time zone recognition dictionary 18A (evaluation function shown in Expression (3)) stored in the storage unit 18 to discriminate the shooting time of the shot image when discriminating the shooting time zone of the shot image. Determine the band.

  Specifically, the time zone determination unit 104 calculates the average luminance (Iav) and the number of low luminance blocks (Nblk) of the captured image, which are calculated by the first calculation unit 116H and the second calculation unit 116I, using Equation (3). By substituting into, the value of the evaluation function f (Iav, Nblk) is calculated.

  Then, when the value of the calculated evaluation function f (Iav, Nblk) is equal to or greater than a predetermined threshold, the time zone determination unit 104 determines that the shooting time zone of the captured image indicates daytime. On the other hand, when the value of the calculated evaluation function f (Iav, Nblk) is less than the threshold, the time zone determination unit 104 determines that the shooting time zone of the captured image indicates night. This threshold value may be determined in advance according to the light intensity of the shooting environment in the shooting time zone to be determined.

  Note that the time zone determination unit 104 is based on the feature amount including the average luminance of the captured image, the number of low-luminance blocks of the captured image, and the variance value of the average luminance for each block in the captured image. The band may be determined.

  In this case, the third calculation unit 116J of the time zone determination unit 104 calculates a dispersion value of the average luminance for each block in the captured image. The third calculation unit 116J calculates a dispersion value of the average luminance for each block divided by the second calculation unit 116I. For example, the time zone determination unit 104 calculates the dispersion value (σ) using the following equation (4).

  In Expression (4), σ represents a dispersion value of average luminance for each block in the captured image. In Expression (4), N indicates the number of blocks included in the captured image (that is, a value of m × n). Ii indicates the average luminance of the i-th block. Iav indicates the average brightness of the entire captured image.

  In this case, the time zone determination unit 104 uses the average luminance of the captured image, the number of low luminance blocks of the captured image, and the variance value of the average luminance for each block in the captured image as the feature amount. What is necessary is just to produce 18A previously.

  That is, the time zone discriminating unit 104 uses a machine learning method using SVM (Support Vector Machine) as in the case of using the average luminance of the photographed image and the number of low-luminance blocks of the photographed image as feature quantities. Thus, the time zone recognition dictionary 18A may be created in advance. In this case, the time zone determination unit 104 determines a group of sample points indicated by the average luminance, the number of low luminance blocks, and the variance value in the reference photographed image as a three-dimensional space determined by the average luminance, the number of low luminance blocks, and the variance value. You can register for.

  Then, in the same manner as described above, the time zone determination unit 104 arranges a separation plane that separates the day sample point group and the night sample point group so that the margin is maximized. Then, the time zone determination unit 104 may calculate an evaluation function indicating this separation plane as the time zone recognition dictionary 18A. The following equation (5) is an evaluation function (time zone recognition dictionary 18A) in the case where the average brightness of the shot image P, the number of low brightness blocks of the shot image P, and the variance value are used as feature quantities at the time zone discrimination. is there.

  In Expression (5), f (Iav, Nblk, σ) is an evaluation function (time) when the average luminance of the captured image, the number of low-luminance blocks of the captured image, and the variance value are used as the feature quantities at the time zone determination. This is a band recognition dictionary 18A). In equation (5), A, B, C, and D are coefficients of the evaluation function.

  Thus, for example, the time zone determination unit 104 may create the evaluation function shown in Expression (5) in advance and store the evaluation function in the storage unit 18 in advance as the time zone recognition dictionary 18A.

  In this case, the time zone determination unit 104 uses the time zone recognition dictionary 18A (the evaluation function shown in Equation (5)) stored in the storage unit 18 to determine the shooting time zone of the shot image. P shooting time zone is determined.

  Specifically, the time zone determination unit 104 calculates the average luminance (Iav) and low luminance of the captured image calculated by each of the first calculation unit 116H, the second calculation unit 116I, and the third calculation unit 116J. The number of blocks (Nblk) and the variance value (σ) are obtained. Then, the time zone determination unit 104 substitutes the average luminance (Iav), the number of low-luminance blocks (Nblk), and the variance (σ) in the equation (5), thereby evaluating the evaluation function f (Iav, Nblk). , Σ) is calculated.

  Then, when the value of the calculated evaluation function f (Iav, Nblk, σ) is equal to or greater than a predetermined threshold, the time zone determination unit 104 determines that the shooting time zone of the captured image indicates daytime. On the other hand, when the value of the calculated evaluation function f (Iav, Nblk, σ) is less than the threshold, the time zone determination unit 104 determines that the shooting time zone of the captured image indicates night. This threshold may be determined in advance.

  As described above, the time zone determination unit 104 determines the time zone in which the captured image acquired by the image acquisition unit 101 was captured, and outputs the determined shooting time zone to the selection unit 106.

  Returning to FIG. 2, the description of the functions of the recognition processing unit 16 will be continued. The position information acquisition unit 105 acquires position information of the own device. For example, the position information acquisition unit 105 can acquire position information from an external device such as a GPS device. The position information acquisition unit 105 outputs the acquired position information to the selection unit 106.

  The selection unit 106 selects target signal identification information corresponding to the time zone determined by the time zone determination unit 104. Further, the selection unit 106 selects target signal identification information corresponding to the position information (which can specify the country or region) acquired by the position information acquisition unit 105. More specifically, the selection unit 106 is a target corresponding to a combination of the time zone determined by the time zone determination unit 104 and the location information (country or region can be specified) acquired by the location information acquisition unit 105. Select signal identification information.

  Next, a signal area recognition method by the signal area recognition unit 102 will be described. The signal area recognition unit 102 extracts a signal pixel area including a signal pixel indicating the signal color of the traffic light from the captured image acquired by the image acquisition unit 101, and expands the extracted signal pixel area into an expanded area in advance. When a fixed area having a predetermined shape is included, the area including the fixed area is recognized as a signal area. More specifically, when the expansion region includes a circular fixed region, the signal region recognition unit 102 recognizes a region including the fixed region as a signal region.

  The signal region recognition unit 102 includes a candidate region recognition unit 111, a signal shape recognition unit 112, and a signal region determination unit 113. The candidate area recognition unit 111 extracts (recognizes) a signal pixel area including a signal pixel indicating the signal color of the traffic light as a candidate area from the captured image acquired by the image acquisition unit 101. The signal shape recognition unit 112 recognizes whether or not a fixed region having a predetermined shape is included in the expanded region obtained by expanding the candidate region recognized by the candidate region recognition unit 111. The signal region determination unit 113 determines a signal region based on the recognition result by the candidate region recognition unit 111 and the recognition result by the signal shape recognition unit 112. Specific contents will be described below.

When the captured image acquired by the image acquisition unit 101 is image data in the (R, G, B) color space, the candidate area recognition unit 111 captures the captured image (input of the RGB color space acquired by the image acquisition unit 101) Image) is changed to image data in the (Y, U, V) color space, and signal pixels are extracted. Here, the conversion between the (R, G, B) color space and the (Y, U, V) color space is performed by the following equation (6).

In this example, a signal recognition dictionary that indicates a range of color values of a signal of a traffic light is preliminarily obtained by learning by collecting captured images (referred to as “signal image samples”) obtained by capturing a signal of the traffic light in advance by the photographing device 12. Create it. For example, a separation plane that separates a (U, V) value distribution indicating a red signal from a (U, V) value distribution other than the red signal is arranged so that a margin (distance) between them is maximized. Then, the evaluation function indicating the separation plane can be calculated as a red signal recognition dictionary for recognizing the red signal. The following equation (7) is an equation representing an evaluation function. In Expression (7), f (U, V) is an evaluation function indicating a red signal recognition dictionary, and a, b, and c are coefficients of this evaluation function. The green signal recognition dictionary and the yellow signal recognition dictionary can be created in the same manner.

  The candidate area recognizing unit 111 sets the (U, V) value of each pixel to each signal color (any one of red, yellow, and blue) for each pixel constituting the captured image in the (Y, U, V) color space. Is substituted into the evaluation function f (U, V) corresponding to, and the value of the evaluation function f (U, V) is obtained. If the value of the calculated evaluation function f (U, V) is equal to or greater than a predetermined threshold value, it is determined that the pixel is a signal pixel having a corresponding signal color. For example, a signal pixel may be extracted by providing a minimum value threshold and a maximum value threshold. You may set the threshold value of the minimum value and maximum value of U of each of red, blue, and yellow, and the minimum value and maximum value of V. Further, the threshold value is set so as not to overlap with the (U, V) values of the pixels other than the signal pixel.

  FIG. 10 is a diagram illustrating a pixel example of a red signal extracted by threshold processing in a (U, V) space of pixel values. Since the signal pixel area including the signal pixel (the pixel indicating the red signal) in the image of FIG. 10 includes the saturated pixel and the noise pixel, the extracted signal pixel area is a part of the actual area indicating the red signal. There is only. Therefore, the candidate area recognition unit 111 performs an expansion process on the extracted signal pixels. In the dilation process, N × N pixel blocks are added from the original signal image to one pixel. For example, when N = 7, a 7 × 7 pixel block is added from the original signal image to one pixel to be expanded. FIG. 11 is a diagram illustrating an example of an image after the signal pixel area of FIG. 10 is expanded. Hereinafter, the signal pixel area after the expansion process is referred to as an “expansion area”.

  The signal shape recognition unit 112 performs signal shape recognition processing on the expansion region. Shape recognition is performed by circular extraction for red, blue and yellow signals. Here, shape recognition is performed on the expansion region by extracting a circle. The circle extraction process can perform circle extraction by Hough conversion. When a circle is extracted by the signal shape recognition unit 112, the signal region determination unit 113 obtains a rectangle that circumscribes the extracted circle region. Then, the signal area determination unit 113 determines this rectangular area as a signal area, and calculates a feature amount (feature amount data indicating a recognition result) corresponding to the signal region. A circle drawn by a thick line in FIG. 12 indicates a red signal circle region extracted by the Hough transform. A rectangular area shown in FIG. 13 indicates a signal area determined by the signal area determination unit 113. The rectangular area shown in FIG. 5 is a diagram showing a candidate area (signal area indicating a red signal) recognized from the captured image of FIG.

  FIG. 14 is a flowchart illustrating an operation example of the recognition processing unit 16 of the present embodiment. Since the specific contents of each step are as described above, detailed description will be omitted as appropriate. As shown in FIG. 14, first, the image acquisition unit 101 acquires a captured image (step S1). Next, the signal area recognizing unit 102 recognizes a signal area indicating a signal of a traffic light among the captured images acquired in step S1 (step S2). In parallel with the above processing, the time zone determination unit 104 determines the time zone in which the captured image acquired in step S1 was captured (step S3).

  Next, the selection unit 106 selects an evaluation function corresponding to the combination of the time period determined in step S3 and the country or region specified by the position information acquired by the position information acquisition unit 105 (step S4). ). Next, based on the signal region (feature data) recognized in step S2 and the evaluation function selected in step S4, the determination unit 103 determines whether the signal region is a region indicating the target signal. Judgment processing for judging whether or not is performed (step S5). The output unit 107 outputs, as a recognition result, the signal area determined to be the area indicating the target signal by the determination process in step S5 (step S6).

  As described above, in the present embodiment, target signal identification information predetermined for identifying a target signal indicating a signal of a traffic signal to be detected, a signal region recognized by the signal region recognition unit 102, and Based on, it is determined whether or not the signal region is a region indicating a target signal. According to the present embodiment, it is possible to recognize a signal of a traffic signal reflected in a captured image with high accuracy.

  Next, an example of the hardware configuration of the imaging device 12 will be described. FIG. 15 is a diagram illustrating an example of a hardware configuration of the imaging device 12.

  The photographing apparatus 12 includes a photographing optical system 2010, a mechanical shutter 2020, a motor driver 2030, a CCD (Charge Coupled Device) 2040, a CDS (Correlated Double Sampling) circuit 2050, an A / D converter 2060, and a timing signal generator. 2070, image processing circuit 2080, LCD (Liquid Crystal Display) 2090, CPU (Central Processing Memory) 2100, RAM (Random Access Memory) 2110, ROM (Read Only Memory 2c) Expansion circuit 2 40 comprises a memory 2150, an operation unit 2160, and an output I / F2170.

  The image processing circuit 2080, CPU 2100, RAM 2110, ROM 2120, SDRAM 2130, compression / decompression circuit 2140, memory 2150, operation unit 2160, and output I / F 2170 are connected via a bus 2200.

  The photographing optical system 2010 condenses the reflected light of the subject. The mechanical shutter 2020 is opened for a predetermined time so that the light condensed by the photographing optical system 2010 is incident on the CCD 2040. The motor driver 2030 drives the photographing optical system 2010 and the mechanical shutter 2020.

  The CCD 2040 forms the light incident through the mechanical shutter 2020 as a subject image, and inputs analog image data representing the subject image to the CDS circuit 2050.

  Upon receiving analog image data from the CCD 2040, the CDS circuit 2050 removes noise components from the image data. The CDS circuit 2050 performs analog processing such as correlated double sampling and gain control. Then, the CDS circuit 2050 outputs the processed analog image data to the A / D converter 2060.

  When receiving analog image data from the CDS circuit 2050, the A / D converter 2060 converts the analog image data into digital image data. The A / D converter 2060 inputs digital image data to the image processing circuit 2080. The timing signal generator 2070 transmits a timing signal to the CCD 2040, the CDS circuit 2050, and the A / D converter 2060 in accordance with a control signal from the CPU 2100, whereby the CCD 2040, the CDS circuit 2050, and the A / D converter 2060. Control the operation timing.

  Upon receiving digital image data from the A / D converter 2060, the image processing circuit 2080 uses the SDRAM 2130 to perform image processing on the digital image data. The image processing includes, for example, CrCb conversion processing, white balance control processing, contrast correction processing, edge enhancement processing, and color conversion processing.

  The image processing circuit 2080 outputs the image data subjected to the above-described image processing to the LCD 2090 or the compression / decompression circuit 2140. The LCD 2090 is a liquid crystal display that displays image data received from the image processing circuit 2080.

  When receiving the image data from the image processing circuit 2080, the compression / decompression circuit 2140 compresses the image data. The compression / decompression circuit 2140 stores the compressed image data in the memory 2150. When the compression / decompression circuit 2140 receives image data from the memory 2150, the compression / decompression circuit 2140 decompresses the image data. The compression / decompression circuit 2140 temporarily stores the decompressed image data in the SDRAM 2130. The memory 2150 stores the compressed image data.

  The output I / F 2170 outputs the image data processed by the image processing circuit 2080 to the information processing apparatus 10 as a captured image P.

  Note that at least a part of the functional unit included in the interface unit 14 and the recognition processing unit 16 described in FIG. 2 may be mounted on the photographing apparatus 12 as a signal processing board (signal processing circuit).

  Next, the hardware configuration of the information processing apparatus 10 according to the above embodiment will be described. FIG. 16 is a block diagram illustrating a hardware configuration example of the information processing apparatus 10 according to the embodiment.

  In the information processing apparatus 10 according to the above-described embodiment, the output unit 80, the I / F unit 82, the input unit 94, the CPU 86, the ROM (Read Only Memory) 88, the RAM (Random Access Memory) 90, the HDD 92, etc. It is connected to and has a hardware configuration using a normal computer.

  The CPU 86 is an arithmetic device that controls processing executed by the information processing apparatus 10 according to the embodiment. The RAM 90 stores data necessary for various processes by the CPU 86. The ROM 88 stores a program for realizing various processes by the CPU 86 and the like. The HDD 92 stores data stored in the storage unit 18 described above. The I / F unit 82 is an interface for transmitting and receiving data to and from other devices.

  A program for executing the various processes executed by the information processing apparatus 10 of the embodiment is provided by being incorporated in advance in the ROM 88 or the like.

  Note that the program executed by the information processing apparatus 10 of the above embodiment is a file in a format that can be installed or executed in these apparatuses, and is a CD-ROM, flexible disk (FD), CD-R, DVD (Digital It may be configured to be recorded on a computer-readable recording medium such as Versatile Disk).

  The program executed by the information processing apparatus 10 of the above embodiment may be configured to be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. In addition, a program for executing each of the above processes in the information processing apparatus 10 of the above embodiment may be configured to be provided or distributed via a network such as the Internet.

  In the program for executing the various processes executed by the information processing apparatus 10 of the embodiment, the above-described units are generated on the main storage device.

  Various information stored in the HDD 92 may be stored in an external device. In this case, the external device and the CPU 86 may be connected via a network or the like.

DESCRIPTION OF SYMBOLS 10 Information processing apparatus 14 Interface part 16 Recognition processing part 18 Storage part 101 Image acquisition part 102 Signal area recognition part 103 Judgment part 104 Time zone discrimination part 105 Position information acquisition part 106 Selection part 107 Output part 111 Candidate area recognition part 112 Signal shape Recognizing unit 113 Signal region determining unit

JP 2009-244946 A

Claims (9)

An acquisition unit for acquiring a captured image;
A signal area recognition unit for recognizing a signal area indicating a signal of a traffic light among the captured images acquired by the acquisition unit;
Based on the target signal identification information predetermined for identifying the target signal indicating the signal of the traffic signal to be detected, and the signal region recognized by the signal region recognition unit, the signal region is the A determination unit that determines whether or not the region indicates a target signal,
Information processing device. The target signal identification information is created in advance using a machine learning method using SVM (Support Vector Machine).
The information processing apparatus according to claim 1. The target signal identification information is an evaluation function indicating a separation surface between a target signal region of the captured image and a misrecognition region indicating a signal region different from the target signal.
The information processing apparatus according to claim 2. The target signal identification information is predetermined for each of a plurality of time zones,
A time zone discriminating unit for discriminating a time zone in which the captured image was shot;
A selection unit that selects the target signal identification information corresponding to the time zone determined by the time zone determination unit;
The information processing apparatus according to any one of claims 1 to 3. The target signal identification information is predetermined for each country or region,
A location information acquisition unit for acquiring location information;
A selection unit that selects the target signal identification information corresponding to the position information acquired by the position information acquisition unit,
The information processing apparatus according to any one of claims 1 to 4. The signal area recognition unit
When a signal pixel region including a signal pixel indicating a signal color of a traffic light is extracted from the photographed image, and a fixed region having a predetermined shape is included in an expanded region obtained by expanding the extracted signal pixel region, the fixed image Recognizing a region including a region as the signal region;
The information processing apparatus according to any one of claims 1 to 5. When the circular region is included in the expansion region, the signal region recognition unit recognizes a region including the fixed region as the signal region.
The information processing apparatus according to claim 6. An acquisition step of acquiring a captured image;
A signal area recognition step for recognizing a signal area indicating a signal of a traffic light among the captured images acquired by the acquisition step;
Based on the target signal identification information predetermined for identifying the target signal indicating the signal of the traffic signal to be detected and the signal region recognized by the signal region recognition step, the signal region is A determination step of determining whether or not the region indicates a target signal,
Information processing method. On the computer,
An acquisition step of acquiring a captured image;
A signal area recognition step for recognizing a signal area indicating a signal of a traffic light among the captured images acquired by the acquisition step;
Based on the target signal identification information predetermined for identifying the target signal indicating the signal of the traffic signal to be detected and the signal region recognized by the signal region recognition step, the signal region is A determination step of determining whether or not the region indicates a target signal.

Images