JP6874315B2 - Information processing equipment, information processing methods and programs - Google Patents

Description

The present invention relates to an information processing device, an information processing method and a program.

A system that assists the driver by analyzing a captured image taken by a photographing device such as an in-vehicle camera and detecting a signal of a traffic light (typically a lit signal) included in the captured image is known. ing. Further, in order to recognize the signal of the traffic light, a device for detecting the signal color of the traffic light is known.

For example, Patent Document 1 discloses a method in which the gain of a camera is adjusted twice to acquire two captured images, and the signal color is detected using these two captured images. In Patent Document 1, the gain for the second shooting is adjusted by using the shot image obtained in the first shooting.

However, in the conventional technique, a plurality of captured images having different gains are required. In addition, the detection accuracy may be significantly reduced due to an error in gain adjustment. Further, in the state before detecting the region of the captured image in which the traffic light is reflected, the position of the signal in the captured image is unknown, so that it is difficult to set the gain that can accurately detect the signal color.

Further, even if the camera gain is optimally set, if an object having the same color and shape as the signal of the traffic light appears, there is a problem that the object is erroneously recognized as a signal of the traffic light. For example, if a part of the red signboard is hidden by a tree and the visible part is in the shape of a circle indicating the color of the red traffic light, the red signboard is recognized as an area indicating the red traffic light. With conventional technology, false recognition cannot be deleted.

That is, with the conventional technique, there is a problem that it is difficult to recognize the signal of the traffic light reflected in the captured image with high accuracy.

The present invention has been made in view of the above, and an object of the present invention is to provide an information processing device, an information processing method, and a program capable of recognizing a signal of a traffic light reflected in a captured image with high accuracy. To do.

In order to solve the above-mentioned problems and achieve the object, the present invention corresponds to the color and shape of the signal of the signal among the acquisition unit that acquires the captured image and the captured image acquired by the acquisition unit. A candidate area recognition unit that recognizes a candidate area indicating an area, a peripheral feature amount data calculation unit that calculates ambient feature amount data indicating a feature amount around the candidate area, and a periphery indicating an area around the signal of the signal. In order to identify the area, the peripheral area identification information predetermined for each country or region and the peripheral feature amount data calculated by the peripheral feature amount data calculation unit are added to the surrounding feature amount data. Corresponds to the determination unit for determining whether or not the corresponding candidate area is the area indicating the signal of the signal, the position information acquisition unit for acquiring the position information, and the position information acquired by the position information acquisition unit. It is an information processing apparatus including a selection unit for selecting the surrounding area identification information.

According to the present invention, the signal of the traffic light reflected in the captured image can be recognized with high accuracy.

FIG. 1 is an explanatory diagram of an example of an information processing device. FIG. 2 is a diagram showing an example of a function of the information processing apparatus of the embodiment. FIG. 3 is a diagram showing an example in which the green light of the traffic light arranged in the horizontal direction is lit and the area corresponding to the green light is recognized as the candidate area. FIG. 4 is a diagram showing an example in which the red light of the traffic light arranged in the horizontal direction is lit and the area corresponding to the red light is recognized as the candidate area. FIG. 5 is a diagram showing an example in which the red light of the traffic light arranged in the vertical direction is lit and the area corresponding to the red light is recognized as the candidate area. FIG. 6 is a diagram showing an example in which the green light of the traffic light arranged in the vertical direction is lit and the area corresponding to the green light is recognized as the candidate area. FIG. 7 is an explanatory diagram of a method of calculating ambient feature amount data. FIG. 8 is an explanatory diagram of a method of calculating ambient feature amount data. FIG. 9 is an explanatory diagram of a method of calculating ambient feature amount data. FIG. 10 is an explanatory diagram of a method of calculating ambient feature amount data. FIG. 11 is an explanatory diagram of another calculation method of ambient feature amount data. FIG. 12 is a diagram showing an example of the distribution of pixel average values in any one time zone. FIG. 13 is a diagram showing an example of the function of the time zone determination unit. FIG. 14 is a diagram showing an example of sample points indicated by the average luminance and the number of low-luminance blocks in the reference captured image. FIG. 15 is a diagram showing a pixel example of a green signal. FIG. 16 is a diagram showing a pixel example after the signal pixel region of FIG. 15 has been expanded. FIG. 17 is a diagram showing a circular region of a green light extracted by the Hough transform. FIG. 18 is a diagram showing a candidate region recognized by the candidate region recognition unit. FIG. 19 is a diagram showing a candidate region (a region showing a green light candidate) recognized from the captured image of FIG. FIG. 20 is a flowchart showing an operation example of the recognition processing unit. FIG. 21 is a diagram showing an example of the hardware configuration of the photographing apparatus. FIG. 22 is a diagram showing a hardware configuration example of the information processing device.

Hereinafter, embodiments of the information processing apparatus, information processing method, and program according to the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is an explanatory diagram of an example of the information processing device 10 of the present embodiment. The information processing device 10 is mounted on a mobile body, for example. A moving body is an object whose position can be moved in real space by self-driving or towing. The moving body is, for example, a vehicle, an airplane, a train, a bogie, or the like. In the present embodiment, the case where the moving body is the vehicle 20 will be described as an example. That is, in the present embodiment, a mode in which the information processing device 10 is mounted on the vehicle 20 will be described as an example.

The vehicle 20 is equipped with a photographing device 12. The photographing device 12 obtains a photographed image of the periphery of the vehicle 20. The photographing device 12 is, for example, a known video camera, a digital camera, or the like. In the present embodiment, the photographing device 12 continuously photographs the periphery of the vehicle 20 to capture a plurality of captured images (that is, a plurality of frames). The photographing device 12 may be integrally configured with the information processing device 10 or may be configured as a separate body from the information processing device 10.

Further, the photographing device 12 is not limited to the form mounted on the vehicle 20. The photographing device 12 may be installed at a position where the traffic light 30 can be photographed, and may be fixed to the ground. When the detection result detected by the information processing device 10 of the present embodiment is used for driving support of the driver of the vehicle 20, the photographing device 12 is preferably mounted on the vehicle 20.

In the present embodiment, the photographing device 12 is equipped with an automatic gain control (AGC) function. Therefore, the photographing device 12 automatically adjusts the sensitivity and obtains a photographed image automatically adjusted so that the brightness of the entire screen of the photographed image is optimized.

The information processing device 10 analyzes the captured image. Then, the information processing device 10 detects the traffic light 30 included in the captured image.

Next, an example of the function of the information processing device 10 will be described. FIG. 2 is a block diagram showing an example of the functions of the information processing device 10. For convenience of explanation, the functions related to the present embodiment are mainly illustrated here, but the functions of the information processing apparatus 10 are not limited to these.

The information processing device 10 includes an interface unit 14, a recognition processing unit 16, and a storage unit 18. The interface unit 14 and the storage unit 18 are electrically connected to the recognition processing unit 16.

The interface unit 14 receives a photographed image from the photographing device 12. The photographing device 12 continuously photographs the periphery of the vehicle 20 over time, and outputs each of the captured images (color images in this example) obtained by the photographing to the interface unit 14 in the order of photographing. The interface unit 14 sequentially receives captured images from the photographing device 12, and sequentially outputs them to the recognition processing unit 16 in the order of acceptance.

The storage unit 18 stores various data. In the present embodiment, the storage unit 18 has a time zone recognition dictionary DB 18A and a peripheral area identification dictionary DB 18B, the time zone recognition dictionary DB 18A holds a time zone recognition dictionary, and the peripheral area identification dictionary DB 18B identifies the surrounding area. Keep a dictionary. Details of these will be described later.

The recognition processing unit 16 analyzes the captured image and recognizes (detects) the signal of the traffic light reflected in the captured image. As shown in FIG. 2, the recognition processing unit 16 includes an image acquisition unit 101, a candidate area recognition unit 102, a peripheral feature amount data calculation unit 103, a determination unit 104, a deletion processing unit 105, and a time zone determination unit. It includes 106, a position information acquisition unit 107, a selection unit 108, and an output unit 109.

Image acquisition unit 101, candidate area recognition unit 102, surrounding feature amount data calculation unit 103, judgment unit 104, deletion processing unit 105, time zone determination unit 106, position information acquisition unit 107, selection unit 108, part of output unit 109 Alternatively, all of them may be realized by, for example, a processing device such as a CPU (Central Processing Unit) executing a program, that is, by software, or by hardware such as an IC (Integrated Circuit). , Software and hardware may be used together.

The image acquisition unit 101 is an example of the “acquisition unit” and acquires a captured image. In this example, the image acquisition unit 101 acquires captured images (moving images) sequentially input from the interface unit 14. That is, the image acquisition unit 101 acquires a photographed image photographed by the photographing device 12 mounted on the vehicle 20.

The candidate area recognition unit 102 recognizes a candidate area indicating an area corresponding to the color and shape of the signal of the traffic light among the captured images acquired by the image acquisition unit 101. The candidate area recognition unit 102 recognizes a region (rectangular region in this example) corresponding to the color and shape of the lit signal (red signal or green signal) as a candidate region.

FIG. 3 is a diagram showing an example in which the green light of the traffic light arranged in the horizontal direction is lit and the area 301 corresponding to the green light is recognized as a candidate area. FIG. 4 is a diagram showing an example in which the red light of the traffic light arranged in the horizontal direction is lit and the area 401 corresponding to the red light is recognized as a candidate area. In Japan and the like, traffic lights are generally arranged in the horizontal direction as shown in FIGS. 3 and 4.

FIG. 5 is a diagram showing an example in which the red light of the traffic light arranged in the vertical direction is lit and the area 501 corresponding to the red light is recognized as a candidate area. FIG. 6 is a diagram showing an example in which the green light of the traffic light arranged in the vertical direction is lit and the area 601 corresponding to the green light is recognized as a candidate area. In the United States and the Tohoku region of Japan, traffic lights are generally arranged in the vertical direction as shown in FIGS. 5 and 6. The specific recognition method of the candidate area will be described later.

The explanation will be continued by returning to FIG. The peripheral feature amount data calculation unit 103 calculates the peripheral feature amount data indicating the peripheral feature amount of the candidate area recognized by the candidate area recognition unit 102 among the captured images acquired by the image acquisition unit 101. A method of calculating ambient feature data will be described with reference to FIGS. 7 to 10. The rectangular regions of regions 801 (a) to 801 (f) represent regions around the candidate regions. When they are not distinguished from each other, they may be simply referred to as "region 801". Here, in any of FIGS. 7 to 10, the candidate region corresponds to the region showing the signal of the traffic light, and the regions 801 (a) to 801 (f) are the peripheral region showing the region around the signal of the traffic light. Although they match, not limited to this, if the candidate area and the area indicating the signal of the traffic light do not match, the area around the candidate area and the surrounding area do not match.

The size of the rectangle in the area 801 is the same as the size of the rectangle in the candidate area. Surrounding feature data calculating unit 103 calculates an average value _{I i} of the pixels in each region 801. More specifically, by summing the brightness values (pixel values) of the pixels in the area 801 and dividing by the number of pixels (M × N) in the rectangle, the average value I (the following) of the pixels in the area 801 is obtained. In the description, it may be referred to as "pixel average value I"). In this example, M is the number of pixels in the horizontal direction of the rectangle, and N is the number of pixels in the vertical direction of the rectangle. The peripheral feature amount data calculation unit 103 obtains the pixel average value I of each region 801 (a) to 801 (f) to obtain the peripheral feature amount data. Here, the ambient feature data is K (K corresponds to the number of regions 801) dimensional vector data, and is expressed as _{P (I a} , I _b , I _c , ..., I _k). I _k is the pixel average value I in each region 801. I _a represents the pixel average value I of the region 801 (a), and I _b represents the pixel average value I of the region 801 (b). The same is true for others. In the examples of FIGS. 7 to 10, the ambient feature data P is a six-dimensional vector data (P (I _a , I _b , I _c , ..., _If )). Further, the number of dimensions of the vector data representing the ambient feature data P is arbitrary. For example, as shown in FIG. 11, 14 regions 801 (a) to 801 (n) are provided, the pixel average value I of each region 801 is obtained, and the surrounding feature amount data P is represented by 14-dimensional vector data. It may be.

The description of FIG. 2 will be continued. The determination unit 104 uses the peripheral area identification information predetermined for identifying the peripheral area indicating the area around the signal of the traffic light (surrounding within a certain range) in the captured image, and the peripheral feature amount data calculation unit 103. Based on the calculated ambient feature data, it is determined whether or not the candidate region corresponding to the ambient feature data is the region indicating the signal of the traffic light. In this example, the surrounding area identification information is predetermined for each of a plurality of time zones. In addition, the surrounding area identification information is predetermined for each country or region. More specifically, the surrounding area identification information is predetermined for each combination of the time zone and the country or region. The surrounding area identification information is created in advance by using a machine learning method using SVM (Support Vector Machine). Furthermore, the peripheral area identification information is an evaluation function indicating a separation surface between the peripheral area and the erroneous recognition area indicating an area other than the peripheral area in the captured image.

For example, in a country or region where traffic lights are arranged in the horizontal direction as shown in FIG. 8 (referred to as "target region (concept including country)" in the following description), two-dimensional ambient feature data P is used. It is assumed that the corresponding candidate area is judged. That is, the surrounding area identification information in this case identifies whether or not the two-dimensional peripheral feature amount data P is the correct answer (whether or not it indicates the surrounding area within a certain range of the signal of the traffic light in the captured image). Information for. Here, for convenience of explanation, the two-dimensional ambient feature data P will be described as an example, but the present invention is not limited to this, and the number of dimensions of the ambient feature data P to be determined whether or not the answer is correct is not limited to this. Is optional.

Here, the two-dimensional vector data calculated by the ambient feature data calculation unit 103 is the area to the right of the area (area 401 in this example) showing the red signal of the traffic light shown in FIG. 8 (area 801 in this example). (A)) and the surrounding area identification information for discriminating whether or not to indicate each of the left adjacent area (in this example, the area 801 (d)) will be described as an example.

When creating such peripheral area identification information, separately from the captured image used for detecting the region showing a red light, the photographing device 12 is used in advance for each of a plurality of time zones in the target area prior to this detection process. Collect images (called red light images) of red light traffic lights. Then, for each of the plurality of time zones, among the red signal images, the pixel average of the rectangular size area to the right of the rectangular size (same as the rectangular size of the candidate area) area (red pixel area) composed of a plurality of red pixels. The value I _{a and the pixel average value I d} of the rectangular size area to the left are obtained. FIG. 12 is a diagram showing an example of the distribution of pixel average values (I _a , I _{d) in any one time zone. When the red pixel region is a region showing a red signal of a traffic light, a relatively high pixel average value Ia} can be obtained in the region to the right of the red pixel region (for example, region 801 (a) in FIG. 8). .. On the other hand, since the region to the left of the red pixel region (for example, region 801 (d) in FIG. 8) is a non-lit yellow region, a relatively low pixel average value _Id can be obtained. _{That is, in this case, the set of feature data P (I a} , I _d ) in the surrounding area (the area around the red light), which is the correct answer data, is represented as the set of black circles in the lower right of FIG. , The set of feature data P (I _a , I _d ) of the misrecognition area other than the surrounding area is represented by the set of white circles on the upper left of FIG.

Here, the determination unit 104 arranges a separation surface that separates the set of feature amount data (correct answer data) in the surrounding area and the set of feature amount data in the erroneous recognition area so that the margin d is maximized. Then, the determination unit 104 can pre-calculate (calculate by learning) the evaluation function indicating this separation surface (straight line L in the example of FIG. 12) as the surrounding area identification dictionary DB18B. In this example, the subject that calculates the evaluation function is the determination unit 104, but the present invention is not limited to this, and the subject that calculates the evaluation function in advance may be a function other than the determination unit 104.

The evaluation function is represented by a linear discriminant function such as the following equation (1). A, B, and C in the following equation (1) are the coefficients of the evaluation function. Here, this evaluation function corresponds to "surrounding area identification information". Similarly, an evaluation function for identifying the peripheral area indicating the area around the yellow light and an evaluation function for identifying the peripheral area indicating the area around the green light can be created in advance.

_{The determination unit 104 substitutes the peripheral feature amount data P (I a} , I _d ) calculated by the peripheral feature amount data calculation unit 103 into the evaluation function, and sets the value of the _{evaluation function f (I a} , I _d). calculate. Then, if the value of the calculated evaluation function f (I _a , I _d ) is equal to or higher than a predetermined threshold value (for example, 0), the determination unit 104 corresponds to the peripheral feature amount data P (I _a , I _d ). It is determined that the candidate region to be used is a region showing a red light. This is because it can be determined that the peripheral feature amount data P (I _a , _Id ) is data indicating the feature amount of the area (surrounding area) around the red light in the captured image. On the other hand, if the value of the calculated evaluation function f (I _a , I _d ) is less than the threshold value, the determination unit 104 indicates that the candidate region corresponding to the _{surrounding feature amount data P (I a} , I _{d) is a red signal.} It is judged that it is not the area indicating. This is because it can be determined that the peripheral feature amount data P (I _a , I _d ) is data indicating the feature amount of the erroneous recognition region in the captured image.

In the above, the case where the ambient feature amount data P to be determined whether or not the answer is correct is two-dimensional vector data has been described as an example, but the number of dimensions can be arbitrarily changed. _{When the number of dimensions is K, the evaluation function f (I a} , I _b , ..., I _k ) indicating the separation surface that separates the set of correct answer data and the set of feature data of the misrecognition region is as follows. It is represented by the equation (2). Following in formula _{(2) (w a, w} b, ..., w k) is a coefficient of the evaluation function.

In this case, the peripheral feature data P (I _a , I _b , ..., I _k ) calculated by the peripheral feature data calculation unit 103 is K-dimensional vector data, and the determination unit 104 determines the peripheral feature amount. By substituting the ambient feature data P (I _a , I _b , ..., I _k ) calculated by the data calculation unit 103 into the above evaluation function, the evaluation function f (I _a , I _b , ..., I k) is substituted. Calculate the value of I _k). Then, if the value of the calculated evaluation function f (I _a , I _b , ..., I _k ) is equal to or greater than a predetermined threshold value, the determination unit 104 determines that the candidate region corresponding to the surrounding feature amount data is a candidate region. It is judged to be the area showing the signal of the traffic light. On the other hand, if the value of the calculated evaluation function (I _a , I _b , ..., I _k ) is less than the threshold value, the determination unit 104 determines that the candidate region corresponding to the surrounding feature amount data is the signal of the traffic light. Judge that it is not the area shown.

Here, an evaluation function is created by pre-learning for each combination of the time zone and the country or region, and stored in the surrounding area identification dictionary DB18B. An individual evaluation function may be stored for each of the red signal, the yellow signal, and the green signal, or only the evaluation function for the signal to be detected may be stored. For example, if all of the red light, yellow light, and green light are to be detected, an evaluation function for red light, an evaluation function for yellow light, and an evaluation function for green light are created in advance for each combination with a country or region. However, it may be stored in the peripheral area identification dictionary DB 18B. Further, for example, if only the red light is the detection target, an evaluation function for the red light may be created for each combination with the country or region and stored in the surrounding area identification dictionary DB 18B. In short, the peripheral area identification information predetermined for identifying the surrounding area indicating the area around the signal of the traffic light may be in a form predetermined for each combination of the time zone and the country or region. ..

The determination unit 104 is a candidate area corresponding to the ambient feature data P based on the evaluation function selected by the selection unit 108 described later and the ambient feature data P calculated by the ambient feature data calculation unit 103. It is determined whether or not is the area indicating the signal of the signal.

When the deletion processing unit 105 determines by the determination unit 104 that the candidate area corresponding to the ambient feature amount data P calculated by the ambient feature amount data calculation unit 103 is not the area indicating the signal of the traffic light, the candidate area is the candidate. Performs the process of deleting the area (excluding the output target).

The output unit 109 outputs the candidate area determined to be the area indicating the signal of the traffic light as the recognition result.

The time zone determination unit 106 determines the time zone in which the captured image acquired by the image acquisition unit 101 was captured. The time zone determination unit 16B determines the shooting time zone of the captured image by analyzing the captured image.

The shooting time zone is a day (24 hours) divided into a plurality of time zones different from each other in the shooting environment. Different shooting environments mean different light intensities. For example, the shooting time zone is daytime or nighttime. The shooting time zone may be daytime, nighttime, or evening. Further, the shooting time zone may be any time zone in which the day is divided into a plurality of time zones having different shooting environments, and is not limited to day, night, evening, and the like. The shooting time zone is not limited to two or three types, and may be four or more types. Further, the shooting time zone may be appropriately set according to the shooting environment of the shot image (for example, season, country, region, northern hemisphere / southern hemisphere).

In the present embodiment, as an example, a case where the shooting time zone indicates daytime or nighttime will be described. The shooting time zone indicates daytime is a time zone in which the light intensity of the shooting environment is equal to or higher than the threshold value. The shooting time zone is night is a time zone in which the light intensity of the shooting environment is less than the threshold value. An arbitrary value may be set in advance for the threshold value of this light intensity. For example, at this light intensity threshold value, the amount of light in the region indicating the signal 30A of the signal device 30 included in the captured image begins to saturate (or is saturated) by the auto gain control function of the photographing device 12 that captured the captured image to be processed. Is released) The light intensity may be determined.

The time zone determination unit 106 determines the shooting time zone of the captured image by using the brightness of the captured image acquired by the image acquisition unit 101. Specifically, as shown in FIG. 13, the time zone determination unit 106 includes a first calculation unit 116H, a second calculation unit 116I, and a third calculation unit 116J.

The first calculation unit 116H calculates the average brightness of the captured image. The first calculation unit 116H calculates the average value of the brightness values of each of the pixels included in the captured image. As a result, the first calculation unit 116H calculates the average brightness of the captured image. When the shooting time is daytime, the average brightness of the shot image is high. On the other hand, when the shooting time zone is night, the average brightness of the shot image is lower than that of daytime.

The second calculation unit 116I divides the captured image into a plurality of blocks. For example, the second calculation unit 116I divides the captured image into m × n blocks. Note that m and n are integers of 1 or more, and at least one of m and n is an integer of 2 or more.

Then, the second calculation unit 116I calculates the average brightness for each of the blocks. For example, the second calculation unit 116I calculates the average value of the brightness values of the pixels included in the block for each block. As a result, the second calculation unit 116I calculates the average brightness for each of the blocks included in the captured image.

Further, the second calculation unit 116I calculates the number of blocks whose average brightness in the captured image is equal to or less than the threshold value as the number of low-luminance blocks in the captured image. An arbitrary value may be predetermined for the threshold value of the average luminance used for calculating the number of low-luminance blocks.

Then, the time zone determination unit 106 determines the shooting time zone based on the feature amount including the average brightness of the captured image and the number of low-luminance blocks of the captured image.

Specifically, the time zone determination unit 106 uses, for example, the average brightness of the captured image and the number of low-luminance blocks of the captured image as feature quantities at the time of time zone determination.

In the present embodiment, the time zone determination unit 106 creates in advance the time zone recognition dictionary DB 18A used for determining the shooting time zone. That is, the time zone determination unit 106 creates the time zone recognition dictionary DB 18A in advance before the time zone determination process of the captured image.

In the present embodiment, the time zone determination unit 106 creates a time zone recognition dictionary 18A in advance by using a machine learning method using an SVM (Support Vector Machine).

Specifically, the time zone determination unit 106 uses a plurality of captured images captured in advance for each shooting time zone as reference captured images.

The reference captured image is a captured image previously captured by the imaging device 12 prior to the detection process, in addition to the captured image used for detecting the region showing the signal of the traffic light. First, the time zone determination unit 106 registers a group of sample points indicated by the average luminance and the number of low-luminance blocks in the reference captured image in a two-dimensional space determined by the average luminance and the number of low-luminance blocks. FIG. 14 is a diagram showing an example of sample points indicated by the average luminance (Iav) and the number of low luminance blocks (Nblk) in the reference captured image.

In FIG. 14, the sample point group 40B is a group of sample points indicated by the average brightness and the number of low-luminance blocks in the reference shooting image taken during the shooting time zone indicating night. Further, the sample point group 40A is a group of sample points indicated by the average brightness and the number of low-luminance blocks in the reference photographed image taken in the shooting time zone indicating daytime.

The time zone discriminating unit 106 provides a separation surface (here, a straight line La) that separates the daytime sample point group 40A and the nighttime sample point group 40B, and each of the daytime sample point group 40A and the nighttime sample point group 40B. The distance d (sometimes referred to as a margin) between the boundary lines (straight line La1 and straight line La2) is maximized. Then, the time zone determination unit 106 calculates an evaluation function indicating this separation surface (straight line La in FIG. 14). The following equation (3) is an equation showing an evaluation function showing this straight line La.

In the above equation (3), f (Iav, Nblk) is an evaluation function when the average brightness of the photographed image and the number of low-luminance blocks of the photographed image are used as feature quantities at the time of time zone determination. In formula (3), A, B, and C are coefficients of the evaluation function. Here, the evaluation function f (Iav, Nblk) or the coefficients A, B, and C correspond to the time zone recognition dictionary 18A.

The time zone determination unit 106 creates an evaluation function represented by the equation (3) in advance and stores it in the storage unit 18 in advance as the time zone recognition dictionary 18A.

When determining the shooting time zone of the captured image, the time zone determination unit 106 uses the time zone recognition dictionary 18A (evaluation function represented by the equation (3)) stored in the storage unit 18 to capture the captured image. Determine the band.

Specifically, the time zone determination unit 106 calculates the average brightness (Iav) and the number of low-luminance blocks (Nblk) of the captured image calculated by the first calculation unit 116H and the second calculation unit 116I in the equation (3). By substituting into, the value of the evaluation function f (Iav, Nblk) is calculated.

Then, when the value of the calculated evaluation function f (Iav, Nblk) is equal to or greater than a predetermined threshold value, the time zone determination unit 106 determines that the imaging time zone of the captured image indicates daytime. On the other hand, when the value of the calculated evaluation function f (Iav, Nblk) is less than the threshold value, the time zone discriminating unit 106 determines that the shooting time zone of the captured image indicates night. This threshold value may be set in advance according to the light intensity of the shooting environment in the shooting time zone to be discriminated.

The time zone determination unit 106 is based on a feature amount including the average brightness of the captured image, the number of low-luminance blocks of the captured image, and the dispersion value of the average brightness of each block in the captured image. The band may be discriminated.

In this case, the third calculation unit 116J of the time zone determination unit 106 calculates the dispersion value of the average brightness for each block in the captured image. The third calculation unit 116J calculates the dispersion value of the average brightness for each block divided by the second calculation unit 116I. For example, the time zone determination unit 106 calculates the variance value (σ) using the following equation (4).

In equation (4), σ indicates the dispersion value of the average brightness for each block in the captured image. In the formula (4), N indicates the number of blocks (that is, the value of m × n) included in the captured image. Ii indicates the average brightness of the i-th block. Iav indicates the overall average brightness of the captured image.

In this case, the time zone determination unit 106 uses a time zone recognition dictionary that uses the average brightness of the captured image, the number of low-luminance blocks of the captured image, and the dispersion value of the average brightness of each block in the captured image as feature quantities. 18A may be created in advance.

That is, the time zone determination unit 106 uses a machine learning method using an SVM (Support Vector Machine), as in the case where the average brightness of the captured image and the number of low-luminance blocks of the captured image are used as feature quantities. Therefore, the time zone recognition dictionary 18A may be created in advance. In this case, the time zone determination unit 106 defines a group of sample points indicated by the average brightness, the number of low-luminance blocks, and the variance value in the reference captured image in a three-dimensional space determined by the average brightness, the number of low-luminance blocks, and the variance value. You can register at.

Then, in the same manner as described above, the time zone determination unit 106 arranges the separation surface for separating the daytime sample point cloud and the nighttime sample point cloud so that the margin is maximized. Then, the time zone determination unit 106 may calculate the evaluation function indicating this separation surface as the time zone recognition dictionary 18A. The following equation (5) is an evaluation function (time zone recognition dictionary 18A) when the average brightness of the captured image P, the number of low-luminance blocks of the captured image P, and the variance value are used as feature quantities at the time of time zone discrimination. is there.

In equation (5), f (Iav, Nblk, σ) is an evaluation function (time) when the average brightness of the captured image, the number of low-luminance blocks of the captured image, and the variance value are used as feature quantities at the time of time zone discrimination. The band recognition dictionary 18A). In formula (5), A, B, C, and D are coefficients of the evaluation function.

As described above, the time zone determination unit 106 may, for example, create the evaluation function represented by the equation (5) in advance and store it in the storage unit 18 in advance as the time zone recognition dictionary 18A.

In this case, the time zone determination unit 106 uses the time zone recognition dictionary 18A (evaluation function shown in the equation (5)) stored in the storage unit 18 to determine the shooting time zone of the captured image. Determine the shooting time zone of.

Specifically, the time zone determination unit 106 has an average brightness (Iav) and low brightness of the captured image calculated by each of the first calculation unit 116H, the second calculation unit 116I, and the third calculation unit 116J. The number of blocks (Nblk) and the variance value (σ) are obtained. Then, the time zone determination unit 106 substitutes the average luminance (Iav), the number of low-luminance blocks (Nblk), and the variance value (σ) into the equation (5), thereby substituting the evaluation function f (Iav, Nblk). , Σ) is calculated.

Then, when the value of the calculated evaluation function f (Iav, Nblk, σ) is equal to or greater than a predetermined threshold value, the time zone discriminating unit 106 determines that the shooting time zone of the captured image indicates daytime. On the other hand, when the value of the calculated evaluation function f (Iav, Nblk, σ) is less than the threshold value, the time zone discriminating unit 106 determines that the shooting time zone of the captured image indicates night. This threshold value may be set in advance.

As described above, the time zone determination unit 106 determines the time zone in which the captured image acquired by the image acquisition unit 101 was captured, and outputs the determined shooting time zone to the selection unit 108.

Returning to FIG. 2, the description of the function of the recognition processing unit 16 will be continued. The position information acquisition unit 107 acquires the position information of the own device. For example, the position information acquisition unit 107 can acquire position information from an external device such as a GPS device. The position information acquisition unit 107 outputs the acquired position information to the selection unit 108.

The selection unit 108 selects the surrounding area identification information corresponding to the time zone determined by the time zone determination unit 106. Further, the selection unit 108 selects the surrounding area identification information corresponding to the position information (country or region can be specified) acquired by the position information acquisition unit 107. More specifically, the selection unit 108 is the periphery corresponding to the combination of the time zone determined by the time zone determination unit 106 and the position information (country or region can be specified) acquired by the position information acquisition unit 107. The area identification information (in this example, the evaluation function of equation (2)) is selected.

Next, a method of recognizing the candidate area by the candidate area recognition unit 102 will be described. The candidate region recognition unit 102 extracts a signal pixel region including a signal pixel indicating the signal color of the traffic light from the captured image acquired by the image acquisition unit 101, and previously expands the extracted signal pixel region into an expanded expansion region. When a standard region having a predetermined shape is included, the region including the standard region is recognized as a signal region. More specifically, when the expansion region includes a circular standard region, the candidate region recognition unit 102 recognizes the region including the standard region as a candidate region. The specific contents will be described below.

When the captured image acquired by the image acquisition unit 101 is image data in the (R, G, B) color space, the candidate area recognition unit 102 uses the captured image (input) in the RGB color space acquired by the image acquisition unit 101. Image) is changed to image data in the (Y, U, V) color space, and signal pixels are extracted. Here, the conversion between the color space of (R, G, B) and the color space of (Y, U, V) is performed by the following equation (6).

In this example, a signal recognition dictionary indicating the range of color values of the signal of the traffic light is obtained in advance by learning by collecting captured images (referred to as "signal image samples") obtained by photographing the signal of the traffic light in advance by the photographing device 12. Create it. For example, a separation surface that separates the (U, V) value distribution indicating a red signal and the (U, V) value distribution other than the red signal is arranged so that the margin (distance) between the two is maximized. Then, the evaluation function indicating the separation surface can be calculated as a red signal recognition dictionary for recognizing the red signal. The following equation (7) is an equation representing the evaluation function. In equation (7), f (U, V) is an evaluation function indicating a red signal recognition dictionary, and a, b, and c are coefficients of this evaluation function. A green signal recognition dictionary and a yellow signal recognition dictionary can be created in the same manner.

The candidate area recognition unit 102 sets the (U, V) value of each pixel constituting the captured image in the (Y, U, V) color space to each signal color (any of red, yellow, and blue). The value of the evaluation function f (U, V) is obtained by substituting it into the evaluation function f (U, V) corresponding to. If the value of the calculated evaluation function f (U, V) is equal to or greater than a predetermined threshold value, it is determined that the pixel is a signal pixel of the corresponding signal color. For example, a signal pixel may be extracted by setting a minimum value threshold value and a maximum value threshold value. Thresholds for the minimum and maximum values of U and the minimum and maximum values of V, respectively, of red, blue, and yellow may be set. Further, the threshold value is set so as not to overlap with the (U, V) values of pixels other than the signal pixel.

FIG. 15 is a diagram showing a pixel example of a green signal extracted by threshold processing in the (U, V) space of pixel values. Since the signal pixel region including the signal pixels (pixels showing the green signal) in the image of FIG. 15 includes saturated pixels and noise pixels, the extracted signal pixel region is only a part of the actual region showing the green signal. .. Therefore, the candidate area recognition unit 102 performs expansion processing on the extracted signal pixels. In the expansion process, an N × N pixel block is added to one pixel from the original signal image. For example, when N = 7, a 7 × 7 pixel block is added from the original signal image to one pixel to be expanded. FIG. 16 is a diagram showing a pixel example after the signal pixel region of FIG. 15 has been expanded. Hereinafter, the signal pixel region after the expansion processing is referred to as an “expansion region”.

The candidate area recognition unit 102 performs signal shape recognition processing on the expansion area. Shape recognition is performed for red, blue, and yellow signals by circular extraction. Here, the shape of the expanded region is recognized by extracting a circle. When a circle is detected, it can be determined that a signal exists. The circle extraction process can perform circle extraction by Hough transform. Then, the candidate area recognition unit 102 obtains a rectangle circumscribing the extracted circular area. This rectangular area is used as a candidate area (recognition result by the candidate area recognition unit 102). The circle drawn by the thick line shown in FIG. 17 indicates the circular region of the green light extracted by the Hough transform. The rectangular area shown in FIG. 18 indicates a candidate area recognized by the candidate area recognition unit 102. The rectangular area shown in FIG. 19 is a diagram showing a candidate area (a region showing a green light candidate) recognized from the captured image of FIG.

The recognition result of FIG. 19 is the result of recognizing the region of the captured image showing the signal of the traffic light (green light in this example). Even for objects other than traffic lights, if they have the same color and shape, there is a problem of misrecognition. For example, if there is a red sign behind the tree, part of the sign will be hidden by the tree. Part of the visible signboard is red and has a round shape. If it has the same color and shape (circular shape) as the red light of the traffic light, it will be erroneously recognized as the red light of the traffic light. Similar problems occur with signal colors of other colors.

In the present embodiment described above, the peripheral area identification information (evaluation function in this example) predetermined for identifying the peripheral area indicating the peripheral area indicating the peripheral area of the signal of the signal and the feature amount around the candidate area are used. Based on the ambient feature data to be shown, it is determined whether or not the candidate region corresponding to the ambient feature data is the region indicating the signal of the signal. Therefore, unless the ambient feature data corresponds to the correct answer data, the said The candidate area corresponding to the ambient feature amount data is not determined to be the area indicating the signal of the signal. That is, according to the present embodiment, the signal of the traffic light reflected in the captured image can be recognized with high accuracy.

FIG. 20 is a flowchart showing an operation example of the recognition processing unit 16 of the present embodiment. Since the specific contents of each step are as described above, detailed description thereof will be omitted as appropriate. As shown in FIG. 20, first, the image acquisition unit 101 acquires a captured image (step S1). Next, the candidate area recognition unit 102 recognizes the candidate area indicating the area corresponding to the color and shape of the signal of the traffic light among the captured images acquired in step S1 (step S2). Next, the peripheral feature data calculation unit 103 calculates the peripheral feature data indicating the peripheral features of the candidate region recognized in step S2 among the captured images acquired in step S1 (step S3). Further, in parallel with the above processing, the time zone determination unit 106 determines the time zone in which the captured image acquired in step S1 was captured (step S4).

Next, the selection unit 108 selects an evaluation function corresponding to the combination of the time zone determined in step S4 and the country or region specified by the position information acquired by the position information acquisition unit 107 (step S5). ). Next, the determination unit 104 recognizes the candidate area (recognized in step S2) corresponding to the peripheral feature amount data P based on the peripheral feature amount data calculated in step S3 and the evaluation function selected in step S5. Judgment processing is performed to determine whether or not the selected candidate region) is a region indicating a signal of a traffic light (step S6). Then, the deletion processing unit 105 performs a process of deleting the candidate area determined not to be the area indicating the signal of the traffic light by the determination process of step S6 (step S7). The output unit 109 outputs the candidate area determined to be the area indicating the signal of the traffic signal as the recognition result by the determination process in step S6 (step S8).

Next, an example of the hardware configuration of the photographing device 12 will be described. FIG. 21 is a diagram showing an example of the hardware configuration of the photographing device 12.

The photographing device 12 includes a photographing optical system 2010, a mechanical shutter 2020, a motor driver 2030, a CCD (Charge Coupled Device) 2040, a CDS (Correlated Double Sample) circuit 2050, an A / D converter 2060, and a timing signal generator. 2070, image processing circuit 2080, LCD (Liquid Crystal Device) 2090, CPU (Central Processing Unit) 2100, RAM (Random Access Memory) 2110, ROM (Read Only Memory) 2120, SRAM (Syndy) It includes an extension circuit 2140, a memory 2150, an operation unit 2160, and an output I / F 2170.

The image processing circuit 2080, CPU 2100, RAM 2110, ROM 2120, SDRAM 2130, compression / decompression circuit 2140, memory 2150, operation unit 2160, and output I / F 2170 are connected via a bus 2200.

The photographing optical system 2010 collects the reflected light of the subject. By opening the mechanical shutter 2020 for a predetermined time, the light focused by the photographing optical system 2010 is incident on the CCD 2040. The motor driver 2030 drives the photographing optical system 2010 and the mechanical shutter 2020.

The CCD 2040 forms an image of the light incident through the mechanical shutter 2020 as an image of the subject, and inputs analog image data showing the image of the subject to the CDS circuit 2050.

When the CDS circuit 2050 receives the analog image data from the CCD 2040, the CDS circuit 2050 removes the noise component of the image data. Further, the CDS circuit 2050 performs analog processing such as correlated double sampling and gain control. Then, the CDS circuit 2050 outputs the processed analog image data to the A / D converter 2060.

When the A / D converter 2060 receives the analog image data from the CDS circuit 2050, the A / D converter 2060 converts the analog image data into digital image data. The A / D converter 2060 inputs digital image data to the image processing circuit 2080. The timing signal generator 2070 transmits a timing signal to the CCD 2040, the CDS circuit 2050, and the A / D converter 2060 in response to the control signal from the CPU 2100, thereby transmitting the timing signal to the CCD 2040, the CDS circuit 2050, and the A / D converter 2060. Controls the operation timing of.

When the image processing circuit 2080 receives the digital image data from the A / D converter 2060, the image processing circuit 2080 uses the SDRAM 2130 to perform image processing of the digital image data. The image processing includes, for example, CrCb conversion processing, white balance control processing, contrast correction processing, edge enhancement processing, color conversion processing, and the like.

The image processing circuit 2080 outputs the image data subjected to the above-mentioned image processing to the LCD 2090 or the compression / decompression circuit 2140. The LCD 2090 is a liquid crystal display that displays image data received from the image processing circuit 2080.

When the compression / decompression circuit 2140 receives the image data from the image processing circuit 2080, the compression / decompression circuit 2140 compresses the image data. The compression / decompression circuit 2140 stores the compressed image data in the memory 2150. Further, when the compression / decompression circuit 2140 receives the image data from the memory 2150, the compression / decompression circuit 2140 decompresses the image data. The compression / decompression circuit 2140 temporarily stores the decompressed image data in the SDRAM 2130. Memory 2150 stores compressed image data.

The output I / F 2170 outputs the image data processed by the image processing circuit 2080 to the information processing device 10 as a captured image.

At least a part of the functional unit included in the interface unit 14 and the recognition processing unit 16 described with reference to FIG. 2 may be mounted on the photographing device 12 as a signal processing board (signal processing circuit).

Next, the hardware configuration of the information processing apparatus 10 according to the above-described embodiment and modification will be described. FIG. 22 is a block diagram showing a hardware configuration example of the information processing apparatus 10 of the above embodiment.

In the information processing device 10 of the above embodiment, the output unit 80, the I / F unit 82, the input unit 94, the CPU 86, the ROM (Read Only Memory) 88, the RAM (Random Access Memory) 90, the HDD 92, and the like are mutually connected by the bus 96. It is connected to and has a hardware configuration using a normal computer.

The CPU 86 is an arithmetic unit that controls the processing executed by the information processing device 10 of the above embodiment. The RAM 90 stores data required for various processes by the CPU 86. The ROM 88 stores a program or the like that realizes various processes by the CPU 86. The HDD 92 stores the data stored in the storage unit 18 described above. The I / F unit 82 is an interface for transmitting and receiving data to and from other devices.

A program for executing the various processes executed by the information processing apparatus 10 of the above embodiment is provided by being incorporated in ROM 88 or the like in advance.

The program executed by the information processing device 10 of the above embodiment is a file in a format that can be installed or executed in these devices, and is a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital). It may be configured to be recorded and provided on a computer-readable recording medium such as a Versail Disc).

Further, the program executed by the information processing apparatus 10 of the above embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading via the network. Further, the program for executing each of the above processes in the information processing apparatus 10 of the above embodiment may be configured to be provided or distributed via a network such as the Internet.

Each of the above-described parts of the program for executing the various processes executed by the information processing device 10 of the above-described embodiment is generated on the main storage device.

The various information stored in the HDD 92 may be stored in an external device. In this case, the external device and the CPU 86 may be connected via a network or the like.

10 Information processing device 14 Interface unit 16 Recognition processing unit 18 Storage unit 101 Image acquisition unit 102 Candidate area recognition unit 103 Surrounding feature amount data calculation unit 104 Judgment unit 105 Deletion processing unit 106 Time zone determination unit 107 Position information acquisition unit 108 Selection unit 109 Output section

JP-A-2009-2449446

Claims (8)

The acquisition unit that acquires the captured image and
Among the captured images acquired by the acquisition unit, a candidate area recognition unit that recognizes a candidate area indicating an area corresponding to the color and shape of the signal of the traffic light, and a candidate area recognition unit.
A peripheral feature data calculation unit that calculates peripheral feature data indicating the peripheral features of the candidate region, and a peripheral feature data calculation unit.
In order to identify the peripheral area indicating the peripheral area indicating the peripheral area of the signal of the traffic light, the peripheral area identification information predetermined for each country or region and the peripheral feature amount data calculated by the peripheral feature amount data calculation unit are used. Based on, a determination unit for determining whether or not the candidate region corresponding to the ambient feature amount data is a region indicating a signal of the traffic light, and a determination unit.
The location information acquisition unit that acquires location information and
A selection unit for selecting the surrounding area identification information corresponding to the position information acquired by the position information acquisition unit is provided.
Information processing device. The surrounding area identification information is predetermined for each of a plurality of time zones.
A time zone determination unit that determines the time zone in which the captured image was taken, and
A selection unit for selecting the surrounding area identification information corresponding to the time zone determined by the time zone determination unit is further provided.
The information processing device according to claim 1. The determination unit corresponds to the peripheral feature amount data based on the peripheral area identification information selected by the selection unit and the peripheral feature amount data calculated by the peripheral feature amount data calculation unit. It is determined whether or not the candidate area is a region indicating the signal of the traffic light.
The information processing device according to claim 1 or 2. The acquisition unit that acquires the captured image and
Among the captured images acquired by the acquisition unit, a candidate area recognition unit that recognizes a candidate area indicating an area corresponding to the color and shape of the signal of the traffic light, and a candidate area recognition unit.
A peripheral feature data calculation unit that calculates peripheral feature data indicating the peripheral features of the candidate region, and a peripheral feature data calculation unit.
Based on the peripheral area identification information predetermined for identifying the peripheral area indicating the peripheral area indicating the peripheral area of the signal of the signal, and the peripheral feature amount data calculated by the peripheral feature amount data calculation unit, the said The peripheral area identification information is prepared in advance by using a machine learning method, and includes a determination unit for determining whether or not the candidate area corresponding to the ambient feature amount data is an area indicating a signal of the signal.
It is an evaluation function indicating a separation surface between the peripheral region and the erroneous recognition region indicating an region other than the peripheral region.
Information processing device. The candidate area recognition unit
Among the shooting Kagee image, if the extracting traffic signal pixel region including a signal pixel indicating the signal color, include standard regions of a predetermined shape to the expansion area expands the signal pixel regions thereof extracted, Recognizing the region including the standard region as the candidate region.
The information processing device according to any one of claims 1 to 4. When the expansion region includes the circular standard region, the candidate region recognition unit recognizes the region including the standard region as the candidate region.
The information processing device according to claim 5. The acquisition step to acquire the captured image and
Among the shooting Kagee image, recognizing the candidate region recognition step the candidate region showing a signal region corresponding to the color and shape of traffic,
A peripheral feature data calculation step for calculating peripheral feature data indicating the peripheral features of the candidate region, and
In order to identify the peripheral area indicating the peripheral area of the traffic light, the peripheral area identification information predetermined for each country or region and the peripheral feature amount data calculated by the peripheral feature amount data calculation step are added to. Based on this, a determination step of determining whether or not the signal region corresponding to the ambient feature amount is a region indicating a signal of the traffic light, and a determination step.
The location information acquisition step to acquire the location information and
Including a selection step of selecting the surrounding area identification information corresponding to the acquired position information.
Information processing method. On the computer
The acquisition step to acquire the captured image and
Among the shooting Kagee image, recognizing the candidate region recognition step the candidate region showing a signal region corresponding to the color and shape of traffic,
The peripheral feature data calculation step for calculating the peripheral feature data indicating the peripheral features of the signal region, and the peripheral feature data calculation step.
In order to identify the peripheral area indicating the peripheral area indicating the peripheral area of the signal of the traffic light, the peripheral area identification information predetermined for each country or region and the peripheral feature amount data calculated by the peripheral feature amount data calculation step. Based on, a determination step for determining whether or not the signal region corresponding to the ambient feature amount is a region indicating the traffic light, and
The location information acquisition step to acquire the location information and
A selection step for selecting the surrounding area identification information corresponding to the acquired position information, and
A program to execute.

Images