JP2007060273A - Environment recognition device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an environment recognition device which is capable of easily making environment recognition with a little throughput, on the basis of color video images which are present ahead of an vehicle and imaged by an on-vehicle camera. <P>SOLUTION: A color information forming unit 10 calculates the average color of the captured image data by the macro block, and determines which color index corresponds to the average color of each macro block. Color arrangement information correlating the color information as the judgement result with the macro blocks is stored in a color index memory 14. A histogram forming unit 15 forms a color histogram, indicating the distribution of the macro blocks given color information corresponding to the target color index, in accordance with the color arrangement information. Based on the color arrangement information or the color histogram, a color application enforcing unit 16 recognizes an object or the state of an environment or a vehicle without subjecting video images to a region dividing process (shape capture). <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an environment recognition device that recognizes an environment in front of a vehicle based on a color image in front of the vehicle imaged by an in-vehicle camera.

  2. Description of the Related Art Conventionally, an apparatus for recognizing an object such as a traveling lane of a host vehicle, a preceding vehicle on the traveling lane, a traffic sign, or the like based on a color image (monocular image) in front of the vehicle captured by an in-vehicle camera is known ( For example, see Patent Document 1.)

  In this device, a three-dimensional space representing colors (here, with brightness, lightness, saturation, etc. as axes) is divided into a plurality of regions, and the median value of each divided region is used as a color index. The color of each pixel constituting the color image is classified. Then, an area division process for dividing the image according to the classified color is executed, and whether the position and shape of the area acquired as a result of the area division process is compatible with knowledge about the object prepared in advance. The object is recognized by executing knowledge processing for determining whether or not the object is detected.

As another method for specifying an object from the result of area division, a method by pattern matching is also known.
JP-A-6-338991

  However, when objects having the same color in the color image are overlapped with each other, the area division using the color information of the image data cannot be accurately divided (acquired shape), and as a result, Even if knowledge processing or pattern matching based on the region shape is performed, there is a problem that it is impossible to correctly determine which object the region corresponds to.

  Further, in the apparatus described in Patent Document 1, since the area division process is performed on a pixel basis, the processing load is large, and particularly in the method of specifying an object by pattern matching, the processing load becomes enormous. There was a problem that.

  In recent years, there has been a demand for extracting an object included in an image and recording a description related to the extracted object in association with an image in order to facilitate searching for a recorded image and providing information. For realization, it is desired to establish a technique for recognizing an object represented in an image with a small amount of processing.

  In order to solve the above-described problems, an object of the present invention is to provide an environment recognition device that can easily perform environment recognition based on a color image in front of a vehicle imaged by an in-vehicle camera with a small amount of processing.

  In the environment recognition apparatus according to claim 1, which is an invention made to achieve the above object, the object specifying means specifies a target object which is an object to be recognized. Further, the image data acquisition means acquires image data based on a color image ahead of the vehicle imaged by the in-vehicle camera, and based on the acquired image data, the average color calculation means sets the color image to a preset size. For each pixel block obtained by the division, an average color of the pixel block is calculated.

  Then, the color information providing unit determines which of the color indexes prepared in advance the average color calculated by the average color calculating unit belongs, and stores the determination result in association with the pixel block as color information. .

  In addition, the color histogram generation unit integrates the number of pixel blocks in which the color information associated with the color information providing unit and the target color index match along one direction on the color image, and the integrated value is first-ordered. A color histogram that is originally arranged is generated in time series for each color index and for each frame of image data.

  Then, the object information adding means generates information on the target object specified by the object specifying means based on the color histogram generated by the color histogram generating means, and the image data acquiring means uses the generated information as object information. The image data is stored in association with the acquired image data.

  That is, the color histogram shows the distribution of the number of pixel blocks to which color information corresponding to the color index of interest is given in the intersecting direction intersecting the integration direction. Accordingly, when the target object has a remarkable feature in the color or the arrangement along the intersecting direction, information (object information) related to the target object such as the presence / absence of the target object is recognized from the color histogram. It becomes possible.

  Further, in the environment recognition apparatus of the present invention, the color information is given not in units of pixels but in units of pixel blocks. In addition, the target object is recognized without performing region segmentation (shape acquisition) using a color histogram obtained by integrating the pixel blocks.

  Therefore, according to the environment recognition apparatus of the present invention, it is possible to greatly reduce the processing load compared to the conventional apparatus that assigns color information in units of pixels and divides the area, and makes it easy to recognize an object in a color image. Can be done.

  In particular, in limited applications such as in-vehicle cameras, the objects that exist in the captured color image are limited, so it is possible to recognize the object from the distribution of color information without specifying the shape of the object. Become.

  By the way, as described in claim 2, the direction in which the color histogram generation means accumulates the number of pixel blocks is the vertical direction in the color image, that is, each element of the color histogram is orthogonal to the vertical direction. It is desirable to make it correspond to the position in the horizontal direction.

In this case, the color histogram represents a distribution along the horizontal direction, that is, the vehicle width direction.
Here, FIG. 13 is a three-dimensional graph (spatio-temporal distribution) in which color histograms acquired during night driving are arranged in time series, focusing on the red color index. However, in the figure, the horizontal axis represents time, the vertical axis represents the number of pixel blocks, and the depth represents the horizontal position in the color image. The near side is the right when viewed from the driver, and the far side is the left when viewed from the driver.

  Peaks P1, P2, and P4 appearing on the right side of the screen (front side in the figure) are peaks P1, P2, and P4 due to streetlights installed in the central separation zone. The peak that appears from the center to the left side of the screen is due to the red light of the traffic light installed at the intersection. In particular, the peaks P3 and P5 observed over a long period of 10 seconds or more are the vehicles at the intersection. This is due to the suspension.

  Thus, by considering the time-series change of the color histogram, not only mere object recognition but also vehicle behavior and state such as streetlight passage and intersection stop can be recognized.

  The direction in which the number of pixel blocks is integrated is not limited to the vertical direction in the color image, but may be the horizontal direction or an intermediate direction between the two. In particular, in the horizontal direction, since it is possible to detect a horizontal edge in a color image, it can be applied to vehicle recognition based on the horizontal edge.

According to a third aspect of the present invention, the color histogram generation means may be configured to set a range of pixel blocks to be integrated according to the target object.
In particular, when the integration direction of the pixel block at the time of generating the color histogram is the vertical direction, setting the range of the pixel block to be integrated only applies to those existing at a certain height position. It will be.

  For this reason, when the presence of the target object is limited to a specific height range, the influence of other objects that are located at a height that cannot be the target object and that has the same color as the target object is reliably determined from the color histogram. Can be eliminated.

  That is, by limiting the range of pixel blocks to be integrated when generating a color histogram according to the target object, the recognition accuracy of the target object using the color histogram can be improved.

  Next, in the environment recognition device according to claim 4, the evaluation value calculation unit for each color has a horizontal position corresponding to each element of the color histogram for each of the color histograms generated by the color histogram generation unit. Among these, using a weighting coefficient set so that the value of the position where the target object is likely to be larger becomes larger, the weighted addition of each element constituting the color histogram is executed, and the calculation result of the weighted addition is The result of integration during the monitoring period set according to the target object is calculated as an evaluation value for each color.

  Then, the object information giving means uses the sum of the evaluation values for each color calculated by the evaluation value calculation means for each color as an integrated evaluation value, and if this integrated evaluation value is larger than a preset evaluation threshold value, It is determined that the target object exists in the file, and the determination result is used as object information.

  According to the environment recognition device of the present invention configured as described above, when the target object has a plurality of colors, it is possible to comprehensively evaluate using all the color indexes corresponding to the plurality of colors. The recognition accuracy of the target object can be further increased.

  For color histograms corresponding to colors unrelated to the target object, for example, by obtaining all integrated evaluation values by adding only the color-specific evaluation values related to the target object by setting all the weighting factors to zero. Can do.

  By the way, as described in claim 5, the color-by-color evaluation value calculation means sets the monitoring period to a value inversely proportional to the vehicle speed when the vehicle speed of the vehicle on which the device is mounted is equal to or greater than a preset lower limit value. Preferably, the vehicle speed is set to a preset fixed value when the vehicle speed is smaller than the lower limit value.

  Note that, for example, when the target object is arranged in the vicinity of the road at a certain distance interval, the monitoring period may be set to a time required to pass the certain distance, that is, as the vehicle speed decreases. The monitoring period may be set to be long.

  However, in this case, when the vehicle speed approaches zero, the monitoring period becomes extremely long, and a large delay occurs in the acquisition of the object information (recognition result). For this reason, when the vehicle speed is smaller than the lower limit value, it is desirable to set the monitoring period to a fixed value so that the monitoring period does not exceed the allowable delay time.

  Next, in the color-by-color evaluation calculation means in the environment recognition device according to claim 6, the variable element generation means is a distance obtained by arranging the standard distances set according to the objects in association with the respective color indexes. Designated by the object designating means as a vector, and a load coefficient vector in which the load coefficient is arranged in association with each element of the color histogram, and the load coefficient matrix is arranged in association with each color index. The vector and the load coefficient matrix suitable for the target object are generated.

  Then, the period vector generating means sets the monitoring period corresponding to each of the color indexes as a period vector, and mounts the distance vector generated by the variable element generating means, the frame rate of the color image, and the device. A period vector is generated based on the vehicle speed.

  Further, the matrix calculation means generates a color histogram matrix obtained by arranging the color histogram generated by the color histogram generation means in association with each color index, and generates the color histogram matrix and the variable element generation means. An evaluation unit vector formed by arranging weighted addition values corresponding to each of the color indexes is calculated by performing a matrix operation on the obtained weight coefficient matrix.

  Further, the accumulation calculating means accumulates each element of the evaluation unit vector calculated by the matrix calculating means in accordance with the contents of the period vector generated by the period vector generating means, thereby obtaining the evaluation value for each color of the color index. An evaluation vector for each color formed by associating with each other is generated.

Then, the object information providing unit calculates the integrated evaluation value of the target object by adding each element of the evaluation vector for each color generated by the cumulative calculation unit.
That is, in the environment recognition apparatus of the present invention, a distance vector and a load coefficient matrix corresponding to the target object are generated, and the evaluation value for each color (each element of the evaluation vector for each color) is calculated at once by vector calculation. ing.

  Therefore, according to the environment recognition apparatus of the present invention, even if the standard distance, the weighting coefficient, the number of evaluation values for each color necessary for calculating the integrated evaluation value, and the like differ depending on the object, these are used as a single evaluation value for each color The calculation means can handle them uniformly, and the processing structure can be simplified.

  Each means constituting the evaluation value calculation means for each color may be realized by processing executed by a microcomputer, but can be easily realized by, for example, a processor capable of vector calculation. Time can be significantly reduced.

  Next, in the environment recognition device according to claim 7, the scene classification dictionary storage unit is likely to exist in each scene and each of a plurality of scenes representing a typical scene of a color image. A scene classification dictionary in which objects are stored in association with each other is stored, and the object designating unit sequentially reads out the objects associated with the scenes designated from the outside from the scene classification dictionary, and the read objects are the target objects. Specify as.

  According to the environment recognition apparatus of the present invention configured as described above, since the objects to be recognized are limited for each designated scene, the objects can be recognized efficiently and accurately. The selection of the scene is performed, for example, by associating the scene with position information in advance and based on position information obtained from navigation, GPS, or the like, or based on past recognition results, the current operation mode, or the like. What should I do?

However, the object specifying means may be configured to sequentially specify all objects to be recognized without using the scene classification dictionary.
Next, in the environment recognition device according to the eighth aspect, the object designating unit designates a target object which is an object to be recognized. Further, the image data acquisition means acquires image data based on a color image ahead of the vehicle imaged by the in-vehicle camera, and based on the acquired image data, the average color calculation means sets the color image to a preset size. For each pixel block obtained by the division, an average color of the pixel block is calculated.

  Then, the color information providing unit determines which of the color indexes prepared in advance the average color calculated by the average color calculating unit belongs, and stores the determination result in association with the pixel block as color information. .

  Then, based on the color information associated with each of the pixel blocks by the color information giving means, the object information giving means generates information about the target object specified by the object specifying means, and the generation information is Object information is stored in association with the image data acquired by the image data acquisition means.

  In this case, since the object information is generated based on the color information of the pixel block, the amount of information to be handled is increased as compared with the case where the object information is generated based on the color histogram. It is also possible to obtain position information for a non-integrated direction.

  As described above, according to the environment recognition apparatus of the present invention, more detailed position information can be obtained than in the case of using a color histogram, so that the object recognition accuracy can be improved. In other words, by combining recognition based on the color histogram and recognition based on the color information, and applying the method based on the color information only to objects that are difficult to recognize using the method based on the color histogram, the processing load is increased. Can be reduced and the recognition accuracy can be improved.

  In addition, as an example of performing object recognition (generation of object information) without using a color histogram, the environment recognition apparatus according to claim 9 is a template storage that stores a plurality of templates representing wiper positions in a color image. Means.

  Then, the binarization unit adds a different value between the pixel block to which the color information corresponding to the black color index is assigned by the color information addition unit and the pixel block to which the other color information is assigned. Generate a digitized image. Then, the wiper detection means detects a wiper at a position other than the stop position when not in use by pattern matching between the template stored in the template storage means and the binarized image generated by the binarization means. The calculating means calculates the frequency at which the wiper detecting means detects the wiper during the preset observation period.

  Then, in the object information giving means, when the frequency calculated by the frequency calculation means exceeds a preset operation determination threshold, information indicating that the wiper is operating is set as object information.

  According to the environment recognition apparatus of the present invention configured as described above, it is possible to prevent erroneous recognition of the operating wiper as a part of the target object, and the recognition accuracy of the object can be further improved.

  Furthermore, in the environment recognition apparatus according to claim 10, the cumulative addition means is generated by the binarization means based on the image data captured from the image acquisition means during the operation of the wiper of the vehicle equipped with the apparatus. The binarized images are grouped by using the address of the macroblock on the edge of the binarized image as an identifier, and the identifier to which color information corresponding to the black color index is assigned, Each time a binarized image belonging to the same group is cumulatively added in units of pixel blocks, a cumulative addition image is generated.

  Then, the template generation means, for the cumulative addition image generated by the cumulative addition means, a pixel block whose cumulative addition value is greater than or equal to a preset presence determination threshold, and a pixel block whose cumulative addition value is smaller than the presence determination threshold A template with different values is generated as a template and stored in the template storage means.

That is, according to the environment recognition apparatus of the present invention, the template stored in the template storage means can be automatically generated without being given from the outside.
As another example of recognizing an object (generating object information) without using a color histogram, in the environment recognition apparatus according to claim 11, the candidate extracting unit converts the red color index by the color information adding unit. A pixel block to which corresponding color information is assigned is taken as a red pixel block, and two red pixel blocks arranged in the horizontal direction are extracted as tail lamp candidates.

  Then, the inter-vehicle distance estimating means determines whether the positional relationship of the tail lamp candidates extracted by the candidate extracting means meets any of the vehicle width conditions set in association with each of a plurality of inter-vehicle distances. Estimate the distance between cars.

Then, the object information providing means uses the vehicle position represented by the position of the tail lamp candidate and the inter-vehicle distance estimated by the inter-vehicle distance estimating means as the object information.
According to the environment recognition apparatus of the present invention configured as described above, simple processing such as extraction of red pixel blocks arranged in the horizontal direction and determination of the positional relationship thereof is performed without performing complicated processing such as region division. The vehicle ahead, which is one of the objects to be recognized, can be detected along with its position.

  Furthermore, in the environment recognition device according to claim 12, the lighting determining means determines the presence / absence of lighting based on the brightness of the tail lamp candidate extracted by the candidate extracting means, and the object information providing means is the lighting determining means. The discrimination result is used as object information.

  Still further, in the environment recognition device according to claim 13, the vehicle color extracting means extracts the color of the macro block located between or above the tail lamp candidates extracted by the candidate extracting means as the color of the preceding vehicle. In the object information giving means, the extraction result in the car color extracting means is used as object information.

  In these cases, since the presence / absence of lighting of the tail lamp and the color of the vehicle are added as object information for the preceding vehicle extracted based on the red pixel block, more detailed object information can be provided.

  Next, in the environment recognition device according to claim 14, when the abnormal situation determination unit does not match the recognition result of the target object in the recognition unit with the information about the target object supplied from outside the device. Then, it is determined that there is an abnormal situation.

  According to the environment recognition device of the present invention configured as described above, it is possible to detect an abnormality that has occurred in either the device or an external device that supplies information related to the target object to the device.

  Next, in the environment recognition device according to claim 15, the image data acquisition unit acquires image data based on a color image in front of the vehicle captured by the in-vehicle camera, and based on the acquired image data, the average color The calculation means calculates an average color of the pixel block for each pixel block obtained by dividing the color image by a preset size.

  Then, the color information providing unit determines which of the color indexes prepared in advance the average color calculated by the average color calculating unit belongs, and stores the determination result in association with the pixel block as color information. .

  In addition, the color histogram generation unit integrates the number of pixel blocks in which the color information associated with the color information providing unit and the target color index match along one direction on the color image, and the integrated value is first-ordered. A color histogram that is originally arranged is generated in time series for each color index and for each frame of image data.

  Then, the first difference calculation means calculates the inter-frame difference of the color histogram generated by the color histogram generation means for each color index, and the first caution state detection means is the first difference calculation means Based on the calculation result, a situation requiring attention that appears on the color screen is detected.

  According to the environment recognition apparatus of the present invention configured as described above, since the attention situation is detected only from the change of the color histogram without performing the object recognition, the processing load is small, and the attention situation is promptly detected. Can be detected.

  Specifically, for example, when the direction of the vehicle suddenly changes or something jumps out in front of the vehicle, the difference between frames of the color histogram becomes a large value, so this situation is necessary. It can be detected as a caution situation.

  Next, in the environment recognition device according to claim 16, the image data acquisition means acquires image data based on a color image in front of the vehicle imaged by the in-vehicle camera, and based on the acquired image data, the average color The calculation means calculates an average color of the pixel block for each pixel block obtained by dividing the color image by a preset size.

  Then, the color information providing unit determines which of the color indexes prepared in advance the average color calculated by the average color calculating unit belongs, and stores the determination result in association with the pixel block as color information. .

  Further, the full screen integrated value generation means integrates the number of pixel blocks in which the color information matched by the color information providing means and the target color index match over the entire color image. Are generated in time series in units of frames of image data.

  Then, the second difference calculating means calculates the inter-frame difference of the full screen integrated value generated by the full screen integrated value generating means, and the second caution state detecting means calculates by the second difference calculating means. Based on the result, a situation requiring attention that appears on the color screen is detected.

  Specifically, as described in claim 17, the full-screen integrated value generating means generates a full-screen integrated value for the red color index, and the second caution state detecting means is the weather supplied from the outside The case where the data indicates that it is raining and the inter-frame difference of the full-screen integrated value calculated by the second difference calculating means is greater than or equal to a preset approach threshold is detected as a caution situation. Can be considered.

  In other words, when it rains, the red color histogram reacts sensitively to the lighting of the stop lamp of the preceding vehicle due to the diffused light on the windshield due to raindrops. This number is detected as a situation requiring attention.

  Next, in the environment recognition device according to claim 18, the image data acquisition means acquires image data based on a color image in front of the vehicle imaged by the in-vehicle camera, and based on the acquired image data, the average color The calculation means calculates an average color of the pixel block for each pixel block obtained by dividing the color image by a predetermined size.

  Then, the color information providing unit determines which of the color indexes prepared in advance the average color calculated by the average color calculating unit belongs, and stores the determination result in association with the pixel block as color information. .

  Then, the risk level calculation means calculates the inter-frame difference of the average color of the pixel blocks calculated by the average color calculation means for all the pixel blocks belonging to the monitoring area that is one area in the color image specified in advance. The integrated value is calculated as the risk level of the monitoring area.

  In the environment recognition apparatus of the present invention configured as described above, the degree of risk that is calculated increases as there is a large movement (screen change) in the monitoring area. This calculated degree of risk can be used, for example, for starting a process for alerting the driver.

In addition, as a monitoring area | region, setting the area | region etc. which a vehicle or a person may jump out for every scene etc. can be considered, for example.
However, in this case, as described in claim 19, when calculating the inter-frame difference of the average color of the pixel block, the risk degree calculation unit weights the pixel block closer to the vanishing point of the color image. It is desirable to perform addition.

  In other words, because of the characteristics of the image, the movement on the screen relative to the actual movement is smaller near the vanishing point and larger as it moves away from the vanishing point. This is because a lower risk level is calculated, and a higher risk level is calculated as the distance from the vanishing point increases.

  By the way, when the image data acquired by the image data acquisition means is encoded using orthogonal transform, the average color calculation means uses the direct current component included in the image data as the average color as described in claim 20. It is desirable to use it.

  In this case, by decoding the DC component, the average color of the pixel block can be easily obtained, and the processing load on the average color calculation means can be greatly reduced.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a block diagram showing a configuration of an in-vehicle system including an environment recognition device to which the present invention is applied.

  As shown in FIG. 1, the in-vehicle system includes an in-vehicle camera 1 that captures a scene in front of the vehicle, an encoding unit 2 that encodes a color image (monocular image) obtained from the in-vehicle camera 1, and an encoding unit 2. And an environment recognition device 3 that outputs recognition results of objects existing in the color image and various data useful for object recognition based on the encoded image data.

  Among these, the in-vehicle camera 1 is composed of a CCD camera or the like that outputs a color image, and is installed in the vicinity of the ceiling in the vehicle interior so that a scene in front of the vehicle can be photographed through the windshield. In this embodiment, it is assumed that one frame image is composed of horizontal: 352 pixels × vertical: 240 pixels.

  The encoding unit 2 is a well-known unit that performs encoding in the MPEG format. The image data encoded by the encoding unit 2 is subjected to two-dimensional DCT (discrete cosine transform), and a pixel block ( The DCT coefficient is described in units of 8 pixels × 8 pixels. Note that the direct current (DC) component of the DCT coefficient in each pixel block corresponds to the average value of the pixel blocks.

  An image of one frame is divided into macroblock units each composed of 16 pixels × 16 pixels (that is, one frame is composed of M (= 22) × N (= 15) macroblocks in the present embodiment). This macro block is represented by four pixel blocks Y0, Y1, Y2, Y3 representing luminance (Y component) and two pixel blocks U, V representing color difference (U component, V component). Yes.

  Next, when the image data fetched from the encoding unit 2 is an intra frame (I frame), the environment recognition apparatus 3 obtains color information representing the average color of the image data (I frame) in units of macroblocks. An information generation unit 10 and a color index memory 14 in which color information generated by the color information generation unit 10 is stored in association with each macroblock are provided.

  Among these, the color information generation unit 10 is based on the image data fetched from the encoding unit 2, the block average color calculation unit 11 that calculates the average color of the image data in units of macroblocks, and the color used in the apparatus. A color dictionary storage unit 12 that stores a color dictionary palette that defines a vocabulary (hereinafter referred to as “color index”), and a block average color calculation unit 11 by referring to the color dictionary palette stored in the color dictionary storage unit 12 The color determination unit 13 determines which color index corresponds to the average color of the macroblock calculated in (1) and generates the determination result as color information.

  The block average color calculation unit 11 decodes the DC component of the DCT coefficient for each of the six pixel blocks Y0 to Y3, U, and V associated with the macroblock in the encoded image data. To find the average color. However, for the four pixel blocks Y0 to Y3 representing the luminance, the average value of the direct current components is set as the average color of the Y components in the macroblock.

  Here, FIG. 2A is a sample image taken by the in-vehicle camera 1, and FIG. 2B is an average color for each macroblock calculated by the block average color calculation unit 11 when the sample image is processed. Is shown in a three-dimensional graph. However, FIG. 2B is a three-dimensional graph for the Y component, and the same is generated for the U component and the V component. In the following, in the image taken by the in-vehicle camera 1, the direction corresponding to the vehicle width direction (the left-right direction in FIG. 2A) is referred to as the horizontal direction, and the direction orthogonal thereto is referred to as the vertical direction. To do.

Next, in the color dictionary storage unit 12, the color index of the next P color (P = 7 in the present embodiment) is used.
{Blue, green, red, yellow, white, gray, black}
Each color index is associated with a specific object to be recognized, and a typical example is shown in Table 1 below. This correspondence is given as one piece of knowledge used when the color application execution unit 16 described later recognizes the correspondence.

In the color dictionary palette, threshold values (upper limit value and lower limit value) set using the selected color system (here, YUV) are associated with each color index.
Here, one of the color indexes is ci (i = 1, 2,..., P), the lower limit value of the Y component of the color index ci is LYci, the upper limit value is LHci, the lower limit value of the U component is LUci, and the upper limit value. Is represented by LHci, the lower limit value of the V component is represented by LVci, and the upper limit value is represented by HVci.

  Note that the color index is not limited to the above seven colors, and other colors may be used or the number of color indexes may be increased or decreased as long as color classification in accordance with human senses is achieved. However, for the number of color indexes, several colors centering on the three primary colors and black and white are appropriate due to the configuration of the threshold.

  If the average color of each YUV component calculated by the block average color calculation unit 11 is AY, AU, AV in the block average color calculation unit 11, each color is determined using the threshold values defined in the color dictionary palette. Using the inequalities shown in (1) to (3) defined for each index, it is determined which color index ci corresponds to the color of the macroblock, and the determination result (that is, one of the color indexes) is It is generated as color information of the macro block. However, the upper limit value and the lower limit value may have only one of them defined.

LYci ≦ AY <HYci (1)
LUci ≦ AU <HUci (2)
LVci ≦ AV <LUci (3)
The color information generated in this way is stored in the color index memory 14 in association with the macroblock as described above. Hereinafter, the color information associated with the macroblock is referred to as a color information array.

  Returning to FIG. 1, the environment recognition device 3 is based on the calculation result (macroblock average color) in the block average color calculation unit 11 and calculates a risk level calculation unit that calculates the risk level of the monitoring area set in advance on the color screen. 24, a histogram generation unit 15 that generates a color histogram representing a distribution of macroblocks to which color information corresponding to the color index of interest is assigned, according to the color information array stored in the color index memory 14, and a color index memory 14 A color image captured by the in-vehicle camera 1 by executing various applications based on the color information array stored in the image, the color histogram generated by the histogram generation unit 15, and various data supplied from the outside of the device 3. 3D map generated based on the recognition result of objects existing inside, color histogram, attention is required And a color application execution unit 16 for outputting such caution notice indicating a situation.

  Among these, the risk level calculation unit 24 calculates the risk level Dk of the monitoring area in the frame k according to the equation (4), where k is a parameter for identifying the frame.

  However, Zdc (EL (i), ND (j), k) is the average color (DCT coefficient of the component identified by EL (i) for the macroblock identified by ND (j) in frame k. DC component). EL (i) represents any one of the six types of components Y0 to Y3, U, and V of the macroblock, and ND (j) represents J macroblocks included in the monitoring area. Represents one of these. Further, W (j) is a weighting function given corresponding to the position of the macroblock.

  That is, the greater the difference power between frames (that is, changes in brightness and color), the greater the burden on vision during driving. Therefore, the sum of the difference powers for all components Y0 to Y3, U, and V It is obtained as the degree of risk Dn at k. Even during normal straight running, the difference between frames is large at the periphery of the image. Therefore, the weight function is set so that the macroblock is closer to the vanishing point in the image, especially near the vanishing point in the image. Set W (j). The monitoring area is set to an area where a new object may appear suddenly, such as a vehicle or a person jumping out.

  As shown in FIG. 3, the histogram generation unit 15 determines the number of macroblocks to which the color index ci of interest is attached along the horizontal direction of the image based on the color information array stored in the color index memory 14. A color histogram Hci (m) is generated by accumulating along the vertical direction of the image for each horizontal coordinate m (m = 1, 2,..., M) of the macroblock. That is, the number of color indexes (P) of color histograms Hci (m) is generated for one frame image.

In the histogram generation unit 15, the vertical range in which the macroblocks are accumulated can be arbitrarily changed according to the setting from the color application execution unit 16.
In FIG. 3, the number of macro blocks in the vertical direction of the color information array is described to be easy to see.

  Returning to FIG. 1, the color application execution unit 16 associates a vocabulary representing a typical object, a typical state, and the like included in each scene representing a typical scene of a color image. A scene classification dictionary storage unit 17 that stores a scene classification dictionary, a template storage unit 18 that includes a microcomputer and stores a template used in a wiper detection process described later, and a color index stored in the color index memory 14 Image data, color histogram Hci (m) generated by the histogram generator 15, scene classification dictionary stored in the scene classification dictionary storage unit 17, template stored in the template storage unit 18, operation mode supplied from the outside Selection data AM, vehicle speed data V, weather data TEN, object index to be described later Based on the stored known or past object index such as Sumemori 20, consisting of processing unit 19 for executing various processes for environmental recognition such as recognition of the detection and the status of the object.

  Here, a description example of the scene classification dictionary stored in the scene classification dictionary storage unit 17 is shown in Table 2.

  The processing unit 19 generates a template used in a stationary object detection process for detecting a stationary object such as a roadside object (for example, a streetlight or a traffic light), a wiper detection process for detecting whether or not the wiper is operating, and a wiper detection process. A template generation process, a 3D map generation process for generating a 3D map based on the color histogram, a caution situation detection process for detecting a situation requiring attention, a forward vehicle detection process for detecting information related to the preceding vehicle, and the like are executed.

  Each of these processes is associated with an operation mode specified from the operation mode selection data AM in advance, and a process to be executed is appropriately selected according to the contents of the operation mode selection data AM.

  The operation mode selection data AM may be input by the driver of the vehicle or automatically generated according to the situation recognized by the device or other in-vehicle device. May be. The vehicle speed data V may be obtained from the output of a vehicle speed sensor mounted on the vehicle, or may be obtained from a navigation device, a GPS receiver, or the like.

Here, the detail of each process which the process part 19 performs is demonstrated.
[Still object detection processing]
First, the stationary object detection process will be described with reference to the flowchart shown in FIG.

  When this process is activated, first, a target object (stationary object) to be recognized is selected (S110). Specifically, when a scene is designated, one of objects corresponding to the designated scene is set as a target object by searching the scene classification dictionary stored in the scene classification dictionary storage unit 17. If no scene is specified, any one of the objects to be recognized is selected in order (for example, in descending order of importance) without using the scene classification dictionary. .

  The scene may be specified based on various information input from the outside, for example, information on the current position, operation mode selection data AM, weather data TEN, etc. It may be specified based on an object recognized as a result of the process or an object recognized by a higher-level recognition process executed later in the apparatus.

  Next, various parameters are set according to the target object selected in S110 (S120). The parameters set here are an accumulation range R that defines a range for accumulating the number of macroblocks in the vertical direction of the color information array in the histogram generation unit 15, and a standard distance used for calculating an accumulation frame number K described later. D, a target color ci (i = 1, 2,..., Psp) that determines the type of color index to be processed, and a weight function Wci used when calculating an evaluation value Sci for each color to be described later. .

  That is, the cumulative range R is present when the target object is likely to exist within a predetermined range in the vertical direction (height direction) on the screen, such as a traffic light or a road sign. This is for generating a color histogram only in an area where there is a high possibility of the color histogram. Therefore, the total range in the vertical direction is the cumulative range R for objects that have no particular feature in the height direction.

  The standard distance D is set based on the average arrangement interval when the target object repeatedly appears like a streetlight or a traffic light, and there are cases where there is no such specific repeating pattern. Set to a constant value.

  As the target color ci, a Psp color characteristic of the target object (an integer of 1 ≦ Psp ≦ P) is set. For example, when the target object is a traffic light, {green, yellow, red} is set as the attention color c1, c2, c3, and when the target object is a vehicle tail lamp, {red} is attention. Set as color c1.

  The weighting function Wci is set so that the weight increases as the position where the target object exists is higher for each of the horizontal positions (macroblock horizontal coordinates m) on the screen. When there are a plurality of attention colors ci, different weight functions are set for the attention colors ci. However, for the same target object, the same weight function Wci may be used for all the target colors ci.

  Next, among the parameters set in S120, when the histogram generation unit 15 generates the color histogram Hci (m) using the accumulation range R, a range for accumulating the number of macroblocks in the vertical direction is set. (S130).

Then, the vehicle speed V is acquired (S140), the acquired vehicle speed V is compared with a preset lower limit value Vmin (S150), and if the vehicle speed V is equal to or higher than the lower limit value Vmin, the equation (5) is used. The cumulative frame number K corresponding to the vehicle speed V is calculated (S160). If the vehicle speed V is smaller than the lower limit value Vmin, the cumulative frame number K is calculated regardless of the vehicle speed V using the equation (6) (S170).
K = D × F / V (5)
K = F × Tconst (6)
However, K is represented by an integer obtained by rounding up the decimal point. Further, F [frm / s] is a frame rate of video processing, and Tconst [s] is a fixed time. For example, it is set to an average stop time when the traffic light is a red signal at an intersection. In this case, the lower limit value Vmin is set so that Tconst = D / Vmin.

  That is, in the equation (5), the number of I frames processed during the time (D / V) required for the vehicle to travel the standard distance D is obtained as the cumulative frame number K. However, when the traffic light stops at an intersection due to a red light, or when the vehicle speed V becomes extremely small due to traffic jams or the like, the cumulative number of frames K calculated by equation (5) becomes very large. In order to cause an unacceptable processing delay, the cumulative number of frames K is obtained regardless of the vehicle speed V using equation (6).

  Next, the color histograms Hci (k, m) for all the target colors ci generated by the histogram generation unit 15 are captured for each K frames (S180). Then, based on the captured color histogram Hci (k, m) and the weighting function Wci (m) set in S120, the evaluation value Sci for each color for each target color ci is calculated using the equation (7). Calculate (S190). However, k in Hci (k, m) indicates a color histogram based on the kth frame.

  Further, in this S190, the integrated evaluation value S is calculated by adding all the evaluation values Sci for each color calculated for each of the attention colors ci as shown in the equation (8) (S200).

  Then, the integrated evaluation value S and an evaluation threshold value Aob set in advance according to the target object are compared (S210). If the integrated evaluation value S is equal to or greater than the evaluation threshold value Aob, the target object selected in S110 is an image. On the other hand, if the integrated evaluation value S is smaller than the evaluation threshold Aob, a recognition result indicating that the target object does not exist in the image is generated (S230). The process ends.

  As the cumulative number of frames K increases, a larger delay occurs until the target object recognition result is obtained. Therefore, a coefficient α (0 <α ≦ 1) is set. In S180 and S190, instead of K, In addition to using α × K, the evaluation threshold may also be set to use α × Aob instead of Aob to suppress delay.

  In this stationary object detection process, for example, when the target object selected in S110 is a streetlight, only one red color is set as the target color ci, and the weight function Wci (m) is shown in FIG. As shown in the figure, the one set so that the weight increases from the center of the screen to the right side (see GAI in the figure) is used. The standard distance D is set to a standard installation interval of street lamps.

  That is, the streetlight that illuminates the road is usually located at the upper part of the center of the road represented by a median strip as a three-dimensional position. In Japan, vehicles are generally left-hand traffic. As a result, in the video of the in-vehicle camera 1 at night, this street light spreads the red color component from the horizontal coordinate center of the screen to the right side, and the number of macroblocks to which the red index is assigned increases.

  In addition, while traveling, each street light passes through in an instant. Therefore, when the red color histogram Hci (k, m) is viewed in time series, the peak of the color histogram appears continuously before and after the street light. (Refer to P1, P2, and P4 in FIG. 13). In addition, since the cumulative number K of frames is set based on the standard distance D set as the standard installation interval of street lamps, this process is executed at any timing if there is a street lamp. However, the existence of the streetlight can be recognized almost certainly.

  In this stationary object detection process, for example, when the target object selected in S110 is a traffic light, three colors of red, yellow, and green are set as the target color ci, and the weight function Wci (m ), As shown in FIG. 5, the one set so that the weight increases from the center of the screen to the left side (see SIG in the figure) is used. In addition, the standard distance D is set to a standard installation interval of intersections in an urban area. Further, since the position of the traffic signal in the height direction is substantially constant, the cumulative range R is also set according to the height.

  When the accumulation range R is set in this way, the component of the target color ci (red, yellow, green) generated by an object other than the traffic light located outside the accumulation range R, that is, a misidentification factor is surely eliminated. Signal recognition accuracy is improved.

  Here, all of red, yellow, and green are set as the attention color ci, but only one of the colors may be set as the attention color ci. However, it goes without saying that the use of a plurality of colors can improve the reliability.

In addition, when these street lights and traffic lights are continuously detected as target objects, vehicles such as passage of intersections, stop at intersections (see P3 and P5 in FIG. 13), occurrence of traffic jams, etc., due to changes in the detection conditions. It is also possible to recognize the driving situation.
[Needs attention detection processing]
Next, the caution situation detection processing will be described along the flowchart shown in FIG.

  In this process, first, the process waits until a color histogram Hci (k, m) for a new frame k is acquired from the histogram generation unit 15 (S310). When the color histogram Hci (k, m) is acquired, equation (9) is obtained. Accordingly, a critical evaluation value Sca is calculated by adding the inter-frame differences from the previously acquired color histogram Hci (k-1, m) for all colors (S320).

  Then, the calculated caution required evaluation value Sca and a preset screen sudden change determination threshold THca are compared (S330), and if the caution required evaluation value Sca is equal to or greater than the screen sudden change determination threshold THca, the entire screen is suddenly changed. A notice of caution is output indicating that there is a noticeable change and the situation requires caution (S340).

Thereafter, the presence / absence of rainfall is determined based on the weather data TEN input from the outside (S350). If it is determined that there is no rain, the process returns to S310.
On the other hand, if it is determined that there is rain, the component of the acquired color histogram Hci (k, m) is set to the red (ci = r) histogram Hr (k, m) according to the equation (10). The red screen total value SAr (k) is calculated by adding all (S360).

Further, assuming that the preset number of frames is Kne, the sum of the current value SAr (Kne) of the red screen total value SAr (k) and the weighted cumulative value of the interframe difference over the number of frames Kne is calculated according to the equation (11). Thus, the approach evaluation value Sne is calculated (S370). However, in consideration of safety, regarding the weight coefficient Wk of the inter-frame difference, when the inter-frame difference value is negative, Wk is made smaller or zero than when it is positive. When positive, it can be considered to increase Wk as k is closer to Kne.

  Then, the calculated approach evaluation value Sne is compared with a preset approach determination threshold value THne (S380). If the approach evaluation value Sne is smaller than the approach determination threshold value THne, the process directly returns to S310, and the approach evaluation value Sne is approached. If it is equal to or greater than the determination threshold THne, a notice of caution that indicates a caution situation in which the vehicle immediately approaches the preceding vehicle is output (S390), and the process returns to S310.

  In other words, a large change in the entire screen can be caused by a sudden change in the direction of travel of the vehicle or a sudden appearance of an obstacle near the vehicle. The degree of overall change is obtained from the cautionary evaluation value Sca.

In addition, when it rains, the red color histogram Hr (k, m) reacts sensitively to the lighting of the stop lamp of the preceding vehicle due to diffused light on the windshield due to raindrops. Since the sum of the number of red blocks increases significantly, the degree of increase is obtained from the approach evaluation value Sne.
[Template generation processing]
Next, template generation processing used in the wiper detection processing described later will be described with reference to the flowchart shown in FIG.

The template represents the shape of the wiper that may appear in an image taken by the in-vehicle camera 1 during the operation of the wiper.
Further, at the time of starting this processing, photographing is performed with the in-vehicle camera 1 in a state where the wiper is operated in a place with a bright background, and image data for template generation obtained by this photographing (see FIG. 8A). Is stored in the color index memory 14.

  When this process is started, first, color arrangement information based on the template generation image data is read from the color index memory 14 for a plurality of frames (here, Kgen) (S410).

  Based on the read color arrangement information, as shown in FIG. 8B, the label “1” is assigned to the macroblock to which the color information corresponding to the black color index is assigned for each frame. A binary array Bshot (k, m, n) in which the label “0” is assigned to the macroblock to which the color information is assigned is generated (S420). Here, k is the frame identification number, m is the horizontal coordinate of the macroblock, and n is the vertical coordinate of the macroblock. The binary array Bshot (k, m, n) can also be regarded as an L (= M × N) dimensional vector.

  Here, in the binary array Bshot (k, m, n), the identification address for identifying the macroblock located at the left end and the top end of the array (screen) is i, and the address of the macroblock located at the left end (m , N) = (1,1), (1,2),... (1, N) correspond to the identification addresses i = 1, 2,... N, and the macroblock address (1,1, N), (2, N),... (M, N) correspond to identification addresses i = N, N + 1,.

  Also, in the binary array Bshot (k, m, n) of the frame k, among the macroblocks identified by the identification number i, the one with the label “1” and the smallest identification number i (near the lower end or An edge address Iedge (k) is set at the left end.

  Then, the binary array Bshot (k, m, n) generated in S420 is regarded as an L (= M × N) -dimensional vector Bshot (k), and the edge address Iedge ( By adding the vectors Bshot (k) having the same value of k), an L-dimensional vector Bw (i) is obtained for each identification address i (S430).

  The vector Bw (i) is also described as the addition value array Bw (i, m, n) in the same manner that the vector Bshot (k) can be described as the binary array Bshot (k, m, n). be able to.

  The values of the respective addition value arrays Bw (i, m, n) obtained in this way are normalized by a preset upper limit value bmax, and the normalized value and a preset threshold value bth (≦ bmax), the value of each macroblock is binarized (S440). Specifically, a macroblock whose normalized value is greater than or equal to the threshold value bth (≦ bmax) is assumed to be part of the wiper, and its value is rewritten to “1”, and the normalized value is greater than the threshold value bth. Small macroblocks are not part of the wiper and their values are rewritten to “0”.

  In this way, the added value array Bw (i, m, n) in which the values of the respective macroblocks are binarized is stored in the template storage unit 18 as the template Bw (i, m, n) (S450). ), This process is terminated.

Thereby, as shown in FIG. 8C, M + N−1 templates Bw (i, m, n) are automatically generated.
[Wiper detection processing]
Next, the wiper detection process will be described with reference to the flowchart shown in FIG.

  When this processing is started, first, based on the color information array of frame k read from the color index memory 14, a binary array Bshot (k, m, n) is generated in the same manner as the processing in S420 above ( S510).

  Evaluation value Q (i) for evaluating the degree of approximation with the binary array Bshot (k, m, n) generated in S510 for all templates Bw (i, m, n) stored in the template storage unit 18 , K) is calculated according to equation (13) (S520).

  However, Ww (i, m, n) is a load coefficient set according to the position of the template and the macroblock, so that the weight of the macroblock located at the left end and the upper end of the screen is higher than other portions. Is set to

  Among the evaluation values Q (i, k) calculated in S520, the evaluation value Q (i, k) having the maximum value (the one with the highest degree of approximation) is extracted as the maximum value Q (imax, k) (S530). The maximum value Q (imax, k) is compared with a preset threshold value Qth (S540). If the maximum value Q (imax, k) is equal to or greater than the threshold value Qth, the template Bw A recognition result indicating that a wiper is present at the position specified by (imax) is generated (S550). If the maximum value Q (imax, k) is smaller than the threshold value Qth, no wiper is present in the image of frame k. A recognition result to that effect is generated (S560). These determination results are stored in the memory for the past Kkeep frames.

  Next, based on the determination result stored in the memory in S560, the frequency Nex determined that the wiper is present in the past K frames is obtained according to the equation (14) (S570), and the frequency Nex is preset. The operation determination threshold value Nth is compared (S580). If the frequency Nex is equal to or greater than the operation determination threshold value Nth, the wiper generates a recognition result indicating that the wiper is operating (S590), and the frequency Nex is smaller than the operation determination threshold value Nth. For example, the wiper generates a recognition result indicating that the wiper is stopped (S600), and the process ends.

  The operation determination threshold value Nth is, for example, α as a constant (0.5. ≦ α ≦ 1.0), fw as the wiper operating speed [number of reciprocations / second], and F as the image frame rate [frame / second]. , (14) can be used.

Nth = α · K · fw / {F · (M + N−1)} (14)
This makes it possible to automatically detect the presence or absence of the wiper operation only from the video even if the input signal from the wiper control device cannot be obtained, and to specify the position of the wiper for each frame. Is possible.
[Front vehicle detection processing]
Next, a front vehicle detection process is demonstrated along the flowchart shown in FIG.

When this process is started, first, based on the color information array read from the color index memory 14, color information corresponding to the red color index is assigned, and a pair of macroblocks MB1 and MB2 arranged in the horizontal direction are extracted as a stop lamp candidate (S710), it sets the parameter L for specifying the predetermined vehicle distance DC _L to 1 (S720).

In the present embodiment, six levels of {5 m, 10 m, 20 m, 50 m, 100 m, 200 m or more} are set as the inter-vehicle distance DC _L. Further, in accordance with the inter-vehicle distance DC _L, it is also set in advance in the vehicle width DW _L on the screen of the vehicle in its vehicle distance DC _L.

Then, based on the macroblock MB1, MB2 extracted as a stop lamp candidate in S710, assume the vehicle in the vehicle width DW _L around the center position of the horizontal direction (S730). In the following description, the horizontal coordinates of the macroblocks MB1 and MB2 are mb1 and mb2, and one of the assumed horizontal coordinates of the macroblock located at the end of the vehicle (left side in the present embodiment) is used as the reference position. Let mb0.

Then, the lamp position di _L (i = 1, 2), which is the position of the stop lamp candidate with respect to the reference position mb0, is calculated by the equation (15) (S740), and further, the lamp position di _L and the vehicle width DW _L Based on the above, the evaluation value Ri _L is calculated by the equation (16) (S750).

di _L = mbi−mb0 (15)
Ri _L = di _L / DW _L (16)
Thereafter, increments the parameter L for specifying the inter-vehicle distance _{DC L (L ← L + 1} ) to (S760), the parameter L is the number of stages Lmax headway distance DC _L (= 6) to determine a larger or not (S770). If the parameter L is equal to or less than the number of steps Lmax, the process returns to S730 and the above-described processes of S730 to S770 are repeatedly executed.

On the other hand, is greater than the parameter L is the number of stages Lmax, based on the evaluation value d1 _L, d2 _L obtained for each inter-vehicle distance, the closest evaluation value d1 _L 0, or to extract the nearest evaluation value d2 _L to 1 According to the parameter L given to the extracted evaluation value di _L , the inter-vehicle distance specified by the parameter L is estimated as the inter-vehicle distance to the preceding vehicle having a stop lamp at the position of the stop lamp candidate, Object information representing the effect is generated (S780).

  Furthermore, the luminance value (Y component) of the macroblocks MB1 and MB2 extracted as the stop lamp candidates is acquired, and the luminance value is larger than the lighting threshold value by comparing the luminance value with a preset lighting threshold value. For example, object information indicating that the tail lamp is turned on is generated, and if the luminance value is equal to or lower than the lighting threshold, object information indicating that the tail lamp is turned off is generated (S790).

  Also, the color information of the macroblocks located above and below the center position of the macroblocks MB1 and MB2 is acquired, and the object information representing that is generated by using the color information as the color of the preceding vehicle (S800). finish.

This process is repeatedly executed until all stop lamp candidates existing in the color information array are extracted.
In this way, in this process, a simple process of extracting a macroblock to which color information corresponding to the red index on the color information array is assigned and comparing it with a virtual vehicle makes it possible to perform complicated processing such as segmentation. A forward vehicle can be detected without executing a process with a large amount of processing.
[3D map generation processing]
In this process, the distance moved during the generation period of the color histogram Hci (m) is calculated based on the vehicle speed V and the frame rate F, and the color histogram Hci (m) is arranged according to the calculated distance. A three-dimensional color solid in which the horizontal position in the image, the value obtained by integrating the macroblocks to which the color information corresponding to the color index of interest is added in the vertical direction in the image, and the distance are assigned to the three axes. Construct a map.

  That is, a three-dimensional color three-dimensional map can be generated by simply arranging the color histograms Hci (m) generated by the histogram generation unit 15, but in this case, the color histograms Hci (m) are arranged at regular time intervals. Therefore, unless the vehicle speed V is always constant, the arrangement of the color histogram in the time axis direction (vehicle traveling direction) does not correspond to the actual distance and is inaccurate as a map.

  However, in the color three-dimensional map generated by this processing, since the color histogram is arranged according to the distance, each color object with respect to the traveling direction of the vehicle (space corresponding to the light source and the block pixel affected by the light source) It is possible to correctly grasp the rough distribution of the.

  A color index (color three-dimensional map) may be mapped on the map by combining the color three-dimensional map generated by this processing and map data obtained from an external device such as a car navigation system. In this case, it is possible to generate a color three-dimensional map with improved direction and position accuracy.

  Next, returning to FIG. 1, the environment recognition apparatus 3 further stores an object index memory 20 that stores recognition results such as objects and situations, which are processing results in the color application execution unit 16, in association with image data (frames). Similarly, based on the three-dimensional map memory 21 that stores the three-dimensional map that is the processing result (three-dimensional map generation processing) in the color application execution unit 16, the stored contents of the object index memory 20, and the object related information from other devices, Based on the stored contents of the object index memory 20 based on the contents stored in the object index memory 20 based on the contents stored in the object index memory 20, the situation description describing the situation around the vehicle is automatically generated. The situation description generator 23 to be generated, the image data from the in-vehicle camera 1, and the color index memory 14 are stored. Information, on the basis of the recognition result of the object stored in the object index memory 20, in accordance with the display mode selection data DM, and a display image data generating unit 25 for generating image data for display.

  Among these, the recognition results stored in the object index memory 20 are the specific objects such as the preceding vehicle, the oncoming vehicle, the road sign, the intersection, the building, the sky, and the coast (the stationary object detection process, the preceding vehicle described above). An object index representing an object (including objects detected by the detection process) and a state recognized from the characteristics of the entire scene or a part thereof such as weather, traffic jam, danger, etc. (the wiper operating state detected by the above-described wiper detection process) Etc.) and an attribute index representing.

  The abnormal situation detection unit 22 includes, for example, intersection information obtained from map data of the navigation device, information indicating the operation state of the operation switch of the wiper, and the like as object related information from other devices. If the detection result is different from the detection result in the stationary object detection process or the wiper detection process, it is indicated that an abnormal situation has occurred in either the apparatus 3 or the apparatus that has supplied the object-related information. An error notification is output.

In the display image data generation unit 25, for example, color index image data, object index image data, and the like are generated according to the display mode selection data DM.
Of these, the color index image data is extracted from the color information array stored in the color index memory 14 as a macroblock to which color information corresponding to the designated color index is assigned, and the extracted macroblock is used as the in-vehicle camera. The image data is used when the image data from 1 is superimposed on the image data.

  The object index image data is image data used when the recognition result and the situation stored in the object index memory 20 are displayed over the image data from the in-vehicle camera 1.

The image data generated by the display image data generation unit 25 is displayed on a monitor connected to the environment recognition device 3 directly or indirectly via a communication line (such as an in-vehicle LAN).
The situation description generated by the situation description generation unit 23, the three-dimensional map data stored in the three-dimensional map memory, and other various notifications (danger notification, abnormality notification, caution notice) are, for example, more advanced images. When executing recognition processing, it is used as knowledge data or the like for removing misrecognition factors.

  In addition, the situation description and the three-dimensional map data are transmitted to another vehicle, a base station that collects the situation description, and the like via a communication device mounted on the vehicle, and used as basic data for various applications. Various notifications are used as notifications to the driver, triggers for starting vehicle control corresponding to the notification contents, and the like.

  As described above, in the present embodiment, color information is assigned in units of macroblocks instead of in units of pixels, and color segmentation information and color histograms expressed in units of macroblocks are used, so that region division (shape acquisition) is performed. It is designed to recognize objects and situations without having to.

  Therefore, according to the environment recognition device 3 of the present embodiment, the processing load for recognizing objects and situations can be greatly reduced as compared with a conventional device that assigns color information in pixel units and divides a region. It is possible to easily recognize objects and situations in a color image.

  In this embodiment, image data encoded in the MPEG format is input to the environment recognition device 3. For this reason, the block average color calculation unit 11 can easily determine the block average color simply by decoding the DC component of the DCT coefficient included in the image data, and greatly reduces the processing for determining the average color. can do.

  Further, in the present embodiment, when the histogram generation unit 15 generates a color histogram, a range in which the macroblocks to which the color information corresponding to the color index of interest is added is integrated in the vertical direction can be arbitrarily set. Has been.

  For this reason, according to the environment recognition device 3 of the present embodiment, the influence of other objects located at a height where the target object cannot exist and having the same color as the target object can be reliably excluded from the color histogram. And the recognition accuracy of the target object using the color histogram can be improved.

  In this embodiment, the average color of the macroblock is changed between frames, and the color histogram Hci (k) and the screen total value SAr (k) of the color of interest are calculated between frames. In addition, the degree of danger and the situation requiring attention are recognized without segmentation (shape acquisition). Therefore, this kind of urgent situation can be recognized quickly, and by using the recognition result, the controllability and safety of the vehicle can be improved.

  In the present embodiment, the recognition result based on the color arrangement information and the color histogram is compared with the target related information obtained from the external device, and it is assumed that an abnormal situation has occurred when there is a contradiction between the two. Therefore, the reliability of the recognition result can be further improved.

  In the present embodiment, the block average color calculation unit 11 is an image data acquisition unit and an average color calculation unit, the color determination unit 13 is a color information addition unit, the histogram generation unit 15 is a color histogram generation unit, and the color application execution unit 16 is object information. The assigning unit, the scene classification dictionary storage unit 17 corresponds to the scene classification dictionary storage unit, the template storage unit 18 corresponds to the template storage unit, the abnormal state detection unit 22 corresponds to the abnormal state determination unit, and the risk level calculation unit 24 corresponds to the risk level calculation unit.

Also, S110 is an object specifying means, S120 to S190 are color-specific evaluation value calculating means, S320 is a first difference calculating means, S330 to S340 are first caution state detecting means, S360 is a full screen value generating means, and S370 is the first. Two-difference calculation means, S380-S390 second caution status detection means, S420-S430 are cumulative addition means, S440-S450 are template generation means, S510 is binarization means, S520-S560 are wiper detection means, S570 is frequency Calculation means, S710 corresponds to candidate extraction means, S720 to S780 correspond to inter-vehicle distance estimation means, S790 corresponds to lighting determination means, and S800 corresponds to vehicle color extraction means.
[Second Embodiment]
Next, a second embodiment will be described.

  In the first embodiment, the stationary object detection process for detecting an object using the color histogram is realized as one of the processes executed by the processing unit 19 of the color application execution unit 16, but in the present embodiment, this process is performed. Are executed in exactly the same manner except that they are executed by a dedicated processor capable of vector operation. For this reason, in this embodiment, only a part different from the first embodiment will be described.

  That is, in the present embodiment, the color application execution unit 16 includes a processor 30 that receives the target object X $, the frame rate F, the vehicle speed V, and the color histogram matrix H and outputs the presence / absence of the target object.

  Note that the color histogram matrix H is an M-row P-column matrix configured by arranging color histograms Hc1 to HcP for all color indexes generated by the histogram generation unit 15 based on image data of a single frame ( (See equation (17)). The color histogram matrix H generated for each frame is sequentially supplied from the histogram generation unit 15.

H = [Hc1, Hc2,..., Hcp] (17)
Similarly to the processing in S110 of the stationary object detection process in the first embodiment, the target object X $ is stored in the scene classification dictionary stored in the scene classification dictionary storage unit 17 when a scene is designated. By searching, one of the objects corresponding to the specified scene is selected as a target object. If no scene is specified, all objects that should be recognized are identified without using the scene classification dictionary. One of them is selected in order (for example, in descending order of importance).

  Then, based on the target object X $, the processor 30 generates the load coefficient matrix W and the distance vector VD corresponding to the target object X $, and the variable element generation unit 31 generates The monitoring unit setting unit 32 for generating the period vector VK according to the distance vector VD and the frame rate F and the vehicle speed V supplied from the outside, and the matrix operation of the color histogram matrix H and the load coefficient matrix W are executed to evaluate the unit vector The matrix calculation unit 33 that generates HV and the elements of the evaluation unit vector HV generated by the matrix calculation unit 33 are accumulated for one or more frames according to the contents of the period vector VK set by the period setting unit 32. And the accumulating unit 34 for generating the color-specific evaluation vector VS, and each element (color-specific The addition unit 35 that generates the integrated evaluation value S by adding the value Sci), and the presence / absence of the target object by comparing the integrated evaluation value S generated by the addition unit 35 with a predetermined determination threshold value And a determination unit 36 for outputting the determination result.

  However, the load coefficient matrix W is a P row in which load coefficient vectors wci (see equation (18)) in which M coefficients w1 to wM corresponding to the horizontal coordinates of the macroblock are arranged are arranged for all color indexes. It consists of an M-column matrix (see equation (19)).

wci = (w1, w2,..., wM) ^T (18)
W = [wc1, wc2,..., WcP] (19)
The distance vector VD is a P-dimensional column vector (see equation (20)) in which standard distances Dc1 to DcP set for each color index are arranged.

VD = (Dc1, Dc2, ... , DcP) T (20)
The period vector VK is composed of a P-dimensional column vector in which the monitoring periods Kc1 to KcP calculated for each color index are arranged, and is calculated by Expression (21).

VK = (Kc1, Kc2,..., KcP) ^T = VD × F / V (21)
The color-specific evaluation vector VS includes a P-dimensional column vector (see (22)) in which the color-specific evaluation values Sc1 to ScP calculated for each color index are arranged.

VS = (Sc1, Sc2,..., ScP) ^T (22)
That is, in the stationary object detection process described above, S110 to S120 are the variable element generation unit 31, S140 to S170 are the period setting unit 32, S180 to S190 are the matrix calculation unit 33 and the accumulation unit 34, and S200 is the addition unit 35, S210. S230 corresponds to the determination unit 36.

  As described above, the processor 30 collectively calculates the evaluation value for each color (each element of the evaluation vector for each color VS) Sci by vector calculation, and the distance vector VK and the load coefficient matrix W that are different for each target object X $. Is automatically generated and reflected in the vector calculation.

  Therefore, according to the environment recognition apparatus 3 of the present embodiment, even if the target object X $ has different standard distances Dci, weighting factors wci, and the number and types of color indexes Sci necessary for calculating the integrated evaluation value S, These can be handled uniformly by the processor 30, and the processing necessary for object recognition can be simplified.

In the present embodiment, the variable element generation unit 31 corresponds to a variable element generation unit, the monitoring period setting unit 32 corresponds to a period vector generation unit, the matrix calculation unit 33 corresponds to a matrix calculation unit, and the accumulation unit 34 corresponds to an accumulation calculation unit.
[Other Embodiments]
As mentioned above, although several embodiment of this invention was described, this invention is not limited to the said embodiment, In the range which does not deviate from the summary of this invention, it is possible to implement in various aspects. is there.

  In the above embodiment, the encoding unit 2 performs encoding in the MPEG format. However, any encoding format can be used as long as encoding is performed so that one component represents the average color of the block. Encoding may be performed.

  In addition, when the encoding unit 2 is configured to perform encoding in an encoding format that does not have a component that represents an average color, the block average color calculation unit 11 includes a macroblock (16 pixels × The average color may be calculated in block units corresponding to (16 pixels).

  In the above embodiment, the vehicle speed V is obtained from another device such as a GPS or a vehicle speed sensor, but the vehicle speed V may be obtained by image processing, for example. In this case, the vehicle speed V can always be calculated independently by the present apparatus regardless of the fluctuation of the frame rate. However, since the vehicle speed V is calculated from the optical flow, the accuracy of the vehicle speed V decreases. The vehicle speed V may be a fixed value such as an average vehicle moving speed (for example, 50 km / h).

The block diagram which shows schematic structure of the vehicle-mounted system to which the environment recognition apparatus was applied. Explanatory drawing regarding calculation of the average color for every macroblock. Explanatory drawing regarding the production | generation of a color histogram. The flowchart which shows the content of a stationary object detection process. Explanatory drawing regarding the object recognition using a color histogram, and a color solid map. The flowchart which shows the content of the attention required condition detection process. The flowchart which shows the content of a template production | generation process. Explanatory drawing regarding the production | generation of a template. The flowchart which shows the content of the wiper detection process. The flowchart which shows the content of a front vehicle detection process. Explanatory drawing regarding front vehicle detection. The block diagram which shows the structure of the processor which performs the object recognition using a color histogram uniformly. A three-dimensional graph in which red color histograms generated during night driving are arranged in time series.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Car-mounted camera, 2 ... Encoding part, 3 ... Environment recognition apparatus, 10 ... Color information generation part, 11 ... Block average color calculation part, 12 ... Color dictionary memory | storage part, 13 ... Color determination part, 14 ... Color index memory 15 ... Histogram generator, 16 ... Color application execution unit, 17 ... Scene classification dictionary storage unit, 18 ... Template storage unit, 19 ... Processing unit, 20 ... Object index memory, 21 ... 3D map memory, 22 ... Abnormal situation detection , 23 ... Situation description generation unit, 24 ... Risk level calculation unit, 25 ... Display image data generation unit, 30 ... Processor, 31 ... Variable element generation unit, 32 ... Monitoring period setting unit, 32 ... Period setting unit, 33 ... Matrix calculator, 34... Accumulator, 35... Adder, 36.

Claims (20)

An object designating means for designating a target object that is an object to be recognized;
Image data acquisition means for acquiring image data based on a color image in front of the vehicle imaged by the in-vehicle camera;
Average color calculation means for calculating an average color of the pixel block for each pixel block obtained by dividing the color image by a preset size based on the image data acquired by the image data acquisition means;
Color information providing means for determining which of the color indexes prepared in advance the average color calculated by the average color calculating means belongs, and storing the determination result as color information in association with the pixel block; ,
The number of the pixel blocks in which the color information matched by the color information providing unit and the color index of interest match is integrated along one direction on the color image, and the integrated values are arranged one-dimensionally. A color histogram generating means for generating a color histogram in time series for each color index and in units of frames of the image data;
Based on the color histogram generated by the color histogram generation means, information on the target object specified by the object specification means is generated, and the generated information is used as object information corresponding to the image data acquired by the image data acquisition means Object information giving means for storing the information;
An environment recognition apparatus comprising:   The direction in which the color histogram generation unit accumulates the number of pixel blocks is the vertical direction in the color image, and each element of the color histogram is associated with a horizontal position orthogonal to the vertical direction. The environment recognition apparatus according to claim 1, wherein:   The environment recognition apparatus according to claim 1, wherein the color histogram generation unit sets a range of pixel blocks to be integrated according to the target object. For each of the color histograms generated by the color histogram generation means, the value increases as the position where the target object is likely to exist among the horizontal positions corresponding to the elements of the color histogram. A weighted addition of the elements of the color histogram is executed using the weighting factor set in the above, and the result of adding the calculation result of the weighted addition only during the monitoring period set according to the target object is classified by color. Equipped with color-specific evaluation value calculation means for calculating as an evaluation value,
The object information providing unit uses the sum of the evaluation values for each color calculated by the evaluation value calculation unit for each color as an integrated evaluation value, and the color image is obtained when the integrated evaluation value is larger than a preset evaluation threshold value. The environment recognition apparatus according to claim 1, wherein it is determined that the target object is present therein, and the determination result is used as the object information. The color-specific evaluation value calculation means includes:
The monitoring period is set to a value inversely proportional to the vehicle speed when the vehicle speed of the vehicle on which the device is mounted is equal to or greater than a preset lower limit value, and is set to a preset fixed value when the vehicle speed is less than the lower limit value. The environment recognition device according to claim 4, wherein the environment recognition device is set to a value. The color-specific evaluation value calculation means includes:
A load coefficient obtained by arranging standard distances set according to objects in association with the color indexes as distance vectors, and arranging the load coefficients in correspondence with the elements of the color histogram. Variable element generation means for generating the distance vector and the load coefficient matrix suitable for the target object designated by the object designation means, using a vector in which the vectors are arranged in association with the color indexes as a load coefficient matrix When,
The monitoring period is associated with each of the color indexes as a period vector, and the distance vector generated by the variable element generation unit, the frame rate of the color image, and the vehicle speed of the vehicle on which the apparatus is mounted. A period vector generating means for generating the period vector based on;
The color histogram generated by the color histogram generation unit is arranged in association with each of the color indexes as a color histogram matrix, and the load generated by the color histogram matrix and the variable element generation unit Matrix calculation means for calculating an evaluation unit vector formed by arraying the weighted addition values corresponding to each of the color indexes by performing matrix calculation on a coefficient matrix;
By accumulating each element of the evaluation unit vector calculated by the matrix calculation means according to the contents of the period vector generated by the period vector generation means, the evaluation value for each color corresponds to each of the color indexes. Accumulating operation means for generating an evaluation vector for each color,
With
The said object information provision means calculates the integrated evaluation value of the said target object by adding each element of the evaluation vector classified by color produced | generated by the said accumulation operation means, The said object object is characterized by the above-mentioned. Environment recognition device. Scene classification dictionary storage means for storing a scene classification dictionary that stores a plurality of scenes representing typical scenes of the color image in association with the scene and objects that are likely to exist in the scene. Prepared,
The object specifying means sequentially reads out an object associated with a scene specified from the outside from the scene classification dictionary storage means and specifies it as the target object. The environment recognition device described.
An object designating means for designating a target object that is an object to be recognized;
Image data acquisition means for acquiring image data based on a color image in front of the vehicle imaged by the in-vehicle camera;
Average color calculation means for calculating an average color of the pixel block for each pixel block obtained by dividing the color image by a preset size based on the image data acquired by the image data acquisition means;
Color information providing means for determining which of the color indexes prepared in advance the average color calculated by the average color calculating means belongs, and storing the determination result as color information in association with the pixel block; ,
Based on the color information associated with each of the pixel blocks by the color information providing means, information on the target object specified by the object specifying means is generated, and the image data is acquired using the generated information as object information Object information providing means for storing the image data in association with the acquired image data;
An environment recognition apparatus comprising: Template storage means for storing a plurality of templates representing positions of wipers in the color image;
Binary that generates a binarized image in which different values are given to a pixel block to which color information corresponding to a black color index is given by the color information giving means and a pixel block to which other color information is given And
Wiper detection means for detecting a wiper at a position other than the stop position when not in use by pattern matching between the template stored in the template storage means and the binarized image generated by the binarization means;
A frequency calculation means for calculating the frequency at which a wiper is detected by the wiper detection means during a preset observation period;
With
The object information providing means uses, as the object information, information indicating that the wiper is operating when the frequency calculated by the frequency calculation means exceeds a preset operation determination threshold. The environment recognition apparatus according to claim 8. A binarized image generated by the binarization unit based on image data captured from the image acquisition unit during operation of a wiper of a vehicle equipped with the apparatus is placed on the edge of the binarized image. Using a macroblock address as an identifier, the macroblock specified by the identifier is grouped with a color information corresponding to a black color index, and binarized images belonging to the same group for each group. Cumulative addition means for generating a cumulative addition image obtained by cumulative addition in units of pixel blocks;
For the cumulative addition image generated by the cumulative addition means, different values are assigned to a pixel block having a cumulative addition value equal to or greater than a preset presence determination threshold value and a pixel block having a cumulative addition value smaller than the presence determination threshold value. Generating a template as the template and storing the template in the template storage unit;
The environment recognition apparatus according to claim 9, further comprising: Candidate extraction means for extracting, as a red pixel block, a pixel block to which color information corresponding to a red color index is assigned by the color information giving means, and extracting two red pixel blocks in a horizontal direction as tail lamp candidates; ,
The inter-vehicle distance is estimated by determining which of the vehicle width conditions set in association with each of the plurality of inter-vehicle distances matches the positional relationship of the tail lamp candidates extracted by the candidate extracting means. Inter-vehicle distance estimation means;
With
9. The object information giving means uses the vehicle position represented by the position of the tail lamp candidate and the inter-vehicle distance estimated by the inter-vehicle distance estimating means as the object information. Environment recognition device. Lighting determination means for determining whether or not lighting is based on the brightness of the tail lamp candidate extracted by the candidate extraction means;
The environment recognition apparatus according to claim 11, wherein the object information providing unit uses the determination result obtained by the lighting determination unit as the object information. Vehicle color extraction means for extracting the color of the macroblock located between or above and below the tail lamp candidates extracted by the candidate extraction means as the color of the preceding vehicle;
The environment recognition apparatus according to claim 11 or 12, wherein the object information giving means uses the extraction result of the vehicle color extraction means as the object information.   Characterized by comprising an abnormal situation judging means for judging that an abnormal situation occurs when the object information stored in the object information providing means and the information on the target object supplied from outside the device do not match. The environment recognition device according to claim 1. Image data acquisition means for acquiring image data based on a color image in front of the vehicle imaged by the in-vehicle camera;
Average color calculation means for calculating an average color of the pixel block for each pixel block obtained by dividing the color image by a preset size based on the image data acquired by the image data acquisition means;
Color information providing means for determining which of the color indexes prepared in advance the average color calculated by the average color calculating means belongs, and storing the determination result as color information in association with the pixel block; ,
The number of the pixel blocks in which the color information matched by the color information providing unit and the color index of interest match is integrated along one direction on the color image, and the integrated values are arranged one-dimensionally. A color histogram generating means for generating a color histogram in time series for each color index and in units of frames of the image data;
First difference calculation means for calculating a difference between frames of the color histogram generated by the color histogram generation means for each color index;
First caution situation detecting means for detecting a caution situation appearing on the color screen based on a calculation result in the first difference calculation means;
An environment recognition apparatus comprising: Image data acquisition means for acquiring image data based on a color image in front of the vehicle imaged by the in-vehicle camera;
Average color calculation means for calculating an average color of the pixel block for each pixel block obtained by dividing the color image by a preset size based on the image data acquired by the image data acquisition means;
Color information providing means for determining which of the color indexes prepared in advance the average color calculated by the average color calculating means belongs, and storing the determination result as color information in association with the pixel block; ,
A full-screen integrated value obtained by integrating the number of the pixel blocks whose color information matched by the color information providing means matches the color index of interest over the entire color image is obtained as a frame of the image data. Full-screen integrated value generating means for generating time series in units;
Second difference calculating means for calculating the inter-frame difference of the full screen integrated value generated by the full screen integrated value generating means;
Based on the calculation result of the second difference calculation means, second attention situation detection means for detecting the attention situation that has appeared on the color screen;
An environment recognition apparatus comprising:   The full-screen integrated value generating means generates the full-screen integrated value for a red color index, and the second caution state detecting means indicates that the weather data supplied from the outside is raining And detecting a case where the inter-frame difference of the full-screen integrated value calculated by the second difference calculating means is equal to or greater than a preset approach threshold as the caution situation. Environment recognition device. Image data acquisition means for acquiring image data based on a color image in front of the vehicle imaged by the in-vehicle camera;
Average color calculation means for calculating an average color of the pixel block for each pixel block obtained by dividing the color image by a preset size based on the image data acquired by the image data acquisition means;
The inter-frame difference of the average color of the pixel block calculated by the average color calculating means is integrated for all the pixel blocks belonging to the monitoring area which is one area in the color image specified in advance. An environment recognition apparatus comprising a risk level calculation means for calculating the risk level of an area.   19. The risk level calculation unit, when integrating the inter-frame difference of the average color of the pixel block, performs a weighted addition in which the pixel block closer to the vanishing point of the color image is weighted. The environment recognition device described in 1. The image data acquired by the image data acquisition means is encoded using orthogonal transformation,
The environment recognition apparatus according to claim 1, wherein the average color calculation unit uses a DC component included in the image data as the average color.