JP2005346287A - Image recognizing method and apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image recognizing method and apparatus for stably recognizing an object to be recognized independently of a change in a distance up to the object to be recognized. <P>SOLUTION: The image recognizing apparatus 3 divides image data expressing an image photographed by a camera 5 into a plurality of partial area images and performs image recognition in each partial area image. Namely visibility indicating the degree of clearness of each partial area image is calculated at first. Then a feature value expressing the features of respective pixels constituting the partial area image are extracted and feature data expressing the distribution of the featured values are generated. Then reference data expressing the distribution of featured values corresponding to the calculated visibility are extracted from a reference data group 17a, the extracted reference data are compared with the generated feature data and whether a prescribed object is included in the partial area image or not is decided on the basis of the compared result. Since optimum reference data can be adopted in accordance with the visibility, the prescribed object can be stably recognized even when a distance up to the object is changed. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image recognition method and apparatus for recognizing a predetermined object.

In recent years, image recognition has become important in order to realize safe driving support and human interface in automobiles.
As a technique related to such image recognition, a recognition target (for example, a face) is decomposed into parts (for example, eyes, nose, mouth, etc.), and recognition is performed for each part. On the other hand, a technique for improving the recognition accuracy is disclosed (for example, see Patent Document 1).
Japanese Patent Laid-Open No. 11-283036

  By the way, in the technique of the said patent document 1, since the recognition target by image recognition is limited to the face, the distance to a recognition target object is comparatively small with less than several meters. That is, the appearance of the recognition object hardly changes with the change in the distance to the recognition object. That is, it is not necessary to consider the change in appearance due to distance, and the recognition target (face) can be recognized stably.

  However, in the case of application to vehicle recognition, it is desired to recognize a vehicle (front vehicle) existing between about 100 meters ahead of the host vehicle. That is, since the change in the distance to the recognition target object (front vehicle) is large, the appearance of the recognition target object (front vehicle) changes greatly according to the distance. For this reason, the problem that the precision which recognizes a recognition target object (front vehicle) changes greatly according to this change in appearance.

  The present invention has been made in view of these problems, and an object of the present invention is to provide an image recognition method and apparatus for stably recognizing a recognition object regardless of a change in distance to the recognition object.

  In the image recognition method of the first invention made to achieve the above object, first, in the first step, the sharpness indicating the sharpness of the partial area image is calculated. In the subsequent second step, feature amounts representing the features of each pixel constituting the partial region image are extracted from the partial region image, and feature data representing the distribution of the feature amount is generated. Further, in the third step, the reference data representing the distribution of the feature amount is generated in advance for each of a plurality of reference images in which the predetermined target is set in advance and the sharpness of the predetermined target is different from each other. The reference data corresponding to the sharpness calculated in the first step is selected from the set of reference data groups, the selected reference data is compared with the feature data generated in the second step, and this comparison is performed. Based on the result, it is determined whether or not the predetermined object is shown in the partial area image.

  That is, in the comparison between the feature data and the reference data for recognizing the predetermined object, the optimum reference data can be adopted according to the sharpness. For this reason, according to the image recognition method of the first invention, the object can be recognized stably even if the distance to the predetermined object changes. This is because the sharpness can be used as an index representing the distance to the predetermined object. In this case, for example, when a person visually recognizes an object (for example, a vehicle), the farther the vehicle is located, the more unclear the vehicle is, and it is difficult to determine whether the object is a vehicle. It is clear from that.

  However, the sharpness may vary depending on the shooting environment in which the input image is taken, such as the weather (clear, cloudy, rainy, etc.) and the time (daytime, evening, night, etc.). However, it is desirable to select different reference data depending on the shooting environment.

  Next, in the image recognition apparatus according to the second aspect of the present invention, reference data representing a distribution of feature amounts is generated in advance for each of a plurality of reference images showing a predetermined object set in advance and having different sharpness of the predetermined object. The first storage means for storing the reference data group prepared by the setting is provided, and the sharpness calculation means calculates the sharpness indicating the degree of sharpness of the partial area image. Further, the feature amount extraction means extracts a feature amount representing the feature of each pixel constituting the partial region image from the partial region image, and generates feature data representing the distribution of the feature amount. The first determination means extracts the reference data corresponding to the sharpness from the first storage means based on the sharpness value calculated by the sharpness calculation means, and extracts the extracted reference data and feature amount The feature data extracted by the means is compared, and based on the comparison result, it is determined whether or not the predetermined object is shown in the partial region image.

That is, the image recognition apparatus according to the second invention is an apparatus that implements the method according to the first invention, and thus can obtain the same effects as the first invention.
For example, color information can be used as the feature amount. As this color information, for example, there is information expressed in a well-known YUV color space or RGB color space.

The sharpness calculation means may be configured to perform a two-dimensional Fourier transform on the partial region image and calculate the sharpness based on the spatial frequency distribution obtained by this conversion.
The sharpness calculation means may be configured to perform edge enhancement processing on the partial region image and calculate the sharpness based on the amount of edges included in the image subjected to the edge enhancement processing.

  Next, in the image recognition device according to the third aspect of the invention, reference data representing a distribution of feature amounts is generated in advance for each of a plurality of reference images showing a predetermined object set in advance and having different object distances of the predetermined object. A second storage unit that stores a reference data group prepared by the measurement, and the distance measurement unit measures an object distance, which is a distance between the object indicated in the partial area image and the image recognition apparatus. To do. Further, the feature quantity extraction unit extracts a feature quantity representing the feature of each pixel constituting the partial area image from the partial area image, and generates feature data representing the distribution of the feature quantity. The second determination means extracts the reference data corresponding to the object distance from the second storage means based on the object distance measured by the distance measurement means, and extracts the extracted reference data and the feature quantity extraction means. The feature data is compared, and based on the comparison result, it is determined whether or not the predetermined object is shown in the partial region image.

  That is, in the comparison between the feature data and the reference data for recognizing the predetermined object, it is possible to adopt the optimum reference data according to the object distance. For this reason, according to the image recognition device of the third aspect of the invention, the object can be recognized stably even if the distance to the predetermined object changes.

(First embodiment)
Hereinafter, an image recognition system configured using an image recognition apparatus will be described as a first embodiment of the present invention.

  The image recognition system 1 according to the first embodiment is mounted on a vehicle. As shown in FIG. 1, the foreground in a range where the driver can see through the front window is photographed, and image data representing the photographed image is obtained. It comprises a camera 5 to be generated and an image recognition device 3 that performs image recognition processing on image data generated by the camera 5.

  The image recognition device 3 is used for the CPU 11 that performs processing based on a predetermined processing program, the ROM 13 that stores various control programs, the RAM 15 that stores various data, and the CPU 11 that performs image recognition processing. A user interface (hereinafter referred to as a user I / F) 21 comprising a database 17 for storing a reference data group 17a to be operated, an operation key 21a for an operator to operate, and a display panel 21b for displaying an operation procedure of the operation key 21a. Are connected to the camera 5, the database 17, and the user I / F 21, and includes an input / output unit 19 that inputs and outputs signals and data to and from the CPU 11 and the RAM 15.

The database 17 is a writable storage medium such as a memory card or hard disk.
Then, in the database 17, as shown in FIG. 6, the predetermined number of shots N <b> 2 (this embodiment) taken at different distances for each of the vehicles having a predetermined number N1 (N1 = 5 in this embodiment) of different vehicle types. In the embodiment, N2 = 6) image data (hereinafter referred to as “reference data creation image data”) is stored in advance (only one vehicle is shown in detail in FIG. 6). This reference data creation image data is used to create a reference data group for performing image recognition. Note that the reference data creation image data is obtained by extracting the vehicle area in a rectangular shape from the image taken by the camera 5 and is normalized so that the extracted vehicle images have the same size. In the present embodiment, normalization is performed so that the extracted car image becomes image data of 256 × 256 pixels. Therefore, an image with a larger distance becomes a larger image.

  Further, the vehicle type identification number c and the distance identification number d are assigned to the reference data creation image data according to the vehicle type and the distance. The vehicle type identification number c is a number indicating the vehicle type in the reference data creation image data, and takes an integer value of 1 to 5. The distance identification number d is a number indicating the shooting distance in the image data for creating the reference data, and takes an integer value of 1, 2,..., 6 from the shortest shooting distance. Hereinafter, the reference data creation image data corresponding to the vehicle type identification number c and the distance identification number d is referred to as image data (c, d).

  Further, the user I / F 21 is configured to output a reference data creation command for instructing the start of creation of a reference data group to the CPU 11 based on a predetermined operation on the operation key 21a.

  In the image recognition device 3 configured as described above, the CPU 11 independently executes a reference data creation process for creating the reference data group 17a and an image recognition process for performing image recognition based on the reference data group 17a. Here, the vehicle type identification number is used to generate a reference data group, and does not recognize the vehicle type using the generated reference data group 17a.

  First, the procedure of the reference data creation process will be described with reference to FIGS. FIG. 2 is a flowchart showing the first half of the reference data creation process, and FIG. 3 is a flowchart showing the second half of the reference data creation process. This reference data creation process is repeatedly executed while the vehicle ignition key is inserted into the key cylinder and the key is turned on.

  In this reference data creation process, the CPU 11 first determines whether or not a reference data creation command has been input via the user I / F 21 in S10. If no reference data creation command is input (S10: NO), the reference data creation process is terminated. On the other hand, if a reference data creation command is input (S10: YES), a variable Pc representing the vehicle type identification number c, a variable Pd representing the distance identification number d, and an object identification number k (described later) are obtained in S20. Each representing variable Pk is set to 1.

  Next, in S30, image data (c, d) corresponding to the vehicle type identification number c indicated by the variable Pc and the distance identification number d indicated by the variable Pd is obtained from the database 17, and obtained in S30 at S40. The sharpness of the image data (c, d) is calculated. That is, two-dimensional fast Fourier transform is performed on the image data (c, d), and the sharpness is calculated based on the spatial frequency distribution obtained by the transform.

  Hereinafter, a specific method for calculating the sharpness will be described with reference to FIG. That is, for example, when the image D1 shown in FIG. 7A is acquired as the image data (c, d), the image data indicating the image D1 is also referred to as a two-dimensional fast Fourier transform (hereinafter also referred to as FFT). That is, by performing Fast Fourier Transform), a spatial frequency distribution D2 shown in FIG. 7B is obtained. The horizontal axis of the spatial frequency distribution D2 indicates the change in the intensity of the color information in the horizontal direction of the image D1, and the vertical axis indicates the change in the intensity of the color information in the vertical direction. That is, in the spatial frequency distribution D2, a component having a higher spatial frequency (high frequency component) is arranged away from the center of the spatial frequency distribution D2 (that is, the intersection of the horizontal axis and the vertical axis). In other words, the more components distributed at positions away from the center, the greater the portion where the intensity change is greater, indicating that the image is clearer.

  Thereafter, when the distribution of the frequency of the spatial frequency component with respect to the distance from the center of the spatial frequency distribution D2 is obtained, the frequency distribution D3 shown in FIG. 7C is obtained. The horizontal axis of the frequency distribution D3 is the FFT order. This order is a multiple of the fundamental frequency in the fast Fourier transform, and the higher the order, the higher the spatial frequency. Thereafter, when this frequency distribution D3 is approximated by a normal distribution, a normal distribution D4 is obtained. Then, the sharpness is calculated by setting the variance value of the normal distribution D4 as the sharpness. That is, the sharpness indicates that the higher the value is, the more high frequency components are included.

  Next, in S50, four objects are extracted from the image data (c, d). An object is a characteristic area that appears in common in the preceding vehicle, and is composed of a rear window, a right tail lamp, a left tail lamp, and a license plate. Further, these four objects have a specific positional relationship (hereinafter referred to as a specific positional relationship) between the objects. That is, in a vehicle, generally, a right tail lamp and a left tail lamp are disposed relative to each other below the rear window, and a license plate is disposed below the right tail lamp and the left tail lamp.

  That is, based on the specific positional relationship, as shown in FIG. 8A, a rectangular rear window object O1, right tail lamp object O2, left tail lamp object O3, and license plate object O4 are obtained from the image D11. Extract. Since a method for extracting an object based on a specific positional relationship between objects is well known (for example, refer to Patent Document 1), description thereof is omitted.

  In S60, a color information (YUV) histogram is calculated for each of the rear window object O1, the right tail lamp object O2, the left tail lamp object O3, and the license plate object O4 extracted in S50. Note that YUV is a type of color representation and is well known, and will not be described.

  That is, histograms (Y histogram, U histogram, V histogram) for each of Y, U, and V components are calculated for image data representing an object. That is, as shown in FIG. 8B, a histogram is generated with the horizontal axis representing any one of Y, U, and V and the vertical axis representing the number of pixels. FIG. 8B is an image diagram showing Y, U, and V histograms. The three Y, U, and V histograms are collectively referred to as a YUV histogram. The generated histogram is stored in the RAM 15.

  Thereafter, in S70, it is determined whether or not the value of the variable Pd representing the distance identification number d is equal to or greater than a predetermined number of images N2. Here, when the value of the variable Pd representing the distance identification number d is less than the predetermined number N2 (S70: NO), the value of the variable Pd representing the distance identification number d is incremented by 1 in S80, and then S20. The above process is repeated after moving to step S2. On the other hand, when the value of the variable Pd representing the distance identification number d is equal to or greater than the predetermined number of shots N2 (S70: YES), the process proceeds to S90.

  Then, in S90, for the image data (c, d) with the vehicle type identification number c indicated by the variable Pc, for each object, for example, YUV with a definition m = 22, 24, 26, 28, 30, 32. The histogram is obtained by linear interpolation with the YUV histogram calculated in S60. That is, for example, the sharpness of six images with different shooting distances in the same vehicle type, that is, the sharpness at distance identification numbers d = 1, 2, 3, 4, 5, and 6, respectively, is “32.25” and “30. 79 ”,“ 28.58 ”,“ 26.99 ”,“ 23.63 ”,“ 21.41 ”, for example, in order to obtain a YUV histogram with a sharpness m = 22, the sharpness 21. Linear interpolation may be performed with the 41 YUV histogram and the YUV histogram with a sharpness of 23.63. Then, the YUV histogram of the sharpness m = 22, 24, 26, 28, 30, 32 obtained by linear interpolation is stored in the RAM 15. The YUV histogram of the vehicle type identification number c, the object identification number k, and the definition m is hereinafter referred to as a YUV histogram (c, k, m) (c = 1, 2, 3, 4, 5, k = 1, 2, 3, 4, m = 22, 24, 26, 28, 30, 32). The object identification number k is a number for identifying an object, and the object identification numbers k = 1, 2, 3, and 4 indicate a rear window, a right tail lamp, a left tail lamp, and a license plate, respectively.

  Thereafter, in S100, it is determined whether or not the value of the variable Pc representing the vehicle type identification number c is equal to or greater than the predetermined number of vehicles N1. If the value of the variable Pc representing the vehicle type identification number c is less than the predetermined number of vehicles N1 (S100: NO), the value of the variable Pc representing the vehicle type identification number c is incremented by 1 in S110, and then S20. The above process is repeated after moving to step S2. On the other hand, when the value of the variable Pc representing the vehicle type identification number c is equal to or greater than the predetermined number N1 (S100: YES), the process proceeds to S120.

  In S120, YUV histograms having the same object and the same definition m are added for all vehicle types. For example, since there is one YUV histogram of the rear window object O1 with the definition 22 for each vehicle type identification number c = 1 to 5, these five YUV histograms are added. This added YUV histogram is hereinafter referred to as an integrated YUV histogram. That is, by the process of S120, six integrated YUV histograms are generated for each of the four objects according to the definition m. The integrated YUV histogram having the object identification number k and the definition m is hereinafter referred to as an integrated YUV histogram (k, m) (k = 1, 2, 3, 4, m = 22, 24, 26, 28, 30). , 32).

  Next, in S130, a three-dimensional Gaussian distribution (hereinafter also referred to as GMM, that is, abbreviation of Gaussian Mixtured Model) approximation is performed from the result of the integrated YUV histogram, and the likelihood is calculated.

  Hereinafter, GMM approximation and likelihood will be described. First, the probability density function p (x; Θ) is a weighted linearity of r probability density distributions p (x; θj) (j = 0, 1, 2,...) As shown in the following equation (1). Suppose that it can be modeled by combining. Here, Θ is a parameter vector expressed as Θ = (θ1, θ2,..., Θr), and θj is a parameter of the probability frequency distribution p (x; θj). Such a distribution is called a mixed distribution.

  The probability density distribution p (x; θj) is expressed by a Gaussian distribution having an average uj and a covariance matrix Σj = σj × I as shown in the following equation (2). The distribution represented by the equation (2) is called GMM (Gaussian Mixed Model).

  Further, the weighting coefficient ωj in the equation (1) is called a mixing parameter and satisfies the condition shown in the following equation (3).

  Similarly, each probability density distribution p (x; θj) satisfies the condition shown in the following equation (4).

  The parameters to be estimated are the probability density distribution parameter Θ and the weighting coefficient Ω = (ω1, ω2,..., Ωr). For the parameter estimation method of the mixed Gaussian distribution, a well-known EM algorithm is used, and estimation is performed so that the likelihood expressed by the following equation (5) is maximized. That is, the parameter Θ and weighting coefficient Ω when the likelihood is maximized are calculated. Note that the likelihood is a scale representing how much the distribution of the observation data of interest matches a certain probability theory model. Here, a learning pattern set including n patterns is set to X = (x1, x2,..., Xn).

  That is, the parameter Θ and the weighting coefficient Ω are calculated for 24 integrated YUV histograms with different objects and sharpness m. Then, the GMM distribution represented by the parameter Θ and the weighting coefficient Ω is stored in the reference data group 17a in the database 17 as a reference distribution for each object and definition. The reference distribution of object identification number k and definition m is hereinafter referred to as reference distribution (k, m) (k = 1, 2, 3, 4, m = 22, 24, 26, 28, 30, 32). ).

  In S130, the likelihood of the integrated YUV histogram (k, m) for the reference distribution (k, m) is calculated based on the equation (5) and stored in the reference data group 17a in the database 17. The likelihood of the integrated YUV histogram (k, m) with respect to the reference distribution (k, m) at the object identification number k and the definition m is hereinafter referred to as likelihood X (k, m) (k = 1, 2, 3, 4, m = 22, 24, 26, 28, 30, 32). That is, six reference distributions (k, m) and likelihoods X (k, m) are calculated for each of the four objects according to the definition m by the process of S130.

  Next, in S140, a variable Pk representing the object identification number k is set to 1, and a variable Pm representing the sharpness m is set to 22. Thereafter, in S150, for the five vehicle types having vehicle type identification numbers c = 1 to 5, the object identification number k indicated by the variable Pk and the YUV histogram (c, k, m) of the sharpness m indicated by the variable Pm are described. The likelihood for the reference distribution (k, m) is calculated based on the equation (5). The likelihood for the reference distribution (k, m) for the YUV histogram (c, k, m) is hereinafter expressed as likelihood Y (c, k, m) (c = 1, 2, 3, 4, 5, k = 1, 2, 3, 4, m = 22, 24, 26, 28, 30, 32). That is, five likelihoods Y (c, k, m) with different vehicle type identification numbers c are calculated by the process of S140.

  In S160, a likelihood ratio is calculated for each of the five likelihoods Y (c, k, m) calculated in S150. This likelihood ratio is calculated by Y (c, k, m) / likelihood X (k, m).

  Next, in S170, the maximum value and the minimum value of the five likelihood ratios calculated in S150 are obtained, and then the maximum value and the minimum value are used as the likelihood ratio upper limit value in the object identification number k and the sharpness m. (K, m) and the likelihood ratio lower limit (k, m) are set. Then, the likelihood ratio upper limit value (k, m) and the likelihood ratio lower limit value (k, m) are stored in the reference data group 17a in the database 17.

  Thereafter, in S180, it is determined whether or not the value of the variable Pk representing the object identification number k is equal to or greater than the total number of objects kmax (kmax = 4 in this embodiment). Here, when the value of the variable Pk representing the object identification number k is less than the total number of objects kmax (S180: NO), 1 is added to the value of the variable Pk representing the object identification number k in S190, and then S150. The above process is repeated after moving to step S2. On the other hand, when the value of the variable Pk representing the object identification number k is equal to or greater than the total number of objects kmax (S180: YES), the process proceeds to S200.

  In S200, it is determined whether or not the value of the variable Pm representing the sharpness m is equal to or greater than the maximum sharpness value mmax (mmax = 32 in the present embodiment). If the value of the variable Pm representing the sharpness m is less than the maximum sharpness value mmax (S200: NO), 2 is added to the value of the variable Pm representing the sharpness m in S210, and then to S150. The process is repeated and the above process is repeated. On the other hand, when the value of the variable Pm representing the sharpness m is equal to or greater than the maximum sharpness value mmax (S200: YES), the reference data creation process is terminated.

  That is, by the reference data creation process, the reference distribution (k, m), the likelihood X (k, m), the likelihood ratio upper limit value (k, m), and the likelihood ratio lower limit value (k, m) are stored in the database 17. Are stored in the reference data group 17a (k = 1, 2, 3, 4, m = 22, 24, 26, 28, 30, 32). That is, the reference data group 17a includes a reference distribution (k, m), a likelihood X (k, m), a likelihood ratio upper limit value (k, m), and a likelihood ratio lower limit value (k, m). .

  Next, the procedure of image recognition processing executed by the CPU 11 will be described with reference to FIGS. 4 and 5. FIG. 4 is a flowchart showing the first half of the image recognition processing, and FIG. 5 is a flowchart showing the second half of the image recognition processing. This image recognition process is repeatedly executed while the ignition key of the vehicle is inserted into the key cylinder and the key is turned on.

  In this image recognition process, the CPU 11 first acquires image data taken by the camera 5 from the camera 5 in S310. In S320, the image data acquired in S310 is divided into regions having substantially the same luminance (region division). Here, the following description will be given on the assumption that the area is divided into n areas (hereinafter, this n is also referred to as a total division number n). Further, in S320, an integer number from 1 to n is assigned as a region identification number s in order to identify the divided n regions. Next, in S330, the variable Ps representing the area identification number s is set to 1, and the process proceeds to S340.

  In S340, the area s identified by the area identification number s indicated by the variable Ps is selected, and then the sharpness of the selected area s is calculated in S350. That is, two-dimensional fast Fourier transform is performed on the image data representing the selected region in the same manner as in S40, and the sharpness is calculated based on the spatial frequency distribution obtained by this transform.

  Next, in S360, four objects, that is, rear window object O1, right tail lamp object O2, left tail lamp object O3, and license plate object O4 are extracted from the area selected in S340 in the same manner as S50. To do. In S360, an integer number from 1 to 4 is assigned as an object identification number k to identify the extracted object. That is, the object identification numbers k = 1, 2, 3, and 4 indicate a rear window, a right tail lamp, a left tail lamp, and a license plate, respectively. Thereafter, in S370, the variable Pk representing the object identification number k is set to 1, and the process proceeds to S380.

  In step S380, the object identified by the object identification number k indicated by the variable Pk is selected. In step S390, the YUV histogram for the object selected in step S380 is calculated.

  Next, in S400, a reference distribution at the sharpness calculated in S350 (hereinafter referred to as an interpolation reference distribution) is calculated by linear interpolation. That is, for example, in order to obtain a reference distribution with a sharpness of 24.5, linear interpolation may be performed with a reference distribution (k, 24) with a definition of 24 and a reference distribution (k, 26) with a definition of 26. Further, in S400, the likelihood for this interpolation reference distribution of the YUV histogram calculated in S390 is calculated. This likelihood is hereinafter referred to as likelihood Z.

  In S410, the likelihood X in the sharpness calculated in S350 (hereinafter referred to as interpolation likelihood) is calculated by linear interpolation. That is, for example, in order to obtain an interpolation likelihood with a definition of 24.5, linear interpolation may be performed with a likelihood X (k, 24) of a definition 24 and a likelihood X (k, 26) of a definition 26. . In S410, the likelihood ratio of the likelihood Z calculated in S400 to the interpolation likelihood (hereinafter referred to as the recognition likelihood ratio) is calculated. That is, the recognition likelihood ratio is calculated by (likelihood Z / interpolation likelihood).

  Next, in S420, the likelihood ratio upper limit value in sharpness calculated in S350 (hereinafter referred to as the interpolation likelihood ratio upper limit value) and the likelihood ratio lower limit value (hereinafter referred to as the interpolation likelihood ratio lower limit value). Is calculated by linear interpolation. That is, for example, in order to obtain the likelihood ratio upper limit value of the definition 24.5, the likelihood ratio upper limit value (k, 24) of the definition 24 and the likelihood ratio upper limit value (k, 26) of the definition 26 are determined. And linear interpolation. Also, the interpolation likelihood ratio lower limit value is obtained in the same manner. Further, in S420, it is determined whether or not the recognition likelihood ratio calculated in S410 is in a range not less than the interpolation likelihood ratio lower limit value and not more than the interpolation likelihood ratio upper limit value. Here, when it is in the range not less than the interpolation likelihood ratio lower limit value and not more than the interpolation likelihood ratio upper limit value (S420: YES), in S430, the object identified by the object identification number k indicated by the variable Pk. Then, the kth object recognition flag F (k) indicating the recognition result is set, and the process proceeds to S450. On the other hand, when it is not in the range not less than the interpolation likelihood ratio lower limit value and not more than the interpolation likelihood ratio upper limit value (S420: NO), the kth object recognition flag F (k) is cleared in S440, and the process proceeds to S450. To do.

  In S450, it is determined whether or not the value of the variable Pk representing the object identification number k is equal to or greater than the total number of objects kmax. If the value of the variable Pk representing the object identification number k is less than the total number of objects kmax (S450: NO), the value of the variable Pk representing the object identification number k is incremented by 1 in S460, and then S380. The above process is repeated after moving to step S2. On the other hand, if the value of the variable Pk representing the object identification number k is equal to or greater than the total number of objects kmax (S450: YES), whether or not all the kth object recognition flags F (k) are set in S470. Determine whether. Here, when all of the k-th object recognition flags F (k) are set (S470: YES), it is determined that “a preceding vehicle is indicated in the area s”, and the process proceeds to S500. On the other hand, when all of the kth object recognition flags F (k) are not set (S470: NO), it is determined that “the preceding vehicle is not shown in the area s”, and the process proceeds to S500.

  In S500, it is determined whether or not the value of the variable Ps representing the area identification number s is equal to or greater than the total number of divisions n. If the value of the variable Ps representing the region identification number s is less than the total number of divisions n (S500: NO), the value of the variable Ps representing the region identification number s is incremented by 1 in S510, and then S340. The above process is repeated after moving to step S2. On the other hand, when the value of the variable Ps representing the region identification number s is equal to or greater than the total number of divisions n (S500: YES), the image recognition process is terminated.

  As described above, according to the image recognition system 1 of the first embodiment, in the comparison between the YUV histogram and the reference distribution for recognizing the preceding vehicle, the optimum reference distribution is adopted according to the sharpness. Can do. For this reason, even if the distance to the preceding vehicle changes, the preceding vehicle can be recognized stably. This is because the sharpness can be used as an index representing the distance to the preceding vehicle. In this case, for example, when a person visually recognizes an object (for example, a vehicle), the farther the vehicle is located, the more unclear the vehicle is, and it is difficult to determine whether the object is a vehicle. It is clear from that.

  In the first embodiment described above, the image recognition device 3 is the image recognition device according to the present invention, the processing at S350 in FIG. 4 is the sharpness calculation means according to the present invention, and the processing at S390 in FIG. The database 17 is the first storage means in the present invention, and the processes in S400 to S490 in FIGS. 4 and 5 are the first determination means in the present invention.

  The image data representing the region s identified by the region identification number s is the partial region image in the present invention, the color information (YUV) is the feature amount in the present invention, and the YUV histogram calculated in the process of S390 is the feature in the present invention. Data, a preceding vehicle is a predetermined object in the present invention, image data for creating reference data is a reference image in the present invention, a reference distribution (k, m) is reference data in the present invention, and a reference data group 17a is a reference data group in the present invention. It is.

Next, the result of image recognition using the image recognition system 1 of the present embodiment will be described below as an example.
First, an input image used in this embodiment is shown in FIG. The input image is a creation image (see FIG. 9A) that is one of the reference data creation image data, and a recognition image 1 that shows a car having a color distribution similar to that of the reference data creation image data. (See FIG. 9B), recognition image 2 (see FIG. 9C) showing a car having a color distribution different from that of the reference data creation image data, and recognition image 3 showing a car other than the car (FIG. 9D) 4). As the recognition image 3, a traffic light image in which a red signal is lit is adopted as an example of a red object other than the tail lamp that is seen while traveling on the road. And about this signal image, the process equivalent to another vehicle image was performed, and also each object was cut out by average position and size, and the image recognition process was performed.

  For each input image, the likelihood ratio calculated for each object (rear window object O1, right tail lamp object O2, left tail lamp object O3 and license plate object O4) (the likelihood ratio calculated in S410) Is shown in FIGS. 10, FIG. 11, FIG. 12 and FIG. 13 show the results for the creation image, the recognition image 1, the recognition image 2 and the recognition image 3, respectively. Also, the vertical axis in the figure is the likelihood ratio, and the horizontal axis is the sharpness. The likelihood ratio is indicated by a broken line, and the upper limit value and the lower limit value are indicated by a solid line.

  As shown in FIGS. 10 to 13, in all input images, the likelihood ratio decreases as the definition increases for any object. This is because the higher the definition is, the more the object data includes data in a wider band, and the integrated YUV histogram has a distribution including various elements in the wider band. That is, it is considered that the reference distribution calculated based on the integrated YUV histogram includes many distributions that do not match the input image data. On the other hand, the likelihood ratio increases as the definition decreases. This is considered to be because the spread of the color distribution in the object becomes uniform when the sharpness decreases, and the distribution band in the object does not differ greatly in all input images.

Further, the recognition image 1 has almost the same recognition result because the photographing environment and color distribution of the car are similar to those of the creation image. In other words, the likelihood ratio for all objects is included between the upper and lower limits.
For the recognition image 2, the rear window reflects sunlight, and the color distribution of the recognition image 2 is different from the creation image. For this reason, the likelihood ratio of the rear window may be smaller than the lower limit value. In addition, in the creation image, since sunlight is shining from the left side of the image, the right tail lamp in the creation image is generally darker than the left tail lamp, whereas in the recognition image 2, the right tail lamp is dark. Since the light is also inserted into the right tail lamp of the image for recognition 2, the right tail lamp of the recognition image 2 is brighter than the right tail lamp of the image for creation. For this reason, the likelihood ratio of the right tail lamp in the recognition image 2 is a value close to the lower limit value. Further, since the other objects (left tail lamp and license plate) are similar to the creation image, the likelihood ratio is included between the upper limit value and the lower limit value.

  For the recognition image 3, a red signal exists at the position of the right tail lamp. For this reason, the likelihood ratio in this object is close to the lower limit value of the right tail lamp. However, since the other objects are significantly different from the creation image, the likelihood ratio is small.

  Therefore, when determining whether or not the vehicle is a preceding vehicle, it is considered that the misrecognition can be reduced by determining not only the result of one object but also the entire constituting object.

  From the results so far, it can be seen that the reference distribution can represent the data distribution of the image data for generating reference data. It was also found that even if the distance to the recognition object changes, the change in distance can be dealt with by changing the upper and lower limits of the likelihood ratio.

(Second Embodiment)
Below, 2nd Embodiment is described based on drawing. In the second embodiment, only parts different from the first embodiment will be described.

  First, the structure of the image recognition system 1 in 2nd Embodiment is demonstrated based on FIG. The image recognition system 1 according to the second embodiment includes a reference data group 17b that is different from the reference data group 17a in that a radar laser 7 that measures the distance to an object that exists in the range captured by the camera 5 is added. Except for the points stored in the database 17, the second embodiment is the same as the first embodiment.

  Further, the reference data creation image data stored in the database 17 is added with information on the shooting distance at which the reference data creation image data was shot (hereinafter referred to as distance information). The distance information is information indicating distances such as 15 m and 50 m, for example.

  Next, reference data creation processing in the second embodiment will be described with reference to FIGS. 15 and 16. The reference data creation process of the second embodiment differs from that of the first embodiment in that S40a, S90a, S120a, instead of the processes of S40, S90, S120, S130, S140, S150, S160, S170, S200, and S210, respectively. , S130a, S140a, S150a, S160a, S170a, S200a and S210a.

That is, when the process of S30 is completed, the distance information added to the image data (c, d) acquired in S30 is acquired in S40a.
When the processing of S80 is completed, for example, the shooting distance r = 20, 40, 60, image data (c, d) to which the vehicle type identification number c indicated by the variable Pc is assigned for each object in S90a. The YUV histogram of 80, 100, 120 [m] is obtained by linear interpolation with the YUV histogram calculated in S60. That is, for example, distance information in six images having the same vehicle type and different shooting distances, that is, distance identification numbers d = 1, 2, 3, 4, 5, and 6, respectively, is “15.5”, “32.5”, Assuming that “48.0”, “70.5”, “88.3”, “106.5” [m], for example, in order to obtain a YUV histogram of the shooting distance r = 20 [m], the distance Linear interpolation may be performed between the YUV histogram of information 15.5 and the YUV histogram of distance information 32.5. Then, a YUV histogram of shooting distances r = 20, 40, 60, 80, 100, 120 [m] obtained by linear interpolation is stored in the RAM 15. The YUV histogram of the vehicle type identification number c, the object identification number k, and the shooting distance r is hereinafter referred to as a YUV histogram (c, k, r) (c = 1, 2, 3, 4, 5, k = 1, 2, 3, 4, r = 20, 40, 60, 80, 100, 120).

  In S100, if the value of the variable Pc representing the vehicle type identification number c is 5 or more (S100: YES), the YUV histogram having the same object and the same shooting distance r is added for all vehicle types in S120a. To do. For example, since there is one YUV histogram for the rear window object O1 and the shooting distance r of 20, for each vehicle type identification number c = 1 to 5, these five YUV histograms are added. This added YUV histogram is hereinafter referred to as an integrated YUV histogram. That is, by the process of S120, six integrated YUV histograms are generated for each of the four objects according to the shooting distance r. The integrated YUV histogram of the object identification number k and the shooting distance r is hereinafter referred to as an integrated YUV histogram (k, r) (k = 1, 2, 3, 4, r = 20, 40, 60, 80, 100). 120).

  Next, in S130a, a parameter Θ and a weighting coefficient Ω are calculated for 24 integrated YUV histograms having different shooting distances r from the object. The GMM distribution represented by the parameter Θ and the weighting coefficient Ω is stored in the reference data group 17b in the database 17 as a reference distribution at each object and shooting distance r. The reference distribution of the object identification number k and the shooting distance r is hereinafter referred to as a reference distribution (k, r) (k = 1, 2, 3, 4, r = 20, 40, 60, 80, 100, 120). ).

  Further, in S130a, the likelihood of the integrated YUV histogram (k, r) for the reference distribution (k, r) is calculated based on the equation (5) and stored in the reference data group 17b in the database 17. The likelihood of the integrated YUV histogram (k, r) with respect to the reference distribution (k, r) at the object identification number k and the shooting distance r is hereinafter referred to as likelihood X (k, r) (k = 1, 2, 3, 4, r = 20, 40, 60, 80, 100, 120). That is, six reference distributions (k, r) and likelihoods X (k, r) are calculated for each of the four objects according to the shooting distance r by the process of S130a.

  Next, in S140a, a variable Pk representing the object identification number k is set to 1, and a variable Pr representing the shooting distance r is set to 20. Thereafter, in S150a, the object identification number k indicated by the variable Pk and the shooting distance r indicated by the variable Pr for the YUV histogram (c, k, r) for the five vehicle types having the vehicle type identification numbers c = 1 to 5 are obtained. The likelihood for the reference distribution (k, r) is calculated based on the equation (5). Note that the likelihood of the YUV histogram (c, k, r) with respect to the reference distribution (k, r) is hereinafter expressed as likelihood Y (c, k, r) (c = 1, 2, 3, 4, 5, k = 1, 2, 3, 4, r = 20, 40, 60, 80, 100, 120). That is, five likelihoods Y (c, k, r) with different vehicle type identification numbers c are calculated by the process of S140a.

  In S160a, a likelihood ratio is calculated for each of the five likelihoods Y (c, k, r) calculated in S150a. This likelihood ratio is calculated by Y (c, k, r) / likelihood X (k, r).

  Next, in S170a, the maximum value and the minimum value of the five likelihood ratios calculated in S150a are obtained, and then the maximum value and the minimum value are used as the likelihood ratio upper limit value in the object identification number k and the shooting distance r. (K, r) and the likelihood ratio lower limit (k, r) are set. The likelihood ratio upper limit value (k, r) and the likelihood ratio lower limit value (k, r) are stored in the reference data group 17b in the database 17.

  If the value of the variable Pk representing the object identification number k is equal to or greater than the total number of objects kmax in S180 (S180: YES), the value of the variable Pr representing the shooting distance r is the maximum shooting distance value in S200a. It is determined whether or not rmax (rmax = 120 in this embodiment) or more. Here, when the value of the variable Pr representing the shooting distance r is less than the maximum shooting distance value rmax (S200a: NO), 20 is added to the value of the variable Pr representing the shooting distance r in S210a, and then to S150a. The process is repeated and the above process is repeated. On the other hand, when the value of the variable Pr representing the shooting distance r is equal to or greater than the shooting distance maximum value rmax (S200a: YES), the reference data creation process is terminated.

  That is, by the reference data creation process, the reference distribution (k, r), likelihood X (k, r), likelihood ratio upper limit (k, r) and likelihood ratio lower limit (k, r) are stored in the database 17. Are stored in the reference data group 17b (k = 1, 2, 3, 4, r = 20, 40, 60, 80, 100, 120). That is, the reference data group 17b includes a reference distribution (k, r), a likelihood X (k, r), a likelihood ratio upper limit value (k, r), and a likelihood ratio lower limit value (k, r). .

  Next, image recognition processing in the second embodiment will be described with reference to FIG. The image recognition process of the second embodiment is different from the first embodiment in that the processes of S320a, S350a, S400a, S410a, and S420a are performed instead of the processes of S320, S350, S400, S410, and S420, respectively. is there.

  First, when the processing in S310 is completed, in S320a, the image acquired in S310 is divided into regions where the distances measured by the radar laser 7 are substantially equal (region division). That is, by measuring the distance by the radar laser 7, the image acquired in S310 and a pair of distance data are generated, and the distance data is applied to an applicable distance range, for example, a range obtained by dividing the distance data into equal distance regions within 1 m. It is divided by extracting the area. Here, the following description will be given on the assumption that the area is divided into n areas (hereinafter, this n is also referred to as a total division number n). Further, in S320a, an integer number from 1 to n is assigned as a region identification number s in order to identify the divided n regions.

  When the process of S340 ends, the shooting distance of the selected area is calculated in S350a. That is, the average of the distance data is calculated over the entire selected area, and this average value is set as the shooting distance.

  When the processing of S390 is completed, a reference distribution (hereinafter referred to as an interpolation reference distribution) at the photographing distance calculated in S350a is calculated by linear interpolation in S400a. That is, for example, in order to obtain a reference distribution with a shooting distance of 32.5, linear interpolation is performed between a reference distribution (k, 20) with a shooting distance r = 20 and a reference distribution (k, 40) with a shooting distance r = 40. That's fine. Further, in S400a, the likelihood for this interpolation reference distribution of the YUV histogram calculated in S390 is calculated. This likelihood is hereinafter referred to as likelihood Z.

  In S410a, the likelihood X at the photographing distance calculated in S350a (hereinafter referred to as interpolation likelihood) is calculated by linear interpolation. That is, for example, in order to obtain the interpolation likelihood with the shooting distance of 32.5, the likelihood X (k, 20) with the shooting distance r = 20 and the likelihood X (k, 40) with the shooting distance r = 40. What is necessary is just to perform linear interpolation. In S410a, the likelihood ratio of the likelihood Z calculated in S400a to the interpolation likelihood (hereinafter referred to as the recognition likelihood ratio) is calculated. That is, the recognition likelihood ratio is calculated by (likelihood Z / interpolation likelihood).

  Next, in S420a, the likelihood ratio upper limit value (hereinafter referred to as the interpolation likelihood ratio upper limit value) and the likelihood ratio lower limit value (hereinafter referred to as the interpolation likelihood ratio lower limit value) at the photographing distance calculated in S350a. Is calculated by linear interpolation. That is, for example, in order to obtain the likelihood ratio upper limit value of the shooting distance 32.5, the likelihood ratio upper limit value (k, 20) of the shooting distance r = 20 and the likelihood ratio upper limit value of the shooting distance r = 40 ( k, 40) and linear interpolation. Also, the interpolation likelihood ratio lower limit value is obtained in the same manner. Further, in S420a, it is determined whether or not the recognition likelihood ratio calculated in S410a is in a range not less than the interpolation likelihood ratio lower limit and not more than the interpolation likelihood ratio upper limit. Here, when it exists in the range more than an interpolation likelihood ratio lower limit and below an interpolation likelihood ratio upper limit (S420a: YES), it transfers to S430. On the other hand, when it is not in the range not less than the interpolation likelihood ratio lower limit value and not more than the interpolation likelihood ratio upper limit value (S420a: NO), the process proceeds to S440.

  As described above, according to the image recognition system 1 of the second embodiment, in the comparison between the YUV histogram and the reference distribution for recognizing the preceding vehicle, the optimum reference distribution is adopted according to the shooting distance. Can do. For this reason, even if the distance to the preceding vehicle changes, the preceding vehicle can be recognized stably.

  In the second embodiment described above, the processing of the radar laser 7 and S350a in FIG. 17 is the distance measuring means in the present invention, the processing of S390 in FIG. 17 is the feature amount extracting means in the present invention, and the database 17 is the second in the present invention. The storage means, the processes of S400a, S420a, S430 and S440 in FIG. 17 and the processes of S450 to S490 in FIG. 5 are the second determination means in the present invention.

The reference distribution (k, r) is the reference data in the present invention, and the reference data group 17b is the reference data group in the present invention.
(Other embodiments)
As mentioned above, although one Embodiment of this invention was described, this invention is not limited to the said embodiment, As long as it belongs to the technical scope of this invention, it can be implemented with a various form.

  In the above embodiment, the sharpness changes according to the distance, but the weather (sunny, cloudy, rain, etc.), time (day, evening, night, etc.), change in the incident direction of sunlight, etc. Since the sharpness may vary depending on the shooting environment in which the input image is shot, it is desirable to select different reference distributions depending on the shooting environment even with the same sharpness.

  In the above-described embodiment, the sharpness is calculated by performing the fast Fourier transform on the image data. However, the edge enhancement processing is performed on the image data, and based on the amount of edges included in the edge enhanced image. You may comprise so that a definition may be calculated. That is, an edge image is generated by using a known Laplacian filter or Sobel filter, and the number of pixels having a value equal to or greater than a predetermined threshold is calculated from the pixels included in the edge image. .

  In the above embodiment, the acquired image and a pair of distance data are generated by distance measurement by the radar laser 7, and the acquired image is divided into regions (S320a), as disclosed in JP-A-9-61094. The region may be divided by capturing a moving object by using a motion vector detection method from two temporally continuous images, calculating the contour thereof, and specifying the outline from the contour. Or you may divide | segment into the area | region which exists in a fixed distance range by calculating a distance image using two stereo camera images. Alternatively, pixels having similar colors may be grouped together using a single camera image.

  In the above embodiment, the reference data generation process is performed. However, the reference data generation process is not necessarily executed by the image recognition system 1. That is, it may be executed offline in advance and only the generated reference data group may be stored in the database 17. In this case, the operation key 21a is unnecessary in the user I / F 21. With the above configuration, the reference data generation process can be omitted, and it is not necessary to store image data (reference data creation image data) in the database 17, so that the capacity of the database 17 can be reduced. Furthermore, since the operation key 21a is unnecessary, the image recognition apparatus 3 can be configured more simply.

  In the above embodiment, the image recognition process is repeatedly executed while the ignition key of the vehicle is inserted into the key cylinder and the key is turned on. However, the image recognition process is executed. The image recognition device 3 may be provided with a recognition process execution switch for this purpose, and when the recognition process execution switch is operated, the CPU 11 may start the image recognition process. More specifically, a cruise control start switch for starting cruise control is provided, and this cruise control start switch is operated so that the CPU 11 starts image recognition processing in conjunction with the start of cruise control. You may comprise.

1 is a block diagram showing a configuration of an image recognition system 1. FIG. The flowchart which shows the procedure of the first half part of a reference | standard data creation process. The flowchart which shows the procedure of the latter half part of a reference | standard data creation process. The flowchart which shows the procedure of the first half part of an image recognition process. The flowchart which shows the procedure of the latter half part of an image recognition process. The figure which shows the structure of the image data for reference | standard data preparation. The figure explaining the calculation method of a definition. The figure explaining an object and a YUV histogram. The figure which shows an input image. The figure which shows the likelihood ratio calculation result in the image for preparation. The figure which shows the likelihood ratio calculation result in the image 1 for recognition. The figure which shows the likelihood ratio calculation result in the image 2 for recognition. The figure which shows the likelihood ratio calculation result in the image 3 for recognition. 1 is a block diagram showing a configuration of an image recognition system 1. FIG. The flowchart which shows the procedure of the first half part of a reference | standard data creation process. The flowchart which shows the procedure of the latter half part of a reference | standard data creation process. The flowchart which shows the procedure of the first half part of an image recognition process.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Image recognition system, 3 ... Image recognition apparatus, 5 ... Camera, 7 ... Radar laser, 11 ... CPU, 13 ... ROM, 15 ... RAM, 17 ... Database, 17a, 17b ... Reference data group, 19 ... Input-output part 21 ... User I / F, 21a ... operation keys, 21b ... display panel.

Claims (6)

An image recognition method for dividing an input image into a plurality of partial area images and performing image recognition for each partial area image,
A first step of calculating a sharpness indicating a degree of sharpness of the partial area image;
A second step of extracting a feature amount representing a feature of each pixel constituting the partial region image from the partial region image and generating feature data representing a distribution of the feature amount;
Reference data prepared by generating in advance reference data representing the distribution of the feature amount for each of a plurality of reference images showing a predetermined object set in advance and having different definition of the predetermined object From the group, select the reference data corresponding to the sharpness calculated in the first step, compare the selected reference data with the feature data generated in the second step, and the comparison result A third step of determining whether or not the predetermined object is indicated in the partial region image;
An image recognition method comprising: An image recognition apparatus that divides an input image into a plurality of partial area images and performs image recognition for each partial area image,
A sharpness calculating means for calculating a sharpness indicating the degree of sharpness of the partial area image;
Feature amount extraction means for extracting feature amounts representing the features of each pixel constituting the partial region image from the partial region image and generating feature data representing the distribution of the feature amount;
Reference data prepared by generating in advance reference data representing the distribution of the feature amount for each of a plurality of reference images showing a predetermined object set in advance and having different definition of the predetermined object First storage means for storing groups;
Based on the sharpness value calculated by the sharpness calculating means, the reference data corresponding to the sharpness is extracted from the first storage means, and extracted by the extracted reference data and the feature quantity extracting means. First determination means for comparing the obtained feature data and determining whether or not the predetermined object is indicated in the partial region image based on the comparison result;
An image recognition apparatus comprising: The feature amount is color information.
The image recognition apparatus according to claim 2. The sharpness calculation means performs a two-dimensional Fourier transform on the partial region image, and calculates the sharpness based on a spatial frequency distribution obtained by the transformation.
The image recognition apparatus according to claim 2, wherein the image recognition apparatus is an image recognition apparatus. The sharpness calculation means performs edge enhancement processing on the partial region image, and calculates the sharpness based on the amount of edges included in the edge enhanced image.
The image recognition apparatus according to claim 2, wherein the image recognition apparatus is an image recognition apparatus. An image recognition apparatus that divides an input image into a plurality of partial area images and performs image recognition for each partial area image,
Distance measuring means for measuring an object distance which is a distance between an object shown in the partial region image and the image recognition device;
Feature amount extraction means for extracting feature amounts representing the features of each pixel constituting the partial region image from the partial region image and generating feature data representing the distribution of the feature amount;
Reference data prepared by generating in advance reference data representing the distribution of the feature amount for each of a plurality of reference images showing a predetermined object set in advance and having different object distances of the predetermined object Second storage means for storing groups;
Based on the object distance measured by the distance measuring means, the reference data corresponding to the object distance is extracted from the second storage means, and the extracted reference data and the features extracted by the feature amount extracting means A second determination unit that compares data and determines whether or not the predetermined object is shown in the partial region image based on the comparison result;
An image recognition apparatus comprising: