JP2017091118A - Image determination device and processing device using the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reflect information obtained from an image itself picked up by imaging means in determining whether the image is not suitable to predetermined processing.SOLUTION: An image determination device comprises: labeling means S4 of acquiring a label region by labeling from a predetermined region of a difference image between two images picked up by imaging means at a predetermined interval of time; index value acquiring means S5 of acquiring an average value of respective distances of each of label regions acquired by the labeling means S4 from remaining label regions from the label region or an index value corresponding thereto; histogram acquiring means S7 of acquiring a histogram indicative of the number of label regions having the index value belonging to each predetermined range of index values for the range; and determination means S8, S9 of determining, on the basis of the histogram, whether at least one of the two images is suitable to predetermined processing.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image determination apparatus and a processing apparatus using the same.

  2. Description of the Related Art Conventionally, various apparatuses including a processing apparatus having a processing unit that performs predetermined processing on an image captured by an imaging unit are provided.

  As an example of such a device, the following Patent Document 1 discloses that a road is photographed with a video camera, the photographed video data is image-processed, and the traffic volume, the average speed of a traveling vehicle, whether or not a traffic jam occurs, and the like. A road monitoring device for determining various traffic quantities is disclosed.

  This conventional road monitoring apparatus is a road monitoring apparatus that detects various amounts of traffic based on video captured by an electronic camera, performs image processing on the video of the road captured by the electronic camera, and Image processing means for detecting a traveling vehicle, environment measurement means for measuring the photographing environment by the electronic camera (environmental sensors such as anemometer, illuminometer, visibility meter, rain and snow gauge), and measurement results of the environment measurement means Based on the environment determination means for determining whether or not the shooting environment by the electronic camera is in an appropriate state for detection of the vehicle, and when it is determined that the environment determination means is in an appropriate state, the image processing Traffic information detecting means for obtaining various traffic quantities based on information obtained by the means.

  In this conventional road monitoring device, the photographing environment by the electronic camera is measured, and only when the photographing environment is in an appropriate state, the traffic quantities are obtained based on the information obtained by the image processing means. Yes. Therefore, according to the conventional road monitoring device, even when a vehicle is erroneously detected by image processing in an environment that is not suitable for photographing, the detected information is not used for calculation of various traffic quantities, and thus is highly reliable. Traffic quantities can be detected with stable accuracy.

JP 2002-56491 A

  However, in the conventional road monitoring device, an image captured by the electronic camera is captured by measuring an environment captured by the electronic camera using an environmental sensor and determining whether the environment is appropriate. It is determined whether it is appropriate for detection. Therefore, in the conventional road monitoring device, information obtained from the image itself is not used at all in determining whether the image taken by the electronic camera is appropriate for vehicle detection. The actual situation of the image was not reflected at all.

  Such a situation applies not only to the road monitoring device but also to various devices including a processing device having a processing unit that performs a predetermined process on an image captured by the imaging unit.

  The present invention has been made in view of such circumstances, and in determining whether an image captured by the imaging unit is an image that is not suitable for a predetermined process, the image itself is captured by the imaging unit. An object of the present invention is to provide an image determination apparatus capable of reflecting obtained information and a processing apparatus using the same.

  The following aspects are presented as means for solving the problems. The image determination apparatus according to the first aspect includes a predetermined region of a difference image between two images captured at a predetermined time interval by an imaging unit, or a difference between an image captured by the imaging unit and a background image. Labeling means for obtaining a label area by labeling from a predetermined area of the image, and for each label area obtained by the labeling means, an average value of each distance from the label area to each of the remaining label areas, or in accordance with this Index value acquisition means for acquiring an index value that is a value (for example, an integrated value of each distance, etc.), and the number of label regions having the index value belonging to the range for each predetermined range of the index value And at least one of the two images picked up by the image pickup means based on the histogram. Determining means for determining whether or not the image is an image that is not suitable for a predetermined process or whether the image captured by the imaging means is an image that is not suitable for a predetermined process; It is provided. The predetermined area may include an area on the road.

  The histogram is the distribution status of the label area in the predetermined area of the difference image between two images captured by the imaging means, or the predetermined area of the difference image between the image captured by the imaging means and the background image. It changes according to the distribution status of the label area in the above, and becomes a feature amount indicating the distribution status. On the other hand, among the images picked up by the image pickup means, an image that is not suitable for the predetermined processing has a certain tendency in the distribution status due to the influence of time-varying noise generated in the image.

  Therefore, according to the first aspect, based on the histogram, whether at least one of the two images captured by the imaging unit is an image that is not suitable for a predetermined process. Alternatively, since it is determined whether or not the image captured by the imaging unit is an image that is not suitable for the predetermined processing, the determination can be appropriately performed.

  The histogram is information obtained from the image itself picked up by the image pickup means. Therefore, according to the first aspect, it is determined whether or not at least one of the two images captured by the imaging unit is an image that is not suitable for a predetermined process, or the imaging unit. In the determination as to whether or not the image captured by the method is an image that is not suitable for the predetermined processing, information obtained from the image itself captured by the imaging unit can be reflected.

  The image determination apparatus according to a second aspect is the image determination apparatus according to the first aspect, wherein the determination means determines whether or not the histogram belongs to a group of histograms corresponding to the histogram of an image not suitable for the predetermined processing. Whether or not at least one of the two images captured by the imaging unit is an image not suitable for the predetermined processing, or is captured by the imaging unit. It is determined whether or not the image is an image not suitable for the predetermined processing.

  In the second aspect, whether or not at least one of the two images captured by the imaging unit is not suitable for the predetermined processing is determined by the determination unit by discriminant analysis or the like. Alternatively, an example is given in which it is determined whether or not the image captured by the imaging means is an image that is not suitable for the predetermined processing. In the first aspect, this determination technique is used. Not limited to.

  In the image determination apparatus according to the third aspect, in the second aspect, the determination is performed by determination analysis using a Mahalanobis distance.

  The third mode is an example of the discrimination method. However, the second mode is not limited to the discrimination method, and for example, a method using a neural network or a support vector machine is adopted. May be.

  In the image determination device according to a fourth aspect, in any one of the first to third aspects, the determination unit is a part of at least one of the two images captured by the imaging unit. Alternatively, on the condition that the brightness of the entire area is darker than the predetermined brightness, at least one of the two images captured by the imaging unit is an image that is not suitable for the predetermined processing. The image captured by the imaging unit is determined to have a condition that the brightness of a part or the whole area of the image captured by the imaging unit is darker than a predetermined brightness. It is determined that the image is not suitable for the predetermined processing.

  In general, when the image is bright, the influence of noise or the like is small and the image is suitable for the predetermined processing, whereas when the image is dark, the influence of noise or the like is large and the image is Often not suitable for a given treatment. Therefore, according to the fourth aspect, the determination unit is configured such that the brightness of a part or the entire region of at least one of the two images captured by the imaging unit is greater than a predetermined brightness. Is determined to be at least one of the two images captured by the imaging unit as an image that is not suitable for the predetermined processing, or is captured by the imaging unit. It is determined that the image captured by the imaging means is an image not suitable for the predetermined processing on the condition that the brightness of a part or the entire area of the captured image is darker than the predetermined brightness As a result, the accuracy of the determination can be improved.

  The image determination apparatus according to a fifth aspect is the image determination apparatus according to any one of the first to fourth aspects, wherein the two images captured by the imaging unit or the image captured by the imaging unit The image is affected by the condition.

  In the fifth aspect, an image affected by weather conditions is cited as an example of an image in which noise or the like is generated. In the first to fourth aspects, the second image captured by the imaging unit is used. A single image or the image picked up by the image pickup means may be an image that generates noise or the like under the influence of factors other than weather conditions.

  An image determination apparatus according to a sixth aspect is an image determination apparatus that determines whether an image captured by an imaging unit and affected by weather conditions is an image that is not suitable for predetermined processing, wherein the weather conditions are Means for performing the determination based on the image without using a signal from a sensor to be detected is provided.

  According to the sixth aspect, since it is determined based on the image whether or not the image captured by the imaging unit is an image that is not suitable for the predetermined processing, the image is captured by the imaging unit. Information obtained from the captured image itself can be reflected. Therefore, according to the sixth aspect, when determining whether or not the image captured by the imaging unit is an image that is not suitable for the predetermined process, in the determination, from the image itself captured by the imaging unit. Information obtained can be reflected.

  And according to the said 6th aspect, determination whether the image imaged by the imaging means and the image which receives the influence of a weather condition is an image which is not suitable for a predetermined process is from the sensor which detects the said weather condition. Since it is performed without using a signal, it is not necessary to provide the sensor, and the cost can be reduced accordingly.

  In the image determination device according to a seventh aspect, in any one of the first to sixth aspects, the predetermined process is a process of detecting a detection target. Examples of the detection target include pedestrians and vehicles.

  The seventh aspect is an example of the predetermined process, but the first to sixth aspects are not limited to this example.

  A processing device according to an eighth aspect includes the image determination device according to any one of the first to seventh aspects, and processing means that performs the predetermined process on the image picked up by the image pickup means. It is a thing.

  According to the eighth aspect, when the image determination device determines that the image is not suitable for the predetermined process, for example, by outputting the fact or canceling the predetermined process, The reliability of the processing result of the predetermined processing can be improved.

  According to the present invention, in determining whether an image captured by the imaging unit is an image that is not suitable for the predetermined process, an image that can reflect information obtained from the image captured by the imaging unit itself A determination apparatus and a processing apparatus using the determination apparatus can be provided.

It is a schematic block diagram which shows the traffic control system which has a pedestrian detection apparatus as a processing apparatus by the 1st Embodiment of this invention. It is a schematic flowchart which shows operation | movement of the pedestrian detection apparatus in FIG. It is a figure which shows an example of the image imaged by the camera in FIG. FIG. 4 is a diagram showing an image shown in FIG. 3 in which a rectangle forming the outline of the processing target region is entered. The figure which shows an example of the image after binarizing the difference image of the two images (one image is an image shown in FIG. 3) continuously imaged by the camera in FIG. It is. It is a figure which shows an example of the image imaged at the night of the wind and rain with the camera in FIG. The figure which shows an example of the image after binarizing the difference image of the two images (one image is an image shown in FIG. 6) continuously imaged by the camera in FIG. It is. It is a figure which shows typically the example of the distance between a label area | region and a label area | region. It is a schematic flowchart which shows an example of the method of calculating | requiring the coefficient for Mahalanobis distance calculation for every class used by step S7 in FIG. It is a schematic flowchart which shows the detail of step S21 in FIG. It is a schematic flowchart which shows operation | movement of the pedestrian detection apparatus as a processing apparatus by the 2nd Embodiment of this invention. It is a schematic flowchart which shows operation | movement of the pedestrian detection apparatus as a processing apparatus by the 3rd Embodiment of this invention. It is a schematic flowchart which shows the detail of step S51 in FIG.

  Hereinafter, an image determination apparatus and a processing apparatus according to the present invention will be described with reference to the drawings.

  [First Embodiment]

  FIG. 1 is a schematic block diagram showing a traffic control system 1 having a pedestrian detection device 11 as a processing device according to a first embodiment of the present invention.

  A traffic control system 1 shown in FIG. 1 is based on a pedestrian detection device 11 that detects a pedestrian on the road and a pedestrian detection signal as a pedestrian detection result from the pedestrian detection device 11. 13 and a signal control device 12 for controlling the vehicle traffic signal device 14.

  For example, when the pedestrian traffic signal 13 is a green signal, the signal control device 12 does not receive a pedestrian detection signal indicating that a pedestrian on the road corresponding to the pedestrian traffic signal 13 is detected. The pedestrian traffic signal 13 is controlled so that the green time of the pedestrian traffic signal 13 is a predetermined time. On the other hand, when the pedestrian traffic signal 13 is a green signal, the pedestrian traffic signal 13 When a pedestrian detection signal indicating that a pedestrian on the corresponding pedestrian crossing has been detected is received, the traffic signal for pedestrians is such that the blue time of the traffic signal for pedestrian 13 is longer than the predetermined time. 13 is controlled. At this time, the signal control device 12 controls the vehicle traffic signal 14 so that the vehicle traffic signal 14 performs an operation suitable for the operation of the pedestrian traffic signal 13. However, the signal control device 12 is not limited to such so-called green time extension control, and may perform other controls based on the pedestrian detection signal. Moreover, the use of the pedestrian detection signal from the pedestrian detection device 11 is not limited to the control of the traffic signals 13 and 14 by the signal control device 12, but may be used for other purposes, or other devices (for example, traffic) The data may be output to a management center device or the like.

  In the present embodiment, the pedestrian detection device 11 converts a camera 21 such as a CCD camera as an imaging unit that captures an area including a road, and an image analog signal from the camera 21 into a digital signal having a predetermined gradation for each pixel. An A / D converter 22 for conversion, an image memory 23 for storing an image converted into a digital signal, and a processing unit 24 for performing processing to be described later on the image from the image memory 23 are provided. Although not shown in the drawing, the processing unit 24 is composed of a microcomputer and other electronic circuits so as to realize the operation described later. Although not shown in the drawings, the camera 21 is installed, for example, on a support column standing on the side of the road.

  Since the camera 21 captures an area including the road, the image captured by the camera 21 is affected by weather conditions. FIG. 3 is a diagram illustrating an example of an image captured on a clear day by the camera 21 in FIG. FIG. 4 is a diagram showing the image shown in FIG. 3 with a rectangle forming the outline of the processing target area. In the pedestrian detection apparatus 11 according to the present embodiment, the processing target area includes a road, and a pedestrian as a detection target is detected in the processing target area. FIG. 5 binarizes the difference image of two (2 frames) images (one image is the image shown in FIG. 3) continuously captured on a clear day by the camera 21 in FIG. It is a figure which shows an example of the image after having performed, and shows the example of the image obtained by step S3 in FIG. 2 mentioned later.

  FIG. 6 is a diagram illustrating an example of an image captured by the camera 21 in FIG. FIG. 7 is an image obtained by binarizing a differential image of two images (one image is the image shown in FIG. 6) continuously captured by the camera 21 in FIG. It is a figure which shows an example, and has shown the example of the image obtained by step S3 in FIG. 2 mentioned later.

  Next, operation | movement of the pedestrian detection apparatus 11 in this Embodiment is demonstrated with reference to FIG. FIG. 2 is a schematic flowchart showing the operation of the pedestrian detection device 11 in FIG.

  When the pedestrian detection device 11 starts operating, the processing unit 24 first samples two consecutive images captured by the camera 21 (step S1), and stores the sampled images in the image memory 23. Next, the processing unit 24 generates a difference image (inter-frame difference image) between these two images (step S2). Note that the difference image used in the present invention is not necessarily limited to a difference image between two consecutive images. For example, when the number of frames per second is large, other difference images such as two images every other frame or plural frames are used. It may be a difference image between two images taken at a predetermined time interval.

  Note that two new images may be sampled in each step S1, and the difference image in step S2 may obtain a difference image between the two images newly sampled in this step S1, or each step. In S1, only one image may be sampled, and in the difference process in step S2, a difference image between the image sampled in the current step S1 and the image sampled in the previous step S1 may be obtained.

  Next, the processing unit 24 binarizes the difference image generated in step S2 (step S3). The threshold value used for the binarization may be a fixed threshold value or a variable threshold value represented by discriminant analysis method.

  Subsequently, the processing unit 24 acquires a label region by labeling from the processing target region of the image binarized in step S3 (step S4). In addition, you may perform the process of step S2, S3 only about the said process target area | region.

  Thereafter, the processing unit 24 averages each distance from the label area to each remaining label area for each label area acquired in step S4 (hereinafter, may be referred to as “inter-label area distance average”). Is acquired as an index value (step S5). Here, the distance between the two label areas is the straight line distance on the image between the centroids of the two label areas on the image, but it is not necessarily the distance between the centroids. It may be an actual distance converted instead of a distance. In place of the average value, a value corresponding to the average value, for example, an integrated value may be acquired as an index value, and a value corresponding to the average value may be used instead of the average value in the following processing. .

  Here, a specific example of the processing in step S5 will be described with reference to FIG. FIG. 8 is a diagram schematically illustrating an example of a label area and a distance between label areas. In order to simplify the description, in the example shown in FIG. 8, it is assumed that only three label regions a, b, and c are obtained from the processing target region in step S4. Actually, in step S4, more label regions, for example, tens to hundreds of label regions are acquired. In FIG. 8, the distance between the label areas (between ab) indicates the linear distance on the image between the center of gravity of the label area a and the center of gravity of the label area b, and the distance between label areas (between ac) is the center of gravity of the label area a. A linear distance on the image between the centroid of the label area c and the label area c (between bc) indicates a linear distance on the image between the centroid of the label area b and the centroid of the label area c. Yes.

  If only three label areas a, b, and c are obtained in step S4 as shown in FIG. 8, the average distance between label areas is calculated for all label areas a, b, and c in step S5. Is done. That is, for the label area a, the average value of distances from the label area a to the remaining label areas b and c (= {distance between label areas (between ab) + distance between label areas (between ac)} / 2 ) Is calculated. This is called the average distance between label areas of the label area a. For the label area b, the average value of the distances from the label area b to the remaining label areas a and c (= {distance between label areas (between ab) + distance between label areas (between bc)} / 2. ) Is calculated. This is called the average distance between label areas of the label area b. Further, for the label area c, the average value of the distances from the label area c to the remaining label areas a and b (= {distance between label areas (between ac) + distance between label areas (between bc)} / 2. ) Is calculated. This is called the average distance between label areas of the label area c.

Next, based on the average distance between label areas (index value) of all the label areas calculated in step S5, the processing unit 24 sets the average distance between label areas in the range for each predetermined range (class). A histogram (referred to as “label area distance average histogram”) indicating the number (frequency) of label areas having the average distance between label areas to which it belongs is created (step S6). Now, the number of the range (class) and p, _x 1 wherein the frequency of each class _{respectively,} x _2, x 3, · · ·, When _{x p,} the label area the distance between the average histogram them elements P-dimensional vector x = (x ₁ , x ₂ , x ₃ ,..., X _p ) ^T , which corresponds to one new sample vector in the p-dimensional multivariable vector space.

  After that, the processing unit 24 performs a predetermined process (this embodiment) on at least one of the two images subjected to the differential process in step S2 based on the distance average histogram between label areas created in step S6. Then, it is determined whether or not the image is not suitable for a pedestrian detection process in step S10 described later (steps S7 to S9).

  The inter-label area distance average histogram changes depending on the distribution status of the label areas in the processing target area of the two images picked up by the camera 21 and subjected to the difference processing in step S2, and becomes a feature amount indicating the distribution status. On the other hand, an image that is not suitable for the predetermined processing among images captured by the camera 21 has a certain tendency in the distribution status due to the influence of temporally changing noise or the like generated in the image. Therefore, according to the present embodiment, at least one of the two images picked up by the camera 21 and subjected to the difference processing in step S2 based on the average distance between histograms of the label areas is the predetermined process. Since it is determined whether or not the image is not suitable for the image, the determination can be performed appropriately.

  In the present embodiment, a group of label area distance average histograms corresponding to the label area distance average histogram of the image that is not suitable for the predetermined processing as the label area distance average histogram created in step S6 ( Whether or not at least one of the two images taken by the camera 21 and subjected to the difference processing in step S2 is an image that is not suitable for the predetermined processing. Is determined (steps S7 to S9). In this embodiment, the discrimination is performed by discriminant analysis using the Mahalanobis distance. However, in the present invention, the determination may be performed by a technique using a neural network or a support vector machine.

  In this embodiment, in order to perform discriminant analysis using the Mahalanobis distance, a plurality of groups (classes) including a group (class) of distance average histograms between label areas of images that are not suitable for the predetermined processing in advance. For each class, that is, a Mahalanobis distance calculation coefficient for each class used in step S7 to be described later, and these are stored in an internal memory (not shown) of the processing unit 24. Stored in advance.

  Here, an example of a method for obtaining a Mahalanobis distance calculation coefficient for each class used in step S7 described later will be described.

  As a result of the inventor's research based on a large number of images actually captured by the camera 21, in the pedestrian detection device as in the present embodiment, the night of wind and rain (see later explanation) (see later explanation) When the pedestrian detection process is performed from the images picked up on the pedestrian, the accuracy of pedestrian detection is greatly reduced, and it has been found that the image is not suitable for the pedestrian detection process. This is because, in an image captured in a rainy night, as shown in FIG. 6, the state of light on a wet road at night changes due to rebounding on a rainy road, changes in ripples due to wind, etc. As shown in FIG. 7, it is considered that the label area caused by a change in light on a wet road or the like in the difference image cannot be distinguished from the label area by a pedestrian.

  Therefore, the present inventor selected 11 classes of image groups from a large number of images captured over 24 hours and 9 months. The first class is an image group captured on a clear day, the second class is an image group captured on a clear night or a cloudy night, the third class is an image group captured on a cloudy day, Class 4 is an image group captured in rainy day, Class 5 is an image group captured in rainy night, Class 6 is an image group captured in rainy day, and Class 7 is wind and rain. The image group captured at night of night, the eighth class image group captured at day of snowfall, the ninth class image group captured at night of snowfall, and the tenth class imaged at daytime of snowfall The eleventh class was an image group captured on the night of snow. In the present invention, the number of classes to be selected is not necessarily limited to eleven.

  Here, the daytime refers to a state in which the lighting of the street light is not confirmed by looking at the display image, and the night means a state in which the lighting of the street light is not confirmed by looking at the display image. Moreover, it is a state where the road surface is dry, seeing the display image, whether it is clear or cloudy. Sunny and cloudy are distinguished only in the daytime. Sunny is a state in which a shadow is seen from the display image, and cloudy is a state in which a shadow is not seen from the display image. Sunny and cloudy are not distinguished at night. As for the wind and rain, in the state where the rain can be confirmed by looking at the display image, the precipitation amount is 2 mm or more according to the Meteorological Agency observation data at the place where the camera 21 is installed, or the precipitation amount is 1 mm according to the Meteorological Agency observation data. This is the state where the average wind speed is 6 m / s or more. Rain refers to a state that is not wind and rain in a state in which the rain can be confirmed by looking at the display image. Snow cover refers to a state where snow can be seen on the entire road surface by looking at the display screen regardless of the weather at that time. Snowfall refers to a state in which it is confirmed that snow is falling on a display image and only a small portion of the snow is falling.

  FIG. 9 is a schematic flowchart showing an example of a method for obtaining a Mahalanobis distance calculation coefficient for each class used in step S7 in FIG. FIG. 10 is a schematic flowchart showing details of step S21 in FIG.

  The method shown in FIGS. 9 and 10 can be realized using, for example, a personal computer. The image groups of the first to eleventh classes are stored in advance in a storage device such as a hard disk for each class.

  When a CPU such as a personal computer starts a process for obtaining a Mahalanobis distance calculation coefficient for each class, the class to be processed at the loop start end L21 is increased to the maximum number of classes (that is, from the first class). (Up to the eleventh class) The processing after the loop start end L21 is performed while sequentially changing and setting.

  When the class to be processed is set at the loop start end L21, the CPU performs a process of calculating the Mahalanobis distance calculation coefficient of the class (the class currently set as the process target at the loop start end L21). (Step S21), the process proceeds to the loop end L22. At the loop end L22, if the class currently set as the processing target at the loop start end L21 is not the last class among the maximum number of classes, the process returns to the loop start end L21 while the class is the last class. For example, the process for obtaining the Mahalanobis distance calculation coefficient for each class is terminated. The coefficient for calculating the Mahalanobis distance for each class obtained in this way is stored in advance in the internal memory (not shown) of the processing unit 24 as described above.

  When the processing of step S21 is started, the CPU stores two consecutively picked-up images to be processed at the loop start end L31 in FIG. 10 in the storage device. The processing after the loop start end L31 is performed while sequentially changing and setting from the image group of the class (the class currently set as the processing target at the loop start end L21) to the maximum number of image frames of the class.

  When two consecutively captured images to be processed are set at the loop start end L31, the CPU samples the two images from the storage device (step S31), and the sampled images are obtained. It is stored in an image memory (not shown) such as the personal computer. Next, the CPU generates a difference image (inter-frame difference image) between these two images (step S32).

  Next, the CPU binarizes the difference image generated in step S32 (step S33). Subsequently, the CPU acquires a label area by labeling from the processing target area of the image binarized in step S33 (step S34).

  Thereafter, the CPU obtains the average value of the distances from the label area to the remaining label areas (average distance between label areas) for each label area obtained in step S34, as in step S5 described above. (Step S35).

Next, in the same manner as Step S6 described above, the CPU determines the average distance between label areas for each predetermined range (class) based on the average distance between label areas of all label areas calculated in Step S35. A histogram (label area distance average histogram) indicating the number (frequency) of label areas having an average distance between label areas belonging to the range is created (step S36). Now, the number of the range (class) and p, _x 1 wherein the frequency of each class _{respectively,} x _2, x 3, · · ·, When _{x p,} the label area the distance between the average histogram them elements P-dimensional vector x = (x ₁ , x ₂ , x ₃ ,..., X _p ) ^T , which corresponds to one known sample vector in a p-dimensional multivariable vector space. Here, “known” means that it is known that the sample vector belongs to the class currently set as the processing target at the loop start end L21.

  After step S36, the process proceeds to the loop end L32. At the loop end L32, the two images currently set as processing targets at the loop starting end L31 are out of the maximum number of image frames of the class (class set as the current processing target at the loop start end L21). If it is not the last two images, the process returns to the loop start end L31, while if the two images are the last two images, the process proceeds to step S37.

In step S37, the CPU determines the average distance between the predetermined number of label areas obtained in step S36 each time for the class (the class currently set as the processing target at the loop start end L21). From the histogram (sample vector x = (x ₁ , x ₂ , x ₃ ,..., X _p ) ^T ), the population of the predetermined number of sample vectors x of the class as the Mahalanobis distance calculation coefficient of the class Mean vector μ = (μ ₁ , μ ₂ , μ ₃ ,..., Μ _p ) ^T and covariance matrix Σ are calculated. The covariance matrix Σ can be calculated by a known formula.

  When step S37 ends, step S21 ends, and the process proceeds to the loop end L22 in FIG. 9 described above.

  As described above, at the loop end L22, if the class currently set as the processing target at the loop start L21 is not the last class of the maximum number of classes, the loop returns to the loop start L21, while the class is If it is the last class, calculation of the Mahalanobis distance calculation coefficient for each class is completed for all of the first to eleventh classes, and the process of obtaining the Mahalanobis distance calculation coefficient for each class is completed. The Mahalanobis distance calculation coefficient for each class determined in this way is stored in advance in an internal memory (not shown) of the processing unit 24 as described above.

In the following description, the Mahalanobis distance of the kth class (k = 1, 2, 3,..., 11) obtained in this manner and stored in advance in an internal memory (not shown) of the processing unit 24. A mean vector and a covariance matrix, which are calculation coefficients, are denoted as μ _k and Σ _k , respectively.

With reference to FIG. 2 again, the specific processing contents after step S7 executed by the processing unit 24 in FIG. 1 will be described. Before that, step S6 will be described again. As described above, in step S6, the processing unit 24 calculates the distance average histogram between label regions based on the two images subjected to the difference processing in step S2, that is, one new Sample vector x = (x ₁ , x ₂ , x ₃ ,..., X _p ) ^T is acquired. It is unknown which class of the first to eleventh classes the sample vector x belongs to.

After step S6, the processing unit 24 calculates the mean vector μ _k and the covariance matrix Σ _k (k) that are the coefficients for calculating the Mahalanobis distance for each class stored in advance in an internal memory (not shown) of the processing unit 24. = 1, 2, 3,..., 11) The Mahalanobis distance to each class of the sample vector x = (x ₁ , x ₂ , x ₃ ,..., X _p ) ^T acquired in step S6. Calculate (step S7).

Sample vector x = (x ₁ , x ₂ , x ₃ ,..., X _p ) The Mahalanobis distance to the k-th (k = 1, 2, 3,..., 11) class of ^T is D _k . When expressed, the Mahalanobis distance D _k can be calculated by an equation of D _k = {(x−μ _k ) ^T Σ _k ⁻¹ (x−μ _k )} ^1/2 . In step S7, the processing unit 24, the Mahalanobis distance _{D 1,} D _2, D 3 by the equation, _..., is to calculate the _{D 11.}

Subsequently, the processing unit 24 compares the Mahalanobis distances D ₁ , D ₂ , D ₃ ,..., D ₁₁ calculated in step S7, and determines which one of the shortest Mahalanobis distances is one ( Step S8). In the discriminant analysis using the Mahalanobis distance, the sample vector x acquired in step S6 is determined to belong to the class having the shortest Mahalanobis distance. For example, the shortest Mahalanobis distance is determined to belong to the third class if D _3, the seventh if the shortest Mahalanobis distance is D ₇ classes (the image group captured at night weathertight Class).

Note that the present inventor has determined that the predetermined number of sample vectors (into the seventh class) that are the basis for calculating the mean vector μ ₇ and the covariance matrix Σ ₇ that are the coefficients for calculating the Mahalanobis distance of the seventh class. Each of the known sample vectors) is regarded as one new sample vector acquired in step S6, and the processing in steps S7 to S9 is performed. As a result, about 95% of the sample vectors are It was confirmed that it was determined to belong to class 7. As a result, according to the discriminant analysis using Mahalanobis in steps S7 to S9, an unknown image actually captured on a rainy night is an image captured on a rainy night with a very high probability of about 95%. It was possible to evaluate that it was discriminated as being.

After step S8, the processing unit 24 determines whether the shortest Mahalanobis distance, which is determined in step S8 is D ₇ (i.e., the night acquired sample vector x is the seventh class (weather at step S6 The difference processing is performed in step S2 depending on whether or not the images belong to the class of the captured image group) and, in turn, whether or not the two images subjected to the difference processing in step S2 are captured in the rainy night) It is determined whether or not the two images are images that are unsuitable (that is, unsuitable) for the pedestrian detection process in step S10 (step S9).

Processing unit 24, when the shortest Mahalanobis distance, which is determined in step S8 is D _7, in step S9, 2 images that have been difference processing in step S2 is suitable for pedestrian detection process in step S10 It is determined that the image is not present, and the process proceeds to step S13 without passing through step S10. On the other hand, the processing unit 24, when the shortest Mahalanobis distance is determined is not a D ₇ in step S8, in step S9, 2 images that have been difference processing in step S2 is the pedestrian detection process in step S10 It is determined that the image is not suitable, and the process proceeds to step S10.

  In step S10, the processing unit 24 performs a pedestrian detection process for detecting a pedestrian in the processing target area based on at least one of the two images subjected to the difference processing in step S2. This pedestrian detection process can be realized by a known method such as pattern recognition for the label area acquired in step S4.

  After step S10, the processing unit 24 determines whether or not a pedestrian has been detected as a result of the pedestrian detection process in step S10 (step S11). If a pedestrian is detected, the process proceeds to step S12. If no pedestrian is detected, the process proceeds to step S13.

  In step S12, the processing unit 24 starts outputting a pedestrian detection signal indicating that a pedestrian has been detected. This pedestrian detection signal is supplied to the signal control device 12 in FIG. If the pedestrian detection signal has already been output, the output is continued. After step S12, the process returns to step S1 to sample a new image.

  In step S13, the processing unit 24 stops outputting a pedestrian detection signal indicating that a pedestrian has been detected. If the output of the pedestrian detection signal is already stopped, the stopped state is continued. After step S13, the process returns to step S1 to sample a new image.

  In the present embodiment, the functions of steps S2 to S9 in FIG. 2 in the processing unit 24 correspond to the image determination apparatus according to an embodiment of the present invention.

  According to the present embodiment, as described above, at least one of the two images picked up by the camera 21 and subjected to the difference processing in step S2 based on the distance average histogram between the label areas is the above-described image. Since it is determined whether or not the image is not suitable for the predetermined process (in this embodiment, the pedestrian detection process in step S10), the determination can be performed appropriately.

  The label area distance average histogram is information obtained from the image captured by the camera 21 itself. Therefore, according to the present embodiment, it is determined whether at least one of the two images captured by the camera 21 and subjected to the difference processing in step S2 is an image that is not suitable for the predetermined processing. The information obtained from the image itself captured by the camera 21 can be reflected.

  In the present embodiment, whether or not the image captured by the camera 21 and affected by the weather condition is an image that is not suitable for the predetermined process is determined by using a signal from the sensor that detects the weather condition. Since it is performed without using it, it is not necessary to provide the sensor, and the cost can be reduced accordingly.

  Further, in the present embodiment, at least one of the two images captured by the camera 21 and subjected to the difference process in step S2 is the predetermined process (in this embodiment, the pedestrian detection process in step S10). If the image is determined not to be suitable for (when YES in step S9), the output of the pedestrian detection signal is stopped without performing the pedestrian detection process in step S10. As a result, the accuracy of person detection increases, and as a result, the control by the signal control device is performed more appropriately.

  Regardless of whether it is YES or NO in step S9, the processing after step S10 is always performed. If YES in step S9, the processing unit 24 indicates that and the pedestrian detection signal is invalid. The output of the signal to the effect (hereinafter referred to as “invalid signal”) to the signal control device 12 is started, and in the case of NO in step S9, the processing unit 24 may stop outputting the invalid signal. Good. In this case, the signal control device 12 receives the pedestrian detection signal from the processing unit 24 as valid only when the invalid signal is not received from the processing unit 24, and receives the invalid signal from the processing unit 24. If it is, the pedestrian detection signal from the processing unit 24 may be ignored.

  [Second Embodiment]

  FIG. 11 is a schematic flowchart showing the operation of the pedestrian detection apparatus as the processing apparatus according to the second embodiment of the present invention. The difference between the pedestrian detection device according to the present embodiment and the pedestrian detection device 11 according to the first embodiment is the point described below.

  In the present embodiment, after step S1, the processing unit 24 calculates the average gray value of the processing target area of one of the two images sampled in step S1 as the processing target area of the image. Is calculated as a brightness (step S42).

  Next, the processing unit 24 determines whether the brightness of the processing target area of the image is darker than the predetermined brightness based on whether the average gray value calculated in step S42 is lower than the predetermined value. (Step S43). The predetermined brightness is set, for example, to a brightness that is a threshold value for lighting / non-lighting a street lamp.

  If it is determined in step S43 that it is darker than the predetermined brightness, the process proceeds to step S2. If it is determined in step S43 that it is not darker than the predetermined brightness, the process proceeds to step S10.

  Thereby, in this Embodiment, the process part 24 is a partial area | region (all area | regions may be sufficient) of one image (both images may be sufficient) of the said 2 images imaged with the camera 21. .) Is required to be darker than the predetermined brightness, and at least one of the two images captured by the camera 21 is subjected to the predetermined processing (in the present embodiment, step S10). It is determined that the image is not suitable for the pedestrian detection process. In addition, it is good also as a necessary condition that the brightness of the one part or the whole area | region of at least one image of the said 2 images imaged with the camera 21 is darker than predetermined brightness.

  In general, when an image is bright, the image is dark while the influence of noise or the like is small and the image is suitable for the predetermined processing (in this embodiment, pedestrian detection processing in step S10). In some cases, the image is not suitable for the predetermined processing due to a large influence of noise or the like. Therefore, according to the present embodiment, the processing unit 24 is configured such that the brightness of a part or the entire area of at least one of the two images captured by the camera 21 is darker than a predetermined brightness. As a necessary condition, it is determined that at least one of the two images captured by the camera 21 is an image that is not suitable for the predetermined processing, so that the accuracy of the determination can be improved. it can.

In the present embodiment, as in the first embodiment, the processing unit 24 calculates the _eleven Mahalanobis distances D ₁ , D ₂ , D ₃ ,..., D ₁₁ in step S7. In step S8, it is determined whether the shortest Mahalanobis distance is any one. However, in the present embodiment, the processing unit 24 calculates the _five Mahalanobis distances D ₂ , D ₅ , D ₇ , D ₉ , and D ₁₁ related to the night in step S7, and the shortest Mahalanobis distance in step S8. It may be determined whether or not. In this case, since the first, third, fourth, sixth, eighth and tenth class Mahalanobis distance calculation coefficients related to daytime are not used, it is necessary to store them in the internal memory of the processing unit 24. Absent.

  Note that the pedestrian detection device according to the present embodiment can be used in place of the pedestrian detection device 11 according to the first embodiment in the traffic control system 1 shown in FIG.

  [Third Embodiment]

  FIG. 12 is a schematic flowchart showing the operation of the pedestrian detection device as the processing device according to the third embodiment of the present invention. The difference between the pedestrian detection device according to the present embodiment and the pedestrian detection device 11 according to the first embodiment is the point described below.

  In the present embodiment, when the operation is started, the processing unit 24 creates an initial background image (step S51). Details of step S51 will be described with reference to FIG. FIG. 13 is a schematic flowchart showing details of step S51 in FIG.

  When the process of step S51 is started, the processing unit 24 sets a count value n indicating the cumulative number of times to zero (step S61). Next, the processing unit 24 initializes the previous accumulated image f (t−1) to an image in which all pixels are zero (step S62). Thereafter, the processing unit 24 samples the current image from the camera 21 as the current image g (t) and loads it into the image memory 23 (step S63).

  Next, the processing unit 24 calculates f (t) = αf (t−1) + (1−α based on the previous accumulated image f (t−1) and the image g (t) sampled in step S63. ) The current cumulative image f (t) is created according to the equation g (t) (step S64). In this equation, α is a weighting coefficient that satisfies 0 <α <1, and is set to a value that reduces the influence of a moving object such as a pedestrian.

  Thereafter, the processing unit 24 increments the count value n stored in its internal memory (not shown) by 1 (step S65), and then determines whether or not the count value n has reached a predetermined cumulative number N. (Step S66). If the predetermined cumulative number N has not been reached, the processing unit 24 sets the current cumulative image f (t) as the previous cumulative image f (t-1) (step S67), and returns to step S63. On the other hand, if it is determined in step S66 that the predetermined cumulative number N has been reached, the processing unit 24 stores the current cumulative image f (t) in the image memory 23 as an initial background image (step S68). Thus, the initial background image creation process in step S51 is completed, and the process proceeds to step S52 in FIG. Note that an image captured by the camera 21 in a state where there is no moving object such as a pedestrian may be sampled, and this image may be used as an initial background image.

  In the following description, the background image means the initial background image stored in step S51 until updated in step S54, and after updated in step S54, the background image updated in step S54 is the latest updated background image. Means an image.

  In step S52, the processing unit 24 samples one image captured by the camera 21 (step S52), and stores the sampled image in the image memory 23. Thereafter, the process proceeds to step S3.

  In step S3, the processing unit 24 binarizes the difference image (background difference image in the present embodiment) generated in step S53. The threshold value used for the binarization may be a fixed threshold value or a variable threshold value represented by discriminant analysis method.

  Subsequently, the processing unit 24 acquires a label region by labeling from the processing target region of the image binarized in step S3 (step S4).

  Next, the processing unit 24 updates the background image (step S54), and then proceeds to step S5. In step S54, the background image is updated by, for example, calculating the image f (t) according to the above-described equation by setting the current image to g (t) and the current background image to f (t-1). This can be done by setting t) as the updated background image.

  Also in the present embodiment, the processing after step S5 is the same as in the case of the first embodiment. In the present embodiment, after step S12 and after step S13, the process returns to step S52 to sample a new image.

  Also in this embodiment, the Mahalanobis distance calculation coefficient for each class used in step S7, which is stored in advance in an internal memory (not shown) of the processing unit 24, is the same as that in the first embodiment. It may be exactly the same as the case. However, in the present embodiment, it is more preferable that the Mahalanobis distance calculation coefficient for each class used in step S7 is obtained using a background difference image.

  Here, an example of a method for obtaining the Mahalanobis distance calculation coefficient for each class used in step S7 using the background difference image will be described.

  The method according to this example can also be realized by using a personal computer or the like, and also in the method according to this example, the image groups of the first to eleventh classes are stored in advance in a storage device such as a hard disk for each class. Keep it.

When a CPU such as a personal computer starts a process for obtaining a Mahalanobis distance calculation coefficient for each class, first, images of the first class of images stored in the storage device are continuously captured. For a series of images, the processing corresponding to steps S51, S52, S53, S3, S4, S54, S5, and S6 in FIG. 12 is repeated to calculate a plurality of label area distance average histograms related to the series of images. In addition, among the first class of image groups stored in the storage device, a plurality of label area distance average histograms for the remaining series of images are respectively calculated. Thus, for the first class of image groups stored in the storage device, a predetermined number of label area distance average histograms (sample vectors x = (x ₁ , x ₂ , x ₃ ,..., X _p ) ^T ) is obtained. Next, the CPU calculates an average vector μ = (μ ₁ , μ ₂ , μ ₃ ,..., Μ _p ) ^T and a covariance matrix of the predetermined number of sample vectors x of the first class. Σ is calculated as a coefficient for calculating the Mahalanobis distance of the first class.

Thereafter, the CPU performs the same processing as the processing described above for the first class of image groups stored in the storage device for the second to eleventh classes stored in the storage device. For each class of image group, the average vector μ = (μ ₁ , μ ₂ , μ ₃ ,..., Μ _p ) ^T and the covariance matrix Σ of the classes of the second to eleventh classes. Is calculated as a Mahalanobis distance calculation coefficient for each class.

  In the present embodiment, the Mahalanobis distance calculation coefficient for each class obtained in this way may be stored in advance in an internal memory (not shown) of the processing unit 24.

  Also in this embodiment, the same advantages as those in the first embodiment can be obtained. Note that the pedestrian detection device according to the present embodiment can be used in place of the pedestrian detection device 11 according to the first embodiment in the traffic control system 1 shown in FIG.

  In the present invention, a modification similar to that obtained by modifying the first embodiment may be applied to the second embodiment.

  Although the embodiments of the present invention have been described above, the present invention is not limited to these embodiments.

  For example, in each of the above-described embodiments, the pedestrian detection device is described as an example of the processing device according to the present invention. However, the present invention is not limited to this, and predetermined processing (vehicle) The present invention can also be applied to various processing apparatuses having processing means for performing detection processing or other processing. For example, the present invention can be applied to a road monitoring device similar to a conventional road monitoring device.

11 Pedestrian Detection Device 21 Camera 24 Processing Unit

Claims (8)

A label area by labeling from a predetermined area of a difference image between two images captured by the imaging means at a predetermined time interval or from a predetermined area of a difference image between the image captured by the imaging means and the background image Labeling means to obtain,
For each label area acquired by the labeling means, an index value acquisition means for acquiring an average value of distances from the label area to the remaining label areas or an index value that is a value corresponding thereto;
A histogram acquisition means for acquiring a histogram indicating the number of the label regions having the index value belonging to the range for each predetermined range of the index value;
Based on the histogram, whether at least one of the two images captured by the imaging unit is an image that is not suitable for a predetermined process, or the image captured by the imaging unit Determination means for determining whether or not the image is not suitable for the predetermined processing;
An image determination apparatus comprising:   The determination unit determines whether the histogram belongs to a group of histograms corresponding to the histogram of images that are not suitable for the predetermined processing, thereby determining the two images captured by the imaging unit. Whether at least one of the images is not suitable for the predetermined processing, or whether the image captured by the imaging unit is not suitable for the predetermined processing The image determination apparatus according to claim 1, wherein:   The image determination apparatus according to claim 2, wherein the determination is performed by determination analysis using a Mahalanobis distance.   The determination means is based on the requirement that the brightness of a part or the whole area of at least one of the two images taken by the imaging means is darker than a predetermined brightness. It is determined that at least one of the two images captured by the image processing is not suitable for the predetermined processing, or a part or the whole of the image captured by the imaging unit 2. The image processing apparatus according to claim 1, wherein the image captured by the imaging unit is determined to be an image that is not suitable for the predetermined process on the condition that the brightness of the region is darker than the predetermined brightness. 4. The image determination device according to any one of 3.   5. The two images captured by the imaging unit or the images captured by the imaging unit are images that are affected by weather conditions. Image judging device. An image determination apparatus that determines whether an image captured by an imaging unit and affected by weather conditions is an image that is not suitable for predetermined processing,
An image determination apparatus comprising: means for performing the determination based on the image without using a signal from a sensor that detects the weather condition.   The image determination apparatus according to claim 1, wherein the predetermined process is a process of detecting a detection target. An image determination apparatus according to any one of claims 1 to 7,
Processing means for performing the predetermined processing on the image picked up by the image pickup means;
A processing apparatus comprising: