JP2012221162A - Object detection device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable an object to be accurately detected even when the object stands still.SOLUTION: A window image extraction unit 20 extracts a window image from a captured image captured by an imaging device 12. A score calculation unit 22, based on a discrimination model for each pedestrian direction class and the window image, calculates a score indicating pedestrian likelihood for each direction class. A pedestrian discrimination unit 26 discriminates whether the window image is a pedestrian image based on a distribution of the calculated score for each direction class.

Description

  The present invention relates to an object detection device and a program, and more particularly to an object detection device and a program for detecting an object from a captured image.

  2. Description of the Related Art Conventionally, there has been known a vehicle periphery monitoring device that uses a visible camera that captures a visible image and detects a predetermined object such as a pedestrian even during night driving (Patent Document 1). In this vehicle periphery monitoring device, it is determined whether a pedestrian or a non-pedestrian is detected by detecting temporal changes in luminance values of left and right feet illuminated by night lights.

  Also, an obstacle detection device that accurately identifies an obstacle from a normal visible image and reduces the load of image processing is known (Patent Document 2). In this obstacle detection device, a time-series motion pattern of a pedestrian is learned.

  In addition, a vehicle periphery capable of quickly determining an object such as a pedestrian to be avoided from contact with the vehicle from an image around the vehicle and presenting information to the driver and controlling vehicle behavior A monitoring device is known (Patent Document 3). In this vehicle periphery monitoring device, the posture is estimated using asymmetry, the degree of inclination of the body, and the change in relative position.

  There is also known an image processing apparatus capable of accurately extracting and identifying a specific attribute of an object from an image captured by a relatively inexpensive imaging unit mounted on a vehicle (Patent Document 4). In this image processing apparatus, the attribute of a pedestrian is determined by a self-organizing map.

JP 2010-205087 A JP 2004-145660 A JP 2007-293487 A JP 2007-323177 A

  However, the technique described in Patent Document 1 is a method limited to nighttime, and has a problem that only pedestrians moving in the front-rear direction are targeted for detection. In addition, attention is paid to movement, and there is a problem that stationary pedestrians and non-pedestrians cannot be distinguished.

  In addition, the technique described in Patent Document 2 focuses on movement, and there is a problem that a stationary pedestrian and a non-pedestrian cannot be identified.

  In addition, the feature amount used in the technique described in Patent Document 3 appears prominently for a moving pedestrian and cannot be applied to a stationary pedestrian. Further, there is a problem that the posture is used for determining whether or not a pedestrian has a high risk of collision, and cannot be applied to a method for removing false detections.

  In addition, the technique described in Patent Document 4 has a problem in that it only determines an attribute and does not determine whether it is a pedestrian or a non-pedestrian.

  The present invention has been made to solve the above-described problem, and an object of the present invention is to provide an object detection device and a program that can detect an object with high accuracy even when the object is stationary. And

  In order to achieve the above object, an object detection apparatus according to a first aspect of the present invention identifies an extraction means for extracting a window image from a captured image obtained by imaging the periphery of the apparatus, and identifies the object for each classification related to the attribute of the object. Score calculating means for calculating a score indicating the likelihood of the object for each classification relating to the attribute, based on the identification model for performing and the window image extracted by the extracting means; and calculating by the score calculating means Identification means for identifying whether or not the window image is an image representing the object based on the score for each classification related to the attribute.

  According to a second aspect of the present invention, there is provided a program for extracting a window image from a captured image obtained by capturing an image of the periphery of the device, an identification model for identifying the object for each classification related to the attribute of the object, Based on the window image extracted by the extraction means, for each classification related to the attribute, a score calculation means for calculating a score indicating the likelihood of the object, and for each classification related to the attribute calculated by the score calculation means It is a program for causing a window image to function as an identification unit for identifying whether or not the window image is an image representing the object based on a score.

  According to the first invention and the second invention, the window image is extracted from the captured image obtained by imaging the periphery of the own apparatus by the extracting means. Based on an identification model for identifying the object for each classification related to the attribute of the object by the score calculation means and the window image extracted by the extraction means, the object for each classification related to the attribute A score indicating the likelihood is calculated.

  Then, the identifying means identifies whether the window image is an image representing the object based on the score for each classification related to the attribute calculated by the score calculating means.

  As described above, based on the identification model for each category relating to the attribute of the object, a score is calculated for each category relating to the attribute, and based on the score for each category relating to the attribute, whether the window image is an image representing the object or not. By identifying, even if the object is stationary, the object can be detected with high accuracy.

  According to a third aspect of the present invention, there is provided an object detection apparatus for extracting a window image from a captured image obtained by imaging the periphery of the own apparatus, and for extracting the window image so as to partially overlap the window image extracted by the extraction means. For each of a plurality of peripheral images with shifted regions, the target object is classified for each attribute-related classification based on an identification model for identifying the target object for each classification regarding the target object attribute and the peripheral image. A score calculation means for calculating a score indicating the likelihood, and a classification of attributes for which the score satisfies a predetermined condition, obtained from a score for each of the attributes for each of the plurality of surrounding images calculated by the score calculation means; Alternatively, identification means for identifying whether the window image is an image representing the object based on a cumulative score for each attribute classification , It is configured to include a.

  According to a fourth aspect of the present invention, there is provided a program for extracting a window image from a captured image obtained by imaging the periphery of its own device, and a region of the window image so as to partially overlap the window image extracted by the extraction unit. For each of the plurality of peripheral images shifted from each other, based on the identification model for identifying the object for each classification related to the attribute of the target object and the peripheral image, the likelihood of the object for each classification related to the attribute A score calculation means for calculating a score indicating the above, and a classification of attributes for which the score satisfies a predetermined condition, obtained from a score for each of the attributes for each of the plurality of surrounding images calculated by the score calculation means, or Identify whether the window image is an image representing the object based on a cumulative score for each category of the attribute Is a program for functioning as that identification means.

  According to the third and fourth aspects of the invention, the window image is extracted from the captured image obtained by capturing the periphery of the own apparatus by the extracting means. For each of a plurality of peripheral images in which the window image area is shifted so as to partially overlap with the window image extracted by the extracting unit, the object is identified for each classification related to the object by the score calculating unit. On the basis of the identification model for doing so and the surrounding image, a score indicating the likelihood of the object is calculated for each classification related to the attribute.

  Then, the classification unit obtains from the score for each category related to the attribute for each of the plurality of peripheral images calculated by the score calculation unit, the classification of the attribute that satisfies the predetermined condition, or for each classification of the attribute Based on the cumulative score, it is determined whether or not the window image is an image representing the object.

  As described above, for each of the plurality of peripheral images in which the window image is shifted, a score is calculated for each classification related to the attribute based on the identification model for each classification regarding the attribute of the object, and the attribute related to each of the plurality of peripheral images is related. By identifying whether the window image is an image representing an object based on the score for each classification, the object can be detected with high accuracy even when the object is stationary.

  According to a fifth aspect of the present invention, there is provided an object identification device including: an extracting unit that extracts a plurality of window images from each of a time series of captured images obtained by imaging the periphery of the own device; Based on the identification model for identifying the object for each classification related to the attribute and the plurality of window images extracted by the extraction means, the object-likeness for each window image is determined for each classification regarding the attribute. Each of the plurality of window images is based on a score for each classification related to the attribute calculated by the score calculation unit for each of the time series of the captured images, Candidate image determination means for determining whether the image is a candidate image representing the image, and is determined to be the candidate image in the time series of the captured image. A tracking means for tracking the window image; and a score for each category related to the attribute calculated for the time series of the window image tracked in the time series of the captured image, wherein the score satisfies a predetermined condition. An identification means for identifying whether the window image is an image representing an object based on a classification of attributes to be satisfied, a cumulative score for each classification of the attributes, or a change in an attribute classification for which the score is a maximum value; It is comprised including.

  According to a sixth aspect of the invention, there is provided a program for extracting a plurality of window images from each of a time series of captured images obtained by imaging the periphery of the device, and attribute of an object for each of the time series of the captured images. Based on an identification model for identifying the object for each classification related to and a plurality of window images extracted by the extracting means, the object likelihood for each window image is indicated for each classification related to the attribute A score calculation means for calculating a score, and for each time series of the captured image, each of the plurality of window images represents an object based on a score for each classification related to the attribute calculated by the score calculation means. Candidate image determination means for determining whether or not the image is a candidate image, in the time series of the captured image, Tracking means for tracking the determined window image, and the score obtained from the score for each category related to the attribute calculated for the time series of the window image tracked in the time series of the captured image, Identification means for identifying whether or not the window image is an image representing an object based on a classification of an attribute that satisfies a condition, a cumulative score for each classification of the attribute, or a change in a classification of an attribute having the maximum score It is a program to make it function as.

  According to the fifth and sixth inventions, the extraction means extracts a plurality of window images from each of the time series of the captured images obtained by imaging the periphery of the own device. Based on each of the time series of the captured images by the score calculation means based on the identification model for identifying the object for each classification related to the attribute of the object, and the plurality of window images extracted by the extraction means. Thus, for each classification related to the attribute, a score indicating the object likeness for each window image is calculated.

  Then, each of the plurality of window images represents an object based on a score for each classification related to the attribute calculated by the score calculation unit for each of the time series of the captured images by the candidate image determination unit. It is determined whether the image is a candidate image. The window image determined to be the candidate image in the time series of the captured image is tracked by the tracking unit.

  Then, the classification of the attribute satisfying a predetermined condition, the score obtained from the score for each classification related to the attribute calculated for the time series of the window image tracked in the time series of the captured image by the identification unit, Whether the window image is an image representing an object is identified based on a cumulative score for each attribute classification or a change in an attribute classification with the maximum score.

  In this way, for the time series of window images tracked in the time series of captured images, a score is calculated for each category related to the attribute based on the identification model for each category related to the attribute of the object, By identifying whether or not the window image is an image representing the object based on the score for each classification relating to the attribute for each of the time series, the object can be accurately detected even when the object is stationary. it can.

  Moreover, said object can be made into a pedestrian or a two-wheeled vehicle.

  Moreover, when a target object is a pedestrian, said attribute can be made into pedestrian direction, age, or sex.

  The storage medium for storing the program of the present invention is not particularly limited, and may be a hard disk or a ROM. Further, it may be a CD-ROM, a DVD disk, a magneto-optical disk or an IC card. Furthermore, the program may be downloaded from a server or the like connected to the network.

  As described above, according to the present invention, a score is calculated for each category related to the attribute based on the identification model for each category related to the attribute of the object, and the window image is processed based on the score for each category related to the attribute. By identifying whether or not the image represents an object, it is possible to obtain an effect that the object can be accurately detected even when the object is stationary.

It is a block diagram which shows the structure of the target object detection apparatus which concerns on 1st Embodiment. It is a figure which shows the image showing the pedestrian of each direction. It is a figure for demonstrating the method of calculating | requiring the direction of a pedestrian. It is a figure which shows the image showing a non-pedestrian. It is a figure for demonstrating the example of the score for every classification | category of the direction calculated with each of a pedestrian image and a non-pedestrian image. It is a flowchart which shows the content of the target object detection processing routine in the target object detection apparatus of 1st Embodiment. It is a block diagram which shows the structure of the target object detection apparatus which concerns on 2nd Embodiment. It is a figure for demonstrating a peripheral partial image. It is a figure for demonstrating the example of the score for every classification | category of the direction calculated with each of a pedestrian image and a non-pedestrian image. (A) The figure which shows the histogram which showed the frequency of the classification | category of the direction which has the maximum score obtained from the periphery partial image extracted from the pedestrian periphery, and (b) From the periphery partial image extracted from the non-pedestrian periphery It is a figure which shows the histogram which showed the frequency of the classification | category of the direction with the obtained maximum score. It is a flowchart which shows the content of the target object detection processing routine in the target object detection apparatus of 2nd Embodiment. It is a flowchart which shows the content of the target object detection processing routine in the target object detection apparatus of 2nd Embodiment. It is a block diagram which shows the structure of the target object detection apparatus which concerns on 3rd Embodiment. It is a figure for demonstrating the change of the classification | category of the direction used as the maximum score in the time series of a pedestrian image, and the time series of a non-pedestrian image. It is a figure which shows the example of a weight according to the pattern of direction change. It is a flowchart which shows the content of the target object detection processing routine in the target object detection apparatus of 3rd Embodiment. It is a flowchart which shows the content of the target object detection processing routine in the target object detection apparatus of 3rd Embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the present embodiment, a case will be described in which the present invention is applied to an object detection device that is mounted on a vehicle and detects a pedestrian as an object.

  As illustrated in FIG. 1, the object detection device 10 according to the first embodiment includes an imaging device 12 that captures a range including an identification target region in front of the host vehicle, and a captured image output from the imaging device 12. The computer 16 which performs the target object detection processing routine which detects a pedestrian based on this, and the display apparatus 18 for displaying the processing result in the computer 16 are provided.

  The imaging device 12 images a range including an identification target region in front of the host vehicle, generates an image signal (not shown), and converts an image signal that is an analog signal generated by the imaging unit into a digital signal. An A / D converter (not shown), and an image memory (not shown) for temporarily storing the A / D converted image signal. The image to be used may be color or monochrome, and may be a visible light image or a near red image.

  The computer 16 is connected to a CPU that controls the entire object detection apparatus 10, a ROM as a storage medium that stores a program for an object detection processing routine described later, a RAM that temporarily stores data as a work area, and these. It is configured to include a bus. In the case of such a configuration, a program for realizing the function of each component is stored in a storage medium such as a ROM or HDD, and each function is realized by executing the program by the CPU. To.

  When the computer 16 is described with functional blocks divided for each function realizing means determined based on hardware and software, as shown in FIG. 1, a captured image captured by the imaging device 12 and input to the computer 16 is acquired. An image acquisition unit 19 that extracts a window image of a predetermined region from the acquired captured image, and a feature amount extraction unit 21 that extracts an image feature amount of the window image extracted by the window image extraction unit 20 And a score calculation unit 22 for calculating a score indicating the pedestrian-likeness by comparing the image feature amount extracted for the window image with the identification model, an identification model storage unit 24 in which the identification model is stored, and the window image Including a pedestrian identification unit 26 that identifies whether or not is an image representing a detection target pedestrian. It can be represented by.

  The window image extraction unit 20 cuts an image from a captured image while moving a predetermined size window (referred to as a search window) by a predetermined amount of movement (referred to as a search step) per step. Here, the cut image is referred to as a window image, and the size of the window image (that is, the size of the search window) is referred to as a window size. A plurality of types of window sizes are set to detect pedestrians of various sizes, and the window image extraction unit 20 extracts window images using search windows of all the set window sizes. The window image extraction unit 20 converts the extracted window image into an image having a preset number of pixels.

  The feature amount extraction unit 21 extracts an image feature amount for each window image. As the image feature amount, Haar-Like Feature, HOG (Histograms of Oriented Gradients), FIND (Feature Interaction Descriptor), etc. can be used. Regarding FIND, non-patent literature (Hui CAO, Koichiro YAMAGUCHI, Mitsuhiko OHTA, Takashi NAITO and Yoshiki NINOMIYA: "Feature Interaction Descriptor for Pedestrian Detection", IEICE TRANSACTIONS on Information and Systems, Volume E93-D No.9, pp. .2651-2655, 2010) may be used, and detailed description thereof is omitted.

  The identification model storage unit 24 stores an identification model that is generated by learning in advance and that is referred to when the score calculation unit 22 calculates a score. Here, the case where the identification model storage unit 24 is provided in the computer 16 will be described. However, the identification model is stored in the storage unit of another external device and connected to the other external device via a network or communication unit. And it is good also as a structure which reads the identification model memorize | stored in the memory | storage means of another external apparatus.

  Here, the identification model stored in the identification model storage unit 24 will be described.

  First, in pedestrian detection, detection performance is improved by using a model learned for each attribute classification such as orientation. This is because, as shown in FIG. 2, there is a difference in the posture of the pedestrian depending on the orientation, so that the pedestrian is stably detected by the identification model for each orientation classification. On the other hand, non-pedestrians such as plants and road surfaces may be erroneously detected as pedestrians. Therefore, if such erroneous detection can be reduced, more stable pedestrian detection can be expected. In the following, attributes are described as orientations for explanation.

  A method that does not use the identification model for each orientation classification can only detect the position and size of the pedestrian. In such a case, motion information is used to detect the direction, but there is a problem that it cannot cope with a stationary pedestrian. When using an identification model for each direction classification, it is possible to detect the direction of a stationary pedestrian because it uses the appearance (pattern information). FIG. 3 shows a method for obtaining the direction of a pedestrian using an identification model for each direction classification.

  For each identification model, a score indicating the certainty of whether there is a pedestrian belonging to the classification of the attribute is calculated, and the classification of the direction in which the score becomes the maximum value is set as the direction of the pedestrian.

  Therefore, in the present embodiment, the identification model storage unit 24 stores the orientation-specific identification models learned for each pedestrian orientation classification (for example, front-rear direction, left direction, right direction).

  In the learning process for the identification model according to orientation, a predetermined number of learning images of the target object in which the pedestrian was previously photographed and non-target learning images in which the person other than the pedestrian was photographed for each pedestrian orientation classification Prepare and classify the learning images for each pedestrian orientation, and for each pedestrian orientation classification, the learning images for the objects classified into the orientation classification and non-objects The learning image is used for learning according to the image feature amount and the teacher label of each learning image, and an identification model for the classification of the direction is generated.

  In addition, the score calculation unit 22 calculates a score for each window image by the classifier for each direction classification based on the extracted image feature amount and the identification model for each direction classification. Boosting or SVM can be used as an identifier. It is also output from the discriminator using the method described in non-patent literature (HT Lin, CJ Lin and RC Weng: “A note on Platt's probabilistic outputs for support vector machines”, Machine Learning, Springer, 2007). A value obtained by converting a score to a probability value may be used as the score.

  Next, the principle of this embodiment will be described.

  For example, in the non-pedestrian image as shown in FIG. 4, the shape of the upper body relatively matches that of the pedestrian, but the lower body has no edge and is significantly different from the shape of the pedestrian. The contour shape of the pedestrian's upper body (especially shoulders and torso) is not as great as the leg, and tends to match the pedestrian model in any direction. In such a case, since almost the same score is calculated in any direction, a phenomenon that a difference in score is difficult to occur between attributes occurs. On the other hand, for pedestrian images, as shown in FIG. 5, there is a large difference in scores for each orientation classification. When a score is calculated by applying an identification model for each orientation classification to a pedestrian image, a high value is shown for a correct orientation classification, and a low value is shown for other orientation classifications. Therefore, in order to determine whether only one orientation category has a high score, it is possible to distinguish between pedestrians and non-pedestrians by evaluating the relationship between the scores in each orientation category.

  In the present embodiment, as will be described below, the pedestrian identification unit 26 performs pedestrian identification processing from the scores calculated for each orientation classification (referred to as N scores) for each window image. .

  First, the non-pedestrian window image is removed according to the score. Specifically, when all scores are lower than a preset threshold, it is determined as a non-pedestrian, and when at least one score is large, it is determined as a pedestrian. The threshold value set at this time may be the same for all orientation categories, or a different value may be set for each orientation category.

  In addition to the threshold processing described above, pedestrian identification processing is performed based on the score distribution in each orientation classification.

  For example, the difference between the maximum score and the minimum score is calculated from the score distribution in each orientation classification, and if the obtained value is less than the threshold, the window image is identified as a non-pedestrian image, and the threshold If it is above, it will identify as a pedestrian image.

  Note that a difference between all scores or a variance value of scores may be calculated from the distribution of scores in each orientation classification, and a threshold determination may be performed in the same manner to identify whether the image is a pedestrian image.

  In addition, the pedestrian identification unit 26 controls the display device 18 to display the identification result superimposed on the captured image.

  Next, an object detection processing routine executed by the computer 16 of the object detection apparatus 10 according to the first embodiment will be described with reference to FIG.

  In step 100, a captured image captured by the imaging device 12 is acquired. Next, in step 102, a search window is set as a captured image with respect to the captured image, and a window is acquired from the captured image using the set search window. Image x is extracted.

  Next, in step 104, image feature amounts are extracted from the window image x extracted in step 102. In step 106, a variable K for identifying the identification model is set to 1 as an initial value, and in step 108, the score is determined based on the K-th orientation-specific identification model and the image feature amount extracted in step 104 above. Is calculated. In step 110, the variable K is incremented by 1. In step 112, it is determined whether or not the variable N is equal to or smaller than a constant N indicating the number of identification models. If the variable K is less than or equal to the constant N, the process returns to step 108. On the other hand, if the variable K is greater than the constant N, it is determined that the score has been calculated for all orientation-specific identification models, and the process proceeds to step 114. Transition.

  In step 114, it is identified whether or not the window image x is a pedestrian image based on the score for each direction classification calculated in step 108.

  In the next step 116, it is determined whether or not the image is identified as a pedestrian image in step 114. If it is identified that the image is not a pedestrian image, the process proceeds to step 120 described later. If it is identified as an image, in step 118, the window image x is recorded as a pedestrian area, and the process proceeds to step 120.

  In step 120, it is determined whether or not the search is completed by scanning the search window for the entire captured image acquired in step 100. If not completed, the process returns to step 102, the window image is extracted from the position where the position of the search window has been moved by a predetermined search step, and the processing of step 102 to step 118 is repeated. When the search for the entire image in the current size search window is completed, the process returns to step 102 in the same manner, the size of the search window is changed, and the processing from step 102 to step 118 is repeated. When the search in all size search windows is completed for the entire captured image, the process proceeds to step 122.

  In step 122, based on the pedestrian area recorded in step 118 as an output of the detection result, the display device 18 is displayed on the captured image so that the detected pedestrian is surrounded by a window. Control and end the process.

  As described above, according to the object detection device 10 of the first embodiment, a score is calculated for each orientation classification based on the identification model for each classification regarding the pedestrian orientation, By identifying whether the window image is an image representing a pedestrian based on the score distribution, it is possible to detect a pedestrian with high accuracy even when the pedestrian is stationary.

  Further, by evaluating the distribution of the scores of the classifications in each direction, it is possible to distinguish between pedestrians and non-pedestrians, and it is possible to eliminate false detections such as plants that occur in pedestrian detection.

  Moreover, since the score with respect to each direction classification is calculated from the pedestrian pattern information, even a stationary pedestrian can be detected with high accuracy.

  Next, a second embodiment will be described. In addition, about the part which becomes the structure similar to 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  In the second embodiment, with respect to the window image, the first point is that whether or not it is a pedestrian image is identified based on the scores calculated for a plurality of peripheral partial images shifted in the periphery. This is different from the embodiment.

  As illustrated in FIG. 7, the object detection device 210 according to the second embodiment includes an imaging device 12, a computer 216, and a display device 18. When the computer 216 is described with functional blocks divided for each function realization means determined based on hardware and software, as shown in FIG. 7, the image acquisition unit 19, the window image extraction unit 20, and the window image will be described later. A peripheral part image extracting unit 220 that extracts a plurality of peripheral part images, a feature amount extracting unit 221 that extracts image feature amounts for each window image and a plurality of peripheral part images of each window image, and each window image and each window image A score calculation unit 222 that calculates a score for a plurality of peripheral partial images, an identification model storage unit 24, and a pedestrian identification unit 226 that identifies whether the window image is an image representing a detection target pedestrian. It can be expressed in a configuration that includes it.

  As shown in FIG. 8, the peripheral partial image extraction unit 220 extracts a plurality of peripheral partial images from a plurality of areas in which the window image areas are shifted so as to partially overlap the window image.

  The feature amount extraction unit 221 extracts, for each window image, the image feature amount of the window image and the image feature amounts of a plurality of peripheral partial images corresponding to the window image.

  The score calculation unit 222 calculates, for each window image, a score for each direction classification based on the extracted image feature amount and the identification model for each direction classification, and for each window image. For the peripheral partial images, a score is calculated in the same manner for each orientation classification based on the extracted image feature amount and the identification model for each orientation classification.

  Next, the principle in the present embodiment will be described.

  As shown in FIG. 9, image differences due to slight changes in the position and size of the cut-out image affect the score for each orientation classification. As described in the first embodiment above, in the non-pedestrian image, the score for each orientation classification may show a very similar value. In such a case, the magnitude relation of the score with respect to the classification in each direction may be reversed by a slight change in the score due to the difference in the image depending on the position and size of the cut-out image. For this reason, a change occurs in the classification of the attribute having the maximum score between the peripheral partial images. On the other hand, as shown in FIG. 9, in the pedestrian image, although the score varies due to the deviation, there is a large difference in the score between the correct orientation and the other directions, so that the attribute classification that becomes the maximum score does not change.

  Therefore, in the present embodiment, as described below, the pedestrian identification unit 226 calculates, for each window image, a score (N) calculated for each orientation classification with respect to the window image and a plurality of peripheral partial images. Pedestrian identification processing is performed.

  First, the window image of a non-pedestrian is removed based on the score of the window image being noticed. Specifically, when all scores are lower than a preset threshold, it is determined as a non-pedestrian, and when at least one score is large, it is determined as a pedestrian. The threshold set at this time may be the same for all attributes, or a different value may be set for each orientation classification.

  In addition to the threshold processing, pedestrian identification processing is performed on the window image using the scores of the peripheral partial images.

  FIGS. 10A and 10B are histograms showing the classification frequency of the direction having the maximum score obtained from the peripheral partial images extracted from the pedestrian periphery. As shown in FIG. 10A, the histogram obtained from the peripheral image around the pedestrian shows a high peak in the correct direction classification. On the other hand, the histogram obtained from the peripheral partial image around the non-pedestrian shown in FIG. In this way, it is possible to identify a pedestrian and a non-pedestrian using the distribution of the classification of the direction in which the maximum score is obtained in a plurality of peripheral partial images with respect to the window image of interest.

  For example, the difference between the maximum score and the minimum score is calculated from the frequency distribution of the classification in the direction in which the maximum score for the window image of interest is obtained, and if the obtained value is less than the threshold, the window image is a non-pedestrian It is identified as an image, and if it is equal to or greater than a threshold value, it is identified as a pedestrian image. The frequency distribution of the classification in the direction in which the maximum score is obtained is an example of the distribution of the classification of attributes whose score satisfies a predetermined condition.

  The histogram created at this time may be obtained by counting the classification of the direction in which the score is the maximum value for each peripheral partial image, or may be a histogram obtained by weighting and counting each score. An entropy value can be used to evaluate the histogram.

  Next, an object detection processing routine executed by the computer 216 of the object detection device 210 according to the second embodiment will be described with reference to FIGS. 11 and 12. In addition, about the process same as 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  In step 100, a captured image captured by the imaging device 12 is acquired. Next, in step 102, a search window is set as a captured image with respect to the captured image, and a window is acquired from the captured image using the set search window. Image x is extracted. In step 104, an image feature amount is extracted from the window image x extracted in step 102.

  In step 106, a variable K for identifying the identification model is set to 1 as an initial value, and in step 108, the score is determined based on the K-th orientation-specific identification model and the image feature amount extracted in step 104 above. Is calculated. In step 110, the variable K is incremented by 1. In step 112, it is determined whether or not the variable N is equal to or smaller than a constant N indicating the number of identification models. If the variable K is less than or equal to the constant N, the process returns to step 108. On the other hand, if the variable K is greater than the constant N, it is determined that the scores have been calculated for all the orientation-specific identification models, and the process proceeds to step 250. Transition.

  In step 250, a plurality of peripheral partial images are extracted from the window image x extracted in step 102. In step 252, a variable S for identifying the peripheral partial image is set to 1 which is an initial value, and in step 254, an image feature amount is extracted from the S-th peripheral partial image extracted in step 250. In step 106, a variable K for identifying the identification model is set to an initial value of 1. In step 255, the image feature extracted for the Kth orientation-specific identification model and the Sth peripheral partial image in step 254 above. A score is calculated based on the quantity. In step 110, the variable K is incremented by 1. In step 112, it is determined whether or not the variable N is equal to or smaller than a constant N indicating the number of identification models. If the variable K is less than or equal to the constant N, the process returns to step 255. On the other hand, if the variable K is greater than the constant N, it is determined that the scores have been calculated for all the orientation-specific identification models, and the process proceeds to step 256. Transition.

  In step 256, the variable S is incremented by 1. In step 258, it is determined whether or not the variable S is equal to or less than a constant M indicating the number of peripheral partial images. If the variable S is less than or equal to the constant M, the process returns to step 254. On the other hand, if the variable S is greater than the constant M, it is determined that the scores have been calculated for all peripheral partial images, and the process proceeds to step 260. To do.

  In step 260, the window image x is determined to be a pedestrian based on the score for each orientation classification calculated for the window image x in step 108 and the score for each orientation classification calculated for each peripheral partial image in step 255. Identify whether it is an image.

  In the next step 116, it is determined whether or not the image is identified as a pedestrian image in step 260. If it is identified that the image is not a pedestrian image, the process proceeds to step 120. If it is determined that there is a window image, the window image x is recorded as a pedestrian area in step 118, and the process proceeds to step 120.

  In step 120, it is determined whether or not the search is completed by scanning the search window for the entire captured image acquired in step 100. If the search has not been completed, the process returns to step 102. On the other hand, if the search in the search windows of all sizes is completed for the entire captured image, the process proceeds to step 122.

  In step 122, based on the pedestrian area recorded in step 118 as an output of the detection result, the display device 18 is displayed on the captured image so that the detected pedestrian is surrounded by a window. Control and end the process.

  As described above, according to the object detection device 210 of the second embodiment, for each of the plurality of peripheral partial images shifted from the window image, based on the identification model for each classification related to the pedestrian orientation, A score is calculated for each orientation classification, and the pedestrian is stopped by identifying whether the window image is an image representing a pedestrian based on the orientation classification score for each of the plurality of peripheral partial images. Pedestrians can be detected with high accuracy.

  In the above-described embodiment, the case where the pedestrian image is identified is described as an example based on the frequency distribution of the classification with the direction of the maximum score obtained for a plurality of peripheral partial images. However, the present invention is not limited to this. For example, from the cumulative score distribution in each orientation classification obtained for a plurality of peripheral partial images, the difference between the maximum score and the minimum score, the difference between all scores, or the variance value of the scores is calculated, and the calculated value A threshold determination may be performed to identify whether the image is a pedestrian image.

  Next, a third embodiment will be described. In addition, about the structure same as the target object detection apparatus 10 of 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  In the third embodiment, a pedestrian candidate area is tracked in captured images that are continuously captured, and a pedestrian image is identified based on a score calculated for the tracked pedestrian candidate area. This is different from the first embodiment.

  As illustrated in FIG. 13, the object detection device 310 according to the third embodiment includes an imaging device 12, a computer 316, and a display device 18. If the computer 316 is described with function blocks divided for each function realization means determined based on hardware and software, as shown in FIG. 13, an image acquisition unit that acquires a time series of captured images continuously captured 319, the window image extraction unit 20, the feature amount extraction unit 21, the score calculation unit 22, the identification model storage unit 24, and the time series of the captured images are determined whether or not the window image is a pedestrian candidate region. A pedestrian candidate determination unit 324 that performs tracking processing for each pedestrian candidate area, and a pedestrian that identifies whether or not the window image that is a pedestrian candidate area that has been tracked is a pedestrian image. And an identification unit 328.

  The window image extraction unit 20 extracts a window image of a predetermined area from each time-series of captured images. The feature amount extraction unit 21 extracts image feature amounts from the window images extracted from the time series of the captured images. The score calculation unit 22 compares the extracted image feature quantity with the orientation-specific identification model for each window image for each time series of the captured image, and calculates a score indicating the pedestrian-likeness.

  The pedestrian candidate determination unit 324 performs threshold processing on each window image for each time series of captured images, similarly to the pedestrian identification unit 26 described in the first embodiment, and performs non-walking. While removing the pedestrian image, pedestrian identification processing is performed based on the score distribution calculated for each orientation classification, and the window image identified as the pedestrian image is set as a pedestrian candidate region.

  Next, the principle in the present embodiment will be described.

  As described in the second embodiment, image differences due to slight changes in position and size affect the score for each orientation classification. In the image captured by the in-vehicle camera, the size and the viewing angle gradually change for the same object due to the movement of the host vehicle. For this reason, as shown in FIG. 14, in the pedestrian image, as in the second embodiment, there is no change in the attribute classification that becomes the maximum score between time series. On the other hand, in non-pedestrian images, the reversal of the magnitude relationship between scores for each orientation classification is likely to occur due to image differences between time series, and the attribute classification that gives the maximum score changes between time series images. .

  Therefore, in the present embodiment, the tracking processing unit 326 performs the tracking process on the window image determined by the pedestrian candidate determination unit 324 as a pedestrian candidate area. The tracking processing unit 326 associates the area of the window image determined as a pedestrian candidate area at the current time t with the pedestrian tracking model tracked up to time t-1. An αβ tracker can be used to update the pedestrian tracking model. In the pedestrian tracking model, a score for each classification of orientation calculated for the tracked pedestrian candidate area at each time is recorded. If no pedestrian tracking model corresponding to the window area determined as the pedestrian candidate area at the current time t is found, a new pedestrian tracking model is generated. In the pedestrian tracking model tracked up to time t−1, when a window image determined to be a pedestrian candidate region corresponding to the current time t is not found, the classification of each direction of the current time t of the pedestrian tracking model is performed. Set the score to NULL.

  In this tracking process, a process for determining whether or not a person is a pedestrian is performed using the maximum score of the scores for each time direction classification according to the method described in the patent document (Japanese Patent Application Laid-Open No. 2005-311691). Also good.

  The pedestrian identification unit 328 uses the pedestrian tracking model tracked by the tracking processing unit 326 to perform pedestrian identification processing from the scores calculated for each orientation classification at each time for each of the pedestrian tracking models. Do.

  For example, based on the pedestrian tracking model, the degree of change in the classification of the orientation that becomes the maximum score is evaluated, and if the degree of change is equal to or greater than a threshold, the window image corresponding to the pedestrian tracking model is a non-pedestrian image If it is less than the threshold, it is identified as a pedestrian image.

  As a method of evaluating the degree of change in direction, the number of changes in the direction classification that becomes the maximum score may be counted based on the score calculated for each direction classification at each time. Further, as shown in FIG. 15, the number of changes may be counted by applying a weight according to the change pattern.

  Next, an object detection processing routine executed by the computer 316 of the object detection device 310 according to the third embodiment will be described with reference to FIGS. In addition, about the process same as 1st Embodiment, the same code | symbol is attached | subjected and detailed description is abbreviate | omitted.

  In step 100, a captured image captured by the imaging device 12 is acquired. Next, in step 102, a search window is set as a captured image with respect to the captured image, and a window is acquired from the captured image using the set search window. Image x is extracted.

  Next, in step 104, image feature amounts are extracted from the window image x extracted in step 102. In step 106, a variable K for identifying the identification model is set to 1 as an initial value, and in step 108, the score is determined based on the K-th orientation-specific identification model and the image feature amount extracted in step 104 above. Is calculated. In step 110, the variable K is incremented by 1. In step 112, it is determined whether the variable N is equal to or smaller than a constant N indicating the number of orientation-specific identification models. If the variable K is less than or equal to the constant N, the process returns to step 108. On the other hand, if the variable K is greater than the constant N, it is determined that the score has been calculated for all orientation-specific identification models, and the process proceeds to step 114. Transition.

  In step 114, it is identified whether or not the window image x is a pedestrian image based on the score for each direction classification calculated in step 108.

  In the next step 116, it is determined whether or not the image is identified as a pedestrian image in step 114. If it is identified that the image is not a pedestrian image, the process proceeds to step 120 described later. If it is identified as an image, in step 350, the window image x is recorded as a pedestrian candidate area, and the process proceeds to step 120.

  In step 120, it is determined whether or not the search is completed by scanning the search window for the entire captured image acquired in step 100. If not completed, the process returns to step 102, a window image is extracted from the position where the position of the search window has been moved by a predetermined search step, and the processing of steps 102 to 116, 350 is repeated. If the search of the entire image in the search window of the current size is completed, the process returns to step 102 in the same manner, the size of the search window is changed, and the processes of steps 102 to 116 and 350 are repeated. If the search in all size search windows is completed for the entire captured image, the process proceeds to step 352.

  In step 352, one pedestrian candidate area is selected from the pedestrian candidate areas recorded in step 350. In step 354, the pedestrian candidate area selected in step 352 is tracked using the pedestrian tracking model obtained by the tracking process up to one hour ago.

  In the next step 356, it is determined whether or not the pedestrian candidate area selected in the above step 352 can be tracked from the pedestrian tracking model up to one hour ago. A tracking model is created, and the process proceeds to step 364 described later.

  On the other hand, if the tracking can be performed from the pedestrian tracking model up to one hour ago, in step 358, the window image which is a pedestrian candidate area tracked at each time is tracked based on the pedestrian tracking model. Whether or not the window image is a pedestrian image is identified based on the time-series data of the scores for each orientation classification calculated in (1).

  In the next step 360, it is determined whether or not the image is identified as a pedestrian image in step 358. If the image is not identified as a pedestrian image, the process proceeds to step 364 described later. If it is identified as an image, the window image at the current time is recorded as a pedestrian area in step 362, and the process proceeds to step 362.

  In step 362, it is determined whether or not the processing in steps 352 to 362 has been completed for all pedestrian candidate areas. If not, the process returns to step 352, and walking that has not been selected yet. A candidate area is selected, and the processes of steps 354 to 362 are repeated. On the other hand, when the processing of step 352 to step 362 is completed for all pedestrian candidate regions, the process proceeds to step 122, and the detection result output is based on the pedestrian region recorded in step 362. Then, the display device 18 is controlled so that the detected pedestrian is displayed surrounded by a window with respect to the captured image at the current time, the process returns to step 100, and the image is captured at the next time by the imaging device 12. The captured image is acquired and the above-described processing is executed.

  As described above, in the object detection processing routine, the above-described processing is repeatedly executed for each time series of captured images captured by the imaging device 12.

  As described above, according to the object detection apparatus 310 according to the third embodiment, the time series of the window images tracked in the time series of the photographed images is used as the identification model for each classification regarding the pedestrian orientation. And calculating a score for each orientation classification and identifying whether the window image is an image representing a pedestrian based on the score for each orientation classification for each time series of tracked window images Even when the pedestrian is stationary, the pedestrian can be detected with high accuracy.

  In addition, by evaluating the time-series direction change state, it is possible to distinguish between pedestrians and non-pedestrians and to eliminate false detections.

  In the above-described embodiment, whether or not the window image is a pedestrian image based on the number of changes in the direction classification that is the maximum score obtained for the tracked pedestrian candidate region (window image). However, the present invention is not limited to this. For example, the relationship of the score with respect to each orientation classification may be determined in time series. The difference between the maximum score and the minimum score for each orientation classification at each time, the difference between all orientation scores, or the variance value is accumulated in time series, and if the accumulated value is less than or equal to a preset threshold, the walking The window image corresponding to the person tracking model may be identified as a non-pedestrian image.

  Further, based on the frequency distribution of the classification of the direction that becomes the maximum score obtained for the tracked pedestrian candidate region (window image), the window image is converted into the pedestrian image as in the second embodiment. You may make it identify whether it is.

  In addition, the difference between the maximum score and the minimum score, the difference between all scores, or the variance value of the scores is calculated from the distribution of cumulative scores in each orientation classification obtained for the tracked pedestrian candidate area (window image). The threshold value may be determined for the calculated value to identify whether the image is a pedestrian image. In addition, when adding scores in time series, the weights may be reduced each time they move away from the current time, and may be determined as a non-pedestrian image if the total value is equal to or less than a preset threshold value.

  Moreover, although the case where the tracking process from the tracking result up to one hour before the pedestrian candidate area at the current time is detected has been described as an example, the present invention is not limited to this and is not limited to this. A corresponding area may be searched from the tracking result, and a pedestrian candidate area at the current time may be detected based on the search result.

  In the third embodiment, the technique using the plurality of peripheral partial images described in the second embodiment may be applied.

  Moreover, in said 1st Embodiment-3rd Embodiment, although the case where direction was used was demonstrated as an example as a pedestrian attribute, it is not limited to this, An adult, a child, An attribute related to age having a classification such as an elderly person or a sex related to classification having a classification such as gender may be used as an attribute. In this case, an age-specific identification model or a sex-specific identification model may be learned and prepared in advance. Moreover, you may make it divide the classification | category of the attribute of a pedestrian like direction + the degree of leg spread (for example, large / medium / small / none).

  In the first to third embodiments described above, the case of performing the process of removing the non-pedestrian's window image by the score threshold process has been described as an example. You may abbreviate | omit the process which removes a pedestrian's window image.

  Moreover, although the case where the detection target object is a pedestrian has been described as an example, the present invention is not limited to this. For example, a two-wheeled vehicle such as a bicycle may be used as the detection target object.

10, 210, 310 Object detection device 12 Imaging device 16, 216, 316 Computer 20 Window image extraction unit 21, 221 Feature extraction unit 22, 222 Score calculation unit 24 Identification model storage unit 26, 226, 328 Pedestrian identification unit 220 Peripheral partial image extraction unit 324 Pedestrian candidate determination unit 326 Tracking processing unit

Claims (12)

Extraction means for extracting a window image from a captured image obtained by imaging the periphery of the device;
Based on an identification model for identifying the object for each classification relating to the attribute of the object and the window image extracted by the extracting means, a score indicating the likelihood of the object is obtained for each classification relating to the attribute. A score calculating means for calculating;
Identification means for identifying whether the window image is an image representing the object based on a score for each classification related to the attribute calculated by the score calculation means;
An object detection apparatus including:   2. The object detection according to claim 1, wherein the identification unit identifies whether the window image is an image representing the object based on a score distribution for each classification related to the attribute calculated by the score calculation unit. apparatus. Extraction means for extracting a window image from a captured image obtained by imaging the periphery of the device;
An identification model for identifying the object for each of the classifications related to the attributes of the object for each of a plurality of peripheral images in which the window image area is shifted so as to partially overlap the window image extracted by the extracting unit And a score calculation means for calculating a score indicating the likelihood of the object for each classification related to the attribute based on the peripheral image;
Based on an attribute classification for which the score satisfies a predetermined condition, or a cumulative score for each attribute classification, obtained from the score for each of the attributes for each of the plurality of surrounding images calculated by the score calculation means Identifying means for identifying whether the window image is an image representing the object;
An object detection apparatus including:   The identification unit is obtained from a score for each classification related to the attribute for each of the plurality of peripheral images calculated by the score calculation unit, and the distribution of the attribute classification for which the score satisfies a predetermined condition, or the classification of the attribute The target object detection apparatus according to claim 3, wherein the window image identifies whether or not the window image is an image representing the target object based on a cumulative score distribution for each. Extraction means for extracting a plurality of window images from each of the time series of captured images obtained by imaging the periphery of the device;
For each of the time series of the captured images, based on an identification model for identifying the object for each classification related to the attribute of the object, and the plurality of window images extracted by the extraction unit, Score calculating means for calculating a score indicating the likelihood of the object for each window image for each classification;
Whether or not each of the plurality of window images is a candidate image representing an object based on a score for each classification related to the attribute calculated by the score calculation means for each of the time series of the captured images. Candidate image determining means for determining;
Tracking means for tracking the window image determined to be the candidate image in a time series of the captured images;
Obtained from the score for each category related to the attribute calculated with respect to the time series of the window image tracked in the time series of the captured image, the category of the attribute for which the score satisfies a predetermined condition, for each category of the attribute Identification means for identifying whether the window image is an image representing an object based on a cumulative score or a change in the classification of the attribute having the maximum score,
An object detection apparatus including:   The identification unit is obtained from a score for each classification related to the attribute calculated with respect to the time series of the window image tracked in the time series of the captured image, and the distribution of attribute classifications in which the score satisfies a predetermined condition The object according to claim 5, wherein the window image is an image representing an object based on a distribution of a cumulative score for each attribute classification, or a change in an attribute classification with the maximum score. Object detection device.   The object detection device according to claim 1, wherein the object is a pedestrian or a two-wheeled vehicle.   The object detection device according to claim 7, wherein, when the object is a pedestrian, the attribute is a pedestrian orientation, age, or gender.   The object detection device according to claim 1, wherein the identification model is learned in advance for each classification related to an attribute of the object. Computer
Extraction means for extracting a window image from a captured image obtained by imaging the periphery of the own device;
Based on an identification model for identifying the object for each classification relating to the attribute of the object and the window image extracted by the extracting means, a score indicating the likelihood of the object is obtained for each classification relating to the attribute. A score calculation means for calculating, and a program for functioning as an identification means for identifying whether or not the window image is an image representing the object based on the score for each classification related to the attribute calculated by the score calculation means . Computer
Extraction means for extracting a window image from a captured image obtained by imaging the periphery of the own device;
An identification model for identifying the object for each of the classifications related to the attributes of the object for each of a plurality of peripheral images in which the window image area is shifted so as to partially overlap the window image extracted by the extracting unit And a score calculation unit that calculates a score indicating the likelihood of the object for each classification related to the attribute based on the peripheral image, and the attribute for each of the plurality of peripheral images calculated by the score calculation unit An identification for identifying whether the window image is an image representing the object based on a classification of an attribute satisfying a predetermined condition, or a cumulative score for each classification of the attribute, obtained from a score for each classification Program to function as a means. Computer
Extraction means for extracting a plurality of window images from each of the time series of captured images obtained by imaging the periphery of the own device;
For each of the time series of the captured images, based on an identification model for identifying the object for each classification related to the attribute of the object, and the plurality of window images extracted by the extraction unit, Score calculating means for calculating a score indicating the likelihood of the object for each window image for each classification;
Whether or not each of the plurality of window images is a candidate image representing an object based on a score for each classification related to the attribute calculated by the score calculation means for each of the time series of the captured images. Candidate image determining means for determining,
Tracking means for tracking the window image determined to be the candidate image in the time series of the captured image, and the time calculated for the time series of the window image tracked in the time series of the captured image The window image is obtained from a score for each attribute-related classification, the attribute classification for which the score satisfies a predetermined condition, a cumulative score for each attribute classification, or a change in the attribute classification for which the score is the maximum value. A program for functioning as an identification means for identifying whether or not an image represents an object.