JP5829155B2 - Pedestrian detection device and program - Google Patents

Description

  The present invention relates to a pedestrian detection apparatus and program, and more particularly to a pedestrian detection apparatus and program for detecting a pedestrian from a captured image.

  In pedestrian detection, pedestrian detection may be undetected due to the variety of pedestrian posture patterns, and pedestrian detection is performed using flow (motion) information because the pedestrian is a moving object. An apparatus is known (for example, Patent Document 1).

  In this pedestrian detection device, a region in which movements in each image obtained by continuously capturing images with a camera are identified as a moving body region, and a lower part of the moving body region in which movements in each image are aligned. , Set the lower area of the size corresponding to the size of the moving body area, determine whether or not the movement of the feature points in the set lower area is uneven, the lower part where the movement is determined to be uneven The moving body area corresponding to the area and the lower area is specified as the whole body area of the pedestrian.

JP 2009-157581 A

  However, since the technique described in Patent Document 1 is a detection method based on a flow, only a moving pedestrian is a detection target. In addition, the technique described in Patent Literature 1 described above groups flow information obtained from an image, and performs pedestrian determination using non-uniformity in the flow of legs in the grouped region and an image pattern. Yes. However, there is a problem that it is difficult to extract a pedestrian as a moving body region because non-uniformity of the flow also occurs in the upper body part due to the movement of the arm swing at the time of crossing. Furthermore, since the pedestrian traveling in parallel with the own vehicle has little change in the appearance of the leg portion, there is a problem that the non-uniformity of the flow in the lower part of the moving body region does not easily occur and is difficult to detect.

  The present invention has been made to solve the above-described problems, and it is an object of the present invention to provide a pedestrian detection apparatus and program capable of stably detecting a pedestrian regardless of the movement of the pedestrian. And

  In order to achieve the above object, the pedestrian detection device of the present invention corresponds to each feature point between the captured images from a plurality of time-series captured images obtained by imaging the periphery of the own device mounted on the moving body. A flow extraction unit that extracts a flow indicating the movement of the feature point, a moving object flow extraction unit that extracts a flow on a moving object from the flow extracted by the flow extraction unit, and a window image extracted from the captured image Score calculation for calculating a score indicating the pedestrian-likeness based on a window image extracting means for performing identification, a pre-learned identification model for identifying a pedestrian, and the window image extracted by the window extracting means And the moving object flow extraction hand based on a predetermined flow size range corresponding to the distance to the pedestrian candidate on the captured image. The pedestrian flow extraction means for extracting a flow having a size within the range from the flow on the moving object in the window image extracted by the above, an extraction result by the pedestrian flow extraction means, and the score calculation means Identification means for identifying whether or not the window image is an image representing the pedestrian based on the score calculated by the above.

  The program according to the present invention shows, for each feature point, the movement of the corresponding feature point between the captured images from a plurality of time-series captured images obtained by imaging the periphery of the own device mounted on the mobile body. A flow extracting means for extracting a flow; a moving object flow extracting means for extracting a flow on a moving object from a flow extracted by the flow extracting means; a window image extracting means for extracting a window image from the captured image; Score calculating means for calculating a score indicating the pedestrian-likeness based on an identification model for identifying a pedestrian and the window image extracted by the window extracting means, pedestrian candidates on the captured image Extracted by the moving object flow extraction means based on a predetermined flow size range corresponding to the distance to Calculated from the flow on the moving object in the window image by the pedestrian flow extraction means for extracting a flow having a size within the range, the extraction result by the pedestrian flow extraction means, and the score calculation means It is a program for functioning as an identification means for identifying whether or not the window image is an image representing the pedestrian based on the score.

  According to the present invention, for each feature point from the plurality of time-series captured images obtained by capturing the periphery of the own device mounted on the moving body by the flow extraction unit, the movement of the corresponding feature point between the captured images. Is extracted. The flow on the moving object is extracted from the flow extracted by the flow extracting means by the moving object flow extracting means.

  Then, a window image is extracted from the captured image by the window image extracting means. A score indicating the pedestrian-likeness is calculated by the score calculation means based on the identification model for identifying the pedestrian learned in advance and the window image extracted by the window extraction means.

  Then, the window extracted by the moving object flow extraction unit based on a predetermined flow size range corresponding to the distance to the pedestrian candidate on the captured image by the pedestrian flow extraction unit. A flow having a size within the range is extracted from the flow on the moving object in the image. The identifying means identifies whether the window image is an image representing the pedestrian based on the extraction result by the pedestrian flow extracting means and the score calculated by the score calculating means.

  As described above, the flow within the predetermined flow size range corresponding to the distance to the pedestrian candidate is extracted from the flow on the moving object in the window image, and the flow extraction result and the pedestrian By identifying whether the window image is an image representing the pedestrian based on the score indicating the likelihood, the pedestrian can be stably detected regardless of the movement of the pedestrian.

  The range of the size of the flow may be determined in advance so that the size of the flow becomes smaller as the distance to the pedestrian candidate on the captured image increases.

  The range of the size of the flow is such that the larger the lateral position of the pedestrian candidate obtained from the position of the window image on the captured image is, the larger the size of the flow is. As such, it can be determined in advance.

  The identifying means according to the present invention corrects the score calculated by the score calculating means according to the number of the flows extracted by the pedestrian flow extracting means, and based on the corrected score, Whether the window image is an image representing the pedestrian or not can be identified.

  The score calculation means according to the present invention calculates the pedestrian-likeness for each orientation based on the identification model learned in advance for each pedestrian orientation and the window image extracted by the window extraction means. The pedestrian flow extraction means calculates a predetermined flow size corresponding to the distance to the pedestrian candidate on the captured image and the direction in which the score is maximized. Based on the range, a flow having a size within the range can be extracted from the flow on the moving object in the window image. As a result, the flow of what seems to be a pedestrian can be further limited and extracted.

  The score calculation means according to the present invention calculates the pedestrian-likeness for each posture based on the identification model learned in advance for each posture of the pedestrian and the window image extracted by the window extraction means. The identification means calculates the presence / absence of a pedestrian movement predetermined for the posture with the maximum score and the number of flows extracted by the pedestrian flow extraction means. The score calculated by the score calculation means is corrected according to the consistency, and it is possible to identify whether the window image is an image representing the pedestrian based on the corrected score. .

  The moving object flow extraction means according to the present invention can extract the flow on the moving object from the flow extracted by the flow extraction means based on the motion of the moving body.

  The storage medium for storing the program of the present invention is not particularly limited, and may be a hard disk or a ROM. Further, it may be a CD-ROM, a DVD disk, a magneto-optical disk or an IC card. Furthermore, the program may be downloaded from a server or the like connected to the network.

  As described above, according to the present invention, a flow within a predetermined flow size range corresponding to the position of the window image is extracted from the flow on the moving object in the window image. By detecting whether the window image is an image representing the pedestrian based on the extraction result and the score indicating the pedestrian-likeness, the pedestrian can be stably detected regardless of the movement of the pedestrian. The effect of being able to be obtained.

It is a block diagram which shows the structure of the pedestrian detection apparatus which concerns on 1st Embodiment. It is a figure which shows the whole body model of the pedestrian of each direction. It is a figure which shows the part model of the pedestrian of each attitude | position. It is a figure for demonstrating the method of calculating | requiring a pedestrian's direction and attitude | position. It is a graph which shows the relationship between the distance of a pedestrian and the magnitude | size of a flow. It is a figure which shows the distance to a pedestrian. It is a figure which shows the window image on a captured image. It is a figure for demonstrating the method of calculating the distance from the lower end position of the window image on a captured image to a pedestrian candidate. It is a flowchart which shows the content of the pedestrian detection process routine in the pedestrian detection apparatus of 1st Embodiment. It is a flowchart which shows the flow of the calculation process of the score using the whole body model in the pedestrian detection apparatus of 1st Embodiment. It is a flowchart which shows the flow of the calculation process of the score using the site | part model in the pedestrian detection apparatus of 1st Embodiment. It is a graph which shows the relationship between the distance of a pedestrian and the magnitude | size of a flow. It is a figure which shows the movement information added to the site | part model.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the present embodiment, a case will be described in which the present invention is applied to a pedestrian detection device that is mounted on a vehicle and detects a pedestrian.

  As illustrated in FIG. 1, the pedestrian detection device 10 according to the first embodiment includes an imaging device 12 that captures a range including an identification target region in front of the host vehicle, and a captured image output from the imaging device 12. The computer 16 which performs the pedestrian detection process routine which detects a pedestrian based on this, and the display apparatus 18 for displaying the process result in the computer 16 are provided.

  The imaging device 12 continuously captures a range including an identification target region in front of the host vehicle, and generates an image signal (not shown), and an image signal that is an analog signal generated by the imaging unit An A / D converter (not shown) for converting the signal into a digital signal, and an image memory (not shown) for temporarily storing the A / D converted image signal. The image to be used may be color or monochrome, and may be a visible light image or a near red image.

  The computer 16 is a CPU that controls the pedestrian detection apparatus 10 as a whole, a ROM as a storage medium that stores a program for a pedestrian detection processing routine, which will be described later, a RAM that temporarily stores data as a work area, and a connection between them. It is configured to include a bus. In the case of such a configuration, a program for realizing the function of each component is stored in a storage medium such as a ROM or HDD, and each function is realized by executing the program by the CPU. To.

  When the computer 16 is described with functional blocks divided for each function realization means determined based on hardware and software, as shown in FIG. 1, a plurality of images captured continuously by the imaging device 12 and input to the computer 16 An image acquisition unit 19 that acquires a captured image of the image, a window image extraction unit 20 that extracts a window image of a predetermined area from the acquired captured image, and an image feature amount extracted from the window image extracted by the window image extraction unit 20 The feature amount extraction unit 21, the score calculation unit 22 that compares the image feature amount extracted for the window image with the identification model and calculates a score indicating the pedestrian-likeness, and the identification model storage unit that stores the identification model 24.

  The window image extraction unit 20 cuts an image from a captured image while moving a predetermined size window (referred to as a search window) by a predetermined amount of movement (referred to as a search step) per step. Here, the cut image is referred to as a window image, and the size of the window image (that is, the size of the search window) is referred to as a window size. A plurality of types of window sizes are set to detect pedestrians of various sizes, and the window image extraction unit 20 extracts window images using search windows of all the set window sizes. The window image extraction unit 20 converts the extracted window image into an image having a preset number of pixels.

  The feature amount extraction unit 21 extracts an image feature amount for each window image. As the image feature amount, Haar-Like Feature, HOG (Histograms of Oriented Gradients), FIND (Feature Interaction Descriptor), etc. can be used. Regarding FIND, non-patent literature (Hui CAO, Koichiro YAMAGUCHI, Mitsuhiko OHTA, Takashi NAITO and Yoshiki NINOMIYA: "Feature Interaction Descriptor for Pedestrian Detection", IEICE TRANSACTIONS on Information and Systems, Volume E93-D No.9, pp. .2651-2655, 2010) may be used, and detailed description thereof is omitted.

  Further, the feature amount extraction unit 21 extracts an image feature amount for a corresponding local region for each part of a part model to be described later in the window image.

  The identification model storage unit 24 stores an identification model that is generated by learning in advance and that is referred to when the score calculation unit 22 calculates a score. Here, the case where the identification model storage unit 24 is provided in the computer 16 will be described. However, the identification model is stored in the storage unit of another external device and connected to the other external device via a network or communication unit. And it is good also as a structure which reads the identification model memorize | stored in the memory | storage means of another external apparatus.

  Here, the identification model stored in the identification model storage unit 24 will be described.

  In the detection of pedestrians using image patterns, since the postures of pedestrians are very diverse, there are cases where pedestrians are not detected or objects similar to pedestrians are erroneously detected. In view of such circumstances, not only detection is performed using a pedestrian model of the entire pedestrian, but also detection is performed using models of each part of the pedestrian (for example, the pedestrian's head, trunk, and leg). There is a case. In this case, undetected and erroneous detection can be reduced by extracting rough pedestrian candidates based on the whole body model and then performing more detailed identification using the pedestrian model of each part.

  Moreover, the pedestrian model of the whole body and each part utilized at this time can improve a performance more by producing | generating a model for every some attitude | position, for example, it shows to FIG. 2 (A)-(C) In this way, a pedestrian model of the whole body is generated for each pedestrian orientation, and several models corresponding to postures are generated from models of each part as shown in FIGS. 3 (A) to (D). Detection performance can be improved. In this way, by configuring the pedestrian model for each orientation and posture, it is possible to know not only the position and size of the pedestrian but also the orientation of the pedestrian and the posture of each part as shown in FIG. become.

  Therefore, in the present embodiment, the identification model storage unit 24 stores, as an identification model, a whole body model for each orientation learned for each pedestrian direction (for example, front-rear direction, left direction, right direction), and pedestrians. The part model for each posture learned for each posture is stored for each part.

  In the learning process for the orientation-based whole body model, for each pedestrian orientation, a predetermined number of pedestrian learning images in which pedestrians have been photographed and non-pedestrian learning images in which other than pedestrians have been photographed are prepared. The learning image of the object is classified for each direction of the pedestrian, and for each direction of the pedestrian, the learning image for the pedestrian classified in the direction and the learning image for the non-pedestrian are used. Then, learning is performed according to the image feature amount and the teacher label of each learning image, and an identification model for the direction is generated.

  In the learning process of the position-specific part model, for each posture of the pedestrian part, a learning image of the pedestrian part in which the part of the pedestrian is photographed in advance, and for non-pedestrian learning in which the part other than the pedestrian is photographed Prepare a predetermined number of images, classify the pedestrian part learning image for each posture of the part, and for each pedestrian part posture, the pedestrian part learning image classified into the posture, Learning is performed in accordance with the image feature amount of each learning image and the teacher label using the non-pedestrian learning image, and an identification model for the posture of the part is generated.

  In addition, the score calculation unit 22 determines, for each window image, whether or not there is a pedestrian in the window image by the discriminator for each direction based on the extracted image feature amount and the whole body model for each direction. A score indicating the certainty level is calculated. Boosting or SVM can be used as an identifier. It is also output from the discriminator using the method described in non-patent literature (HT Lin, CJ Lin and RC Weng: “A note on Platt's probabilistic outputs for support vector machines”, Machine Learning, Springer, 2007). A value obtained by converting a score to a probability value may be used as the score.

  Moreover, the score calculation part 22 determines as a non-pedestrian, when each of the magnitude | size of the score calculated using all the whole body models classified by direction is lower than a threshold value, and when that is not right, it is after that as a pedestrian candidate. Continue processing. The threshold value set here may be the same for all orientations, or a different value may be set for each orientation. At this time, the direction of the pedestrian in the window image is determined based on the score obtained using the whole body model for each direction with respect to the window image set as the pedestrian candidate. In the example of FIG. 3 above, as a result of the score calculation using the whole body model according to the orientation in the three directions, the score when the left-facing model is used is the highest, and this window image is labeled “left-handed pedestrian”. can do.

  Next, the score calculation part 22 calculates a score using the site | part model for every attitude | position of each part of a pedestrian with respect to the window image made into the pedestrian candidate by the whole body model.

  The score calculation unit 22 uses the image feature amount extracted for the local region corresponding to the region model in the window image in the local region of the window image for each posture for each region as in the case of the whole body. A score indicating the certainty of whether or not there is a pedestrian is calculated. And the score calculation part 22 calculates an integrated score using the maximum score obtained from the whole body model for every direction and the maximum score obtained from each part model. For the calculation of the integrated score, an SVM, a linear discriminant function, or the like can be used. If the integrated score obtained here is lower than a preset threshold value, it is determined as a non-pedestrian, and if not, the subsequent processing is continued with the window image as a pedestrian candidate. Here, the calculation result using the whole body model is used for calculating the integrated score, but only the calculation result using each part model may be used.

  At this time, as in the whole body model, the posture in the local region is determined based on the result obtained by using each part model for the local region of the window image that is a pedestrian candidate. In the example of FIG. 4 above, the part model is applied to the leg region, and the score when using the flip-up model is the highest, and this local region can be labeled as “bounce-up pattern”. .

  Note that the same feature quantity and discriminator may be used for identification of the whole body model and the part model, or different feature quantities and discriminators may be used.

  Further, the computer 16 extracts an optical flow (hereinafter referred to as a flow) indicating the movement of the feature points between the captured images from the plurality of acquired captured images for each feature point, and the flow extraction unit 26. The moving object flow extracting unit 28 for extracting the flow on the moving object from the flow extracted by the above, the moving object flow limiting unit 30 for limiting the flow on the moving object according to the position of the window image, and the window image It can represent with the structure containing the pedestrian identification part 32 which identifies whether it is an image showing the pedestrian of a detection target. The moving object flow restriction unit 30 is an example of a pedestrian flow extraction unit.

  The flow extraction unit 26 calculates the optical flow in the captured image using the captured image at the current time t and the captured image at the time t−1. The optical flow can be obtained by detecting the movement of each feature point from the change in the position of the feature point in each image and calculating the flow vector of each feature point. As a feature point detection method, a Harris operator, FAST, Tomasi-Kanade method, or the like can be used. As a feature point tracking method, a Lucas & Kanade algorithm or the like can be used.

  The moving object flow extraction unit 28 extracts the flow on the moving object calculated from the feature points corresponding to the moving object from the extracted flows. The moving object flow extraction unit 28 can extract a flow that does not follow the movement of the vehicle as a flow on the moving object from among the flows calculated from the feature points. In addition, the own vehicle motion may be acquired from a sensor attached to the vehicle, or may be estimated from the calculated flow by a method based on Structure from Motion.

  Next, the principle of this embodiment will be described.

  The flow is limited before the score is corrected using the flow on the moving object obtained by the moving object flow extraction unit 28. The flow obtained by the moving object flow extraction unit 28 is calculated from feature points extracted from objects at various distances, and the flow on the stationary object is erroneously extracted as the flow on the moving object. There may be. Therefore, stable pedestrian detection can be performed by correcting a score, which will be described later, using only a flow related to a pedestrian in the window image. Specifically, the flow used for score correction is limited from among the flows on the moving object obtained from the window image of the target pedestrian candidate. As shown in FIG. 5, the size of the flow obtained from the pedestrian varies depending on the position and direction (traveling direction) of the pedestrian. The flow on the moving object to be used is limited based on the corresponding flow size range. For example, when the position of the window image (the position in the height direction) corresponds to the position of a pedestrian candidate 50 m ahead, if a flow having a size of 10 pixels exists in the window image, the above-described FIG. Obviously, it can be regarded as a flow relating to an object other than a pedestrian, so that it can be prevented from being used for score correction described below. Note that FIG. 5 shows that the lateral position of the pedestrian is 2.5 m away from the lateral position of the host vehicle, the speed of the host vehicle is 30 km / h, the resolution of the camera is SXGA, and the horizontal angle of view. The flow size corresponding to the distance to the pedestrian in the case of 55 degrees and 10 frames / s is shown. Moreover, since the magnitude | size of the flow obtained from a pedestrian changes with vehicle speed and a camera parameter, you may make it limit a flow also using these.

  Further, the range of the size of the flow to be limited includes the distance from the position (position in the height direction) in the captured image of the window image to the pedestrian candidate, the camera parameters, and the assumed pedestrian height. It is possible to obtain based on

  Therefore, in the present embodiment, for the window image in which the integrated score calculated by the score calculation unit 22 by the moving object flow restriction unit 30 is equal to or greater than the threshold value, the lower end position (in the height direction) of the window image on the captured image. Based on the range of the size of the flow predetermined according to the position), the flow on the moving object in the window image is limited to the flow within the range. In addition, about each lower end position (position of a height direction) on the picked-up image of a window image, the range of the magnitude | size of the flow on a pedestrian is calculated from the distance D (refer FIG. 6, FIG. 7) according to the said lower end position. Find it in advance. The distance according to the lower end position of the window image is determined by the camera parameters (camera mounting height, camera depression angle, camera vertical angle of view) and parameters according to the assumed road plane (image height, y coordinate on the image). It is possible to obtain based on That is, as shown in FIG. 8, from the y coordinate of the pedestrian's foot position (bottom edge position), using the camera mounting height, camera depression angle, camera vertical angle of view, etc., according to the following equation (1) The distance D is calculated.

However, θ _aod is the camera depression angle, θ _vert is the camera vertical field angle, and H _camera is the camera mounting height [m]. H _image is the image height [pixel], D is the distance [m], and y is the y coordinate [pixel] on the image.

  As will be described below, the pedestrian identification unit 32 corrects the integrated score for each window image using the presence or absence of a flow in the window image, and performs pedestrian identification processing.

  Since the moving object flow restriction unit 30 restricts the flow according to the position of the window image of the target pedestrian candidate, in the correction of the integrated score, when there is a limited flow for the window image, Considering that the possibility of pedestrians will be higher, correction is made to increase the value of the integrated score.

  For example, if the number of flows limited according to the window image is equal to or greater than a preset number, the integrated score may be uniformly increased, or the integrated score may be increased according to the number of flows. May be.

  The pedestrian identification unit 32 determines that the window image is an image representing a pedestrian if the corrected integrated score is greater than or equal to the threshold value, and if the corrected integrated score is less than the threshold value, the window image is a pedestrian. It is determined that the image does not represent. Moreover, the pedestrian identification part 32 controls the display apparatus 18 so that an identification result may be superimposed and displayed on a captured image.

  Even if the pedestrian is stationary, if the integrated score calculated by the score calculation unit 22 is large, it is determined that the window image representing the pedestrian is an image representing the pedestrian.

  Next, a pedestrian detection processing routine executed by the computer 16 of the pedestrian detection apparatus 10 according to the first embodiment will be described with reference to FIG.

  In step 100, the whole body model for each orientation and the part model for each posture are read from the identification model storage unit 24. In step 102, a captured image at time t-1 and a captured image at time t, which are continuously captured by the imaging device 12, are acquired. Next, in step 104, from the captured images at time t-1, t, Calculate the flow.

  In step 106, a search window is set as a captured image for the captured image at time t, and a window image x is extracted from the captured image using the set search window.

  Next, in step 108, the image feature amount is extracted from the window image x extracted in step 106, and the image feature amount is extracted from the local region corresponding to the part model of each part.

  In step 110, a score is calculated based on the orientation-specific whole body model and the image feature amount extracted in step 108.

  Step 110 is realized by the processing routine shown in FIG.

  In step 150, N orientation-specific whole body models are set. In step 152, the variable K for identifying the whole body model is set to an initial value of 1. In step 154, the score is calculated based on the Kth orientation-specific whole body model and the image feature amount extracted in step 108. Is calculated. In step 156, the score calculated in step 154 is recorded in a memory (not shown).

  In step 158, the variable K is incremented by 1. In step 160, it is determined whether or not the variable K is equal to or smaller than a constant N indicating the number of orientation-specific whole body models. If the variable K is less than or equal to the constant N, the process returns to step 154. On the other hand, if the variable K is greater than the constant N, it is determined that the scores have been calculated for all the whole body models by orientation, and the processing routine is executed. finish.

  Then, in step 112 of the pedestrian detection processing routine of FIG. 9, it is determined whether or not the window image x is a non-pedestrian image based on the score for each direction calculated in step 154. When all the scores for each direction are less than the threshold, it is determined that the window image x is a non-pedestrian image, and the process proceeds to step 130 described later. On the other hand, if at least one of the scores for each direction is equal to or greater than the threshold value, it is determined that the window image x is not a non-pedestrian image, and the process proceeds to step 114. Also, the direction in which the score is maximized is determined as the direction of the pedestrian candidate.

  In step 114, a score is calculated based on the part model for each posture and the image feature amount of the local area extracted in step 108.

  Step 114 is realized by the processing routine shown in FIG.

  In step 170, HM position-specific part models for each of the M parts are set. In step 172, a variable H for identifying a part is set to 1 as an initial value, and in step 174, a variable K for identifying a posture of the part model is set to 1 as an initial value.

  In step 176, a score is calculated based on the Kth posture-specific part model of the Hth part and the image feature amount of the local region corresponding to the Hth part extracted in step 108 above. . In step 178, the score calculated in step 176 is recorded in a memory (not shown).

  In step 180, the variable K is incremented by 1. In step 182, it is determined whether or not the variable K is equal to or less than a constant NM indicating the number of posture-specific part models of the H-th part. If the variable K is less than or equal to the constant NM, the process returns to step 176. On the other hand, if the variable K is greater than the constant NM, it is determined that the scores have been calculated for all the posture-specific part models of the Hth part. Then, the process proceeds to Step 184.

  In step 184, the variable H is incremented by 1. In step 186, it is determined whether or not the variable H is equal to or smaller than a constant M indicating the number of parts. If the variable H is less than or equal to the constant M, the process returns to step 174. On the other hand, if the variable H is greater than the constant M, it is determined that the scores have been calculated for all the parts, and the processing routine ends.

  Then, in step 116 of the pedestrian detection processing routine of FIG. 9, the maximum score of the score for each direction calculated in step 154 and the maximum score of the score for each posture for each part calculated in step 176 are calculated. To calculate an integrated score.

  In the next step 118, based on the integrated score calculated in step 116, it is determined whether or not the window image x is a non-pedestrian image. If the integrated score is less than the threshold value, it is determined that the window image x is a non-pedestrian image, and the process proceeds to step 130 described later. On the other hand, if the integrated score is greater than or equal to the threshold value, it is determined that the window image x is not a non-pedestrian image, and the process proceeds to step 120. In addition, for each part, the posture with the maximum score is determined as the posture of the part of the pedestrian candidate.

  In step 120, the flow in the window image x in the flow calculated in step 104 is limited to the flow within the range of the flow size determined according to the position of the window image x.

  In the next step 122, the integrated score calculated in step 116 is corrected according to the number of flows limited in step 120. In step 124, it is determined whether or not the window image x is a pedestrian image based on the integrated score corrected in step 122.

  In the next step 126, it is determined whether or not the image is identified as a pedestrian image in step 124. If it is identified that the image is not a pedestrian image, the process proceeds to step 130 described later. If it is identified as an image, the window image x is recorded as a pedestrian area in step 128 and the process proceeds to step 130.

  In step 130, it is determined whether or not the search is completed by scanning the search window for the entire captured image at time t acquired in step 102. If not completed, the process returns to step 106, the window image is extracted from the position where the position of the search window has been moved by a predetermined search step, and the processing of step 108 to step 128 is repeated. If the search for the entire image in the search window of the current size is completed, the process returns to step 106 in the same manner, the size of the search window is changed, and the processing from step 106 to step 118 is repeated. When the search in all size search windows is completed for the entire captured image, the process proceeds to step 132.

  In step 132, as an output of the detection result, based on the pedestrian area recorded in step 128, the display device 18 is displayed so that the detected pedestrian is surrounded and displayed on the captured image. Control and end the process.

  As described above, according to the pedestrian detection apparatus 10 of the first embodiment, a flow within a predetermined flow size range corresponding to the position of the window image is moved within the window image. By extracting from the flow on the object and identifying whether the window image is an image representing a pedestrian based on the flow extraction result and the score calculated using the identification model, Regardless, pedestrians can be detected stably. In addition, by determining the consistency between the window image determined to be a pedestrian candidate based on the image pattern and the flow information obtained from the image, it is possible to reduce undetected and false detections and enable stable pedestrian detection. Become.

  Moreover, by calculating a score using a whole body model composed of a plurality of directions and a part model for each part posture as an identification model, and identifying whether the window image is an image representing a pedestrian, It can be detected stably.

  Next, a second embodiment will be described. In addition, since the pedestrian detection apparatus which concerns on 2nd Embodiment becomes the structure similar to 1st Embodiment, it attaches | subjects the same code | symbol and abbreviate | omits description.

  In the second embodiment, the flow on the moving object in the window image is limited in consideration of the lateral position of the pedestrian assumed in the window image of the pedestrian candidate. The form is different.

  FIG. 12 shows the relationship between the distance and the flow size when a pedestrian is present in a lateral position (5 m) different from the example of FIG. As shown in FIG. 5 and FIG. 12, the size of the flow differs depending on the lateral position of the pedestrian. That is, a larger flow can be obtained from the image as the lateral position from the camera center becomes farther.

  Therefore, in the present embodiment, the moving object flow restriction unit 30 determines the pedestrian candidate's position from the position of the window image on the captured image for the window image whose integrated score calculated by the score calculation unit 22 is greater than or equal to the threshold value. The horizontal position is calculated, and based on the range of the flow size predetermined according to the height position on the captured image of the window image and the calculated horizontal position of the pedestrian candidate, The flow on the moving object is limited to the flow in the range. For each position (position in the height direction) on the captured image of the window image, the range of the flow size on the pedestrian from each combination of the distance corresponding to the position and each value of the horizontal position. Can be obtained in advance.

  In addition, about the other structure and effect | action of a pedestrian detection apparatus which concern on 2nd Embodiment, since it is the same as that of 1st Embodiment, description is abbreviate | omitted.

  As described above, the flow within the predetermined flow size range corresponding to the position in the height direction of the window image and the lateral position of the pedestrian candidate assumed from the window image is represented as a moving object in the window image. Based on the flow extraction result and the score calculated using the identification model, whether the window image represents a pedestrian or not is identified based on the flow extraction result. Therefore, a pedestrian can be detected stably.

  Next, a third embodiment will be described. In addition, since the pedestrian detection apparatus which concerns on 3rd Embodiment becomes a structure similar to 1st Embodiment, it attaches | subjects the same code | symbol and abbreviate | omits description.

  The third embodiment is different from the first embodiment in that the flow on the moving object in the window image is limited in consideration of the orientation of the pedestrian obtained based on the score. ing.

  In the present embodiment, pedestrian orientation information obtained by the score calculation unit 22 is used. As shown in FIG. 5 described above, the size of the calculated flow varies depending on the direction of the pedestrian (sideways, forward, and backward).

  Therefore, in the present embodiment, the moving object flow restriction unit 30 uses the orientation of the pedestrian obtained by the score calculation unit 22 and the window image in which the integrated score calculated by the score calculation unit 22 is equal to or greater than the threshold, Based on a predetermined range of the flow size according to the height position on the captured image of the window image, the flow on the moving object in the window image is limited to the flow in the range. For each position (position in the height direction) on the captured image of the window image, the range of the flow size on the pedestrian is determined from each combination of the distance according to the position and the direction of the pedestrian. Find it in advance.

  In addition, about the other structure and effect | action of the pedestrian detection apparatus which concern on 3rd Embodiment, since it is the same as that of 1st Embodiment, description is abbreviate | omitted.

  Thus, the flow within the range of the predetermined flow size corresponding to the direction of the pedestrian candidate based on the position in the height direction of the window image and the score is determined from the flow on the moving object in the window image. The pedestrian is extracted regardless of the movement of the pedestrian by identifying whether the window image is an image representing the pedestrian based on the extraction result of the flow and the score calculated using the identification model. Can be detected stably. Moreover, since the advancing direction can be estimated if the target pedestrian is moving by using the direction of the pedestrian, the flow used for score correction can be further limited.

  Note that, similarly to the second embodiment described above, the range of the flow size may be limited in consideration of the lateral position obtained for the window image.

  Next, a fourth embodiment will be described. In addition, since the pedestrian detection apparatus which concerns on 3rd Embodiment becomes a structure similar to 1st Embodiment, it attaches | subjects the same code | symbol and abbreviate | omits description.

  The fourth embodiment is different from the first embodiment in that the score is corrected using the motion information obtained in advance for the posture related to the part obtained based on the score.

  In the identification model storage unit 24 of the pedestrian detection device according to the fourth embodiment, as shown in FIGS. 13 (A) to (C), the general pattern recognition is performed on the part model for each posture of each part. In addition to the model to be used, information on whether or not the posture of the part model has a movement is held. For example, in the case of the leg model 1 shown in FIG. 13A, the information that the probability that the pedestrian in this posture pattern is moving is 0.9. This movement information can be obtained from a large amount of learning data. Note that when calculating the score of a window image, only the model used for pattern recognition is used without using motion information.

  Here, the principle of the present embodiment will be described.

  Flow (motion) information is effective when detecting a moving pedestrian. Among the leg part models shown in FIG. 3, the part models shown in FIGS. 3A and 3B are characterized by a high possibility that a pedestrian is moving. 3D has a feature that it is difficult to determine whether it is stationary or moving. If there is a high possibility that the body is moving in the posture recognized by a certain part obtained from the image pattern, and if the flow is surely detected, the certainty (score) that the object is a pedestrian is increased. Thus, undetected pedestrians can be reduced.

  In addition, when there is a low possibility of moving in a posture recognized at a certain part and no flow is detected, the certainty of a stationary pedestrian can be increased. On the contrary, if the flow is detected despite the low possibility of moving in the posture recognized at a certain part, the false detection can be reduced by lowering the certainty of being a pedestrian.

  Thus, stable detection can be performed by using the consistency between the motion information added to the posture obtained from the image pattern (appearance) information and the flow information obtained from the image.

  Therefore, in the present embodiment, as described below, the pedestrian identification unit 32 uses the consistency between the posture obtained for each part and the presence / absence of the flow for each window image to calculate the integrated score. Correction is performed and pedestrian identification processing is performed.

  In the score calculation unit 22, the posture of each part is detected by the position-specific part model of each part. For example, consider a case where the leg region is detected as the leg model 2 (bounce up) shown in FIG. In addition to the model used for pattern identification, the leg model 2 holds motion information, and the motion information of the leg model 2 has a probability that the pedestrian is moving in this posture, which is 0.95. Have information that is very likely. At this time, if there is no flow in the window image, the pedestrian candidate is not moving and inconsistency with the motion information of the leg model 2 occurs. On the other hand, if there is a flow in the window image, the pedestrian candidate is moving and matches the motion information of the leg model 2. Therefore, the score can be corrected from the motion information and the flow information according to the following equation (2).

Here, S ′ is the corrected integrated score, and S is the integrated score obtained by the score calculation unit 22. K is the number of parts, and Pm _i is motion information (probability of motion) held by the model corresponding to the posture of the part i obtained by the score calculation unit 22. α (> 0) is a correction parameter. If the value of α is large, the amount of correction is large, and if it is small, the amount of correction is small. N _f is the number of flows included in the window image limited by the moving object flow limiting unit 30, and th is a preset threshold value for the number of flows.

According to the above equation (1), when the number of flows is small (N _f ≦ th), if the probability Pm of motion with respect to the posture obtained for a certain part is high (0.5 or more), the score is small. When the number of flows is large (N _f ≧ th), if the probability Pm with motion is high (0.5 or more), the score is corrected so as to increase.

  In addition, about the other structure and effect | action of a pedestrian detection apparatus which concerns on 4th Embodiment, since it is the same as that of 1st Embodiment, description is abbreviate | omitted.

  As described above, according to the pedestrian detection apparatus according to the fourth embodiment, information on movement is added in advance to the position-specific part model of each part, and each part obtained from the pedestrian detection based on the image pattern. By determining the consistency between the posture and the flow information obtained from the image, stable pedestrian detection is possible.

  Further, since detection is performed using a plurality of pedestrian models learned for each posture of each part, in addition to the position and size of the pedestrian, pedestrian posture information can be known. On the other hand, it is possible to infer from the posture of each part of the pedestrian at a certain time whether the possibility that the pedestrian is moving is high or low. Therefore, by using the pedestrian's posture information obtained from the image pattern and the flow information obtained from the entire image, the consistency between the image pattern recognition result and the flow information is judged, and the pedestrian detection performance is improved. be able to.

  As in the second and third embodiments, the range of the flow size is limited by further considering the horizontal position obtained for the window image or the orientation of the pedestrian candidate. You may do it.

  Next, a fifth embodiment will be described. In addition, since the pedestrian detection apparatus which concerns on 5th Embodiment becomes a structure similar to 1st Embodiment, it attaches | subjects the same code | symbol and abbreviate | omits description.

  The fifth embodiment is different from the first embodiment in that the distance to the pedestrian candidate is calculated from the window image size.

  As shown in FIGS. 6 and 7, the distance to the pedestrian candidate can be calculated not only from the position of the window image in the captured image but also from the window image size. Therefore, in the present embodiment, for a window image in which the integrated score calculated by the score calculation unit 22 by the moving object flow restriction unit 30 is equal to or greater than a threshold, the size of the window image on the captured image (the size in the height direction) ), The flow on the moving object in the window image is limited to the flow in the range based on the range of the flow size determined in advance. For each size (size in the height direction) on the captured image of the window image, the range of the flow size on the pedestrian is determined in advance from the distance D (see FIGS. 6 and 7) corresponding to the size. Find it. The distance to the pedestrian candidate according to the size of the window image can be obtained by assuming the height of the pedestrian (for example, the average height of a Japanese person is about 170 cm). That is, the distance D is calculated according to the following equation (3).

Here, W _height is the height [pixel] of the window image, and Ped _height is the assumed pedestrian height [m].

  In addition, about the other structure and effect | action of a pedestrian detection apparatus which concerns on 5th Embodiment, since it is the same as that of 1st Embodiment, description is abbreviate | omitted.

  In the first to fifth embodiments, the case where the distance and the lateral position of the pedestrian candidate are calculated from the position and size of the target window image has been described as an example. However, the present invention is not limited to this. It is not a thing. For example, when a device capable of measuring a distance, such as a stereo camera or a laser radar, can be used, the distance and lateral position of the pedestrian candidate may be obtained using information obtained from them.

  In the first to fifth embodiments described above, the case of performing the process of removing the non-pedestrian window image by the score threshold process has been described as an example. You may abbreviate | omit the process which removes a pedestrian's window image.

DESCRIPTION OF SYMBOLS 10 Pedestrian detection apparatus 12 Imaging device 16 Computer 18 Display apparatus 19 Image acquisition part 20 Window image extraction part 21 Feature-value extraction part 22 Score calculation part 24 Identification model memory | storage part 26 Flow extraction part 28 Moving object flow extraction part 30 Moving object flow Limiting part 32 Pedestrian identification part

Claims (10)

Flow extraction means for extracting, from each of a plurality of time-series captured images obtained by imaging the periphery of the device mounted on the moving body, a flow indicating the movement of the corresponding feature points between the captured images for each feature point;
A moving object flow extracting means for extracting a flow on the moving object from the flow extracted by the flow extracting means;
Window image extraction means for extracting a window image from the captured image;
Score calculating means for calculating a score indicating the pedestrian-likeness based on an identification model for identifying a pedestrian learned in advance and the window image extracted by the window image extracting means;
The flow on the moving object in the window image extracted by the moving object flow extraction means based on a predetermined flow size range corresponding to the distance to the pedestrian candidate on the captured image. Pedestrian flow extraction means for extracting a flow having a size within the range,
Identification means for identifying whether the window image is an image representing the pedestrian based on the extraction result by the pedestrian flow extraction means and the score calculated by the score calculation means;
A pedestrian detection device.   The pedestrian detection device according to claim 1, wherein the flow size range is determined in advance such that the flow size decreases as the distance to the pedestrian candidate on the captured image increases.   The range of the size of the flow is such that the larger the lateral position of the pedestrian candidate obtained from the position of the window image on the captured image is, the larger the size of the flow is. The pedestrian detection device according to claim 1 or 2, which is previously determined.   The identification unit corrects the score calculated by the score calculation unit according to the number of the flows extracted by the pedestrian flow extraction unit, and based on the corrected score, the window image The pedestrian detection device according to any one of claims 1 to 3, wherein whether or not an image representing the pedestrian is identified. The score calculation means, based on the identification model learned in advance for each pedestrian orientation and the window image extracted by the window image extraction means, shows a pedestrian-like score for each orientation. To calculate
The pedestrian flow extraction means is based on a predetermined flow size range corresponding to the distance to the pedestrian candidate on the captured image and the direction in which the score is maximized. The pedestrian detection device according to any one of claims 1 to 4, wherein a flow having a size within the range is extracted from a flow on the moving object in a window image. The score calculation means is a score indicating the pedestrian-likeness for each posture based on the identification model learned in advance for each posture of the pedestrian and the window image extracted by the window image extraction means. To calculate
According to the consistency between the presence or absence of movement of a pedestrian predetermined for the posture with the maximum score and the number of flows extracted by the pedestrian flow extraction means, the identification means The score calculated by the score calculation means is corrected, and based on the corrected score, it is identified whether the window image is an image representing the pedestrian or not. The pedestrian detection device described.   The pedestrian flow extraction unit is configured to generate the moving object flow based on a predetermined flow size range corresponding to a distance from the window image position on the captured image to the pedestrian candidate. The pedestrian detection device according to any one of claims 1 to 6, wherein a flow having a size within the range is extracted from a flow on the moving object in the window image extracted by an extraction unit.   The pedestrian flow extraction means is configured to generate the moving object flow based on a predetermined flow size range corresponding to the distance to the pedestrian candidate obtained from the size of the window image on the captured image. The pedestrian detection device according to any one of claims 1 to 6, wherein a flow having a size within the range is extracted from a flow on the moving object in the window image extracted by an extraction unit.   The said moving object flow extraction means extracts the flow on the said moving object from the flow extracted by the said flow extraction means based on the exercise | movement of the said mobile body. Pedestrian detection device. Computer
A flow extraction unit that extracts, from each of a plurality of time-series captured images obtained by imaging the periphery of the device mounted on the moving body, a flow indicating the movement of the corresponding feature point between the captured images for each feature point;
A moving object flow extracting means for extracting a flow on the moving object from the flow extracted by the flow extracting means;
Window image extraction means for extracting a window image from the captured image;
Score calculating means for calculating a score indicating the pedestrian-likeness based on an identification model for identifying a pedestrian learned in advance and the window image extracted by the window image extracting means;
The flow on the moving object in the window image extracted by the moving object flow extraction means based on a predetermined flow size range corresponding to the distance to the pedestrian candidate on the captured image. Pedestrian flow extraction means for extracting a flow having a size within the range, and
A program for functioning as an identification unit for identifying whether the window image is an image representing the pedestrian based on an extraction result by the pedestrian flow extraction unit and the score calculated by the score calculation unit .