JP2005311691A - Apparatus and method for detecting object - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To determine with high accuracy whether or not an object in an image obtained by photographing an object imaging area is really an object for detection. <P>SOLUTION: From a stereo image obtained by photographing the object imaging area, a candidate area is obtained which is predicted as a solid body and has a calculated identification value of no less than a predetermined value. The obtained candidate area is collated with past candidate area series a0-a5 and b0-b4 stored in a storage. On the basis of the collation, the storage is updated so as to add a candidate area (a0-a6 and b0-b5) in case the above corresponds, or to generate a new series (c0) in case there is no corresponding series. A mean of the identification values is obtained for each series a0-a6, b0-b4 and c0. For each corresponding time, the obtained mean value of the identification values is compared with a plurality of thresholds having a smaller value as a retroactive time becomes longer, and thus, it is determined whether or not the series of the candidate area is the detection object. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an object detection apparatus and method, and in particular, determines whether or not an object is truly a detection target in an image using an image obtained by capturing an imaged region in time series. The present invention relates to an object detection apparatus and method for detection.

  Realization of automobile safety systems such as a pedestrian information presentation system, warning system, and automatic brake system is expected to prevent accidents and reduce damage to pedestrians who are vulnerable to traffic. Development of a pedestrian detection sensor is necessary to realize these systems.

  Many conventional techniques for detecting people such as pedestrians using images are used for monitoring and security purposes to detect intruders and objects, and the background subtraction method, time subtraction method, and a combination of these methods are mainly used. .

  First, in the background subtraction method, a background image without an intruding object is captured and stored in advance, and a region having a brightness different from that of the background is detected as an intruding object based on a difference between the background image and a captured image at the time of monitoring. . Next, in the time difference method, a brightness change region is detected as a moving object, that is, an intruding object, based on a difference between two captured images at a predetermined time interval. These methods are based on a static background image captured by a fixed camera, and are difficult to apply to images captured by a moving camera such as an in-vehicle camera.

  There are also some research examples of pedestrian detection methods applicable to in-vehicle camera images. These methods detect pedestrians mainly by the following three-step process.

  First, in the first stage, candidate areas are extracted by a relatively fast and simple process. Here, the candidate area is a small area in an image where a pedestrian may exist, and includes a small area where no pedestrian exists.

  Non-Patent Document 1 extracts a three-dimensional object on a road as a candidate area. When each point (parallax point) of the parallax image generated from the stereo camera image is projected onto the real space, the property of increasing the spatial density of the parallax points around the three-dimensional object is used. Non-Patent Literature 2 hierarchically models a pedestrian contour pattern in advance, and extracts candidate areas based on the degree of coincidence with the model. Non-Patent Document 3 extracts a candidate area using a property that a pedestrian area in a far-infrared image has a relatively large density value.

  In the next second stage, whether or not the candidate region extracted in the first stage is a pedestrian is provisionally determined using a discriminator. The generation of classifiers is mainly performed by machine learning. In Non-Patent Document 1 and Non-Patent Document 2, a neural network is used, and in Non-Patent Document 3, a support vector machine is used.

  It should be noted that the extraction accuracy of candidate areas can be improved by narrowing down candidate areas according to different criteria in the first stage and the second stage. Furthermore, the overall processing time can be reduced by limiting the number of candidate areas in the first stage process that is relatively fast.

  In the third stage, the provisional determination result in the second stage is corrected in consideration of temporal continuity, and the final determination is performed. Non-Patent Document 2 tracks using an αβ filter, and finally determines that it is a pedestrian when it is tracked a predetermined number of times, and finally determines that it is not a pedestrian when it is lost a predetermined number of times. In Non-Patent Document 3, the first stage and second stage processes are performed every five frames, and only the position prediction is performed in the frames in between. In the position prediction, the predicted position by the Kalman filter is corrected by the mean shift filter. That is, it is assumed that the provisional determination performed every 5 frames is correct, and the determination is continued as the final determination.

  Further, although not premised on application to a moving camera, several inventions have been proposed for final determination as to whether or not a detection target is made using a past frame as in the third stage processing. As an example, the following two will be described.

  First, Patent Document 1 extracts a partial region that satisfies a predetermined condition, that is, a candidate region, from a captured image by an extraction unit, associates partial regions of different captured images by an association unit, and performs one-dimensional pattern conversion unit. Extracted by generating a plurality of one-dimensional patterns based on partial areas, generating a two-dimensional pattern from a plurality of one-dimensional patterns of different captured images in a two-dimensional pattern conversion means, and comparing the two-dimensional pattern with a dictionary pattern It is determined whether the partial area is a specific object.

Second, Patent Document 2 extracts a moving object, that is, a candidate area by background difference method and dynamic binarization or template matching in the first processing process, and extracts a feature amount of the moving object in the second processing process. Then, the feature quantity sequence extracted in the third processing step is stored in association with the moving object, and whether or not the moving object is the specific moving object from the feature quantity sequence stored in the fourth processing process is determined. judge. As a determination method, there are a method in which the moving object trajectory continues for a predetermined time or more is used as the specific moving body, and a method in which the variation in position change of the moving body is within a predetermined range is used as the specific moving body. Has been described.
L. Zhao and CEThorpe, "Stereo and Neural Network-Based Pedestrian Detection," IEEE Transactions on ITS, Vol. 1 No. 3, pp 148-154, 2000. DMGavrila and J. Giebel, "Shape-based Pedestrian Detection and Tracking," Proc. IEEE Intelligent Vehicles Symposium, 2002. F.Xu and K. Fujimura, "Pedestrian Detection and Tracking with Night Vision," Proc. IEEE Intelligent Vehicles Symposium, 2002. Japanese Patent Laid-Open No. 10-091794 JP 2002-157599 A

  A pedestrian is a set of very diverse patterns (appearances) because the appearance in the image changes depending on the posture, pose, clothes, luggage, and the like. The various patterns change from moment to moment depending on the movement of the camera and the movement of the pedestrian. Therefore, the evaluation value used for the determination, that is, the identification value varies with time, and it is easy to make an erroneous determination if the final determination is made based only on the identification value in the frame.

  Non-Patent Document 3 has a problem in that it is easy to make an erroneous determination because a temporary determination result obtained by processing in one frame is continued from the next frame. Further, when erroneous determination is made, there are problems that erroneous detection continues or that an undetected object is not detected until the next frame in which determination is performed.

  The method of Patent Literature 1 performs determination by comparing a time-series feature amount with a time-series model to be detected. When a fixed camera is used, candidate regions can be extracted relatively stably, so that time-series feature values can be obtained correctly. When a moving camera is used, there is a problem in that a time-series feature amount cannot be obtained correctly because temporary extraction of a candidate area to be extracted or positional deviation is likely to occur. In addition, if a detection target having various appearances such as a pedestrian is handled as a time-series pattern, the diversity of the pattern is further increased and the identification accuracy may be lowered.

  The methods of Non-Patent Document 2 and Patent Document 2 finally determine that the object is a specific object when a predetermined number of frames are continuously tracked. That is, the final determination is performed based on the continuity of the provisional determination result by the processing within one frame. As described above, in the detection target whose appearance changes with time, the provisional determination within one frame is likely to be erroneous. If errors in temporary determination frequently occur, the continuity of the temporary determination results is not established, and there is a problem that correct final determination cannot be made. There is also a problem that the detection is delayed because it always takes a predetermined time before the final determination that it is a detection target.

  Patent Document 2 also proposes a method of determining based on variation in position change. This stores the amount of change in position between frames for a predetermined time, and determines that the object is a specific object when the variation is small. There is a problem that the final determination cannot be performed correctly when the relative motion of the camera and the detection target is acceleration or when there is a deviation in the candidate extraction position.

  The present invention has been made in view of the above-described facts, and an object detection apparatus and method capable of more accurately determining whether or not an object in an image obtained by imaging an imaged region is truly a detection target. The purpose is to provide.

  In order to achieve the above object, an object detection apparatus according to a first aspect of the present invention includes an imaging unit that outputs an image obtained by imaging a region to be imaged in time series, and time series by the imaging unit. Acquisition means for acquiring a region having an identification value indicating a degree of identification as a detection target based on a model of the detection target created in advance from each of the images output to the time series, and the time series Storage means for storing a series of areas which are one or more areas acquired by the acquisition means from each of the images output to each of the images, the areas acquired this time by the acquisition means, The collation means for collating whether or not the series of areas stored in the storage means corresponds, and as a result of the collation by the collation means, when the area acquired this time and the series of areas correspond, This time If the region acquired this time and the sequence of the region are associated with each other and the region acquired this time does not correspond to the sequence of the region, the region acquired this time is stored as a new sequence, Update means for updating storage means; a plurality of average values of identification values of areas calculated retroactively to each of a plurality of different predetermined times in the series of the areas stored and updated in the storage means; And determining means for determining whether or not the series of the region is a detection target by comparing a plurality of threshold values that become smaller as the length of the region becomes longer at each corresponding time.

  The object detection method according to the sixth aspect of the present invention outputs, in time series, images obtained by capturing an imaged region in time series, and creates in advance from each of the images output in time series. Based on the detected model of the detection target, an area having an identification value indicating a degree to be identified as the detection target is greater than or equal to a predetermined value, acquired from each of the images output in time series, and each It is verified whether a series of areas that are one or more associated areas matches the area acquired this time, and as a result of the matching, the area acquired this time corresponds to the series of areas In this case, the region acquired this time and the sequence of the region are associated with each other. If the region acquired this time and the sequence of the region do not correspond to each other, the region acquired this time is set as a new sequence. And the area acquired this time In the sequence of assigned areas, the average values of the identification values of the areas calculated retroactively to a plurality of different predetermined times and the threshold values that become smaller as the retroactive time becomes longer are compared at corresponding times. By doing so, it is determined whether or not the series of regions is a detection target. Since the operation and effect of the present invention are the same as those of the invention described in claim 1, the operation and effect of the invention described in claim 1 will be described below, and the operation and effect of the invention described in claim 8 will be described. Description is omitted.

  That is, the image pickup means according to the first aspect of the present invention outputs, in time series, images obtained by picking up the imaged region in time series. For example, a stereo camera can be used as the imaging means.

  An acquisition unit is an area in which an identification value indicating a degree of identification as a detection target is greater than or equal to a predetermined value based on a model of the detection target created in advance from each of the images output in time series by the imaging unit To get. As the identification value, for example, an output before binarization of the classifier generated by machine learning can be used.

  For each candidate area extracted by the extraction means, an extraction means for extracting a candidate area that may include a detection target from the images output in time series by the imaging means, An identification value calculation unit that calculates an identification value indicating a degree of identification as a detection target, and a candidate area in which the identification value calculated by the identification value calculation unit is equal to or greater than a predetermined value And selecting means for selecting.

  That is, the extraction unit extracts candidate areas that may include a detection target from the images output in time series by the imaging unit. For example, in the above example, a distance image is created based on the parallax of the left and right stereo images, and a point (parallax point) where the parallax is obtained is projected onto a map representing the real space. For example, when this map is created from an image obtained by placing an imaging means on a vehicle and imaging a road as an imaged area, the spatial density of parallax points increases around a three-dimensional object on the road. . Due to this property, a three-dimensional object on a road including a pedestrian is extracted as a candidate for detection by extracting a region having a high spatial density of parallax points on the map.

  The identification value calculation means calculates an identification value indicating the degree of identification as the detection target based on a detection target model created in advance for each candidate region extracted by the extraction means.

  The detection target model created in advance, for example, a pedestrian model, can be generated by machine learning using a pedestrian and many other image samples. Machine learning is performed as follows. First, the size of the sample image is normalized by interpolation processing and converted into a feature amount such as an edge. It is desirable that this feature amount expresses the difference between the pedestrian and other objects more clearly, and the identification accuracy can be improved by selecting an optimum feature amount based on the detection target and the properties of the captured image. Then, after the sample is converted into the feature amount, a discriminator, that is, a model, can be generated using a known method such as a neural network or a support vector machine. The model needs to be generated only once in advance.

  The processing procedure of the identification value calculation means after generating the model will be described. For each candidate area extracted by the extraction means, each candidate area is normalized to the size of the area and converted into a feature amount in the same manner as when generating the model, and is processed using the model generated in advance. As a result, an identification value indicating the degree of identification as a detection target is obtained. Template matching, which is a simple method that does not use machine learning, can also regard an output correlation value as an identification value.

  Then, the selection means selects a candidate area whose identification value calculated by the identification value calculation means is greater than or equal to a predetermined value as an area where the identification value is greater than or equal to the predetermined value. Here, the predetermined value is determined such that a candidate area that can be reliably determined not to be a pedestrian is rejected. By reducing the number of candidate areas, it is possible to prevent erroneous correspondence in the subsequent collating means. Further, the determination by the determination means at the subsequent stage can be performed more accurately. This is because when the identification value is temporarily reduced due to a deviation in the extraction position of the candidate area, the candidate area is rejected and is not reflected in the determination. When the imaging means is in a moving state, the candidate area is likely to be misaligned, and the identification value is likely to be erroneous due to a change in appearance. Therefore, this selection means works particularly effectively.

  As described above, the acquisition means (extraction means, identification value calculation means, and selection means) processes each of the images output in time series by the imaging means, so that even if the imaging means moves, it remains stationary. Even in this case, the detection target candidate can be extracted and the identification value can be output without being affected by the movement of the background. In the methods of Non-Patent Document 2 and Patent Document 2, the final determination is performed using the processing result in the past frame (time). This processing result is binary information indicating the presence / absence of a detection target. On the other hand, the information used in the final determination to be described later in the present invention is an identification degree indicating a degree of identification as a detection target, that is, continuous value information. For this reason, the present invention can make a final determination more accurate than the prior art.

  As described above, the example in which the acquisition unit is configured by the extraction unit, the identification value calculation unit, and the selection unit has been described. That is, a method has been described in which a candidate region is extracted from a captured image and a process of outputting an identification value based on a model is realized in two stages of an extraction unit and an identification value calculation unit. However, a configuration that is not clearly divided into two stages is possible. For example, all the small areas in the image can be input to the discriminator, and the small area whose output identification value is a predetermined value or more can be regarded as a candidate area. Further, it is possible to extract the final candidate region by sequentially limiting the candidates by multi-step processing.

  The storage means stores a series of areas that are one or more areas acquired by the acquisition means from each of the images output in time series and associated with each other.

  The collation means collates whether the area acquired this time by the acquisition means corresponds to the series of areas stored in the storage means, and the update means obtains the current acquisition as a result of the collation by the collation means. The region acquired this time corresponds to the sequence of the region, the region acquired this time and the sequence of the region are associated with each other, and the region acquired this time and the sequence of the region do not correspond Updates the storage means so as to store the area acquired this time as a new series.

  For example, the appearance position of the area in the current image or the probability distribution of the appearance position in the current image is predicted from the series of past areas stored in the storage unit, and the currently acquired area exists near the predicted appearance position or near this position. The result of the collation is determined by determining whether or not the region acquired this time is within a predetermined range including the position where the appearance probability is a predetermined value or more. When an area exists, the area is associated with a series of stored areas.

  As a method for obtaining the predicted position and the probability distribution of the predicted position, there are known methods such as an αβ filter and a Kalman filter.

  As another association method, there is also a method in which the latest area in a series of past areas is stored as a template, and association is performed based on a correlation value with the template. Further, the correlation can be performed by combining the correlation values between the predicted position and the template.

  Here, the series of areas to be collated is not limited to that associated with the area in the immediately preceding frame. A sequence in which there is no region associated with predetermined frames is also included. Therefore, even when a candidate area is temporarily lost, it is possible to make an appropriate determination using the identification values in the frames before and after the associated areas.

  If the region acquired this time corresponds to the series of the regions as a result of the collation by the collating unit, the updating unit adds information necessary for the collation regarding the region and the identification value to the series and stores them.

  Here, the information necessary for the collation is determined according to the collation method, and is the position information in the collation based on the predicted position described above, and created from the latest candidate area in the method based on the correlation with the template. It is a template.

  In addition, when the region acquired this time and the sequence of the region do not correspond as a result of the verification by the verification unit, the update unit stores the region acquired this time as a new sequence, Update the stored contents of the storage means. That is, if there is no series associated with the collating means, a new series is generated, and information necessary for collation and an identification value are similarly stored.

  The determination means stores a plurality of average values of the identification values of the areas calculated retroactively to each of a plurality of different predetermined times in the series of the areas stored and updated in the storage means, and decreases as the retroactive time increases. By comparing a plurality of threshold values for each corresponding time, it is determined whether or not the series of the region is a detection target.

  Here, the plurality of average values of the identification values calculated retroactively to a predetermined plurality of times are, for example, the average value of the identification values of the region for the past three images, the average value of the identification values for the past four images, and the past 5 It means the average value of the identification values of the image area.

  When the determination is made as described above, the number of images required for detection is small for an object with a large identification value, and the number of images required for detection is large for an object with a small identification value. As a result, an object that is difficult to make an error, that is, an object with a large identification value is detected immediately with a small number of images, and an object that is easy to make an error, that is, an object with a small identification value is accurately determined based on many images. The

  In addition, when performing object detection with a fixed camera, the background subtraction method and the time subtraction method can be applied, so that the candidate region can be detected relatively stably. On the other hand, when imaging is performed in a state where the imaging means is moving, it is difficult to distinguish the background from the foreground, so that the extracted region is likely to be temporarily lost or misaligned. However, even if a region is temporarily missing in the present invention, since the regions before and after the lack are matched in time series by the collating means, the identification value of the region that is not missing is used, so that the correct determination can be made. it can.

  As described above, the present invention corresponds to a plurality of average values of identification values of areas calculated retroactively to a plurality of different predetermined times in a series of areas, and a plurality of threshold values that decrease as the retroactive time increases. Since it is determined at each time whether or not the series of the region is a detection target, continuous information called the average value of the identification values is used, so that the determination accuracy can be improved. Although it may be considered that if the identification value is mistaken temporarily, the error may greatly affect the average value.However, in the present invention, since the acquisition unit acquires only the region where the identification value is equal to or larger than the predetermined value, a large error is caused. It can be rejected, eliminating this difficulty.

  By the way, the storage means further stores determination information by the determination means corresponding to the series of areas, and the determination means stores the determination information stored in the storage means corresponding to the series of areas. In consideration of the information, it may be determined whether the sequence of the area is a detection target.

  In this case, the determination means corresponds to a plurality of average values of the identification values and a plurality of threshold values that become smaller as the retroactive time becomes longer at a predetermined time close to the current time among the different predetermined times. The determination information stored in the storage unit may be further taken into consideration at a predetermined time far from the current time among the plurality of different predetermined times.

  Note that the determination information may be at least one of a determination result by the determination unit and the number of times that the region series is determined as a detection target.

  In this way, since the determination information by the determination means stored corresponding to the series of the regions is further taken into account, it is possible to prevent the final determination result from being frequently inverted. Specifically, there is an effect of making it difficult to lose sight of the detection target once detected.

  As described above, the present invention corresponds to a plurality of average values of area identification values calculated retroactively to a plurality of different predetermined times in a series of areas, and a plurality of threshold values that decrease as the retroactive time increases. Since it is determined whether or not the region series is a detection target, the determination accuracy can be improved.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  As shown in FIG. 1, the object detection apparatus according to the present embodiment is connected to the imaging unit 101 that outputs, in time series, images obtained by imaging a monitoring area in time series, and is output in time series. A candidate extraction unit 102 that processes the image for each frame and outputs the position and shape information of the detection target candidate region, and a discriminator in which parameters representing the detection target model calculated in advance by machine learning or the like are stored 104, an identification unit that is connected to the imaging unit 101, the candidate extraction unit 102, and the discriminator 104, and outputs a discrimination value using the parameters stored in the discriminator 104 for each candidate region designated by the candidate extraction unit 102 A value calculation unit 103.

  Further, the object detection apparatus according to the present embodiment is connected to the candidate extraction unit 102 and the identification value calculation unit 103, selects only candidate regions whose identification value is equal to or greater than a predetermined value, and identifies the selected candidate region The candidate selection unit 105 that outputs the value and position information, the candidate region of the current frame output by the candidate selection unit 105 and the sequence of candidate regions of the past frame stored in the storage unit 107, and their correspondence If the candidate area of the current frame corresponds to a sequence of past candidate areas, information on the candidate area of the current frame is added to that series, and if the candidate area of the current frame does not correspond to the series of past candidate areas, A matching unit 106 that controls the storage unit 107 to generate a new sequence and store information on the candidate region of the current frame, and an identification value for each sequence with reference to the sequence of candidate regions stored in the storage unit 107 Calculating unit 108 for calculating the average value of Comprises based on the average value of the identification value calculated by the calculation unit 108, a determination unit 109 whether or not the pedestrian for each sequence of the candidate region.

  Next, an object detection method according to the present embodiment will be described with reference to FIG.

  In step 201 of image input, a stereo image is input. In candidate extraction step 202, a distance image is generated from the stereo image, and a three-dimensional object is extracted as a candidate region. FIG. 3A shows the extracted candidate area as a dotted rectangle. In this example, in addition to the pedestrian 300, a vehicle 302, a sign 304, a guardrail 306, and a building 308 are extracted as three-dimensional objects.

  In step 203 for calculating the identification value, the identification value, that is, the similarity to the detection target is output for each candidate area. In the candidate selection step 204, only candidate regions whose identification value is greater than or equal to a predetermined value are selected. FIG. 3B shows an identification value calculated for each candidate area, and a candidate area having an identification value of −0.5 or more is selected. The selected candidate areas are A, B, and C indicated by solid rectangles.

  In step 205 of collation, the selected candidate area is collated with the stored series of past candidate areas to perform association. FIG. 4A shows the position of the stored series (relative position with respect to the camera) in real space, and two series of a0 to a5 and b0 to b4 are stored. The ellipse in the figure shows the range of appearance positions in the current frame predicted from the two sequences. In this example, candidate area A and candidate area B extracted in the current frame are associated with sequences ai and bi (i is an integer), respectively, and candidate C has no corresponding sequence.

  In the sequence update step 206, the stored sequence is updated based on the collation result. FIG. 4B shows the updated sequence. For the series ai and bi in which there is a corresponding candidate in the current frame, the index i is incremented by 1, and A and B are added as the latest candidates a0 and b0. Candidate C for which no corresponding sequence exists generates a new sequence c0.

  In the determination step 207, the average of the respective identification values is obtained for the series ai, bi, and ci, and it is determined whether or not the person is a pedestrian based on a determination criterion described later. FIG. 3C shows that the candidate B is determined to be a pedestrian.

  Next, each step 201-207 is demonstrated in detail.

  First, step 201 (operation of the imaging unit 101) will be described. The imaging unit 101 is composed of a pair of cameras (stereo cameras) that capture the same imaged area, and each camera images the same imaged area in time series from positions separated by a predetermined interval. Output in time series.

  In step 202 (operation of candidate extraction unit 102), candidate areas are extracted. Specifically, as shown in FIG. 8, in step 150, a distance image is generated. That is, the input two stereo images are associated for each pixel, and a distance image is generated based on a shift amount (parallax) between the corresponding pixels.

  A pixel with which correspondence between two stereo images is obtained, that is, a pixel with a distance to the target is called a parallax point. In step 152, the parallax point is projected onto a map representing the real space. Assume that a road scene is captured by an in-vehicle stereo camera and an image as shown in FIG. 5A is obtained. FIG. 5B is a map of the real space where the parallax points of the distance image created from this image are projected. As shown in FIG. 5B, the spatial density of the parallax points increases around the three-dimensional object on the road such as the pedestrian 300, the vehicle 302, the guard rail 304, and the sign 306. Therefore, the candidate extraction unit 102 extracts a solid object on the road as a pedestrian candidate by extracting an area where the spatial density of the parallax points on the map is large in step 154, and in step 156, the position and shape are extracted. The information of is output. The position information is a position on the image and a position on the real space. The former is used for specifying the position in the image by the identification value calculation unit, and the latter is used for the association by the matching unit 106. The shape information is information on the shape and size in the image. Since the shape of the candidate area can be limited to a rectangle having a predetermined aspect ratio in detection of a pedestrian, the shape information to be output is only a scale factor.

  In the present embodiment, the candidate extraction method for extracting a three-dimensional object from a stereo image is used, but other methods can also be used. For example, a far-infrared camera can be used as the moving image capturing means. In the image picked up by the far-infrared camera, the density value of the high-temperature object is large, so the candidate extraction means uses a simple binarization or dynamic binarization method as a candidate for high-temperature objects including pedestrians. Can be extracted. Further, other sensors such as a millimeter wave radar and a laser radar can be used as the candidate extraction means.

  Next, step 203 (operation of the identification value calculation unit 103) will be described.

  In the present embodiment, a human model as a detection target is generated in advance by machine learning and stored in the classifier 104, and the identification value calculation unit 103 calculates an identification value using this. FIG. 6 is a conceptual diagram showing a process of generating a human model by machine learning. As shown in FIG. 6, a large number of samples of pedestrians and non-pedestrians are prepared, and normalized so that these pixel sizes are suitable for identification.

  In the case of identifying a pedestrian as a detection target, the horizontal size is 10 to 40 pixels, and the vertical size is about 2 to 3 times the horizontal size. The sample whose size is normalized is converted into a feature amount. As the feature amount, for example, an image density value or an edge can be used. The feature amounts shown in FIG. 6 are edges in four directions, ie, the up and down direction, the left and right direction, and the two diagonal directions. Further, the resolution of the edge is reduced to reduce the number of pixels, which is known as a feature quantity called a four-direction plane feature. When the feature quantity is expressed as a multidimensional vector, one sample is one point in the multidimensional space, that is, the feature quantity space. In FIG. 6, the multi-dimensional feature amount space is schematically shown in two dimensions, with the pedestrian sample indicated by ● and the non-pedestrian sample indicated by +. Machine learning is equivalent to obtaining the boundary between the distribution of pedestrians and non-pedestrians in the feature space by learning the distribution of a large number of samples. In a support vector machine, which is one of machine learning methods, samples near the boundary are automatically extracted from a large number of samples. The feature vector of the extracted sample is called a support vector, and the identification boundary is represented by this support vector. The discriminator 104 stores a support vector and its multiplier, and this is a detection target (pedestrian) model generated by learning.

  FIG. 9 is a processing flowchart of step 203. The identification value calculation unit 103 normalizes the size in the candidate region in the image extracted by the candidate extraction unit 102 in step 160 of FIG. 9 in the same manner as the model generation, and calculates the feature vector x in step 162. To do.

  Here, M is the number of dimensions of the feature vector.

  In step 164, an identification value f is calculated. That is, the support vector stored in the discriminator 104 is set as its multiplier. Here, N is the number of support vectors. The identification value f is obtained by the following equation.

Here, t _i is a label of the sample, t _i = 1 if the support vector s _i is a pedestrian sample, and t _i = −1 if s _i is a non-pedestrian sample. H is a constant. K () is a kernel function. In the case of a Gaussian kernel, it is defined as

  In addition to the Gaussian kernel, a sigmoid kernel or a polynomial kernel can be used as a kernel function.

  In step 166, the identification value f is output.

  The identification value calculation unit 103 calculates an identification value at a plurality of positions around the position specified by the candidate extraction unit 102 and a plurality of scales close to the specified scale, and the maximum value of the plurality of calculated identification values The average value can also be output as the identification value of the candidate area. Thereby, even when the accuracy of the position and scale of the candidate area specified by the candidate extraction unit 102 is low, the identification value can be obtained correctly.

  Next, in step 204 (operation of the candidate selection unit 105), only candidate regions whose identification value is greater than or equal to a predetermined value are selected, and the identification value and collation unit 106 outputs necessary information. Here, the predetermined value is determined so that only candidate areas that can be reliably determined not to be pedestrians can be rejected, and the number of candidate areas can be reduced without rejecting candidates who are truly pedestrians. This candidate selection provides the following two effects. First, if the number of candidate areas is reduced, it is possible to prevent erroneous correspondence in the subsequent collating means. Second, since the identification value calculated in error is not reflected in the average value used for the subsequent determination, more accurate determination is possible. The reason why the discriminant value is calculated by apologizing is that even if the candidate region is truly a pedestrian, its position is extracted with a shift, or learning of the discriminator is not appropriate. The second effect can be obtained by providing means for converting the identification value using a characteristic conversion function as shown in FIG. 7 instead of the candidate selection unit 105. This is because the influence of the identification value temporarily reduced by mistake can be reduced by clipping the lower limit value of the identification value to Fmin.

  Next, step 205 (operation of the collation unit 106) will be described.

The storage unit 107 stores information necessary for collation and information necessary for determination for each series of candidate regions extracted in the past. The following is an example of the contents stored in one series.
(1) F0, F1, ... Fn
(2) J0, J1, ... Jn
(3) X, Y, Z
(4) L
(5) Internal parameter Here, (1) Fi (i = 0,..., N) is an identification value before i frames, and a value 0 indicating that there is no identification value when there is no corresponding candidate area Is remembered.

  (2) Ji (i = 0, .., n) is a determination result before i frame, and is 1 when determined as a pedestrian, and is stored as 0 when determined as a non-pedestrian. The

  (3) X, Y, and Z are position vectors immediately before the series.

  (4) L is the number of lost times.

  (5) is an internal parameter necessary for collation. When a Kalman filter described later is used, it is an internal state variable such as a velocity vector (Vx, Vy, Vz) or a parameter such as a covariance matrix thereof.

  In this step 205, the collation unit 106 collates the sequence stored in the storage unit 107 with the candidate area output from the candidate selection unit 105, associates them, and stores them based on the result of the association. The stored contents of the unit 107 are updated.

  First, the matching unit 106 determines the correspondence C (i, j) (i = 1,..., Nc) for all combinations of the candidate area output from the candidate selection unit 105 and the series stored in the storage unit 107. ; j = 1, ..., Ns) is calculated.

  Here, i is the number of the candidate area, and Nc is the number thereof. j is the number of the series and Ns is the number.

  After obtaining the degree of correspondence, association is performed based on the flowchart shown in FIG.

  In step 801 in FIG. 10, the variables are initialized. Rc (i) (i = 1,..., Nc) is a variable for storing the sequence number corresponding to the i-th candidate region, and is initialized to Rc (i) = 0 for all i. Rc (i) = 0 indicates that there is no sequence corresponding to the i-th candidate region. Rs (j) (j = 1,..., Ns) is a variable for storing the number of the candidate area corresponding to the jth series, and is initialized to Rs (j) = 0 for all j. Rs (j) = 0 indicates that there is no candidate region corresponding to the j-th sequence. Mc (i) (i = 1,.., Nc) is a variable for storing the degree of correspondence with the series having the maximum degree of correspondence with respect to the i-th candidate region, and Mc (i) = Initialize with T. T is a threshold value of the degree of correspondence for the candidate area and the series to correspond. In step 802, a variable i is initialized to 1.

  In the loop of Steps 803, 804, 805, 806, and 807, the sequence number Rc (i) having the maximum correspondence Mc (i) with respect to the i-th candidate region is determined. That is, in step 803, the variable j is initialized to 1, and in step 804, whether or not the correspondence C (i, j) between the i-th candidate region and the j-th sequence is greater than Mc (i). By determining, it is determined whether or not the degree of correspondence is greater than the threshold T and before the j-th sequence.

  If it is determined that C (i, j) is not greater than Mc (i), the process proceeds to step 806. If it is determined that C (i, j) is greater than Mc (i), in step 805, Rc (i) is set to j, i.e., the j-th sequence as the sequence corresponding to the i-th candidate region. J indicating the sequence of. Thereby, it can be understood that the sequence corresponding to the i-th candidate region is a sequence identified by j stored in Rc (i).

  In step 805, the correspondence degree C (i, j) between the i-th candidate region and the j-th sequence is stored in Mc (i). As a result, the degree of correspondence between the i-th candidate region and the sequence corresponding to the i-th candidate region is grasped by C (i, j) stored in Mc (i).

  In step 806, the variable j is incremented by 1. In step 807, it is determined whether the variable j is larger than the total number Ns of sequences. It can be determined whether or not it has been determined for the series.

  If it is not determined that the variable j is larger than the total number Ns of series, it is not determined for all the series which series corresponds most to the i-th candidate area, so that the process returns to step 804. (Steps 804 to 807) are executed. Since Mc (i) = T is initialized in step 801, if there is no correspondence C (i, j) (j = 1,..., Ns) greater than T, the initial value Rc (i) = 0 is not updated, that is, it is determined that there is no corresponding sequence.

  On the other hand, if it is determined that the variable j is greater than the total number Ns of series, it is determined for all the series which is the series most corresponding to the i-th candidate area, that is, the i-th candidate area. Since the sequence number Rc (i) having the maximum correspondence Mc (i) is determined, the process proceeds to step 808.

  In step 808, it is determined whether Rc (i) is greater than zero. If Rc (i) is not greater than 0, that is, 0, there is no sequence corresponding to the i-th candidate region, so the processing for the i-th candidate region is terminated and the process proceeds to step 814.

  If Rc (i) is greater than 0, there is a sequence corresponding to the i-th candidate region, so whether or not the Rc (i) -th sequence is already associated with another candidate region in step 809. If they are not associated with each other, variables are set in step 810 so that the Rc (i) -th sequence is associated with the i-th candidate region.

  If it is determined in step 809 that the Rc (i) -th sequence is already associated with the Rs (Rc (i))-th candidate region, the degree of correspondence is compared in step 811. If the i-th candidate region has a higher degree of correspondence with the Rc (i) -th sequence than the already associated Rs (Rc (i) -th) candidate region, Rs (Rc (i )) Set variables so that there is no sequence corresponding to the second candidate area. If the i-th candidate area is smaller than the Rs (Rc (i))-th candidate area already associated with the Rc (i) -th series, the i-th candidate area has a smaller degree of correspondence in step 813. Variables are set so that there is no series corresponding to the candidate area.

  In step 814, the variable i is incremented by 1. In step 815, it is determined whether or not the variable i is associated with all candidate areas by determining whether the variable i is larger than the total number Nc of candidate areas.

  If it is not determined that the variable i is larger than the total number Nc of candidate areas, there is a candidate area that has not been associated yet, so the process returns to step 803 and the above processing (step 803 to step 815) is executed. . When it is determined that the variable i is larger than the total number Nc of candidate areas, since the association with all candidate areas is completed, this process is terminated.

  Incidentally, the calculation of the correspondence level and the update of the information stored in the storage unit 107 can be performed using, for example, a Kalman filter. When the Kalman filter is used, information stored in the storage unit 107 is an internal parameter that represents a position vector in the previous frame and a state in the previous frame. The internal parameter is a covariance matrix of a position vector or an internal variable vector, system noise, or the like. The Kalman filter can predict the position vector and state in the current frame from the state of the previous frame. Since the covariance matrix of the position vector is predicted as the state of the current frame, the correspondence between the series and the candidate area can be obtained as, for example, the Mahalanobis distance between the predicted position vector and the position vector of the candidate area. However, since the Mahalanobis distance represents the degree of separation, the degree of correspondence is obtained by adding its reciprocal or a negative sign.

  As shown in FIG. 11, the information stored in the storage unit 107 is updated as follows. In step 902, a variable k for identifying a series is initialized to 0. In step 904, the variable k is incremented by 1. In step 906, it is determined whether or not there is a candidate area corresponding to the series k identified by the variable k. If there is no candidate area corresponding to the series k identified by the variable k, in step 908, The number of lost times corresponding to the series k is incremented by 1, and it is determined in step 910 whether the number of lost times is equal to or greater than a predetermined value. If the number of lost times is equal to or greater than a predetermined value, in step 912, the series k is deleted. If the number of lost times is less than the predetermined value, in step 914, the stored content is updated to the position vector and state predicted when the correspondence degree is calculated.

  If it is determined in step 906 that the corresponding candidate area exists, the position in the current frame is determined in step 916 using the position vector of the corresponding candidate area and the stored position vector and state in the previous frame. The vector and state are estimated, and in step 918, the stored content is updated to the estimated value.

  Except when the series is deleted (steps 914 and 918), the identification value Fi (i = 0,... N) and the determination result Ji (i = 0,... N) are also updated. However, J0, which is the determination result of the current frame, is updated after being determined in the subsequent stage.

  In step 920, it is determined whether or not the variable k is greater than or equal to the total number of series Nk. If the variable k is not greater than or equal to the total number of series Nk, the process returns to step 904 and the above processing (steps 904 to 920) is performed. If the variable k is greater than or equal to the total number Nk of sequences, it is determined in step 922 whether there is a candidate area for which there is no associated sequence. Is generated. At this time, Fi (i = 0,.., N) is initialized to 0, Ji (i = 0, .., n) is initialized to 0, and the number of lost times is initialized to zero. Further, the position vector stores the position vector of the candidate area, and the internal state is set to an appropriate initial state.

  Next, step 207 (operations of the calculation unit 108 and the determination unit 109) will be described in detail. The calculation unit 108 refers to the information stored in the storage unit 107 and calculates an evaluation value necessary to determine whether or not the person is a pedestrian. The determination unit 109 performs determination based on the evaluation value calculated by the calculation unit 108 and outputs the result. The evaluation values calculated by the calculation unit 108 are shown below.

  Si is a weighted average of identification values up to i frames before, and NJ is the number of times that it was determined to be a pedestrian between n frames up to 1 frame. The weight of the weighted average can be defined as follows:

In a frame (F _i = 0) in which no corresponding candidate area exists, the identification value and the weight are both 0, and thus are not reflected in the average value. Also, the weight can be defined as in the following equation.

  Here, r> 1.

  According to this definition, an identification value close to the current frame can be reflected to a larger average value, that is, an evaluation value.

  The determination unit 109 uses these evaluation values to make a determination based on a predetermined determination condition. Some examples of determination conditions when n = 6 are shown.

First example
A plurality of average values (evaluation values) S _{0 to} S ₆ of identification values of candidate areas calculated retroactively to a plurality of different times up to n = 6, and a plurality of threshold values T _{0 to} T that become smaller as the retrospective time becomes longer. ₆ is compared at each corresponding time.
(1) S0> T0
(2) S1> T1
(3) S2> T2
(4) S3> T3
(5) S4> T4
(6) S5> T5
(7) S6> T6
However, Ti> Ti + 1 (i = 0, ..., n-1) (The smaller the retroactive time, the smaller)
When one or more of the conditions (1) to (7) are satisfied, it is determined that the person is a pedestrian.

Second example
A plurality of average values (evaluation values) S _{0 to} S ₆ of identification values of candidate areas calculated retroactively to a plurality of different times up to n = 6, and a plurality of threshold values T _{0 to} T that become smaller as the retrospective time becomes longer. ₆ and a value determined by determination information (determination result (J1 + 1)) are compared for each corresponding time.
(1) S0> T0 / (J1 + 1)
(2) S1> T1 / (J1 + 1)
(3) S2> T2 / (J1 + 1)
(4) S3> T3 / (J1 + 1)
(5) S4> T4 / (J1 + 1)
(6) S5> T5 / (J1 + 1)
(7) S6> T6 / (J1 + 1)
However, Ti> Ti + 1 (i = 0, ..., n-1) (The smaller the retroactive time, the smaller)
A third example that determines that a person is a pedestrian when one or more of conditions (1) to (7) are met
A plurality of average values (evaluation values) S _{0 to} S ₆ of identification values of candidate areas calculated retroactively to a plurality of different times up to n = 6, and a plurality of threshold values T _{0 to} T that become smaller as the retrospective time becomes longer. ₆ and the value determined by the determination information (number of times determined to be a pedestrian (NJ + 6)) are compared at each corresponding time.
(1) S0> T0 × 6 / (NJ + 6)
(2) S1> T1 × 6 / (NJ + 6)
(3) S2> T2 × 6 / (NJ + 6)
(4) S3> T3 × 6 / (NJ + 6)
(5) S4> T4 × 6 / (NJ + 6)
(6) S5> T5 × 6 / (NJ + 6)
(7) S6> T6 × 6 / (NJ + 6)
However, Ti> Ti + 1 (i = 0, ..., n-1) (The smaller the retroactive time, the smaller)
A fourth example that determines that a person is a pedestrian when one or more of conditions (1) to (7) are met
At a predetermined time (n = 0 to 3) close to the current time among a plurality of different predetermined times up to n = 6, a plurality of average values S _{0 to} S _{6 of the} identification value become smaller as the retroactive time becomes longer. A plurality of threshold values T _{0 to} T ₆ are compared for each corresponding time, and at a predetermined time (n = 4 to 6) far from this time among a plurality of different predetermined times up to n = 6, determination information ( Further consider the number of times NJ) determined to be a pedestrian.
(1) S0> T0
(2) S1> T1
(3) S2> T2
(4) S3> T3
(5) S4> T4 and NJ ≧ 1
(6) S5> T5 and NJ ≧ 2
(7) S6> T6 and NJ ≧ 3
However, Ti> Ti + 1 (i = 0, ..., n-1)
If one or more of the conditions (1) to (7) are met, it is determined that the person is a pedestrian. Is determined, J0 = 0 and the contents of the storage unit 107 are updated.

  In the present embodiment described above, since the candidate extraction unit 102 and the identification value calculation unit 102 process the output image for each frame, the motion of the background regardless of whether the camera is moving or stationary. It is possible to extract a candidate area to be detected without being influenced by and to output the identification value. In the method of Non-Patent Document 2 and the like, the final determination is performed using the processing result in the past frame (time), and this processing result is binary information indicating the presence or absence of the detection target. On the other hand, in the present embodiment, the past information used in the final determination is continuous value information representing the degree of similarity with the pedestrian. For this reason, this embodiment can perform final determination more accurate than the prior art.

  In the present embodiment, candidate selection section 105 selects a candidate area whose identification value is greater than or equal to a predetermined value. However, a predetermined value is set so that candidate areas that can be determined to be non-pedestrians are rejected. Therefore, by reducing the number of candidate areas, it is possible to prevent erroneous correspondence in the subsequent collation unit 106 and to perform the determination in the determination unit 109 more accurately. This is because, when the identification value is accidentally temporarily reduced due to a deviation in the extraction position of the candidate area, the candidate area is rejected and is not reflected in the determination. When a pedestrian is detected by a moving camera, the position of the candidate area is likely to be misaligned, and the identification value is likely to be erroneous due to the temporal change in appearance. Therefore, this candidate selection works particularly effectively.

  Furthermore, in this embodiment, the sequence of candidate areas to be collated is not limited to that associated with the candidate area in the immediately preceding frame, as described with reference to FIG. A sequence in which there is no candidate area associated with a predetermined frame is also included. Therefore, even when the candidate area is temporarily lost, it is possible to make an appropriate determination using the identification values in the previous and subsequent frames associated with each other.

  In the present embodiment, since the determination is performed based on the time average of the identification value, the final determination result is not easily affected by the time variation of the identification value. In addition, an object with a large identification value requires fewer frames for detection, and an object with a small identification value requires more frames for detection. Therefore, an object that is less likely to be erroneously detected is immediately detected with a small number of frames, and an object that is likely to be erroneously determined is more accurately determined based on a large number of frames.

  When performing object detection with a fixed camera, the background subtraction method or the time subtraction method can be applied, so that the candidate region can be detected relatively stably. In the object detection by the moving camera, it is difficult to distinguish the background from the foreground, and therefore, the extracted candidate region is likely to be temporarily lost or misaligned. In addition, when the position of the candidate area is shifted, the identification value is also erroneously calculated. Furthermore, even when the position of the candidate area is correctly extracted, there is a problem that the identification value is likely to be erroneous when the pattern of appearance like a pedestrian is frequently used and fluctuates over time.

  As described above, it has been difficult in the past to accurately determine and detect whether or not an object whose appearance pattern is diverse and time-varying from an image captured by a moving camera is a detection target. In the embodiment, this is made possible.

 In the embodiment described above, the value output from the classifier generated by machine learning is used as the identification value. In addition, a value representing the likelihood of detection of a candidate area can be used as an identification value. For example, a representative sample to be detected may be used as a template, and a correlation value with the template may be used.

It is a block diagram of the object detection apparatus concerning this Embodiment. It is the flowchart which showed the object detection method concerning this Embodiment. It is explanatory drawing explaining selection of a candidate basin. It is explanatory drawing explaining the update of the series of a candidate area | region. It is explanatory drawing explaining extraction of a candidate area | region. It is explanatory drawing explaining the production | generation of a pedestrian model. It is a figure which shows the conversion function which converts an identification value. It is a flowchart which shows a candidate extraction process. It is a flowchart which shows an identification value calculation process. It is a flowchart which shows a collation process. 7 is a flowchart illustrating an update process of information stored in a storage unit 107.

Explanation of symbols

101 Imaging unit (imaging means)
102 Candidate extraction unit (acquisition means, extraction means)
103 Identification value calculation unit (acquisition means, identification value calculation means)
104 Classifier (model)
105 Candidate selection unit (acquisition means, selection means)
106 Verification unit (verification unit, update unit)
107 Storage unit (storage means)
108 Calculation unit (determination means)
109 Determination unit (determination means)

Claims (6)

Imaging means for outputting in time series images obtained by imaging the imaged region in time series;
Acquisition of an area where an identification value indicating a degree of identification as a detection target is greater than or equal to a predetermined value based on a model of the detection target created in advance from each of the images output in time series by the imaging unit Means,
Storage means for storing a series of areas that are one or more areas acquired by the acquisition means from each of the images output in time series and associated with each other;
Collation means for collating whether the area acquired this time by the acquisition means corresponds to the series of areas stored in the storage means;
As a result of the collation by the collation means, when the region acquired this time corresponds to the sequence of the region, the region acquired this time and the sequence of the region are associated with each other, and the region acquired this time And an update unit that updates the storage unit so as to store the area acquired this time as a new series,
A plurality of average values of the identification values of the areas calculated retroactively to each of a plurality of different predetermined times in the series of the areas stored and updated in the storage means, and a plurality of threshold values that become smaller as the retroactive time becomes longer , For each corresponding time, determination means for determining whether the series of the region is a detection target,
An object detection apparatus comprising: The storage means further stores information of determination by the determination means corresponding to the series of the areas,
The determination means further considers determination information stored in the storage means corresponding to the series of areas, and determines whether the series of areas is a detection target;
The object detection apparatus according to claim 1.   The determination means includes a plurality of average values of the identification values and a plurality of threshold values that become smaller as the retroactive time becomes longer at a predetermined time close to the current time among the different predetermined times. 3. The object detection apparatus according to claim 2, wherein the determination information stored in the storage unit is further considered at a predetermined time far from the current time among the plurality of different predetermined times.   4. The object detection apparatus according to claim 2, wherein the determination information is at least one of a result of determination by the determination unit and a number of times that the series of regions is determined as a detection target. The acquisition means includes
Extraction means for extracting candidate areas that may include a detection target from images output in time series by the imaging means;
For each candidate area extracted by the extraction means, an identification value calculation means for calculating an identification value indicating a degree of identification as the detection target based on a model of the detection target created in advance;
Selection means for selecting a candidate area whose identification value calculated by the identification value calculation means is greater than or equal to a predetermined value as an area where the identification value is greater than or equal to the predetermined value;
The object detection apparatus according to claim 1, further comprising: Output images obtained by imaging the imaged area in time series, in time series,
From each of the images output in time series, based on a detection target model created in advance, an area having an identification value indicating a degree of identification as the detection target is greater than or equal to a predetermined value,
Check whether or not a series of areas that are one or more areas previously acquired from each of the images output in time series correspond to the areas acquired this time correspond to each other,
As a result of the collation, when the region acquired this time corresponds to the sequence of the region, the region acquired this time and the sequence of the region are associated with each other, and the region acquired this time and the region If the series does not correspond, the area acquired this time is stored as a new series,
In the series of areas associated with the area acquired this time, a plurality of average values of the identification values of the areas calculated retroactively to each of a plurality of different predetermined times, and a plurality of threshold values that become smaller as the retroactive time becomes longer, To determine whether or not the series of the region is a detection target, by comparing each corresponding time,
Object detection method.

Images