JP2012058780A - Device and method for creating environment map and device and method for action prediction - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device and a method for action prediction, capable of highly accurately estimating a transferring movement of a movable body.SOLUTION: An inventive action prediction device creates a likelihood map representing the propriety of the next velocity vector of a pedestrian, on the basis of a photographed image of the pedestrian, and then, converts the likelihood map into a probability density map representing the probability density regarding a position of the pedestrian after a unit time. The action prediction device arranges a plurality of hypotheses in the field having a high probability density on the probability density map, and calculates a plurality of predicted moving routes corresponding to the hypotheses. The hypothesis in this case corresponds to a single point on the probability density map and means that the pedestrian moves at the velocity vector corresponding to the point during the next unit time. Therefore, a predicted moving route can be obtained from the velocity vector.

Description

  The present invention relates to an environment map creation apparatus and method for creating an environment map related to a moving object such as a pedestrian, and an action prediction apparatus and method for predicting the action of a moving object.

  As a conventional behavior prediction apparatus and method, for example, one described in Patent Document 1 is known. In the thing of patent document 1, a pedestrian is detected from the pixel pattern information etc. of a moving object, both leg support time and the maximum stride of the pedestrian are obtained, and whether or not the pedestrian stops based on these information The future position of the pedestrian is predicted using the determination result, and the range where the pedestrian can exist is specified.

JP 2009-12521 A

  However, the following problems exist in the prior art. That is, in an urban area where there are many pedestrians, pedestrians often move rapidly. For this reason, the estimation accuracy cannot be increased by the method of linearly estimating the movement behavior of the pedestrian from the movement immediately before the pedestrian.

  An object of the present invention is to provide an environment map creation device and method, and a behavior prediction device and method that can estimate the movement behavior of a moving object with high accuracy.

  The present invention relates to an environment map creation device for creating an environment map related to a moving body, the moving body detecting means for detecting the moving body, and the probability of the position after a predetermined time of the moving body detected by the moving body detecting means. Probability density map creation means for creating a probability density map representing density as an environment map is provided.

  As described above, in the environment map creating apparatus of the present invention, a probability density map representing the probability density for the position of the moving object after a predetermined time is created. Accordingly, when predicting the course of a moving object using such a probability density map, for example, by selecting a plurality of points with high probability on the probability density map, a plurality of predicted paths are obtained for one moving object. be able to. Thereby, the estimation accuracy of the moving action of the moving body can be improved.

  Preferably, a mixed normal distribution generating means for expressing the probability density map by a mixed normal distribution is further provided. In this case, by obtaining a plurality of normal distribution parameters (center value, height, etc.) constituting the mixed normal distribution, a plurality of predicted courses can be reliably obtained for one moving body.

  Preferably, the apparatus further comprises group determination means for determining whether or not the plurality of moving bodies belong to one group when a plurality of moving bodies are detected by the moving body detection means, and the probability density map creation means When the determination means determines that a plurality of mobile objects belong to one group, a probability density map is created with the group as one mobile body. When a plurality of moving bodies act as a group, the moving speed and moving direction of each moving body should be the same. Therefore, when it is determined that a plurality of moving bodies belong to one group, a probability density map is created by using the group as one moving body, and the path of the moving body is determined using such a probability density map. In the case of prediction, it is possible to accurately predict the course when a plurality of moving bodies act as a group.

  The present invention also relates to a behavior predicting apparatus for predicting the behavior of a moving body, the moving body detecting means for detecting the moving body, and the probability of the position after a predetermined time of the moving body detected by the moving body detecting means. A probability density map creating means for creating a probability density map representing density; and a course predicting means for predicting the course of the moving object using the probability density map created by the probability density map creating means. Is.

  As described above, in the behavior prediction apparatus of the present invention, a probability density map representing the probability density for the position of the moving object after a predetermined time is created, and the course of the moving object is predicted using the probability density map. At this time, for example, by selecting a plurality of points with high probability on the probability density map, a plurality of predicted courses can be obtained for one moving object. Thereby, the estimation accuracy of the moving action of the moving body can be improved.

  Preferably, the course prediction means predicts the course of the moving object by arranging a plurality of hypotheses on the probability density map. In this case, a plurality of predicted courses can be reliably obtained for one moving body.

  Further, the apparatus further comprises a mixed normal distribution generating means for expressing the probability density map by a mixed normal distribution, and a parameter acquiring means for obtaining a parameter of the mixed normal distribution, and the course predicting means includes a moving object and other adjacent to the moving object. The course of the moving body may be predicted based on the parameters of the mixed normal distribution related to the moving body. In this case, a plurality of predicted courses can be reliably obtained for the moving body and other moving bodies adjacent to the moving body.

  At this time, preferably, the parameter obtaining unit obtains a parameter of the mixed normal distribution based on a discretized map obtained by discretizing the probability density map. In this case, the parameters of the mixed normal distribution can be easily obtained with a small amount of calculation.

  Preferably, the apparatus further comprises group determination means for determining whether or not the plurality of moving bodies belong to one group when a plurality of moving bodies are detected by the moving body detection means, and the probability density map creation means When the determination means determines that a plurality of mobile objects belong to one group, a probability density map is created with the group as one mobile body. In this case, as described above, it is possible to accurately predict the course when a plurality of moving bodies act as a group.

  Furthermore, the course prediction means uses the path of the moving body predicted using the probability density map as an initial value, and models the states of the plurality of moving bodies with a conditional random field, so that a plurality of moving bodies can be Simultaneously with the determination of whether or not it belongs to a group, the course of a plurality of moving bodies may be predicted. In this case, the time required only to determine whether or not a plurality of moving bodies belong to one group is eliminated, so that it is possible to predict the course when the plurality of moving bodies act in a group in a short time.

  The present invention is also an environment map creation method for creating an environment map relating to a moving body, the step of detecting the moving body, and a probability density map representing a probability density for a position after a predetermined time of the moving body. And a step of creating as a feature.

  As described above, in the environment map creation method of the present invention, a probability density map representing a probability density for a position after a predetermined time of a moving object is created, so that a plurality of predicted courses are obtained for one moving object as described above. Obtainable. Thereby, the estimation accuracy of the moving action of the moving body can be improved.

  Furthermore, the present invention is a behavior prediction method for predicting the behavior of a mobile object, the step of detecting the mobile object, and the step of creating a probability density map representing the probability density for the position of the mobile object after a predetermined time; And a step of predicting the course of the moving object using the probability density map.

  Thus, in the behavior prediction method of the present invention, by creating a probability density map representing the probability density for the position of the moving object after a predetermined time, and predicting the course of the moving object using the probability density map, As described above, a plurality of predicted courses can be obtained for one moving body. Thereby, the estimation accuracy of the moving action of the moving body can be improved.

  ADVANTAGE OF THE INVENTION According to this invention, the movement action of a moving body can be estimated with high precision. As a result, for example, when tracking a moving object using a technique for estimating the moving behavior of the moving object, the tracking performance can be improved.

It is a block diagram which shows schematic structure of 1st Embodiment of the action prediction apparatus concerning this invention. It is a flowchart which shows the detail of the process sequence performed by the pedestrian course prediction part shown in FIG. It is a figure which shows an example of the likelihood map which expressed the validity of the next velocity vector of a pedestrian by the polar coordinate. It is a figure which shows an example of the likelihood map which expressed the validity of the next velocity vector of a pedestrian by the square coordinate. It is a figure which shows the probability density map corresponding to the likelihood map shown in FIG. It is a figure which shows the state which has arrange | positioned five hypotheses on the probability density map shown in FIG. It is the schematic which shows a mode that the course of a pedestrian's movement is predicted sequentially. It is a flowchart which shows the detail of the process sequence performed by the pedestrian course prediction part in 2nd Embodiment of the action prediction apparatus concerning this invention. FIG. 6 is a diagram illustrating a state in which the probability density map illustrated in FIG. 5 is expressed by a mixed normal distribution. It is a figure which shows each normal probability distribution which comprises the mixed normal distribution shown in FIG. It is the schematic which shows an example of the following speed vector about two pedestrians. It is the schematic which shows a mode that the position of the next unit time of each pedestrian is calculated | required repeatedly. It is a flowchart which shows the detail of the process sequence which calculates | requires the parameter of mixed normal distribution by a simple method. It is a figure which simplifies two-dimensionally and shows an example of a probability density map and the discretization map obtained by discretizing the said probability density map. It is a block diagram which shows schematic structure of 3rd Embodiment of the action prediction apparatus concerning this invention. It is a flowchart which shows the detail of the process sequence performed by the group determination part shown in FIG. It is a figure which shows an example of the collation pattern of the state where pedestrians hold hands. It is a flowchart which shows the detail of the other process procedure performed by the group determination part shown in FIG. It is the schematic which shows a mode that each pedestrian which forms the group, and each pedestrian which does not form the group move with progress of time. It is a block diagram which shows schematic structure of 4th Embodiment of the action prediction apparatus concerning this invention. It is a flowchart which shows the detail of the process sequence performed by the group determination / pedestrian course prediction part shown in FIG. It is the schematic which shows an example of the graph model of a conditional random field.

  Preferred embodiments of an environment map creation device and method, an action prediction device and method according to the present invention will be described below in detail with reference to the drawings. In the drawings, the same or equivalent elements are denoted by the same reference numerals, and redundant description is omitted.

  First, a first embodiment of the behavior prediction apparatus according to the present invention will be described with reference to FIGS. FIG. 1 is a block diagram showing a schematic configuration of a first embodiment of the behavior prediction apparatus according to the present invention. In the figure, a behavior prediction device 1 of the present embodiment is a device that detects a pedestrian by image recognition, creates an environment map related to the pedestrian, and predicts the movement behavior of the pedestrian using the environment map.

  The behavior prediction apparatus 1 includes a camera 2 that images a pedestrian, an ECU (Electronic Control Unit) 3, and an output display 4. The camera 2 may be fixedly installed at a high place such as a building, or may be mounted on a vehicle or the like.

  The ECU 3 includes a CPU, a memory such as a ROM and a RAM, an input / output circuit, and the like. The ECU 3 includes an image processing unit 5, a storage unit 6, and a pedestrian course prediction unit 7.

  The image processing unit 5 performs image processing such as filter processing, binarization processing, and feature extraction processing on the captured image of the pedestrian acquired by the camera 2 to generate an image frame (image data) including the pedestrian. To do. The storage unit 6 stores parameters and the like used for calculation processing by the pedestrian course prediction unit 7.

  The pedestrian course prediction unit 7 performs a predetermined process based on the image frame generated by the image processing unit 5 to predict the course of the pedestrian, and outputs and displays the prediction result on the output display 4.

  FIG. 2 is a flowchart showing details of a processing procedure executed by the pedestrian course prediction unit 7. In the figure, first, a likelihood map (potential map) representing the validity of the next speed vector (direction and speed) of the pedestrian is created based on the image frame generated by the image processing unit 5 (step S101). .

  Here, FIG. 3 shows an example of a likelihood map expressing the validity of the next velocity vector of the pedestrian in polar coordinates. The likelihood map shown in FIG. 3 includes an energy term relating to the condition that the distance between pedestrians is kept above a certain level, an energy term relating to the condition that the pedestrian is trying to maintain a constant speed, and the pedestrian is the destination. Represents an energy function defined by a weighted linear sum with an energy term relating to the condition of trying to proceed toward.

In FIG. 3, white points A ₁ and A ₂ indicate speed vectors that can advance to the next unit time (for example, 0.2 seconds) when the pedestrians P ₁ and P ₂ maintain the current speed and direction. Yes. Here, 60 degrees on the left and right with respect to the directions indicated by the points A ₁ and A ₂ are changed in increments of 10 degrees and further changed by 0.3 m / s before and after the speed indicated by the points A ₁ and A ₂ . It represents the validity of the velocity vector. At this time, the validity (possibility) as a velocity vector increases in the order of the black portion, hatched portion, dot portion, and white portion.

  FIG. 4 shows an example of a likelihood map expressing the validity of the next speed vector of the pedestrian in square coordinates. Note that the likelihood map shown in FIG. 4 does not correspond to the likelihood map shown in FIG. In FIG. 4, the x-axis of the likelihood map indicates the lateral (left / right) direction position when the pedestrian P advances to the next unit time, and the y-axis of the likelihood map indicates the pedestrian P is the next unit time. The forward position when proceeding to is shown. The next velocity vector of the pedestrian is represented as an xy coordinate position when the pedestrian P has advanced to the next unit time. The lower the likelihood map height (deeper), the more appropriate the velocity vector.

  Subsequently, the likelihood map obtained in step S101 is converted into a probability density map (step S102). The probability density map is an environment map representing the probability density for the position of the pedestrian after a unit time.

なお、ωＥ（ｖ_ｉ ^ｔ｜Ｓ^ｔ−１）が尤度マップを表すパラメータである。 FIG. 5 is obtained by converting the likelihood map shown in FIG. 4 into a probability density map. This probability density map is represented in the form of a Markov random field or a Gibbs random field. The following conversion formula is used for conversion from the likelihood map to the probability density map.

Note that ωE (v _i ^t | S ^t−1 ) is a parameter representing the likelihood map.

In such a probability density map, a map having a higher probability is more appropriate as the next velocity vector of the pedestrian. In other words, the area S _1, S ₂ low profile in the likelihood map shown in FIG. 4, corresponding to the area S _1, S ₂ with a high probability density in the probability density map shown in Figure 5.

  Subsequently, a plurality (n) of hypotheses are arranged in a region having a high probability density on the probability density map obtained in step S102 (step S103). At this time, a plurality of hypotheses are arranged randomly or evenly according to the probability of the probability density map or on the maximum point of the probability density map. FIG. 6 shows five hypotheses K arranged on the probability density map shown in FIG.

  Subsequently, n predicted movement paths corresponding to the n hypotheses are calculated (step S104). The hypothesis in this case is that it corresponds to one point on the probability density map, and the pedestrian advances by the next unit time at the speed vector corresponding to that point. Then, a predicted movement path is obtained from the velocity vector.

  Subsequently, it is determined whether or not n is larger than 1 (procedure S105). When n is larger than 1, n is subtracted by 1 (procedure S106). If n is not greater than 1, step S106 is omitted.

  Subsequently, it is determined whether or not the process of steps S101 to S104 has been repeated a specified number of times (step S107). If the process of steps S101 to S104 has not been repeated the specified number of times, the process returns to step S101.

  Then, as shown in FIG. 7, the process for obtaining the predicted movement path of the pedestrian P is repeated a plurality of times. At this time, if the unit time is 0.2 seconds, the course of the pedestrian P up to 2 seconds later is predicted when the processing of steps S101 to S104 is repeated 10 times. Further, as the step proceeds, the number of hypotheses arranged on the probability density map decreases from n to one. As a result, a plurality of predicted movement paths are obtained for the pedestrian P.

  On the other hand, when the processes of steps S101 to S104 are repeated a specified number of times, a plurality of predicted movement paths of pedestrians up to a predetermined time later are output to the output display 4 (step S108).

  In the above, the camera 2 and the image processing unit 5 of the ECU 3 constitute a moving body detection unit that detects a moving body. The above steps S101 and S102 of the pedestrian course prediction unit 7 of the ECU 3 include probability density map creating means for creating a probability density map representing the probability density for the position after a predetermined time of the moving body detected by the moving body detecting means. Constitute. The steps S103 to S107 constitute a course prediction unit that predicts the course of the moving object using the probability density map created by the probability density map creation unit.

  As described above, in the present embodiment, the likelihood map of the pedestrian velocity vector obtained from the image captured by the camera 2 is converted into a probability density map, and a plurality of different hypotheses are arranged on the probability density map. As a result, a plurality of predicted movement paths can be obtained for one pedestrian. Thus, for example, when there is an obstacle in the direction of travel of the pedestrian and the vehicle avoids both the left and right sides of the obstacle, both the left and right courses of the obstacle are predicted as the travel path of the pedestrian. Therefore, it is possible to prevent the prediction of the movement path of the pedestrian from greatly deviating. Therefore, the movement behavior of the pedestrian can be estimated with high accuracy.

  A second embodiment of the behavior prediction apparatus according to the present invention will be described with reference to FIGS. FIG. 8 is a flowchart showing details of a processing procedure executed by the pedestrian course prediction unit 7 in the second embodiment of the behavior prediction apparatus according to the present invention.

  In the figure, first, steps S101 and S102 are executed in the same manner as the process shown in FIG. Subsequently, the probability density map obtained in step S102 is expressed by a mixed normal distribution (step S111). FIG. 9 represents the probability density map shown in FIG. 5 with a mixed normal distribution. As shown in FIG. 10, the mixed normal distribution is expressed by a combination of a plurality of normal probability distributions.

  Subsequently, parameters of the mixed normal distribution are obtained (procedure S112). The parameters of the mixed normal distribution include the center value μ, the spread σ, and the height w of the normal probability distribution. These parameters are determined using known EM algorithms. As shown in FIG. 10, w is a weight value determined depending on the height of each normal distribution. As this weight value increases, the possibility that the pedestrian moves according to the distribution increases.

  Subsequently, the next velocity vector (predicted movement course) of the pedestrian is obtained from the parameters of the mixed normal distribution (step S113). Then, it is determined whether or not the next speed vector has been obtained for all pedestrians (step S114). If the next speed vector has not been obtained for all pedestrians, the process returns to step S101.

For example, assume that there are pedestrians A and B as shown in FIG. It is assumed that the next velocity vector of pedestrian A at this time is represented by a mixed normal distribution having two normal distributions. For this reason, the velocity vector corresponding to the median μ of the two normal distributions indicates two course candidates that the pedestrian A may proceed next. Let these velocity vectors be s ₁ and s ₂ . In addition, it is assumed that the next velocity vector of pedestrian B at this time is also represented by a mixed normal distribution having two normal distributions. For this reason, similarly to the above, two velocity vectors indicating the course candidates that the pedestrian B may move to the next are obtained. Let these velocity vectors be t ₁ and t ₂ .

  When it is determined in step S114 that the next velocity vector has been obtained for all pedestrians, a plurality of combinations relating to the movements of an arbitrary pedestrian and surrounding pedestrians are considered, and the product of these weight values is calculated ( Procedure S115). Then, the top m items having a large product of the weight values are selected from a plurality of combinations (step S116).

In the example shown in FIG. 11, since pedestrians A and B both travel at two speed vectors, there are two positions where pedestrians A and B reach in the next unit time. For this reason, there are 2 × 2 = 4 combinations of positions that pedestrians A and B can take after a unit time. At this time, if the weight values of the velocity vectors s ₁ , s ₂ , t ₁ , t ₂ are w (s ₁ ), w (s ₂ ), w (t ₁ ), w (t ₂ ), four combinations are possible. The product of the weight values in is: w (s ₁ ) × w (t ₁ ), w (s ₁ ) × w (t ₂ ), w (s ₂ ) × w (t ₁ ), w (s ₂ ) × w Calculated by (t ₂ ).

  Then, m (for example, two) patterns having a large product of weight values are selected from the four combination patterns (see FIG. 12). As a result, two patterns can be left as combinations of positions that the pedestrians A and B can take after a unit time.

  Subsequently, it is determined whether or not the processes of steps S101, S102, and S111 to S116 have been repeated a specified number of times (step S117). If these processes have not been repeated the specified number of times, the process returns to step S101.

  Thereby, as shown in FIG. 12, a new probability density map is obtained for the two selected patterns, and this probability density map is expressed by a mixed normal distribution, and the position of the next unit time of each pedestrian. Will be calculated.

  On the other hand, when the processes of steps S101, S102, and S111 to S116 are repeated a specified number of times, a plurality of predicted movement paths of pedestrians up to a predetermined time later are output to the output display 4 (step S108).

  In the above processing procedure, the top m patterns having a large weight value product are selected from a plurality of combination patterns related to the movements between adjacent pedestrians. However, the weight value product is a predetermined value or more. A certain pattern may be selected.

  In the above, the procedure S111 constitutes a mixed normal distribution generating unit that expresses the probability density map by a mixed normal distribution. The procedure S112 constitutes a parameter acquisition unit for obtaining a parameter of the mixed normal distribution. The above steps S113 to S117 constitute a course prediction means for predicting the course of the moving object using the probability density map created by the probability density map creation means.

  In the present embodiment as described above, the parameters of the mixed normal distribution matching the probability density map are obtained, and the next velocity vector of a plurality of pedestrians is obtained based on the parameters. Thus, a plurality of predicted movement paths can be obtained.

  At this time, only the top m patterns having a large weight value product are always selected from a plurality of combination patterns related to the movement of an arbitrary pedestrian and surrounding pedestrians, and the next velocity vector of each pedestrian is obtained again. Therefore, the amount of calculation can be reduced and the increase in calculation time can be suppressed.

  In the present embodiment, the weight values at each time in the plurality of predicted travel routes finally obtained may be summed for each predicted travel route, and only the top several items having a large total value may be output. . In addition, since a weight value corresponds to the calculated predicted movement route, it is possible to perform processing using the weight value in the subsequent stage.

  By the way, in the present embodiment, the parameters of the mixed normal distribution are obtained using the EM algorithm. However, since the EM algorithm is a type of algorithm that is converged by repeated calculation, the amount of calculation must be increased. Therefore, it is desirable to easily obtain the parameters of the mixed normal distribution by a method other than the EM algorithm.

  FIG. 13 is a flowchart showing details of a processing procedure for obtaining a parameter of a mixed normal distribution by a simple method.

  In the figure, first, the probability density map is discretized to 10 × 10, for example, and the maximum point in the discretized map is extracted (step S121). At this time, in the discretization map, when the value of the point of interest is larger than the values of the front, rear, left and right points, the point of interest becomes the maximum point. Note that the number of maximum points may be limited to the top k.

For example, when the probability density map shown in FIG. 14A is discretized, a discretized map as shown in FIG. 14B is obtained. In FIG. 14, the map is represented in a two-dimensionally simplified manner. Then, maximum points E ₁ and E ₂ are extracted in the discretization map.

  Subsequently, the center of the maximal point in the discretization map is set as an initial value, and the maximal point is obtained using a quasi-Newton method or the like on the probability density map before being discretized (step S122). In addition, as long as it is a nonlinear optimization method, not only a quasi-Newton method but any solution method may be used. The maximum point obtained at this time is the median value μ of one normal distribution of the mixed normal distribution (see FIG. 14).

  Subsequently, the points around the maximum point in the discretized map are viewed. Since the point around the local maximum is smaller than the value of the local maximum, it is labeled as the same group as the local maximum. Then, the surroundings of the newly labeled points are viewed in the same manner, and those with smaller values are sequentially labeled. As a result, a region including one local maximum point and connecting the surrounding points is completed. Do the same for all local maxima. Then, the variance covariance matrix σ of the probability density map before being discretized is obtained from the region including these local maximum points (step S123).

  Subsequently, the height w of the normal distribution is obtained from the median μ of the normal distribution obtained in step S122 and the variance covariance matrix σ of the probability density map obtained in step S123 (step S124). At this time, a height w that minimizes the difference between the normal distribution and the probability density map is obtained. As the difference between the normal distribution and the probability density map, the sum of squares of the difference between the points on the normal distribution and the points on the probability density map is used.

  By the above procedure, the parameters of the mixed normal distribution can be easily obtained with a small amount of calculation. Although the parameters of the mixed normal distribution obtained by this method are not strict, it has been confirmed by experiments that it is sufficiently practical for the task of predicting pedestrian behavior.

  A third embodiment of the behavior prediction apparatus according to the present invention will be described with reference to FIGS. FIG. 15: is a block diagram which shows schematic structure of 3rd Embodiment of the action prediction apparatus concerning this invention.

  In the figure, the ECU 3 of the pedestrian movement estimation apparatus 1 according to the present embodiment includes a plurality of pedestrians in one group (group) in addition to the image processing unit 5, the storage unit 6, and the pedestrian course prediction unit 7. It has a group determination unit 8 that determines whether or not it belongs.

  FIG. 16 is a flowchart showing details of a processing procedure executed by the group determination unit 8. Pedestrians forming a group often talk to each other, and in that case, the pedestrians face each other. In addition, pedestrians forming a group may hold hands with each other.

  Therefore, in the figure, first, based on the image frame generated by the image processing unit 5, the face orientations and positions of a plurality of pedestrians are detected (step S131). Subsequently, it is determined whether or not the distance between the pedestrians is equal to or less than a predetermined value and the faces of the pedestrians face each other (step S132).

  When the distance between the pedestrians is equal to or smaller than a predetermined value and the faces of the pedestrians face each other, it is determined that the pedestrians form a group (step S133). Therefore, when two pedestrians face each other while maintaining a distance of a predetermined value or less, it is determined that both pedestrians form a group. In addition, when one or both of these two pedestrians and another pedestrian are facing each other with a distance of a predetermined value or less, the three pedestrians form a group. It is determined that

When the procedure S132 is not satisfied, it is determined whether or not pedestrians are holding hands (procedure S134). For example, as shown in FIG. 17, this determination is performed by comparing a region V between the pedestrians P ₁ and P ₂ with a pattern in which hands are obtained in advance by learning. As a pattern matching method, V & J method, HOG, or the like can be used.

  When pedestrians are holding hands, it is determined that each pedestrian forms a group (step S133). On the other hand, when the pedestrians are not holding hands, it is determined that each pedestrian does not form a group (step S135).

  FIG. 18 is a flowchart showing details of another processing procedure executed by the group determination unit 8. It is estimated that the pedestrians forming the group are at a certain distance for a certain time or more.

  Therefore, in the figure, first, the positions of a plurality of pedestrians are detected based on the image frames generated by the image processing unit 5 (step S141). Subsequently, it is determined whether or not the situation in which the distance between the pedestrians is equal to or less than a predetermined value continues for a predetermined time (for example, 3 seconds) (step S142).

  When the situation where the distance between each pedestrian is less than or equal to a predetermined value continues for a predetermined time, it is determined that each pedestrian forms a group (step S143), and the distance between each pedestrian is less than or equal to a predetermined value. When such a situation does not continue for a predetermined time, it is determined that each pedestrian does not form a group (step S144).

Accordingly, as shown in FIG. 19 (a), when the situation where the distance between the pedestrians P ₁ and P ₂ is not more than a predetermined value continues for a long time, the pedestrians P ₁ and P ₂ form a group. It is determined that On the other hand, as shown in FIG. 19 (b), when the distance between the pedestrian P _1, P ₂ is not followed long situation is less than the predetermined value, the pedestrian P _1, P ₂ form a group It is determined that it is not.

  Returning to FIG. 15, when the group determination unit 8 determines that the plurality of pedestrians form one group, the pedestrian course prediction unit 7 regards the group as a virtual single pedestrian. To predict the course. The course prediction at this time is performed in the same manner as in the first embodiment or the second embodiment described above.

  In the above, the group determining unit 8 constitutes a group determining unit that determines whether or not a plurality of moving bodies belong to one group when a plurality of moving bodies are detected by the moving body detecting unit.

  As described above, in the present embodiment, when a plurality of pedestrians belong to one group, the course is predicted on the assumption that the group is a single pedestrian. When a person acts as a group as a group, it is possible to prevent the pedestrian's prediction of the movement path from being erroneous.

  A fourth embodiment of the behavior prediction apparatus according to the present invention will be described with reference to FIGS. FIG. 20 is a block diagram showing a schematic configuration of the fourth embodiment of the behavior prediction apparatus according to the present invention.

  In the figure, the ECU 3 of the behavior prediction apparatus 1 of this embodiment has a group determination / pedestrian path prediction unit 9 instead of the group determination unit 8 and the pedestrian path prediction unit 7 shown in FIG.

  FIG. 21 is a flowchart showing details of a processing procedure executed by the group determination / pedestrian course prediction unit 9. In the figure, first, initial values of the movement paths of a plurality of pedestrians are obtained (step S151). The calculation of the initial value of the movement path is performed using the environment map related to the pedestrian, as in the first embodiment or the second embodiment.

  Subsequently, the determination of whether each pedestrian belongs to one group and the prediction of each pedestrian's movement course are performed simultaneously using a conditional random field graph model (step S152).

  FIG. 22 shows an example of a conditional random field graph model. In FIG. 22, Φ is a term representing the movement and appearance characteristics of the pedestrian P. Ψ is a term representing the relationship between the position and orientation of two pedestrians P. Υ is a term indicating whether the two pedestrians P form a group. χ is a term for confirming a group including three pedestrians P when one pedestrian P common to a group of two or two is included.

なお、Φ、Υ、Ψ、χは、確率を出力する関数である。 These terms are formulated as follows.

Note that Φ, Υ, Ψ, and χ are functions that output probabilities.

  In the above formula, H represents a predicted course of pedestrians, L represents a group state between pedestrians, I represents an input image sequence, and Θ represents an observed parameter. Here, when I and Θ are given, it is the meaning of the above formula that H and L are determined such that logP is maximized.

Φ^motionは歩行者の動きを表し、Φ^appは歩行者の外見上の特徴を表している。Θ_speedは歩行者の速度を表し、Θ_appは歩行者の確からしさを表している。Ψ^posは２人の歩行者の位置関係を表し、Ψ^angは２人の歩行者の向きの関係を表している。 Each term of the above formula is expressed as follows.

Φ ^motion represents the movement of the pedestrian, and Φ ^app represents the appearance characteristic of the pedestrian. Θ _speed represents the _speed of the pedestrian, and Θ _app represents the likelihood of the pedestrian. Ψ ^pos represents the positional relationship between the two pedestrians, and ψ ^ang represents the relationship between the orientations of the two pedestrians.

  In addition, the term of Υ indicating whether or not two pedestrians form a group, and the term of χ that confirms that the group includes three pedestrians is “0” or “1”. The last term (logZ (I, Θ)) of the above formula is an adjustment term.

Here, as a representative, Φ ^motion (h _i | Θ _speed ) of the first term on the right side of the equation (A) will be described in detail.

を意味する。つまり、各歩行者についての確率を計算し、その確率を全部足し合わせて場面全体の確率を計算する。 i is a number representing each target pedestrian. If there are n pedestrians in the scene, Σ in the first term on the right side of the above equation (A) is

Means. That is, the probability for each pedestrian is calculated, and the probabilities for the entire scene are calculated by adding all the probabilities.

h _i is a future course of the pedestrian i, means a change in position every unit time, and is expressed as h _i = {p ₁ , p _{2 ,.} The subscript is a time step. The unit time is set to 0.2 sec, when the repeating 10 _{steps, h i = {p 0.2,} p 0.4, ..., p 1.8, p 2.0} can be expressed as. The subscript represents the elapsed time. Here, since p represents a place, if a pedestrian moves on a plane, it is represented by coordinates such as p = {x, y}.

Θ _speed given as a parameter is the _speed of the pedestrian i at the time of each unit time obtained from h _i . That is, Θ _speed means the _speed of pedestrian i after 0.2 seconds, the speed after 0.4 seconds, ..., the speed after 2.0 seconds. h _i, since it represents the position of the pedestrian i for every elapse each time unit, can also be calculated velocity at that point in time.

Now, Φ ^motion (h _i | Θ _speed ) means the probability that the course at that time is h _i when Θ _speed is given. The probability is obtained by determining the relationship between the walking speed and direction of each pedestrian from past observation results. Although not specifically described in detail, this is defined as the formula (B). In other words, ask for one of the probability assumed h _i can occur much to every pedestrian, adding the probabilities for all of the pedestrian. The fact that this sum combined probability is maximum, because most likely probability that scene is that high, it comes to the path to be obtained when the resulting h _i truly thereon.

Since Φ ^motion , Φ ^app , Ψ ^pos , Ψ ^ang are functions whose values change depending on h _i , the most probable probability in the scene when the respective probabilities are maximized so that a high h _i, ie ^{Φ motion, Φ app, Ψ pos} , that [psi ^ang all elements considered the most appropriate h _i.

A probability function is similarly defined for the group state, and the probabilities represented by Υ, Ψ ^pos , Ψ ^ang , and χ vary depending on the group state.

  As described above, we prepared a function that changes the probability depending on the future course of each pedestrian and the state of the group between pedestrians, and added all the probabilities together for all pedestrians. The above formula (A) means that the course and the state of the group in which the thing is maximized are obtained.

  As a calculation method for solving such an expression, an existing method such as a dual decomposition method is used.

  In the above, the group determination / pedestrian course prediction unit 9 configures a course prediction unit that predicts the course of the moving object using the probability density map created by the probability density map creation unit.

  As described above, in this embodiment, by using a conditional random field graph model, prediction of each pedestrian's movement path and determination of whether each pedestrian belongs to one group, Can be done simultaneously. This eliminates the time required to determine whether or not each pedestrian belongs to a single group, so that the movement path when multiple pedestrians act in a group can be predicted efficiently in a short time. Can do.

  In this embodiment, in order to perform the calculation quickly, the initial values of the movement paths of the plurality of pedestrians are obtained using the environment map described above. However, the method is not limited to this method, and the movements of the plurality of pedestrians are not limited. The initial value of the course may also be obtained using a conditional random field graph model.

  As mentioned above, although several suitable embodiment of the environment map preparation apparatus and method, action prediction apparatus, and method concerning this invention has been described, this invention is not limited to the said embodiment.

  For example, the above embodiment creates an environment map for a pedestrian and predicts the behavior of the pedestrian, but creates an environment map for a moving body other than the pedestrian and predicts the behavior of the moving body. Anyway.

  Moreover, in the said embodiment, although the pedestrian was detected from the captured image acquired with the camera 2, in addition to this, moving bodies, such as a pedestrian, are detected by sensors, such as laser radar and a millimeter wave radar, or GPS. Also good.

DESCRIPTION OF SYMBOLS 1 ... Behavior prediction apparatus, 2 ... Camera (moving body detection means), 3 ... ECU, 4 ... Image processing part (moving body detection means), 7 ... Pedestrian course prediction part (Probability density map creation means, Mixed normal distribution generation) Means, parameter acquisition means, course prediction means), 8... Group determination section (group determination means), 9... Group determination / pedestrian path prediction section (course prediction means).

Claims (11)

An environment map creation device for creating an environment map related to a moving object,
Moving body detecting means for detecting the moving body;
Probability density map creation means for creating, as the environment map, a probability density map that represents a probability density for a position after a predetermined time of the mobile body detected by the mobile body detection means. apparatus.   The environment map creating apparatus according to claim 1, further comprising mixed normal distribution generating means for expressing the probability density map by a mixed normal distribution. Further comprising group determination means for determining whether or not the plurality of moving bodies belong to one group when the plurality of moving bodies are detected by the moving body detection means;
The probability density map creating means creates the probability density map with the group as one moving body when the group determining means determines that the plurality of moving bodies belong to one group. The environment map creation device according to claim 1 or 2. A behavior prediction device for predicting behavior of a moving body,
Moving body detecting means for detecting the moving body;
A probability density map creating means for creating a probability density map representing a probability density for a position after a predetermined time of the moving body detected by the moving body detecting means;
A behavior prediction apparatus comprising: a course prediction unit that predicts a course of the moving object using the probability density map created by the probability density map creation unit.   5. The behavior prediction apparatus according to claim 4, wherein the course prediction unit predicts a course of the moving object by arranging a plurality of hypotheses on the probability density map. Mixed normal distribution generating means for expressing the probability density map by a mixed normal distribution;
Parameter obtaining means for obtaining a parameter of the mixed normal distribution;
5. The behavior according to claim 4, wherein the course predicting unit predicts a course of the moving body based on a parameter of the mixed normal distribution related to the moving body and another moving body adjacent to the moving body. Prediction device.   The behavior prediction apparatus according to claim 6, wherein the parameter obtaining unit obtains a parameter of the mixed normal distribution based on a discretized map obtained by discretizing the probability density map. Further comprising group determination means for determining whether or not the plurality of moving bodies belong to one group when the plurality of moving bodies are detected by the moving body detection means;
The probability density map creating means creates the probability density map with the group as one moving body when the group determining means determines that the plurality of moving bodies belong to one group. The behavior prediction apparatus according to any one of claims 4 to 7.   The course predicting means uses the course of the moving body predicted by using the probability density map as an initial value, and models the states of the plurality of moving bodies by a conditional random field, so that the plurality of moving bodies are integrated. The behavior prediction apparatus according to claim 4, wherein the route of the plurality of moving objects is predicted simultaneously with the determination of whether or not it belongs to one group. An environment map creation method for creating an environment map related to a moving object,
Detecting the moving body;
Creating a probability density map representing a probability density for a position after a predetermined time of the moving object as the environment map. A behavior prediction method for predicting the behavior of a moving body,
Detecting the moving body;
Creating a probability density map representing the probability density for a position of the mobile object after a predetermined time;
Predicting the course of the moving object using the probability density map.

Images