WO2016183793A1

WO2016183793A1 - Systems and methods for pedestrian walking path prediction

Info

Publication number: WO2016183793A1
Application number: PCT/CN2015/079256
Authority: WO
Inventors: Xiaogang Wang; Shuai Yi; Hongsheng LI
Original assignee: Xiaogang Wang
Priority date: 2015-05-19
Filing date: 2015-05-19
Publication date: 2016-11-24
Also published as: CN107646111A; CN107646111B

Abstract

Disclosed is a system for pedestrian walking path prediction from video frames. In one embodiment, the video frames may comprise at least one of a first group of frames including scene obstacles, a second group of frames including moving pedestrians, and a third group of frames including stationary pedestrians. The system comprises: an energy distribution generation device generating an energy distribution for the video frames, wherein the generated energy distribution comprising a combination of at least one of a scene layout energy distribution for the first group of frames, a moving pedestrian distribution for the second group of frames, and a stationary group distribution for the third group of frames. The system further comprises a walking path generation device that is electrically communicated with the energy distribution generation device, and determines a most probable walking path for current individuals, by minimizing an energy cost along a walking route according to the generated energy distribution.

Description

Systems and Methods for Pedestrian Walking Path Prediction

Technical Field

The disclosures relate to a system and a method for pedestrian walking path prediction, in particular, relate to a system and a method for understanding pedestrian behaviors from stationary crowd groups.

Background

Pedestrian behavior modeling and analysis is important in video surveillance and has drawn increasing attentions in recent years. It can be used for various applications including pedestrian walking path prediction, traffic flow segmentation, crowd counting and segmentation, and abnormal event detection.

Pedestrian behavior modeling is challenging, especially for scenes with crowds. Previous studies have shown that the walking behavior of an individual can be influenced by a variety of factors including scene layout (e.g. entrances, exits, walls, and obstacles) , pedestrian beliefs (the choice of source and destination) , and interactions with other moving pedestrians. However, an important factor, i.e. stationary crowd groups, is missing in literature of modeling pedestrian behaviors.

The stationary crowd groups have considerable influence on pedestrians and are crucial in pedestrian behavior modeling. As shown in Fig. 1 (d) , the walking path of a pedestrian is affected by a stationary crowd group. However, without modeling the stationary crowd group, it is difficult to explain why the pedestrian detours when approaching the destination, as shown in Fig. 1 (f) .

Studies also show that stationary crowd groups have greater influences on pedestrian behaviors than moving crowds. A pedestrian usually changes the walking speed rather than direction to avoid collision with other moving crowds. However, when moving crowds become stationary, the walking pedestrian is forced to change his or her direction and the walking path is influenced significantly.

As shown in Fig. 1 (e) , stationary crowd groups can serve as multiple roles for different pedestrians. For pedestrians that are leaving or joining a stationary crowd group, it can be regarded as the source or the destination. For other pedestrians that are moving near the stationary crowd group, it can be regarded as an obstacle. Although both stationary crowd groups and fixed scene obstacles can block traffic, a pedestrian can choose to walk through the stationary crowd group or to detour from it, while scene obstacles are solid and cannot be penetrated. Moreover, as shown in Figs. 1 (a) - (d) , the spatial distribution of stationary crowd groups might change over time, which leads to the dynamic variations of traffic patterns. Therefore, static models cannot be used for stationary crowd group modeling.

Summary

The following presents a simplified summary of the disclosure in order to provide a basic understanding of some aspects of the disclosure. This summary is not an extensive overview of the disclosure. It is intended to neither identify key or critical elements of the disclosure nor delineate any scope of particular embodiments of the disclosure, or any scope of the claims. Its sole purpose is to present some concepts of the disclosure in a simplified form as a prelude to the more detailed description that is presented later.

Pedestrian behavior modeling and analysis is important for crowd scene understanding and has various applications in video surveillance. Stationary crowd groups are a key factor influencing pedestrian walking patterns but was largely ignored in literature. In this application, a novel model is proposed for pedestrian behavior modeling by including stationary crowd groups as a key component. Through inference on the interactions between stationary crowd groups and pedestrians, the proposed model can be used to investigate pedestrian behaviors. The effectiveness of the proposed model is demonstrated through multiple applications, including walking path prediction, destination prediction, personality classification, and abnormal event detection.

In one aspect, the present application is proposed to at least address the problem of pedestrian walking path prediction. It aims at automatically simulating which walking route is one pedestrian likely to choose given the source and destination, based on the scene structure, moving pedestrians in the scene, and stationary groups in the scene.

In the proposed solutions of the present application, the factor of stationary crowd groups is introduced for the first time to model pedestrian behaviors. Both walking through and walking bypass pedestrians can be well modeled. The proposed model can be dynamically updated with time to adapt the change of stationary crowd groups.

Based on the proposed model, the present application can investigate the influence of stationary crowd groups on pedestrian behaviors. By learning model parameters, it is observed that stationary crowd groups have greater influence on pedestrian walking paths than moving crowds, which shows the importance of monitoring stationary groups in a traffic control system. Moreover, by modeling the interactions among stationary groups and moving pedestrians, a personality attribute is proposed to classify pedestrians into different categories. This attribute is a key factor that makes each individual behave differently. One interesting observation is that people are more likely to behave in a conservative way when the scene is not that crowded. In contrast, a crowded scene leads to aggressive walking patterns because of the lack of space.

In accordance with one or more embodiments and corresponding disclosure, various non-limiting aspects are described in connection with systems and methods for pedestrian walking path prediction. In an embodiment, a system is provided to proceed with the video frames comprising at least one of a first group of frames including scene obstacles, a second group of frames including moving pedestrians, and a third group of frames including stationary pedestrians. The system comprises: an energy distribution generation device generating an energy distribution for the video frames, the generated energy distribution comprising a combination of at least one of a scene layout energy distribution for the first group of frames, a moving pedestrian distribution for the second group of frames, and a stationary group distribution for the third group of frames； and a walking path generation device determining a most probable walking path for current individuals, by minimizing an energy cost along a walking route according to the generated energy distribution.

In an embodiment, a method for pedestrian walking path prediction from video frames is disclosed, which may comprise: obtaining the video frames； segmenting the obtained video frames into at least one of a first group of frames including at least one scene obstacle, a second group of frames including moving pedestrians, and a third group of frames including stationary pedestrians； generating, from the segmented video frames, an energy distribution for the video frames, the generated energy distribution comprises a combination of at least one of a scene layout energy distribution for the first group of frames, a moving pedestrian distribution for the second group of frames, and a stationary group distribution for the third group of frames； and determining a most probable walking path for current individuals, by minimizing a cost along a walking route according to the generated energy distribution.

In a yet further embodiment, a system for pedestrian walking path prediction from video frames is disclosed. The video frames comprise at least one of a first group of frames including scene obstacles, a second group of frames including moving pedestrians, and a third group of frames including stationary pedestrians, and the system may comprises: a memory that stores executable components； and a processor electrically coupled to the memory to execute the executable components to perform operations of the system. The executable components comprise: an energy distribution generation component generating an energy distribution for the video frames, the generated energy distribution comprising a combination of at least one of a scene layout energy distribution for the first group of frames, a moving pedestrian distribution for the second group of frames, and a stationary group distribution for the third group of frames； and a walking path generation component determining a most probable walking path for current individuals, by minimizing an energy cost along a walking route according to the generated energy distribution.

The following description and the annexed drawings set forth certain illustrative aspects of the disclosure. These aspects are indicative, however, of but a few of the various ways in which the principles of the disclosure may be employed. Other aspects of the disclosure will become apparent from the following detailed description of the disclosure when considered in conjunction with the drawings.

Brief Description of the Drawing

Exemplary non-limiting embodiments of the present invention are described below with reference to the attached drawings. The drawings are illustrative and generally not to an exact scale. The same or similar elements on different figures are referenced with the same reference numbers.

Figs. 1 (a) -1 (f) illustrate a schematic diagram of the pedestrian behaviors prediction, wherein Figs. 1 (a) - (b) show two video frames, Figs. 1 (c) - (d) illustrate energy maps calculated from Figs. 1 (a) - (b) using the proposed model., Fig. 1 (e) is an illustration of multiple roles of a stationary crowd group, and Fig. 1 (f) illustrates an energy map calculated from Fig. 1 (b) without modeling the factor of stationary crowd groups.

Fig. 2 illustrates a schematic block diagram of a system for pedestrian walking path prediction according to one embodiment of the present application.

Fig. 3 illustrates a schematic block diagram of the energy distribution generation device as shown in Fig. 1 according to one embodiment of the present application.

Fig. 4 illustrates an example of a scene layout map/distribution according to one embodiment of the present application.

Fig. 5 illustrates an example of a moving pedestrian influence map/distribution according to one embodiment of the present application.

Fig. 6 illustrates an example of a stationary crowd group influence map/distribution according to one embodiment of the present application.

Fig. 7 illustrates an example of path generation according to one embodiment of the present application.

Fig. 8 illustrates a schematic block diagram of a system for pedestrian walking path prediction according to a further embodiment of the present application.

Fig. 9 is a schematic diagram illustrating a flow process for a method for pedestrian walking path prediction according to a further embodiment of the present application.

Fig. 10 a schematic diagram illustrating a flow process for the step of generating as shown in Fig. 9 according to one embodiment of the present application.

Fig. 11 is a schematic diagram illustrating a flow process for a method for pedestrian walking path prediction according to a further embodiment of the present application.

Fig. 12 illustrates a system for pedestrian walking path prediction from video frames according to a yet another embodiment of the present application.

Fig. 13 illustrates a system for pedestrian walking path prediction from video frames according to a yet another embodiment of the present application.

Detailed Description

Reference will now be made in detail to some specific embodiments of the invention including the best modes contemplated by the inventors for carrying out the invention. Examples of these specific embodiments are illustrated in the accompanying drawings. While the invention is described in conjunction with these specific embodiments, it will be understood that it is not intended to limit the invention to the described embodiments. On the contrary, it is intended to cover alternatives, modifications, and equivalents as may be included within the spirit and scope of the invention as defined by the appended claims. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. The present invention may be practiced without some or all of these specific details. In other instances, well-known process operations have not been described in detail in order not to unnecessarily obscure the present invention.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a" , "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising, " when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

Fig. 2 illustrates a schematic block diagram of a system 1000 for pedestrian walking path prediction according to one embodiment of the present application. As shown, the system 1000 comprises an energy distribution generation device 100 and a walking path generation device 200 electrically coupled to the device 100. The energy distribution generation device 100 is configured to receive the video frames. The video frames may be obtained from a surveillance system or other video frames. The received video frames may comprise at least one of a first group of frames including at least one scene obstacle, a second group of frames including moving pedestrians, and a third group of frames including stationary pedestrians. The energy distribution generation device 100 generates an energy distribution for the video frames, which may be a combination of at least one of a scene layout energy distribution for the first group of frames, a moving pedestrian distribution for the second group of frames, and a stationary group distribution for the third group of frames. The walking path generation device 200 is configured to calculate a most probable walking path for current individuals, by minimizing a cost along a walking route according to the generated energy distribution.

Fig. 3 illustrates a schematic block diagram of the energy distribution generation device 100 as shown in Fig. 2 according to one embodiment of the present application. As shown, the energy distribution generation device 100 may comprise a scene factor segmentation unit 101, a scene layout distribution modeling unit 102, a moving pedestrian distribution modeling unit 103, a stationary group distribution modeling unit 104 and an energy distribution combination unit 105.

In the present application, different influence factors should be factorized and modeled separately, so the factor segmentation unit 101 is configured to separate each influence factors, for example, scene obstacle, moving pedestrians, and stationary pedestrians. In particular, the scene factor segmentation unit 101 is configured to receive a plurality of video frames and segment each input video frame into multiple types of frames, such as frames including scene obstacle, frames including moving pedestrians, and frames including stationary pedestrians. The segmentation may be used any of the conventional technical means in the art.

Scene layout map/distribution modeling unit 102 is electronically communicated with the factor segmentation unit 101 or electronically coupled to the factor segmentation unit 101. The scene layout map/distribution modeling unit 102 operates to receive the frames including the separated scene obstacle, and model these frames into a scene layout map/distribution. Pedeatrian’a walking behavior is constrained by scene layout. The pedestrians cannot walk freely in a scene due to the constraints of walls and other static obstacles, and thus they cannot be observed at some locations. Moreover, people tends to keep a distance from these obstacles and are not likely to walk very close to them, and thus the probability of observing the pedestrian decreases when getting close to the obstacle regions. Accordingly, to set the scene layout influence map/distribution, the scene layout map/distribution modeling unit 102 is configured to firstly detect all the scene obstacle regions, such as walls, pillars in the received image frames. Because pedestrians can never walk through a scene obstacle, the scene layout map/distribution modeling unit 102 sets the energy value inside the scene obstacle regions as 0, and locations near the scene obstacle boundary should have a low energy value because people are not likely to walk near the obstacles. The energy values of the locations near the scene obstacles will decrease according to the distance to the scene obstacle boundaries.

The scene layout influence map/distribution is therefore modeled as

where SL is a set of locations occupied by scene obstacles which are unreachable, d₁ (x, SL) ＝min_y∈SL

measures the distance from the current location x to its nearest scene obstacle location y, and θ₁ is a parameter indicating the influence bandwidth (which also can be viewed as the importance) of the scene layout term. If x∈SL, there is an obstacle at location x, and d₁ (x, SL) ＝0. In this case, f_SL (x, Θ) is equal to 0, which means that pedestrians cannot appear at location x. When

d₁ (x, SL) >0. f_SL (x, Θ) gets close to 0 when the current location x approaches to scene obstacles. An example of a scene layout map/distribution is shown in Fig. 4, in which Fig. 4 (a) illustrates the scene background, Fig. 4 (b) illustrates the energy values along the white horizontal lines in Figs. 4 (c) and 4 (d) , in which the above curve presents the energy values with θ₁, and the bottom curve presents the energy values with θ2. Figs. 4 (c) and (d) show two scene influence maps calculated by setting θ₁ as 0.01 and 0.05, respectively. Energy drops near the scene boundaries.

Pedestrian’s walking behavior can be influenced by the interactions with other moving pedestrians. The moving pedestrian map/distribution modeling unit 103 will model the moving pedestrians into moving pedestrian map/distribution. In particular, the moving pedestrian map/distribution modeling unit 103 operates to locate all the moving pedestrians from frames including moving pedestrians, which are received from the factor segmentation unit 101. And then, the moving pedestrian map/distribution modeling unit 103 sets the energy value at the location of the current moving pedestrian as 0. Since the pedestrians are likely to keep a private space from other moving pedestrians, locations near the moving pedestrian should have a low energy value. Accordingly, the moving pedestrian map/distribution modeling unit 103 decreases the energy values of the locations near the moving pedestrians, and sums the energy values of all moving pedestrians together.

The moving pedestrian influence map/distribution is modeled as

Where MP_i (i∈ [1； m] ) is the ith moving pedestrian,

is the spatial location of MP_i at current time t,

is used to estimate the spatial location of MP_i at time t+1,

measures the distance from the current location x to the moving pedestrian MP_i, and θ₂is the influence bandwidth of the moving pedestrian term. An example of a moving pedestrian influence map/distribution is shown in Fig. 5, in which Fig. 5 (a) illustrates a video frame, Fig. 5 (b) illustrates the energy values along the dash horizontal lines in Figs. 5 (c) and 5 (d) . Figs. 5 (c) - (d) illustrate two moving pedestrian influence maps calculated by setting θ₂ as 0.01 (the top curve) and 0.05 (the bottom curve) , respectively. Energy drops around moving pedestrian.

In addition, the pedestrian’s walking behavior can be influenced by the interactions with other stationary groups. The stationary group map/distribution modeling unit 104 will model the stationary pedestrians into stationary group map/distribution. In particular, the stationary group map/distribution modeling unit 104 receives frames including stationary pedestrians from the factor segmentation unit 101, and then detects all the stationary groups from the received frames.

Pedestrians are likely to keep a private space from other stationary groups, and thus the stationary group map/distribution modeling unit 104 makes locations near the stationary group have a low energy value, i.e., to decrease the energy values of the locations near the stationary groups. In addition, the pedestrian may walk through the stationary group, and sparse groups are more likely to walk through. Accordingly, the stationary group map/distribution modeling unit 104 sets the energy value inside the stationary group region as a fix value smaller than 1. Sparse stationary groups are assigned with larger energy values in the stationary group map/distribution modeling unit 104. Finally, the effect (i.e., the energy values) of all stationary groups will be summed together.

Stationary crowd groups are modeled in two aspects. First, for pedestrians that bypass a stationary crowd group, this stationary crowd group acts similarly as a scene obstacle. The group has a repulsive force around the group region to keep moving pedestrians away. Second, for pedestrians that walk through a stationary crowd group, there should be a penalty inside the group region. This is the key difference with the scene layout factor, where obstacles cannot be penetrated. The penalty is related to crowd density. It is more difficult to walk through denser stationary crowds.

The Stationary Group influence map/distribution is modeled as

where SG_i (i∈ [1, n] ) is the ith stationary crowd group region automatically detected by using the conventional approaches,

measures the distance from x to the stationary crowd group region SG_i, θ₃ is the influence bandwidth of the stationary crowd group term, and d₄ (SG_i) ∈ (0,+∞) is used to measure the sparsity of stationary crowd group region SG_i. d₄ is calculated as the average distance among group members. Larger d₄ represents lower crowd density. The weight θ₄ controls the influence of group sparsity on estimation result.

If x∈SG_i, the location x is inside SG_i, and d₃ (x, SG_i) ＝0. f_SG (x； Θ) at locations x∈SG_i inside the group is constant and is positively correlated with group sparsity d₄ (SG_i) . f_SG (x； Θ) is in the range of (0, 1) , which means that the probability of observing a pedestrian walking through the group region decreases because of the influence of the stationary group, but it is still larger than 0.

If

x is outside SG_i, and d₃ (x, SG_i)＞0. The influence value increases from group boundary to faraway locations. An example of a stationary crowd group influence map/distribution is shown in Fig. 6, In Fig. 6, the left views illustrate three stationary crowd group influence maps calculated from the same frame by using different θ3 and θ4. The right views illustrate the energy values along two vertical lines (a) and (b) in (Left) . In the energy curves obtained along lines (a) and (b) , the top curve represents the energy values with θ₃ ＝0.08, θ₄＝0.005； θ₃＝0.08 and θ₄＝0 in the middle curve, and θ₃＝0.15 and θ4＝0 in the bottom curve. Comparing the two the most top curves, it shall be noticed that the stationary group regions may have non-zero energy values by setting a nonzero θ₄. Different groups may have different energy values due to the density differences. By setting θ₄＝0, the differences disappear and the energy values inside the groups turn to zero.

After the three map/distributions are generated, i.e. the scene layout map/distribution, the moving pedestrian map/distribution, and the stationary group map/distribution, all the three map/distributions will be combined into the final general energy map/distribution by the general energy map/distribution combination unit 105. According to one embodiment of the present application, the scene layout energy map/distribution f_SL, the moving pedestrian map/distribution f_MP and the stationary group map/distribution f_SG will be multiplied together to form the final general energy map/distribution M (x； Θ) by rule of

M (x； Θ) ＝f_SL (x； θ₁) f_MP (x； θ₂) f_SG (x； θ₃， θ₄) 4)

Where Θ＝ [θ₁, θ₂, θ₃, θ₄] are weighting parameters for different terms. M is also a probability map/distribution and can be used as the probability of pedestrian appearing at each location. It can be extended by including new channels.

After the energy distribution M for the video frames is generated, the walking path generation device 200 will use fast marching algorithm to calculate a most probable walking path for the current individual using fast marching algorithm.

To generate pedestrian walking paths, the well known Fast Marching may beused. Given the source x_s and the destination x_d, an optimal path

is calculated based on the energy map/distribution M:

where

is the most efficient and probable route from x_s to x_d according to the current energy map/distribution M. Several examples are shown in Fig. 6. When using a personalized map/distribution MP, the optimal path is just for the specific individual. When using a general map/distribution M, the optimal path can be regarded as an average path for ordinary pedestrians. Fig. 7 illustrates an example of path generation. The point 701 is a source, the points 702 are destinations, and the curves between the point 701 and the points 702 are optimal walking routes calculated by Equation 5) .

Fig. 8 illustrates a schematic block diagram of a system 1000’ for pedestrian walking path prediction according to a further embodiment of the present application. Besides the energy distribution generation device 100， the system 1000’ further comprise a personalized energy map/distribution modeling device 300 and a walking path generation device 200’.

The functions and operations for the device 100 in this embodiment are the same of those in the embodiment as shown in Fig. 2, and thus the detailed description thereof are omitted.

People might behave differently under the same situation. It is modeled by a personality parameter P. Different personalized energy map/distributions MP are generated based on the general energy map/distribution M with different P values. For different individuals, the personalized energy map/distribution modeling device 300 will transform general energy map/distribution into personalized energy map/distribution MP by rule of:

M_P (x； Θ) ＝exp (P×ln M (x； Θ) ) 6)

If P is large for a pedestrian, the influence bandwidth of all the terms (θ₁, θ₂, θ₃) would equivalently increase for this individual. The energy values are small at locations near obstacles and stationary crowd groups. It denotes that this pedestrian cares more about these influence factors and is likely to walk a longer way to avoid close contact with these obstacles. In contrast, smaller P means that the pedestrian is walking aggressively and cares less about obstacles. To be specific, if P＝1, personalized energy map/distribution is the same as general energy map/distribution； P>1 means this pedestrian cares more about these influence factors and is likely to walk a longer way to avoid close contact with these obstacles, and the energy values of different locations are decreased in the same scale； If P<1, it means pedestrian is walking aggressively and cares less about obstacles, and the energy values of different locations are increased in the same scale.

Walking path generation device 200’ will use fast marching algorithm to calculate a most probable walking path for the current individual using fast marching algorithm. To generate pedestrian walking paths, the well known Fast Marching may be used.Given the source x_s and the destination x_d, an optimal path

is calculated based on the energy map/distribution M_P :

where

is the most efficient and probable route from x_s to x_d according to the current energy map/distribution M_p. Several examples are shown in Fig. 6. When using a personalized map/distribution M_P, the optimal path is just for the specific individual.

Hereinabove， the systems 1000 and 1000’ according to the embodiments of the present application have been discussed. Now， a method 2000 and 2000’ for pedestrian walking path prediction from video frames will be discussed.

Referring Fig. 9, the method 2000 for pedestrian walking path prediction according to one embodiment of the present application is illustrated. At step S20, the method 2000 obtains the video frames. The video frames may be obtained from a surveillance system or other video frames. At step S40, it segments the obtained video frames into at least one of a first group of frames including at least one scene obstacle, a second group of frames including moving pedestrians, and a third group of frames including stationary pedestrians. At step S60, it generates, from the segmented video frames, an energy distribution for the video frames, the generated energy distribution comprises a combination of at least one of a scene layout energy distribution for the first group of frames, a moving pedestrian distribution for the second group of frames, and a stationary group distribution for the third group of frames.

Fig. 10 is a schematic diagram illustrating a flow process for the step of generating as shown in Fig. 9 according to one embodiment of the present application. In particular, at step S601, it models one or more frames from the first group of frames into the scene layout energy distribution/map. The scene layout energy distribution is modeled by rule of: setting energy values inside scene obstacle regions as 0, and decreasing energy values of locations near the scene obstacles according to distances to boundaries of the scene obstacle, which is the same as the functions of the scene layout map modeling unit 102.

At step S602, the process locates all moving pedestrians from the second group of frames and models the located moving pedestrians into the moving pedestrian distribution. The moving pedestrian distribution is modeled by rule of: setting the energy value at the location of the current moving pedestrian as 0； and decreasing the energy values of the locations near the moving pedestrians, which is the same as the functions of the moving pedestrian map modeling unit 103.

At step S603, the process detects all stationary groups from the third group of frames to model the stationary pedestrians into the stationary group map/distribution. In one embodiment of the present application, the stationary group distribution is modeled by rule of: setting the energy value inside a stationary group region as a fix value smaller than 1； and assigning sparse stationary groups with larger energy values in the stationary group distribution, which is the same as the functions of the stationary group map modeling unit 104.

At step S604, the scene layout map/distribution, the moving pedestrian map/distribution, and the stationary group map/distribution are combined into the energy map/distribution. The scene layout energy distribution, the moving pedestrian distribution, and the stationary group distribution are multiplied together to form the energy distribution.

Rerunning to Fig. 9 again, the process 200 then proceeds to step S80, in which a most probable walking path for current individuals is determined, by minimizing a cost along a walking route according to the generated energy distribution. The detailed description for step S80 may be the same as that of the walking path generation device 200.

Fig. 11 is a schematic diagram illustrating a flow process for a method for pedestrian walking path prediction according to a further embodiment of the present application. The steps S20～60 are the same as those discussed in reference to Fig. 9, and thus the detailed description thereof will be omitted herein.

At step S70, the energy distribution obtained at step S60 is transformed into a personalized energy distribution, in which a larger energy value denotes that a pedestrian cares more about these influence factors and is likely to walk a longer way to avoid close contact with these obstacles, and a smaller energy value means that the pedestrian is walking aggressively and cares less about obstacles. And then at step S80’， the most probable walking path for current individuals is determined according to the generated personalized energy distribution, which is the same as the operations for the walking path generation device 200’.

As will be appreciated by one skilled in the art, the present invention may be embodied as a system, method or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment and hardware aspects that may all generally be referred to herein as a “unit” ， “circuit， ” “module” or “system.” Much of the inventive functionality and many of the inventive principles when implemented, are best supported with or integrated circuits (ICs) , such as a digital signal processor and software therefore or application specific ICs. It is expected that one of ordinary skill, notwithstanding possibly significant effort and many design choices motivated by, for example, available time, current technology, and economic considerations, when guided by the concepts and principles disclosed herein will be readily capable of generating ICs with minimal experimentation. Therefore, in the interest of brevity and minimization of any risk of obscuring the principles and concepts according to the present invention, further discussion of such software and ICs, if any, will be limited to the essentials with respect to the principles and concepts used by the preferred embodiments.

In addition, the present invention may take the form of an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software. Furthermore, the present invention may take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code embodied in the medium. Fig. 12 illustrates a system 3000 for pedestrian walking path prediction from video frames. The system 3000 comprises a memory 3001 that stores executable components and a processor 3002, electrically coupled to the memory 3001 to execute the executable components to perform operations of the system 3000. The executable components may comprise: an energy distribution generation component 1201 for generating an energy distribution for the video frames, the generated energy distribution comprising a combination of at least one of a scene layout energy distribution for the first group of frames, a moving pedestrian distribution for the second group of frames, and a stationary group distribution for the third group of frames； and a walking path generation device 1202 for determining a most probable walking path for current individuals, by minimizing a energy cost along a walking route according to the generated energy distribution. Although they are not shown, the energy distribution generation component 1201 may comprise: a scene factor segmentation component that segments each of the video frames into the first, the second and the third groups of the frames； a scene layout map modeling component that receives one or more frames from the first group of frames and models the received frames into the scene layout energy distribution； a moving pedestrian map modeling component that locates all moving pedestrians from the second group of frames and models the located moving pedestrians into the moving pedestrian distribution； a stationary group map modeling component that detects all stationary groups from the third group of frames, and models the stationary pedestrians into the stationary group distribution； and a general energy distribution combination unit that combines the scene layout distribution, the moving pedestrian distribution, and the stationary group distribution, into the energy distribution. Since the functions for the components 1201～1202 are the same as those for the devices 100～200, respectively, the detailed descriptions thereof are omitted herein.

Fig. 13 illustrates a system 3000’ for pedestrian walking path prediction from video frames. The system 3000 comprises a memory 3001 that stores executable components and a processor 3002, electrically coupled to the memory 3001 to execute the executable components to perform operations of the system 3000. The executable components may comprise: an energy distribution generation component 1201 for generating an energy distribution for the video frames, the generated energy distribution comprising a combination of at least one of a scene layout energy distribution for the first group of frames, a moving pedestrian distribution for the second group of frames, and a stationary group distribution for the third group of frames； and a personalized energy distribution modeling component 1203 that transforms the energy distribution into a personalized energy distribution, in which a larger energy value denotes that a pedestrian is likely to walk a longer way to avoid close contact with the obstacles, and a smaller energy value means that the pedestrian is walking aggressively and cares less about the obstacles. The system 3000’ further comprises a walking path generation component 1202’ configured to calculate the most probable walking path for current individuals according to the generated personalized energy distribution. Since the functions for the

components

1201， 1202’ and 1203 are the same as those for the

devices

100， 300 and 200’， respectively， the detailed descriptions thereof are omitted herein.

Although the preferred examples of the present invention have been described, those skilled in the art can make variations or modifications to these examples upon knowing the basic inventive concept. The appended claims are intended to be considered as comprising the preferred examples and all the variations or modifications fell into the scope of the present invention.

Claims

A system for pedestrian walking path prediction from video frames, wherein, the video frames comprise at least one of a first group of frames including scene obstacles, a second group of frames including moving pedestrians, and a third group of frames including stationary pedestrians, and the system comprises:

an energy distribution generation device generating an energy distribution for the video frames, the generated energy distribution comprising a combination of at least one of a scene layout energy distribution for the first group of frames, a moving pedestrian distribution for the second group of frames, and a stationary group distribution for the third group of frames； and

a walking path generation device being electrically communicated with the energy distribution generation device, and determining a most probable walking path for current individuals, by minimizing an energy cost along a walking route according to the generated energy distribution.
The system according to claim 1, wherein the energy distribution generation device comprises:

a scene factor segmentation unit that segments each of the video frames into the first, the second and the third groups of the frames；

a scene layout map modeling unit that receives one or more frames from the first group of frames and models the received frames into the scene layout energy distribution；

a moving pedestrian map modeling unit that locates all moving pedestrians from the second group of frames and models the located moving pedestrians into the moving pedestrian distribution；

a stationary group map modeling unit that detects all stationary groups from the third group of frames, and models the stationary pedestrians into the stationary group distribution； and

an energy distribution combination unit that combines the scene, moving pedestrian, and stationary group distributions into the energy distribution.
The system according to claim 2, wherein the energy distribution combination unit multiplies the scene layout energy distribution, the moving pedestrian distribution, and the stationary group distribution together to form the energy distribution.
The system according to claim 2, wherein the scene layout map modeling unit models the received frames into the scene layout energy distribution by rule of:

setting energy values inside scene obstacle regions as 0, and

decreasing energy values at locations near the scene obstacles according to distances to boundaries of the scene obstacle.
The system according to claim 2, wherein the moving pedestrian map modeling unit models the located moving pedestrians into the moving pedestrian distribution by rule of:

setting energy values at the location of the current moving pedestrian as 0；

decreasing energy values at the locations near the moving pedestrians； and

accumulating the energy values together.
The system according to claim 2, wherein the stationary group map modeling unit models the stationary pedestrians into the stationary group distribution by rule of:

setting energy values inside a stationary group region as a fix value smaller than 1；

assigning sparse stationary groups with larger energy values in the stationary group distribution； and

accumulating the energy values together.
The system according to any one of claims 1-6, further comprising:

a personalized energy distribution modeling device that transforms the energy distribution into a personalized energy distribution, in which a larger energy value denotes that a pedestrian is likely to walk a longer way to avoid close contact with the obstacles, and a smaller energy value means that the pedestrian is walking aggressively and cares less about the obstacles.
The system according to claim 7, wherein the walking path generation device is further configured to calculate the most probable walking path for current individuals according to the generated personalized energy distribution.
A method for pedestrian walking path prediction from video frames, comprising:

obtaining the video frames；

segmenting the obtained video frames into at least one of a first group of frames including at least one scene obstacle, a second group of frames including moving pedestrians, and a third group of frames including stationary pedestrians；

generating, from the segmented video frames, an energy distribution for the video frames, the generated energy distribution comprises a combination of at least one of a scene layout energy distribution for the first group of frames, a moving pedestrian distribution for the second group of frames, and a stationary group distribution for the third group of frames； and

determining a most probable walking path for current individuals, by minimizing a cost along a walking route according to the generated energy distribution.
The method according to claim 9, wherein the generating comprises:

modeling one or more frames from the first group of frames into the scene layout energy distribution；

locating all moving pedestrians from the second group of frames and model the located moving pedestrians into the moving pedestrian distribution；

detecting all stationary groups from the third group of frames to model the stationary pedestrians into the stationary group distribution； and

combining the scene layout distribution, the moving pedestrian distribution, and the stationary group distribution into the energy distribution.
The method according to claim 10, wherein the scene layout energy distribution, the moving pedestrian distribution, and the stationary group distribution are multiplied together to form the energy distribution.
The method according to claim 10, wherein the scene layout energy distribution is modeled by rule of:

setting energy values inside scene obstacle regions as 0, and

decreasing energy values of locations near the scene obstacles according to distances to boundaries of the scene obstacle.
The method according to claim 10, wherein the moving pedestrian distribution is modeled by rule of:

setting energy values at the location of the current moving pedestrian as 0, decreasing energy values of the locations near the moving pedestrians, and

accumulating the energy values together.
The method according to claim 10, wherein the stationary group distribution is modeled by rule of:

setting energy values inside a stationary group region as a fix value smaller than 1；

assigning sparse stationary groups with larger energy values in the stationary group distribution, and

accumulating the energy values together.
The method according to any one of claims 9-14, further comprising:

transforming the energy distribution into a personalized energy distribution, in which a larger energy value denotes that a pedestrian is likely to walk a longer way to avoid close contact with the obstacles, and a smaller energy value means that the pedestrian is walking aggressively and cares less about obstacles.
The method according to claim 15, wherein the determining a most probable walking path further comprises:

determining the most probable walking path for current individuals from the generated personalized energy distribution.
A system for pedestrian walking path prediction from video frames, wherein, the video frames comprise at least one of a first group of frames including scene obstacles, a second group of frames including moving pedestrians, and a third group of frames including stationary pedestrians, and the system comprises:

a memory that stores executable components； and

a processor electrically coupled to the memory to execute the executable components to perform operations of the system, wherein, the executable components comprise:

an energy distribution generation component generating an energy distribution for the video frames, the generated energy distribution comprising a combination of at least one of a scene layout energy distribution for the first group of frames, a moving pedestrian distribution for the second group of frames, and a stationary group distribution for the third group of frames； and

a walking path generation component determining a most probable walking path for current individuals, by minimizing a energy cost along a walking route from the generated energy distribution.
The system according to claim 17, wherein the energy distribution generation component comprises:

a scene factor segmentation component that segments each of the video frames into the first, the second and the third groups of the frames；

a scene layout map modeling component that receives one or more frames from the first group of frames and models the received frames into the scene layout energy distribution；

a moving pedestrian map modeling component that locates all moving pedestrians from the second group of frames and models the located moving pedestrians into the moving pedestrian distribution；

a stationary group map modeling component that detects all stationary groups from the third group of frames, and models the stationary pedestrians into the stationary group distribution； and

a general energy distribution combination component that combines the scene layout distribution, the moving pedestrian distribution, and the stationary group distribution, into the energy distribution.
The system according to any one of claim 17 or 18, further comprising:

a personalized energy distribution modeling component that transforms the energy distribution into a personalized energy distribution, in which a larger energy value denotes that a pedestrian is likely to walk a longer way to avoid close contact with the obstacles, and a smaller energy value means that the pedestrian is walking aggressively and cares less about the obstacles.
The system according to claim 19, wherein the walking path generation component is further configured to calculate the most probable walking path for current individuals according to the generated personalized energy distribution.