IT201800003102A1 - A PROCEDURE TO FORECAST A TRAJECTORY AND A FRUSTUM OF VIEW, CORRESPONDING SYSTEM AND IT PRODUCT - Google Patents

Description

DESCRIPTION of the industrial invention entitled:

"A procedure for predicting a trajectory and a frustum of view, corresponding system and IT product"

TEXT OF THE DESCRIPTION

Field of the Invention

The present invention generally refers to the field of intelligent lighting, in particular to a process for predicting a trajectory and a frustum of view of a person, a corresponding computer system and product.

Throughout this description, reference will be made to various documents by reproducing in square brackets (eg, [X]) a number that identifies the document in a "List of Cited Documents" that appears at the end of the description.

Background of the Invention

Anticipating the trajectories that could occur in the future is important for various reasons: in artificial vision or computer vision, predicting the path helps modeling the dynamics for tracking targets

[40, 47, 48, 59] and understanding of behavior [3, 30, 33, 35, 47]; in robotics, autonomous systems should plan routes that will avoid collisions and be respectful of human proxemics [13, 21, 31, 36, 53, 62].

For self-driving systems, the state of the art route planning concepts consider only the current status and positions of the vehicles and pedestrians involved, which leads to an unnatural and artificial movement configuration or pattern of the entire traffic flow. The classical prediction approaches [38] adopted Kalman filters [29], linear regression models [37] or Gaussian [44, 45, 57, 58], autoregressive models [2] and time series analysis [43]. These approaches ignore human-to-human interactions, which instead play an important role.

The consideration of other pedestrians in the scene and their innate avoidance of collisions was initially experimented with [24]. This initial seed was further developed through [34], [35] and [40], which respectively introduced a theoretical data-driven, continuous, and game model. Particularly, these approaches successfully employ substantially suggestions to pursue prediction such as human-to-human interaction and the intended destination of people. More recent works encode human-to-human interactions in a "social" descriptor [4] or propose human attributes [60] for forecasting in crowds. More implicitly, other procedures [3, 55] incorporate proxemic reasoning into prediction by grouping hidden variables that represent the probable position of a pawn in a Long and Short Term Memory (LSTM). However, these previous works that extrapolate the movements detected by the observation of previous time intervals are not adequate to solve the task, since human behavior is driven by environmental influence as well as by the dynamics of simple laws of inertia.

Summary of the Invention

The present invention seeks to provide a process that predicts a trajectory of a person by considering an area of attention to visibility driven by the pose of the head, that is, within the person's attention cone. In particular, an object of the present invention is to provide a method which considers the pose of the head, together with the positional information, as a suggestion for carrying out the prediction. The present invention also seeks to provide a corresponding computer system and product.

According to a first aspect, the description provides a method which comprises the steps of receiving from at least one image sensor image signals of at least one person; detect, from the image signals, a two-dimensional position of at least one person; estimate, as a function of the image signals received by at least one image sensor, a pose of the head of at least one person; generate a frustum of sight of at least one person from the estimated head pose; enter the two-dimensional position and the frustum of view in a recurrent neural network; generate a predicted trajectory of movement of at least one person.

The frustum of view (or frustum of vision) is the region of space that can appear on a computer screen, thus representing roughly what the field of view is for a notional camera. The designation "frustum" refers to the cone, which represents the person's line of sight together with the typical opening angle of the human field of view.

The exact shape of the region covered by a view frustum can vary depending on e.g. of the considered optical system. In most applications it can be considered as a frustum of a rectangular pyramid.

Preferably, the process also comprises generating a predicted view frustum of at least one person. This allows you to reason about where people are likely to look, providing a fine granulated level of long-term prediction never before achieved in crowded scenarios. Preferably, the pose of the head of said at least one person is two-dimensional.

Preferably, the method also comprises passing the predicted trajectory of movement and the predicted view frustum to a panel control server.

Preferably, the method also comprises, the panel control server which responds to the predicted motion trajectory and the predicted view frustum, and which controls the panel to display and / or close a video content.

Preferably, the method also comprises passing the predicted trajectory and the predicted view frustum to a traffic control system.

Preferably, the method also comprises a traffic control system which responds to the predicted motion trajectory and the predicted view frustum, and which assigns a vehicle trajectory.

Preferably, the recurrent neural network is a Long and Short Term Memory (LSTM) network.

According to a further aspect, the description provides a system comprising:

- at least one image sensor for generating image signals of at least one person,

- a person detector and the head pose estimator joined to detect a two-dimensional position of at least one person from the image signals; estimate, as a function of the image signals received by at least one image sensor, a pose of the head of at least one person; and generate a frustum of sight of at least one person from the pose of the estimated head,

- a recurrent neural network coupled to the person detector and the joint head pose estimator, to process the two-dimensional position and the frustum of view of said at least one person and generate a predicted trajectory of movement of the at least one person.

Preferably, the recurrent neural network also generates a predicted view frustum of at least one person.

Preferably, the system further comprises a panel control server for responding to the predicted motion trajectory and predicted view frustum, and controls the panel to display video content.

Preferably, the system further comprises a traffic control system for responding to the predicted motion trajectory and the predicted view frustum, and assigns a vehicle trajectory.

Preferably, the recurrent neural network is a Long and Short Term Memory (LSTM) network.

In a still further aspect, the disclosure provides a non-transient computer-readable recording medium that stores a computer product, which, when executed by a processor, causes a computer to perform the above processes.

Brief Description of the Drawings

Embodiments are explained by way of

example referring to the attached drawings, in which:

Fig. 1a shows a graphical explanation of tracklet e

and anchor point vislet

Fig. 1b shows a graphical explanation of a

grouping of a visual frustum;

Fig. 1c shows a graphical explanation of angles for

correlation analysis;

Fig. 2a shows the analysis between the angle discrepancy ω between the pose of the head and the movement, the speed

regularized pedestrian and average errors of approaches

different on the UCY sequence;

Fig. 2b shows the correlation between the angle of movement β and the orientation angle of the head α when the

speed;

Fig. 3a shows the qualitative result of the MX-LSTM;

Fig. 3b shows a qualitative study of Ablation on

a Single MX-LSTM;

Fig. 4 shows an exemplary flow chart of

an embodiment;

Fig. 5 shows an exemplary flow chart of

another embodiment.

Detailed Description of the Invention

Example embodiments incorporating one

or more aspects of the invention are described and illustrated

in the drawings. It must be understood that they can be used

other embodiments and which can be carried out

structural or logical changes without departing

within the scope of the present invention. The following

detailed description, therefore, should not be considered in

a limiting sense. The same reference numerals are used to refer to the same or similar portions or components.

The present invention considers the pose of the head, in conjunction with positional information, as a suggestion for making the prediction. In particular, tracklets (sequences of coordinates (x; y)) and vislets, i.e., reference points indicating the panoramic orientation of the head, are the entrance to the new MiXing LSTM (MX-LSTM), a model based on LSTM that learns how tracklet and vislet flows are related, mixing them together in LSTM hidden state recursion by means of cross-flow complete covariance matrices, optimized during back-propagation. The MX-LSTM is able to encode how the movements of the head and the dynamics of people are connected. For example, it captures the fact that the rotation of the head towards a particular direction can anticipate a drift of trajectory with an acceleration (as in the case of a person who leaves a group after a conversation). This happens thanks to a new optimization of the LSTM parameters using a complete Gaussian covariance through a log-Cholesky parameterization in the back propagation, which ensures positive semidefinite matrices. Vislet information is also used to construct a scene context, i.e. where people are and how they move, via a shared state grouping as in [3, 55], which is further improved here using the head pose discarding the people that an individual cannot see.

The MX-LSTM also predicts head-on orientations, allowing you to reason about where people are likely to look, providing a fine granulated level of long-term prediction never before achieved in crowded scenarios.

In adopting standard protocols for trajectory prediction [3, 34, 40] and in using head pose information provided by a standard head pose estimator [32], the MX-LSTM defines the new state-deli art in both in the UCY sequences (Zara01, Zara02 and UCY) and in the dataset or dataset of the CentroCittà or TownCentre. In particular, the MX-LSTM has the ability to predict people when they move slowly, the Achilles heel of all the other approaches proposed so far.

Here, we present the MX-LSTM, able to jointly predict positions and orientations of an individual's head thanks to the presence of two streams of information: Tracklet and vislet.

Given a subject i, a tracklet (see Fig.1a) is formed by consecutive positions (x, y) on the ground plane,

while a vislet is made up of points

of anchorqqio <with> indicating a

reference point at a fixed distance r from the corresponding towards which the face is oriented (The spacing is not influential in this invention, and it can be any value; in this invention, for display purposes, we set it to 0.5). In practice, it is a fixed dimension vector that originates from whose direction implicitly indicates the panoramic angle

of the head. In principle, it would be possible to directly code the orientation of the head with an angle. We prefer the representation of the vislet because it does not show discontinuity (between 360 ° and 0 °) and because it is closer to the representation of the position (x, y) and therefore more suitable for the vislet-position interaction.

In the prediction documentation [3, 53, 59] it is assumed that the prediction follows an "observation" period in which basic truth or ground truth data is fed into the machine. Here, the observation tracklets and vislets are fed into the MX-LSTM, which mixes the two streams together to understand their relationship, providing a joint prediction. In the experiments we evaluate the cases where the past vislets are of groundtruth, but also the case “in context”, where the head pose is provided by a real head detector. In this way, the MX-LSTM will not require any additional annotation compared to previous approaches.

A single MX-LSTM is instantiated for each i-pedestrian, accepting tracklets and vislets with two separate <embedding or embedding functions:>

where the embedding function ∅ consists of a linear projection through the embedding weights Wx and Wα into a D-dimensional vector, multiplied by a non-linearity RELU, where D is the dimension of the hidden space.

The social grouping introduced in [3] is an effective way to let LSTM capture the fact of how people move in a crowded scene while avoiding collisions. This work considers an isotropic area of interest around the single pedestrian, in which the hidden states of the neighbors are considered, including those behind the pedestrian. According to our invention, we improve this module by using the vislet information by selecting which individuals to consider, building an attention view frustum (VFOA), i.e. a triangle that originates from aligned with and with an opening given by the angle y and a depth d; these parameters were learned through cross-validation on the training partition of the TownCentre dataset.

Our social grouping of the frustum of view is a No X No X D tensor, in which the space around the pawn is divided into a grid of No X No cells as in [3], in which the VFOA is positioned, which acts as the new region of interest where people have to be considered. Grouping occurs as follows:

where the indices m and n pass over the No X No grid and condition j ∈ VFOAi is satisfied when subject j is in subject i's VFOA. The grouping vector is subsequently incorporated into a D-dimensional vector via

Finally, the MX-LSTM recursion equation is

In principle (but we will change the wording in the following part), the hidden state is hardened to contain the parameters of a multivariate distribution

Four-dimensional Gaussian as follows:

vectorized of In practice

contain the covariances between the (x, y) coordinate distributions of tracklets and vislets. The distribution is then sampled to generate the joint prediction of tracklet and vislet points

In other words, it is able to predict in the

same time trajectories and poses of the head.

The weight parameters of the LSTM are found by minimizing the multivariate Gaussian log of likelihood for the i-th trajectory

where Tobs is the time frame until the ground truth data is observed through the LSTM, while Tobs + 1, Tpred are the time frames for which prediction is required. The loss of Eq. 7 is minimized on all training sequences, and to prevent overfitting we include a regularization term l2

The optimization provides the weight matrices of the MX-LSTM, which in turn produce the set of Gaussian parameters, including the complete covariance ∑ The latter is necessary to strengthen the LSTM in coding the relations between the coordinate distributions ( x, y) of tracklets and vislets.

In general, the estimation of a complete covariance matrix through the optimization of an objective function (such as the log likelihood of Eq. (7)) is a difficult numerical problem [41], since it must be ensured that the resulting estimate si an appropriate covariance, that is, a positive semi-definite matrix (p.s.d.).

LSTMs involving log-likelihood losses on Gaussian distributions have so far been limited to two dimensions for simple Gaussian distributions [3] or a mixture of Gaussian distributions [19], in which the 2 x 2 covariance matrices have been obtained by simply optimizing l scalar correlation index

which becomes the covariance term of ∑ with

[19]. In the case of higher dimensional problems, pair-level correlation terms cannot be optimized and used to construct ∑, since the optimization process for each correlation term is mutually independent, while positive-definite is a simultaneous limitation on multiple variables [42]. This lack of coordination provides matrices far from being s.d.p., which in turn require correction procedures by projection into the s.d.p. closer using, for example, a cost function based on the Frobenious norm [7, 25]. These procedures are expensive [41], and difficult to incorporate into the optimization process [12], especially in the case of LSTM, where non-linearities due to embedding weights make the analytical derivation difficult to formulate. To date, no loss of LSTM has involved complete covariances of size> 2.

Our solution involves a non-limited optimization, where an appropriate parameterization of the variables to be learned strengthens the positive semidefinite limitation, which is easier to express, greatly improving the convergence properties of the optimization algorithm.

In practice, we consider the Choleski family of parametrizations [42]: let ∑ denote a positive covariance matrix defined n × n (in our case, n = 4). Since ∑ is symmetric, only n (n 1) / 2 parameters are required to represent it. The Choleski factorization is given by:

∑ = L <T> L, (8)

where L is an upper triangular matrix n x n. In practice, the optimization process would focus on finding the n (n 1) / 2 distinct scalar values for L, which subsequently serve to solve the covariance given Eq. (8). A problem with Cholesky factorization is its non-uniqueness: any matrix obtained by multiplying a subset of the rows of L by -1 is valid; as a consequence, the non-uniqueness of the solution makes the optimization process difficult to converge. To make L unique, its diagonal elements must all be positive. To this end, the Cholesky Logarithmic parameterization [42] assumes that the values found by the optimizer of the main covariance diagonal are the logarithm of the values of L: Formally, the values found through the optimizer can be written as

In practice, after the estimation of the parameters Wx, Wa WH, WLSTM, WO, or the values of θL are extracted through

(9)

torized by θL. Subsequently, i

diagonal values of θL are exponentiated to form L and obtain ∑ through Eq. (8).

To date, no quantitative studies have focused on how knowledge of head pose affects trajectory prediction. Here, we show a preliminary analysis of common prediction datasets with an emphasis on the head pose that motivated the design of the MX-LSTM.

In particular, we focus on the dataset UCY [34], composed of the sequences Zara01, Zara02, and UCY, which provides the annotations for the panoramic angle of the pose of the head of all pedestrians. We also consider the Town Center dataset [6], where we manually annotated the pose of the head, using the same annotation protocol as [34]. We discover the following facts (Here the analysis on the UCY sequence is presented, which is similar to what we have observed on the other sequences):

1) People often don't watch their steps. To demonstrate this fact, for each single trajectory composed of T plots (omitting the single indices), we calculate all the αt, βt, and ωt of Fig.1c. Αt is the panoramic angle of the pose of the head with respect to a given reference system; similarly, βt is the angle of movement, and ωt shows the discrepancy between the two. For each single trajectory, we calculate the average On the multiple y axis Fig. 2a, we show the value ω (in degrees) of all the sequences, in an increasing order. From the figure, we omit those sequences where the velocity is below 0.45m / sec .: in those cases the individual is substantially stationary and the motion vector xt + 1 - xt carries little if any meaning, and consequently the angle βt cannot be considered. The ω value ranges from 0.02 ° to 72 °. We conclude that in 25% of the video sequences the misalignment between the pose of the head and the direction of the step is greater than 20 °.

2) The pose of the head and the movements are (statistically) correlated; On the same figure, we refer to the speed curve, where each y-point provides the average speed of the trajectory ordered on the axis. For purposes of readability, the curve was smoothed with a moving average filter of size 10. As shown, there is an inverse proportionality relationship between the speed of ω and the pedestrian: the alignment between the head towards the direction of movement is higher when the speed is higher; when the person slows down the head pose is noticeably misaligned. The relationship is statistically significant: we consider Pearson's circular correlation coefficient [28] between the angles αt and βt, calculated on all the plots of the sequences considered for that figure. On the whole data, the correlation is 0.83 (p-value <0.01). We also verify how the correlation changes with the velocity: Fig. 2b shows the correlation values with respect to the velocity, calculated by grouping the angles αt and βt around a certain velocity value; in particular, each correlation value to the speed τ was calculated considering all the samples in the interval [t - 0.01R, τ 0.01R], where R is the entire speed range. All reported values have statistical significance (p-value <0.01). The plot clearly shows that the correlation is lower at low speeds, where the discrepancy between the angles αt and βt is generally higher. The challenge here is to check whether this discrepancy can be learned via the MX-LSTM to improve prediction. More intriguingly, the MX-LSTM should learn how these relationships evolve over time, which has not yet been verified, since the analysis carried out so far considers each instant of time as independent of each other.

3) The prediction errors are generally higher when the pedestrian speed is lower; In Fig. 2 the Mean Average Displacement (MAD) [40] of the following approaches are reported: SF [59], LTA [53], LSTM Vanilla and LSTM Sociale [3], together with our MX approach -LSTM. In general, slower speeds lead to more errors, as when people walk very slowly their behavior becomes less predictable, due to physical reasons (less inertia) but also behavioral (people who walk slowly are usually involved in other activities, such as talking to other people, looking around). Conversely, it is shown here that the MX-LSTM performs very well even at lower speeds, hitting very close to zero errors with static people.

In summary, head pose is related to movement, especially when people are moving quickly. When people move slowly, the correlation is weaker but significant, prediction errors are greater, and the pose of the head is noticeably out of alignment with the movement. These facts justify and motivate our goal with the MX-LSTM, to capture the information on the pose of the head in conjunction with the movement and use it for better prediction.

Next, both quantitative and qualitative experiments are presented here. The quantitative results validate the proposed MX-LSTM model, setting the new state of the art for trajectory prediction; results are also provided for an ablation study showing the importance of the different parts of the MX-LSTM. Finally, we present the very first results on the prediction of the head pose. The qualitative results reveal the interaction between tracklet and vislet learned from the MX-LSTM.

Our model is evaluated against all published approaches that have made their code publicly available: Social Strength (SF) model [59], Linear Path Avoidance (LTA) [40], LSTM Vanilla and LSTM Social (S -LSTM) [3].

The experiments follow the widely used evaluation protocol of [40], in which the algorithm first observes 8 ground-truth (GT) "observation" plots of a trajectory, which predict the following 12. For the three UCY Sequences are Three models were trained: for each we used two sequences as training data and then we tested the third sequence. For the Town Center dataset the model was trained and tested on the respective sets provided. The grid for social grouping (Eq. (3)) has cells No × No with No = 32. The opening angle of the view frustum has been cross-validated on the TownCenter training partition and kept fixed for the remaining evidence (y = 40 °), while the depth d is simply linked by the social grouping grid. The trajectory prediction performance is analyzed with the Mean Average Displacement (MAD) error (Euclidean distance between predicted points and GT, averaged over the sequence), and Final Average Displacement (FAD) error (distance between last point predicted and corresponding point GT) [40].

The results are shown in Table 1. The MX-LSTM exceeds the state of the art procedures in each individual sequence and with both metrics, with an average improvement of 32.7%. The highest relative gain is achieved in the Zara02 dataset, where complex non-linear paths are mostly caused by standing talkgroups and people walking close to them, avoiding collisions. People who slow down and look at shop windows also pose a challenge. As shown in Fig. 2, slow moving and interacting pedestrians cause problems for competing processes, while the MX-LSTM clearly overcomes these disadvantages by indicating a better model.

Please note that different processes rely on different input data: both the SF and LTA require the destination point of each individual, while the SF also requires annotations regarding social groups; MX-LSTM requires each individual's head pose for the first 8 plots, but this can be estimated via a head pose estimator. This motivates our next experiment: we automatically estimate the bounding box of the head given the positions of the feet on the ground plane, assuming on the plane of the floor, assuming that people are 1.80m tall. Next, we apply the head pose estimator of [32] which provides continuous angles that can be used as an input to our approach now called “MX-LSTM-HPE”. As shown by the scores in Table 1, the MX-LSTM-HPE is not affected by small errors in the pose of the head entered, with an average decrease in performance of only 5%. Note that the MX-LSTM-HPE still outperforms all competing processes on all datasets even with the noisy estimated head pose information.

How accurate should the head pose estimate be, in order for the MX-LSTM-HPE to perform convincingly, for example exceeding the Social LSTM? We answer this question by corrupting the true estimate of the head pose with additive Gaussian noise where

is the correct head pose and standard deviation.

The MX-LSTM-HPE outperforms the social LSTM by up to 24◦ noise.

In addition to the models in the documentation, we test three variations of the MX-LSTM to capture the net contributions of the different parts that characterize our approach.

Block Diagonal MX-LSTM (BD-MX-LSTM): it serves as evidence of the importance of estimating complete covariances to understand the interaction between tracklet and vislet. Basically, the approach estimates two two-dimensional covariances (The 2 x 2 covariance is estimated using two variances σ1, σ2 and a correlation term p as presented in [19] Eq. (24) and (25).) ∑x and ∑a for trajectory and vislet modeling respectively, without capturing cross-flow covariances. The equations that differ from the MX-LSTM w.r.t. will be provided as supplemental material.

MX-LSTM Without Frustum: this variation of the MX-LSTM uses the social grouping as in [3], in which the area of interest where the hidden states of people are grouped in the social tensor all around the individual. In other words, no frustum selecting the people to be considered is used here.

Single MX-LSTM: In this case, no social grouping is taken into account, so the embedding operation of Eq. (4) is absent, and the weight matrix W H disappears. In practice, this variant learns independent patterns for each person, each considering the tracklet and vislet points.

Table 1, last three columns, reports numerical results for all MX-LSTM simplifications on all datasets. The main facts that emerge are: 1) the highest variations are with the Zara02 sequence, where the MX-LSTM doubles the performance of the worst approach (MX-LSTM Single); 2) the worst performances are in general the Single MX-LSTM, which shows that social reasoning is in fact necessary; 3) social reasoning is systematically improved with the help of the vislet-based view frustum; 4) the complete covariance estimate plays a role in decreasing the error that is already small with the adoption of vislets.

Table 1. Mean and Final Average Displacement errors (in meters) for all proceedings on all datasets. The first 5 columns are the comparative procedures and our proposed model trained and tested with GT annotations. The MX-LSTM-HPE is our tested model with the output of a real head pose estimator [32]. The last 3 columns are variations of our trained and tested approach on GT annotations.

Table 2. Average angular error (in degrees) for the state of the art head pose estimator [32], and the MX-LSTM model fed with GT Annotations and estimated values (MX-LSTM-HPE).

In summarizing the results so far, having vislets as input allows to definitively increase the trajectory prediction performance, even if the vislets are estimated with noise. Vislets should be used to understand social interactions with social grouping by constructing a view frustum that tells which people are currently being observed by each individual. All these features are efficiently performed through the MX-LSTM: in fact the training time is the same as when having a social grouping LSTM.

As done with the trajectories, we also provide a prediction of each individual's head pose at each plot which is a distinctive attribute of our process. We evaluate the performance of this estimate in terms of mean angular error eα, which is the absolute mean difference between the estimated pose (angle αt in Fig. 1c) and the annotated GT.

Table 2 shows numerical results of the static head pose estimator [32] (HPE), the MX-LSTM using GT head pose, and the MX-LSTM powered with the HPE output during the observation period (MX-LSTM -HPE). In all cases our prediction output is comparable to that of the HPE, but in our case we don't use appearance hints - i.e. we don't look at the images at all. In the case of Zara01, the MX-LSTM is even better than the static prediction which demonstrates the prediction power of our model. In our opinion this is due to the fact that in this sequence the trajectories are mostly very linear and fast, and the heads are mostly aligned with the direction of movement. When we provide estimates to the MX-LSTM model during the observation period, the angular error increases, as expected. Despite this, the error is surprisingly limited.

Fig. 3 shows qualitative results on the Zara02 dataset, which was shown as the most complex scenario in quantitative experiments. Fig. 3a presents MX-LSTM results: a group scenario is taken into consideration, with the focus on the girl in the lower left corner. In the left column, the ground-truth prediction vislets show that the girl converses with the group members, with near-zero movement and the panoramic angle of her head swinging. The behavior of the S-LSTM, which incorrectly predicts the girl leaving the group. This error confirms the problem of concurrent procedures in predicting the movement of slowly moving or static people as discussed above, and further confirmed by the results of quantitative experiments. In the central column, the observation sequence provided to the MX-LSTM (quasi static with oscillating vislets) is shown. The related prediction shows oscillating vislets, and almost no movement, confirming that the MX-LSTM has learned this particular social behavior. If we provide the MX-LSTM with an artificial observation sequence with no trajectory but with the vislets oriented to the west (third column), where no person is present, the MX-LSTM predicts a trajectory starting from the group.

The two lines of Fig. 3b analyze the Single MX-LSTM, in which no social grouping is taken into consideration. Thus, here each pedestrian is unaffected by the surrounding people, and the relationship between the tracklets and the vislets in the prediction can be observed without any confounding factors. Fig. 3b first row shows three situations in which the vislets of the observation sequence are artificially made pointing north, resulting not aligned with the trajectory. In this case the MX-LSTM Single predicts a slowing trajectory that deviates north, especially in the second and third figures. If the observation has the legal vislets (barely visible since they are aligned with the trajectory), the resulting trajectory has a different behavior, closer to the GT. The second row is similar, with the realized observation vislets pointing south. The prediction with the modified vislets is also shown. The only difference is in the lower left figure: here the observation vislets pointing south are in agreement with the movement, such that the resulting predicted trajectory does not decelerate as in the other cases, but accelerates southwards.

Fig. 4 shows an exemplary flow chart of one embodiment. As illustrated in Fig. 4, at least one installed camera 100 in a shopping mall captures the images and sends them to a detector 101. The detector is a joint people detector and head pose estimator, as described in [23], which processes the tagged images and outputs the spatial positions and view frustums of at least one person in the images. A person's spatial position and frustum of view, such as two entrances, enter the LSTM 102 system, or call it an MX-LSTM system. The LSTM 102 system processes the signals and predicts the person's future trajectory and frustum of sight. Further, the aforementioned information of a person's position and view frustum is passed to the advertising control panel of the server 103. It uses future information of where a person will be and where he will look to change the advertising panel 104 at runtime. For example, the monitor is only turned on when a person will look at it. Thus, it saves energy. Combined with complex face detection technology, this system could also be used to select the appropriate content to be presented on the screen for a certain person.

Fig. 5 shows an exemplary flow chart of another embodiment. Similarly to the above embodiment, the camera 203 takes images of at least one person, the joint person detector and head pose estimator 202 emit the spatial position and orientation information of the head, and the LSTM 201 uses this information and predicts the future position and frustum of the person's view. Based on future information where a person will be, a traffic flow system 200 for autonomous vehicles will adjust the vehicle's path accordingly. In this embodiment, the predicted trajectories can be used to avoid potential collisions with pedestrians.

As main contributions, in this invention: We show that trajectory prediction can be greatly improved by considering head pose estimates; We propose a new LSTM architecture, the MX-LSTM, which exploits positional information (tracklet) and orientation (vislet) thanks to an optimization of d-varied Gaussian parameters including complete covariances with d> 2; We motivate the need for MX-LSTM which shows that head poses are related to trajectories, even at low speeds, where most prediction approaches fail; We define a new type of social grouping, in the sense of [3, 55], exploiting the information of vislet; we define forecast results of the state of the art on different datasets; We present the results of the MX-LSTM of Head Pose Prediction, which show new long-term behavioral analysis capabilities.

List of documents cited

[1] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In ICML, 2004. 2

[2] H. Akaike. Fitting autoregressive models for prediction. Annals of the institute of Statistical Mathematics, 21 (1): 243–247, 1969. 2

[3] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese. Social LSTM: Human trajectory prediction in crowded spaces. In CVPR, 2016. 1, 2, 3, 4, 5, 6, 7

[4] A. Alahi, V. Ramanathan, and L. Fei-Fei. Sociallyaware large-scale crowd forecasting. In CVPR, 2014. 2

[5] S. O. Ba and J.-M. Odobez. A probabilistic framework for joint head tracking and pose estimation. In ICPR, 2004.

2

[6] B. Benfold and I. Reid. Stable multi-target tracking in realtime surveillance video. In CVPR, 2011. 4

[7] S. Boyd and L. Xiao. Least-squares covariance matrix adjustment. SIAM Journal on Matrix Analysis and Applications, 27 (2): 532–546, 2005. 4

[8] J. F. Caminada and W. J. M. van Bommel. Philips engineering report 43, 1980. 1, 2

[9] C. Chen and J.-M. Odobez. We are not contortionists: Coupled adaptive learning for head and body orientation estimation in surveillance video. In CVPR, 2012. 1

[10] H. Coskun, F. Achilles, R. Di Pietro, N. Navab, and F. Tombari. Long short-term memory kalman filters: Recurrent neural estimators for pose regularization. In ICCV, 2017. 2

[11] N. Davoudian and P. Raynham. What do pedestrians look at at night? Lighting Research and Technology, 44 (4): 438-448, 2012. 1, 2

[12] J. E. Dennis Jr and R. B. Schnabel. Numerical methods for unconstrained optimization and nonlinear equations. SIAM, 1996. 4

[13] A. D. Dragan, N. D. Ratliff, and S. S. Srinivasa. Manipulation planning with goal sets using constrained trajectory optimization. In ICRA, 2011. 1, 2

[14] S. Fotios, J. Uttley, C. Cheal, and N. Hara. Using eyetracking to identify pedestrians' critical visual tasks, Part 1. Dual task approach. Lighting Research & Technology, 47 (2): 133–148, 2015. 1, 2

[15] S. Fotios, J. Uttley, and B. Yang. Using eye-tracking to identify pedestrians ’critical visual tasks. part 2. fixation on pedestrians. Lighting Research & Technology, 47 (2): 149-160, 2015. 1, 2

[16] T. Foulsham, E. Walker, and A. Kingstone. The where, what and when of gaze allocation in the lab and the natural environment. Vision research, 51 (17): 1920-1931, 2011. 1, 2

[17] N. Gourier, J. Maisonnasse, D. Hall, and J. L. Crowley. Head pose estimation on low resolution images. In CLEAR, 2006.2

[18] A. Graves. Supervised sequence labeling with recurrent neural networks, volume 385. Springer, 2012. 2

[19] A. Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv: 1308.0850, 2013. 2, 4, 6

[20] K. Gregor, I. Danihelka, A. Graves, D. J. Rezende, and D. Wierstra. DRAW: A recurrent neural network for image generation. arXiv preprint arXiv: 1502.04623, 2015. 2

[21] E. T. Hall. The hidden dimension. Doubleday & Co, 1966. 1

[22] Y. D. B. Z. Hang Su, Jun Zhu. Forecast the plausible paths in crowd scenes. In IJCAI, 2017. 1, 2

[23] I. Hasan, T. Tsesmelis, F. Galasso, A. Del Bue, and M. Cristani. Tiny head pose classification by bodily cues. In ICIP, 2017. 2

[24] D. Helbing and P. Molnar. Social force model for. Physical review E, 51 (5): 4282, 1995. 2

[25] N. J. Higham. Computing a nearest symmetric positive semidefinite matrix. Linear algebra and its applications, 103: 103–118, 1988. 4

[26] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9 (8): 1735–1780, 1997. 1, 2

[27] J. Intriligator and P. Cavanagh. The spatial resolution of visual attention. Cognitive psychology, 43 (3): 171–216, 2001. 1, 2

[28] S. R. Jammalamadaka and A. Sengupta. Topics in circular statistics, volume 5. World Scientific, 2001. 5

[29] R. E. Kalman et al. A new approach to linear filtering and prediction problems. ASME Journal of Basic Engineering, 1960. 2

[30] K. Kitani, B. Ziebart, J. Bagnell, and M. Hebert.

Activity forecasting. In ECCV, 2012. 1, 2

[31] M. Kuderer, H. Kretzschmar, C. Sprunk, and W. Burgard. Feature-based prediction of trajectories for socially compliant navigation. In Robotics: science and systems, 2012.

1

[32] D. Lee, M.-H. Yang, and S. Oh. Fast and accurate test pose estimation via random projection forests. In ICCV, 2015. 1, 2, 6, 7

[33] N. Lee and K. M. Kitani. Predicting wide receiver trajectories in american football. In WACV, 2016. 1, 2

[34] A. Lerner, Y. Chrysanthou, and D. Lischinski. Crowds by example. In Computer Graphics Forum, 2007. 1, 2, 4, 5

[35] W.-C. But, D.-A. Huang, N. Lee, and K. M. Kitani. Forecasting interactive dynamics of pedestrians with fictitious play. In CVPR, 2017. 1, 2

[36] J. Mainprice, R. Hayne, and D. Berenson. Goal set inverse optimal control and iterative replanning for predicting human reaching motions in shared workspaces. IEEE Trans. on Robotics, 32 (4): 897–908, 2016. 1, 2

[37] P. McCullagh and J. A. Nelder. Generalized linear models, no. 37 in monograph on statistics and applied probability, 1989. 2

[38] B. T. Morris and M. M. Trivedi. A survey of visionbased trajectory learning and analysis for surveillance. IEEE Trans. on Circuits and Systems for Video Technology, 18 (8): 1114-1127, 2008. 2

[39] A. E. Patla and J. N. Vickers. How far ahead do we look when required to step on specific locations in the travel path during locomotion? Experimental brain research, 148 (1): 133–138, 2003. 1, 2

[40] S. Pellegrini, A. Ess, K. Schindler, and L. Van Gool. You’ll never walk alone: Modeling social behavior for multitarget tracking. In ICCV, 2009. 1, 2, 5, 6, 7

[41] J. C. Pinheiro and D. M. Bates. Unconstrained parametrizations for variance-covariance matrices. Statistics and Computing, 6 (3): 289–296, 1996. 2, 4

[42] M. Pourahmadi. Covariance estimation: The GLM and regularization perspectives. Statistical Science, pages 369– 387, 2011. 4

[43] M. B. Priestley. Spectral analysis and time series. Academic press, 1981. 2

[44] J. Qui ̃nonero-Candela and C. E. Rasmussen. A unifying view of sparse approximate gaussian process regression. Journal of Machine Learning Research, 6 (12): 1939–1959, 2005.

2

[45] C. E. Rasmussen. Gaussian processes for machine learning. In Adaptive Computation and Machine Learning, 2006.

2

[46] N. M. Robertson and I. D. Reid. Estimating gaze direction from low-resolution faces in video. In ECCV, 2006.

1, 2

[47] A. Robicquet, A. Sadeghian, A. Alahi, and S. Savarese. Learning social etiquette: Human trajectory understanding in crowded scenes. In ECCV, 2016. 1

[48] A. Sadeghian, A. Alahi, and S. Savarese. Tracking the untrackable: Learning to track multiple cues with longterm dependencies. arXiv preprint arXiv: 1701.01909, 2017. 1, 2

[49] R. Stiefelhagen, M. Finke, J. Yang, and A. Waibel. From gaze to focus of attention. In VISUAL, 1999. 1, 2

[50] H. Su, Y. Dong, J. Zhu, H. Ling, and B. Zhang. Crowd scene understanding with coherent recurrent neural networks. In IJCAI, 2016. 1, 2

[51] L. Sun, Z. Yan, S. M. Mellado, M. Hanheide, and T.

Duckett. 3DOF pedestrian trajectory prediction learned from long-term autonomous mobile robot deployment data. arXiv preprint arXiv: 1710.00126, 2017. 1

[52] Fr Tosato, M. Spera, M. Cristani, and V. Murino. Characterizing humans on riemannian manifolds. IEEE TPAMI, 35 (8): 1972–1984, 2013. 2

[53] P. Trautman and A. Krause. Unfreezing the robot: Navigation in dense, interacting crowds. In IROS, 2010. 1, 3, 5

[54] P. Vansteenkiste, G. Cardon, E. D’Hondt, R. Philippaerts, and M. Lenoir. The visual control of bicycle steering: The effects of speed and path width. Accident Analysis & Prevention, 51: 222–227, 2013. 1, 2

[55] D. Varshneya and G. Srinivasaraghavan. Human trajectory prediction using spatially aware deep attention models. In NIPS, 2017. 1, 2

[56] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In CVPR, 2015.

2

[57] J. M. Wang, D. J. Fleet, and A. Hertzmann. Gaussian process dynamical models for human motion. IEEE TPAMI, 30 (2): 283–298, 2008. 2

[58] C. K. I. Williams. Prediction with gaussian processes: From linear regression to linear prediction and beyond. In Learning in graphical models, pages 599–621. Springer, 1998. 2

[59] K. Yamaguchi, A. C. Berg, L. E. Ortiz, and T. L. Berg. Who are you with and where are you going? In CVPR, 2011. 1, 3, 5, 6, 7

[60] S. Yi, H. Li, and X. Wang. Understanding pedestrian behaviors from stationary crowd groups. In CVPR, 2015. 2

[61] B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey.

Maximum entropy inverse reinforcement learning. In AAAI, 2008. 2

[62] B. D. Ziebart, N. Ratliff, G. Gallagher, C. Mertz, K. Peterson, J. A. Bagnell, M. Hebert, A. K. Dey, and S. Srinivasa. Planning-based prediction for pedestrians. In IROS, 2009. 1

Claims (13)

CLAIMS 1. Process comprising - receiving from at least one image sensor image signals of at least one person, - detect, from the image signals, a two-dimensional position of at least one person, - estimate, as a function of the image signals received by at least one image sensor, a pose of the head of at least one person, - generate a frustum of sight of at least one person from the estimated head pose, - enter the two-dimensional position and the frustum of view in a recurrent neural network, - generate a predicted trajectory of movement of at least one person. 2. A method according to claim 1, further comprising, - generate a predicted frustum of view of at least one person. A method according to claim 2, further comprising passing the predicted trajectory of motion and the predicted view frustum to a panel control server. The method of claim 3, further comprising, the panel control server responding to the predicted motion trajectory and the predicted view frustum, and controlling the panel to display and / or close a video content. A method according to claim 2, further comprising passing the predicted trajectory of movement and the predicted view frustum to a traffic control system. The method of claim 5, further comprising the traffic control system which responds to the predicted motion trajectory and the predicted view frustum, and which assigns a vehicle trajectory. Method according to any of claims 1 to 6, wherein the recurrent neural network is a Long and Short Term Memory (LSTM) network. 8. System comprising: - at least one image sensor for generating image signals of at least one person, - a person detector and the head pose estimator joined to detect a two-dimensional position of at least one person from the image signals; estimate, as a function of the image signals received by at least one image sensor, a pose of the head of at least one person; and generate a frustum of sight of at least one person from the pose of the estimated head, - a recurrent neural network coupled to the person detector and the joint head pose estimator, to process the two-dimensional position and the frustum of view of said at least one person and generate a predicted trajectory of movement of the at least one person. 9. System according to claim 8, in which the recurrent neural network also generates a predicted view frustum of at least one person. A system according to claim 9, further comprising a panel control server which responds to the predicted motion trajectory and the predicted view frustum, and which controls the panel for displaying video content. The system according to claim 9, further comprising a traffic control system for responding to the predicted motion trajectory and the predicted view frustum, and assigning a vehicle trajectory. System according to any of claims 8-11, wherein the recurrent neural network is a Long and Short Term Memory (LSTM) network. 13. Non-transient computer readable recording medium that stores a computer product, which, when executed by a processor, causes a computer to perform the method according to any of claims 1 to 7.