KR20210130072A - Method and apparatus of processing image - Google Patents

Abstract

An image processing method according to an embodiment includes defining relationships between objects based on the feature vectors of the objects whose motion are to be predicted in the image of a first viewpoint, estimating a dynamic interaction between the entities at a first time point based on the relationships between the objects based on the estimated dynamic interaction, predicting the movement of the objects that change at a second time point, and outputting a result reflecting the predicted motion at the second time point.

Description

Image processing method and apparatus {METHOD AND APPARATUS OF PROCESSING IMAGE}

The following embodiments relate to an image processing method and apparatus.

For tasks such as forecasting, it is important to understand the interactions between entities such as, for example, the joints of the human body, players on a sports team, and the like. However, interactions between individuals are not generally observed and are not easy to quantify. In particular, it is very difficult to reflect such changes over time in the interactions between entities because the relationship between entities generally changes over time.

The above-mentioned background art is possessed or acquired by the inventor in the process of deriving the disclosure of the present application, and it cannot be said that it is necessarily known technology disclosed to the general public prior to the present application.

According to an embodiment, an image processing method includes: defining relationships between entities based on feature vectors of entities of a target whose motion is to be predicted in an image of a first view; estimating a dynamic interaction between the entities at the first time point based on the relationships between the entities; estimating the movement of the objects changed at a second time point based on the estimated dynamic interaction; and outputting a result reflecting the predicted motion at the second time point.

The relationships between the entities apply to the connection relationships between the entities, the positions of the entities, the postures of the entities, the moving directions of the entities, the moving speed of the entities, the trajectory of the entities, and the entities. It may be determined based on at least one of a rule, a movement pattern of the entities according to the rule, a law applied to the entities, and a movement pattern of the entities according to the rule.

The defining of the relationships between the entities may include applying the feature vector to a Graph Neural Network (GNN) including nodes corresponding to the entities and edges corresponding to the relations between the entities. The method may include generating hidden state information corresponding to relationships between the entities at a first time point.

The graph neural network may include a fully-connected graph neural network for generating hidden state information corresponding to states of relationships between pairs of the entities based on the feature vector.

The estimating of the dynamic interaction may include: generating prior information corresponding to the entities based on the hidden state information; generating posterior information predicted to correspond to the entities based on the prior information and the hidden state information; and generating a latent variable corresponding to a dynamic interaction between the entities based on the prior information and the post-information.

The prior information may be determined based on a past history of relationships between the entities up to a time point prior to the first time point, and feature vectors of the entities input up to the first time point.

The generating of the dictionary information may include generating the dictionary information by transferring the hidden state information as forward state information to a forward long short-term memory (LSTM).

The generating of the post-information may include generating the post-information by transmitting the prior information and the hidden state information as backward state information to a backward LSTM.

The generating of the latent variable may include: sampling a result of combining the prior information and the post-information; and generating a latent variable corresponding to a dynamic interaction between the entities at the first time point based on the sampling result.

The generating of the latent variable may include optimizing the latent variable based on the prior information.

The objects of the subject may include at least one of body parts of a single user, joints of a single user, multiple pedestrians, multiple vehicles, and athletes of a sports team.

Predicting the movement of the objects may include predicting the movement of the object changed at the second time point by decoding the estimated dynamic interaction.

The outputting the result of reflecting the predicted motion may include: processing the objects included in the image of the first viewpoint into an image of a second viewpoint by reflecting the predicted motion; and outputting the image of the second viewpoint.

The outputting the result of reflecting the predicted motion may include: processing the objects included in the image of the first viewpoint into an image of a second viewpoint by reflecting the predicted motion; recognizing whether a dangerous situation has occurred based on the image of the second viewpoint; and outputting an alarm corresponding to the dangerous situation.

The image processing method may further include determining the objects of the object for which the motion is to be predicted.

According to an embodiment, an image processing apparatus includes: a communication interface for receiving an image of a first view including objects of a target for which a motion is to be predicted; extracting feature vectors of the entities from the image of the first viewpoint, and estimating dynamic interactions between the entities at the first viewpoint based on relationships between the entities defined based on the feature vectors, and a processor for predicting motions of the objects that are changed at a second time point by the estimated dynamic interaction; and an output device for outputting a result reflecting the motion predicted at the second time point.

The processor is configured to determine based on a past history of relationships between the entities up to a time before the first time point, and feature vectors corresponding to the entities input up to the first time point, based on the feature vector. a prior that generates prior information; an encoder that generates a latent variable corresponding to a dynamic interaction between the entities based on the feature vector and the prior information; And based on the latent variable, it may include a decoder (decoder) for predicting the movement of the objects that change at the second time point.

The encoder includes: a fully connected graph neural network (GNN) that generates, based on the feature vector, hidden state information corresponding to the state of relationships between the pairs of entities; a forward LSTM for generating prior information corresponding to the objects of the target in the image of the first view based on the hidden state information; a reverse LSTM for generating posterior information predicted according to dynamic interactions between the entities based on the prior information and the hidden state information; and Multi-MLP (MLP) for generating a latent variable corresponding to a dynamic interaction between the entities at the first time point based on the prior information transmitted through the forward LSTM and the post-information transmitted through the backward LSTM -Layer Perceptron).

The image processing apparatus may include at least one of a head up display (HUD) device, a 3D digital information display (DID), a 3D mobile device, and a smart vehicle.

1 is a flowchart illustrating an image processing method according to an exemplary embodiment;
2 to 3 are diagrams for explaining components of an image processing apparatus and operations of the components according to an exemplary embodiment;
4 is a flowchart illustrating an image processing method according to another exemplary embodiment;
5 is a diagram illustrating a result of reflecting a human body motion predicted by an image processing method according to an exemplary embodiment;
6 is a diagram illustrating a movement trajectory reflecting the movement of basketball players predicted by an image processing method according to an embodiment;
7 is a view for explaining a process of generating an image of a future viewpoint by reflecting the movement of pedestrians predicted by the image processing method according to an embodiment;
8 is a block diagram of an image processing apparatus according to an exemplary embodiment;

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes may be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It should be understood that all modifications, equivalents and substitutes for the embodiments are included in the scope of the rights.

The terms used in the examples are used for description purposes only, and should not be construed as limiting. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are given the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In the description of the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

In addition, in describing the components of the embodiment, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the components from other components, and the essence, order, or order of the components are not limited by the terms. When it is described that a component is “connected”, “coupled” or “connected” to another component, the component may be directly connected or connected to the other component, but another component is between each component. It will be understood that may also be "connected", "coupled" or "connected".

Components included in one embodiment and components having a common function will be described using the same names in other embodiments. Unless otherwise stated, descriptions described in one embodiment may be applied to other embodiments as well, and detailed descriptions within the overlapping range will be omitted.

1 is a flowchart illustrating an image processing method according to an exemplary embodiment. Referring to FIG. 1 , an image processing apparatus according to an embodiment defines relationships between entities based on feature vectors of entities of a target whose motion is to be predicted in an image of a first view. do (110). Here, the 'object' for which movement is to be predicted includes not only living things such as people, animals, plants, etc., but also various types of inanimate objects capable of movement, movement transformation, and/or shape transformation, such as automobiles, motorcycles, bicycles, etc. can be understood as meaning A subject may be understood, for example, as a clustered form of entities described below.

According to an exemplary embodiment, 'objects' may be understood as all objects having an organic connection relationship or an organic relationship with a motion prediction target. The 'entities' may correspond to some components and/or some of the objects for which motion is to be predicted.

The objects of interest may correspond to, for example, but not necessarily limited to, body parts of a single user, joints of a single user, multiple pedestrians, multiple vehicles, and athletes of a sports team, etc. . For example, if a target for which a movement is to be predicted is a 'sports team', the objects of the target may correspond to 'athletes in competition' of the corresponding sports team. Alternatively, if the object for which the motion is to be predicted is 'a plurality of pedestrians', the objects of the object may correspond to five different pedestrians moving in front of the vehicle. Alternatively, if the object for which the motion is to be predicted is 'user A' or 'user B's hand', the object of the object may correspond to each body part of user A or finger joints of user B. In addition, if the object whose motion is to be predicted is the material W, the object of the object may correspond to elements constituting the material W.

The 'feature vector' of the entities may include, for example, a position and a velocity of each of the entities, but is not necessarily limited thereto. The feature vector of the entities may correspond to, for example, a movement trajectory of the entities.

For example, the joints of the human body are restricted in movement by a skeleton, the players of a sports team move according to practiced formations, and the means of transport are enforced by traffic laws or traffic rules. can be moved according to As such, 'relationships between entities' are, for example, connection relationships between entities, positions of entities, postures of entities, moving directions of entities, moving speeds of entities, movement trajectories of entities, and applied to entities. It may be determined based on a rule, a movement pattern of the entities according to the rule, a law applied to the entities, and a movement pattern of the entities according to the law.

에 해당할 수 있다. In step 110 , the image processing apparatus applies a feature vector to, for example, a graph neural network (GNN) illustrated in FIG. 3 , thereby providing a hidden state corresponding to relationships between entities at a first time point. information can be generated. In this case, the graph neural network may include, for example, nodes corresponding to entities and edges corresponding to relationships between entities. The graph neural network may be, for example, a fully-connected graph neural network for generating hidden state information corresponding to the state of relationships between pairs of entities, based on a feature vector. Hidden state information generated by the image processing device through a graph neural network is, for example, 'relation embeddings' to be described later.

may correspond to

In an embodiment, a 'hidden state' corresponds to an internal state of nodes constituting a neural network (eg, a graph neural network), and 'hidden state information' corresponds to an internal state of nodes constituting a neural network. It may correspond to information representing an internal state. For example, temporal information in which information processed at a previous time point is accumulated by a feedback structure of a neural network may be embedded in the hidden state. The hidden state information may be, for example, information in a vector form such as a hidden state vector. Also, the hidden state information may include, for example, a feature vector corresponding to the first view and at least one target object included in an image frame of a view preceding the first view.

Hereinafter, for convenience of explanation, the previous casting time of the first time point is a 'time t-1' corresponding to the past, the first time point is a 'time t' corresponding to the present, and the second time point corresponds to the future It can also be expressed as 't+1 time point'.

The image processing apparatus estimates a dynamic interaction between the objects at the first time point based on the relationships between the objects defined in step 110 (step 120). Dynamic interaction may be expressed in the form of, for example, a latent variable representing a relationship between entities.

는 타임 스텝(time step) t에서의 개체들의 특징 벡터(feature vector)를 나타낼 수 있다. 개체들의 특징 벡터는 예를 들어, 위치 및 속도를 나타낼 수 있다. It is common to study surrogate tasks, such as predicting trajectories over time, to understand interactions between entities. Specifically, given N objects to be modeled,

may represent a feature vector of entities at a time step t. The feature vectors of entities may represent, for example, position and velocity.

In general, the framework of Neural Relational Inference (NRI), which can interpret relationships between entities in the process of predicting system dynamics, can predict trajectories by predicting a series of interactions between entities. As a result, accurately predicted interactions enable accurate trajectory predictions, so interactions can be used to improve prediction of future trajectories. However, neural relationship inference (NRI) methods are based on the assumption that these relationships remain static in the observed trajectories, and in many systems, relationships between entities generally change over time. Using neural relational inference (NRI), averaged interactions are retrieved over time, and averaged interactions are difficult to accurately represent the underlying system.

의 형태로 표현될 수 있다. 여기서, e는 모델링되는 관계 유형들(relation types)의 개수에 해당할 수 있다. 잠재 변수는 '잠재 관계 변수'라고도 부를 수 있다. For example, interactions between entities are latent variables for pairs i and j of all entities.

can be expressed in the form of Here, e may correspond to the number of relation types to be modeled. A latent variable can also be referred to as a 'latent relational variable'.

와 개체들의 미래 궤적들을 모두 예측하기 위해 신경 관계 추론(NRI) 방법은 변이형 자동 인코더(Variational Auto-Encoder; VAE)를 학습할 수 있다. These relationships do not have predefined meanings, but a model by relationship types can learn how to assign meaning to each type. latent variable

In order to predict both and future trajectories of entities, the Neural Relational Inference (NRI) method can learn Variational Auto-Encoder (VAE).

The observed variable represents the trajectory x of the individuals, and the latent variable represents the relationships z between the individuals. According to the traditional variant type automatic encoder (VAE), the evidence lower bound (ELBO) may be maximized as in Equation 1 below.

및 θ는 인코더 파라미터 및 디코더 파라미터를 나타낼 수 있다. 수학식 1의 공식은 세 가지 주요 확률 분포들로 구성되며, 이에 대하여는 후술하기로 한다.here,

and θ may represent an encoder parameter and a decoder parameter. The formula of Equation 1 consists of three main probability distributions, which will be described later.

Variant automatic encoder (VAE) is one of generative models, and aims to generate data by learning a probability distribution P(x). The encoder of the variant automatic encoder (VAE) may receive the training data (x) as an input and output a parameter for the probability distribution of the latent variable (z). For example, in the case of a Gaussian normal distribution, μ and σ ² can be output. The encoder can find, for example, an ideal probability distribution p(z|x) that, given the data, can sample a latent variable (z) that the decoder can reconstruct well into the original data. In addition, the decoder of the variant automatic encoder (VAE) may receive a vector sampled from the probability distribution p(z) for the latent variable, and reconstruct the original image using the sampled vector. The decoder may receive the sample extracted from the encoder as an input and play a role of reconstructing it back to the original.

To recap, a variant autoencoder (VAE) finds a distribution of latent variables that explains the given data well (performed by the encoder) through optimization (performed by the encoder), and reconstructs the same image as the original image from the latent variable (performed by the decoder) well. can do.

In an embodiment, a 'dynamic Neural Relational Inference (dNRI)' method that restores interactions between entities at all time points in consideration of relationships between entities that change over time may be used. In an embodiment, by enabling a continuous latent variable to predict separate relationship graphs for each time step, it is possible to solve the problem of the above-described (static) neural relationship inference method and improve prediction accuracy.

More specifically, the dynamic neural relationship inference (dNRI) method according to an embodiment may estimate the interaction between entities by using a latent variable model. Here, a 'latent variable' may indicate the strength of a relationship between entities. In an embodiment, the movement trajectories of the observed entities may be restored as accurately as possible using the estimated relationship strength. The dynamic neural relationship inference (dNRI) method can estimate latent variables at all time points, unlike the neural relationship inference (NRI) method.

In addition, in an embodiment, a sequential relation prior that depends on the history of an input trajectory and an approximate relation posterior that considers both past and future variable states ), a sequential latent-variable model can be applied to the framework of neural relational inference (NRI).

As will be described with reference to FIGS. 5 and 6 below, in an embodiment, a result of applying a dynamic neural relationship inference (dNRI) method to a difficult motion capture and sports trajectory data set can be evaluated. The dynamic neural relationship inference (dNRI) method can significantly improve the recovery of the observed trajectory when compared with the static neural relationship inference (NRI) method, and the different types of trajectories that the static neural relationship inference (NRI) method cannot achieve Relationships that change different phases can be predicted.

에 해당할 수 있다. 사전 정보는 예를 들어, 개체들 간의 관계들의 강도를 나타낼 수 있다. 또한, 사전 정보는 예를 들어, 제1 시점의 이전 시점까지 개체들 간의 관계들의 과거 이력, 및 제1 시점까지 입력된 개체들의 특징 벡터들에 기초하여 결정될 수 있다. 영상 처리 장치는 예를 들어, 히든 스테이트 정보를 순방향 상태 정보로서 도 3의 310에 도시된 순방향(forward) LSTM(Long Short-Term Memory)인 LSTM_prior로 전달함으로써 사전 정보를 생성할 수 있다. In operation 120 , the image processing apparatus may generate prior information corresponding to the objects, for example, based on the hidden state information. The prior information is, for example, a fryer described later

may correspond to The prior information may indicate, for example, the strength of relationships between entities. Also, the prior information may be determined based on, for example, a past history of relationships between entities up to a time point before the first time point, and feature vectors of entities input up to the first time point. The image processing apparatus may generate the prior information by, for example, transferring the hidden state information as forward state information to an LSTM _{prior that is a forward long short-term memory (LSTM) shown in 310 of FIG. 3 .}

에 해당할 수 있다. In operation 120 , the image processing apparatus may generate posterior information predicted to correspond to the objects based on the prior information and the hidden state information. The image processing apparatus may generate post information by, for example, transferring the prior information and the hidden state information as backward state information to LSTM _{enc which is a backward LSTM shown in 310 of FIG. 3 .} The posterior information is, for example, an approximate posterior to be described later.

may correspond to

를 생성할 수 있다. 이때, 영상 처리 장치는 사전 정보에 기초하여 잠재 변수를 최적화할 수 있다. In operation 120 , the image processing apparatus may generate a latent variable corresponding to a dynamic interaction between entities based on the prior information and the post information. The image processing apparatus may sample a result of combining the pre-information and post-information. Based on the sampling result, the image processing apparatus may be configured to provide a latent variable corresponding to a dynamic interaction between objects at a first time point.

can create In this case, the image processing apparatus may optimize the latent variable based on the prior information.

를 예측할 수 있다. The image processing apparatus predicts the movement of the objects that change at the second time point based on the dynamic interaction estimated in operation 120 (operation 130). The image processing apparatus may predict the motion of the object changed at the second time point by decoding the estimated dynamic interaction, for example, by the decoder 330 illustrated in FIG. 3 . The decoder, for example, from the sampling result for the relational variables generated by the encoder, the trajectory distribution

can be predicted

The image processing apparatus outputs a result reflecting the motion predicted at the second time point ( 140 ). In operation 140 , the image processing apparatus may implicitly or explicitly output a result reflecting the motion predicted at the second time point. In operation 140, the image processing apparatus may, for example, reflect predicted motions on objects included in the image of the first viewpoint, process it into an image of the second viewpoint, and output the image of the second viewpoint. have. Alternatively, the image processing apparatus may process the objects included in the image of the first viewpoint into an image of the second viewpoint by reflecting the predicted motion. The image processing apparatus may recognize whether a dangerous situation has occurred based on the image of the second viewpoint, and output an alarm corresponding to the dangerous situation.

According to an embodiment, by estimating dynamic interactions between objects over time from an input image, for example, object segmentation, tracking, and data annotation for deep learning processing It can be used for automation. More specifically, the image processing apparatus estimates the dynamic interaction between objects over time from the input image, so that, for example, future movements of athletes in an athletic event, children or pedestrians playing in front of an autonomous driving vehicle, or a vehicle It is possible to predict their future movements. Estimating the future movements of athletes in an athletic event enables the detection of key game scenes, so that images of key game scenes are effectively transmitted to the spectators watching the game. In addition, the image processing apparatus may prevent an accident by predicting the movement of vehicles or pedestrians to warn a driver or related parties of an accident that will occur in the future. In addition, the image processing apparatus predicts a pattern of finger movement when typing to enhance the performance of recognizing a typing pattern for a virtual keyboard, or the future of elements in physics. By predicting motion, it can contribute to the development of related research fields such as material relationship analysis.

2 is a diagram illustrating an example of a calculation graph for explaining an operation of an image processing apparatus according to an exemplary embodiment. Referring to FIG. 2 , the image processing apparatus 200 according to an embodiment may include a Prior 210 , an Encoder 230 , and a Decoder 250 .

In the dynamic neural relationship inference (dNRI) method according to an embodiment, it is important to capture these changes in the prior variance because relationships between entities are expected to be different for each time step. To this end, in an embodiment, it may be learned by an auto-regressive model for prior probabilities of relationships between entities.

Based on the input feature vector (eg, x ^{t -1} , x, x ^{t + 1} ), the prior 210 determines the past history of relationships between entities up to the previous time point of the first time point, and the first time point. It is possible to generate the prior information determined based on the feature vectors corresponding to the entities input up to .

Prior 210 may adjust according to previous relations at each time step t, as well as inputs for previous relations and times 0 to t.

The fryer 210 may be expressed as, for example, Equation 2 below.

The structure of the fryer 210 used in one embodiment will be described in detail with reference to FIG. 3 below.

For example, in a graph neural network, if an edge is hard-coded to indicate that there is no interaction, the prior value for that edge, i.e. the prior information corresponding to that edge, is the expected sparsity of the relationship for a given problem. (sparsity) may be selected. The prior information may adjust the loss so that the encoder 230 is biased according to a sparsity level. The dictionary information may guide, for example, the encoder 230 to generate hidden state information corresponding to each pair of entities at each time step.

The encoder 230 may generate a latent variable corresponding to a dynamic interaction between entities based on the feature vector and the prior information generated by the prior 210 .

는 관측된 변수들 x의 미래 상태들의 함수에 해당할 수 있다. 따라서, 인코더(230)의 핵심 구성 요소는 변수들의 상태들을 반대로 처리하는 LSTM이다. The role of encoder 230 in one embodiment is to approximate the distribution of relationships at each time step as a function of the total input as opposed to past input history. Actual posterior distribution for latent variables

may correspond to a function of the future states of the observed variables x. Thus, a key component of the encoder 230 is the LSTM, which reverses the states of the variables.

The encoder 230 may be implemented using, for example, a fully connected graph neural network (GNN) architecture with one node per entity as shown at 310 of FIG. 3 . The encoder 230 may be used to generate posterior relation probabilities for all relation types predicted after learning the embedding for each pair of entities. Given the distribution provided by the encoder 230, the sampled relationships can be used as input to the decoder. The encoder 230 should be able to differentiate the sampling procedure so that model weights can be updated through back propagation. However, standard sampling is indistinguishable from a categorical distribution, and consequently, a sample can be drawn from a concrete distribution. This distribution is a continuous approximation to the discontinuous category distribution, and sampling in this distribution can be performed in the form of Equation 3 below.

는

에 대한 예측된 사후 로짓(posterior logits)이고,

는 Gumbel (0,1) 분포의 표본이며, τ는 분포의 평활도(smoothness)를 제어하는 온도 파라미터에 해당할 수 있다.here,

Is

is the predicted posterior logits for ,

is a sample of the Gumbel (0,1) distribution, and τ may correspond to a temperature parameter controlling the smoothness of the distribution.

까지 그래디언트(gradients)를 역전파할 수 있다.This process approximates discrete sampling in a different way and encoder 210 parameters in decoder reconstruction.

You can backpropagate gradients up to .

Components of the encoder 230 will be described in detail with reference to FIG. 3 below.

,

,

)에 기초하여, 제2 시점에서 변화되는 개체들의 움직임을 예측할 수 있다. The decoder 250 is a latent variable(s) generated in the encoder (eg,

,

,

), it is possible to predict the movement of the objects that change at the second time point.

과 같이 표현될 수 있다. 디코더(250)는 변수들 x 의 미래 상태들을 예측하는 데에 도움을 주기 위해 인코더(230)에서 샘플링된 잠재 변수들을 사용할 수 있다. 디코더(250)에 입력되는 잠재 변수 z는 타임 스텝마다 변화될 수 있다. The decoder 250 is

can be expressed as Decoder 250 may use the latent variables sampled at encoder 230 to help predict future states of variables x. The latent variable z input to the decoder 250 may be changed for each time step.

The decoder 250 according to an embodiment may be expressed as Equation 4 below.

In practice, this may involve choosing a graph neural network model for all edges at each time step instead of using the same model throughout the sequence. Through this, the decoder 250 may improve the ability to model a dynamic system by adjusting the prediction based on the system state.

Similar to the encoder 230 , the decoder 250 may also be configured based on a graph neural network (GNN). However, unlike the encoder 230 , the decoder 250 may train a separate graph neural network (GNN) for all edge types. When conveying a message (or information) for a given edge (i, j), the edge model used may correspond to the prediction generated by the latent variable input to the decoder 250 . In addition, in one embodiment, the edge type may be hard-coded to indicate 'no interaction', and in this case, a message transmitted through the corresponding edge during calculation may not exist. .

The decoder 250 may be, for example, a Markovian decoder, in which case the graph neural network (GNN) of the decoder 250 is simply a function of previous predictions and iterative in decoders depending on all previous states. The hidden state can be updated using a graph neural network.

3 is a diagram illustrating components of an image processing apparatus according to an exemplary embodiment. Referring to FIG. 3 , the image processing apparatus 300 according to an embodiment may include a fryer, an encoder 310 , and a decoder 330 .

For example, the encoder may receive the movement trajectory of each entity as an input and encode it as a latent variable representing the relationships between entities. The encoded latent variables may be optimized through the prior information, and decoded as movement trajectories of entities in the next frame through the decoder 330 .

Inputs to the prior and encoder 310 may be fed through a fully connected graph neural network (GNN) to generate embeddings for all pairs of entities at all time steps.

An input to the prior at each time step may generate embeddings per time and per edge through a graphic neural network (GNN) structure expressed by Equations 5 to 8 below.

The graphic neural network (GNN) structure represented by Equations 5 to 8 may implement the form of a neural message transmitted to the graph. Here, vertices v may indicate entities, and edges e may indicate relationships between entities.

는 256 개의 히든/출력 유닛들 및 ELU 활성화 기능을 갖춘 2-계층 MLP일 수 있다. 또한, 프라이어 및 인코더에 의해 사용되는 LSTM 모델들은 64개의 히든 유닛들을 사용할 수 있다. In addition, in the graphic neural network (GNN) represented by Equations 5 to 8, for example,

may be a two-layer MLP with 256 hidden/output units and ELU activation capability. Also, the LSTM models used by the prior and encoder can use 64 hidden units.

와

는 예를 들어, 128개의 히든 유닛들과 ReLU 활성화 기능을 가진 3-계층 MLP일 수 있다. 이 경우, 인코더의 로짓은 256 개의 히든 유닛들과 모델링되는 관계 유형들의 개수와 동일한 수의 출력 유닛들을 가진 3-계층 MLP를 통해 h _emb를 전달하여 생성될 수 있다. in fryer and encoder 310

Wow

may be, for example, a 3-layer MLP with 128 hidden units and a ReLU activation function. In this case, the logit of the encoder can be generated by passing h _emb through a 3-layer MLP with 256 hidden units and a number of output units equal to the number of relation types being modeled.

According to an embodiment, the image processing apparatus may use a recurrent decoder for both static neural relationship inference and dynamic neural relationship inference.

In the above equations, each h may represent information on intermediate hidden states for entities or relationships during calculation. The result of this calculation may be an embedding that captures the state of the relationships between entities i and j at time t. Each of these embeddings can be supplied as an LSTM. Intuitively, such an LSTM can model the evolution of relationships between entities over time.

The input to the prior and encoder 310 is a forward LSTM _{prior that encodes a past history of relationships between entities. and backwards LSTM enc} that encodes a future history of relationships between entities.

All models f shown in FIG. 3 may represent a Multilayer Perceptron (MLP).

MLP may transform the hidden state at each time step into logits of prior variance. These final two steps may be expressed as, for example, Equations 9 and 10 below.

의 이전 타임 스텝들에 대한 관계들에 대한 프라이어의 의존성을 인코딩할 수 있다.In one embodiment, instead of passing as input the previous relation predictions for the prior, the hidden state corresponding to time step t

may encode the dependency of the prior on relations to previous time steps of .

를 재사용하고, 관계 임베딩

의 대표값들을 역방향(backwards) LSTM_enc을 통해 전달할 수 있다. 실시예에 따라서, 인코더는 예를 들어, LSTM(Long-Short term Memory) 이외에도, GRU 및 순환 신경망(Recurrent Neural Network; RNN) 등과 같은 순환 구조의 신경망에 의해 구성될 수도 있다. The encoder embeds the relationship described above.

Reuse and embed relationships

Representative values of can be transmitted through the _{backwards LSTM enc.} According to an embodiment, the encoder may be configured by, for example, a neural network having a recurrent structure, such as a GRU and a Recurrent Neural Network (RNN), in addition to Long-Short Term Memory (LSTM).

The final approximate posterior of the encoder can be obtained by concatenating this reverse state with the forward state provided by the prior, and passing the combined result to the MLP. . The above-described operation of the encoder may be expressed as Equations 11 and 12 below.

를 사용할 수 있다. Since encoder and prior share parameters, in one embodiment, encoder parameters for encoder and prior

can be used

In the prior and encoder 310, the prior is calculated only as a function of past history, whereas the approximate posterior by the encoder may be calculated as a function of the past and future. A set of edge variables is sampled from an approximate forsterer, which can be used to select edge models for a decoder graph neural network (GNN).

The decoder 330 may develop a hidden state using such a graph neural network (GNN) and previous predictions, and use the hidden states to predict the state of entities in the next time step.

및

를 트레이닝하는 과정을 설명하기에 앞서, 신경 관계 추론(NRI) 방법에 따른 트레이닝 과정을 살펴보기로 한다. Hereinafter, the parameters of the prior and encoder 310 and decoder 330 .

and

Before explaining the process of training , let's look at the training process according to the neural relation inference (NRI) method.

를 예측하기 위해 현재 입력 x를 처리할 수 있다. 다음으로, 인코더는 구체적인 근사치에서 이 분포까지 관계들의 집합을 샘플링할 수 있다. 이러한 샘플들

가 주어지면, 최종 단계는 원래 궤도(original trajectory) X를 예측하는 것이다. 이를 통해 디코딩 성능을 향상시키고 디코더가 예측된 에지에 의존하는지 확인할 수 있다. 예를 들어, 트레이닝 시간에 디코더에게 제한된 개수(예를 들어, 10개)의 단계들에 대한 정답 입력들(ground-truth inputs)을 제공한 다음, 이전 예측의 함수로 궤적의 나머지를 예측할 수 있다. First, the encoder calculates the posterior relation probability for all pairs of entities.

We can process the current input x to predict Next, the encoder can sample a set of relationships from a specific approximation to this distribution. these samples

Given , the final step is to predict the original trajectory X. This can improve decoding performance and ensure that the decoder relies on predicted edges. For example, at training time we can provide the decoder with ground-truth inputs for a limited number of steps (e.g. 10) and then predict the remainder of the trajectory as a function of the previous prediction. .

The ELBO described in Equation 1 above may include the following two terms. First, the reconstruction error represents the average of the Gaussian distribution having a fixed variance σ of the predicted output, and as a result, it can be expressed in the form of Equation 13 below.

In addition, KL-divergence represents the variance between a uniform prior and a predicted approximate posterior, and may be expressed, for example, in the form of Equation 14 below.

Here, H may represent an entropy function. The constant term is a result of the fact that the prior is uniform, which can lead to marginalization of one of the encoder terms in loss.

The neural relationship inference (NRI) model is an unsupervised model, which can infer interactions purely from observation data and express them explicitly. To this end, a variant autoencoder model can be formulated in which the latent code represents the basic interaction graph in the form of an adjacency matrix. Both the encoder model and the reconstruction model can be based on a graph neural network. Unlike dynamic neural relationship inference (dNRI) models, static neural relationship inference (NRI) models assume that interactions remain the same over time. The Neural Relationship Inference (NRI) formula assumes that the relationships between all entities are static. However, this assumption is too strong for many applications. The way entities interact is subject to change over time. For example, basketball players may adjust their position relative to the position of other team members at different points in time.

Accordingly, in an embodiment, a dynamic Neural Relational Inference (dNRI) method may be used to reveal dynamic interactions and better track entities whose relationships change over time.

를 예측할 수 있다. 개별 관계

는 모델이 궤도를 통해 그들의 관계들이 변화하는 개체들에 응답하도록 할 수 있으므로 미래 상태들을 예측하는 능력이 향상될 수 있다. 동적 신경 관계 추론(dNRI) 방법을 사용하려면 시간이 지남에 따른 개체들 간의 관계들의 진화를 추적해야 하는데, 이는 정적 신경 관계 추론 (NRI)에는 필요하지 않은 것이다.More specifically, in one embodiment separate relations for every time step t

can be predicted individual relationship

can allow the model to respond to entities whose relationships change through trajectories, thus improving the ability to predict future states. Dynamic neural relationship inference (dNRI) methods require tracking the evolution of relationships between entities over time, which is not required for static neural relationship inference (NRI).

In one embodiment, the purpose of each model component may be reconsidered to predict individual relationships at every time step. As noted earlier, the fryer can in fact correspond to a tunable component of the loss function. Conversely, in order to make the fryer more useful in a sequential context, according to an embodiment, if all previous states are given to the image processing apparatus, relationships between objects may be predicted at all time points. For example, in static neural relation inference (NRI) methods, in contrast to the encoder using a single edge prediction set covering the entire set of input trajectories, in one embodiment the state information of the system at any point in time based on the past and the future. You can use the encoder while understanding This state information can be passed from the encoder to the prior during training due to the KL-divergence term of the loss function. These changes may allow the fryer to better predict future relationships. As a result of sequential relational prediction, the decoder can become more flexible. Through this, in an embodiment, different models may be used for each time point according to how the system changes. All these changes can lead to more expressive models that improve predictive performance.

를 생성할 수 있다. 이러한 관계 임베딩

은 순방향 LSTM 및/또는 역방향 LSTM으로 전달되고, 이를 통해 프라이어

및 근사 포스테리어

가 계산될 수 있다. 그런 다음, 인코더는 근사 포스테리어

에서의 샘플링을 통해 관계 변수

를 생성할 수 있다. 디코더는 이러한 샘플들(예를 들어,

)이 주어지면, 궤도 분포

를 예측할 수 있다. For example, an input trajectory x input to the image processing apparatus 300 according to an embodiment passes through a graph neural network (GNN) model and embeds relationships for every time t and all pairs of entities (i, j)

can create embedding these relationships

is passed to the forward LSTM and/or reverse LSTM through which the fryer

and approximate posteriors

can be calculated. Then, the encoder makes an approximate posterior

Relational variables through sampling from

can create The decoder can use these samples (e.g.,

), given the orbital distribution

can be predicted

In an embodiment, unlike in the case of static neural relationship inference (NRI), ground truth states may always be provided to the decoder as an input during training. In one embodiment, it is observed that the test provides the correct answer for a fixed number of steps, and then the prediction can be used as input to the remaining trajectories.

According to an embodiment, the reconstruction error in the ELBO may be calculated in the same manner as described in Equation 13, and the KL-divergence may be calculated as Equation 15 below.

이 주어지면, 관계들

에 대한 프라이어 분포를 계산할 수 있다. 또한, 관계 예측

을 얻기 위해 프라이어로부터 샘플링하고, 변수들

의 다음 상태를 추정하기 위해 이전 예측뿐만 아니라 관계 예측

을 사용할 수 있다. 이러한 과정은 전체 이동 궤도가 예측될 때까지 계속될 수 있다.According to one embodiment, it is possible to predict future states of the system at the time of testing. This means that the encoder cannot be used to predict edges, since we do not have adequate information about the future. Thus, in one embodiment, the previous prediction

Given this, the relationships

We can calculate the prior distribution for . In addition, relationship prediction

sample from the fryer to obtain

Predict the relationship as well as the previous prediction to estimate the next state of

can be used This process may continue until the entire movement trajectory is predicted.

4 is a flowchart illustrating an image processing method according to another exemplary embodiment. Referring to FIG. 4 , the image processing apparatus according to an embodiment may determine objects of a target whose motion is to be predicted in an image of a first view ( 410 ). The image processing apparatus may directly receive a selection of an object included in the input image from a user, for example, or the image processing apparatus may automatically set the object included in the input image directly.

The image processing apparatus may extract the feature vectors of the objects determined in operation 410 (operation 420).

The image processing apparatus may generate hidden state information corresponding to relationships between entities at a first time point by applying the feature vector extracted in operation 420 to the graph neural network (operation 430 ).

The image processing apparatus may generate dictionary information based on the hidden state information generated in operation 430 (operation 440).

The image processing apparatus may generate post-information predicted to correspond to the objects based on the prior information generated in step 440 and the hidden state information generated in step 430 ( S450 ).

The image processing apparatus may generate a latent variable corresponding to a dynamic interaction between entities based on the prior information generated in operation 440 and the post-information generated in operation 450 ( 460 ).

The image processing apparatus may predict the movement of the objects that change at the second time point based on the latent variable generated in operation 460 (operation 470).

The image processing apparatus may output a result reflecting the motion predicted at the second time point ( 480 ).

In one embodiment, in order to demonstrate the strength of dynamic neural relationship inference (dNRI) compared to static neural relationship inference (NRI), human motion capture data sets and movement trajectory data of basketball players as shown in FIGS. 5 and 6 below. Experiments can be performed on sets.

In FIGS. 5 and 6 , several sample trajectories and predicted relationships can be further visualized to show the operation of the models. Also, all models can hard-code the first edge type to indicate no interaction. For evaluation purposes, the models are given n initial time steps of the input and have to predict some future steps. When evaluating a static model, one embodiment may use two different inference procedures.

In FIGS. 5 and 6, the inference procedure marked 'S NRI, Dyn inf ' represents the result of evaluating the relationship prediction using the most recent n orbital predictions, and the inference procedure marked 'Static NRI' predicts the relationship types. The result of evaluating the relationship prediction using the input of the initial n time steps provided for These relationships can be used to decode the entire movement trajectory.

5 is a diagram illustrating a result of reflecting the movement of the human body predicted by the image processing method according to an exemplary embodiment. Referring to FIG. 5 , a dynamic neural relationship inference (dNRI) model 510, a static neural relationship inference model 530, and a “dynamic” inference for a test trajectory of a motion capture object using four relationship types. The results of sampling the motion predicted through the static neural relationship inference (S. NRI (Dyn. Inf)) model 550 with In FIG. 5 , a human skeleton in a solid line indicates a correct answer state, and a human skeleton in a dotted line may correspond to a motion prediction result according to each inference model. Each of these frames can be predicted every 20 time steps after the most recent correct answer is provided.

5 shows four predicted time steps for a static model and a dynamic model. It can be seen that all models can capture a general jumping motion, but the dynamic neural relationship inference (dNRI) model 510 according to one embodiment tracks the positions of the leg and hip joints much more accurately. As can be seen in FIG. 5 , the dynamic neural relationship inference (dNRI) model 510 is able to predict frames for the future of the gait cycle without deviating too far from the ground-truth skeleton. However, static models 530 and 550 start to make significant errors as more frames are predicted in the future. These errors can occur at the points where significant deformations of the skeleton begin to appear.

6 is a diagram illustrating movement trajectories reflecting the movement of basketball players predicted by the image processing method according to an exemplary embodiment. Referring to FIG. 6 , results of an experiment using trajectory data of basketball players for each of the above-described models of FIG. 5 are shown.

The left graph 610 of FIG. 6 shows each model (Dynamic Neural Relationship Inference (dNRI) Model 510, Static NRI Model 530 and "Dynamic" inferences for the trajectory data of basketball players. It may represent the prediction error of the static neural relationship inference (S. NRI (Dyn. Inf)) model 550). The three figures 630 , 650 , 670 of FIG. 6 show samples of trajectory predictions using the respective models.

Figure 630 corresponds to the correct answer result, the figure 650 corresponds to the inference result of the static neural relationship inference (NRI) model, and the figure 670 corresponds to the inference result of the dynamic neural relationship inference (dNRI) model according to an embodiment. can

The two-dimensional positions and velocities of an attacking team composed of five players may be included in the movement trajectories shown in each of the drawings 630 , 650 , and 670 of FIG. 6 . The two-dimensional positions and velocities of the attacking team can be pre-processed by, for example, 49 frames played for about 8 seconds. At this time, all models can be trained in the first 40 frames of the training trajectory. In evaluation, these models are provided with the first 40 input frames, and the task of predicting the next 9 frames can be performed. In one embodiment, for example, we can train models that predict two types of relationships.

From the results of FIG. 6 , when predicting the movement trajectories of basketball players, it can be understood that the movement trajectories predicted by the dynamic neural relationship inference (dNRI) model are more accurate than the static neural relationship inference (Static NRI) model. As described above, according to the image processing method according to an embodiment, the movement of each player may be predicted based on the relationship between the players during an athletic match.

7 is a view for explaining a process of generating an image of a future viewpoint by reflecting the movement of pedestrians predicted by the image processing method according to an embodiment. Referring to FIG. 7 , for example, an image 710 in which children playing with a ball in front of another vehicle parked in front of a moving vehicle are photographed is shown.

In an embodiment, a Dynamic Neural Relational Inference method may be used to process a system in which relationships between entities are expected to change over time. In addition, modeling the relationships between dynamic entities can improve performance in both human motion capture and sports trajectory prediction tasks. The model can also be applied to other areas where dynamic relationships are expected, including traffic systems and biological neural networks.

Alternatively, according to an embodiment, a motion of a target object predicted in a video by an image processing method may be modeled and utilized for various applications.

The image processing apparatus according to an embodiment defines relationships between objects based on feature vectors of objects (eg, children playing with a ball) extracted from the image 710 to predict a motion. can do. In this case, relationships between entities may be defined by hidden state information generated in response to states of relationships between pairs of entities.

The image processing apparatus may generate dictionary information corresponding to the objects, such as the image 720 , based on the hidden state information. At this time, the prior information reflects the movement of the children covered by the parked vehicle, and is determined based on the past histories of the objects identified from the image before the image 710 and the feature vectors of the objects corresponding to the image 710 . can

The image processing apparatus may generate a latent variable corresponding to a dynamic interaction between entities based on hidden state information generated from the image 710 and prior information such as the image 720 .

Based on the latent variable, the image processing apparatus may generate an image 730 predicted to reveal the motion patterns of children covered by a parked vehicle in the image 710, or after the image 710 is captured. Predicted future images (image 740 and image 750 ) may be generated and displayed.

According to an embodiment, by applying the above-described image processing method to prediction of an object or objects of a target that are not visible in the current frame, prediction information about a pedestrian is provided to the driver during driving or autonomous driving, or when a danger occurs It can help drivers drive safely by warning drivers of danger.

The image processing method according to an embodiment may be used for video segmentation or video tracking, such as instance segmentation and amodal segmentation, in various electronic products.

8 is a block diagram of an image processing apparatus according to an exemplary embodiment. Referring to FIG. 8 , an image processing apparatus 800 according to an embodiment includes a communication interface 810 , a processor 830 , and a memory 850 . Communication interface 810 , processor 830 , and memory 850 may communicate with each other via communication bus 805 . The image processing apparatus 800 may correspond to, for example, a head-up display (HUD) device, a 3D digital information display (DID), a 3D mobile device, and a smart vehicle.

The communication interface 810 receives an image of a first view including objects of a target whose motion is to be predicted.

The processor 830 extracts feature vectors of objects from the image of the first view. The processor 830 estimates the dynamic interaction between the entities at the first time point based on the relationships between the entities defined based on the feature vector. The processor 830 predicts the movement of the objects changed at the second time point by the estimated dynamic interaction. The processor 830 may include, for example, a prior, an encoder, and a decoder as shown in FIGS. 2 and 3 .

The output device 850 outputs a result reflecting the motion predicted at the second time point. The output device 850 may correspond to, for example, a display device such as a display of a HUD, or a sound device such as a speaker.

According to an embodiment, the communication interface 810 may receive feature vectors of entities extracted from the image of the first view. In this case, the processor 830 does not perform the feature vector extraction process, but estimates the dynamic interaction between the entities at the first time point based on the relationships between the entities defined based on the feature vector, and the estimated dynamic It is possible to predict the movement of the objects changed at the second point in time due to the interaction.

The memory 850 may store, for example, feature vectors of objects in the image of the first viewpoint received through the communication interface 810 or the image of the first viewpoint received through the communication interface 810 . In addition, the memory 850 may store the prior information generated by the processor 830 , a latent variable corresponding to the operational interaction between entities estimated by the processor 830 , and/or the second predicted by the processor 830 . It is possible to store the movement of objects that change at 2 viewpoints.

Also, the memory 850 may store various pieces of information generated in the process of the processor 830 described above. In addition, the memory 850 may store various data and programs. The memory 850 may include a volatile memory or a non-volatile memory. The memory 850 may include a mass storage medium such as a hard disk to store various data.

Also, the processor 830 may perform at least one method described above with reference to FIGS. 1 to 7 or an algorithm corresponding to the at least one method. The processor 830 may be a hardware-implemented data processing device having a circuit having a physical structure for executing desired operations. For example, desired operations may include code or instructions included in a program. For example, a data processing device implemented as hardware includes a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , an Application-Specific Integrated Circuit (ASIC), and a Field Programmable Gate Array (FPGA).

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and carry out program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

800: image processing unit
805: communication bus
810: communication interface
830: Processor
850: memory

Claims (20)

defining relationships between entities based on feature vectors of entities of a target whose motion is to be predicted in an image of a first view;
estimating a dynamic interaction between the entities at the first time point based on the relationships between the entities;
estimating the movement of the objects changed at a second time point based on the estimated dynamic interaction; and
outputting a result reflecting the predicted motion at the second time point
containing,
Image processing method. According to claim 1,
The relationships between the entities are
Connection relationships between the entities, the positions of the entities, the postures of the entities, the moving direction of the entities, the moving speed of the entities, the trajectory of the entities, a rule applied to the entities, determined based on at least one of a movement pattern of the entities according to the rule, a law applied to the entities, and a movement pattern of the entities according to the law,
Image processing method. According to claim 1,
The step of defining the relationships between the entities
By applying the feature vector to a Graph Neural Network (GNN) comprising nodes corresponding to the entities and edges corresponding to relationships between the entities, the relation between the entities at the first time point is applied. generating hidden state information corresponding to the
containing,
Image processing method. 4. The method of claim 3,
The graph neural network is
A fully-connected graph neural network for generating, based on the feature vector, hidden state information corresponding to the state of relationships between pairs of entities.
comprising,
Image processing method. 4. The method of claim 3,
The step of estimating the dynamic interaction is
generating prior information corresponding to the entities based on the hidden state information;
generating posterior information predicted to correspond to the entities based on the prior information and the hidden state information; and
generating a latent variable corresponding to a dynamic interaction between the entities based on the prior information and the post-information;
Including, an image processing method. 6. The method of claim 5,
The prior information
Determined based on a past history of relationships between the entities up to a time point prior to the first time point, and feature vectors of the entities input up to the first time point,
Image processing method. 6. The method of claim 5,
The step of generating the dictionary information is
generating the dictionary information by transferring the hidden state information as forward state information to a forward long short-term memory (LSTM);
containing,
Image processing method. 6. The method of claim 5,
The step of generating the post-information is
generating the post-information by transferring the prior information and the hidden state information as backward state information to a backward LSTM;
containing,
Image processing method. 6. The method of claim 5,
The step of generating the latent variable is
sampling a result of combining the pre-information and the post-information; and
generating, based on the sampling result, a latent variable corresponding to the dynamic interaction between the entities at the first time point;
containing,
Image processing method. 10. The method of claim 9,
The step of generating the latent variable is
optimizing the latent variable based on the prior information
containing,
Image processing method. According to claim 1,
The subject objects are
comprising at least one of body parts of a single user, joints of a single user, multiple pedestrians, multiple vehicles, and athletes of a sports team.
Image processing method. According to claim 1,
Predicting the movement of the objects includes:
predicting the movement of the object that is changed at the second time point by decoding the estimated dynamic interaction
containing,
Image processing method. According to claim 1,
The step of outputting a result reflecting the predicted motion
processing the objects included in the image of the first viewpoint into an image of a second viewpoint by reflecting the predicted motion; and
outputting the image of the second viewpoint
containing,
Image processing method. According to claim 1,
The step of outputting a result reflecting the predicted motion
processing the objects included in the image of the first viewpoint into an image of a second viewpoint by reflecting the predicted motion;
recognizing whether a dangerous situation has occurred based on the image of the second viewpoint; and
Outputting an alarm corresponding to the dangerous situation
containing,
Image processing method. According to claim 1,
Determining the objects of the object for which the motion is to be predicted
further comprising,
Image processing method. A computer program stored in a computer-readable recording medium in combination with hardware to execute the method of any one of claims 1 to 15. a communication interface for receiving an image of a first view including objects of a target for which motion is to be predicted;
extracting feature vectors of the entities from the image of the first viewpoint, and estimating dynamic interactions between the entities at the first viewpoint based on relationships between the entities defined based on the feature vectors, and a processor for predicting motions of the objects that are changed at a second time point by the estimated dynamic interaction; and
An output device for outputting a result reflecting the motion predicted at the second time point
containing,
image processing device. 18. The method of claim 17,
the processor
Based on the feature vector, a past history of relationships between the entities up to the time before the first time point, and prior information determined based on the feature vectors corresponding to the entities input up to the first time point creating a prior;
an encoder that generates a latent variable corresponding to a dynamic interaction between the entities based on the feature vector and the prior information; and
A decoder predicting the movement of the entities that change at the second time point based on the latent variable
containing,
image processing device. 19. The method of claim 18,
the encoder is
a fully connected graph neural network (GNN) that generates, based on the feature vector, hidden state information corresponding to states of relationships between the pairs of entities;
a forward LSTM for generating prior information corresponding to the objects of the target in the image of the first view based on the hidden state information;
a reverse LSTM for generating post-information predicted according to a dynamic interaction between the entities based on the prior information and the hidden state information; and
Based on the prior information transmitted through the forward LSTM and the post-information transmitted through the backward LSTM, MLP (Multi- Layer Perceptron)
containing,
image processing device. 18. The method of claim 17,
The image processing device
comprising at least one of a Head Up Display (HUD) device, a 3D Digital Information Display (DID), a 3D mobile device, and a smart vehicle,
image processing device.

Images