KR20210101570A - Method and Apparatus of Avatar Placement in Remote Space from Telepresence - Google Patents

Abstract

Disclosed are a method and apparatus of placing an avatar in a remote space for telepresence. A method of placing avatar in a remote space according to an embodiment of the present invention includes the steps of: collecting user preference data, which is placement data preferred by people, through a user preference survey; defining a similarity between a person and an avatar between corresponding points for two spaces by learning the collected user preference data with a deep neural network-based neural network; and placing the user's avatar in a remote space based on the similarity between the person and the avatar between corresponding points for the two defined spaces.

Description

{Method and Apparatus of Avatar Placement in Remote Space from Telepresence}

The present invention relates to a remote space avatar arrangement technology for telepresence, and more particularly, to provide telepresence as if people living in different places are living together in a remote space that preserves the context and meaning of a person located in a real space. The present invention relates to a method and apparatus for disposing an avatar in a remote space capable of disposing an avatar at a corresponding point in space.

Rapidly advancing technologies are increasingly enabling 3D telepresence experiences, which allow users to interact with remote users through their virtual avatars, which are images of the remote user overlaid on local physical space. Significantly improves presence. For two-way communication, the local user is sometimes represented as an avatar that the remote user can see.

One scenario for 3D telepresence is for a user to wear a head-mounted display (HMD) with a virtual avatar that moves just like a remote user. This can be achieved by capturing the remote user's appearance and reconstructing it directly into a virtual avatar. This scenario works well if both remote spaces have the same configuration. Otherwise, spatial differences may negate the motion of the avatar. That is, the avatar can sit in the air or penetrate furniture. Because human living spaces are diverse, when the two spaces are different, the user's motion performed in the remote environment must be modified according to the configuration of the local environment, and the local user can properly recognize what the remote user is doing through his or her avatar and be able to interact with her avatar.

This problem is a kind of motion retargeting problem in computer animation research. While traditional motion retargeting problems mainly deal with changes in personality or environmental geometry, in telepresence applications, retargeting can take into account the various elements of a person's motion in real life, including the user's motion, attention, and relationship with surrounding people or objects. should be considered At a higher level, other important aspects should also be considered, such as the naturalness of the avatar animation, ensuring the availability and accessibility of interactive resources, and user immersion. The problem of reassigning objects in motion to best preserve the meaning of motion is a challenging task, as it is difficult to recognize some potential properties and compromise between multiple elements.

Recent technological advances facilitate interactive immersive telepresence that merges two remote spaces into one. A conventional technology according to an embodiment proposes an HMD-based telepresence system that superimposes an image of a remote user in a local space, and another conventional technology according to another embodiment transmits 3D data of a remote user and an object to the local space. Although these studies focus on capturing remote users and reconstructing them in a designated local space, they do not consider the problems arising from the difference between the two spaces.

This problem has been addressed in part by the techniques of one embodiment that have developed a system for projecting images of remote users at specific local locations where the communication is manually assigned, eg, sofa-to-sofa. In the related art, another embodiment of the present invention proposes a method of selecting an appropriate furniture object to which an avatar should be located by configuring communication between objects in two spaces, and adjusting body poses according to different shapes of objects. Another conventional technique of another embodiment proposes a method of strictly arranging free spaces to overlap as much as possible so that a remote person in the free space can be transmitted to the free space. However, the strict layout approach is valid only when the two free spaces are similar.

Problems caused by spatial differences often occur in VR. To allow a user to walk through a large virtual scene within a narrower physical space, researchers have developed a technique to micro-rotate the virtual space in order to induce the user to change the direction of walking or to optimally distort the virtual space to fit the physical space. developed.

The problem of human motion retargeting shares the same goal as the motion retargeting problem in that the existing human motion is modified for different situations. Early methods mostly solved the problem of modifying a given motion for new characters with different body dimensions or different environments. With respect to an interactive motion between two people or an interactive motion between a person and an object, a retargeted motion can be meaningful only if the meanings of the interaction between the two people are preserved. To this end, many approaches have defined spatial relationships between interacting entities.

Retargeting a placement of a person to another environment in a day-to-day scenario should also preserve the meaning of that location. Compared with previous motion retargeting studies, this problem has to consider much more factors related to human motion as described above.

In order to solve the problem of character placement and behavior in the scene, it is necessary to represent the human space use and behavior in relation to the surrounding people and the surrounding environment, which is increasingly interested in computer graphics research.

The conventional technique of one embodiment estimates both the 3D geometry and the human pose usable in the scene by simply fitting the pose set to the 3D geometry in a single indoor image. Another prior art embodiment technique observes real human interaction with objects in order to learn to predict the likelihood of an action in a 3D scene. Another prior art embodiment technique has developed a human-centered interactive expression that connects the human pose attributes, which can generate both a human pose and a furniture arrangement corresponding to a desired motion input, with the geometry and arrangement of an object.

At the object interaction level, researchers have developed various methods for representing and estimating object tolerance or function. A prior art embodiment technique has proposed a geometric description of the function of a 3D object derived from the interaction between objects in the context of a given scene. According to another conventional technique, a method for predicting a human pose for a 3D shape by identifying a contactable local shape in a 3D shape and searching for an acceptable human pose has been developed.

Interpersonal distance is another important factor in human placement. Spatial science is the study of human behavior in space use according to egocentric distance, and many studies have reported that spatial science is effective between people and virtual avatars. In addition to interpersonal distance, the prior art according to an embodiment is inspired by the theory of human territory and emphasizes the importance of agent orientation in modeling group dynamics in social interaction.

Planning and generating motion within a scene taking into account multiple factors is a complex task and has not been extensively studied in computer graphics. Notable recent studies of this kind have synthesized whole-body motions, including movement, body position, action execution, and gaze behavior, for general demonstration tasks in furniture and indoors. In order to make a plausible movement, the prior art considers various conditions such as visual constraints, movement accessibility, and behavioral validity among obstacles.

In addition, Korean Patent No. 10-1659849 and US Patent No. US9843772 B2, which are technologies according to an embodiment of the prior art, are technologies applied by the applicant of the present invention, and provide a telepresence method using an avatar, and each spatial information, It is a technology for generating an avatar that responds to the user's motion by reflecting the user's location information and the user's gaze information in each space. Here, the technology divides the space into patches, determines the location of the avatar in units of patches, and uses only the distance and angle of the person-avatar, and class information and geometric information of the patch as factors for measuring the spatial similarity, and It has a motion change function that is adaptive to the environment of the space.

However, the prior art does not consider changes in the importance of key features according to interactions with people and situations.

Embodiments of the present invention provide a remote space in which an avatar can be placed at a corresponding point in a remote space that preserves the context and meaning of a person located in a real space in order to provide telepresence as if people living in different places live together. An avatar arrangement method and apparatus are provided.

A remote space avatar arrangement method according to an embodiment of the present invention includes: collecting user preference data, which is arrangement data preferred by people, through a user preference survey; defining a similarity between a person and an avatar between corresponding points for two spaces by learning the collected user preference data with a deep neural network-based neural network; and arranging the user's avatar in the remote space based on the degree of similarity between the person and the avatar between the defined corresponding points for the two spaces.

The user preference data includes the relative distance and gaze direction between the person and the avatar, the height value around the location where the person is located, the sum of the types and distances of the furniture around the location where the person is located, the sum of the types of furniture and the distance seen in the front view of the person, It may include at least one of whether a person is seated.

In the step of defining the degree of similarity, feature modeling is performed, including spatial features that specify interaction features between other people or objects, pose acceptance features for human poses, and functional features of the surrounding space, and based on the feature modeling, two By learning the similarity between the disposition features for the space, the similarity between the person and the avatar between the corresponding points for the two spaces can be defined.

The neural network may include a neural network that learns a nonlinear characteristic of a difference or distance between the two spaces.

The disposing may include modeling the movement of the avatar as a finite state machine comprising three states of static placement, quasi-normal movement, and movement, and placing the user's avatar in a remote space using the modeled finite state machine. have.

In the disposing of the avatar, when the disposition of the avatar is finished, the disposition state of the avatar is changed to a quasi-normal moving state, the state of the avatar is changed to the moving state when the user starts walking, and when the user stops, the avatar is moved. The state can be switched to a static batch state.

A remote space avatar arrangement apparatus according to an embodiment of the present invention includes: a collecting unit for collecting user preference data, which is arrangement data preferred by people, through a user preference survey; a definition unit for learning the collected user preference data with a deep neural network-based neural network to define a degree of similarity between a person and an avatar between corresponding points for two spaces; and an arrangement unit for disposing the user's avatar in the remote space based on the degree of similarity between the person and the avatar between corresponding points for the two defined spaces.

The user preference data includes the relative distance and gaze direction between the person and the avatar, the height value around the location where the person is located, the sum of the types and distances of the furniture around the location where the person is located, the sum of the types of furniture and the distance seen in the front view of the person, It may include at least one of whether a person is seated.

The definition unit performs feature modeling including spatial features specifying interaction features between other people or objects, pose acceptance features for a person's pose, and functional features of the surrounding space, and arrangement for two spaces based on the feature modeling By learning the similarity between the features, it is possible to define the similarity between the person and the avatar between the corresponding points for the two spaces.

The neural network may include a neural network that learns a nonlinear characteristic of a difference or distance between the two spaces.

The arrangement unit may model the movement of the avatar as a finite state machine having three states of static placement, quasi-normal movement, and movement, and may place the user's avatar in a remote space using the modeled finite state machine.

When the arrangement of the avatar is finished, the arrangement unit changes the arrangement state of the avatar to a semi-normal moving state, changes the state of the avatar to the moving state when the user starts walking, and changes the state of the avatar to the moving state when the user stops. You can switch to a static batch state.

According to embodiments of the present invention, in order to provide telepresence as if people living in different places are living together, the avatar may be arranged at a corresponding point in a remote space that preserves the context and meaning of a person located in the real space. .

According to embodiments of the present invention, since immersive communication between users in a remote space is enabled, it is possible to reduce the time, cost, and energy consumption of traveling to meet in person, thereby increasing economic and environmental benefits.

According to embodiments of the present invention, by arranging the avatar in the real space, the user's coexistence and interaction with the user in the remote space are enabled in the real space.

The present invention can be applied to a telepresence application that gives a sense of coexistence to people living apart from each other, and can also be applied to a system that overcomes physical distance and communicates in various situations such as remote conferences, education, and social events.

1 shows a screenshot of the system of the present invention.
2 shows an exemplary diagram of a target scenario for 3D telepresence.
3 is a diagram illustrating an example of a user survey screen.
4 shows some representative samples of user preference surveys.
5 is a diagram illustrating an example of a histogram of an average nearest neighbor distance of a user arrangement.
6 shows an exemplary diagram of features expressing the meaning of arrangement.
7 is a diagram illustrating an example for explaining a process of generating different features in a scene.
8 shows an exemplary diagram of a network structure of a batch similarity predictor.
9 shows a finite state machine for avatar motion.
10 is a diagram illustrating an example of a procedure for creating two local areas.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be embodied in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains It is provided to fully inform those who have the category of the invention, and the invention is only defined by the category of the claims.

The terminology used herein is for the purpose of describing the embodiments, and is not intended to limit the present invention. As used herein, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” refers to a referenced component, step, operation and/or element of one or more other components, steps, operations and/or elements. The presence or addition is not excluded.

Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used with the meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. The same reference numerals are used for the same components in the drawings, and repeated descriptions of the same components are omitted.

Embodiments of the present invention, in order to provide telepresence as if people living in different places live together, arranging the avatar at a corresponding point in a remote space that preserves the context and meaning of a person located in the real space. do.

Here, the present invention precedes a user preference survey on the location of an avatar corresponding to a person located in various situations in order to find an optimal correspondence point that can vary for each person according to the difference in situation and space, and consists of five main characteristic elements. The relative distance and gaze direction between the person and the avatar, the height value around the place where the person is located, the sum of the furniture types and distances around the place where the person is located, the sum of the furniture types and distances seen in the person's front view, and whether or not a person is seated can be utilized Define the similarity between two correspondence points by learning user preference data expressed as key feature elements with a deep neural network-based neural network, for example, Triplet MatchNet, and use the spatial sampling-based optimization technique to optimize the correspondence point with the highest similarity. You can find and place your avatar.

That is, the present invention can provide a technique for arranging an avatar at a location where various interactions that can occur in another person's remote space are possible, and disposes the avatar at an arbitrary location on a continuous surface without dividing the space into patches. can do it As factors for measuring spatial similarity, factors such as the distribution of objects around the person (avatar), information on objects entering the person's (avatar) field of view, and the line/sitting posture of the person (avatar) can be considered quantitatively.

The present invention addresses an important sub-problem of determining the placement of an avatar in a room-scale indoor environment, among various problems to be explored for generating an avatar motion. The telepresence scenario of the present invention shows two remote rooms with different layouts and furniture arrangements as shown in FIG. 2 . By teleporting person X in space A to avatar X' in space B, person Y in space B wearing the HMD can see avatar X'. Similarly, X can see Y's avatar Y' in space A. This can make each user feel that the remote user is visiting the place and can interact with them as if they were physically in the same room. In addition, the present invention assumes that two people live daily lives in a telepresence environment. They sometimes interact with each other and may act independently. To simplify matters, the present invention assumes that there is only one person in each space. Now, assuming that X in space A moves from one point to another, it is important to place X' in space B to best represent X.

This has to do with understanding the meaning of X in the pX of space A, where pX can mean an arrangement that includes the position and orientation of X, and for this purpose a number of different properties such as interaction, pose and space should be taken into account.

Interaction: X can be in pX to interact with people or objects near pX, in which case X' should be placed where the same interaction is possible.

Pose: X can take certain poses that pX provides, eg sitting in a chair, ideally placing X' where it can accommodate the same poses.

Space: The indoor space is divided into various functional spaces such as dining and learning spaces, where X' may need to be placed in the same functional space as X.

The present invention uses a feature set that quantifies these properties.

After defining these characteristics, the present invention provides a method for finding the appropriate corresponding arrangement of X'. This can be formulated as an optimization problem defined as in Equation 1 below.

[Equation 1]

Here, x(pX'|pY, B) denotes a feature vector of pX' representing the meanings given by pY and space B, and Sim(·,·) denotes a similarity function. You can measure how close you are. The avatar placement then relies on modeling an appropriate similarity function.

If the configuration of the two remote spaces, i.e., the space of Y and Y', the shape of the arrangement, and the furniture arrangement are the same, all features will point to a similar position with respect to the arrangement of X'. Otherwise, each feature may increase the preference for a different location. Therefore, learning to measure similarity is related to estimating the relative importance among features, and the relative importance varies depending on the situation. For example, when two people communicate, interaction-related characteristics may dominate other characteristics, but pose and spatial-related characteristics will be more important when users act independently. Thus, the similarity measure will be a non-linear function of the feature.

Since similarity measures are in the realm of human cognition, we take an approach of learning measures from data sets obtained from user research on how users can find their avatars in various situations. In particular, a ranking approach can be chosen to solve the optimization problem, and a triplet matchnet structure can be used to train a neural network that can estimate the similarity of two places in different spaces. The trained neural network significantly outperforms the baseline linear model in terms of test accuracy.

The learned similarity function can be used to perform static placement of the avatar. Since practical telepresence applications must also deal with the motion of the user, the present invention may further utilize various methods for generating the corresponding movement of the avatar as the user moves in an indoor space.

In order to investigate the effectiveness of the method according to an embodiment of the present invention in a telepresence scenario, a prototype VR and AR-based telepresence system in which two users located in different spaces communicate with each other through each other's avatars were built, and various tests were conducted. 1 shows a screen shot of the system of the present invention. FIG. 1A shows a screenshot of a virtual reality telepresence space, and FIG. 1B shows a screenshot of an augmented reality telepresence space. As shown in the plan view and perspective view of Fig. 1A, the female avatar Y' on the left and the male avatar X' on the right move to represent the movements of users X and Y in two remote spaces, and in the top view and perspective view of Fig. 1B As shown, the user X and the user Y can see the other's avatars Y' and X' through the augmented reality telepresence space viewed from the self-centered view.

contribute

The present invention is the first study to determine the placement and movement of a virtual avatar in a general indoor space, and preserves various aspects of the meaning of the user's placement and movement in other remote spaces. To this end, the present invention identifies a feature set to be considered for user placement and provides a neural network-based similarity predictor using a training set obtained from a user preference survey. The present invention can implement a VR and AR-based telepresence system as a prototype, and confirm the effectiveness of the method of the present invention.

Batch Similarity Learning

A central factor for finding the optimal arrangement of the avatar is a similarity function that estimates the similarity of arrangement between the avatar and the person from the telepresence point of view. To this end, we can train a neural network that outputs the difference between two feature vectors representing the meaning of placement. Since the similarity depends on human perception, the present invention first conducts a user preference survey to collect people's preferred placement data for various spatial and human placement configurations, and reviews the collected placement data to provide features related to telepresence. Identifies the set. A neural network can be trained to learn the ranking of a batch using the Triple Matchnet structure.

Data acquisition by user preference survey

In various scenarios, user research can be conducted to collect preferred avatar positions corresponding to user positions. To this end, a total of 864 questions can be generated by first preparing 24 pairs of house models, and the specific process may be as follows. In the present invention, four types of indoor 3D models available in Google Sketch UP can be selected, and the functional area such as a suitable number of furniture items and a space for cooking, learning, and rest is sufficiently wide. You can then double the number of models by changing the furniture arrangement and the interior, for example, the props. Four pairs can be excluded as having their own variants to compose 24 space pairs from 28 pairs in a total of 8 spaces. In order to know that the user prefers to place an avatar in a different indoor environment for each pair of spaces, the present invention can place Person X, Person Y, Avatar Y' in different positions and orientations to generate 36 questions. Distance and direction between the person and the avatar, interactions with objects (e.g. watching TV, looking out a window, etc.), poses (e.g. sitting or standing), functional areas (e.g., A question can be created in each space to have a kitchen, living room, free space), resulting in a total of 24 Х 36 = 864 questions.

3 shows an exemplary view of a user survey screen, where the participant can view the top view of space A (bottom left) and space B (bottom right) by placing a person and an avatar at the same time as a 2D monitor, and participant X ( Views of Avatar X' (top right) and Avatar X' (top left) may be presented to the participant. The participant selects the placement of the avatar X' corresponding to the placement of the person X, the placement of the avatar X' answered by the 10 participants in the lower right corner may overlap, and the yellow and purple gazes indicate X and Y, respectively. For each question, the present invention asks the participants to place the avatar X' in their preferred location, and asks the participants a 2D floor plan of each space and an egocentric view of the person and the avatar as shown in FIG. can provide Participants can explore the space by changing and rotating the avatar with the mouse. During experimentation, the present invention allows users to thoroughly inspect the space according to the guidelines "Place the avatar in a location that best represents the person's location in a distant space". Participants may be asked to place their avatars by viewing only the scene's perspective, and no additional information is provided, such as exactly what X and Y are seeing or what they are doing. The reason for not providing additional information is to provide an avatar placement algorithm that does not depend on such high-level contextual information.

A total of 10 responses can be obtained for each question, resulting in a total user response of 864 Х 10 = 8640. About 200 people take part in the survey, and participants can answer an average of 40 questions.

Figure 4 shows some representative samples of user preference survey. As a result, the participants' responses were similar. On the other hand, participants' answers may vary when the configuration of the two spaces is significantly different due to the separate arrangement of people or avatars, the arrangement of different spaces, or the lack of furniture in a particular category. That is, as shown in Fig. 4, when person X and avatar Y' seem to interact, the user places avatar X' in a similar position, and when person X and avatar Y' act independently, avatar X' The user placement of can make a bigger difference.

In order to understand the clustering/dispersion pattern of user data, a histogram of the average nearest neighbor distance of the user placement in each question can be calculated as shown in FIG. 5 . The nearest neighbor of each user batch may be identified as the closest of the remaining nine user batches for the same question, and the distance may be defined as the weighted sum of the position and angle differences. The average distance between user placement and nearest neighbor can be used for histogram analysis, which shows an untruncated normal distribution pattern that constantly changes the clustering tendency of user survey data.

Characteristic modelling

Determining the placement of avatar X' corresponding to the position of person X requires defining the characteristics of the placement that represent the meaning of that placement, and preserving the characteristics when placing the avatar. After reviewing the user preference survey results, low-level features for interaction features, pose acceptance features, and spatial features can be identified as shown in FIG. 6 to indicate the meaning of human placement in an indoor environment.

As for the interaction characteristic, a person often interacts with another person or object, and the spatial relationship between that person and the entity the other person is present is an important characteristic to be preserved in telepresence. According to Hall's Proxemics, the distance between two people defines the closeness of their relationship. The direction a person's head is pointing is also important, as the visibility of other people or objects also strongly influences the interaction.

The present invention can model the interaction characteristics of two categories. One is the interpersonal relationship between the opponent and the avatar of another party, that is, x _ip = {d _rel , θ ₁ , θ ₂ }. Here, d _rel may mean a distance between two people, and θ _1,2 may mean an angular difference from the front direction of the other party to the position of the other party. Of course, the angle difference may not always be negative.

Another category is the visual attention feature. The present invention assumes that one person can attend up to 12 object categories (eg, sofa, chair, table, TV, air conditioner, refrigerator, sink, lamp, piano, cabinet, shelf, window). Rather than specifying a single object that is difficult to recognize in real situations, all objects within a narrow field of view (eg, 40 degrees) within a certain distance are candidate objects of visual interest, and objects closer to the human position and center of gaze have higher visual attention. Assume you get a value. Therefore, the i-th element of the visual attention function x _va may be defined as in <Equation 2> below.

[Equation 2]

와

는 각각 최대 거리(예를 들어, 4m) 및 각도(예를 들어, 40도)를 의미할 수 있다.Here, d _{i ,j} and θ _i,j mean the distance and angle of the j-th object of the i-th category (i = 1 ... 12),

Wow

may mean a maximum distance (eg, 4 m) and an angle (eg, 40 degrees), respectively.

Unlike objects, the placement of avatars through _{interpersonal characteristics x ip} can always be considered even when the other party or avatar is invisible to the other party. The rationale for this choice is that even if two people are not directly interacting, the present invention assumes that people are constantly taking the other's position in mind. Once trained, the optimizer of the present invention will highlight this function appropriately depending on the situation.

Speaking of pose receptive characteristics, a person's pose, such as a sitting or standing posture, is an important factor in characterizing human behavior, requiring placement of the avatar in a position that accepts the user's pose to reflect the remote user's pose. there is Because typical furniture items that accept a person's pose (eg, floor, chair, bed) have different height fields, pose acceptance is expressed as a height field around the location. Specifically, a circular area with a radius of 0.5 m around the person is divided into a center, that is, a circle with a radius of 0.25 m, and divided into 16 peripheral sectors in the outer ring. Then, the pose receptive feature x _pa is expressed as the average height of each subdivided area. We also add a _{binary feature x ss} to indicate whether the person or avatar is sitting or standing. For example, x _ss may be set to sit on a seat when a person or avatar is placed in a furniture category such as a sofa or chair where the person or avatar can sit, and may be set to stand otherwise.

The pose receptive feature may not only characterize possible poses at a given location, but may also define the spatial relationship between the person (or avatar) and nearby furniture, eg, standing next to a table or standing in front of a table.

Regarding spatial characteristics, the purpose of spatial characteristics of the present invention is to characterize the functional characteristics of the surrounding space. For this, household distribution is an important factor. In order to model spatial features, the present invention can take a rather simplistic approach that summarizes the distances of furniture items belonging to the same category within a certain distance. Therefore, the i-th element of the spatial feature x _sp may be defined as in Equation 3 below.

[Equation 3]

는 공간적 특징 범위의 최대 거리(예를 들어, 3m)를 의미하며, d_i,j는 i번째 카테고리(i = 1···12)의 j번째 오브젝트의 거리를 의미할 수 있다.here,

may mean the maximum distance (eg, 3m) of the spatial feature range, and d _i,j may mean the distance of the j-th object of the i-th category (i = 1 ... 12).

All of the feature vectors of the batch can be expressed as a _{45-dimensional vector x = [x ip} , x _va , x _pa , x _ss , x _{sp ].}

similarity learning

As described above, the present invention can obtain several samples from a user preference survey and define feature vectors that characterize their placement in an indoor scene. The arrangement characteristic of person X' may be expressed as ^{x 0} , and the arrangement characteristic of the avatar X' obtained through the user preference survey ^{may be expressed as x + .} For each sample of _{x i} ⁰ _{, x i,j} ⁺ , (i = 1...864, j = 1...10) can be obtained.

A user-selected batch may indicate that one is better than another, but is not an absolute correct answer. That is, the user sample x ⁺ may not have been selected if there had been a better placement. Therefore, this problem needs to be solved in terms of ranking among data. That is, it can be solved by training a similarity function d(·, ·) that can determine that ^{d(x 0} , x ⁺ ) < d(x ⁰ , x ^{- ) when x -} is a feature of a batch not selected by the user can

The basic approach to this is to find a two-linear similarity function, d(x ⁰ ,x) = (x ⁰ -x) ^T W(x ⁰ -x). Many techniques have been developed for similarity learning or distance measurement learning in a two-line format, and some of them can support learning relative similarity like the problem of the present invention. This approach can be effective if the differences in the data provide enough information for ranking. Unfortunately, this does not fall within the scope of the present invention as the significance of the sub-features ^{depends on the configuration of x 0 .} In order to reflect the nonlinear characteristic of data similarity to the problem of the present invention, a deep neural network that learns the similarity between two batch features can be trained.

Regarding the dataset, similar pairs (x ⁰ ,x ⁺ ) were obtained from the user preference survey, but less similar pairs (x ⁰ ,x ^- ) are needed for training. In order to generate different features x ⁻ in the scene, as shown in FIG. 7 , one can generate a random arrangement in the scene and calculate the features corresponding to that arrangement. The figure on the left of Fig. 7 shows the placement of Person X, and the figure on the right of Fig. 7 shows the user-selected similar arrangement (brown color) of Avatar X' and the created less similar arrangement of Avat X'.

A random batch sample that is too close to one of the user-selected positions with respect to distance, angle and pose may be discarded. Random placement can be performed over the entire room space for 100 samples and placed close to 10 additional samples selected by the user to create challenging and different data. So for each x ⁰ we have 10 positive data x ⁺ and 110 negative data x ^- , which can make a total of 1100 tuples (x ⁰ , x ⁺ , x ^- ) in training.

The present invention can provide a neural network that learns the nonlinear characteristics of the difference (or distance) between two places. Here, the neural network may be trained based on a triplet match network (MatchNet) framework.

Triplet MatchNet can learn the difference between input features in a supervised learning method as shown in FIG. 8A. In the training phase, the model receives a dataset of three types of input {x}:(x ⁰ , x ⁺ , x ^- ), then the Triplet MatchNet inputs to predict that ^{x +} is ^{more similar to x 0} than x ^- Learn the differences between features. The differences can be ranked. A typical Triplet MatchNet model consists of FeatureNet for feature abstraction and MetricNet for difference estimation. Triplet MatchNet sets the distance d ⁺ between similar pairs (x ⁰ , x ⁺ ) to low (e.g. 0) and the distance d ^{- between different pairs (x 0} , x ^- ) to high (e.g. 1 ) can be optimized to push At the same time, we use the comparison between the two distances so that ^{d + is less than d -.} In the present invention, two optimization goals can be achieved with the loss functions φ({x}) and ψ({x}), and the two loss functions can be expressed as <Equation 4> and <Equation 5> below.

[Equation 4]

[Equation 5]

Here, the total loss function may be defined as φ({x})+ψ({x}).

The present invention slightly extends the general Triplet MatchNet architecture to provide additional hints for neural networks that can better estimate differences for the problem of the present invention. In addition to the triplet input, our model can be used as an additional input by explicitly calculating the ^{difference |x 0} -x ^{± | between two features.} 8B shows the overall structure of the network model, and an explicit distance input may be transmitted through another FeatureNet called Distance FeatureNet.

The neural network model of the present invention may be composed of two basic modules, a black box module (BBM) and a sub-feature processing module (SFPM), as shown in FIG. 8C . SFPM takes one of the input feature sets, χ ∈ {x ⁰ , x ⁺ , x ^- , |x ⁰ -x ⁺ |, |x ^{0 -x -} |}, and converts the _{input into subfeatures x ip} , x of the same type. Split into _pa , x _va , x _ss , and x _{sp .} The high-order sub-features of the local correlations x _pa , x _va and x _{sp can be processed through a separate BBM.} Then, the output of the BBM and the low-dimensional sub-features x _ip and x _ss is linked to become the final output of the SFPM. Two SFPMs are used as the FeatureNet and Distance FeatureNet at the bottom, and a simple BBM is used as the MetricNet at the top.

The model of the present invention has five types of BBM, and the size of the layer may be as shown in <Table 1> below, and the present invention can compare BBM with SFPM and FeatureNet (BBM FeatureNet), and each layer of BBM is In the SFPM, the same number of cells as the corresponding layer cells can be maintained.

In Table 1, x denotes input cells, h1 and h2 denote cells of a subsequent layer, y denotes output cells, and N denotes 2 in the case of a general Triplet MatchNet and 3 in the case of the model of the present invention. can do.

Place and move your avatar

Based on the learned similarity function between the two placements, it is possible to provide a way to position or move the avatar to best represent the placement or movement of the remote user.

The present invention assumes that the user moves quasi-normally within a small area or walks towards another destination. To express these behaviors, the movement of the avatar is modeled as a finite state machine consisting of three states: static placement, quasi-normal movement, and movement. Fig. 9 shows a finite state machine for avatar movement, wherein in a static placement state, the avatar moves to a position most suitable for placement of a remote user. At the end of deployment, the state transitions to a quasi-normal movement state. A quasi-normal movement state corresponds to a situation in which the user moves slowly within an area, in which case the avatar's position is a compromise between showing the displacement of the user's position in the frame of reference and preventing the avatar from penetrating into the environment. it is decided to When the user starts walking, the avatar state transitions to the moving state, and the present invention provides several methods for preserving various aspects of the meaning of the user's movement. When the user pauses, the avatar state transitions to a static placement state to update the placement. An avatar placement and movement policy according to each state will be described as follows.

Regarding the static placement, the present invention uses a sampling-based optimization method to find the optimal placement of the avatar X' by considering the placement of the person X in different spaces. For brevity, if d(x(pX'|pY,B),x(pX|pY'), which is the difference between the placement of avatar X' and person X, is defined as D(pX',pX), The optimization problem of the invention can be expressed as in Equation 6 below.

[Equation 6]

The present invention solves the optimization by dividing it into two steps. First, the present invention samples each space as a 0.25-meter grid size 2D grid map and takes 24 orientation samples, one for every 15 degrees for each grid. Compare the difference between the shape vector of person X and the feature vector of avatar X' in every batch of samples to find the best sample with the lowest difference in a given grid. Next, starting with the best sample placement, the present invention finds the optimal placement of avatar X' using particle swarm optimization (PSO). In the present invention, inertia w = 0.73 for PSO parameters, compression coefficients c ₁ , c ₂ = 1.49, 10 particles and 10 maximum epochs can be used.

Next, Avatar X' is quickly moved to the optimal placement. To help Person Y perceive this transmission, the placement of avatar X' is interpolated from the old placement to the new placement. It also visualizes the avatar with added transparency during interpolation to indicate that this fast transfer is not the original action taken by the user. When static placement is complete, the avatar's state transitions to quasi-normal movement.

Explaining quasi-normal movement, people tend to change positions gradually rather than standing still, even when talking to the other person or performing tasks in a stationary position. The present invention calls this behavior q-quasi-normal movement. An effective way to express the meaning of the user's movement in quasi-normal movement is to move the position of the avatar to preserve the user's relative displacement with respect to the local reference coordinates. To realize this, we identify the surrounding local area of the user and avatar immediately after static placement and define the relationship between the two areas. When the user makes a quasi-normal movement within the local area, the user's location is mapped to the corresponding location of the avatar in real time.

10 is a diagram illustrating an example of a procedure for creating two local areas. In FIG. 10, yellow indicates the user's local area and blue indicates the corresponding area of the avatar. First, the local area that the user can freely access within a certain distance is obtained by emitting a line from the user's location. This user's local area mesh is then strictly transformed into the frame of reference coordinates of the avatar. Next, collision check is performed between the vertices of the local area mesh and the obstacles in the avatar space, and the collided vertices move toward the center to a collision-free position to define the local area mesh of the avatar. The relationship between two local area points is determined using the bidirectional coordinates of the triangles in the local area mesh. In the method of the present invention, the orientation of the user with respect to the frame of reference is applied directly to the orientation of the avatar. The radius of the local area may be set to 0.7 m for a standing posture and close to 0 for a sitting posture.

In terms of movement, while the user is walking, the avatar is in a moving state of movement. In this state, the avatar moves to convey the meaning of the user's walking. If the user's destination is known in advance, one of the simple but effective solutions is to have the avatar move to the object destination calculated by Equation (6) in relation to the user's destination. However, it is very difficult to accurately predict the user's destination, so the present invention provides three alternatives.

The first method, Walk-In-Place (WIP), is to have the avatar walk in places where the state transitions from quasi-normal movement to movement. This is simply a baseline method of indicating that the avatar is in a mobile state.

The WIP method cannot provide information about the user's destination movement path. Accordingly, the present invention provides a continuous path update (CPU), which is a second method for making the avatar walk to a position corresponding to the intermediate position of the walking user in order to reflect the meaning of the user's walking path. For this purpose, the characteristic of the intermediate position in the path is obtained with respect to the sample point located in the forward direction from the position. Assuming a sample position {pX1,...,pXs} corresponding to the current placement pX of the walking user and a sample position {pX'1,...,pX's} corresponding to the candidate placement pX' of the avatar, two locations can be defined as in <Equation 7> below.

[Equation 7]

Then, the avatar arrangement is determined by solving the optimization problem as shown in Equation 8 below.

[Equation 8]

Here, pX' _prev may mean the placement position of the avatar in the previous time step. The weight w(pX',pX' _prev ) is set in proportion to the difference in position and orientation between _{pX' and pX' prev in} order to determine the optimal placement closer to the previous placement for the continuity of the avatar's path.

Take a sampling-based optimization scheme for real-time performance. Specifically, the present invention not only randomly traverses the entire space but also samples the placement around _{pX' prev at 15 degree intervals.} The path update of Equation (8) may be performed for a predetermined time, for example, every 100 ms.

The last method, redirection (orientation correction), is similar to the second method, but in this method we only sample around _{pX 'prev at 15 degree intervals to ensure that the avatar only turns and does not jump out of space.} This method ensures a natural movement of the avatar rather than conveying intermediate route information.

In all of the strategies described above, the user's pose is applied directly to the user's avatar, so the altered position and orientation of the avatar's origin creates foot skating. A method of removing foot skating artifacts will improve the quality of the avatar motion. While the quasi-normal movement and redirection strategy of the present invention is collision-free, the teleport induced by the static placement and continuous path update strategy may cause the avatar to pass through walls or furniture in the room.

The placement algorithm must run quickly to immediately place the avatar in a real-time application. For this purpose, spatial-specific features such as _{x pa} , x _{sp , and 2D obstacle meshes are precomputed for each space.} Online courses may use multithreading (eg 16 threads) to manipulate static placement and move procedures. Each static batch takes 0.4 seconds (0.3 seconds for grid level optimization, 0.1 seconds for PSO), and path optimization can take around 100ms.

As described above, in the method according to an embodiment of the present invention, in order to provide telepresence as if people living in different places are living together, avatars are arranged at corresponding points in a remote space that preserves the context and meaning of people located in the real space. can do.

In addition, since the method according to an embodiment of the present invention enables immersive communication between users in a remote space, it is possible to reduce the time, cost, and energy consumption to travel to meet in person, and thus increase economic and environmental benefits. have.

In addition, the method according to an embodiment of the present invention enables coexistence and interaction with a user in a remote space in the user's real space by arranging the avatar in the real space.

The method of the present invention may be implemented as an apparatus or a system, and each functional process may be configured as a functional component. For example, the device according to an embodiment of the present invention has a collection unit that collects user preference data, which is placement data preferred by people through a user preference survey, and learns the collected user preference data with a deep neural network-based neural network. and a defining unit defining a degree of similarity between the person and the avatar between corresponding points in the two spaces, and a disposing unit disposing the user's avatar in a remote space based on the similarity between the person and the avatar between the defined corresponding points in the two spaces.

Here, the user preference data includes the relative distance and gaze direction between the person and the avatar, the height value around the place where the person is located, the sum of the furniture types and distances around the place where the person is located, and the furniture types and distances seen in the front view of the person. It may include at least one of sum and whether a person is seated.

The definition unit performs feature modeling including spatial features specifying interaction features between other people or objects, pose acceptance features for a person's pose, and functional features of the surrounding space, and arrangement for two spaces based on the feature modeling By learning the similarity between the features, it is possible to define the similarity between the person and the avatar between the corresponding points for the two spaces.

The neural network may include a neural network that learns a nonlinear characteristic of a difference or distance between the two spaces.

The arrangement unit may model the movement of the avatar as a finite state machine having three states of static placement, quasi-normal movement, and movement, and may place the user's avatar in a remote space using the modeled finite state machine.

When the arrangement of the avatar is finished, the arrangement unit changes the arrangement state of the avatar to a semi-normal moving state, changes the state of the avatar to the moving state when the user starts walking, and changes the state of the avatar to the moving state when the user stops. You can switch to a static batch state.

Of course, the device according to the embodiments of the present invention may include all the contents described with reference to FIGS. 1 to 10 , which will be apparent to those skilled in the art.

The system or apparatus described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, the systems, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA). ), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiments may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (12)

collecting user preference data, which is placement data preferred by people, through a user preference survey;
defining a similarity between a person and an avatar between corresponding points for two spaces by learning the collected user preference data with a deep neural network-based neural network; and
arranging the user's avatar in the remote space based on the degree of similarity between the person and the avatar between corresponding points for the two defined spaces;
A remote space avatar placement method comprising:
According to claim 1,
The user preference data is
Among the relative distance and gaze direction between the person and the avatar, the height value around the place where the person is located, the sum of the furniture types and distances around the place where the person is located, the sum of the furniture types and distances seen in the front view of the person, and whether or not a person is seated A remote space avatar placement method comprising at least one.
According to claim 1,
The step of defining the similarity is
Feature modeling is performed, including spatial features specifying the interaction features between other people or objects, pose acceptance features for poses of people, and functional features of the surrounding space, and based on the feature modeling, it is possible to determine the difference between the arrangement features for two spaces. A remote space avatar arrangement method, characterized in that by learning the degree of similarity, the degree of similarity between the person and the avatar between the corresponding points in the two spaces is defined.
According to claim 1,
The neural network is
and a neural network for learning a nonlinear characteristic of a difference or distance between the two spaces.
According to claim 1,
The arranging step
A remote space, characterized in that the movement of the avatar is modeled as a finite state machine composed of three states of static placement, quasi-normal movement, and movement, and the user's avatar is placed in a remote space using the modeled finite state machine. How to place your avatar.
6. The method of claim 5,
The arranging step
When the arrangement of the avatar is finished, the arrangement state of the avatar is changed to a semi-normal moving state, the state of the avatar is changed to a moving state when the user starts walking, and when the user stops, the state of the avatar is changed to a static arrangement state A remote space avatar placement method, characterized in that switching to .
a collection unit for collecting user preference data, which is placement data preferred by people through a user preference survey;
a definition unit for learning the collected user preference data with a deep neural network-based neural network to define a degree of similarity between a person and an avatar between corresponding points for two spaces; and
An arrangement unit for disposing the user's avatar in a remote space based on the degree of similarity between the person and the avatar between corresponding points for the two defined spaces.
A remote space avatar placement device comprising a.
8. The method of claim 7,
The user preference data is
Among the relative distance and gaze direction between the person and the avatar, the height value around the place where the person is located, the sum of the furniture types and distances around the place where the person is located, the sum of the furniture types and distances seen in the front view of the person, and whether or not a person is seated A remote space avatar placement device comprising at least one.
8. The method of claim 7,
the definition section
Feature modeling is performed, including spatial features specifying the interaction features between other people or objects, pose acceptance features for poses of people, and functional features of the surrounding space, and based on the feature modeling, it is possible to determine the difference between the arrangement features for two spaces. A remote space avatar arrangement apparatus, characterized in that by learning the degree of similarity, the degree of similarity between the person and the avatar between the corresponding points for the two spaces is defined.
8. The method of claim 7,
The neural network is
and a neural network for learning a nonlinear characteristic of a difference or distance between the two spaces.
8. The method of claim 7,
the arrangement part
A remote space, characterized in that the movement of the avatar is modeled as a finite state machine composed of three states of static placement, quasi-normal movement, and movement, and the user's avatar is placed in a remote space using the modeled finite state machine. Avatar placement device.
12. The method of claim 11,
the arrangement part
When the arrangement of the avatar is finished, the arrangement state of the avatar is changed to a semi-normal moving state, the state of the avatar is changed to a moving state when the user starts walking, and when the user stops, the state of the avatar is changed to a static arrangement state A remote space avatar placement device, characterized in that it converts to .

Images