KR102650349B1 - Extended reality device and method for controlling the same - Google Patents

Abstract

The present invention relates to an XR device and its control method, and more specifically, is applicable to the 5G communication technology field, robot technology field, autonomous driving technology field, and AI (Artificial Intelligence) technology field.

Description

XR device and its control method {EXTENDED REALITY DEVICE AND METHOD FOR CONTROLLING THE SAME}

The present invention relates to an XR device and its control method, and more specifically, is applicable to the 5G communication technology field, robot technology field, autonomous driving technology field, and AI (Artificial Intelligence) technology field.

VR (Virtual Reality) technology provides objects and backgrounds in the real world only as CG (Computer Graphics) images, while AR (Augmented Reality) technology provides virtually created CG images on top of images of real objects, and MR (Mixed Reality) technology provides ) technology is a computer graphics technology that mixes and combines virtual objects in the real world. The aforementioned VR, AR, MR, etc. are all simply referred to as XR (extended reality) technologies.

In particular, recently, virtual services have been applied to users using XR technology. For example, XR devices installed in stores or at home display virtual avatars on the screen.

However, the screen of the XR device installed in the store is a large screen, and the camera used to create the virtual avatar is installed on one side of the screen. Therefore, it was impossible to photograph the user's face from various angles using only the camera installed on the screen of the XR device. Additionally, if the user's face cannot be captured accurately, there is a problem that the virtual avatar's face is very different from the actual user's face.

Of course, you can consider using multiple cameras or a camera with a wide angle of view, but there is a problem that the cost increases exponentially.

One embodiment of the present invention provides technology for accurately recognizing the user's face regardless of the size of the screen of the XR device.

Another embodiment of the present invention provides a technology that can capture a user's face from multiple angles using only one camera installed at an arbitrary location on an XR device.

In addition, another embodiment of the present invention seeks to minimize the possibility of errors that may occur in the process of creating a virtual avatar on an XR device.

However, it is not limited to the above-described purpose, and the scope of the present invention may be expanded to other purposes that can be inferred by a person skilled in the art based on the entire contents of this specification.

A method of controlling an XR device that provides at least one virtual avatar according to an embodiment of the present invention to achieve the above-described object, etc. includes the steps of recognizing a user located around the XR device using a camera, Displaying at least one real object and a first virtual avatar around the XR device, generating an indicator for creating a second virtual avatar, and displaying the generated indicator to the XR device. displaying at a first position within the screen, and moving the indicator from the first position to a second position (the second position is different from the first position and is located around the camera within the screen). ) and, while the indicator moves from the first position to the second position, photographing the user using the camera, and based on the photographed result, the at least one real object and displaying the second virtual avatar.

And, an XR device that provides at least one virtual avatar according to an embodiment of the present invention includes a camera that recognizes a user located around the XR device, and at least one real object around the XR device. and a display module that displays a first virtual avatar, and a controller that controls the camera and the display module.

For example, the controller generates an indicator for creating a second virtual avatar, controls the generated indicator to be displayed at a first position in the screen of the XR device, and moves the indicator from the first position to a second position. position (the second position is different from the first position and is located around the camera in the screen), and while the indicator moves from the first position to the second position, the camera Controls to photograph the user using and controls to display the at least one real object and the second virtual avatar based on the photographed results.

According to one of the various embodiments of the present invention, it is possible to provide technology for accurately recognizing the user's face regardless of the size of the screen of the XR device.

According to another embodiment of the present invention, there is an advantage in providing a technology that can photograph a user's face from multiple angles with only one camera installed at an arbitrary location on an XR device.

And, according to another embodiment of the present invention, there is a technical effect of minimizing the possibility of errors that may occur in the process of creating a virtual avatar on an XR device.

However, it is not limited to only the technical effects described above, and the scope of the present invention may be expanded to other technical effects that can be inferred by a person skilled in the art based on the entire contents of this specification.

Figure 1 shows a case where an XR device according to an embodiment of the present invention is implemented as an HMD type.
Figure 2 shows a case where the XR device according to an embodiment of the present invention is implemented as an AR glass type.
Figure 3 shows a case where an embodiment of the present invention is applied to a clothing-related device.
Figure 4 shows an example of a flow chart illustrating a process for controlling an XR device according to an embodiment of the present invention.
Figure 5 shows the appearance of an XR device and an installed camera according to an embodiment of the present invention.
Figure 6 shows a process in which an XR device generates the face of a virtual avatar from one image according to an embodiment of the present invention.
Figure 7 shows a plurality of images acquired from various angles by an XR device according to an embodiment of the present invention.
Figure 8 shows a process in which an XR device induces the user's facial posture according to an embodiment of the present invention.
FIG. 9 illustrates a process in which an XR device according to an embodiment of the present invention creates a face of a virtual avatar using a plurality of images acquired in FIG. 8.
Figure 10 shows a problem depending on the location of a user away from the XR device according to an embodiment of the present invention.
FIG. 11 shows an example of a solution to solve the problem shown in FIG. 10.
FIG. 12 shows another example of a solution for solving the problem shown in FIG. 10.
FIG. 13 shows another example of a solution for solving the problem shown in FIG. 10.
Figure 14 shows a process in which an XR device guides the user's gaze according to an embodiment of the present invention.
Figure 15 shows a process for solving a problem when the XR device according to an embodiment of the present invention changes from the direction in which the user's gaze was guided.
Figure 16 shows a process by which an XR device guides the user's gaze for more than a certain period of time according to an embodiment of the present invention.
Figure 17 shows a process for solving a problem when the XR device according to an embodiment of the present invention does not match the direction in which the user's gaze and upper body face.
Figure 18 shows a process in which an XR device confirms the face of a created avatar according to an embodiment of the present invention.
Figure 19 shows a process for an XR device to capture a specific part of a user's face in more detail according to an embodiment of the present invention.
Figure 20 shows a process for the XR device to more accurately capture a specific part of the user's face according to an embodiment of the present invention.
And, Figure 21 shows an example of a flow chart showing a process for controlling an XR device according to an embodiment of the present invention.

Hereinafter, embodiments disclosed in the present specification will be described in detail with reference to the attached drawings. However, identical or similar components will be assigned the same reference numbers regardless of reference numerals, and duplicate descriptions thereof will be omitted. The suffixes “module” and “part” for components used in the following description are given or used interchangeably only for the ease of preparing the specification, and do not have distinct meanings or roles in themselves. Additionally, in describing the embodiments disclosed in this specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed descriptions will be omitted. In addition, the attached drawings are only for easy understanding of the embodiments disclosed in this specification, and the technical idea disclosed in this specification is not limited by the attached drawings, and all changes included in the spirit and technical scope of the present invention are not limited. , should be understood to include equivalents or substitutes.

Of course, the following examples of the present invention are only intended to embody the present invention and do not limit or limit the scope of the present invention. Anything that can be easily inferred by an expert in the technical field to which the present invention belongs from the detailed description and embodiments of the present invention will be interpreted as falling within the scope of the rights of the present invention.

The above detailed description should not be construed as limiting in any respect and should be considered illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Extended reality described in this specification collectively refers to virtual reality (VR), augmented reality (AR), and mixed reality (MR). VR technology provides objects and backgrounds in the real world only as CG images, AR technology provides virtual CG images on top of images of real objects, and MR technology provides computer technology that mixes and combines virtual objects in the real world. It is a graphic technology.

MR technology is similar to AR technology in that it shows real objects and virtual objects together. However, in AR technology, virtual objects are used to complement real objects, whereas in MR technology, virtual objects and real objects are used equally.

XR technology can be applied to HMD (Head-Mount Display), HUD (Head-Up Display), mobile phones, tablet PCs, laptops, desktops, TVs, digital signage, etc., and devices with XR technology applied are called XR Devices. It can be called.

However, recently, rather than clearly distinguishing between VR, AR, and MR technologies, they are sometimes referred to as XR (extended reality) technologies. Accordingly, embodiments of the present invention are applicable to all VR, AR, MR, and XR technologies.

Meanwhile, hardware (HW)-related element technologies applied to VR, AR, MR, and XR technologies include, for example, wired/wireless communication technology, input interface technology, output interface technology, and computing device technology. In addition, software (SW)-related element technologies include, for example, tracking and matching technology, voice recognition technology, interaction and user interface technology, location-based service technology, search technology, and AI (Artificial Intelligence) technology.

In particular, embodiments of the present invention use the above-described HW/SW related element technologies, etc. to solve problems such as communication problems with other devices, efficient memory use problems, slow data processing speed due to inconvenient UX/UI, and video problems. , it seeks to solve at least one of the following: acoustic problems, motion sickness, or other problems.

Figure 1 shows a case where an XR device according to an embodiment of the present invention is implemented as an HMD type. Various embodiments to be described later may be implemented with the HMD type shown in FIG. 1.

The HMD type XR device 100a shown in FIG. 1 includes a communication unit 110, a control unit 120, a memory unit 130, an I/O unit 140a, a sensor unit 140b, and a power supply. Includes unit 140c, etc. In particular, the communication unit 110 in the XR device 10a establishes wired and wireless communication with the mobile terminal 100b.

And, Figure 2 shows a case where the XR device according to an embodiment of the present invention is implemented as an AR glass type. Various embodiments to be described later can also be implemented with the AR glass type shown in FIG. 2.

As shown in FIG. 2, AR glasses may include a frame, a control unit 200, and an optical display unit 300.

As shown in FIG. 2, the frame may have the form of glasses worn on the face of the user 10's body, but is not necessarily limited thereto, and may be in the form of goggles, etc., worn in close contact with the face of the user 10. You can also have

Such a frame may include a front frame 110 and first and second side frames.

The front frame 110 has at least one opening and may extend in a first horizontal direction (x), and the first and second side frames extend in a second horizontal direction (y) that intersects the front frame 110. They can be extended and extended parallel to each other.

The control unit 200 may generate an image to be shown to the user 10 or an image containing a series of images. Such a control unit 200 may include an image source that generates an image and a plurality of lenses that diffuse and converge light generated from the image source. In this way, the image generated by the control unit 200 may be output to the optical display unit 300 through the guide lens P200 located between the control unit 200 and the optical display unit 300.

This control unit 200 may be fixed to either the first or second side frames. For example, the control unit 200 may be fixed to the inside or outside of one of the side frames, or may be built into the inside of one of the side frames and formed integrally.

The optical display unit 300 may play a role in allowing the image generated by the control unit 200 to be displayed to the user 10, and while allowing the image to be displayed to the user 10, the external environment can be viewed through the opening. In order to do so, it may be formed of a translucent material.

This optical display unit 300 is inserted into and fixed to the opening included in the front frame 110, or is located on the back of the opening (i.e. between the opening and the user 10) and fixed to the front frame 110. It can be provided. In the present invention, as an example, a case where the optical display unit 300 is located on the back of the opening and is fixed to the front frame 110 is shown as an example.

As shown in FIG. 2, in such an XR device, when the control unit 200 makes an image incident on the incident area (S1) of the optical display unit 300, the image light passes through the optical display unit 300. , the image generated by the control unit 200 can be displayed to the user 10 by being emitted to the emission area S2 of the optical display unit 300.

Accordingly, the user 10 can view the external environment through the opening of the frame 100 and simultaneously view the image generated by the control unit 200.

Figure 3 shows a case where an embodiment of the present invention is applied to a clothing-related device.

Embodiments of this specification can be applied not only to XR devices but also to various clothing-related devices.

To further explain in relation to an embodiment of the present invention, as shown in FIG. 3, when the user 100 approaches a clothing-related device (ex: styler, air dresser, etc.), a camera or sensor installed in the clothing-related device Recognizes user access.

Additionally, the display 200 installed on the front of the clothing-related device displays an avatar related to the recognized user, and additionally displays graphics showing the user 100 virtually trying on the desired clothing, hat, etc. As shown in FIG. 3, it can be confirmed that the actual user 100 is not wearing a hat, but the avatar displayed on the display 200 is wearing a virtual hat. Furthermore, the display 200 can also operate as a mirror when the user is not recognized.

Lastly, in FIG. 3, it is assumed that the display 200 is exposed to the front of the clothing-related device, but the present invention is also designed so that the display is located inside when the user 100 opens the door of the clothing-related device. falls within the scope of rights.

Meanwhile, although this specification describes XR devices and various multimedia devices as examples, any device with a display function is sufficient for the present invention.

Figure 4 shows an example of a flow chart illustrating a process for controlling an XR device according to an embodiment of the present invention.

As shown in FIG. 4, according to one embodiment of the present invention, an XR device providing at least one virtual avatar operates in the following order. Of course, it is also possible for some steps to be changed, added, or deleted according to the needs of those skilled in the art.

The XR device uses a camera to recognize users located around the XR device (S510). The XR device can be installed, for example, in an offline store where clothes are sold, but can also be installed in other places.

The XR device displays at least one real object and a first virtual avatar around the XR device (S520). However, the first virtual avatar here means, for example, a state in which the user's face around the XR device is not reflected.

The XR device generates an indicator for creating a second virtual avatar (S530), and displays the generated indicator at the first location on the screen of the XR device (S540).

The XR device moves the indicator from the first position to the second position (S550). The second location is different from the first location and is located around the camera within the screen. This will be described in detail later with reference to FIG. 8 and the like.

The XR device photographs the user using the camera while the indicator moves from the first position to the second position (S560), and based on the photographed result, the at least one actual The object and the second virtual avatar are displayed (S570). However, the second virtual avatar here is different from the first virtual avatar described above. The second virtual avatar means, for example, a state in which the user's face around the XR device is reflected.

Although not shown in FIG. 4, the XR device determines whether the user is separated from the XR device by a predetermined distance or more. Furthermore, as a result of the determination, if the distance is greater than the preset distance, a graphic image related to the reference location is displayed together with the first virtual avatar. The first virtual avatar is characterized in that it moves according to the user's movement. In relation to this, it will be described later with reference to FIGS. 10 to 12.

The preset constant distance is characterized in that it changes depending on the screen size of the XR device, for example.

Furthermore, the XR device determines whether the user's eyes or face are facing the indicator while the indicator moves from the first position to the second position, and as a result of the determination, the user's eyes or face If it is not pointing towards the indicator, change the size or color of the indicator. This will be described later with reference to FIG. 15.

Furthermore, the XR device displays time information or status information necessary for face scanning only while the indicator is located in the second position. This will be described in more detail below with reference to FIG. 16.

Additionally, the XR device compares a first vector based on the user's face with a second vector based on the user's upper body only while the indicator is located at the second position. Furthermore, the XR device displays a message guiding the user to move his or her upper body only when the first vector and the second vector differ by more than a preset angle. This will be described later with reference to FIG. 17.

Additionally, only while the indicator is located in the second position, the XR device displays a graphic image corresponding to the specific part of the user's face to be analyzed. The specific part includes, for example, at least one of the user's eyes, nose, or mouth. This will be described in more detail in Figure 19 below.

In addition, the color of the outermost line of the graphic image when the analysis of the specific part of the face fails and the color of the outermost line of the graphic image when the analysis of the specific part of the face is successful are different from each other. do. This will be described later with reference to FIG. 20.

Meanwhile, although the method invention is explained in Figure 4, it can also be implemented as a device invention. The XR device according to an embodiment of the present invention includes a camera, a display module, and a controller, as previously described in FIGS. 1 and 2, etc.

The camera recognizes users located around the XR device.

The display module displays at least one real object and a first virtual avatar around the XR device.

And, the controller controls the camera and the display module.

More specifically, for example, the controller generates an indicator for creating a second virtual avatar, controls the generated indicator to be displayed at a first location on the screen of the XR device, and displays the indicator at the first location. to a second position (the second position is different from the first position and is located around the camera in the screen), while the indicator moves from the first position to the second position. , controlling to photograph the user using the camera, and controlling to display the at least one real object and the second virtual avatar based on the photographed result.

Figure 5 shows the appearance of an XR device and an installed camera according to an embodiment of the present invention.

As shown in FIG. 5, the XR device 600 according to an embodiment of the present invention is equipped with a camera 610 on the side, and uses the camera 610 to recognize nearby users 620. do. Through this, it is possible to create a virtual avatar similar to the face of the user 620.

However, the larger the screen size of the XR device 600, the larger the camera 610 must be physically located at a considerable distance from the center point of the screen. Since the user 620 typically gazes at the center of the screen of the XR device 600, there was a problem in that the front view and various angles of the user 620 could not be photographed with only the single camera 610. In this situation, the face of the user 620 cannot be accurately detected, which raises the problem that the face of the virtual avatar is very different from the face of the actual user 620.

Figure 6 shows a process in which an XR device generates the face of a virtual avatar from one image according to an embodiment of the present invention.

For example, as shown in (a) of FIG. 6, assume that the user's frontal face has been photographed. Using the image shown in (a) of FIG. 6, a 3D face model is created and a rendering task is performed, as shown in (b) of FIG. 6. As a result, as shown in (c) of FIG. 6, a virtual avatar face is created.

However, in order to create a virtual avatar that is closest to the user's actual face, it is very important to capture the user's frontal face, and it is also important to capture the face from various angles as possible.

Figure 7 shows a plurality of images acquired from various angles by an XR device according to an embodiment of the present invention.

Figure 7 assumes that a camera is installed on the side of the XR device (see previous Figure 5). Therefore, when the user gazes at the center point of the screen of the XR device, the image shown in (a) of FIG. 7 is obtained, and the image shown in (b) of FIG. 7 is obtained only when the user gazes at the camera installed on the side of the XR device. It is possible to obtain. The XR device according to an embodiment of the present invention proposes a solution for acquiring various images shown in FIG. 7 as quickly as possible, more accurately, and without user resistance. This will be described in more detail below in FIG. 8.

Figure 8 shows a process in which an XR device induces the user's facial posture according to an embodiment of the present invention.

As shown in FIG. 8, the angle of the user's face located around the XR device is controlled using a specific indicator (ex: Red Dot). In order to create a virtual avatar that is close to the user's face, we want to naturally induce the facial posture of the user located around the XR device.

First, as shown in (a) of FIG. 8, the XR device according to an embodiment of the present invention uses the camera 930 to determine whether a nearby user 901 exists. Of course, this function can also be implemented using an object proximity sensor other than a camera.

When it is recognized that a nearby user 901 exists, the XR device displays the first virtual ataba 900 and the indicator 920 at the first location. However, the first virtual avatar 900 is an arbitrary graphic image previously stored in memory, and does not reflect the face of the user 901.

Furthermore, a guide message 910 that allows the user's face to move naturally is displayed.

Then, the XR device slowly moves the indicator to the second position 921, and the face of the user 901 also moves naturally. In particular, by designing the second location to be located around the camera 930, the camera 930 is capable of capturing the front face of the user 901.

Figure 8 (a) assumes that the camera 930 is located on the left side of the XR device, but as shown in Figure 8 (b), even if the camera 950 is located on the right side of the XR device One embodiment of the present invention can be applied.

As shown in (b) of FIG. 8, the position 940 where the indicator is first displayed is the same as (a) in FIG. 8, but the point to which it is finally moved (941) is different from (a) in FIG. 8. That is, the indicator moves to the midpoint 941 on the right side of the screen of the XR device, which is the location of the camera 950 installed in the XR device, and thus it is possible to capture the front of the user's face.

Furthermore, Figure 8(a) assumes that the camera 930 is located on the left side of the XR device, but as shown in Figure 8(c), the camera 970 is located at the top center of the XR device. Even in this case, an embodiment of the present invention can be applied.

As shown in (c) of FIG. 8, the position 960 where the indicator is first displayed is the same as (a) of FIG. 8, but the point to which it is finally moved (961) is different from (a) of FIG. 8. That is, the indicator moves to the upper central point 961 of the screen of the XR device, which is the location of the camera 970 installed in the XR device, and thus it is possible to capture the front of the user's face.

FIG. 9 illustrates a process in which an XR device according to an embodiment of the present invention creates a face of a virtual avatar using a plurality of images acquired in FIG. 8.

As shown in FIG. 8, when the direction of the user's face gradually changes, the XR device is capable of capturing images in various facial directions of the user, as shown in (a) of FIG. 9.

Therefore, as shown in (b) of FIG. 9, it is possible to obtain more detailed facial texture information, and generate the 3D virtual avatar face shown in (c) of FIG. 9 through a rendering process.

However, the previously described embodiments assumed that the distance between the user located around the XR device and the XR device was constant. However, in reality, the distance between the XR device and the user may vary. Even in this case, if the position where the indicator is displayed is the same, a problem arises in which the user's face from various angles and the user's face from the front cannot be obtained. Specific scenarios of related problems will be described later in Figure 10, and various solutions to solve them will be described in more detail in Figures 11 to 13 below.

Figure 10 shows a problem depending on the location of a user away from the XR device according to an embodiment of the present invention.

As shown in FIG. 10 , arbitrary users may be standing at various distances from the XR device 1100.

For example, when the user is located very close to or very far from the XR device 1100, it is impossible to accurately capture the user's face from various angles. In particular, when the user is located far away from the XR device 1100, it is impossible to capture the user's face from various angles just by moving the indicator on the screen of the XR device.

We present two solutions to solve this. The solution presented in FIGS. 11 and 12 is a method of guiding the user to move to a certain location, and the solution presented in FIG. 13 is a method of guiding the user to look at a specific object among real objects outside the screen of the XR device. am.

FIG. 11 shows an example of a solution to solve the problem shown in FIG. 10.

As shown in (a) of FIG. 11, when the XR device 1200 and the user 1210 are far away, the movement of the indicator on the screen of the XR device 1200 and the corresponding movement of the user's face can be used at various angles. The user's face cannot be captured.

Therefore, in order to solve the problem shown in (a) of FIG. 11, a solution is required that allows the user to move from the first point 1230, which is far away, to the second point 1231, which is closer to the XR device. A specific method related to this will be described later with reference to FIG. 12.

FIG. 12 shows another example of a solution for solving the problem shown in FIG. 10.

The avatar shown in FIG. 12 is a concept corresponding to the first avatar described previously, and the user's face is not yet reflected.

However, the first avatar is also displayed at a specific point 1320 corresponding to the point 1310 where the user is first recognized. And, according to the user's movement, the first avatar also moves within the screen of the XR device.

Accordingly, if it is impossible to capture the user's various faces at the point 1310 where the user is first recognized by simply operating the above-described indicator, the first avatar moves to the reference position 1321. Therefore, it is possible to guide the user to move to the target point 1311 as well.

However, the solutions of FIGS. 11 and 12 have the limitation that the user must move to the reference position, and there is a possibility that the user may not move to the exact reference position. A solution to solve this problem will be described later in FIG. 13.

FIG. 13 shows another example of a solution for solving the problem shown in FIG. 10.

First, it is assumed that the XR device has specific photo information in the direction from which the user is looking at the XR device.

In particular, if the user is located remotely from the XR device and cannot photograph the user from various angles just by looking at a specific point on the screen of the XR device, the XR device 1400 may use the above-described specific photo information 1410 displays.

Accordingly, the user does not look at the indicator on the screen of the XR device, but looks at the actual object 1420 at the corresponding point. Through this, it is possible to photograph the user's face from more diverse angles.

According to the above-described embodiments, the user's face also moves according to the movement of the indicator on the screen of the XR device. However, unlike the user's facial movement, there is a possibility that the user's gaze may move.

A solution to solve this problem will be described later with reference to FIGS. 14 and 15.

Figure 14 shows a process in which an XR device guides the user's gaze according to an embodiment of the present invention.

As shown in FIG. 14, when the XR device 1500 recognizes the presence of the user 1510, it displays the first virtual avatar 1520 and displays the user's face by moving an indicator (ex: red dot). Guide you to move.

However, by displaying an additional message 1530 to direct the user's gaze toward the camera, it is possible to accurately capture the user's eyes appropriate for the direction of the user's face.

However, the embodiment of FIG. 14 has a problem in that the face direction and gaze direction match only when the user is looking at the camera. A solution to solve this problem is described later in FIG. 15.

Figure 15 shows a process for solving a problem when the XR device according to an embodiment of the present invention changes from the direction in which the user's gaze was guided.

It is assumed that the XR device shown in FIG. 15 has an eye tracking function applied.

As shown in (a) of FIG. 15, the user's gaze 1600 is located within a certain error range according to the movement of the indicator 1610. Furthermore, providing a feedback effect to the user by displaying the user's gaze 1600 on the screen of the XR device also falls within the scope of the present invention.

However, as shown in (b) of FIG. 15, when the user's gaze 1601 deviates from the preset error range of the indicator 1611, the indicator size is changed significantly as shown in (c) of FIG. 15. Do (1621). Therefore, it is possible to guide the user's gaze back to the indicator within the shortest possible time. Of course, in Figure 15 (c), the size of the indicator is enlarged, but changing the color of the indicator or outputting warning audio also falls within the scope of the present invention.

However, there may be cases where the user's face is not in a normal state at the moment the user's frontal face is photographed, and a solution to solve this problem will be described later in FIG. 16.

Figure 16 shows a process by which an XR device guides the user's gaze for more than a certain period of time according to an embodiment of the present invention.

Because users around the XR device may blink, multiple photos are taken during a preset time (3 seconds), and a virtual avatar is designed to be created from a single photo with no change in facial expression.

As shown in (a) of FIG. 16, when the indicator is located around the camera, a counting number 1710 is displayed to guide the user to maintain the facial expression. Accordingly, the user can recognize that the facial expression must be maintained for the corresponding number of seconds.

Furthermore, displaying a bar indicating the face scanning status in a circular (1720) type also falls within the scope of rights of one embodiment of the present invention.

Therefore, the XR device is based on the image in a normal state ((d) in Figure 16) with the least change in facial expression among the multiple images ((b), (c), and (d) in Figure 16) captured during the corresponding time. It is possible to create a virtual avatar. FIG. 17 shows a process for solving a problem when the XR device according to an embodiment of the present invention does not match the direction in which the user's gaze and upper body are facing.

As shown in (a) of FIG. 17, when the indicator is located near the camera of the XR device, the user's gaze is also directed toward the indicator, so it is possible to capture the frontal face.

However, unlike the user's gaze, there is a possibility that the user's upper body is not facing the indicator, and in this case, there is a problem that the virtual avatar also differs from the user's actual appearance.

Therefore, when the direction of the user's face and the direction of the user's upper body do not match, a graphic image 1800 is displayed to guide the user's upper body to match the direction of the face.

The solution for determining this is shown in Figures 17 (b) and (c).

As shown in (b) of FIG. 17, the XR device according to an embodiment of the present invention compares a first vector 1810 based on the user's face and a second vector 1820 based on the user's upper body. Only when the first vector and the second vector differ by more than a preset angle (for example, 15 degrees), the graphic image 1800 shown in (a) of FIG. 17 or the user's upper body is moved. Designed to display guiding messages.

On the other hand, as shown in (c) of FIG. 17, when the first vector based on the user's face and the second vector based on the user's upper body match or differ by at least less than the preset angle, FIG. The graphic image 1800 shown in (a) of 17 is not displayed.

As described above, it is possible to display the second virtual avatar based on the user's image information photographed and captured from various angles. It has been described above that unlike the first virtual avatar, the second virtual avatar reflects the user's facial information. However, despite the above-described embodiments, there may be cases where there is a difference between the second virtual avatar and the user's actual face, and a method for confirming this will be described later with reference to FIG. 18.

Figure 18 shows a process in which an XR device confirms the face of a created avatar according to an embodiment of the present invention.

The second virtual avatar created according to the above-described embodiments is displayed as shown in FIG. 18.

Furthermore, as the face of the user 1900 rotates, the second virtual avatar 1910 is also rotated in the same direction.

The user 1900 checks both the front and side views of the second virtual avatar 1910 and, if it is confirmed that it is similar to the user, selects the option 1940 to apply the virtual avatar. This can be selected based on the user's gesture, and it is assumed that the XR device has a motion detection function.

On the other hand, the user 1900 checks both the front and side views of the second virtual avatar 1910 and, if it is confirmed that it is not similar to the user, selects the option 1930 to try taking a picture again without applying the virtual avatar. do. This can be selected based on the user's gesture, and it is assumed that the XR device has a motion detection function.

Figure 19 shows a process for an XR device to capture a specific part of a user's face in more detail according to an embodiment of the present invention.

As shown in (a) of FIG. 19, it has been described above that the XR device can capture the user's frontal face by gazing at the indicator 2010 around the camera of the XR device. However, if the XR device does not provide feedback to the user about which part of the user's face it is capturing, the user has no choice but to focus their attention on the entire face.

To solve this problem, as shown in (b) of FIG. 19, a guide message indicating that the user's face outline 2020 is detected is displayed in the first step. Therefore, at this time, there is no problem even if the user moves the area within the face comfortably.

Furthermore, as shown in (c) of FIG. 19, a guide message indicating that the user's eyes are detected is displayed in the second step. Therefore, at this time, the user must keep his eyes properly open, but other parts of the face other than the eyes can be moved comfortably.

Finally, as shown in (d) of FIG. 19, a guide message indicating that the user's mouth is detected is displayed in the third step. Therefore, at this time, the user must keep the mouth closed, but other parts of the face other than the mouth can be moved comfortably.

However, there is a problem that the embodiment of Figure 19 alone does not know how long the user should maintain the corresponding facial expression. A solution to solve this problem will be described later in Figure 20. Figure 20 shows a process for an XR device to more accurately capture a specific part of a user's face according to an embodiment of the present invention.

As shown in (a) of FIG. 20, when recognition of a specific part of the user's face is successful, a graphic image corresponding to the specific part is displayed as an outline 2110 of the first color.

On the other hand, as shown in (b) of FIG. 20, when recognition of a specific part of the user's face fails (or the eyes are frowned), a graphic corresponding to the specific part is displayed as an outline 2120 in a second color. Display the image.

And, Figure 21 shows an example of a flow chart showing a process for controlling an XR device according to an embodiment of the present invention.

The XR device according to an embodiment of the present invention determines whether the indicator is located at the second location, which is the final target location (S2210). As described above, the second location corresponds, for example, to the vicinity of the in-screen camera of the XR device.

As a result of the determination, if the indicator is located at the second position, the XR device compares a first vector based on the user's face and a second vector based on the user's upper body (S2220).

Additionally, the XR device determines whether the first vector and the second vector have a difference of more than a preset angle (S2230), and displays a guide message only if the difference is more than the preset angle (S2240). This has been previously described in detail in FIG. 17.

110: Communication unit
120: Control unit
130: Memory unit
140a: I/O unit
140b: Sensor unit
140c: Power supply unit

Claims (20)

In a method of controlling an XR device providing at least one virtual avatar,
Recognizing a user located around the XR device using a camera;
Displaying at least one real object and a first virtual avatar around the XR device;
Creating an indicator for creating a second virtual avatar;
Displaying the generated indicator at a first location on the screen of the XR device;
moving the indicator from the first position to a second position, wherein the second position is different from the first position and is located around the camera in the screen;
While the indicator moves from the first position to the second position, photographing the user using the camera;
Based on the photographed result, displaying the at least one real object and the second virtual avatar;
Determining whether the user is separated from the XR device by a predetermined distance or more; and
As a result of the determination, if the distance is greater than the predetermined distance, displaying a graphic image related to the reference location together with the first virtual avatar.
Includes,
The first virtual avatar moves according to the user's movement, and the preset distance changes according to the screen size of the XR device. delete delete According to paragraph 1,
determining whether the user's eyes or face are facing the indicator while the indicator moves from the first position to the second position; and
If, as a result of the determination, the user's eyes or face are not facing the indicator, changing the size or color of the indicator
A control method of an XR device including. According to paragraph 4,
Displaying time information or status information required for face scanning only while the indicator is located at the second position.
A control method of an XR device including. According to clause 5,
Comparing a first vector based on the user's face and a second vector based on the user's upper body only while the indicator is located at the second position
A control method of an XR device including. According to clause 6,
Only when the first vector and the second vector differ by more than a preset angle,
Displaying a message guiding the user to move his/her upper body
A control method of an XR device including. In clause 7,
Displaying a graphic image corresponding to a specific part of the user's face to be analyzed only while the indicator is located in the second position.
A control method of an XR device including. According to clause 8,
The specific part above is,
A method of controlling an XR device including at least one of the user's eyes, nose, or mouth. According to clause 9,
XR, characterized in that the color of the outermost line of the graphic image when the analysis of a specific part of the face fails and the color of the outermost line of the graphic image when the analysis of the specific part of the face is successful are different from each other Device control method. In an XR device providing at least one virtual avatar,
A camera that recognizes a user located around the XR device;
A display module that displays at least one real object and a first virtual avatar around the XR device; and
Including a controller that controls the camera and the display module,
The controller is,
Create an indicator for creating a second virtual avatar,
Controlling the generated indicator to be displayed at a first position in the screen of the XR device,
Moving the indicator from the first position to a second position, wherein the second position is different from the first position and is located around the camera in the screen,
While the indicator moves from the first position to the second position, control to photograph the user using the camera, and
Based on the captured results, control to display the at least one real object and the second virtual avatar,
The controller is,
Determine whether the user is more than a preset distance from the XR device,
As a result of the determination, if the distance is greater than the preset distance, control to display a graphic image related to the reference location together with the first virtual avatar,
The first virtual avatar moves according to the user's movement, and the preset distance changes depending on the screen size of the XR device. delete delete According to clause 11,
The controller is,
While the indicator is moving from the first position to the second position, determine whether the user's eyes or face are facing the indicator, and
XR device, characterized in that if the user's eyes or face are not facing the indicator as a result of the determination, the size or color of the indicator is changed. According to clause 14,
The controller is,
An XR device characterized in that the indicator is controlled to display time information or status information required for face scanning only while it is located in the second position. According to clause 15,
The controller is,
An XR device characterized in that it compares a first vector based on the user's face and a second vector based on the user's upper body only while the indicator is located at the second position. According to clause 16,
The controller is,
Only when the first vector and the second vector differ by more than a preset angle,
An XR device characterized in that it controls to display a message guiding the user to move his or her upper body. According to clause 17,
The controller is,
XR device characterized in that it controls to display a graphic image corresponding to a specific part to be analyzed in the user's face only while the indicator is located in the second position. According to clause 18,
The specific part above is,
An XR device including at least one of the user's eyes, nose, or mouth. According to clause 19,
XR, characterized in that the color of the outermost line of the graphic image when the analysis of a specific part of the face fails and the color of the outermost line of the graphic image when the analysis of the specific part of the face is successful are different from each other device.