JP2022084658A - Method for generating 3d object arranged in extended real space - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method for generating a 3D object which is arranged in an extended real space.

SOLUTION: A method for generating a 3D object which is arranged in an extended real space to be displayed in a display unit of a user device generates the 3D object which is arranged in the extended real space to be displayed in the display unit of the user device, displays an image of the 3D object in an imaged real space to detect a hand gesture of a user in the extended real space, and controls a posture of the 3D object on the basis of the hand gesture.

SELECTED DRAWING: Figure 7

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to a method of generating a 3D object arranged in augmented reality space. For details, 3D placed in the augmented reality space displayed on the display of mobile terminals such as smartphones.
Regarding how to create an object. The present invention also relates to a method of generating a 3D object arranged in an augmented reality space displayed on a display unit of a wearable device such as a smart glass or a head-mounted display.

Conventionally, various techniques have been proposed for generating characters represented by two-dimensional (2D) or three-dimensional (3D) still images or moving images (animation).

In many cases, a method is used in which parameters are set for the parts constituting the object to be expressed as an image, and the state of each part is changed by adjusting the parameters.

As an example, in the generation of a three-dimensional object (for example, a character), changes in the state of each part (for example, opening and closing of the mouth, opening and closing of eyes, eyebrows, etc.) with respect to its components (for example, face, mouth, eyebrows, etc.) Parameters related to vertical movement of the eyebrows, horizontal movement of the eyebrows, vertical movement of the eyeballs, horizontal movement of the eyeballs, degree of anger, degree of laughter, degree of shyness) and rotation angle of the object (for example, face) (-180 ~ 18
A method for setting parameters related to 0) is disclosed (for example, Patent Document 1).

Recently, a service that provides an augmented reality space that expands the user's real space by adding information such as images and annotations to the real world through a display of a wearing device has become widespread. One example is a service that superimposes and displays a digital image such as an animated character on the real space captured by the camera of a smartphone, and the user can see if the character actually exists in the real space. You can have a feeling like that.

Japanese Unexamined Patent Publication No. 2009-104570

However, digital images such as characters that produce such an augmented reality space are often not originally intended to be placed in the real space, and provide characters that match different real spaces depending on each user. That is difficult. 3D of the above patent document
The object generation technique is an extension of the two-dimensional illustration editing tool, and is not intended to generate characters to be placed in augmented reality space.

Therefore, an object of the present invention is to provide a technique for generating a 3D object arranged in an augmented reality space.

In one embodiment of the present invention, a method of generating a 3D object arranged in an augmented reality space displayed on a display unit of a user device, wherein an image of the 3D object is displayed in the captured real space, and the above-mentioned A method of detecting a user's hand gesture in augmented reality space and controlling the posture of the 3D object based on the hand gesture.

According to the present invention, it is possible to provide a technique for generating a 3D object arranged in an augmented reality space.

The system block diagram which concerns on 1st Embodiment is shown. The functional block diagram of the server which concerns on 1st Embodiment is shown. The functional block diagram of the user terminal which concerns on 1st Embodiment is shown. The functional block diagram of the control part which concerns on 1st Embodiment is shown. The details of the storage which concerns on 1st Embodiment are shown. The flowchart of the 3D object generation method which concerns on 1st Embodiment is shown. The flowchart of the process associated with the attitude control of the user terminal which concerns on 1st Embodiment is shown. An example of a 3D object generation screen according to the first embodiment is shown.

<First Embodiment>
A specific example of the method for generating a 3D object according to the first embodiment of the present invention will be described below with reference to the drawings. It should be noted that the present invention is not limited to these examples, and is indicated by the scope of claims, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims. In the following description, the same elements will be designated by the same reference numerals in the description of the drawings, and duplicate description will be omitted.

FIG. 1 shows a system configuration diagram according to the first embodiment of the present invention. As shown in FIG. 1, the system according to the present embodiment includes a server 1 and a user terminal 2A and a user terminal 2B connected to the server 1 via a network such as the Internet. Although the user terminals 2A and 2B (hereinafter collectively referred to as user terminals 2) are shown in FIG. 1 for convenience of explanation, any number of user terminals can be connected to the network.

The server 1 can provide a service associated with the user terminals 2A and 2B to provide an augmented reality space by arranging a 3D object such as an animation character in the real space. For example, the server 1 provides a service that provides an augmented reality space by superimposing a character image on a real space imaged by a camera built in a user terminal 2 (for example, a mobile phone terminal or a smartphone).

More specifically, the user can take an image of the user's room with a camera built in the smartphone, for example, and the user can display the favorite character image superimposed on the background image of the room. As a result, the user can feel as if his / her favorite character lives in his / her room.

For example, the user can download the service application from the server 1 or another platform to the user terminal 2, and send / receive data related to the server 1 and the service via the application. Alternatively, the user can also send and receive data related to this service by accessing the Web application stored in the server 1 by the Web browser built in the user terminal 2. In the following embodiment, a method of providing this service via a downloaded application will be described as an example.

The user terminal 2 may be, for example, a smartphone, a tablet, a mobile terminal or other information terminal, or may be a general-purpose computer such as a workstation or a personal computer. Further, it may be a wearable device such as a smart glass or a head-mounted display that can be worn on the user's body.

FIG. 2 shows a functional configuration diagram of the server according to the first embodiment. The configuration shown in the figure is an example, and may have other configurations.

As shown, the server 1 is connected to a database (not shown) to form part of the system. The server 1 may be a general-purpose computer such as a workstation or a personal computer, or may be logically realized by cloud computing.

The server 1 has at least a control unit 10, a memory 11, a storage 12, a transmission / reception unit 13, and so on.
Input / output units 14 and the like are provided, and these are electrically connected to each other through a bus 15.

The control unit 10 is an arithmetic unit that controls the operation of the entire server 1, controls the transmission and reception of data between each element, and performs information processing necessary for application execution and authentication processing.
For example, the control unit 10 is a CPU (Central Processing Unit), and executes each information processing by executing a program or the like stored in the storage 12 and expanded in the memory 11.

The memory 11 is a DRAM (Dynamic Random Access Memory).
Main storage composed of volatile storage devices such as y), flash memory and HDD (Hard)
Includes auxiliary storage configured with a non-volatile storage device such as Disc Drive). The memory 11 is used as a work area or the like of the processor 10, and also stores a BIOS (Basic Input / Output System) executed when the server 1 is started, various setting information, and the like.

The storage 12 stores various programs such as application programs. A database (not shown) storing data used for each process may be built in the storage 12.

The transmission / reception unit 13 connects the server 1 to the network. The transmission / reception unit 13 is set to Blue.
It may be provided with a short-range communication interface of echoth (registered trademark) and BLE (Bluetooth Low Energy).

The input / output unit 14 is an information input device such as a keyboard and a mouse, and an output device such as a display.

The bus 15 is commonly connected to each of the above elements and transmits, for example, an address signal, a data signal, and various control signals.

FIG. 3 shows a functional configuration diagram of a user terminal according to the first embodiment. The configuration shown in the figure is an example, and may have other configurations.

As described above, whether the user terminal 2 can be various information terminals or general-purpose computers will be described below by taking a smartphone as an example. The user terminal 2 is at least the control unit 20,
A memory 21, a storage 22, a transmission / reception unit 23, an input unit 24, and the like are provided, and these are electrically connected to each other through a bus 25.

The control unit 20 is an arithmetic unit that controls the operation of the entire user terminal 2, controls the transmission and reception of data between each element, and performs information processing necessary for application execution and authentication processing. For example, the control unit 20 is a CPU (Central Processing Unit), and executes each information processing by executing a program or the like stored in the storage 22 and expanded in the memory 21.

The memory 21 is a DRAM (Dynamic Random Access Memory).
Main memory composed of volatile storage devices such as y), flash memory and HDD (Hard)
Includes auxiliary storage configured with a non-volatile storage device such as Disc Drive). The memory 21 is used as a work area or the like of the control unit 20, and also stores a BIOS (Basic Input / Output System) executed when the user terminal 2 is started, various setting information, and the like.

The storage 22 stores various programs such as application programs. A database (not shown) storing data used for each process may be built in the storage 22.

The transmission / reception unit 23 connects the user terminal 2 to the network. The transmission / reception unit 23 is B.
Bluetooth® and BLE (Bluetooth Low Energy)
) May be provided with a short-range communication interface.

The input unit 24 is an information input device such as a touch panel.

The bus 25 is commonly connected to each of the above elements and transmits, for example, an address signal, a data signal, and various control signals.

Further, the user terminal 2 includes a camera 26. The camera 26 is built in the user terminal 2 and is provided so as to capture an image of the real space and usually match the line-of-sight direction of the user holding the user terminal 2. The camera 26 also has a sensor (eg, for example) that captures the user's gesture.
It has a function as a 2D / 3D camera, ultrasonic wave, depth, IR sensor, etc.).

The image of the real space captured by the camera 26 is combined with the image of a 3D object such as a character by the image control unit 27, and the image of the 3D object is superimposed and displayed on the image of the real space by the display unit 28. .. The image control unit 27 is, for example, a GPU (Graphic).
s Processing Unit), which mainly executes arithmetic processing related to image processing. Further, in the case of a smartphone, the display unit 28 is, for example, a display such as an LCD or an organic EL. The display unit 28 can also function as a finder of the camera 26.

GPS (Global Positioning System) 29 positions the current position information. The GPS 29 can receive radio waves for positioning from GPS satellites and can position latitude / longitude information that is an absolute position. It is also possible to receive communication radio waves from a plurality of base stations for mobile phones and perform positioning of the current position information by a plurality of base station positioning functions.

The acceleration sensor 30 detects acceleration (change in speed per unit time). The acceleration sensor 30 can detect acceleration in the triaxial directions (x, y, z directions). For example, if the front-back direction of the smartphone is the x-axis, the left-right direction is the y-axis, the up-down direction is the z-axis, the front direction is the x-axis positive direction, the left direction is the y-axis positive direction, and the downward direction is the z-axis direction. 30 detects acceleration in each direction, and also detects a rotation angle around the x-axis (roll angle), a rotation angle around the y-axis (pitch angle), and a rotation angle around the z-axis (yaw angle).

The geomagnetic sensor 31 detects the direction of the geomagnetism in the three-axis directions (the above-mentioned x, y, z-axis directions) and displays the horizontal direction. The azimuth is indicated by a clockwise angle with respect to the calculated true north azimuth N detected by the geomagnetic sensor.

In this example, a smartphone is mentioned, but as described above, a wearable device can also be used as a user terminal. In this case, on the wearable device,
It can be equipped with the same functions as those built into the smartphone in this example.

FIG. 4 shows a functional configuration diagram of the control unit according to the first embodiment. In this embodiment, the details of the functional configuration diagram of the control unit 20 of the user terminal 2 will be described, but the same function can be provided in the control unit 10 of the server 1 and processed by the control unit 10. Hereinafter, each functional configuration is realized by executing the program stored in the memory 21, the storage 12, and the like in the control unit 20.

The gesture detection unit 41 detects, for example, the gesture of the user's hand. Specifically, the gesture detection unit 41 normalizes the image including the gesture of the user's hand captured by the camera 26 of the user terminal 2 (smartphone) by a method such as filter removal, and the edge image of the normalized image. An index including the position and direction in the three-dimensional coordinate system (x, y, z) of the camera is assigned to each pixel of the above, and the index is stored in the database outside the user terminal in advance based on the assigned index. The indexed gesture image is searched to determine the gesture image of the hand with the matched index.

The character image management unit 42 manages the 3D object of the character stored in the storage 22, and controls, for example, the position and posture of the character in response to the gesture detection.

FIG. 5 shows the details of the storage according to the first embodiment. Although the details of the storage 22 of the user terminal 2 will be described in the present embodiment, the same information can be stored in the memory 21, the memory 11 of the server 1, the storage 12, and the like.

The storage 12 stores the character information 51. As character information, data related to the generation of a 3D object of a character displayed in the extended real space (for example, character image data, 3D shape data, texture data, animation data, coordinate information, color information, and the like. ) Is stored in a predetermined format. Further, the character information has bone object information necessary for each part of the character (for example, hair, face, neck, torso, arms, hands, legs, etc.) to move with each other as animation data, and each of them has bone object information. For parts, position (x, y, z coordinates), connection source / destination parts, mass / width / height, rotation limit (maximum / minimum values of x, y, z coordinates), and movement limit (x). , Y, z coordinates maximum value /
It can have information such as (minimum value). It is also possible to store a plurality of character information so that the user can select a plurality of types of characters. In addition, the storage 12 stores various programs, applications, operation systems, and the like, including an image processing program.

FIG. 6 shows a flowchart of a method for generating a 3D object according to the first embodiment.
The process relating to the method of generating the 3D object can be realized, for example, by executing the program expanded in the memory 22 of the user terminal 2 in the control unit 21, but in addition, the server 1
It can be realized by executing the program expanded in the memory 12 of the above in the control unit 11, or it can be realized by executing these programs in cooperation with each other.

First, the user request receiving unit 41 of the control unit 20 receives a character selection request to be displayed in the augmented reality space from the user associated with the user terminal 2 via an application or the like (S101). If there is only one type of character to display, this step can be omitted.

Subsequently, the display control unit 27 performs a process of displaying the character selected by the user in the augmented reality space (S102). For example, the display control unit 27 maps the 3D shape data to a predetermined space based on the character information stored in the storage 22, and maps the texture data to the 3D shape to the texture data converted from the character image data. This creates a 3D object for the character. Then, by setting the real space captured by the camera 26 as the background image of the predetermined space, an image in which the 3D object of the character is superimposed on the real space is generated, the extended real space is provided, and the expanded real space is provided. Is displayed on the display. For example, as shown in FIG. 8, the character's 3D object is displayed in the center in augmented reality space and faces the front. Here, the display control unit 72 provides a plurality of indicators for adjusting parameters that change the state of each component (for example, face, eye, nose, mouth, eyebrows, ears, hairstyle, etc.) of the character's 3D object. It can be displayed, receive instructions from the user via an indicator for adjusting parameters, and in response, perform a process of changing the state of a specified component. The display control unit 27 can also display an indicator for correcting the color of the texture of the character's 3D object in the augmented reality space. Here, the color correction includes corrections such as brightness, saturation, color tone, sharpness, and contrast.

Subsequently, the gesture detection unit 41 of the control unit 20 confirms whether or not the gesture has been detected (
S103). For example, the user attempts to change the pose of the character displayed on the display using his or her hand, as shown in FIG. 8 (a). As an example of gesture detection, the gesture detection unit 41 normalizes the image including the gesture of the user's hand captured by the camera 26 of the user terminal 2 (smartphone) by a method such as filter removal, and the edge image of the normalized image. An index including the position and direction in the three-dimensional coordinate system (x, y, z) of the camera is assigned to each pixel of the above, and the index is stored in the database outside the user terminal in advance based on the assigned index. The indexed gesture image is searched and the gesture image with the matching index is determined. In the example of FIG. 8,
The gesture detection unit 41 detects a gesture that the user picks something with a finger (so-called pinch operation) and moves in a predetermined direction.

When the gesture detection unit 41 detects the gesture of the user's hand, the control unit 20 and the image control unit 27 control the posture (posing) of the character according to the gesture (the gesture detection unit 41).
S104). In this process, for example, as shown in FIG. 8B, the image control unit 27 controls to lift the left hand of the 3D object of the character. The details of this process will be described with reference to FIG.

Next, the image control unit 27 displays the 3D object of the character after the attitude control in the augmented reality space (S105). For example, as shown in FIG. 8B, the image of the displayed object is animated to lift the left hand.

FIG. 7 shows a flowchart of the process related to the posture control of the character according to the first embodiment. First, the gesture detection unit 41 of the control unit 20 detects the gesture of the hand to the user (
S201). Here, the gesture detection unit 41 makes a gesture image including a movement corresponding to the movement of the user's hand, that is, a gesture of moving in a predetermined direction by performing an operation of picking something with a finger (pinch operation). Determine the corresponding image.

Next, the gesture detection unit 41 confirms the coordinates of the user's hand based on the detected gesture (S202). As mentioned above, here the gesture detector 41 is captured and
The three-dimensional coordinates (x,) of the camera given to each pixel of the edge image of the normalized image.
Refer to the position and direction in y, z). As shown in FIGS. 8 (a) and 8 (b), the gesture detection unit 41 determines the position and direction of the three-dimensional coordinates of the start point position of the user's hand and the position and the three-dimensional coordinates of the end point position of the user's hand. The direction is confirmed. Here, depending on the detected gesture, it is possible to proceed to the main process by using a specific gesture as a trigger without proceeding with the main process. For example, when the user simply holds his / her hand around the character, the operation of pinching the character's part with a finger (pinch operation) is detected without taking any action, and the process proceeds to the subsequent steps. It may be that.

Subsequently, the gesture detection unit 41 confirms the coordinates of the 3D object corresponding to the coordinates of the detected user's hand (S203). Specifically, as shown in FIGS. 8 (a) and 8 (b), the three-dimensional coordinates of the character's 3D object corresponding to the detected start point position of the user's hand are confirmed.

Then, the gesture detection unit 41 identifies a part of the 3D object corresponding to the coordinates of the confirmed 3D object (S204). In the example shown in FIG. 8A, it can be seen that the three-dimensional coordinates of the character's 3D object corresponding to the starting point position of the user's hand are the left hand of the character.

Then, the display control unit 27 controls to change the position of the component of the 3D object of the character. For example, the display control unit 27 in the examples shown in FIGS. 8 (a) and 8 (b).
Controls to raise the character's left hand according to the gesture of the detected hand.

According to the above example, the three-dimensional coordinates of the camera are used to specify the position of the user's hand and the relative position of the character component, but the world coordinates and local coordinates are applied and the coordinates are appropriately converted. You can also do it. It is also possible to provide two-dimensional coordinates instead of three-dimensional coordinates. Further, by displaying the hand image as the gesture image of the detected user's hand, the user can intuitively adjust the pose of the character, but it is also possible not to display the hand image. ..

As described above, according to the present embodiment, it is possible to adjust the pose of the character by using hand tracking while the 3D object of the character is placed in the augmented reality space, so that the pose of the character figure is as if it were in the real space. As if to decide
You can create a 3D object of your character.

According to the present embodiment, a 3D object of a single character is arranged in an augmented reality space, and a process of changing its constituent elements is performed. For example, a plurality of characters are arranged in the same augmented reality space. The user can use 3 of any of the multiple characters.
Hand tracking can be used to adjust the pose of the D object. As a result, the user can enjoy the pose of his / her favorite character while feeling as if a plurality of characters having different appearances and heights coexist in the real space.

Further, in the present embodiment, the server 1 can provide the augmented reality space provided to one user terminal 2A to another user terminal 2B via a network. Here, the user terminal 2A transmits a request to the server 1 to transmit information about a character 3D object arranged in the augmented reality space displayed on the display unit 28 of the own terminal to another terminal 2B. The user terminal 2B is a 3D character displayed on the user terminal 2A.
When the information about the object is received, the character is displayed in the augmented reality space displayed on the display unit of the own terminal 2B. At this time, as the background image displayed in the augmented reality space, the real space captured by the camera built in the user terminal 2B can be used. Then, the user who operates the user terminal 2B can enjoy the state of the change in the appearance of the character instructed by the user operated by the user terminal 2A in real time in the augmented reality space of the own terminal.

The embodiments described above are merely examples for facilitating the understanding of the present invention, and are not intended to limit the interpretation of the present invention. It goes without saying that the present invention can be modified and improved without departing from the spirit thereof, and the present invention includes its equivalents.

1 server 2 user terminal

Claims (5)

It is a method of generating a 3D object to be placed in the augmented reality space displayed on the display unit of the user device.
An image of a 3D object is displayed in the captured real space,
Detecting the user's hand gesture in the augmented reality space,
Controlling the posture of the 3D object based on the hand gesture,
Method. Controlling the posture of the 3D object is
Check the detected position coordinates of the 3D object image displayed in the augmented reality space.
Check the position coordinates of the 3D object corresponding to the detected position coordinates, and check
Identify the parts of the 3D object that correspond to the position coordinates of the confirmed 3D object.
A method comprising changing the component position of the 3D object based on the detected hand gesture. The method according to claim 1, wherein the 3D object is a 3D object related to a character. The method according to claim 2, wherein the part is a movable part of the 3D object. The method of claim 4, wherein the component is any of the hair, arms, fingers, or legs of the 3D object.

Images