US20210134040A1 - Avatar motion generating method and head mounted display system - Google Patents
Avatar motion generating method and head mounted display system Download PDFInfo
- Publication number
- US20210134040A1 US20210134040A1 US16/676,416 US201916676416A US2021134040A1 US 20210134040 A1 US20210134040 A1 US 20210134040A1 US 201916676416 A US201916676416 A US 201916676416A US 2021134040 A1 US2021134040 A1 US 2021134040A1
- Authority
- US
- United States
- Prior art keywords
- motion
- avatar
- input event
- data
- predefined
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/20—3D [Three Dimensional] animation
- G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/90—Constructional details or arrangements of video game devices not provided for in groups A63F13/20 or A63F13/25, e.g. housing, wiring, connections or cabinets
- A63F13/98—Accessories, i.e. detachable arrangements optional for the use of the video game device, e.g. grip supports of game controllers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/20—Input arrangements for video game devices
- A63F13/22—Setup operations, e.g. calibration, key configuration or button assignment
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/20—Input arrangements for video game devices
- A63F13/23—Input arrangements for video game devices for interfacing with the game device, e.g. specific interfaces between game controller and console
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/80—2D [Two Dimensional] animation, e.g. using sprites
Definitions
- the present disclosure generally relates a method for generating the motion of an avatar, in particular, to an avatar motion generating method and a head mounted display system.
- VR virtual reality
- AR augmented reality
- MR mixed reality
- XR extended reality
- the motion sensing result of a hand-held controller can be used to estimate the pose of a hand and generate a corresponding motion accordingly.
- the estimation of the pose may not be accurate in some situations.
- an abnormal motion may be presented on the avatar. Therefore, there are still lots of technical problems that should be improved for the motion simulation of the avatar.
- the present disclosure is directed to an avatar motion generating method and a head mounted display system, in which the motion of an avatar may be changed to a predefined motion in response to some situations.
- an avatar motion generating method includes, but not limited to, the following steps. First avatar motion is generated based on one of predefined motion data, motion sensing data and a combination thereof at the first period of time. Second avatar motion is generated based on another of the predefined motion data, the motion sensing data and the combination thereof at the second period of time.
- a head mounted display system wearable on a user's head includes, but not limited to, a memory and a processor.
- the memory is used for storing a program code.
- the processor is coupled to the memory and loads the program code to perform the following steps.
- the processor generate first avatar motion a determined result, switches between predefined motion data and motion sensing data based on one of predefined motion data, motion sensing data and a combination thereof at the first period of time, and generates second avatar motion based on another of the predefined motion data, the motion sensing data and the combination thereof at the second period of time.
- FIG. 1 is a block diagram illustrating a head mounted display system according to one of the exemplary embodiments of the disclosure.
- FIG. 2 is a flowchart illustrating an avatar motion generating method according to one of the exemplary embodiments of the disclosure.
- FIG. 3 is a timing diagram illustrating the switching between different motion data according to one of the exemplary embodiments of the disclosure.
- FIG. 4 is a timing diagram illustrating the switch between different motion data according to one of the exemplary embodiments of the disclosure.
- FIG. 5 is a schematic diagram illustrating the human body portions of an avatar according to one of the exemplary embodiments of the disclosure.
- FIG. 1 is a block diagram illustrating a head mounted display (HMD) system 100 according to one of the exemplary embodiments of the disclosure.
- the HMD system 100 includes, but not limited to, an input apparatus 110 , a memory 140 and a processor 150 .
- the HMD system 100 is adapted for VR, AR, MR, XR or other reality-related technology.
- the input apparatus 110 is used for obtaining user data.
- the user data is related to a sensing result of a user.
- the sensing result may be related to the voice, the motion, the facial expression and/or the operation of the user.
- the user data could be a combination of voice data, motion sensing result, and image data.
- the input apparatus 110 includes an image sensor 111 .
- the image sensor 111 may be a camera, such as a monochrome camera or a color camera, a deep camera, a video recorder, or other sensor capable of capturing images.
- the image sensor 111 may be used to capture toward one or more human body portions of the user, to generate an image including the human body portion of the user.
- the human body portions may include a face, a hand, a head, an ankle, a leg or a waist of the user.
- the image sensor 111 may be used to capture toward the face of a user, to generate a facial image including one or both eye, one or both ear, a nose, a mouse, a face shape, a hair shape, an eyebrow, a bread or a combination thereof of the user.
- the input apparatus 110 includes an audio receiving apparatus 112 .
- the audio receiving apparatus 112 may include, but not limited to, a microphone, an analog-to-digital converter, a filter, and an audio processor.
- the microphone of the audio receiving apparatus 112 may receive sound waves (e.g., generated by human voices, ambient sounds, etc.) and converting them into sound data.
- the audio receiving apparatus 112 is used to receive the voice of a user and generate voice data. In some embodiments, the HMD system 100 may not have the audio receiving apparatus 112 .
- the input apparatus 110 includes a motion sensor 113 .
- the motion sensor 113 may be an accelerometer, a gyroscope, a magnetometer, a laser sensor, an inertial measurement unit (IMU), an infrared ray (IR) sensor, or any combination of aforementioned sensors.
- the motion sensor 113 is used for sensing the motion of one or more human body portions of the user, to generate a sequence of motion sensing result of the sensor 110 (e.g. sensed strength values, etc.) at several time points.
- the motion sensing result includes a 3-degree of freedom (3-DoF) data
- the 3-DoF data is related to the rotation data of the human body portion in a three-dimensional (3D) space, such as accelerations in yaw, roll and, pitch.
- the motion sensing result includes a relative position and/or displacement of a human body portion in the 2D/3D space.
- the motion sensor 113 could be embedded in a handheld controller or a wearable apparatus, such as a wearable controller, a smartwatch, an ankle sensor, a head-mounted display (HMD), or the likes.
- a wearable controller such as a wearable controller, a smartwatch, an ankle sensor, a head-mounted display (HMD), or the likes.
- HMD head-mounted display
- the HMD system 100 further includes a display 130 .
- the display 130 may be a liquid-crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, or other displays.
- the display 130 is used for displaying images.
- the display 130 may be a display of an external apparatus (such as a smartphone, a tablet, or the likes), and the external apparatus can be placed on the main body of the HMD system 100 .
- the memory 140 may be any type of a fixed or movable Random-Access Memory (RAM), a Read-Only Memory (ROM), a flash memory, a similar device or a combination of the above devices.
- RAM Random-Access Memory
- ROM Read-Only Memory
- flash memory a similar device or a combination of the above devices.
- the memory 140 records program codes, device configurations, buffer data or permanent data (such as image data, voice data, motion sensing data, predefined motion data, motion sensing result, etc.), and these data would be introduced later.
- the processor 150 is coupled to the input apparatus 110 , the display 130 , and the memory 140 .
- the processor 150 is configured to load the program codes stored in the memory 140 , to perform a procedure of the exemplary embodiment of the disclosure.
- functions of the processor 150 may be implemented by using a programmable unit such as a central processing unit (CPU), a microprocessor, a microcontroller, a digital signal processing (DSP) chip, a field-programmable gate array (FPGA), etc.
- the functions of the processor 150 may also be implemented by an independent electronic device or an integrated circuit (IC), and operations of the processor 150 may also be implemented by software.
- the processor 150 may not be disposed at the same apparatus with the input apparatus 110 .
- the apparatuses respectively equipped with input apparatus 110 and the processor 150 may further include communication transceivers with compatible communication technology, such as Bluetooth, Wi-Fi, infrared ray (IR), or physical transmission line, to transmit or receive data with each other.
- the audio receiving apparatus 112 and the processor 150 may be disposed in an HMD while the image sensor 111 is disposed outside the HMD.
- the processor 150 may be disposed in a server while the image sensor 111 and the audio receiving apparatus 112 being disposed outside the server.
- FIG. 2 is a flowchart illustrating an avatar motion generating method according to one of the exemplary embodiments of the disclosure.
- the processor 150 may generate first avatar motion based on one of predefined motion data, motion sensing data and a combination thereof at a first period of time (step S 210 ).
- the predefined motion data is pre-stored in the memory 140 without motion sensing result of a human body portion of the user.
- the predefined motion data is not generated based on the currently detected motion sensing result of the user.
- the predefined motion could be the motion of a hand for shooting darts or shooting hoops.
- the predefined motion data may include specific motion data (such as 3-DoF data or 6-DoF data, accelerations in yaw, roll and, pitch, the displacement, etc.) for each predefined motion of different human body portions. Taking the foot of the user as an example, the predefined motion may be lifting, pointing, kicking, stepping, or jumping.
- specific motion data such as 3-DoF data or 6-DoF data, accelerations in yaw, roll and, pitch, the displacement, etc.
- the motion sensing data is generated based on the currently detected motion sensing result of the user.
- the motion sensing data is related to one or more essential motions, and the motion sensing data may include specific motion data based on the motion sensing result (such as 3-DoF or 6-DoF data, accelerations in yaw, roll and, pitch, the displacement, etc.,) for each essential motion.
- the processor 150 may determine a pose of one or more human body portions based on the motion sensing result and inverse kinematics to generate the motion sensing data. Based on the inverse kinematics, given position information of the end of the arm can be used to estimate joint parameters (such as the angle between the upper arm and the forearm, the angle between the shoulder and the upper arm, etc.). The given position information can be provided by the motion sensing result. For example, 6-DoF data obtained from the motion sensor 113 disposed at a handheld control can be used to determine the position of the hand of the user.
- the joint parameters can be determined based on the determined position from the motion sensing result of the motion sensor 113 and/or the image sensor 111 , so as to further determine the pose of the arm based on kinematics equations. It should be noted that the poses of the legs, the upper body, the head, and other human body portions can be estimated based on inverse kinematics, too, and their detailed description would be omitted.
- the motion sensor 113 may be disposed at multiple human joints or body portions, or the image sensor 111 may capture all of or the desired human body portions of the user, the motion sensing result of the motion sensor 113 and/or the image sensor 111 can be used for estimating the pose of the corresponding human body portion directly.
- the processor 150 may generate the combination of the predefined motion data and the motion sensing data.
- the combination could be a weight combination of the predefined motion data and the motion sensing data.
- the position based on the predefined motion data is given with a first weight
- the position based on the motion sensing data is given with a second weight
- the weight combination would be the weighted calculation on the positions based on the predefined motion data and the motion sensing data with the first and the second weights.
- the combination would be another predefined motion data based on the original predefined motion data and the motion sensing data.
- the positions and the rotation situations based one the original predefined motion data and the motion sensing data would be considered as references to determine the position and the rotation situation in the another predefined motion data.
- the avatar motion is the motion of the avatar in the virtual environment.
- the processor 150 may select one of the predefined motion data, the motion sensing data and the combination thereof for the first avatar motion at the first period. In other words, a predefined motion, a sensing motion of the user, or the combination thereof would be performed on the avatar at the first period.
- the processor 150 may generate second avatar motion based on another of the predefined motion data, the motion sensing data and the combination thereof at a second period of time (step S 230 ). Specifically, compared with the first avatar motion at the first period, another avatar motion (i.e., the second avatar motion) is generated based on a different motion data at the second period. For example, the first avatar motion is generated based on the motion sensing data, and the second avatar motion is generated based on the predefined motion data. For another example, the first avatar motion is generated based on the combination of both motion data and the second avatar is generated based on the predefined motion data.
- the estimated pose merely based on the motion sensing data may result in an abnormal motion on the avatar.
- the predefined motion data may present a natural and normal motion of the human.
- the use of the predefined motion data may prevent the abnormal motion on the avatar.
- the processor 150 may receive an input event from the user data obtained by the input apparatus 110 at the first period.
- the input event may be encountering a scenario/level of the game, a motion of the user being detected, user data (such as voice data, image data, motion sensing result, text data, input operation of a user, etc.) being received, etc.
- user data such as voice data, image data, motion sensing result, text data, input operation of a user, etc.
- the reception of the voice data by the audio receiving apparatus 112 could be an input event.
- the detection of the motion sensing result by the motion sensor 113 could be another input event.
- the selection of cinema scenario could be an input event.
- each predefined triggering condition may be corresponding to a movement of one human body portion, a rotation of one human body portion, a pose of one human body portion, a gesture of a hand, an emotion, a scenario, a level of a game, or a keyword/keyphrase.
- the predefined triggering condition could be that, for example, the head of the user moves horizontally; the head of the user rotates clockwise and horizontally; the hand of the user exists in the image data; the hand of the user makes a fist; the lips of the user make a smile; entering a basketball game scenario; the keywords “very happy” is detected in the voice data.
- the processor 150 may use the image identification, the semantic analysis, the motion state analysis, or other technologies to compare the input event with the corresponding predefined triggering condition. The determined result would be the input event is met a corresponding predefined triggering condition or input event is not met a corresponding predefined triggering condition.
- the processor 150 may generate the second avatar motion at the second period in response to the input event being met anyone of the predefined triggering conditions.
- the processor 150 may switch between predefined motion data and motion sensing data according to the determined result.
- the predefined triggering conditions correspond to the conditions for the change of motion data. If the determined result is that the input event is met one of the predefined triggering conditions, the motion data for generating the motion of the avatar may be changed from the motion sensing data to the predefined motion data or from the predefined motion data to the motion sensing data.
- the processor 150 detects that the input event is that the head of the user moves horizontally, the processor 150 may change the motion data from the motion sensing data to the predefined motion data, so that a predefined walk motion (i.e., the second avatar motion) for legs may be presented on the avatar.
- the processor 150 detects that the input event is that the hand of the user exists in the camera image obtained from the image sensor 111 , the processor 150 may change the motion data from the predefined motion data to the motion sensing data, so that the hand of the avatar may move based on the motion sensing result (i.e., the second avatar motion).
- the processor 150 detects that the input event is that a throw motion of the hand of the user is detected when the hand of the avatar holds a dart, the processor 150 may change the motion data from the motion sensing data to the predefined motion data, so that the hand of the avatar may shoot the dart based on the predefined motion (i.e., the second avatar motion).
- the processor 150 may obtain a behavior state of the avatar from a state machine, and the behavior state is related to the currently performing behavior of the avatar.
- the behavior state of the state machine may be idle, standing, sitting, walking, running, jumping, or other states.
- the processor 150 may compare the input event with the current behavior state in response to the determined result being that the input event is met one of the predefined triggering conditions, and the compared result may trigger the change of the motion data.
- the processor 150 may provide a priority for the input event, and compare the priorities of the input event and the behavior state. It is assumed that the priority of the input event and the priority of the current behavior state are fixed. It means that each input event has a corresponding priority, and each behavior state has another corresponding priority.
- the processor 150 may switch between the predefined motion data and the motion sensing data in response to the input event having higher priority than the behavior state, and the behavior state would be modified accordingly. For example, the current behavior state is standing, and the standing state for the legs of the avatar has a lower priority than the input event which is that the head of the user moves horizontally. Then, the modified behavior state would be the walking state. On the other hand, in response to the input event not having a higher priority than the behavior state, the change of the motion data would be no need.
- the processor 150 may provide a weight for the input event, and compare the weights of the input event and the behavior state. It is assumed that the weight of the input event and the weight of the current behavior state are variable based on different scenarios.
- the processor 150 may switch between the predefined motion data and the motion sensing data in response to the input event having higher weight than the behavior state. For example, at the first time point, the weight of the input event is 80, the weight of the behavior state is 90, and the change of the motion data would not happen. Then, at the second time point, the weight of the input event is 90, the weight of the behavior state is 70, and the change of the motion data may happen.
- the motion data to present the behavior state may be the predefined motion data or the motion sensing data based on the actual situation.
- the processor 150 may switch between the predefined motion data and the motion sensing data in response to the input event not exist. For example, no motion is detected from the handheld controller, the hand of the user does not exist in the camera image, or the user chooses to exit from the basketball game, etc.
- some input events may be predefined with continuation times. When a corresponding continuation time expires, the processor 150 may determine that the input event does not exist. Then, the processor 150 may return to a previous behavior state from the behavior state modified by the input event. For example, the input event is a laugh from the voice data obtained from the audio receiving apparatus 112 , a predefined laugh motion would be performed on the avatar's hand for 2 seconds, and then the motion of the avatar's hand may be based on motion sensing data.
- the processor 150 may further receive one or more second input events (i.e., other input events). If the input event still exists, the processor 150 may compare the second input event with the input event to generate a comparing result. For example, the timing, the priorities, or the weights of these input events may be compared. The processor 150 may generate third avatar motion of the human body portion of the avatar based on motion data of one of the input event and the second input event according to the comparing result. For example, the priority of the second input event is higher than the input event, the processor 150 may choose the second input event to be the reference of the motion data. For another example, the weight of the input event is higher than the second input event, the processor 150 may choose the input event.
- the processor 150 may choose the input event.
- the processor 150 may generate the motion of the human body portion of the avatar based on the predefined motion data and the motion sensing data at different time periods. Specifically, the motion of the avatar presented in the display 130 can be generated according to the determined motion data. For example, the motion of the leg of the user is lifting, and the avatar may lift its leg based on the motion sensing data. For another example, the legs of the avatar perform the predefined walk motion. However, regarding one human body of the avatar, the processor 150 may choose one of the predefined motion data and the motion sensing data one time.
- FIG. 3 is a timing diagram illustrating the switching between different motion data according to one of the exemplary embodiments of the disclosure. It is assumed that the motion of the avatar is generated based on the motion sensing data SMD at first time period td 1 . At the time point tp 1 , an input event ie 1 is received, and the processor 150 determines to switch the motion data from the motion sensing data SMD to the predefined motion data PMD. Then, the motion of the avatar is generated based on the predefined motion data PMD at the time period td 2 . At the time point tp 2 , another input event ie 2 is received, and the processor 150 determines to switch the motion data from the predefined motion data PMD to the motion sensing data SMD. Then, the motion of the avatar is generated based on the motion sensing data SMD at the time period td 3 .
- the processor 150 may generate transition avatar motion based on both the predefined motion data and the motion sensing data. It means that a transition would be inserted between two motion data. For example, the processor 150 may find some intermediate positions between the final position of the avatar's hand based on the motion sensing data and the initial position of the avatar's hand based on the predefined motion data as parameters of the transition motion data. Similarly, the pose and the rotation manner of the human body portion of the transition avatar motion is determined based on a transition between the motion sensing data and the predefined motion data.
- FIG. 4 is a timing diagram illustrating the switch between different motion data according to one of the exemplary embodiments of the disclosure. Referring to FIG. 4 , the difference from the embodiment of FIG. 3 is that, in response to the input event ie 1 being received, the processor 150 generates the motion of the avatar based on the transition motion data TMD 1 , which is the combination of the predefined motion data PMD and the motion sensing data SMD, between time points tp 1 and tp 3 .
- the processor 150 generates the motion of the avatar based on the predefined motion data PMD.
- the processor 150 in response to the input event ie 2 being received, the processor 150 generates the motion of the avatar based on the transition motion data TMD 2 , which is the combination of the predefined motion data PMD and another motion sensing data SMD, between time points tp 2 and tp 4 .
- the processor 150 generates the motion of the avatar based on the motion sensing data SMD.
- the aforementioned embodiments introduce the switch of motion data between the motion sensing data and the predefined motion data for one human body motion. It means that, for one human body portion, the motion sensing data and the predefined motion data may used at different time periods. In some embodiment, for multiple human body motion, the processor 150 may generate the motion of a first human body portion of the avatar based on the predefined motion data and the motion of a second human body portion of the avatar different from the first human body portion based on the motion sensing data at the same time period.
- FIG. 5 is a schematic diagram illustrating the human body portions of an avatar according to one of the exemplary embodiments of the disclosure.
- the head mounted display system 100 may provide handheld controllers with the motion sensor 113 and a head mounted display with the image sensor 111 and the motion sensor 113 to detect the motion of the human body portions B 1 ⁇ B 3 . Therefore, the motion of the human body portions B 1 ⁇ B 3 may be generated based on the motion sensing data from the motion sensing result of the image sensor 111 and the motion sensor 113 . On the other hand, the motion of the human body portions B 4 and B 5 may be generated based on the predefined motion data.
- the body of the avatar may be further divided into more or less human body portions.
- the human body portions B 1 is further divided into two portions, which are the arm and the palm.
- the human body portions B 1 and B 2 are generated based on the motion sensing data, and other human body portions B 3 ⁇ B 5 are generated based on the predefined motion data.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- User Interface Of Digital Computer (AREA)
- Processing Or Creating Images (AREA)
Abstract
An avatar motion generating method and a head mounted display system are provided. In the method, an input event is received, and the input event is related to sensing result of a user. First avatar motion is generated based on one of predefined motion data, motion sensing data and a combination thereof at the first period of time. Second avatar motion is generated based on another of the predefined motion data, the motion sensing data and the combination thereof at the second period of time. Accordingly, the motion of the avatar could be smooth and natural.
Description
- The present disclosure generally relates a method for generating the motion of an avatar, in particular, to an avatar motion generating method and a head mounted display system.
- Technologies for simulating senses, perception and/or environment, such as virtual reality (VR), augmented reality (AR), mixed reality (MR) and extended reality (XR), are popular nowadays. The aforementioned technologies can be applied in multiple fields, such as gaming, military training, healthcare, remote working, etc.
- In order to let the user perceive the simulated environment as a real environment, motion of the body portions in the real world would be tracked, so that the displaying image and the motion of an avatar on a VR, AR, MR or XR display can be changed in response to the motion of the user. For example, the motion sensing result of a hand-held controller can be used to estimate the pose of a hand and generate a corresponding motion accordingly. However, the estimation of the pose may not be accurate in some situations. Eventually, an abnormal motion may be presented on the avatar. Therefore, there are still lots of technical problems that should be improved for the motion simulation of the avatar.
- Accordingly, the present disclosure is directed to an avatar motion generating method and a head mounted display system, in which the motion of an avatar may be changed to a predefined motion in response to some situations.
- In one of the exemplary embodiments, an avatar motion generating method includes, but not limited to, the following steps. First avatar motion is generated based on one of predefined motion data, motion sensing data and a combination thereof at the first period of time. Second avatar motion is generated based on another of the predefined motion data, the motion sensing data and the combination thereof at the second period of time.
- In one of the exemplary embodiments, a head mounted display system wearable on a user's head includes, but not limited to, a memory and a processor. The memory is used for storing a program code. The processor is coupled to the memory and loads the program code to perform the following steps. The processor generate first avatar motion a determined result, switches between predefined motion data and motion sensing data based on one of predefined motion data, motion sensing data and a combination thereof at the first period of time, and generates second avatar motion based on another of the predefined motion data, the motion sensing data and the combination thereof at the second period of time.
- It should be understood, however, that this Summary may not contain all of the aspects and embodiments of the present disclosure, is not meant to be limiting or restrictive in any manner, and that the invention as disclosed herein is and will be understood by those of ordinary skill in the art to encompass obvious improvements and modifications thereto.
- The accompanying drawings are included to provide a further understanding of the disclosure, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the disclosure and, together with the description, serve to explain the principles of the disclosure.
-
FIG. 1 is a block diagram illustrating a head mounted display system according to one of the exemplary embodiments of the disclosure. -
FIG. 2 is a flowchart illustrating an avatar motion generating method according to one of the exemplary embodiments of the disclosure. -
FIG. 3 is a timing diagram illustrating the switching between different motion data according to one of the exemplary embodiments of the disclosure. -
FIG. 4 is a timing diagram illustrating the switch between different motion data according to one of the exemplary embodiments of the disclosure. -
FIG. 5 is a schematic diagram illustrating the human body portions of an avatar according to one of the exemplary embodiments of the disclosure. - Reference will now be made in detail to the presently preferred embodiments of the disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts.
-
FIG. 1 is a block diagram illustrating a head mounted display (HMD)system 100 according to one of the exemplary embodiments of the disclosure. Referring toFIG. 1 , theHMD system 100 includes, but not limited to, aninput apparatus 110, amemory 140 and aprocessor 150. TheHMD system 100 is adapted for VR, AR, MR, XR or other reality-related technology. - The
input apparatus 110 is used for obtaining user data. The user data is related to a sensing result of a user. The sensing result may be related to the voice, the motion, the facial expression and/or the operation of the user. Alternatively, the user data could be a combination of voice data, motion sensing result, and image data. - In one embodiment, the
input apparatus 110 includes animage sensor 111. Theimage sensor 111 may be a camera, such as a monochrome camera or a color camera, a deep camera, a video recorder, or other sensor capable of capturing images. - In some embodiments, the
image sensor 111 may be used to capture toward one or more human body portions of the user, to generate an image including the human body portion of the user. For example, the human body portions may include a face, a hand, a head, an ankle, a leg or a waist of the user. - In some embodiments, the
image sensor 111 may be used to capture toward the face of a user, to generate a facial image including one or both eye, one or both ear, a nose, a mouse, a face shape, a hair shape, an eyebrow, a bread or a combination thereof of the user. - In one embodiment, the
input apparatus 110 includes anaudio receiving apparatus 112. Theaudio receiving apparatus 112 may include, but not limited to, a microphone, an analog-to-digital converter, a filter, and an audio processor. The microphone of the audio receivingapparatus 112 may receive sound waves (e.g., generated by human voices, ambient sounds, etc.) and converting them into sound data. - In some embodiments, the
audio receiving apparatus 112 is used to receive the voice of a user and generate voice data. In some embodiments, theHMD system 100 may not have the audio receivingapparatus 112. - In one embodiment, the
input apparatus 110 includes amotion sensor 113. Themotion sensor 113 may be an accelerometer, a gyroscope, a magnetometer, a laser sensor, an inertial measurement unit (IMU), an infrared ray (IR) sensor, or any combination of aforementioned sensors. In the embodiment of the disclosure, themotion sensor 113 is used for sensing the motion of one or more human body portions of the user, to generate a sequence of motion sensing result of the sensor 110 (e.g. sensed strength values, etc.) at several time points. For one example, the motion sensing result includes a 3-degree of freedom (3-DoF) data, and the 3-DoF data is related to the rotation data of the human body portion in a three-dimensional (3D) space, such as accelerations in yaw, roll and, pitch. For another example, the motion sensing result includes a relative position and/or displacement of a human body portion in the 2D/3D space. - It should be noticed that the
motion sensor 113 could be embedded in a handheld controller or a wearable apparatus, such as a wearable controller, a smartwatch, an ankle sensor, a head-mounted display (HMD), or the likes. - In some embodiments, the
HMD system 100 further includes adisplay 130. Thedisplay 130 may be a liquid-crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, or other displays. In the embodiment of the disclosure, thedisplay 130 is used for displaying images. It should be noted that, in some embodiments, thedisplay 130 may be a display of an external apparatus (such as a smartphone, a tablet, or the likes), and the external apparatus can be placed on the main body of theHMD system 100. - The
memory 140 may be any type of a fixed or movable Random-Access Memory (RAM), a Read-Only Memory (ROM), a flash memory, a similar device or a combination of the above devices. Thememory 140 records program codes, device configurations, buffer data or permanent data (such as image data, voice data, motion sensing data, predefined motion data, motion sensing result, etc.), and these data would be introduced later. - The
processor 150 is coupled to theinput apparatus 110, thedisplay 130, and thememory 140. Theprocessor 150 is configured to load the program codes stored in thememory 140, to perform a procedure of the exemplary embodiment of the disclosure. - In some embodiments, functions of the
processor 150 may be implemented by using a programmable unit such as a central processing unit (CPU), a microprocessor, a microcontroller, a digital signal processing (DSP) chip, a field-programmable gate array (FPGA), etc. The functions of theprocessor 150 may also be implemented by an independent electronic device or an integrated circuit (IC), and operations of theprocessor 150 may also be implemented by software. - It should be noticed that the
processor 150 may not be disposed at the same apparatus with theinput apparatus 110. However, the apparatuses respectively equipped withinput apparatus 110 and theprocessor 150 may further include communication transceivers with compatible communication technology, such as Bluetooth, Wi-Fi, infrared ray (IR), or physical transmission line, to transmit or receive data with each other. For example, theaudio receiving apparatus 112 and theprocessor 150 may be disposed in an HMD while theimage sensor 111 is disposed outside the HMD. For another example, theprocessor 150 may be disposed in a server while theimage sensor 111 and theaudio receiving apparatus 112 being disposed outside the server. - To better understand the operating process provided in one or more embodiments of the disclosure, several embodiments will be exemplified below to elaborate the operating process of the
HMD system 100. The devices and modules in theHMD system 100 are applied in the following embodiments to explain the avatar motion generating method provided herein. Each step of the method can be adjusted according to actual implementation situations and should not be limited to what is described herein. -
FIG. 2 is a flowchart illustrating an avatar motion generating method according to one of the exemplary embodiments of the disclosure. Referring toFIG. 2 , theprocessor 150 may generate first avatar motion based on one of predefined motion data, motion sensing data and a combination thereof at a first period of time (step S210). Specifically, the predefined motion data is pre-stored in thememory 140 without motion sensing result of a human body portion of the user. In other words, the predefined motion data is not generated based on the currently detected motion sensing result of the user. For example, the predefined motion could be the motion of a hand for shooting darts or shooting hoops. The predefined motion data may include specific motion data (such as 3-DoF data or 6-DoF data, accelerations in yaw, roll and, pitch, the displacement, etc.) for each predefined motion of different human body portions. Taking the foot of the user as an example, the predefined motion may be lifting, pointing, kicking, stepping, or jumping. - It should be noted that there may be more than one predefined motion for presenting the current state or emotion on the avatar. For example, there may be 10 predefined motions for an idle state of the avatar, and the
processor 150 may select one of the 10 predefined motions to be the reference of the predefined motion data. In some embodiments, there may be merely one predefined motion for presenting one current state or one emotion on the avatar. For example, the motion of a hand for shooting darts. - On the other hand, the motion sensing data is generated based on the currently detected motion sensing result of the user. The motion sensing data is related to one or more essential motions, and the motion sensing data may include specific motion data based on the motion sensing result (such as 3-DoF or 6-DoF data, accelerations in yaw, roll and, pitch, the displacement, etc.,) for each essential motion.
- In one embodiment, the
processor 150 may determine a pose of one or more human body portions based on the motion sensing result and inverse kinematics to generate the motion sensing data. Based on the inverse kinematics, given position information of the end of the arm can be used to estimate joint parameters (such as the angle between the upper arm and the forearm, the angle between the shoulder and the upper arm, etc.). The given position information can be provided by the motion sensing result. For example, 6-DoF data obtained from themotion sensor 113 disposed at a handheld control can be used to determine the position of the hand of the user. The joint parameters can be determined based on the determined position from the motion sensing result of themotion sensor 113 and/or theimage sensor 111, so as to further determine the pose of the arm based on kinematics equations. It should be noted that the poses of the legs, the upper body, the head, and other human body portions can be estimated based on inverse kinematics, too, and their detailed description would be omitted. - In some embodiments, the
motion sensor 113 may be disposed at multiple human joints or body portions, or theimage sensor 111 may capture all of or the desired human body portions of the user, the motion sensing result of themotion sensor 113 and/or theimage sensor 111 can be used for estimating the pose of the corresponding human body portion directly. - In one embodiment, the
processor 150 may generate the combination of the predefined motion data and the motion sensing data. In some embodiments, the combination could be a weight combination of the the predefined motion data and the motion sensing data. For example, the position based on the predefined motion data is given with a first weight, the position based on the motion sensing data is given with a second weight, and the weight combination would be the weighted calculation on the positions based on the predefined motion data and the motion sensing data with the first and the second weights. In another embodiment, the combination would be another predefined motion data based on the original predefined motion data and the motion sensing data. For example, the positions and the rotation situations based one the original predefined motion data and the motion sensing data would be considered as references to determine the position and the rotation situation in the another predefined motion data. - The avatar motion is the motion of the avatar in the virtual environment. The
processor 150 may select one of the predefined motion data, the motion sensing data and the combination thereof for the first avatar motion at the first period. In other words, a predefined motion, a sensing motion of the user, or the combination thereof would be performed on the avatar at the first period. - The
processor 150 may generate second avatar motion based on another of the predefined motion data, the motion sensing data and the combination thereof at a second period of time (step S230). Specifically, compared with the first avatar motion at the first period, another avatar motion (i.e., the second avatar motion) is generated based on a different motion data at the second period. For example, the first avatar motion is generated based on the motion sensing data, and the second avatar motion is generated based on the predefined motion data. For another example, the first avatar motion is generated based on the combination of both motion data and the second avatar is generated based on the predefined motion data. - In general, the estimated pose merely based on the motion sensing data may result in an abnormal motion on the avatar. It should be noted that it can be sure that the predefined motion data may present a natural and normal motion of the human. In some situations, the use of the predefined motion data may prevent the abnormal motion on the avatar. However, in other situations, it would need to reflect the actual motion of the user based on the motion sensing data. Therefore, a change of motion data among the motion sensing data, predefined motion data and the combination thereof may help in different situations.
- In one embodiment, two different avatar motions for one human body portion would be generated at different time periods. In this embodiment, the
processor 150 may receive an input event from the user data obtained by theinput apparatus 110 at the first period. Specifically, the input event may be encountering a scenario/level of the game, a motion of the user being detected, user data (such as voice data, image data, motion sensing result, text data, input operation of a user, etc.) being received, etc. For example, the reception of the voice data by theaudio receiving apparatus 112 could be an input event. For another example, the detection of the motion sensing result by themotion sensor 113 could be another input event. Alternatively, the selection of cinema scenario could be an input event. - In one embodiment, the
processor 150 may determine whether the input event is met one of predefined triggering conditions to generate a determined result. Specifically, each predefined triggering condition may be corresponding to a movement of one human body portion, a rotation of one human body portion, a pose of one human body portion, a gesture of a hand, an emotion, a scenario, a level of a game, or a keyword/keyphrase. The predefined triggering condition could be that, for example, the head of the user moves horizontally; the head of the user rotates clockwise and horizontally; the hand of the user exists in the image data; the hand of the user makes a fist; the lips of the user make a smile; entering a basketball game scenario; the keywords “very happy” is detected in the voice data. Regarding different types of the predefined triggering conditions, theprocessor 150 may use the image identification, the semantic analysis, the motion state analysis, or other technologies to compare the input event with the corresponding predefined triggering condition. The determined result would be the input event is met a corresponding predefined triggering condition or input event is not met a corresponding predefined triggering condition. - Then, the
processor 150 may generate the second avatar motion at the second period in response to the input event being met anyone of the predefined triggering conditions. In one embodiment, theprocessor 150 may switch between predefined motion data and motion sensing data according to the determined result. The predefined triggering conditions correspond to the conditions for the change of motion data. If the determined result is that the input event is met one of the predefined triggering conditions, the motion data for generating the motion of the avatar may be changed from the motion sensing data to the predefined motion data or from the predefined motion data to the motion sensing data. For example, theprocessor 150 detects that the input event is that the head of the user moves horizontally, theprocessor 150 may change the motion data from the motion sensing data to the predefined motion data, so that a predefined walk motion (i.e., the second avatar motion) for legs may be presented on the avatar. For another example, theprocessor 150 detects that the input event is that the hand of the user exists in the camera image obtained from theimage sensor 111, theprocessor 150 may change the motion data from the predefined motion data to the motion sensing data, so that the hand of the avatar may move based on the motion sensing result (i.e., the second avatar motion). Still another example, theprocessor 150 detects that the input event is that a throw motion of the hand of the user is detected when the hand of the avatar holds a dart, theprocessor 150 may change the motion data from the motion sensing data to the predefined motion data, so that the hand of the avatar may shoot the dart based on the predefined motion (i.e., the second avatar motion). - In another embodiment, the
processor 150 may obtain a behavior state of the avatar from a state machine, and the behavior state is related to the currently performing behavior of the avatar. The behavior state of the state machine may be idle, standing, sitting, walking, running, jumping, or other states. Theprocessor 150 may compare the input event with the current behavior state in response to the determined result being that the input event is met one of the predefined triggering conditions, and the compared result may trigger the change of the motion data. - In one embodiment, the
processor 150 may provide a priority for the input event, and compare the priorities of the input event and the behavior state. It is assumed that the priority of the input event and the priority of the current behavior state are fixed. It means that each input event has a corresponding priority, and each behavior state has another corresponding priority. Theprocessor 150 may switch between the predefined motion data and the motion sensing data in response to the input event having higher priority than the behavior state, and the behavior state would be modified accordingly. For example, the current behavior state is standing, and the standing state for the legs of the avatar has a lower priority than the input event which is that the head of the user moves horizontally. Then, the modified behavior state would be the walking state. On the other hand, in response to the input event not having a higher priority than the behavior state, the change of the motion data would be no need. - In another embodiment, the
processor 150 may provide a weight for the input event, and compare the weights of the input event and the behavior state. It is assumed that the weight of the input event and the weight of the current behavior state are variable based on different scenarios. Theprocessor 150 may switch between the predefined motion data and the motion sensing data in response to the input event having higher weight than the behavior state. For example, at the first time point, the weight of the input event is 80, the weight of the behavior state is 90, and the change of the motion data would not happen. Then, at the second time point, the weight of the input event is 90, the weight of the behavior state is 70, and the change of the motion data may happen. - It should be noticed that the motion data to present the behavior state may be the predefined motion data or the motion sensing data based on the actual situation.
- In one embodiment, the
processor 150 may switch between the predefined motion data and the motion sensing data in response to the input event not exist. For example, no motion is detected from the handheld controller, the hand of the user does not exist in the camera image, or the user chooses to exit from the basketball game, etc. In some embodiments, some input events may be predefined with continuation times. When a corresponding continuation time expires, theprocessor 150 may determine that the input event does not exist. Then, theprocessor 150 may return to a previous behavior state from the behavior state modified by the input event. For example, the input event is a laugh from the voice data obtained from theaudio receiving apparatus 112, a predefined laugh motion would be performed on the avatar's hand for 2 seconds, and then the motion of the avatar's hand may be based on motion sensing data. - In some embodiments, the
processor 150 may further receive one or more second input events (i.e., other input events). If the input event still exists, theprocessor 150 may compare the second input event with the input event to generate a comparing result. For example, the timing, the priorities, or the weights of these input events may be compared. Theprocessor 150 may generate third avatar motion of the human body portion of the avatar based on motion data of one of the input event and the second input event according to the comparing result. For example, the priority of the second input event is higher than the input event, theprocessor 150 may choose the second input event to be the reference of the motion data. For another example, the weight of the input event is higher than the second input event, theprocessor 150 may choose the input event. - When the
processor 150 determines the motion data, theprocessor 150 may generate the motion of the human body portion of the avatar based on the predefined motion data and the motion sensing data at different time periods. Specifically, the motion of the avatar presented in thedisplay 130 can be generated according to the determined motion data. For example, the motion of the leg of the user is lifting, and the avatar may lift its leg based on the motion sensing data. For another example, the legs of the avatar perform the predefined walk motion. However, regarding one human body of the avatar, theprocessor 150 may choose one of the predefined motion data and the motion sensing data one time. -
FIG. 3 is a timing diagram illustrating the switching between different motion data according to one of the exemplary embodiments of the disclosure. It is assumed that the motion of the avatar is generated based on the motion sensing data SMD at first time period td1. At the time point tp1, an input event ie1 is received, and theprocessor 150 determines to switch the motion data from the motion sensing data SMD to the predefined motion data PMD. Then, the motion of the avatar is generated based on the predefined motion data PMD at the time period td2. At the time point tp2, another input event ie2 is received, and theprocessor 150 determines to switch the motion data from the predefined motion data PMD to the motion sensing data SMD. Then, the motion of the avatar is generated based on the motion sensing data SMD at the time period td3. - Sometimes, the change of motion data may result in an abnormal behavior on the avatar. In one embodiment, the
processor 150 may generate transition avatar motion based on both the predefined motion data and the motion sensing data. It means that a transition would be inserted between two motion data. For example, theprocessor 150 may find some intermediate positions between the final position of the avatar's hand based on the motion sensing data and the initial position of the avatar's hand based on the predefined motion data as parameters of the transition motion data. Similarly, the pose and the rotation manner of the human body portion of the transition avatar motion is determined based on a transition between the motion sensing data and the predefined motion data. - Then, the
processor 150 generates the motion of the human body portion of the avatar based on the transition avatar motion at the third period of time between the first period and the second period in which the first avatar motion is generated at the first period and the second avatar motion is generated at the second period.FIG. 4 is a timing diagram illustrating the switch between different motion data according to one of the exemplary embodiments of the disclosure. Referring toFIG. 4 , the difference from the embodiment ofFIG. 3 is that, in response to the input event ie1 being received, theprocessor 150 generates the motion of the avatar based on the transition motion data TMD1, which is the combination of the predefined motion data PMD and the motion sensing data SMD, between time points tp1 and tp3. Then, at the time point tp3, theprocessor 150 generates the motion of the avatar based on the predefined motion data PMD. In addition, in response to the input event ie2 being received, theprocessor 150 generates the motion of the avatar based on the transition motion data TMD2, which is the combination of the predefined motion data PMD and another motion sensing data SMD, between time points tp2 and tp4. Then, at the time point tp4, theprocessor 150 generates the motion of the avatar based on the motion sensing data SMD. - The aforementioned embodiments introduce the switch of motion data between the motion sensing data and the predefined motion data for one human body motion. It means that, for one human body portion, the motion sensing data and the predefined motion data may used at different time periods. In some embodiment, for multiple human body motion, the
processor 150 may generate the motion of a first human body portion of the avatar based on the predefined motion data and the motion of a second human body portion of the avatar different from the first human body portion based on the motion sensing data at the same time period. -
FIG. 5 is a schematic diagram illustrating the human body portions of an avatar according to one of the exemplary embodiments of the disclosure. Referring toFIG. 5 , it is assumed that the body of the avatar is divided into 5 human body portions B1˜B5. The head mounteddisplay system 100 may provide handheld controllers with themotion sensor 113 and a head mounted display with theimage sensor 111 and themotion sensor 113 to detect the motion of the human body portions B1˜B3. Therefore, the motion of the human body portions B1˜B3 may be generated based on the motion sensing data from the motion sensing result of theimage sensor 111 and themotion sensor 113. On the other hand, the motion of the human body portions B4 and B5 may be generated based on the predefined motion data. - It should be noticed that, base on different design requirements, the body of the avatar may be further divided into more or less human body portions. For example, the human body portions B1 is further divided into two portions, which are the arm and the palm. For another example, merely the human body portions B1 and B2 are generated based on the motion sensing data, and other human body portions B3˜B5 are generated based on the predefined motion data.
- It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the present disclosure without departing from the scope or spirit of the disclosure. In view of the foregoing, it is intended that the present disclosure cover modifications and variations of this disclosure provided they fall within the scope of the following claims and their equivalents.
Claims (20)
1. An avatar motion generating method, comprising:
generating first avatar motion based on one of predefined motion data, motion sensing data and a combination thereof at a first period of time;
receiving an input event in the first period of time, wherein the input event is related to motion sensing result sensing from the user, and the motion sensing result is determined based on the motion sensing data;
obtaining a behavior state of an avatar from a state machine, wherein the behavior state represents currently performing behavior of the avatar;
determining the input event is met one of a plurality of predefined triggering conditions for a transition among the predefined motion data, the motion sensing data, and the combination thereof, and further comparing a first value of the input event with a second value of the behavior state of the avatar;
in response to the first value of the input event being higher than the second value of the behavior state of the avatar and the input event being met one of the plurality of predefined trigger conditions, generating second avatar motion based on another of the predefined motion data, the motion sensing data, the combination thereof, and the comparing result with the behavior state of the avatar at a second period of time; and
in response to the first value of the input event being lower than the second value of the behavior state of the avatar but the input event being met one of the plurality of predefined triggering conditions, generating the second avatar motion based on the same one of the predefined motion data, the motion sensing data, and the combination thereof as the first period at the second period of time.
2. The avatar motion generating method according to claim 1 , further comprising:
performing the step of generating the second avatar motion in response to the input event being met anyone of the plurality of predefined triggering conditions.
3. (canceled)
4. The avatar motion generating method according to claim 1 , wherein the step of comparing the input event with the behavior state comprises:
switching between the predefined motion data and the motion sensing data in response to the input event having higher priority than the behavior state, wherein a priority of the input event and a priority of the behavior state are fixed, the priority of the input event is the first value, and the priority of the behavior state is the second value.
5. The avatar motion generating method according to claim 1 , wherein the step of comparing the input event with the behavior state comprises:
switching between the predefined motion data and the motion sensing data in response to the input event having higher weight than the behavior state, wherein a weight of the input event and a weight of the behavior state are variable, the weight of the input event is the first value, and the weight of the behavior state is the second value.
6. The avatar motion generating method according to claim 2 , wherein the step of generating the second avatar motion comprises:
switching between the predefined motion data and the motion sensing data in response to the input event not exist.
7. The avatar motion generating method according to claim 1 , further comprising:
generating a transition motion data according to both the predefined motion data and the motion sensing data; and
generating transition avatar motion based on the transition motion data at a third period of time between the first period and the second period.
8. The avatar motion generating method according to claim 2 , further comprising:
receiving a second input event;
comparing the second input event with the input event to generate a comparing result; and
generating third avatar motion based on motion data of one of the input event and the second input event according to the comparing result.
9. The avatar motion generating method according to claim 1 , wherein the step of generating the first avatar motion based on one of the predefined motion data, the motion sensing data and the combination thereof at the first period of time, further comprises:
generating motion of a first human body portion of an avatar based on the predefined motion data and motion of a second human body portion of the avatar based on the motion sensing data at the same period of time.
10. The avatar motion generating method according to claim 1 , further comprising:
determining a pose of a human body portion based on motion sensing result and inverse kinematics to generate the motion sensing data.
11. A head mounted display system, wearable on a user's head, the head mounted display system comprising:
a memory, storing a program code; and
a processor, coupled to the memory, and loading the program code to perform:
generating first avatar motion based on one of predefined motion data, motion sensing data and a combination thereof at a first period of time;
receiving an input event in the first period of time, wherein the input event is related to motion sensing result sensing from the user, and the motion sensing result is determined based on the motion sensing data;
obtaining a behavior state of an avatar from a state machine, wherein the behavior state represents currently performing behavior of the avatar;
determining the input event is met one of a plurality of predefined triggering conditions for a transition among the predefined motion data, the motion sensing data, and the combination thereof, and further comparing a first value of the input event with a second value of the behavior state of the avatar;
in response to the first value of the input event being higher than the second value of the behavior state of the avatar and the input event being met one of the plurality of predefined trigger conditions, generating second avatar motion based on another of the predefined motion data, the motion sensing data, the combination thereof, and the comparing result with the behavior state of the avatar at a second period of time; and
in response to the first value of the input event being lower than the second value of the behavior state of the avatar but the input event being met one of the plurality of predefined triggering conditions, generating the second avatar motion based on the same one of the predefined motion data, the motion sensing data, and the combination thereof as the first period at the second period of time.
12. The head mounted display system according to claim 11 , wherein the processor is configured to perform:
performing the step of generating the second avatar motion in response to the input event being met anyone of the plurality of predefined triggering conditions.
13. (canceled)
14. The head mounted display system according to claim 11 , wherein the processor is further configured to perform:
switching between the predefined motion data and the motion sensing data in response to the input event having higher priority than the behavior state, wherein a priority of the input event and a priority of the behavior state are fixed the priority of the input event is the first value, and the priority of the behavior state is the second value.
15. The head mounted display system according to claim 11 , wherein the processor is further configured to perform:
switching between the predefined motion data and the motion sensing data in response to the input event having higher weight than the behavior state, wherein a weight of the input event and a weight of the behavior state are variable, the weight of the input event is the first value, and the weight of the behavior state is the second value.
16. The head mounted display system according to claim 12 , wherein the processor is further configured to perform:
switching between the predefined motion data and the motion sensing data in response to the input event not exist.
17. The head mounted display system according to claim 12 , wherein the processor is further configured to perform:
generating a transition motion data according to both the predefined motion data and the motion sensing data; and
generating transition avatar motion based on the transition motion data at a third period of time between the first period and the second period.
18. The head mounted display system according to claim 12 , wherein the processor is further configured to perform:
receiving a second input event;
comparing the second input event with the input event to generate a comparing result; and
generating third avatar motion based on motion data of one of the input event and the second input event according to the comparing result.
19. The head mounted display system according to claim 11 , wherein the processor is further configured to perform:
generating motion of a first human body portion of an avatar based on the predefined motion data and motion of a second human body portion of the avatar based on the motion sensing data at the same period of time.
20. The head mounted display system according to claim 11 , wherein the processor further performs:
determining a pose of a human body portion based on motion sensing result and inverse kinematics to generate the motion sensing data.
Priority Applications (7)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/676,416 US10997766B1 (en) | 2019-11-06 | 2019-11-06 | Avatar motion generating method and head mounted display system |
EP19213016.9A EP3819011A1 (en) | 2019-11-06 | 2019-12-02 | Avatar motion generating method and head mounted display system |
JP2019219321A JP2021077310A (en) | 2019-11-06 | 2019-12-04 | Avatar motion generation method and head-mounted display system |
TW108145008A TW202119198A (en) | 2019-11-06 | 2019-12-10 | Avatar motion generating method and head mounted display system |
CN201911288899.0A CN112764525B (en) | 2019-11-06 | 2019-12-12 | Virtual avatar motion generation method and head-mounted display system |
TW109135053A TWI851831B (en) | 2019-10-24 | 2020-10-08 | Human body portion tracking method and human body portion tracking system |
CN202011133133.8A CN112712545A (en) | 2019-10-24 | 2020-10-21 | Human body part tracking method and human body part tracking system |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/676,416 US10997766B1 (en) | 2019-11-06 | 2019-11-06 | Avatar motion generating method and head mounted display system |
Publications (2)
Publication Number | Publication Date |
---|---|
US10997766B1 US10997766B1 (en) | 2021-05-04 |
US20210134040A1 true US20210134040A1 (en) | 2021-05-06 |
Family
ID=68766513
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/676,416 Active US10997766B1 (en) | 2019-10-24 | 2019-11-06 | Avatar motion generating method and head mounted display system |
Country Status (5)
Country | Link |
---|---|
US (1) | US10997766B1 (en) |
EP (1) | EP3819011A1 (en) |
JP (1) | JP2021077310A (en) |
CN (1) | CN112764525B (en) |
TW (1) | TW202119198A (en) |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR102570009B1 (en) * | 2019-07-31 | 2023-08-23 | 삼성전자주식회사 | Electronic device and method for generating argument reality object |
JP7197217B2 (en) * | 2021-06-04 | 2022-12-27 | 株式会社コナミアミューズメント | Game system, computer program used therefor, and control method |
KR20230021924A (en) * | 2021-08-06 | 2023-02-14 | 삼성전자주식회사 | Electronic device and method for displaying character object based on a priority of states in the electronic device |
TWI808787B (en) * | 2022-06-15 | 2023-07-11 | 英業達股份有限公司 | Methods of automatically classifying transition motions |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6329994B1 (en) * | 1996-03-15 | 2001-12-11 | Zapa Digital Arts Ltd. | Programmable computer graphic objects |
US20090213123A1 (en) * | 2007-12-08 | 2009-08-27 | Dennis Allard Crow | Method of using skeletal animation data to ascertain risk in a surveillance system |
US20120223952A1 (en) * | 2011-03-01 | 2012-09-06 | Sony Computer Entertainment Inc. | Information Processing Device Capable of Displaying A Character Representing A User, and Information Processing Method Thereof. |
Family Cites Families (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP4559092B2 (en) | 2004-01-30 | 2010-10-06 | 株式会社エヌ・ティ・ティ・ドコモ | Mobile communication terminal and program |
JP4848515B2 (en) | 2006-06-12 | 2011-12-28 | 国立大学法人九州工業大学 | Avatar motion control system, program and method thereof |
US8988437B2 (en) * | 2009-03-20 | 2015-03-24 | Microsoft Technology Licensing, Llc | Chaining animations |
US9245176B2 (en) * | 2012-08-01 | 2016-01-26 | Disney Enterprises, Inc. | Content retargeting using facial layers |
JP2016509292A (en) * | 2013-01-03 | 2016-03-24 | メタ カンパニー | Extramissive spatial imaging digital eyeglass device or extended intervening vision |
US9325936B2 (en) | 2013-08-09 | 2016-04-26 | Samsung Electronics Co., Ltd. | Hybrid visual communication |
CN105578954B (en) | 2013-09-25 | 2019-03-29 | 迈恩德玛泽控股股份有限公司 | Physiological parameter measurement and feedback system |
US9633463B2 (en) | 2014-09-24 | 2017-04-25 | Intel Corporation | User gesture driven avatar apparatus and method |
US10684485B2 (en) * | 2015-03-06 | 2020-06-16 | Sony Interactive Entertainment Inc. | Tracking system for head mounted display |
GB2548154A (en) | 2016-03-11 | 2017-09-13 | Sony Computer Entertainment Europe Ltd | Virtual reality |
US20180359448A1 (en) | 2017-06-07 | 2018-12-13 | Digital Myths Studio, Inc. | Multiparty collaborative interaction in a virtual reality environment |
JP6583395B2 (en) * | 2017-11-24 | 2019-10-02 | マツダ株式会社 | Driving assistance device |
GB2571956B (en) | 2018-03-14 | 2022-04-27 | Sony Interactive Entertainment Inc | Head-mountable apparatus and methods |
US11017576B2 (en) * | 2018-05-30 | 2021-05-25 | Visyn Inc. | Reference model predictive tracking and rendering |
JP7030669B2 (en) | 2018-10-09 | 2022-03-07 | 株式会社コロプラ | An information processing method, a device, and a program for causing a computer to execute the information processing method. |
-
2019
- 2019-11-06 US US16/676,416 patent/US10997766B1/en active Active
- 2019-12-02 EP EP19213016.9A patent/EP3819011A1/en not_active Withdrawn
- 2019-12-04 JP JP2019219321A patent/JP2021077310A/en active Pending
- 2019-12-10 TW TW108145008A patent/TW202119198A/en unknown
- 2019-12-12 CN CN201911288899.0A patent/CN112764525B/en active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6329994B1 (en) * | 1996-03-15 | 2001-12-11 | Zapa Digital Arts Ltd. | Programmable computer graphic objects |
US20090213123A1 (en) * | 2007-12-08 | 2009-08-27 | Dennis Allard Crow | Method of using skeletal animation data to ascertain risk in a surveillance system |
US20120223952A1 (en) * | 2011-03-01 | 2012-09-06 | Sony Computer Entertainment Inc. | Information Processing Device Capable of Displaying A Character Representing A User, and Information Processing Method Thereof. |
Also Published As
Publication number | Publication date |
---|---|
CN112764525B (en) | 2024-03-12 |
EP3819011A1 (en) | 2021-05-12 |
TW202119198A (en) | 2021-05-16 |
CN112764525A (en) | 2021-05-07 |
US10997766B1 (en) | 2021-05-04 |
JP2021077310A (en) | 2021-05-20 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10997766B1 (en) | Avatar motion generating method and head mounted display system | |
US10223064B2 (en) | Method for providing virtual space, program and apparatus therefor | |
US10877562B2 (en) | Motion detection system, motion detection method and computer-readable recording medium thereof | |
US10216261B2 (en) | Method of suppressing VR sickness, system for executing the method , and information processing device | |
US11049325B2 (en) | Information processing apparatus, information processing method, and program | |
CN113031840B (en) | False triggering prevention method and device for wrist-worn device, electronic device and storage medium | |
EP3809236A1 (en) | Avatar facial expression generating system and method of avatar facial expression generation | |
US11460912B2 (en) | System and method related to data fusing | |
US11029753B2 (en) | Human computer interaction system and human computer interaction method | |
US10948978B2 (en) | Virtual object operating system and virtual object operating method | |
US11087520B2 (en) | Avatar facial expression generating system and method of avatar facial expression generation for facial model | |
TWI815593B (en) | Method and system for detecting hand gesture, and computer readable storage medium | |
JP6278546B1 (en) | Information processing method, apparatus, and program for causing computer to execute information processing method | |
EP3912696A1 (en) | Method of interacting with virtual creature in virtual reality environment and virtual object operating system | |
US11892625B2 (en) | Method for determining posture of user, host, and computer readable medium | |
TW202117499A (en) | Virtual object operating system and virtual object operating method | |
US11783492B2 (en) | Human body portion tracking method and human body portion tracking system | |
KR102520841B1 (en) | Apparatus for simulating of billiard method thereof | |
JP2018206353A (en) | Information processing method, apparatus, and program for implementing that information processing method in computer | |
CN117133045A (en) | Gesture recognition method, device, equipment and medium | |
US20200089940A1 (en) | Human behavior understanding system and method | |
EP4016252A1 (en) | System and method related to motion tracking | |
US20210157395A1 (en) | Motion sensing data generating method and motion sensing data generating system | |
EP3832435A1 (en) | Motion tracking system and method | |
TW202122972A (en) | Motion sensing data generating method and motion sensing data generating system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
AS | Assignment |
Owner name: XRSPACE CO., LTD., TAIWAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HONG, WEI-ZHE;HSIEH, PEI-WEN;SIGNING DATES FROM 20190819 TO 20190820;REEL/FRAME:050988/0154 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: SMAL); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |