WO2013139181A1

WO2013139181A1 - User interaction system and method

Info

Publication number: WO2013139181A1
Application number: PCT/CN2013/070608
Authority: WO
Inventors: 刘广松
Original assignee: 乾行讯科（北京）科技有限公司
Priority date: 2012-03-19
Filing date: 2013-01-17
Publication date: 2013-09-26
Also published as: CN102789313B; CN102789313A

Abstract

Disclosed are a user interaction system and method. The system comprises an information operation processing unit which is used for providing a 3D interface display signal for a 3D display unit; the 3D display unit which is used for displaying a 3D interface for a user according to the 3D interface display signal; and an action acquisition unit which is used for acquiring limb spatial movement information generated when the user browses the 3D interface, and sending the limb spatial movement information to the information operation processing unit. After the embodiments of the present invention are applied, the user can be immersed in a private and interesting information interaction space and conduct natural information interaction in this space, thereby enhancing the user's interaction experience. Based on this, a large quantity of meaningful applications can be derived, so that the user interacts with the digital information world more naturally.

Description

TECHNICAL FIELD The present invention relates to the field of electronic application technologies, and in particular, to a user interaction system and method. Background technique

In 1959, American scholar B. Shackel first proposed the concept of human-computer interaction engineering. Since the late 1990s, with the rapid development and popularization of high-speed processing chips, multimedia technology and Internet technology, human-computer interaction research has focused on intelligent interaction, multi-modal (multi-channel) multimedia interaction. Virtual interaction and Human-machine interaction and other aspects, that is, the human-centered human-machine with the advancement of society and the era of information explosion, people rely more and more on a variety of consumer electronic devices (such as mobile terminals, individuals) Digital Assistant (PM), etc.) obtain a variety of information. For example: Call to communicate with others, browse the web to get news and check emails. The widely used human-computer interactions include hardware devices such as traditional keyboard and mouse, and touch screens that have become popular in recent years.

People are not satisfied with the existing human-computer interaction methods. It is expected that the new generation of human-computer interaction can be as natural, accurate and fast as human-to-human interaction. So in the 1990s, the study of human-computer interaction entered the multi-modal stage, called Human-Computer Nature Interaction (HCNI sk, Human-Machine Nature Interact ion, HMNI).

Virtual reality technology uses computer simulation to generate a virtual world in a three-dimensional space, providing users with simulations of visual, auditory, tactile and other senses, allowing users to observe 3D in a timely and unrestricted manner as if they were immersive. Things in space, and virtual worlds The elements in the world interact. Virtual reality technology has a virtual reality that transcends reality. It is a new computer technology developed with multimedia technology. It uses 3D graphics generation technology, multi-sensor interaction technology and high-resolution display technology to generate 3D realistic virtual environment.

However, how to apply virtual reality technology to various applications of user interaction is still a huge challenge. SUMMARY OF THE INVENTION In view of this, an embodiment of the present invention provides a user interaction system to increase user experience. The embodiment of the invention also proposes a user interaction method to enhance the user experience.

The technical scheme of the present invention is as follows:

A user interaction system, the system includes an information operation processing unit, a three-dimensional display unit information operation processing unit, for providing a three-dimensional stereoscopic display unit with a three-dimensional stereoscopic display signal, and a three-dimensional display unit for The three-dimensional interface display signal displays a three-dimensional interface to the household;

a motion capture unit, configured to capture a body space movement information made by the user browsing the three-dimensional interface, and send the limb space movement information to an information operation processing unit;

The information operation processing unit is further configured to determine an interaction command corresponding to the movement information of the user's limb space, and provide a three-dimensional stereoscopic display unit with a corresponding three-dimensional interface display signal corresponding to the execution of the interactive operation command.

The information operation processing unit is a mobile terminal, a computer or a cloud computing-based information service platform. The system further includes a viewing angle sensing unit worn on the user's head;

a view sensing unit, configured to sense user head motion information, and send the user head motion information to an information operation processing unit;

An information operation processing unit, further configured to determine a user's real-time view according to the user's head motion information Angle information, and real-time display to the three-dimensional display unit based on the three-dimensional interface display message number of the user in real-time view.

The system further includes a sound processing unit;

a sound processing unit, configured to capture voice input information of the user, send the voice input information to the information operation processing unit; and play the voice play information provided by the information operation processing unit to the user; the information operation processing unit is further used for Determining a voice operation command according to the voice input information, and providing a three-dimensional stereo interface unit with a three-dimensional stereo interface signal for performing the voice operation; and providing the sound processing unit with a voice play related to the three-dimensional interface display signal information.

The three-dimensional stereoscopic display unit is a head mounted device.

The three-dimensional display unit and the perspective-aware unit are physically integrated as a portable user wearable unit.

The three-dimensional display unit, the angle-of-view sensing unit, and the sound processing unit are physically integrated into a portable, wearable whole.

The three-dimensional stereoscopic display unit, the viewing angle sensing unit, and the motion capture unit are physically integrated into a portable user-capable device or a casual wearable device.

The three-dimensional stereoscopic display unit, the viewing angle sensing unit, the sound processing unit, and the motion capture unit are physically integrated into a portable user wearable device or a portable wearable device.

The motion capture unit is a portable wearable device, or is fixed at a location where the user can capture the user action.

The information operation processing unit is further configured to display a spatial virtual pointer element corresponding to the user's hand on the three-dimensional interface;

a motion capture unit, configured to capture, in real time, user hand position shape information that is generated by the user in response to browsing the three-dimensional interface;

An information operation processing unit, configured to determine an interaction operation command corresponding to the user hand position pattern information according to the user hand position pattern information, and output an image signal of the space virtual pointer element in real time, Therefore, the motion trajectory of the spatial virtual pointer element on the three-dimensional interface is consistent with the trajectory of the user's hand, and the user is presented with a three-dimensional interface display signal for performing the interactive command corresponding to the position information of the user's hand in real time. .

A user interaction method, the method comprising:

Providing a three-dimensional interface display signal;

Displaying a three-dimensional stereo interface to the user according to the three-dimensional interface display signal;

Capturing the user's browsing of the three-dimensional interface to make the body space movement information;

An interactive operation command corresponding to the movement information of the user's limb space is determined, and a three-dimensional interface corresponding to the hand-held, multi-digit, and the '_ command is displayed in real time.

The information about the movement of the limb space made by the user to browse the three-dimensional interface is: capturing a precise positioning operation and/or an inaccurate positioning operation made by the user browsing the three-dimensional interface.

The precise positioning operation comprises: a user hand movement control space virtual pointer element freely moving in a three-dimensional direction in the three-dimensional stereo interface; recognizing two different states of the user hand and a spatial change of the corresponding hand when the state changes The position in the three-dimensional interface, wherein the two states of the hand include a "fist state" and a state in which only the index finger is extended; a button of the three-dimensional interface is clicked; or a specific area on the three-dimensional interface is selected.

The precise positioning operation includes: hand hovering, hand swiping from right to left, hand moving from left to right 3⁄4, hand swiping from top to bottom, hand swiping from bottom to top, or hands apart or gathering, or Wave your hand.

The method further includes the step of initially obtaining an initial correction setting of the user interaction habit.

The method further includes:

Sensing user head motion information;

Determining a real-time perspective of the user according to the user's head motion information;

The three-dimensional interface display signal of the user in real-time view is provided in real time.

The method further includes:

Capturing the user's voice input information; Determining a voice operation command according to the voice input information;

Providing a three-dimensional interface signal after performing the voice operation as a command.

As can be seen from the above technical solutions, in the embodiments of the present invention, a novel user interaction system and method is proposed. After applying the embodiment of the present invention, the user can be immersed in a private and interesting virtual information interaction space, and perform natural information interaction in the space. Many of the products developed based on embodiments of the present invention will be consumer electronics products with market competitiveness. The unique natural interaction solution of the embodiment of the present invention will promote the development of products and applications in the field of consumer-level virtual reality and augmented reality, greatly improve the user's interactive experience, and can generate a series of meaningful applications, recognizing greatly Enhanced user experience.

Moreover, the embodiment of the present invention proposes a natural interaction solution integrating voice interaction, gesture interaction, natural angle transformation and the like. With this solution, users can naturally interact with elements in the 3D virtual information natural interaction space interface to obtain information or entertain, creating an immersive, unique and attractive user experience.

In addition, an embodiment of the present invention provides a three-dimensional virtual information natural interaction interface of a natural interaction technology, and the interaction interface includes a plurality of three-dimensional solid elements that can perform natural interaction. During the interaction process, the interface is designed to provide interactive feedback to the user in real time through sound, interactive changes in light and shadow, etc., enhancing the user's natural interaction fun and experience. With the solution proposed by the embodiment of the present invention, the user can naturally control the virtual pointer corresponding to the user's hand in the natural interaction interface of the three-dimensional virtual information, and naturally interact with the natural interactive interface of the three-dimensional virtual information.

In addition, the embodiments of the present invention can be applied to any display device and an interactive interface. The solution of adding a pointer corresponding to the user's hand in real time on the interactive interface can facilitate the user to perform a series of precise touch interaction operations. Moreover, this kind of interaction is very natural, conforms to the basic gesture interaction mode of human nature, and reduces the learning cost of the user on the operating device. This interaction is in line with the natural interaction between the human body and the portable information processing hardware device, enabling people to concentrate more on their concerns. Information rather than the hardware device itself.

Not only that, the embodiment of the present invention can be applied to any human-machine interaction information device, and its versatility will bring great convenience to people. BRIEF DESCRIPTION OF DRAWINGS FIG. 1 is a schematic structural diagram of a weekly interaction system according to an embodiment of the present invention;

2 is a schematic flowchart of a user interaction method according to an embodiment of the present invention;

3 is a schematic diagram of a three-dimensional virtual information natural interaction interface according to an embodiment of the present invention; FIG. 4 is a schematic diagram of a three-dimensional virtual information natural interaction interface according to another embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS In order to make the technical solutions and advantages of the embodiments of the present invention more clearly, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

In general, users need to enter a virtual environment through special interactive devices.

A complete virtual reality system can be composed of virtual environment, virtual environment processor with high-performance computer as the core, vision system with helmet display as the core, auditory system with speech recognition sound synthesis and sound localization as the core, and azimuth tracking. The device, the data glove and the data clothing are the main body position and posture tracking devices, and the functional units such as taste, smell, touch and force feedback system.

In one embodiment, embodiments of the present invention propose a new and complete consumer-grade virtual reality device solution. With this solution, users can fully immerse themselves in a new immersive 3D human-machine interaction interface through a first-person perspective with a 3D stereoscopic display and related sensing devices that provide natural interaction. Natural information interactions such as changes in speech, gestures, and head angles are performed. FIG. 1 is a schematic structural diagram of a user interaction system according to an embodiment of the present invention.

As shown in FIG. 1, the system includes an information operation processing unit 101, a three-dimensional stereoscopic display unit 102, and a motion capture unit 103, wherein:

An information operation processing unit 101, configured to provide a three-dimensional stereoscopic display unit to the three-dimensional display unit ¹ ;

The three-dimensional display unit 102 is configured to display a three-dimensional interface to the user according to the three-dimensional interface display signal;

The action capture unit 103 is configured to capture the body space movement information made by the user browsing the three-dimensional interface, and send the limb space movement information to the information operation processing unit 101;

The information operation processing unit 101 is further configured to determine an interactive operation command corresponding to the user's limb space movement information, and provide the three-dimensional stereoscopic display unit 102 with a three-dimensional stereo interface display signal corresponding to the execution of the interactive operation command in real time.

Preferably, the system further includes a perspective perception unit 104 that is worn on the user's head.

The view sensing unit 104 is configured to sense user head motion information, and send the user head motion information to the information operation processing unit 101;

The information operation processing unit 101 is further configured to determine a real-time viewing angle of the user according to the user's head motion information, and provide a three-dimensional stereoscopic interface display signal based on the real-time viewing angle of the user to the three-dimensional stereoscopic display unit 102 in real time.

Preferably, the system further includes a sound processing unit 105.

The sound processing unit 105 is configured to capture voice input information of the user, send the voice input information to the information operation processing unit 101, and play the voice play information provided by the information operation processing unit 101 to the user;

The information processing unit 101 is further configured to input information to determine the voice according to the voice operation command to the three-dimensional display unit ¹⁰² provides interface signals performs three-dimensional operation of the voice command; and a voice processing unit to: provide 105 The three-dimensional interface displays signal related Voice playback information.

The three-dimensional display unit 102 can be embodied as a head mounted device, such as a glasses-type three-dimensional display device. The glasses-type three-dimensional display device forms a parallax by the user's eyes by controlling the slight difference in the screen displayed by the micro-displays corresponding to the left and right eyes, and the brain interprets the parallax of both eyes and judges the object's distance and stereoscopic vision.

The information operation processing unit 1 01 may be any device capable of providing a three-dimensional interface display signal. The information operation processing unit 1 Q1 can be derived from any information acquisition device such as a mobile terminal, a computer, or a cloud-based information service platform.

The information operation processing unit 01 can perform a certain operation (such as mobile phone dialing, browsing webpage, etc.) by processing the corresponding interactive processing command through its built-in operating system, and update the corresponding three-dimensional stereo interface display signal in real time through wired or wireless mode, and output the three-dimensional image. The stereoscopic interface display signal is displayed to the three-dimensional display unit 102.

Preferably, the communication manner between the information operation processing unit 101 and the three-dimensional stereoscopic display unit 102 can be implemented in various specific forms, including but not limited to: wireless broadband transmission, wif i transmission, Bluetooth transmission, infrared transmission, and mobile Communication transmission, USB transmission or wired transmission, etc.

Correspondingly, the communication manner between the information operation processing unit 101 and the action capture unit 103, the view perception unit 104 or the sound processing unit 105 may also have various specific implementation forms, including but not limited to: wireless broadband transmission, wi Fi transmission, Bluetooth transmission, infrared transmission, mobile communication transmission, USB transmission or wired transmission, and the like.

The sound processing unit 1 05 may include an array sound collection sensor, a speaker module, and a data transmission module. The sound processing unit 105 may capture the voice information input by the user by using the sound collection sensor, and transmit the captured user voice data information to the information operation processing unit 101 to perform further recognition processing by the information operation processing unit 101; the sound processing unit 1 05 is also used to receive and process various speech signals from the information operation processing unit 101 to provide various audible feedback to the user.

Specifically, the motion capture unit 103 may include an optical depth sensor and a data transmission module. The depth image of the user's hands or one hand is obtained in real time by the optical depth sensor, and these depth images are transmitted to the information operation processing unit 101 in real time through the data transmission module, and the information operation processing unit is

101 performs analysis processing to obtain user gesture interaction intention information.

Specifically, the viewing angle sensing unit 04 may include a micro electronic sensor such as a gyroscope, an accelerometer, an electronic compass, and a data transmission module. The view sensing unit 104 can be fixedly worn on the user's head for sensing the user's head motion information, and transmitting the corresponding data information to the information operation processing unit 101 through the data transmission module, and further analyzing and obtaining the user by the information operation processing unit 101. Real-time viewing direction and change information.

The information operation processing unit 101 receives relevant data information provided in real time by the processing sound processing unit 105, the motion capturing unit 103, and the angle sensing unit 104, and updates in real time to provide a three-dimensional interface display signal to the three-dimensional display unit 102.

The information operation processing unit 101 has corresponding computing capabilities and is capable of communicating with other units. The information operation processing unit 101 receives and analyzes the data information from the units such as the sound processing unit 105, the motion capture unit 103, and the view perception unit 104, analyzes the user interaction intention in real time, and combines the three-dimensional natural interaction interface exclusive to the system to utilize the three-dimensional graphics. The generating technology real-time renders the current user perspective and the three-dimensional virtual environment updated in real time under the natural interaction operation, and converts the three-dimensional stereoscopic display signal into the three-dimensional stereoscopic display unit 102 in real time.

In a specific implementation, in order to facilitate application to various specific application scenarios, the three-dimensional display unit 102 can be a portable head mounted device; and the view sensing unit 104 can be a portable head mounted device.

Optionally, the three-dimensional stereoscopic display unit 102 and the viewing angle sensing unit 104 can also be integrated into a portable head mounted device.

Alternatively, the three-dimensional stereoscopic display unit 102, the viewing angle sensing unit 104, and the sound processing unit 105 may be integrated into one portable head mounted device or wearable device.

Optionally, the three-dimensional stereoscopic display unit 102, the perspective perception unit 104, and the gesture recognition module may be The block is integrated into a portable head mounted device or wearable device;

Optionally, the three-dimensional stereoscopic display unit 102, the viewing angle sensing unit 104, the motion capturing unit 103, and the sound processing unit 105 may be integrated into a portable head mounted device or a wearable device.

Alternatively, the motion capture unit -03 may be a portable wearable device or fixed at the user's external location to capture the user action location.

The motion capture unit 103 can be worn on the chest, even on the head (e.g., glasses), etc., to facilitate capturing human motion from the 。.

In one embodiment, when the user wears the three-dimensional display unit 102 and the view perception unit 104, and is connected with the information operation processing unit 101 and other units, the user can feel that he has entered a natural interactive virtual three-dimensional information space. .

3 is a schematic diagram of a three-dimensional virtual virtual information natural interaction interface according to an embodiment of the present invention; FIG. 4 is a schematic diagram of a three-dimensional virtual information natural interaction interface according to another embodiment of the present invention.

The three-dimensional information space can contain a lot of user-defined content. For example, it may include a user's virtual pet, a user's favorite news section, a user's mail, and the like. All virtual environments are rendered in real time by the information processing unit 1 01 using 3D graphics generation technology, and presented to the user in a three-dimensional dynamic form.

After the user performs an action of changing the angle of view such as looking up or turning, the sensor in the view sensing unit 104 acquires the processing related data information in real time, and transmits the quantized measurement result of the user's real-time action state to the information operation processing unit 101. The information operation processing unit 101 further analyzes and processes the real-time viewing angle information of the user, and the obtained real-time viewing angle information is used for real-time rendering of the three-dimensional virtual environment in a new viewing angle state, so that the user can perform the action of changing the viewing angle such as looking up or turning the head. The virtual environment that is seen can be adjusted in real time according to the perspective change of the household, so that the user can feel that in the virtual space, the content of the corresponding viewing direction of the virtual space can be naturally viewed through the action of changing the head, such as a head-turning head. In addition, the feeling of head-turning in the virtual space is as close as possible to the feeling of making corresponding actions in real space. Users can also interact with the three-dimensional virtual space environment they see through voice. For example, when the user says "menu", the voice signal is collected and analyzed by the array microphone of the sound processing unit 105 as a data signal, and transmitted to the information operation processing unit 101 for analysis. The statistical matching analysis algorithm analyzes that the user's interaction intention is The interactive menu is opened, and the information operation processing unit 101 will control the three-dimensional virtual twisted space interaction interface to wipe the interactive command, and an interactive menu will appear in the user's field of view.

After the user's hand enters the detection range of the motion capture unit 103, the motion capture unit 103 captures and transmits the depth image sequence of the user's hand to the information operation processing unit 101 in real time, and the information operation processing unit 101 performs real-time through a series of software algorithms. The depth image sequence of the weekly hand is analyzed to obtain the movement trajectory of the user's hand, and the user's gesture interaction intention is determined by the analysis.

At the same time, the corresponding virtual pointer in the user's field of view corresponds to the motion and position of the user's hand, and the user is fed back with corresponding gestures.

Illustratively, the following is a description of the system workflow by taking the user's hand up, moving, and clicking actions as an example.

After the user raises the hand into the detection range of the motion capture unit 103, the motion capture unit 103 acquires the depth image sequence of the current field of view in real time and transmits it to the information operation processing unit 101, and the corresponding match recognition in the information operation processing unit 101. The algorithm analyzes the received depth image sequence in real time, analyzes the user's hand and recognizes it as an effective feature tracking object, and uses its depth image data to analyze its three-dimensional position information in real time. At this time, the three-dimensional virtual space in the user's field of view A virtual pointer will appear in the environment corresponding to the user's hand, and the user controls the movement of the virtual pointer just like controlling his own hand.

Preferably, there is a virtual button at the upper left of the user's field of view, and the user controls the virtual pointer to move to the virtual button by hand. At this time, the virtual pointer shape changes, prompting the user that the position can be clicked. At this time, the user makes a click action or the form of the hand changes from the state of extending only the index finger to the state of the fist, and the information operation processing unit 101 analyzes the user's hand from the action capture unit 103. The depth image sequence analyzes the motion trajectory and morphological changes of the user's hand, and analyzes and judges that the user's interaction intention is click and confirm through a series of redundant motion matching algorithms. At this time, the information operation processing unit 101 controls the three-dimensional image. The three-dimensional virtual interaction space interface performs the interaction intention, and presents the interaction result after the execution of the interaction intention to the user in real time.

In the embodiment of the present invention, various electronic devices (such as portable electronic devices) in the prior art are limited to the disadvantage of using a physical touch screen or a keyboard as an interactive means, and the three-dimensional display unit 102 is provided with three-dimensional information by the information operation processing unit 101. The stereoscopic interface displays signals and interacts by recognizing the user's limb space movement information made by the user for the three-dimensional interface.

Moreover, the embodiment of the present invention simultaneously proposes a humanized interaction scheme based on recognition of a human limb motion (preferably a human gesture) for the aforementioned three-dimensional stereo interface, and the interaction scheme can seamlessly integrate the aforementioned three-dimensional interface and the body motion of the human body. Manipulate the information. By optimizing some basic and typical operational recognition processes, a stable interactive development platform is developed for developers to develop a wide range of applications.

Still further, embodiments of the present invention provide an accurate interactive solution. Through the interaction scheme, the user can interact with any interactive interface through a touch operation conforming to the natural interaction mode of the human body ₍

In one embodiment, the motion capture unit 1 0 3 is selected as an infrared depth imaging sensor device, and the user limb space movement information is an image signal containing depth of field information captured by the infrared depth imaging sensor device.

The information operation processing unit 101 receives and analyzes the depth image information of the user's limb movement from the motion capture unit 103, and analyzes the user interaction intention (ie, the interactive operation command) through software algorithm analysis, and provides the three-dimensional display unit in real time. Corresponding to the three-dimensional interface display signal after the execution of the interactive operation command.

In one embodiment, the information operation processing unit 101 firstly analyzes the real-time position information of the user's limb based on the received real-time image information. And store the user's limb for a certain length of time (than Historical location information such as the hand is for further user interaction intent judgment. The further identified user interaction intent includes a simple moving operation of the user with one or both hands (default), a one-hand or two-handed drag operation or a one- or two-handed click, a stay, a swing operation, and the like.

In one embodiment, the information operation processing unit 101 is further configured to display a spatial virtual pointer element corresponding to the user's hand on the three-dimensional interface; the action capturing unit 103 is configured to capture the user in response to browsing the three-dimensional interface in real time. And the user's hand position form information is made;

The information operation processing unit 101 is configured to determine an interaction operation command corresponding to the user hand position pattern information according to the user hand position pattern information, and output an image signal of the space virtual pointer element in real time, thereby realizing the three-dimensional interface The motion trajectory of the space virtual pointer element is consistent with the trajectory of the weekly hand movement, and provides the user with a three-dimensional interface display signal after performing the interactive operation command corresponding to the position information of the user's hand in real time.

For example, if the user's hand swipe from right to left across the field of view of the motion capture unit 103, the action capture unit 103 records and transmits the image data to the information operation processing unit 101 in real time. The information operation processing unit ί 01 is analyzed from the image data through a series of software algorithms. The user gesture track is swiped from right to left, and then determined by the software algorithm as an interactive command (for example: return to the previous page), Process this command data stream and give feedback to the user.

In the actual interaction process, the information operation processing unit (U can recognize a series of interactive commands. For example: "Start interaction / OK / select / click",, "Move (up and down, left, right front, back)", " The gesture actions of zooming in, "zooming out", "rotating", "exiting/ending the interaction", etc., are converted into interactive operation commands in real time and correspondingly executed, and then the corresponding interactive display state is output to the user.

An exemplary complete interaction process is described below to better illustrate embodiments of the present invention.

If the hand of the household is swiped from the right to the left across the field of view of the motion capture unit 103, and the "user's hand from right to left" is preset, the limb action corresponds to the "return to the previous page" interactive command.

(The correspondence between the limb motion and the interactive operation command can be saved in advance in the information operation processing unit 101) First, the action capturing unit 103 records and transmits image data to the information operation processing unit 101 in real time. The information operation processing unit 101 analyzes from the image data through a series of software algorithms to draw the user gesture track from right to left, and then determines by the software algorithm that the gesture corresponds to the command of "returning to the previous page", and then executes " Returns the command processing of the previous page, and outputs the display state after the "return to the previous page" is executed.

Preferably, the information operation processing unit 101 has self-learning capability and a certain user-defined extension function, and the user can improve the gesture recognition capability of the system according to his own gesture habits and can customize various operations according to the user's own preferences. Gestures and ways of doing things. The user interaction recognition software presets a lot of parameters, such as human skin color information, arm length information, etc. In the initial case, the initial values of these parameters are based on statistical averages to satisfy most users, and the system is implemented by software algorithms. Self-learning ability, that is, as the user continues to use, the software can modify some of the parameters according to the user's own characteristics to make the recognition interaction more inclined to target specific user characteristics, and improve the gesture recognition ability of the system.

In addition, the user identification interaction software should also provide a user-defined interface, such as a user-specific gesture track representing a user-defined operation command, thereby realizing the system's personalized customizable features.

More specifically, the user's interaction with the three-dimensional interface is divided into two categories: one is to identify inexact positioning operations, such as "page turning", "forward", "back", etc. Another type is to achieve precision. Positioning operations, such as clicking a button in the interactive interface or selecting a specific area.

The precise positioning operation may include: a user hand movement control space virtual pointer element freely moving in a three-dimensional direction in the three-dimensional stereo interface; recognizing two different states of the user hand and a space virtual virtual 4 pin element corresponding to the hand when the state changes The position in the three-dimensional interface, wherein the two states of the hand include a state of clenching and a state of extending only the index finger; clicking a button of the three-dimensional interface; or selecting a specific area on the three-dimensional interface.

For the identification of inexact positioning operations, it is only necessary to record the movement track information of the analysis hand. For example, the inaccurate positioning operation may include, for example, a hand swiping from right to left, a hand swiping from left to right, a hand swiping from top to bottom, and a hand swiping from bottom to top, and separating, gathering, etc. .

In order to realize the recognition of the precise operation, it is required to track the movement track of the user's hand in real time and correspond to the pointer element on the interactive interface to determine the position of the user to accurately interact with the element on the interactive interface, and the information operation processing unit 101 analyzes and determines the household. The hand trajectory is intended to derive interactive commands, from 3⁄4 to achieve accurate interface.

In the above process, the correspondence between the user's gesture and each specific interactive operation command may be preset. Moreover, such correspondence is preferably editable so that new interactivity commands can be added, or gestures corresponding to interworking commands can be changed based on user habits.

For another example, the technical solution of the present invention will be described below with the purpose of recognizing the user interaction with one-handed click. First, the user raises the signal acquisition capture range into the motion capture unit 103 with one hand (such as the right hand). The user performs a forward click action according to his own habits, and assumes that the entire click action takes 0.5 seconds, and the motion capture unit 103 transmits the collected image information of the user's hand to the information operation processing unit 101 in real time. The information operation processing unit 101 accepts the image information data transmitted in real time, and stores the history image information data for a certain period of time, assuming that the stored history information data has a duration of ls. The software in the motion capture unit 103 analyzes the image information data of the user's hand in the past one second in real time, and obtains the spatial displacement information of the user's hand in the last one second. The logical algorithm determines that the movement trajectory of the user's hand is in accordance with the simple movement, and the total movement trajectory of the user's hand after the 0.5 second is representative that the probability of the user making the click action is sufficiently high (ie, the probability value conforms to a certain pre- A critical criterion is set) to be recognized as a one-click operation. Therefore, at this moment, the information operation processing unit 101 analyzes and obtains a click interaction, and the user has performed a click operation for 0.5 seconds in the past one second, and a click operation at G. 5 seconds. The click operation interaction intention obtained by this analysis is compiled and transmitted to the display signal source through the communication module. It is worth noting that the position of the user's hand is recognized as the default movement operation within 0.5 seconds before this moment. Therefore, the pointer corresponding to the user's hand on the interactive interface is constantly updated in position. When the user first uses this interaction scheme, it is preferred to go through a specific initial correction setting process to make the system software parameters conform to the interaction habit of the user. The initial calibration setting process can include:

Firstly, the user is instructed to extend the two hands into the detection area of the motion capture unit through the three-dimensional interactive display interface, and the image recognition of the user's hands is performed, and the relevant shape parameters for identifying the user's hand are established. Then, through the interactive display interface, the user defines the spatial extent of the hand during the interaction operation, for example, respectively, indicating that the user places the hand on the spatial plane corner point and the two points before and after, and determines the spatial extent of the user's hand interaction operation through image sampling analysis. The parameter value.

Then, the information operation processing unit 101 determines the relative position information of the user hand at each point by the correction setting process transmitted by the action capturing unit 103 to determine the key related parameter in the recognition interaction algorithm, and instructs the user to perform one-handed several times. Or the click operation of the two hands, drag and drop to extract the key parameter information related to the corresponding interaction intention criterion. At this point, the initial calibration setting process ends and is saved as a callable information file for storage. The user can directly call the corresponding file in the future.

Through the initial calibration setting process to determine the key parameters in the recognition interaction algorithm, the interaction scheme can satisfactorily satisfy the interaction habits of any user, and provide a personalized and accurate interactive operation experience for different users.

Based on the above analysis, the embodiment of the present invention also proposes a user interaction method „

2 is a schematic flow chart of a method for interacting with a household according to an embodiment of the present invention.

As shown in Figure 1, the method is wrapped:

Step 201: Provide a three-dimensional interface display signal.

Step 202: Display a three-dimensional interface according to the three-dimensional interface display signal to the user. Step 203: Accomplish the movement information of the limb space made by the user browsing the three-dimensional interface. Step 204: Determine an interaction operation command corresponding to the user's limb space movement information, and provide a three-dimensional interface display signal corresponding to the execution of the interaction operation command in real time.

In one embodiment, capturing the movement information of the limb space made by the user browsing the three-dimensional interface is specifically: capturing a precise positioning operation and z or non-precision determination of the user on the three-dimensional interface Bit operation. The precise positioning operation may include: clicking a button on the three-dimensional interface or selecting a specific area on the interaction interface, and the non-precise positioning operation may specifically include: a hand swiping from right to left, a hand swiping from left to right, a hand Swiping from top to bottom, hand swiping from bottom to top, or two hands apart, gathering, and other specific regular gesture trajectories.

Preferably, the method further comprises an initial correction setting step of acquiring user interaction habits in advance. Includes:

Firstly, the user is instructed to extend the Han hand into the detection area of the motion capture unit through the three-dimensional interface, and the image is sampled and recognized by the user's hands, and the relevant shape parameters for identifying the user's hand are established. Then, through a three-dimensional interface, the user defines the spatial extent of the hand during the interaction, for example, respectively indicating that the user places his hand in each corner of the space (upper left corner, upper right corner, lower left corner, lower right corner, etc.), and before and after At two points, the spatial range phase of the interactive operation of the user hand is determined by image analysis, and then the information operation processing unit determines the relative position information of the user hand at each point by analyzing the correction setting process transmitted by the motion capture unit to determine the recognition interaction. The algorithm measures the key parameters related to the scale, and instructs the user to perform the single-hand or two-hand click operation, and the drag operation extracts the key parameter information related to the corresponding interaction intention criterion. At this point, the initial calibration setting process ends and is saved as a callable information file for storage. The user can directly call the corresponding file in the future.

In one embodiment, the method further comprises:

Sensing user head motion information;

The three-dimensional interface display signal based on the real-time view of the user is provided in real time.

In another embodiment, the method further comprises:

Capturing the user's voice input information;

Determining a voice command according to the voice input information;

A three-dimensional interface signal is provided after the voice command is executed. In summary, in the embodiments of the present invention, a novel weekly interaction device and method are proposed. In the embodiment of the present invention, the information operation processing unit provides a three-dimensional stereoscopic interface display signal to the three-dimensional stereoscopic display unit; the three-dimensional stereoscopic display unit displays a three-dimensional stereoscopic interface to the user according to the three-dimensional stereoscopic interface display signal; and the motion capture unit captures the user browsing the three-dimensional interface. The limb space movement information made by the stereoscopic interface transmits the limb space movement information to the information operation processing unit.

It can be seen that after applying the embodiment of the present invention, the user can be immersed in a private and interesting virtual information interaction space, and natural information interaction is performed in the space.

Many of the products developed based on embodiments of the present invention will be consumer electronics products with market competitiveness. The unique natural interaction solution of the embodiment of the present invention will promote the development of products and applications in the field of consumer-level virtual reality and augmented reality, and greatly enhance the user's interactive experience, and can generate a series of meaningful applications, thereby greatly Enhanced user experience.

In addition, an embodiment of the present invention provides a three-dimensional virtual information natural interaction interface of a natural interaction technology, and the interaction interface includes a plurality of three-dimensional solid elements that can perform natural interaction. During the interaction process, the interface is designed to provide interactive feedback to the user in real time through sound, light and shadow changes of interactive elements, and enhance the user's natural interaction fun and experience.

With the solution proposed by the embodiment of the present invention, the user can naturally control the virtual pointer corresponding to the user's hand in the natural interactive interface of the three-dimensional virtual information, and naturally interact with the natural interactive interface of the three-dimensional virtual information.

In addition, the embodiments of the present invention can be applied to any display device and an interactive interface. The solution of adding a pointer corresponding to the user's hand in real time on the interactive interface can facilitate the user to perform a series of precise touch interaction operations. Moreover, this kind of interaction is very natural, conforms to the basic gesture interaction mode of human nature, and reduces the learning cost of the user's equipment. This interaction is in line with the natural interaction between the human body and the portable information processing hardware device, enabling people to concentrate more on the information they care about.

!3⁄4 is not the hardware device itself „

Not only that, the embodiment of the present invention can be applied to any human-machine interaction information device, and its versatility will bring great convenience to people.

The above is only a preferred embodiment of the embodiments of the present invention, and is not intended to limit the scope of the embodiments of the present invention. All modifications, equivalents, improvements, etc., made within the spirit and scope of the embodiments of the present invention are intended to be included within the scope of the present invention.

Claims

K A user interaction system, characterized in that the system comprises an information operation processing unit three-dimensional display unit and a motion capture unit, wherein:

An information operation processing unit, configured to provide a three-dimensional stereoscopic display signal to the three-dimensional stereoscopic display unit; and a three-dimensional stereoscopic display unit, configured to display a three-dimensional stereoscopic interface to the user according to the three-dimensional stereoscopic interface display signal;

a motion capture unit, configured to capture a body space movement information made by the user browsing the three-dimensional interface, and send the start space movement information to an information operation processing unit;

The information operation processing unit is further configured to determine an interaction operation command corresponding to the movement information of the user's limb space, and provide a three-dimensional vertical interface display signal corresponding to the execution of the interaction command to the three-dimensional display unit in real time.

2. The user interaction system according to claim 1, wherein the information operation processing unit is a mobile terminal, a computer or a cloud computing-based information service platform.

3. The user interaction system according to claim 1, wherein the system further comprises a perspective sensing unit worn on a user's head;

The information operation processing unit is further configured to determine a real-time viewing angle of the user according to the user head motion information, and provide a three-dimensional stereoscopic display unit based on the real-time viewing angle of the user in real time to the three-dimensional stereoscopic display unit.

4. The user interaction system of claim 3, wherein the system further comprises a sound processing unit;

a sound processing unit, configured to capture voice input information of the user, send the voice input information to the information operation processing unit; and play the voice play information provided by the information operation processing unit to the user; the information operation processing unit is further used for Determining a voice operation command according to the voice input information, And providing a three-dimensional stereoscopic display unit with the three-dimensional stereo interface signal after executing the voice operation command; and providing the sound processing unit with the voice playback information related to the three-dimensional stereo interface display signal.

The user interaction system according to claim 1, wherein the three-dimensional stereoscopic display unit is a head mounted device.

6. The user interaction system according to claim 3, wherein the three-dimensional stereoscopic display unit and the perspective perception unit are physically integrated as a portable user wearable unit.

7. The user interaction system according to claim 4, wherein the three-dimensional stereoscopic display unit, the viewing angle sensing unit, and the sound processing unit are physically integrated as a portable user wearable unit.

8. The user interaction system according to claim 3, wherein the three-dimensional stereoscopic display unit, the viewing angle sensing unit, and the motion capture unit are physically integrated into a portable user wearable device or a portable wearable device.

9. The user interaction system according to claim 4, wherein the three-dimensional stereoscopic display unit, the viewing angle sensing unit, the sound processing unit, and the motion capture unit are physically integrated into a portable peripheral wearable device or portable Wear the device.

The user interaction system according to claim 1, wherein the motion capture unit is a portable wearable device, or is fixed at a user action location outside the user.

1 1. The user interaction system according to claim 1, wherein:

An information operation processing unit, configured to determine an interaction operation command corresponding to the user hand position pattern information according to the user hand position pattern information, and output an image signal of the space virtual pointer element in real time, thereby realizing a space on the three-dimensional interface The motion trajectory of the virtual pointer element is consistent with the trajectory of the user's hand, and the user is provided with an interactive operation command corresponding to the position information of the user's hand in real time. The three-dimensional interface displays the signal.

12. A user interaction method, the method comprising:

Displaying a three-dimensional interface display signal;

Capturing the movement information of the limb space made by the user browsing the three-dimensional interface;

An interactive command corresponding to the movement information of the user's limb space is determined, and a three-dimensional vertical body interface display signal corresponding to the command of the interaction is provided in real time.

The user interaction method according to claim 12, wherein the capturing of the limb space movement information by the user browsing the three-dimensional interface is: capturing a precise positioning operation that the user browses the three-dimensional interface and is proud of, Do and/or inaccurate positioning operations.

The user interaction method according to claim 13, wherein the _ precise positioning operation comprises:

The user hand moves the control space virtual pointer element to move freely in the three-dimensional direction in the three-dimensional interface;

Identifying two different states of the user's hand and the position of the spatial virtual pointer element of the corresponding hand in the three-dimensional interface when the state changes, wherein the two states of the hand include a state of clenching and a state of extending only the index finger;

Click on the 3D interface or

Select a specific area on the 3D interface.

15. The user interaction method according to claim 13, wherein the inaccurate positioning operation comprises: hand hovering, hand swiping from right to left, hand swiping from left to right, hand from above Move to the bottom, move your hand from bottom to top, or separate or gather together, or wave your hand.

16. The user interaction method according to claim 12, wherein the method further comprises

17. The user interaction method according to claim 12, wherein the method further comprises Sensing user head motion information; providing a three-dimensional interface display signal based on the real-time perspective of the user in real time.

18. The user interaction method according to claim 12, wherein the method further comprises capturing voice input information of the user;

Determining a voice command according to the voice input information;

A three-dimensional interface signal is provided after the voice command is executed.