CN111047925A - Action learning system and method based on room type interactive projection - Google Patents
Action learning system and method based on room type interactive projection Download PDFInfo
- Publication number
- CN111047925A CN111047925A CN201911244297.5A CN201911244297A CN111047925A CN 111047925 A CN111047925 A CN 111047925A CN 201911244297 A CN201911244297 A CN 201911244297A CN 111047925 A CN111047925 A CN 111047925A
- Authority
- CN
- China
- Prior art keywords
- user
- learning
- action
- motion
- projection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B5/00—Electrically-operated educational appliances
- G09B5/06—Electrically-operated educational appliances with both visual and audible presentation of the material to be studied
- G09B5/067—Combinations of audio and projected visual presentation, e.g. film, slides
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1423—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display
Abstract
The utility model provides a motion learning system and method based on room formula interactive projection, this disclosure has constructed an immersive motion learning virtual environment, the user puts into a room both sides and is equipped with the mirror, wall and ground all have the projected room in, the user can observe self action, change the content that each face projection presents simultaneously in the in-process of study motion, user's sight point need not fix on a certain projection, can change along with the action nature, do not need extra action of turning round, can guarantee better teaching quality.
Description
Technical Field
The disclosure belongs to the technical field of virtual reality, and relates to an action learning system and method based on room-type interactive projection.
Background
The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.
Virtual Reality (VR) technology has been developed vigorously in recent years, and is a powerful potential educational resource with characteristics of imagination, interactivity, immersion, and the like. With the further development of motion capture and virtual reality display equipment, the virtual reality technology can break the limitation of teaching resources and scenes, create effective learning environment and attractive entertainment experience, and enhance the visual experience and emotional experience of users. Training is generally considered to be one of the most natural application fields of virtual reality, and virtual reality technology can provide better visual perception and more comprehensive action display for teaching in sports, and provide higher level of interaction and immersion.
According to the knowledge of the inventor, at present, there is a teaching and training technical scheme based on partial VR for learning accurate action through various interaction methods, however, the problems of the present action learning system are mainly shown in the following:
(1) the coach can not learn the action intuitively due to the shielding of the body of the coach, and the user can not watch and learn the action from all angles.
(2) In sports, dancing and other movements, the user often turns around and turns around, and the characteristic makes the viewpoint of the user not fixed for a long time and needs to turn along with the body, which is not considered by the current VR system.
(3) Due to the lack of precision of motion capture devices, users cannot see the motion of their own driven three-dimensional virtual character in real time for complex motion motions, or for delays.
(4) Feedback and exercise are the two most important conditions for learning motor skills. The existing learning system has no good action reference and action correction mechanism, and a user does not know whether the action made by the user is standard or not, how much the action is deviated and how to correct the action.
Disclosure of Invention
The invention aims to solve the problems and provides an action learning system and method based on room-type interactive projection, the invention constructs an immersive action learning virtual environment, a user is placed in a room with mirrors on two sides of the room, the wall surface and the ground are both provided with projections, the user can observe self action, simultaneously the content presented by projection of each surface is changed in the process of learning movement, the viewpoint of the user does not need to be fixed on the projection of a certain surface, the viewpoint can be naturally changed along with the action, no extra turning action is needed, and better teaching quality can be ensured.
According to some embodiments, the following technical scheme is adopted in the disclosure:
a room-based interactive projection action learning system, comprising:
the multi-projection splicing and fusion module comprises a plurality of clients, wherein each client is used for being responsible for projection and display control of a part of areas in a room and is configured to splice projection pictures of a plurality of projectors on the wall and the ground of the room so as to smoothly transit a fusion zone; performing geometric correction, edge fusion and brightness adjustment on the projection picture to generate a splicing fusion parameter graph and form a rendering picture;
the motion capturing and positioning module is configured to acquire a color image and a depth image of a user, obtain coordinate values of joints of bones of the user in a camera coordinate system through mapping with virtual bones, and then convert the coordinate values into a ground projection screen coordinate system so as to determine the position of the user in a room; capturing a gesture of a user using a motion capture device;
the interaction module is configured to receive interaction mode selection information of a user and play a rendering picture and a learning task according to the mode;
a feedback module configured to make feedback on the learning task in a visual form;
and the communication module is configured to complete communication among a plurality of hosts connected with the projector, and realize direct communication with the client and message transmission among the clients.
As an alternative embodiment, the clients include at least wall clients and floor clients.
The wall client comprises a plurality of hosts and a plurality of projectors and is used for projecting virtual scenes onto different walls, acquiring user gestures and judging the accuracy of user actions; the ground client comprises at least one host and a plurality of projectors, and is used for projecting the virtual scene onto different wall surfaces, acquiring the foot position of the user and judging the accuracy of the foot action of the user;
the server comprises at least one host, an RGBD camera and a set of motion capture devices, and is used for providing user posture and position data and realizing message transmission between clients.
As an alternative embodiment, the motion capture and positioning module acquires a color image and a depth image of the user by using an RGBD camera, obtains coordinate values of joints of the bones of the user in a camera coordinate system through mapping with virtual bones, and then converts the coordinate values into a ground projection screen coordinate system, so as to determine the position of the user in the room; the gestures of the user are captured using a motion capture device, a network connection is established using a network protocol, and calculated based on a joint position data stream.
As an alternative embodiment, the interaction module includes a mode selection module configured to support the user to select different learning modes according to the user's own situation, specifically including a learning mode and an exercise mode.
As an alternative embodiment, the interaction module comprises a human-computer interaction module configured to support voice interaction and posture interaction modes, and the user controls the progress of action learning, the angle of watching the virtual trainer and the playing speed of the standard action by selecting the proper interaction mode.
As an alternative embodiment, the system further comprises a learning quality evaluation module configured to evaluate the motion state of each joint of the user through a motion similarity matching algorithm.
A motion learning method based on room-type interactive projection comprises the following steps:
receiving a selection mode of a user, capturing position and motion information of the user, and providing data drive for a model in a virtual scene for logic judgment;
providing a virtual coach picture, providing decomposition actions to be learned, driving a student model in a virtual scene through motion capture, comparing the student model with the actions of a virtual teacher, and providing an evaluation report.
As an alternative embodiment, the RGBD camera is used to acquire the bone information of the user, specifically, the bone information is the coordinate values (x, y, z) of the foot joint point in the camera coordinate system at the current time. And packaging the names and the coordinates of the joints into a data packet, and sending the data packet to the ground client. The user wears an inertial sensor-based motion capture device that transmits motion data of the whole body joint points in a BVH data stream. The wall client receives the data stream and drives the model in the scene by using the unity plug-in. Besides the skeleton information, the messages sent and received by the client also include the progress of the animation played by the current coach model, the projection on which the current action should be displayed, and the current learning state of the user.
The realization of multi-projection splicing and fusion needs to export a coded parameter map after interactive geometric correction and brightness fusion are carried out through multi-channel correction software. One stores the deformation mapping table, and the other stores the brightness color fusion mapping table. In the development engine Unity3D, the images are subjected to image deformation and luminance/color fusion using shader decoding.
As an alternative implementation mode, the user skeleton information is sent to a ground client, so that whether the user gait is correct or not is judged; the wall client acquires the node information of the whole body of the user through the motion capture equipment, so that whether the user motion is standard or not is judged, and similarity matching is carried out on the user motion and the template motion.
As an alternative embodiment, the specific process of learning includes:
(1) before the action starts to learn, the user is prompted to arrive at an appointed initial position by voice, and the user sends a voice command to enter an exercise mode; playing animation and starting learning;
(2) judging whether to continue or not according to the learning state of the user in each step, if the user does not do the animation learning, not continuing to play the animation learning, but strengthening the current action of the learning; the learning state refers to the similarity degree between footsteps and joints of the body and the action template, the positions of the feet of the user are continuously obtained through the bones of the user, and the action capturing equipment continuously obtains the joint movement data and broadcasts the joint movement data to all clients on the wall surface;
(3) at the end of an action, the user issues a voice command and chooses to continue learning the next action, either repeating the move or returning to the main menu.
In the step (1), the specific process of the exercise mode includes:
(a) after the voice prompt exercise is started, the user starts to exercise at any position in the learning area;
(b) the user drives the virtual model in real time, the system is matched with the actions in the template library through an action matching algorithm to obtain the most similar result, the action of the user in practice is known, and the virtual coach model plays corresponding teaching animation;
(c) at the end of an action, the user may issue voice command controls to continue learning or return to the main menu.
In the step (b), the specific method of action matching is as follows:
taking the data of the animation recorded in advance as a standard reference template R, and taking the data as an M-dimensional vector;
taking the bone data of the user collected in real time as a template T for testing, wherein the template T is an N-dimensional vector;
m and N are animation frame numbers, the length of the action made by the user is not necessarily the same as that of a standard animation, M is not necessarily equal to N, but the dimension of each component is the same, and the dimension is the number of joints;
constructing an MxN distance matrix D, matrix elements Dij＝dist(ri，tj)，ri，tjSequence points of the standard action template and sequence points of the user action template, dijIs a certain element in the matrix, i is between 0-M, j is between 0-N; dist () is a distance computation function that computes the euclidean distance between sequence points; and generating a loss matrix M according to the distance matrix, wherein the calculation method comprises the following steps:
M11＝D11
Mij＝Min(Mi-1,j-1,Mi-1,j,Mi,j-1)+Mi,j
last row and last column M of the loss matrixmnIs the cumulative distance between the two sequences; finding a shortest path from the lower left corner to the upper right corner in the distance matrix to minimize the sum of the element values on the path, i.e. obtaining a distance d from the lower left corner by using a dynamic programming method11To dmnThe shortest path of (2).
A computer readable storage medium having stored therein a plurality of instructions adapted to be loaded by a processor of a terminal device and to perform the steps of the method for motion learning based on interactive projection of a room.
A terminal device comprising a processor and a computer readable storage medium, the processor being configured to implement instructions; the computer readable storage medium is used for storing a plurality of instructions adapted to be loaded by a processor and for executing the steps of the room-based interactive projection motion learning method.
Compared with the prior art, the beneficial effect of this disclosure is:
in an immersive virtual environment formed by multi-surface projection, a user does not need to fix a viewpoint for learning, the user can rotate the head according to a correct posture, and a virtual coach appears on a proper projection screen according to the current orientation of the user by the system; positioning and step tracking a user by using an RGBD (red green blue) camera, and acquiring data of a plurality of nodes on the whole body of the user by using motion capture equipment, wherein the data is used for driving a virtual model and evaluating learning quality; the invention provides two modes: a learning mode and an exercise mode. The learning mode provides detailed instructions and guidance, such as ground gait instructions, with more detailed action per action; the system in the exercise mode may match the most likely action based on the user action, giving feedback.
The user does not need to wear a heavy helmet display, the body burden in the learning process is reduced, and better immersion experience is achieved. The user does not need extra training and only needs to use a natural interaction mode.
The mirrors can be arranged on different walls (such as the left side and the right side) of a room, so that the user can compare the own action with the standard action and quickly correct the posture. And a good learning effect is obtained.
This public expense supports multiple study mode, and the user can select the mode that suits oneself most according to self condition, and is progressive. The method can be applied to action learning such as dancing and sports, and supports multi-user collaborative learning.
Drawings
The accompanying drawings, which are included to provide a further understanding of the disclosure, illustrate embodiments of the disclosure and together with the description serve to explain the disclosure and are not to limit the disclosure.
FIG. 1 is a diagram illustrating the hardware architecture of the present embodiment;
FIG. 2 is a flow chart of the system according to the present embodiment;
FIG. 3(a) is a luminance color fusion parameter graph;
FIG. 3(b) is a deformation map parameter map;
FIG. 4(a) is a user calibration action diagram;
FIG. 4(b) is a diagram of a user-driven robot model;
FIG. 5 is a diagram of an embodiment of the present invention applied to learning.
The specific implementation mode is as follows:
the present disclosure is further described with reference to the following drawings and examples.
It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
An action learning system based on room-type interactive projection creates an immersive action learning virtual environment and is built in a solid room consisting of wall projection, ground projection and a mirror. The reflection function of the mirror is fully utilized to enable the user to observe the self action. The user faces the mirror and follows the projected content in the mirror for learning. The system changes the content presented by the projection of each surface in the process of learning the movement according to the movement characteristics, and the viewpoint of the user does not need to be fixed on the projection of a certain surface and can naturally change along with the movement. During the moving of the viewpoint, the user can always see the virtual trainer without additional turning. In addition to this, the user can see his real posture in the mirror, which is advantageous for correcting his movements. The ground projection indicates the dynamic motion of the footsteps in a visual way, and the user can quickly master the motion footsteps. After the user wears the motion capture equipment, the virtual avatar in the scene can be driven in real time, and the system feeds back the accuracy and the deviation degree of the user action in time. The system judges the learning progress of the user, takes the learning progress as a reference of the teaching progress, and assists correction through visual and auditory prompts. And the motion similarity matching algorithm is used for quantifying the learning effect and giving an evaluation result. The user can use the system to learn and practice for many times, and the aim of learning and exercising can be easily achieved.
Specifically, the system comprises a multi-projection splicing and fusion module, a motion capture and positioning module, a mode selection module, an interaction module, a feedback module and a communication module. The multi-projection splicing and fusion module splices projection pictures of a plurality of projectors on the wall and the ground and is in gentle transition fusion with the projector. And performing geometric correction, edge fusion and brightness adjustment on the projection picture by using multi-channel correction software to generate a splicing fusion parameter graph, and finally rendering the picture and displaying the picture from different projectors by decoding and calculating the parameter graph.
The motion capturing and positioning module acquires a color image and a depth image of a user by using an RGBD (red, green and blue) camera, obtains coordinate values of joints of bones of the user in a camera coordinate system through mapping with virtual bones, and converts the coordinate values into a ground projection screen coordinate system so as to determine the position of the user in a room; the user's gestures are captured using a motion capture device, a network connection is established using a network protocol, and joint position data is streamed into the program for computation.
The mode selection module supports a user to select different learning modes according to self conditions. The user with the completely zero base can select a learning mode, the number of prompts is large under the mode, if the ground has the prompt of footsteps, different actions can be stopped, the actions are more subdivided, and the learning progress is completely controlled by the user. After the user is gradually skilled, the user can select an exercise mode, and few prompts are provided in the exercise mode, if no step prompt is provided, the movement is not stopped;
the interaction module can control the progress of action learning, the angle of watching a virtual coach, the playing speed of standard actions and the like by selecting a proper interaction mode through a plurality of interaction modes such as voice interaction, posture interaction and the like.
And the feedback module feeds back the learning in a visual form. If the ground has a footprint outline indicating the position of the next action step, the user can trigger the animation to play after stepping on the footprint. The left foot and the right foot, the actual step and the virtual step, and the gravity center are also distinguished by different expressions. In order to enhance the visual feedback effect, the teacher model is rendered as a semi-transparent shell to be superposed on the virtual body of the user, and the joint point position with large position deviation can light up a red light to remind the user to correct.
The learning quality evaluation module scores the performance of the user through an action similarity matching algorithm. In the exercise mode, the score of the current action is given by the action ending, and the score can be detailed to each joint, so that the exercise can be continued more specifically.
The communication module can complete communication among a plurality of hosts connected with the projector, and comprises the steps of establishing network connection, sending messages, forwarding messages, analyzing messages and the like. The server has a message forwarding function and realizes direct communication with the client and message transmission between the clients.
The interactive display method of the action learning system based on the room type interactive projection comprises the following steps:
(1) and starting the server and waiting for connection application of each client. And opening the client side and applying for establishing connection with the server. Establishing a network connection and communicating messages.
(2) The user walks into the room and stands at the appointed starting position according to the ground prompt, and the guidance words of the system introduce the system to lead the user to quickly master the use method of the system;
(3) by utilizing the motion capturing and positioning module, the client can acquire the position and motion information of a user, and the position and motion information is used for logical judgment and providing data drive for a model in a virtual scene;
(4) a user selects a mode in a main menu according to the set gesture interactive action, and if the mode is a learning mode, the user jumps to the step (5), and if the mode is an exercise mode, the user jumps to the step (6);
(5) in the learning mode, the user follows the virtual coach to learn and decompose the movement. The user wears to move and catches the student model that equipment can drive in the virtual scene, can compare with virtual mr's action. Prompting the ground step method;
(6) in the practice mode, prompts to the user are reduced, so that the user can compare the user in the mirror with a coach in the virtual scene more intensely, and feedback and scoring of action deviation are given.
(7) And judging whether the system use is finished or not. If yes, ending; otherwise, jumping to the step (4);
in the step (1), the RGBD camera acquires skeleton information of the user, the skeleton information is sent to the appointed client through the server, and besides the skeleton information, messages sent and received by the client also include the progress of animation playing of the current coach model, the projection on which the current action is to be displayed, the current learning state of the user and the like.
In the step (3), the server sends the user skeleton information to the ground program client so as to judge whether the user gait is correct. The wall client acquires the node information of the whole body of the user through the motion capture equipment, so that whether the user motion is standard or not is judged, and similarity matching is carried out on the user motion and the template motion.
In the step (5), the specific method includes:
(5-1) before learning, the user is reminded of reaching the appointed initial position by voice, and the user sends out a voice command, plays the animation and starts learning.
And (5-2) judging whether to continue or not according to the learning state of the user at each step of action, and if the user does not do the action in place, not continuing to play the animation for learning, but strengthening the current action for learning. The learning state refers to the similarity degree of steps and joints of the body with the motion template, the position of the feet is obtained through an RGBD camera, and joint data is obtained through motion capture equipment.
(5-3) one action learning is finished, the user can give a voice instruction to choose to continue learning the next action, or repeat the move, or return to the main menu.
In the step (6), the specific method includes:
(6-1) after the start of the exercise is voice-prompted, the user can start the exercise at an arbitrary position within the learning area.
(6-2) the user drives the virtual model in real time, the system is matched with the actions in the template library through an action matching algorithm to obtain the most similar result, the action of the user in practice is known, and the virtual coach model plays corresponding teaching animation;
(6-3) one action is finished, and the user can send out voice command control, continue learning or return to the main menu.
In the step (6-2), the specific method for action matching is as follows:
the data of the Taijiquan animation recorded in advance is taken as a standard reference template R and is an M-dimensional vector,
R＝r1，r2，r3，……，rm
the bone data of the user collected in real time is used as a template T for testing, is an N-dimensional vector,
T＝t1，t2，t3，……，tn
m and N are animation frame numbers, the length of the action made by the user and the standard animation is not necessarily the same, M is not necessarily equal to N, but the dimension of each component is the same. Dimension refers to the number of joints that a motion capture device (e.g., nuotang) can capture, with 17 nodes detectable by nuotang (hip, right thigh, right calf, right foot, left thigh, left calf, left foot, right shoulder, right forearm, right hand, left shoulder, left forearm, left chest, head). In order to compare the similarity between the templates, the distance between them needs to be calculated, and the smaller the distance, the higher the similarity.
Firstly, an m × n distance matrix D is constructed, the matrix element Dij＝dist(ri，tj) Dist () is a distance computation function that computes the euclidean distance between sequence points. And generating a loss matrix M according to the distance matrix, wherein the calculation method comprises the following steps:
①M11＝D11
②Mij＝Min(Mi-1,j-1,Mi-1,j,Mi,j-1)+Mi,j
last row and last column M of the loss matrixmnIs the cumulative distance between the two sequences. Finding a shortest path from the lower left corner to the upper right corner in the distance matrix such that the sum of the element values on the path is minimized. I.e. using a dynamic programming method to find a value d11To dmnThe shortest path of (2) is better, the path is often near the diagonal, so the area for finding the path using the constraint condition and the global path window constraint can only be found upwards or rightwards or upwards and cannot be far away from the diagonal.
As a typical embodiment, as shown in fig. 1, the hardware architecture of the present embodiment mainly includes three hosts (clients), a notebook computer (server), 7 projectors, a somatosensory device Kinectv2, and a set of Noitom motion capture devices. All devices are in the same local area network. The three hosts are respectively responsible for ground projection, left wall projection and right wall projection. The laptop is connected to Kinectv2 and the Noitom inertial sensor motion capture device and acts as a server, sending, receiving and forwarding data. There is a total of one server and three clients.
As shown in fig. 2, the system flow chart of the present embodiment:
(1) starting a projector set, wearing motion capture equipment by a user, starting wireless data connection, and starting a Kinect;
(2) starting a server program;
(3) starting a client program;
(4) establishing network communication using a TCP protocol;
(5) a user selects a mode to be performed on a main interface through voice interaction and posture interaction;
(6) judging whether the user selects a learning mode, if so, skipping to the step (7); otherwise, jumping to the step (10);
(7) the user selects an action through voice interaction and posture interaction;
(8) the ground projection can generate footwork prompt of the current learning action, and the wall projection can generate animation of playing corresponding action by a virtual coach;
(9) after one action is learned, whether the learning is finished or not is judged. If yes, skipping to step (14), otherwise, skipping to step (7).
(10) The user exercises any action, and the system compares the exercise action of the user with the action template library through a matching degree algorithm.
(11) The virtual coach projected on the wall plays the animation of the matched action.
(12) Given the score, the score for each joint can be viewed.
(13) And judging whether to finish the exercise. If yes, skipping to step (14), otherwise, skipping to step (7).
(14) And judging whether the user wants to log out of the system. If yes, ending; otherwise, jumping to the step (5).
As shown in fig. 3, the deformation mapping parameter map and the luminance-color fusion parameter map can be obtained by using multi-channel correction software. Because the images of the wall surface and the ground are spliced by a plurality of projectors, the rendered pictures need to be pre-deformed by using the parameter graphs and then fused and spliced. The size of the parameter map is the resolution of the screen, since the resolution of the projector is 1024 × 768, i.e. the wall is 2048 × 768 and the floor is 2048 × 1536. The method comprises the steps of writing a shader in a unity engine, reading a parameter map, decoding the parameter map to obtain a mapping relation, multiplying the mapping relation by a pixel value of an original image to obtain a pixel value of a rendered image, and achieving predeformation.
FIG. 4(a) shows a user performing a calibration of a motion capture device. Firstly, the hands droop, naturally stand, then the T-position with the hands lifted laterally, and finally the S-position with the hands lifted horizontally and half squat.
Fig. 4(b) is a default robot model in the motion capture software, which is driven by the user in real time. The user is now performing a calibration of the T-pos.
As shown in fig. 5, a user wears the motion capture device and places the device in a room with two mirrors and three projections to perform taijiquan learning. The floor has red and blue footprints, representing the left and right feet, respectively. Two models are displayed in two wall projections. Wherein, wearing blue training clothes is a virtual model driven by a user, wearing white training clothes is a virtual coach. The user learns by looking at the projection in the mirror and on the wall. In FIG. 5, the user is exercising the final step of the twenty-four simplified second move of the Taijiquan, horsehair. The user's viewpoint now falls on the wall projection.
As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.
The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure, and various modifications and changes may be made to the present disclosure by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present disclosure should be included in the protection scope of the present disclosure.
Although the present disclosure has been described with reference to specific embodiments, it should be understood that the scope of the present disclosure is not limited thereto, and those skilled in the art will appreciate that various modifications and changes can be made without departing from the spirit and scope of the present disclosure.
Claims (11)
1. A motion learning system based on room-type interactive projection is characterized in that: the method comprises the following steps:
the multi-projection splicing and fusion module comprises a plurality of clients, wherein each client is used for being responsible for projection and display control of a part of areas in a room and is configured to splice projection pictures of a plurality of projectors on the wall and the ground of the room so as to smoothly transit a fusion zone; performing geometric correction, edge fusion and brightness adjustment on the projection picture to generate a splicing fusion parameter graph and form a rendering picture;
the motion capturing and positioning module is configured to acquire a color image and a depth image of a user, obtain coordinate values of joints of bones of the user in a camera coordinate system through mapping with virtual bones, and then convert the coordinate values into a ground projection screen coordinate system so as to determine the position of the user in a room; capturing a gesture of a user using a motion capture device;
the interaction module is configured to receive interaction mode selection information of a user and play a rendering picture and a learning task according to the mode;
a feedback module configured to make feedback on the learning task in a visual form;
and the communication module is configured to complete communication among a plurality of hosts connected with the projector, and realize direct communication with the client and message transmission among the clients.
2. The system of claim 1, wherein the interactive projection-based motion learning system comprises: the clients at least comprise wall clients and ground clients;
or further, the wall client comprises a plurality of hosts and a plurality of projectors, and is used for projecting the virtual scene onto different walls, acquiring the user posture and judging the accuracy of the user action; the ground client comprises at least one host and a plurality of projectors, and is used for projecting the virtual scene onto different wall surfaces, acquiring the foot position of the user and judging the accuracy of the foot action of the user.
3. The system of claim 1, wherein the interactive projection-based motion learning system comprises: the motion capturing and positioning module acquires a color image and a depth image of a user by using an RGBD (red, green and blue) camera, obtains coordinate values of joints of bones of the user in a camera coordinate system through mapping with virtual bones, and converts the coordinate values into a ground projection screen coordinate system so as to determine the position of the user in a room; the gestures of the user are captured using a motion capture device, a network connection is established using a network protocol, and calculated based on a joint position data stream.
4. The system of claim 1, wherein the interactive projection-based motion learning system comprises: the interaction module comprises a mode selection module, a learning module and a training module, wherein the mode selection module is configured to support a user to select different learning modes according to self conditions, and specifically comprises a learning mode and a training mode;
or, the interaction module comprises a human-computer interaction module configured to support voice interaction and posture interaction modes, and the user controls the progress of action learning, the angle of watching a virtual coach and the playing speed of standard actions by selecting a proper interaction mode;
or the system further comprises a learning quality evaluation module which is configured to evaluate the motion state of each joint of the user through a motion similarity matching algorithm.
5. A motion learning method based on room type interactive projection is characterized by comprising the following steps: the method comprises the following steps:
receiving a selection mode of a user, capturing position and motion information of the user, and providing data drive for a model in a virtual scene for logic judgment;
providing a virtual coach picture, providing decomposition actions to be learned, driving a student model in a virtual scene through motion capture, comparing the student model with the actions of a virtual teacher, and providing an evaluation report.
6. The method as claimed in claim 5, wherein the method comprises: acquiring skeleton information of a user by using an RGBD (red green blue) camera, wherein the skeleton information is specifically coordinate values (x, y, z) of foot joint points under a camera coordinate system at the current moment;
packing names and coordinates of joints into data packets, wearing an inertial sensor-based motion capture device by a user, and transmitting motion data of the joint points of the whole body by the device in a BVH data stream mode;
receiving a data stream, driving a model in a scene by using a unity plug-in, and performing playing projection;
or, according to the skeleton information of the user, judging whether the user gait is correct or not;
and acquiring the node information of the whole body of the user through the motion capture equipment, judging whether the motion of the user is standard or not, and performing similarity matching with the motion of the template.
7. The method as claimed in claim 5, wherein the method comprises: the specific process of learning comprises the following steps:
(1) before the action starts to learn, the user is prompted to stop at a designated initial position by voice, the user sends a voice command, animation is played, and learning is started;
(2) judging whether to continue or not according to the learning state of the user in each step, if the user does not do the animation learning, not continuing to play the animation learning, but strengthening the current action of the learning; the learning state refers to the similarity degree between footsteps and joints of the body and the action template, the positions of the feet of the user are continuously obtained through the bones of the user, and the action capturing equipment continuously obtains the joint movement data and broadcasts the joint movement data to all clients on the wall surface;
(3) at the end of an action, the user issues a voice command and chooses to continue learning the next action, either repeating the move or returning to the main menu.
8. The method as claimed in claim 5, wherein the method comprises: the specific process of the exercise mode comprises the following steps:
(a) after the voice prompt exercise is started, the user starts to exercise at any position in the learning area;
(b) the user drives the virtual model in real time, the system is matched with the actions in the template library through an action matching algorithm to obtain the most similar result, the action of the user in practice is known, and the virtual coach model plays corresponding teaching animation;
(c) at the end of an action, the user may issue voice command controls to continue learning or return to the main menu.
9. The method as claimed in claim 8, wherein the method comprises: in the step (b), the specific method of action matching is as follows:
taking the data of the animation recorded in advance as a standard reference template R, and taking the data as an M-dimensional vector;
taking the bone data of the user collected in real time as a template T for testing, wherein the template T is an N-dimensional vector;
m and N are animation frame numbers, the length of the action made by the user is not necessarily the same as that of a standard animation, M is not necessarily equal to N, but the dimension of each component is the same, and the dimension is the number of joints;
constructing an MxN distance matrix D, matrix elements Dij＝dist(ri，tj)，ri，tjRespectively is a sequence point of a standard action template and a sequence point of a user action template, dij is a certain element in the matrix, i is between 0 and M, and j is between 0 and N; dist () is a distance computation function that computes the euclidean distance between sequence points; and generating a loss matrix M according to the distance matrix, wherein the calculation method comprises the following steps:
M11＝D11
Mij＝Min(Mi-1,j-1,Mi-1,j,Mi,j-1)+Mi,j
last row and last column M of the loss matrixmnIs the cumulative distance between the two sequences; finding a shortest path from the lower left corner to the upper right corner in the distance matrix such that the sum of the element values on the pathMinimum, i.e. one derived from d by means of dynamic programming11To dmnThe shortest path of (2).
10. A computer-readable storage medium characterized by: stored with instructions adapted to be loaded by a processor of a terminal device and to perform the steps of a room-based interactive projection motion learning method according to any of claims 5-9.
11. A terminal device is characterized in that: the system comprises a processor and a computer readable storage medium, wherein the processor is used for realizing instructions; a computer readable storage medium for storing a plurality of instructions adapted to be loaded by a processor and for performing the steps of a method for interactive projection-based motion learning according to any of the claims 5-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911244297.5A CN111047925B (en) | 2019-12-06 | 2019-12-06 | Action learning system and method based on room type interactive projection |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911244297.5A CN111047925B (en) | 2019-12-06 | 2019-12-06 | Action learning system and method based on room type interactive projection |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111047925A true CN111047925A (en) | 2020-04-21 |
| CN111047925B CN111047925B (en) | 2021-06-25 |
Family
ID=70234834
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911244297.5A Active CN111047925B (en) | 2019-12-06 | 2019-12-06 | Action learning system and method based on room type interactive projection |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111047925B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113190104A (en) * | 2021-01-18 | 2021-07-30 | 郭奕忠 | Method for realizing man-machine interaction by recognizing human actions through visual analysis by intelligent equipment |
| WO2022127634A1 (en) * | 2020-12-16 | 2022-06-23 | 高勇 | Interactive projection input/output device |
| CN115002428A (en) * | 2022-04-28 | 2022-09-02 | 北京连屏科技有限公司 | Immersive space system |
Citations (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5930741A (en) * | 1995-02-28 | 1999-07-27 | Virtual Technologies, Inc. | Accurate, rapid, reliable position sensing using multiple sensing technologies |
| CN102662472A (en) * | 2012-04-10 | 2012-09-12 | 苏州中科启慧软件技术有限公司 | Body movement based learning method and cloud service system thereof |
| CN104575129A (en) * | 2013-10-27 | 2015-04-29 | 西安天动数字科技有限公司 | Dance teaching system |
| CN104808798A (en) * | 2015-05-14 | 2015-07-29 | 哈尔滨工业大学 | Kinect-based multi-screen interactive folk dance entertainment system |
| CN204965778U (en) * | 2015-09-18 | 2016-01-13 | 华中师范大学 | Infant teaching system based on virtual reality and vision positioning |
| CN105791698A (en) * | 2016-04-22 | 2016-07-20 | 周口职业技术学院 | System and method of determining dance training |
| WO2016168117A2 (en) * | 2015-04-14 | 2016-10-20 | John James Daniels | Wearable electric, multi-sensory, human/machine, human/human interfaces |
| CN106097787A (en) * | 2016-08-18 | 2016-11-09 | 四川以太原力科技有限公司 | Limbs teaching method based on virtual reality and teaching system |
| CN106485055A (en) * | 2016-09-22 | 2017-03-08 | 吉林大学 | A kind of old type ii diabetes patient moving training system based on Kinect sensor |
| US20170291086A1 (en) * | 2016-04-11 | 2017-10-12 | Charles Chungyohl Lee | Motivational kinesthetic virtual training program for martial arts and fitness |
| CN107341612A (en) * | 2017-07-07 | 2017-11-10 | 上海理工大学 | A kind of action evaluation method of the rehabilitation training based on taijiquan |
| KR20180057177A (en) * | 2016-11-22 | 2018-05-30 | 순천향대학교 산학협력단 | Multi sides booth system for virtual reality and the thereof |
| CN108376487A (en) * | 2018-02-09 | 2018-08-07 | 冯侃 | Based on the limbs training system and method in virtual reality |
| CN108777081A (en) * | 2018-05-31 | 2018-11-09 | 华中师范大学 | A kind of virtual Dancing Teaching method and system |
| CN109785686A (en) * | 2019-01-22 | 2019-05-21 | 哈尔滨拓博科技有限公司 | A kind of multimedia system for dancing classroom |
| CN109817031A (en) * | 2019-01-15 | 2019-05-28 | 张赛 | A kind of limb motion teaching method based on VR technology |
| CN109859324A (en) * | 2018-12-29 | 2019-06-07 | 北京光年无限科技有限公司 | A kind of motion teaching method and device based on visual human |
| CN110297584A (en) * | 2018-03-21 | 2019-10-01 | 佟志强 | The immersion interaction classroom CAVE towards children education |
-
2019
- 2019-12-06 CN CN201911244297.5A patent/CN111047925B/en active Active
Patent Citations (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5930741A (en) * | 1995-02-28 | 1999-07-27 | Virtual Technologies, Inc. | Accurate, rapid, reliable position sensing using multiple sensing technologies |
| CN102662472A (en) * | 2012-04-10 | 2012-09-12 | 苏州中科启慧软件技术有限公司 | Body movement based learning method and cloud service system thereof |
| CN104575129A (en) * | 2013-10-27 | 2015-04-29 | 西安天动数字科技有限公司 | Dance teaching system |
| WO2016168117A2 (en) * | 2015-04-14 | 2016-10-20 | John James Daniels | Wearable electric, multi-sensory, human/machine, human/human interfaces |
| CN104808798A (en) * | 2015-05-14 | 2015-07-29 | 哈尔滨工业大学 | Kinect-based multi-screen interactive folk dance entertainment system |
| CN204965778U (en) * | 2015-09-18 | 2016-01-13 | 华中师范大学 | Infant teaching system based on virtual reality and vision positioning |
| US20170291086A1 (en) * | 2016-04-11 | 2017-10-12 | Charles Chungyohl Lee | Motivational kinesthetic virtual training program for martial arts and fitness |
| CN105791698A (en) * | 2016-04-22 | 2016-07-20 | 周口职业技术学院 | System and method of determining dance training |
| CN106097787A (en) * | 2016-08-18 | 2016-11-09 | 四川以太原力科技有限公司 | Limbs teaching method based on virtual reality and teaching system |
| CN106485055A (en) * | 2016-09-22 | 2017-03-08 | 吉林大学 | A kind of old type ii diabetes patient moving training system based on Kinect sensor |
| KR20180057177A (en) * | 2016-11-22 | 2018-05-30 | 순천향대학교 산학협력단 | Multi sides booth system for virtual reality and the thereof |
| CN107341612A (en) * | 2017-07-07 | 2017-11-10 | 上海理工大学 | A kind of action evaluation method of the rehabilitation training based on taijiquan |
| CN108376487A (en) * | 2018-02-09 | 2018-08-07 | 冯侃 | Based on the limbs training system and method in virtual reality |
| CN110297584A (en) * | 2018-03-21 | 2019-10-01 | 佟志强 | The immersion interaction classroom CAVE towards children education |
| CN108777081A (en) * | 2018-05-31 | 2018-11-09 | 华中师范大学 | A kind of virtual Dancing Teaching method and system |
| CN109859324A (en) * | 2018-12-29 | 2019-06-07 | 北京光年无限科技有限公司 | A kind of motion teaching method and device based on visual human |
| CN109817031A (en) * | 2019-01-15 | 2019-05-28 | 张赛 | A kind of limb motion teaching method based on VR technology |
| CN109785686A (en) * | 2019-01-22 | 2019-05-21 | 哈尔滨拓博科技有限公司 | A kind of multimedia system for dancing classroom |
Non-Patent Citations (1)
| Title |
|---|
| 任文: "基于Kinect运动捕捉技术的辅助训练系统的研究", 《电子设计工程》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022127634A1 (en) * | 2020-12-16 | 2022-06-23 | 高勇 | Interactive projection input/output device |
| TWI796039B (en) * | 2020-12-16 | 2023-03-11 | 高勇 | Interactive projection input and output devices |
| CN113190104A (en) * | 2021-01-18 | 2021-07-30 | 郭奕忠 | Method for realizing man-machine interaction by recognizing human actions through visual analysis by intelligent equipment |
| CN115002428A (en) * | 2022-04-28 | 2022-09-02 | 北京连屏科技有限公司 | Immersive space system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111047925B (en) | 2021-06-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11948260B1 (en) | Streaming mixed-reality environments between multiple devices | |
| US11132533B2 (en) | Systems and methods for creating target motion, capturing motion, analyzing motion, and improving motion | |
| TWI732194B (en) | Method and system for eye tracking with prediction and late update to gpu for fast foveated rendering in an hmd environment and non-transitory computer-readable medium | |
| CN111047925B (en) | Action learning system and method based on room type interactive projection | |
| CN110650354B (en) | Live broadcast method, system, equipment and storage medium for virtual cartoon character | |
| US6624853B1 (en) | Method and system for creating video programs with interaction of an actor with objects of a virtual space and the objects to one another | |
| US20160267699A1 (en) | Avatar control system | |
| US20180165862A1 (en) | Method for communication via virtual space, program for executing the method on a computer, and information processing device for executing the program | |
| CN102129343B (en) | Directed performance in motion capture system | |
| US20180374383A1 (en) | Coaching feedback system and method | |
| US20180373328A1 (en) | Program executed by a computer operable to communicate with head mount display, information processing apparatus for executing the program, and method executed by the computer operable to communicate with the head mount display | |
| US20100302253A1 (en) | Real time retargeting of skeletal data to game avatar | |
| KR20130098770A (en) | Expanded 3d space based virtual sports simulation system | |
| US20180299948A1 (en) | Method for communicating via virtual space and system for executing the method | |
| Song et al. | An immersive VR system for sports education | |
| JP2001236505A (en) | Method, device and system for estimating coordinate | |
| Magdin | Simple mocap system for home usage | |
| Chague et al. | Digital cloning for an increased feeling of presence in collaborative virtual reality environments | |
| Li | Development of immersive and interactive virtual reality environment for two-player table tennis | |
| KR102262894B1 (en) | System and method for ballet lesson using projection mapping | |
| Beever | Exploring Mixed Reality Level Design Workflows | |
| Törmänen | Comparison of entry level motion capture suits aimed at indie game production | |
| WO2023026529A1 (en) | Information processing device, information processing method, and program | |
| KR100684401B1 (en) | Apparatus for educating golf based on virtual reality, method and recording medium thereof | |
| CN117438038A (en) | Action intervention autism disorder children training method and system based on mixed reality |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |