CN112799507A - Human body virtual model display method and device, electronic equipment and storage medium - Google Patents
Human body virtual model display method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112799507A CN112799507A CN202110055323.0A CN202110055323A CN112799507A CN 112799507 A CN112799507 A CN 112799507A CN 202110055323 A CN202110055323 A CN 202110055323A CN 112799507 A CN112799507 A CN 112799507A
- Authority
- CN
- China
- Prior art keywords
- image
- human body
- virtual model
- key point
- gesture recognition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
Abstract
The invention provides a human body virtual model display method, a human body virtual model display device, electronic equipment and a storage medium, wherein the obtained user image is subjected to gesture recognition processing to obtain a gesture recognition result; determining image processing operation corresponding to the gesture recognition result, and executing the image processing operation on a pre-constructed human body virtual model to obtain a displayed target human body virtual model; and based on a 3D image display technology, performing image generation and image rendering processing on the target human body virtual model to obtain and display a 3D image of the target human body virtual model. The scheme of the invention can identify the user gesture to operate the human body virtual model, and carry out naked eye 3D display based on the 3D image display technology, thereby realizing flexible and efficient operation of the human body virtual model according to the user gesture and improving the display effect of the human body virtual model.
Description
Technical Field
The invention relates to the technical field of 3D display, in particular to a human body virtual model display method and device, electronic equipment and a storage medium.
Background
With the development of 3D display technology, the display device is widely applied to the fields of movie and television, medicine, military and the like, and particularly in the medical field, a human body model is often displayed in the teaching process.
In the prior art, a 3D human body model is constructed through three-dimensional model software, the constructed model is displayed on a display device, and a user operates the 3D human body model through a control device.
However, in the prior art, the human body virtual model is controlled by hardware equipment, so that a user cannot flexibly and efficiently operate the human body virtual model, and the model display effect is poor.
Disclosure of Invention
The invention provides a method and a device for displaying a human body virtual model, electronic equipment and a storage medium, which realize the operation of the human body virtual model by recognizing user gestures and the naked eye 3D display on display equipment, and improve the display effect of the human body virtual model.
In a first aspect, the present invention provides a method for displaying a virtual human body model, including:
performing gesture recognition processing on the obtained user image to obtain a gesture recognition result;
determining image processing operation corresponding to the gesture recognition result, and executing the image processing operation on a pre-constructed human body virtual model to obtain a displayed target human body virtual model;
and based on a 3D image display technology, performing image generation and image rendering processing on the target human body virtual model to obtain and display a 3D image of the target human body virtual model.
Optionally, the performing gesture recognition processing on the acquired user image to obtain a gesture recognition result includes:
determining an image area of a hand in the user image;
performing hand key point identification on the image area to obtain a hand key point sequence in the image area; the hand key point sequence comprises the key point type, the position and the form of the hand in each image frame in the image area;
and determining the corresponding gesture according to the hand key point sequence to obtain a gesture recognition result.
Further, before determining the corresponding gesture according to the hand key point sequence and obtaining a gesture recognition result, the method further includes:
in the hand key point sequence, determining whether the key point type, position and form of any image frame meet the triggering condition;
and if so, executing the step of determining the corresponding gesture according to the hand key point sequence to obtain a gesture recognition result.
Further, the determining a gesture corresponding to the hand key point sequence according to the hand key point sequence to obtain a gesture recognition result includes:
determining the change trend of each hand key point of the hand in the continuous image frames according to the key point type, the position and the form of the hand of the continuous image frames in the hand key point sequence;
and determining gesture types and gesture tracks corresponding to the hand key point sequences according to the variation trend of each hand key point to obtain gesture recognition results.
Further, the determining an image processing operation corresponding to the gesture recognition result, and executing the image processing operation on a pre-constructed virtual human body model to obtain a target virtual human body model to be displayed includes:
calling a pre-stored image processing function of a corresponding type from a function library according to the gesture type;
processing the pre-stored image processing function according to the gesture track to obtain an image processing function to be executed;
and executing the image processing function to be executed on the pre-constructed human body virtual model to obtain a target human body virtual model to be displayed.
Optionally, the performing image generation and image rendering processing on the target virtual human body model based on the 3D image display technology to obtain and display a 3D image of the target virtual human body model includes:
constructing a plurality of virtual cameras, and calculating a sub-pixel mapping matrix of each virtual camera;
determining a camera image of each virtual camera presenting the target human body virtual model by utilizing each sub-pixel mapping matrix;
performing light and shadow rendering processing on each camera image according to the model material of the target human body virtual model to obtain a corresponding rendered image;
and creating and displaying a 3D image of the target human body virtual model according to each rendering image.
In a second aspect, the present invention provides a virtual human model display apparatus, including:
the gesture recognition unit is used for performing gesture recognition processing on the acquired user image to obtain a gesture recognition result;
the execution unit is used for determining image processing operation corresponding to the gesture recognition result, executing the image processing operation on a pre-constructed human body virtual model and obtaining a displayed target human body virtual model;
and the display unit is used for carrying out image generation and image rendering processing on the target human body virtual model based on a 3D image display technology to obtain and display a 3D image of the target human body virtual model.
Optionally, the gesture recognition unit is specifically configured to:
determining an image area of a hand in the user image;
performing hand key point identification on the image area to obtain a hand key point sequence in the image area; the hand key point sequence comprises the key point type, the position and the form of the hand in each image frame in the image area;
and determining the corresponding gesture according to the hand key point sequence to obtain a gesture recognition result.
Further, the gesture recognition unit is specifically further configured to:
in the hand key point sequence, determining whether the key point type, position and form of any image frame meet the triggering condition;
and if so, executing the step of determining the corresponding gesture according to the hand key point sequence to obtain a gesture recognition result.
Further, the gesture recognition unit is specifically further configured to:
determining the change trend of each hand key point of the hand in the continuous image frames according to the key point type, the position and the form of the hand of the continuous image frames in the hand key point sequence;
and determining gesture types and gesture tracks corresponding to the hand key point sequences according to the variation trend of each hand key point to obtain gesture recognition results.
Optionally, the execution unit is specifically configured to:
calling a pre-stored image processing function of a corresponding type from a function library according to the gesture type;
processing the pre-stored image processing function according to the gesture track to obtain an image processing function to be executed;
and executing the image processing function to be executed on the pre-constructed human body virtual model to obtain a target human body virtual model to be displayed.
Optionally, the display unit is specifically configured to:
constructing a plurality of virtual cameras, and calculating a sub-pixel mapping matrix of each virtual camera;
determining a camera image of each virtual camera presenting the target human body virtual model by utilizing each sub-pixel mapping matrix;
performing light and shadow rendering processing on each camera image according to the model material of the target human body virtual model to obtain a corresponding rendered image;
and creating and displaying a 3D image of the target human body virtual model according to each rendering image.
In a third aspect, the present invention provides an electronic device comprising: at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executes computer-executable instructions stored by the memory, causing the at least one processor to perform the presentation method of any one of the first aspects.
In a fourth aspect, the present invention provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, the presentation method according to the first aspect and various possible designs of the first aspect are implemented.
In a fifth aspect, the invention provides a computer program product comprising a computer program that, when executed by a processor, implements the presentation method of any one of the first aspects.
According to the human body virtual model display method and device, the electronic equipment and the storage medium, gesture recognition processing is carried out on the obtained user image to obtain a gesture recognition result; determining image processing operation corresponding to the gesture recognition result, and executing the image processing operation on a pre-constructed human body virtual model to obtain a displayed target human body virtual model; and based on a 3D image display technology, performing image generation and image rendering processing on the target human body virtual model to obtain and display a 3D image of the target human body virtual model. Compared with the prior art, the method for displaying the human body virtual model can recognize the gesture of the user to operate the human body virtual model, and can perform naked eye 3D display based on the 3D image display technology, so that the human body virtual model can be flexibly and efficiently operated, and the display effect of the human body virtual model is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a schematic diagram of a network architecture upon which the present disclosure is based;
fig. 2 is a schematic flow chart of a method for displaying a virtual human body model according to an embodiment of the present disclosure;
fig. 3 is a schematic diagram of a gesture recognition process provided in the embodiment of the present disclosure;
FIG. 4 is a schematic structural diagram of a virtual human model display apparatus according to an embodiment of the present disclosure;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Currently, 3D display technology has become a research hotspot and is widely applied to a plurality of fields, such as movie and television, medicine, military and the like. In the medical field, teaching demonstration by adopting a human body virtual model is a development trend of medical teaching.
In the prior art, a human body virtual model is usually constructed by using three-dimensional model construction software, the constructed model is rendered and displayed on a display device, a user operates the displayed model through a hardware control device, and the display device synchronously displays the operated human body virtual model according to the user operation.
For example, a liver virtual model is built by three-dimensional model building software, the liver virtual model is rendered and displayed on a display device, a user rotates the liver virtual model displayed on the display device by using a mouse to display other positions of the liver, and the display device synchronously displays the other positions of the liver according to the operation of the user on the mouse.
However, because the user needs to manually operate the hardware control device to display the human body virtual model, the user needs to spend a lot of time on operating the hardware control device, and even the user can only teach beside the hardware control device, the user cannot flexibly and efficiently operate the human body virtual model, and the effect of model display is seriously influenced.
For the problems, the inventor researches and discovers that hardware control equipment can be omitted, a shooting device is used for obtaining a gesture image of a user, the gesture recognition result of the user is determined according to the gesture image, the gesture recognition result is converted into operation on a human body virtual model, the operated human body virtual model is rendered based on a 3D image display technology, a 3D image of the human body virtual model with stronger reality sense is obtained, and the 3D image is displayed on a 3D screen. Therefore, the human body virtual model can be operated by recognizing the user gesture, naked eye 3D display is carried out based on the 3D image display technology, the human body virtual model can be operated flexibly and efficiently, and the display effect of the human body virtual model is improved.
Fig. 1 is a schematic diagram of a network architecture on which the present disclosure is based, and as shown in fig. 1, the network architecture on which the present disclosure is based may include a virtual human body model display apparatus 1, a shooting device 2 and a display device 3.
The human virtual model display apparatus 1 and the shooting device 2 may be implemented by hardware or software for data transmission through a network, and may be used to execute the display method described in the following embodiments.
When the virtual human model display device 1 is a hardware, it may be a cloud server with an operation function. When the human virtual model display apparatus 1 is software, it can be installed in electronic devices with computing function, wherein the electronic devices include, but are not limited to, laptop portable computers, desktop computers, and the like.
The shooting device 2 may be a binocular camera, a depth camera, an intelligent camera, or other hardware devices.
Meanwhile, the virtual human model display apparatus 1 may utilize a photographing function of the photographing device 2 to acquire a user image photographed by the photographing device 2.
For example, the depth camera shoots a user image, the virtual human body model display device obtains the user image and performs gesture recognition on the user image to determine a gesture recognition result, corresponding virtual human body model operation is determined according to the gesture recognition result, the virtual human body model operation is executed on the virtual human body model, the virtual human body model after operation is subjected to three-dimensional processing and rendering processing by using a 3D display technology to obtain a 3D image, and the 3D image is displayed on display equipment for a user to watch.
The human body virtual model display method, the human body virtual model display device, the electronic device and the storage medium provided by the application are further described as follows:
fig. 2 is a schematic flow chart of a method for displaying a virtual human body model according to an embodiment of the present disclosure. As shown in fig. 2, a method for displaying a virtual human body model provided in an embodiment of the present disclosure includes:
s21, performing gesture recognition processing on the acquired user image to obtain a gesture recognition result;
in this embodiment, since the model is not operated by using the hardware control device, but the virtual human body model is operated by recognizing the gesture of the user, the gesture recognition processing is performed on the acquired user image, and the gesture recognition result of the user is determined to operate the virtual human body model.
In a possible implementation manner, the performing gesture recognition processing on the acquired user image to obtain a gesture recognition result includes: determining an image area of a hand in the user image; performing hand key point identification on the image area to obtain a hand key point sequence in the image area; the hand key point sequence comprises the key point type, the position and the form of the hand in each image frame in the image area; and determining the corresponding gesture according to the hand key point sequence to obtain a gesture recognition result.
That is to say, the hand position in each frame of user image is determined, and hand key point information is acquired, wherein the hand key point information comprises key point types, positions and forms, so as to obtain a hand key point sequence. And processing the key point information in the hand key point sequence to obtain a gesture recognition result.
Because it is determined that the gesture recognition result needs to first determine whether the gesture triggers a preset condition, and then determine a specific gesture recognition result based on the preset condition, the determining the gesture corresponding to the gesture according to the hand key point sequence further includes, before obtaining the gesture recognition result: in the hand key point sequence, determining whether the key point type, position and form of any image frame meet the triggering condition; and if so, executing the step of determining the corresponding gesture according to the hand key point sequence to obtain a gesture recognition result.
After the fact that the user gesture meets the triggering conditions is determined, the corresponding method is adopted to process the hand key point sequence to determine a gesture recognition result, and then the gesture recognition result is determined according to the change trend of the hand key points. Determining the corresponding gesture according to the hand key point sequence to obtain a gesture recognition result, wherein the gesture recognition result comprises: determining the change trend of each hand key point of the hand in the continuous image frames according to the key point type, the position and the form of the hand of the continuous image frames in the hand key point sequence; and determining gesture types and gesture tracks corresponding to the hand key point sequences according to the variation trend of each hand key point to obtain gesture recognition results.
Fig. 3 is a schematic diagram of a gesture recognition process according to an embodiment of the present disclosure, as shown in fig. 3, after a virtual human model starts to be displayed, first obtaining hand key point information, and determining whether a trigger condition is satisfied according to the hand key point information; if the condition is not met, acquiring the hand key point information again; if the condition is met, determining the variation trend of each hand key point according to the hand key point information; and finally, obtaining a gesture recognition result according to the variation trend of each hand key point.
Gesture recognition is further illustrated below by a few specific gesture recognition examples:
in an example I, the user right hand controls the movement of the human body virtual model to move the heart in the human body virtual model to the middle position of the display device, the human body virtual model display device obtains a user image shot in the process, a right hand key point sequence is obtained from the user image, the gesture of the user is determined to meet the triggering condition of one-hand movement operation according to the right hand key point sequence, and the user right hand is further determined to move 30cm along the positive direction of the X axis according to the right hand key point sequence.
And in the second example, the right hand of the user makes a door pushing action on the heart in the human body virtual model along the positive direction of the Z axis to select the heart, the human body virtual model display device acquires the user image shot in the process, acquires a right-hand key point sequence from the user image, and determines that the right hand of the user moves by 15cm along the positive direction of the Z axis according to the right-hand key point sequence.
And in the third example, the right hand of the user controls the selected heart to rotate by a certain angle around the Y axis so as to display the other side of the heart, the human body virtual model display device acquires the user image shot in the process, acquires a right-hand key point sequence from the user image, determines that the gesture of the user meets the triggering condition of one-hand rotation operation according to the right-hand key point sequence, and further determines that the right hand of the user rotates by 95 degrees around the Y axis according to the right-hand key point sequence.
And example four, the two hands of the user magnify the selected heart to display heart details, the human body virtual model display device acquires the user image shot in the process, acquires a two-hand key point sequence from the user image, determines that the user gesture meets a two-hand magnification operation triggering condition according to the two-hand key point sequence, and further determines that the right hand and the left hand of the user are relatively far away from each other by 20cm in the Y-axis direction according to the two-hand key point sequence.
S22, determining image processing operation corresponding to the gesture recognition result, and executing the image processing operation on a pre-constructed human body virtual model to obtain a target human body virtual model to be displayed;
in this embodiment, since the gesture recognition result is used to operate the virtual human body model, the gesture recognition result needs to be associated with the operation of the virtual human body model to implement the operation of the virtual human body model by the gesture, and the virtual human body model is essentially an image, so that the image processing operation corresponding to the gesture recognition result is determined, and the image processing operation is performed on the pre-constructed virtual human body model to obtain the target virtual human body model to be displayed.
In a possible implementation manner, the determining an image processing operation corresponding to the gesture recognition result, and performing the image processing operation on a pre-constructed virtual human body model to obtain a target virtual human body model to be displayed includes: calling a pre-stored image processing function of a corresponding type from a function library according to the gesture type; processing the pre-stored image processing function according to the gesture track to obtain an image processing function to be executed; and executing the image processing function to be executed on the pre-constructed human body virtual model to obtain a target human body virtual model to be displayed.
Specifically, different gestures correspond to different model operations, and a real space in which a hand is located and a virtual space in which a virtual model of a human body is located need to be converted to realize control of the model by the gestures, so in this embodiment, a pre-stored image processing function of a corresponding type is called from a function library according to the gesture type, a gesture track is input to the pre-stored image processing function to obtain an image processing function to be executed, the pre-constructed virtual model of the human body is controlled to execute the image processing function to be executed, and the virtual model of the human body is controlled to perform model operations corresponding to the gestures.
Continuing to take the first example as an example, since it is determined that the gesture of the user satisfies the one-handed movement operation triggering condition and the right hand of the user moves by 30cm in the positive direction of the X axis, the moving image processing function is called, the right hand of the user moves by 30cm in the positive direction of the X axis and is input into the moving image processing function to obtain the image processing function to be executed, the image processing function to be executed is executed on the virtual human model, and the virtual human model moves horizontally by 2m to the right side.
S23, based on the 3D image display technology, image generation and image rendering processing are carried out on the target human body virtual model, and a 3D image of the target human body virtual model is obtained and displayed.
In this embodiment, because the 3D image effect of the pre-constructed virtual human body model displayed on the display device is poor, and especially the display effect of the operated target virtual human body model needs to be improved, the image generation and the image rendering processing are performed on the target virtual human body model based on the 3D image display technology.
In a possible implementation manner, the performing image generation and image rendering processing on the target virtual human body model based on a 3D image display technology to obtain and display a 3D image of the target virtual human body model includes: constructing a plurality of virtual cameras, and calculating a sub-pixel mapping matrix of each virtual camera; determining a camera image of each virtual camera presenting the target human body virtual model by utilizing each sub-pixel mapping matrix; performing light and shadow rendering processing on each camera image according to the model material of the target human body virtual model to obtain a corresponding rendered image; and creating and displaying a 3D image of the target human body virtual model according to each rendering image.
Specifically, since the virtual human body model needs to display a 3D effect on the display device and guarantee the reality of the display effect, a 3D image corresponding to the virtual human body model needs to be constructed, a plurality of virtual cameras are firstly constructed and a sub-pixel mapping matrix of each virtual camera is calculated, and a plurality of camera images are obtained by performing pixel mapping on the existing model pixels by using the sub-pixel mapping matrix.
And then, according to the physical characteristics of the model material, performing light and shadow rendering processing on the camera images by using a rendering equation, and calculating the pixel brightness in the camera images by using the rendering equation shown in the following formula I.
Lo＝Le+∮fr·Li·(wi·n)dwiFormula one
Wherein L isoRepresenting the rendered pixel luminance, LeRepresenting the luminance of the pixel, frIndicating the ratio of the reflection of the light from the incident direction to the exit direction of the pixel, wiN represents the attenuation of the incident ray by the angle of incidence.
And performing simulation processing on the model surface in the processed camera image by using a sub-surface scattering technology, and calculating the radiation exitance in each camera image by using a rendering equation shown in the following formula II.
Wherein M ise(x, y) represents the radiant exitance at (x, y), E (x, y) represents the irradiance at (x, y), Rd(x, y) represents a two-dimensional diffuse reflection profile.
The above formula can also be expressed by convolution, since MeThe two-dimensional convolution produced by (x, y) is computationally complex and can be represented by Rd(x, y) is expressed as an approximation consisting of a series of separable functions, the operation being approximated by a number of one-dimensional convolution sequences, which are much less computationally complex, as shown in equation three below.
Wherein, aiRepresenting a one-dimensional convolution sequence.
Since the two-dimensional diffuse reflection profile is radially symmetric, the same one-dimensional convolution sequence is used in both directions. While a gaussian function with an average of 0 is a suitable simulation, it can be expressed as the following equation four.
Wherein, wiRepresenting the weight, G (x, y; sigma)i) Representing a gaussian function.
And creating a 3D image according to the rendered image, and displaying the 3D image on display equipment so that the left eye and the right eye of a viewer can see different images, thereby realizing the display of the 3D effect of the human virtual model.
The embodiment provides a human body virtual model display method, which comprises the steps of carrying out gesture recognition processing on an obtained user image to obtain a gesture recognition result; determining image processing operation corresponding to the gesture recognition result, and executing the image processing operation on a pre-constructed human body virtual model to obtain a displayed target human body virtual model; and based on a 3D image display technology, performing image generation and image rendering processing on the target human body virtual model to obtain and display a 3D image of the target human body virtual model. The method for displaying the virtual human body model can recognize the user gesture to operate the virtual human body model, and can perform naked eye 3D display based on a 3D image display technology, so that the virtual human body model can be flexibly and efficiently operated according to the user gesture, and the display effect of the virtual human body model is improved.
Corresponding to the human virtual model display method in the foregoing embodiment, fig. 4 is a schematic structural diagram of a human virtual model display apparatus provided in the embodiment of the present disclosure. For ease of illustration, only portions that are relevant to embodiments of the present disclosure are shown. Referring to fig. 4, the virtual human model display apparatus includes:
the gesture recognition unit 31 is configured to perform gesture recognition processing on the acquired user image to obtain a gesture recognition result;
the execution unit 32 is configured to determine an image processing operation corresponding to the gesture recognition result, and execute the image processing operation on a pre-constructed virtual human body model to obtain a displayed target virtual human body model;
and a display unit 33, configured to perform image generation and image rendering processing on the target virtual human body model based on a 3D image display technology, to obtain and display a 3D image of the target virtual human body model.
Optionally, the gesture recognition unit 31 is specifically configured to:
determining an image area of a hand in the user image;
performing hand key point identification on the image area to obtain a hand key point sequence in the image area; the hand key point sequence comprises the key point type, the position and the form of the hand in each image frame in the image area;
and determining the corresponding gesture according to the hand key point sequence to obtain a gesture recognition result.
Further, the gesture recognition unit 31 is specifically further configured to:
in the hand key point sequence, determining whether the key point type, position and form of any image frame meet the triggering condition;
and if so, executing the step of determining the corresponding gesture according to the hand key point sequence to obtain a gesture recognition result.
Further, the gesture recognition unit 31 is specifically further configured to:
determining the change trend of each hand key point of the hand in the continuous image frames according to the key point type, the position and the form of the hand of the continuous image frames in the hand key point sequence;
and determining gesture types and gesture tracks corresponding to the hand key point sequences according to the variation trend of each hand key point to obtain gesture recognition results.
Optionally, the execution unit 32 is specifically configured to:
calling a pre-stored image processing function of a corresponding type from a function library according to the gesture type;
processing the pre-stored image processing function according to the gesture track to obtain an image processing function to be executed;
and executing the image processing function to be executed on the pre-constructed human body virtual model to obtain a target human body virtual model to be displayed.
Optionally, the display unit 33 is specifically configured to:
constructing a plurality of virtual cameras, and calculating a sub-pixel mapping matrix of each virtual camera;
determining a camera image of each virtual camera presenting the target human body virtual model by utilizing each sub-pixel mapping matrix;
performing light and shadow rendering processing on each camera image according to the model material of the target human body virtual model to obtain a corresponding rendered image;
and creating and displaying a 3D image of the target human body virtual model according to each rendering image.
The embodiment provides a human body virtual model display device, which performs gesture recognition processing on an acquired user image to obtain a gesture recognition result; determining image processing operation corresponding to the gesture recognition result, and executing the image processing operation on a pre-constructed human body virtual model to obtain a displayed target human body virtual model; and based on a 3D image display technology, performing image generation and image rendering processing on the target human body virtual model to obtain and display a 3D image of the target human body virtual model. The virtual human model display device provided by the embodiment can recognize user gestures to operate the virtual human model, and can display the virtual human model in naked eyes 3D based on a 3D image display technology, so that the virtual human model can be flexibly and efficiently operated according to the user gestures, and the display effect of the virtual human model is improved.
Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure, and as shown in fig. 5, an electronic device 60 according to this embodiment may include: memory 61, processor 62.
A memory 61 for storing a computer program (e.g., an application program, a functional module, etc. for implementing the above-mentioned human virtual model display method), computer instructions, etc.;
the computer programs, computer instructions, etc. described above may be stored in one or more memories 61 in partitions. And the computer programs, computer instructions, data, etc. described above may be invoked by the processor 62.
A processor 62 for executing the computer program stored in the memory 61 to implement the steps of the method according to the above embodiments.
Reference may be made in particular to the description relating to the preceding method embodiment.
The memory 61 and the processor 62 may be separate structures or may be an integrated structure integrated together. When the memory 61 and the processor 62 are separate structures, the memory 61 and the processor 62 may be coupled by a bus 63.
The electronic device of this embodiment may execute the technical solution in the method shown in fig. 2, and for a specific implementation process and a technical principle, reference is made to the relevant description in the method shown in fig. 2, which is not described herein again.
In addition, embodiments of the present application further provide a computer-readable storage medium, in which computer-executable instructions are stored, and when at least one processor of the user equipment executes the computer-executable instructions, the user equipment performs the above-mentioned various possible methods.
Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be integral to the processor. The processor and the storage medium may reside in an ASIC. Additionally, the ASIC may reside in user equipment. Of course, the processor and the storage medium may reside as discrete components in a communication device.
Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Furthermore, the present application also provides a computer program product, including a computer program, which when executed by a processor implements the solution of the above human virtual model display method.
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents does not depart from the spirit of the disclosure. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.
Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
Claims (10)
1. A human body virtual model display method is characterized by comprising the following steps:
performing gesture recognition processing on the obtained user image to obtain a gesture recognition result;
determining image processing operation corresponding to the gesture recognition result, and executing the image processing operation on a pre-constructed human body virtual model to obtain a displayed target human body virtual model;
and based on a 3D image display technology, performing image generation and image rendering processing on the target human body virtual model to obtain and display a 3D image of the target human body virtual model.
2. The method according to claim 1, wherein the performing gesture recognition processing on the acquired user image to obtain a gesture recognition result comprises:
determining an image area of a hand in the user image;
performing hand key point identification on the image area to obtain a hand key point sequence in the image area; the hand key point sequence comprises the key point type, the position and the form of the hand in each image frame in the image area;
and determining the corresponding gesture according to the hand key point sequence to obtain a gesture recognition result.
3. The method according to claim 2, wherein before determining the corresponding gesture according to the hand key point sequence and obtaining a gesture recognition result, the method further comprises:
in the hand key point sequence, determining whether the key point type, position and form of any image frame meet the triggering condition;
and if so, executing the step of determining the corresponding gesture according to the hand key point sequence to obtain a gesture recognition result.
4. The method according to claim 3, wherein the determining the corresponding gesture according to the hand key point sequence to obtain a gesture recognition result comprises:
determining the change trend of each hand key point of the hand in the continuous image frames according to the key point type, the position and the form of the hand of the continuous image frames in the hand key point sequence;
and determining gesture types and gesture tracks corresponding to the hand key point sequences according to the variation trend of each hand key point to obtain gesture recognition results.
5. The method according to claim 4, wherein the determining an image processing operation corresponding to the gesture recognition result and performing the image processing operation on the pre-constructed virtual human body model to obtain the target virtual human body model to be displayed comprises:
calling a pre-stored image processing function of a corresponding type from a function library according to the gesture type;
processing the pre-stored image processing function according to the gesture track to obtain an image processing function to be executed;
and executing the image processing function to be executed on the pre-constructed human body virtual model to obtain a target human body virtual model to be displayed.
6. The method according to claims 1-5, wherein the performing image generation and image rendering processing on the target virtual human body model based on 3D image display technology to obtain and display a 3D image of the target virtual human body model comprises:
constructing a plurality of virtual cameras, and calculating a sub-pixel mapping matrix of each virtual camera;
determining a camera image of each virtual camera presenting the target human body virtual model by utilizing each sub-pixel mapping matrix;
performing light and shadow rendering processing on each camera image according to the model material of the target human body virtual model to obtain a corresponding rendered image;
and creating and displaying a 3D image of the target human body virtual model according to each rendering image.
7. A virtual mannequin display device, comprising:
the gesture recognition unit is used for performing gesture recognition processing on the acquired user image to obtain a gesture recognition result;
the execution unit is used for determining image processing operation corresponding to the gesture recognition result, executing the image processing operation on a pre-constructed human body virtual model and obtaining a displayed target human body virtual model;
and the display unit is used for carrying out image generation and image rendering processing on the target human body virtual model based on a 3D image display technology to obtain and display a 3D image of the target human body virtual model.
8. An electronic device, comprising: at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the method of any one of claims 1-6.
9. A computer-readable storage medium having computer-executable instructions stored thereon which, when executed by a processor, implement the method of any one of claims 1-6.
10. A computer program product comprising a computer program, characterized in that the computer program realizes the method of any of claims 1-6 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110055323.0A CN112799507B (en) | 2021-01-15 | 2021-01-15 | Human body virtual model display method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110055323.0A CN112799507B (en) | 2021-01-15 | 2021-01-15 | Human body virtual model display method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112799507A true CN112799507A (en) | 2021-05-14 |
| CN112799507B CN112799507B (en) | 2022-01-04 |
Family
ID=75809707
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110055323.0A Active CN112799507B (en) | 2021-01-15 | 2021-01-15 | Human body virtual model display method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112799507B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115331309A (en) * | 2022-08-19 | 2022-11-11 | 北京字跳网络技术有限公司 | Method, apparatus, device and medium for recognizing human body action |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2002150292A (en) * | 2001-09-14 | 2002-05-24 | Toshiba Corp | Dynamic image processing device |
| US20080273764A1 (en) * | 2004-06-29 | 2008-11-06 | Koninklijke Philips Electronics, N.V. | Personal Gesture Signature |
| US20110214055A1 (en) * | 2010-02-26 | 2011-09-01 | General Electric Company | Systems and Methods for Using Structured Libraries of Gestures on Multi-Touch Clinical Systems |
| CN105446481A (en) * | 2015-11-11 | 2016-03-30 | 周谆 | Gesture based virtual reality human-machine interaction method and system |
| CN106097442A (en) * | 2016-08-24 | 2016-11-09 | 广东华邦云计算股份有限公司 | A kind of intelligent simulation dressing system and application process thereof |
| CN106980383A (en) * | 2017-03-31 | 2017-07-25 | 哈尔滨工业大学 | A kind of dummy model methods of exhibiting, module and the virtual human body anatomical model display systems based on the module |
| CN107515674A (en) * | 2017-08-08 | 2017-12-26 | 山东科技大学 | It is a kind of that implementation method is interacted based on virtual reality more with the mining processes of augmented reality |
| CN108733287A (en) * | 2018-05-15 | 2018-11-02 | 东软集团股份有限公司 | Detection method, device, equipment and the storage medium of physical examination operation |
| CN109032358A (en) * | 2018-08-27 | 2018-12-18 | 百度在线网络技术（北京）有限公司 | The control method and device of AR interaction dummy model based on gesture identification |
| CN111651050A (en) * | 2020-06-09 | 2020-09-11 | 浙江商汤科技开发有限公司 | Method and device for displaying urban virtual sand table, computer equipment and storage medium |
| CN111880657A (en) * | 2020-07-30 | 2020-11-03 | 北京市商汤科技开发有限公司 | Virtual object control method and device, electronic equipment and storage medium |
| CN112154402A (en) * | 2019-08-29 | 2020-12-29 | 深圳市大疆创新科技有限公司 | Wearable device, control method thereof, gesture recognition method and control system |
-
2021
- 2021-01-15 CN CN202110055323.0A patent/CN112799507B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2002150292A (en) * | 2001-09-14 | 2002-05-24 | Toshiba Corp | Dynamic image processing device |
| US20080273764A1 (en) * | 2004-06-29 | 2008-11-06 | Koninklijke Philips Electronics, N.V. | Personal Gesture Signature |
| US20110214055A1 (en) * | 2010-02-26 | 2011-09-01 | General Electric Company | Systems and Methods for Using Structured Libraries of Gestures on Multi-Touch Clinical Systems |
| CN105446481A (en) * | 2015-11-11 | 2016-03-30 | 周谆 | Gesture based virtual reality human-machine interaction method and system |
| CN106097442A (en) * | 2016-08-24 | 2016-11-09 | 广东华邦云计算股份有限公司 | A kind of intelligent simulation dressing system and application process thereof |
| CN106980383A (en) * | 2017-03-31 | 2017-07-25 | 哈尔滨工业大学 | A kind of dummy model methods of exhibiting, module and the virtual human body anatomical model display systems based on the module |
| CN107515674A (en) * | 2017-08-08 | 2017-12-26 | 山东科技大学 | It is a kind of that implementation method is interacted based on virtual reality more with the mining processes of augmented reality |
| CN108733287A (en) * | 2018-05-15 | 2018-11-02 | 东软集团股份有限公司 | Detection method, device, equipment and the storage medium of physical examination operation |
| CN109032358A (en) * | 2018-08-27 | 2018-12-18 | 百度在线网络技术（北京）有限公司 | The control method and device of AR interaction dummy model based on gesture identification |
| CN112154402A (en) * | 2019-08-29 | 2020-12-29 | 深圳市大疆创新科技有限公司 | Wearable device, control method thereof, gesture recognition method and control system |
| CN111651050A (en) * | 2020-06-09 | 2020-09-11 | 浙江商汤科技开发有限公司 | Method and device for displaying urban virtual sand table, computer equipment and storage medium |
| CN111880657A (en) * | 2020-07-30 | 2020-11-03 | 北京市商汤科技开发有限公司 | Virtual object control method and device, electronic equipment and storage medium |
Non-Patent Citations (4)
| Title |
|---|
| X. JI等: "Design of Human Machine Interactive System based on Hand Gesture recognition", 《2019 6TH INTERNATIONAL CONFERENCE ON INFORMATION SCIENCE AND CONTROL ENGINEERING (ICISCE)》 * |
| 杨志伟等: "基于纹理空间与屏幕空间混合的心脏次表面绘制方案", 《贵州大学学报(自然科学版)》 * |
| 邹晨等: "VR环境中产品设计手势的定义与合成", 《工程图学学报》 * |
| 陈龙等: "基于手势传感与智能设备的交互方法研究", 《计算机与数字工程》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115331309A (en) * | 2022-08-19 | 2022-11-11 | 北京字跳网络技术有限公司 | Method, apparatus, device and medium for recognizing human body action |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112799507B (en) | 2022-01-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11335379B2 (en) | Video processing method, device and electronic equipment | |
| US11538229B2 (en) | Image processing method and apparatus, electronic device, and computer-readable storage medium | |
| WO2018188499A1 (en) | Image processing method and device, video processing method and device, virtual reality device and storage medium | |
| US9886102B2 (en) | Three dimensional display system and use | |
| Francone et al. | Using the user's point of view for interaction on mobile devices | |
| CN108594999B (en) | Control method and device for panoramic image display system | |
| CN108139801B (en) | System and method for performing electronic display stabilization via preserving light field rendering | |
| US8379955B2 (en) | Visualizing a 3D volume dataset of an image at any position or orientation from within or outside | |
| Fender et al. | Optispace: Automated placement of interactive 3d projection mapping content | |
| US11042730B2 (en) | Method, apparatus and device for determining an object, and storage medium for the same | |
| US20140160122A1 (en) | Creating a virtual representation based on camera data | |
| Jeong et al. | Development of a real-time integral imaging display system based on graphics processing unit parallel processing using a depth camera | |
| CN111467803B (en) | Display control method and device in game, storage medium and electronic equipment | |
| WO2021263035A1 (en) | Object recognition neural network for amodal center prediction | |
| CN112799507B (en) | Human body virtual model display method and device, electronic equipment and storage medium | |
| CN107065164A (en) | Image presentation method and device | |
| US20230368432A1 (en) | Synthesized Camera Arrays for Rendering Novel Viewpoints | |
| Wischgoll | Display systems for visualization and simulation in virtual environments | |
| CN108510433B (en) | Space display method and device and terminal | |
| US9035945B1 (en) | Spatial derivative-based ray tracing for volume rendering | |
| US11636578B1 (en) | Partial image completion | |
| CN114241127A (en) | Panoramic image generation method and device, electronic equipment and medium | |
| Boubekeur | ShellCam: Interactive geometry-aware virtual camera control | |
| Weiskopf | An immersive virtual environment for special relativity | |
| US20240119570A1 (en) | Machine learning model training using synthetic data for under-display camera (udc) image restoration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |