CN113112612B - Positioning method and system for dynamic superposition of real person and mixed reality - Google Patents
Positioning method and system for dynamic superposition of real person and mixed reality Download PDFInfo
- Publication number
- CN113112612B CN113112612B CN202110414359.3A CN202110414359A CN113112612B CN 113112612 B CN113112612 B CN 113112612B CN 202110414359 A CN202110414359 A CN 202110414359A CN 113112612 B CN113112612 B CN 113112612B
- Authority
- CN
- China
- Prior art keywords
- real
- module
- character
- image
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 230000000007 visual effect Effects 0.000 claims abstract description 10
- 230000000694 effects Effects 0.000 claims abstract description 8
- 239000000872 buffer Substances 0.000 claims description 56
- 239000000463 material Substances 0.000 claims description 26
- 230000015572 biosynthetic process Effects 0.000 claims description 13
- 238000003786 synthesis reaction Methods 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 12
- 230000009286 beneficial effect Effects 0.000 claims description 6
- 230000002194 synthesizing effect Effects 0.000 claims description 2
- 230000008859 change Effects 0.000 abstract description 13
- 230000003993 interaction Effects 0.000 abstract description 5
- 238000005516 engineering process Methods 0.000 description 15
- 210000000988 bone and bone Anatomy 0.000 description 10
- 238000013519 translation Methods 0.000 description 10
- 238000004364 calculation method Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000005094 computer simulation Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000000873 masking effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/20—3D [Three Dimensional] animation
- G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Graphics (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Architecture (AREA)
- Geometry (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention discloses a method and a system for positioning a real person and a mixed reality by dynamic superposition, wherein the method comprises the following steps: the human body acquisition module acquires a real person image through camera shooting, acquires three-dimensional data information and position information of the person, and establishes a human body three-dimensional model of the real person image, wherein the human body three-dimensional model comprises the three-dimensional data information and the position information; the virtual generation module generates virtual content; the display change of virtual contents at the corresponding positions is adjusted through the real-time change of the position information of the real character image, so that a final three-dimensional scene is obtained; and displaying the final three-dimensional scene, so that real-time dynamic activities and changes of a plurality of real character images shot by the camera according with visual laws in the virtual content are realized. And positioning the user by acquiring the position information of the real character, and displaying the real user and the virtual three-dimensional scene content in real time and dynamically combined mode according to the positioning information, so that real character interaction with the virtual three-dimensional scene content in real time is realized.
Description
Technical Field
The invention relates to the technical field of computer vision, in particular to a positioning method and a system for dynamic superposition of a real person and mixed reality.
Background
At present, rapid development of related fields is driven by AR (augmented reality), VR (virtual reality) and MR (mixed reality). Some games manufactured by related technologies exist in places such as a part of market banks and the like, customers are attracted to interact with machines to play the games, so that the customers have a participation sense, and the brand image can be improved. The essence of the method is that the virtual three-dimensional scene manufactured by a computer and the moving image of the person shot by the camera on site are digitally synthesized in real time, so that the person and the virtual background can be synchronously changed, and seamless fusion of the two is realized, so that a perfect synthesized picture is obtained.
In order to achieve the effect, a color key image matting technology is used in a three-dimensional virtual studio, a foreground character is placed in front of a background plate with a specific color, then a background part with the specific color in a shot picture is scratched, and a rendered three-dimensional virtual background is used for replacing the background with the specific color to form an output synthesized picture.
However, the existing technology has a plurality of defects: the virtual object is above the person, and the virtual object shields the real person in a default state, which does not accord with the visual law of the real world and the front-back position relationship. Most of the current AR and MR technologies have the problem that Hololens can greatly influence the substitution sense of customers on program scenes, the sense of reality is poor, and the users cannot really feel and experience.
Disclosure of Invention
Based on the above, the present invention provides a method and a system for positioning a real person and a mixed reality by dynamic superposition, which comprises the following steps:
S1: the human body acquisition module acquires a real person image through camera shooting, acquires three-dimensional data information and position information of the person, and establishes a human body three-dimensional model of the real person image, wherein the human body three-dimensional model comprises the three-dimensional data information and the position information;
s2: the virtual generation module generates virtual content;
s3: the image synthesis module synthesizes the real figure image and the virtual content according to the three-dimensional data information and the position information of the human body three-dimensional model and the virtual content by combining calculation, so as to obtain a preliminary three-dimensional scene;
S41: coordinating the display of the real character image and the virtual content by utilizing a character shielding module, so that the virtual content can be displayed behind the real character image, and a final three-dimensional scene is obtained;
And/or the number of the groups of groups,
S42: coordinating the display of the real character image and the virtual content by utilizing a character shielding module, and adjusting the display change of the virtual content at the corresponding position through the real-time change of the position information of the real character image to obtain a final three-dimensional scene;
s5: and displaying the final three-dimensional scene, so that real-time dynamic activities and changes of the real character image shot by the camera according with the visual law in the virtual content are realized.
Preferably, step S41 and/or step S42 further comprises the steps of:
the character shielding module defines a buffer zone module through a drawing pipeline;
The buffer zone module comprises a buffer zone of pixel information and a depth buffer zone, wherein the pixel information is used for identifying the shape of an object from a camera source, so that the pixel information is beneficial to the subsequent use of the pixel information to shield virtual content, and the depth buffer zone is used for providing depth position information of a real person image.
The character can be accurately extracted from the initial three-dimensional scene extracted by using the mask, and the depth reference plane in the initial three-dimensional scene can be accurately positioned by the character shielding module by combining the depth position information of the depth buffer zone to calculate;
And redrawing the preliminary three-dimensional scene to enable virtual content to be displayed behind the real character image, thereby obtaining a final three-dimensional scene.
Preferably, the method further comprises the following steps: and keeping the buffer area of the pixel information and the depth buffer area consistent with the pixel and the running frequency of the camera.
Preferably, the step S2 further includes the steps of:
editing materials of a 3D scene material library through an environment generation unit;
and adding a positioning coordinate system for the edited material through a positioning coordinate system unit, and extracting the positioning coordinates of the display area.
Preferably, the image synthesis module of step S3 further includes the following steps:
S31: setting display skeleton points for the human body three-dimensional model according to the three-dimensional data information through a picture superposition unit;
S32: superimposing the virtual content on an upper layer of the real character image;
S33: and matching and binding according to the space coordinates and the positioning coordinates of the displayed skeleton points, and obtaining a preliminary three-dimensional scene after the space coordinates and the positioning coordinates are overlapped.
The invention also provides a system for dynamically superposing the real character and the mixed reality, which comprises: comprising the following steps: a human body acquisition module, a virtual generation module, an image synthesis module and a character shielding module,
The human body acquisition module acquires a real person image through camera shooting, acquires three-dimensional data information and position information of the person, and establishes a human body three-dimensional model of the real person image, wherein the human body three-dimensional model comprises the three-dimensional data information and the position information;
the virtual generation module generates virtual content;
the image synthesis module synthesizes the real figure image and the virtual content according to the three-dimensional data information and the position information of the human body three-dimensional model and the virtual content by combining calculation, so as to obtain a preliminary three-dimensional scene;
the person shielding module coordinates the display of the real person image and the virtual content, so that the virtual content can be displayed behind the real person image, and a final three-dimensional scene is obtained;
And/or the number of the groups of groups,
And the person shielding module coordinates the display of the real person image and the virtual content, and adjusts the display change of the virtual content at the corresponding position through the real-time change of the position information of the real person image to obtain a final three-dimensional scene.
Preferably, the virtual generation module comprises a 3D scene material library, an environment generation unit and a positioning coordinate system unit, and the environment generation unit edits the materials of the 3D scene material library;
The positioning coordinate system unit is used for adding a positioning coordinate system for the edited material and extracting the positioning coordinate of the display area.
Preferably, the system further comprises a display module and a storage module, wherein the display module is used for displaying the three-dimensional scene, and the storage module is used for storing data and information generated by the system.
The invention also provides a device for dynamically superposing the real person and the mixed reality, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and is characterized in that the processor realizes the steps of the positioning method for dynamically superposing the real person and the mixed reality when executing the computer program.
The invention also proposes a computer readable storage medium storing a computer program which, when executed, implements the steps of the method for positioning a dynamic superposition of a real person and a mixed reality.
The embodiment of the invention has the following beneficial effects:
Positioning a user by acquiring the position information of the real character, and displaying the real user and the virtual three-dimensional scene content in a real-time and dynamic combined manner according to the positioning information, so that the real character can interact with the virtual three-dimensional scene content in real time, and the user experience and experience are improved; the character shielding module is used for shielding the virtual content, the virtual content can be displayed behind the body of the real character image, the virtual content accords with the visual law and the front-back position relationship of the real world, the mixed reality scene looks more real and natural, and the seamless fusion of the virtual and the reality is realized.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Wherein:
FIG. 1 is a schematic flow diagram of a method according to one embodiment;
FIG. 2 is a schematic illustration of the various joints of the method described in one embodiment;
FIG. 3 is a schematic diagram of the implementation of the method in one embodiment;
FIG. 4 is another schematic diagram of the implementation of the method in one embodiment.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative and intended to explain the present invention and should not be construed as limiting the invention.
With the increasing demand of people for intelligent devices, augmented Reality (AR) technology, virtual Reality (VR) technology and Mixed Reality (MR) technology are widely applied in multiple fields, especially in the field of remote interaction technology, so that original remote interaction modes and interaction scenes are greatly enriched.
The AR technology is a technology for skillfully fusing virtual information with the real world, and widely uses various technical means such as multimedia, three-dimensional modeling, real-time tracking and registering, intelligent interaction, sensing and the like, and applies virtual information such as characters, images, three-dimensional models, music, videos and the like generated by a computer to the real world after simulation, wherein the two information are mutually complemented, so that the enhancement of the real world is realized.
VR technology is a computer simulation system that creates and experiences a virtual world by using a computer to create a simulated environment into which a user is immersed. The virtual reality technology is to use data in real life, combine electronic signals generated by computer technology with various output devices to convert the electronic signals into phenomena which can be perceived by people, wherein the phenomena can be real and cut objects in reality or substances which can not be seen by naked eyes, and the phenomena are shown by a three-dimensional model.
MR means that a new environment and a visual three-dimensional world are created by combining a real world and a virtual world, and MR technology can realize free switching between virtual and real, so that reality can be maintained in the virtual, and the reality can be converted into the virtual.
As shown in fig. 1, the invention provides a positioning method for dynamic superposition of a real person and mixed reality, which comprises the following steps:
S1: the human body acquisition module acquires a real person image through camera shooting, acquires three-dimensional data information and position information of the person, and establishes a human body three-dimensional model of the real person image, wherein the human body three-dimensional model comprises the three-dimensional data information and the position information;
S2: the virtual generation module generates virtual content; specifically, the virtual content is virtual three-dimensional scene content presented through a mixed reality scene;
s3: the image synthesis module synthesizes the real figure image and the virtual content according to the three-dimensional data information and the position information of the human body three-dimensional model and the virtual content by combining calculation, so as to obtain a preliminary three-dimensional scene;
S41: coordinating the display of the real character image and the virtual content by utilizing a character shielding module, so that the virtual content can be displayed behind the real character image, and a final three-dimensional scene is obtained;
And/or the number of the groups of groups,
S42: coordinating the display of the real character image and the virtual content by utilizing a character shielding module, and adjusting the display change of the virtual content at the corresponding position through the real-time change of the position information of the real character image to obtain a final three-dimensional scene;
S5: and displaying the final three-dimensional scene, so that real-time dynamic activities and changes of the real character image shot by the camera according with the visual law in the virtual content are realized. The three-dimensional scene of the finally presented mixed reality is displayed according to the standing position of the real person, and the real person is positioned according to real time so as to achieve a good display effect of combining the real person and the mixed reality scene.
The method comprises the steps of obtaining the position information of a real person, positioning a user, determining the position information of the real person according to the positioning information, and realizing the shielding of virtual content by the person through the person shielding module. And the real character and the virtual three-dimensional scene content are dynamically overlapped in real time according to the position of the real character to be displayed, so that the real character and the virtual three-dimensional scene content are interacted in real time.
The human body acquisition module comprises image pickup equipment and Kinect, acquires RBG-D image streams of a scene with a person, and outputs real person images with transparent background frame by frame, wherein the image pickup equipment comprises a camera;
The method comprises the steps of acquiring RBG-D image streams of a scene with a person by using Kinect, extracting a color image and a depth image of each frame, then cutting the color image and the depth image, wherein the cutting range of the color image and the depth image is the same, and the ratio and the area of the images are different as the resolution of a Kinect depth camera is 512 x 424 and the resolution of a color camera is 1920 x 1080, so that the two images are required to be cut out in a superposition range; serializing the cut depth image and the color image to obtain a depth image array and a color image array; carrying out manned judgment on pixel points on the depth image according to a human body index coordinate system, carrying out unmanned point or manned point judgment on each pixel point, carrying out point-by-point mapping on the color image and the depth image, setting the color of the unmanned point as transparent, and setting the color of the manned point as primary color, so as to obtain a color image data stream; and (5) performing deserialization to obtain a real character image with transparent background.
The virtual generation module comprises a 3D scene material library, an environment generation unit and a positioning coordinate system unit, and the step S2 further comprises the following steps: editing materials of a 3D scene material library through an environment generation unit; and adding a positioning coordinate system for the edited material through a positioning coordinate system unit, and extracting the positioning coordinates of the display area. The virtual content generated in step S2 thus contains a coordinate system and positioning coordinates.
The image synthesis module further comprises the following steps:
S31: setting display skeleton points for the human body three-dimensional model according to the three-dimensional data information through a picture superposition unit;
S32: superimposing the virtual content on an upper layer of the real character image;
S33: and matching and binding according to the space coordinates and the positioning coordinates of the displayed skeleton points, and obtaining a preliminary three-dimensional scene after the space coordinates and the positioning coordinates are overlapped.
As shown in fig. 2, in a preferred embodiment, the picture overlaying unit further comprises the steps of:
Setting a presentation skeleton point by the setting section; and matching the space coordinates of the displayed skeleton points with the positioning coordinates through the positioning part, and adjusting the corresponding positions between the real character image and the virtual content.
The method for matching the space coordinates and the positioning coordinates of the bone points comprises the following steps: leading in a real figure image to a positioning coordinate system, comparing the positioning coordinate with the space coordinate of the displayed skeleton point, and respectively making difference between an abscissa value and an ordinate value to obtain a transverse translation distance and a longitudinal translation distance of the displayed skeleton point; or setting the initial coordinate point showing the bone point in the positioning coordinate system as the origin of the positioning coordinate system, and showing the transverse translation distance of the bone point as the abscissa value of the positioning coordinate and the longitudinal translation distance as the ordinate value of the positioning coordinate.
In the image superposition unit of the image synthesis module, the method for extracting the positioning coordinates of the display area comprises the following steps: and searching a longitudinal longest line and a transverse longest line in the display area, wherein the coordinate of the intersection point of the longitudinal longest line and the transverse longest line is a positioning coordinate.
The space coordinate value of the displayed skeleton point is a scaling value of the real character image, the space coordinate value can be obtained by comparing the distance between the Kinect and the human body with the set distance, if the space coordinate value is larger than the preset coordinate value, the space coordinate value is larger than the default coordinate value, the human body foreground picture needs to be reduced, and otherwise, the human body foreground picture is enlarged.
In a preferred embodiment, step S41 and/or step S42 further comprises the steps of:
the character shielding module defines a buffer zone module through a drawing pipeline;
The buffer zone module comprises a buffer zone of pixel information and a depth buffer zone, wherein the pixel information is used for identifying the shape of an object from a camera source, so that the pixel information is beneficial to the subsequent use of the pixel information to shield virtual content, and the depth buffer zone is used for providing depth position information of a real person image.
The character can be accurately extracted from the initial three-dimensional scene extracted by using the mask, and the depth reference plane in the initial three-dimensional scene can be accurately positioned by the character shielding module by combining the depth position information of the depth buffer zone to calculate;
And redrawing the preliminary three-dimensional scene to enable virtual content to be displayed behind the real character image, thereby obtaining a final three-dimensional scene.
Specifically, the above steps are explained in detail by the following description.
Most AR applications we see are similar to fig. 3, i.e. the virtual content is above the image of the real person, and the virtual content in default state obscures the real person, which does not conform to the visual law of the real world and the front-back positional relationship. Most of the AR and MR technologies currently suffer from this problem, including Hololens. The visual laws of the real world and the front-back positional relationship should be as shown in fig. 4, i.e., the virtual object may be presented behind a real character.
In step S3, the image synthesis module is used to synthesize the real person image and the virtual content, so as to obtain a preliminary three-dimensional scene, realize the movement and change of the real person image shot by the image pickup device in the virtual content, and only cover the virtual content on the upper layer of the camera, so that although the AR effect of virtual and reality superposition is realized, the result is that an erroneous shielding relationship is generated. The false occlusion relationship is essentially a problem of depth ordering (Depth Ordering).
The depth ordering refers to the reference planes of objects in the same picture at different depths, so that people can see different objects at different depths, and each person is on the reference planes at different depths. It is assumed that there are 3 objects (1 virtual content and 2 real objects) on the same picture, at different depths, and the virtual content is in the middle of the other 2 real objects, and the exact position of the virtual content is already known by the drawing pipeline (Render-pipeline), so that only one buffer module needs to be simply used.
Whereas in the case where the drawing pipeline does not know the exact location of the virtual content, the drawing pipeline needs to know the location of each object in the scene, thus adding two split buffers.
The partition buffer contains a buffer and a depth buffer of pixel information that is used to identify the shape of an object from the camera source, facilitating subsequent use of the pixel information to occlude virtual content.
The depth buffer provides depth position information of the person, the depth buffer being used to represent an estimated depth value in a camera source occluding the virtual content.
It should be noted that the pixel information buffer and the depth buffer are consistent with the pixel and the operation frequency of the camera.
The buffer of pixel information and the depth buffer need to be made into frames of the camera at the same frequency. If the camera is operating at a frequency of 60 per second/frame, the character occlusion module will also generate a buffer of the pixel information and the depth buffer at a frequency of 60 per second/frame. At the same time, it is also ensured that the generated buffer of pixel information and the depth buffer are consistent with the pixels of the camera.
In another embodiment, the person shielding module further includes a neural network computing module, where the neural network computing module is configured to train the three-dimensional scene model to be used for viewing the preliminary three-dimensional scene to a very small extent, and after the preliminary three-dimensional scene is enlarged, many details that are not viewed will be revealed. Therefore, in order to compensate for the loss of detail of the preliminary three-dimensional scene, some other processing is needed, and the detail of the specific loss is known through the mask extraction as a guide of the camera.
In the step, the primary three-dimensional scene extracted by using the mask can accurately extract the characters from the scene, and the character masking module can accurately position the depth reference surface in the scene of the primary three-dimensional scene by combining the depth estimation data calculation of the depth buffer zone, so that the depth ordering problem is solved. And finally, redrawing the initial three-dimensional scene.
In another embodiment, the person occlusion module also has an occlusion in the case of multiple persons. The processing method is similar to depth ordering, and can also process shielding under the condition of multiple persons.
In actual operation, the person shielding module is integrated on the control module, and the control module may be an a12 processor, where the person shielding function needs to be implemented by combining the a12 processor with machine learning, and the segmentation buffer area and the depth buffer area may be generated through photos.
The storage module is used for storing the data and information generated by the method, and the character shielding module or the storage module can be an upper computer or a database server.
In another embodiment, in the positioning method for dynamic superposition of real characters and mixed reality, real-time dynamic superposition of multiple people and virtual contents can be performed, and the method comprises the following steps:
S1: the human body acquisition module acquires a plurality of real person images through camera shooting, acquires three-dimensional data information and position information of the plurality of persons, and establishes a human body three-dimensional model containing the three-dimensional data information and the position information of the plurality of real person images;
s2: the virtual generation module generates virtual content;
S3: the image synthesis module synthesizes the real figure image and the virtual content according to the three-dimensional data information and the position information of the three-dimensional human models and the virtual content in a combined mode to obtain a preliminary three-dimensional scene;
S41: coordinating the display of the plurality of real character images and the virtual content by utilizing a character shielding module, so that the virtual content can be displayed behind the real character images, and a final three-dimensional scene is obtained;
And/or the number of the groups of groups,
S42: coordinating the display of the real character images and the virtual contents by utilizing a character shielding module, and adjusting the display change of the virtual contents at the corresponding positions through the real-time change of the position information of the real character images to obtain a final three-dimensional scene;
s5: and displaying the final three-dimensional scene, so that real-time dynamic activities and changes of a plurality of real character images shot by the camera according with visual laws in the virtual content are realized.
Since there are a plurality of real person images, adjustment is required in the step of the image synthesizing module, and assuming that there are 2 human bodies, a real person image a and a real person image B are obtained in step S1. In the setting part of the picture superposition unit, a plurality of display skeleton points are required to be set, and the display skeleton points are assumed to be an A skeleton point and a B skeleton point; and the positioning part of the picture superposition unit is used for matching the space coordinates of the displayed skeleton points with the positioning coordinates and adjusting the corresponding positions between the real character images and the virtual contents. The method for matching the space coordinates and the positioning coordinates of the bone points comprises the following steps: leading in a real character image A to a positioning coordinate system, comparing the positioning coordinate with the space coordinate of the displayed skeleton point A, and respectively making difference between an abscissa value and an ordinate value to obtain a transverse translation distance and a longitudinal translation distance of the displayed skeleton point A; or setting the initial coordinate point showing that the bone point A is positioned in the positioning coordinate system as the origin of the positioning coordinate system, and showing that the transverse translation distance of the bone point A is the abscissa value of the positioning coordinate and the longitudinal translation distance is the ordinate value of the positioning coordinate. Meanwhile, the real character image B is imported into a positioning coordinate system, the positioning coordinate is compared with the space coordinate of the display bone point A, and the horizontal translation distance and the longitudinal translation distance of the display bone point B are obtained by respectively making differences between the horizontal coordinate value and the vertical coordinate value. I.e. with reference to the positioning coordinate system of 1 bone point, other bone points and so on.
In another embodiment, in the positioning method for dynamic superposition of real characters and mixed reality of the present invention, real-time dynamic superposition of multiple real character images and multiple virtual contents can be performed, and multiple virtual contents are generated in a virtual generation module, then the method further includes the following steps: editing a plurality of materials of a 3D scene material library through an environment generation unit; and adding a positioning coordinate system for the edited materials through a positioning coordinate system unit. At this time, the materials correspond to the virtual contents, and the virtual contents adopt a uniform positioning coordinate system, namely, 1 positioning coordinate system is used as a reference, and other skeleton points and the like.
The invention also provides a system for dynamically superposing the character and the virtual content in real time, which comprises: a human body acquisition module, a virtual generation module, an image synthesis module, a character shielding module, a display module and a storage module,
The human body acquisition module acquires a real person image through camera shooting, acquires three-dimensional data information and position information of the person, and establishes a human body three-dimensional model of the real person image, wherein the human body three-dimensional model comprises the three-dimensional data information and the position information;
the virtual generation module generates virtual content;
the image synthesis module synthesizes the real figure image and the virtual content according to the three-dimensional data information and the position information of the human body three-dimensional model and the virtual content by combining calculation, so as to obtain a preliminary three-dimensional scene;
the person shielding module coordinates the display of the real person image and the virtual content, so that the virtual content can be displayed behind the real person image, and a final three-dimensional scene is obtained;
And/or the number of the groups of groups,
And the person shielding module coordinates the display of the real person image and the virtual content, and adjusts the display change of the virtual content at the corresponding position through the real-time change of the position information of the real person image to obtain a final three-dimensional scene.
Specifically, the virtual generation module comprises a 3D scene material library, an environment generation unit and a positioning coordinate system unit, and the environment generation unit edits the materials of the 3D scene material library;
The positioning coordinate system unit is used for adding a positioning coordinate system for the edited material and extracting the positioning coordinate of the display area.
The system of the embodiment also comprises a display module and a storage module, wherein the display module is used for displaying the three-dimensional scene, and the storage module is used for storing data and information generated by the system.
The embodiment of the invention also provides equipment for dynamically superposing the real person and the mixed reality, which comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, wherein the processor realizes the steps of the positioning method for dynamically superposing the real person and the mixed reality when executing the computer program.
The embodiment of the invention also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program which can be loaded and executed by a processor, and the computer program realizes the steps of the positioning method for dynamically superposing the real character and the mixed reality when being executed.
It should be noted that the above-mentioned positioning method for dynamic superposition of real person and mixed reality, system for dynamic superposition of real person and mixed reality, device for dynamic superposition of real person and mixed reality, and computer-readable storage medium belong to a general inventive concept, and the contents thereof are mutually applicable.
Those skilled in the art will appreciate that all or part of the processes in the methods of the above embodiments may be implemented by a computer program for instructing relevant hardware, where the program may be stored in a non-volatile computer readable storage medium, and where the program, when executed, may include processes in the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (SYNCHLINK) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.
Claims (10)
1. The method for dynamically superposing the real person and the mixed reality is characterized by comprising the following steps of:
step S1: the human body acquisition module acquires a real person image through camera shooting, acquires three-dimensional data information and position information of the person, and establishes a human body three-dimensional model of the real person image, wherein the human body three-dimensional model comprises the three-dimensional data information and the position information;
step S2: the virtual generation module generates virtual content;
Step S3: an image composition module comprising the steps of: s31: setting display skeleton points for the human body three-dimensional model according to the three-dimensional data information through a picture superposition unit; s32: superimposing the virtual content on an upper layer of the real character image; s33: matching and binding according to the space coordinates and the positioning coordinates of the displayed skeleton points, and obtaining a preliminary three-dimensional scene after the space coordinates and the positioning coordinates are overlapped;
Step S41 and/or step S42 include: the character shielding module defines a buffer zone module through a drawing pipeline; the buffer zone module comprises a buffer zone of pixel information and a depth buffer zone, wherein the pixel information is used for identifying the shape of an object from a camera source, so that the pixel information is beneficial to the subsequent use of the pixel information to shield virtual contents, and the depth buffer zone is used for providing depth position information of a real person image; the character can be accurately extracted from the initial three-dimensional scene extracted by using the mask, and the depth reference plane in the initial three-dimensional scene can be accurately positioned by the character shielding module by combining the depth position information of the depth buffer zone to calculate; redrawing the preliminary three-dimensional scene to enable virtual content to be displayed behind the real character image, and obtaining a final three-dimensional scene; the buffer area of the pixel information is consistent with the depth buffer area, and the pixel and the running frequency of the camera are consistent;
step S5: and displaying the final three-dimensional scene, so that real-time dynamic activities and changes of the real character image shot by the camera according with the visual law in the virtual content are realized.
2. The method for dynamically superimposing a real character on a mixed reality according to claim 1, wherein the step S41 and/or the step S42 further comprises the steps of:
the character shielding module defines a buffer zone module through a drawing pipeline;
The buffer zone module comprises a buffer zone of pixel information and a depth buffer zone, wherein the pixel information is used for identifying the shape of an object from a camera source, so that the pixel information is beneficial to the subsequent use of the pixel information to shield virtual contents, and the depth buffer zone is used for providing depth position information of a real person image;
The character can be accurately extracted from the initial three-dimensional scene extracted by using the mask, and the depth reference plane in the initial three-dimensional scene can be accurately positioned by the character shielding module by combining the depth position information of the depth buffer zone to calculate;
And redrawing the preliminary three-dimensional scene to enable virtual content to be displayed behind the real character image, thereby obtaining a final three-dimensional scene.
3. The method for dynamically superimposing a real character and a mixed reality according to claim 2, further comprising the steps of:
And keeping the buffer area of the pixel information and the depth buffer area consistent with the pixel and the running frequency of the camera.
4. The method for dynamically superimposing a real character on a mixed reality according to claim 1, wherein the step S2 further comprises the steps of:
editing materials of a 3D scene material library through an environment generation unit;
and adding a positioning coordinate system for the edited material through a positioning coordinate system unit, and extracting the positioning coordinates of the display area.
5. The method for dynamically superimposing a real person and a mixed reality according to claim 1, wherein the image synthesizing module of step S3 further comprises the steps of:
S31: setting display skeleton points for the human body three-dimensional model according to the three-dimensional data information through a picture superposition unit;
S32: superimposing the virtual content on an upper layer of the real character image;
S33: and matching and binding according to the space coordinates and the positioning coordinates of the displayed skeleton points, and obtaining a preliminary three-dimensional scene after the space coordinates and the positioning coordinates are overlapped.
6. A system for dynamic superposition of a real character and a mixed reality, comprising: a human body acquisition module, a virtual generation module, an image synthesis module and a character shielding module,
The human body acquisition module acquires a real person image through camera shooting, acquires three-dimensional data information and position information of the person, and establishes a human body three-dimensional model of the real person image, wherein the human body three-dimensional model comprises the three-dimensional data information and the position information;
the virtual generation module generates virtual content;
the image synthesis module includes: setting display skeleton points for the human body three-dimensional model according to the three-dimensional data information through a picture superposition unit; superimposing the virtual content on an upper layer of the real character image; matching and binding according to the space coordinates and the positioning coordinates of the displayed skeleton points, and obtaining a preliminary three-dimensional scene after the space coordinates and the positioning coordinates are overlapped;
The character shielding module defines a buffer zone module through a drawing pipeline; the buffer zone module comprises a buffer zone of pixel information and a depth buffer zone, wherein the pixel information is used for identifying the shape of an object from a camera source, so that the pixel information is beneficial to the subsequent use of the pixel information to shield virtual contents, and the depth buffer zone is used for providing depth position information of a real person image; the character can be accurately extracted from the initial three-dimensional scene extracted by using the mask, and the depth reference plane in the initial three-dimensional scene can be accurately positioned by the character shielding module by combining the depth position information of the depth buffer zone to calculate; redrawing the preliminary three-dimensional scene to enable virtual content to be displayed behind the real character image, and obtaining a final three-dimensional scene; the buffer area of the pixel information is consistent with the depth buffer area, and the pixel and the running frequency of the camera are consistent.
7. The system for dynamic superposition of real characters and mixed reality according to claim 6, wherein the virtual generation module comprises a 3D scene material library, an environment generation unit and a positioning coordinate system unit, and the material of the 3D scene material library is edited by the environment generation unit;
The positioning coordinate system unit is used for adding a positioning coordinate system for the edited material and extracting the positioning coordinate of the display area.
8. The system for dynamic superposition of real person and mixed reality according to claim 7, further comprising a display module for displaying the three-dimensional scene and a storage module for storing system-generated data and information.
9. An apparatus for dynamic superposition of a real person and a mixed reality, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method for positioning a dynamic superposition of a real person and a mixed reality according to any of claims 1 to 5 when the computer program is executed by the processor.
10. A computer readable storage medium storing a computer program, characterized in that the computer program when executed implements the steps of the method for positioning a real person dynamically superimposed with mixed reality according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110414359.3A CN113112612B (en) | 2021-04-16 | 2021-04-16 | Positioning method and system for dynamic superposition of real person and mixed reality |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110414359.3A CN113112612B (en) | 2021-04-16 | 2021-04-16 | Positioning method and system for dynamic superposition of real person and mixed reality |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113112612A CN113112612A (en) | 2021-07-13 |
| CN113112612B true CN113112612B (en) | 2024-06-04 |
Family
ID=76718194
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110414359.3A Active CN113112612B (en) | 2021-04-16 | 2021-04-16 | Positioning method and system for dynamic superposition of real person and mixed reality |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113112612B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113490063B (en) * | 2021-08-26 | 2023-06-23 | 上海盛付通电子支付服务有限公司 | Method, device, medium and program product for live interaction |
| CN114793276A (en) * | 2022-04-22 | 2022-07-26 | 上海爱客博信息技术有限公司 | 3D panoramic shooting method for simulation reality meta-universe platform |
| CN115147520A (en) * | 2022-06-07 | 2022-10-04 | 聚好看科技股份有限公司 | Method and equipment for driving virtual character based on visual semantics |
| CN115866354A (en) * | 2022-11-25 | 2023-03-28 | 广州美术学院 | Interactive virtual reality-based non-material heritage iconic deduction method and device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107393018A (en) * | 2017-07-27 | 2017-11-24 | 北京中达金桥技术股份有限公司 | A kind of method that the superposition of real-time virtual image is realized using Kinect |
| CN108830940A (en) * | 2018-06-19 | 2018-11-16 | 广东虚拟现实科技有限公司 | Hiding relation processing method, device, terminal device and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11373272B2 (en) * | 2015-06-05 | 2022-06-28 | MindAptiv, LLC | Digital gradient signal processing system and method for signals comprising at least three dimensions |
-
2021
- 2021-04-16 CN CN202110414359.3A patent/CN113112612B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107393018A (en) * | 2017-07-27 | 2017-11-24 | 北京中达金桥技术股份有限公司 | A kind of method that the superposition of real-time virtual image is realized using Kinect |
| CN108830940A (en) * | 2018-06-19 | 2018-11-16 | 广东虚拟现实科技有限公司 | Hiding relation processing method, device, terminal device and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| 混合硬件跟踪定位的空间信息虚实配准;武雪玲;任福;杜清运;;地理与地理信息科学(03);全文 * |
| 虚拟演播室系统在传统教学演播室中的应用;许建平;;中国现代教育装备(09);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113112612A (en) | 2021-07-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113112612B (en) | Positioning method and system for dynamic superposition of real person and mixed reality | |
| US11238644B2 (en) | Image processing method and apparatus, storage medium, and computer device | |
| US10096157B2 (en) | Generation of three-dimensional imagery from a two-dimensional image using a depth map | |
| US8655152B2 (en) | Method and system of presenting foreign films in a native language | |
| US10621777B2 (en) | Synthesis of composite images having virtual backgrounds | |
| CN111371966B (en) | Method, device and storage medium for synthesizing foreground character shadow in virtual studio | |
| CN105191287A (en) | Method of replacing objects in a video stream and computer program | |
| CN109598796A (en) | Real scene is subjected to the method and apparatus that 3D merges display with dummy object | |
| CN112954292B (en) | Digital museum navigation system and method based on augmented reality | |
| CN108833877B (en) | Image processing method and device, computer device and readable storage medium | |
| CN113132707A (en) | Method and system for dynamically superposing character and virtual decoration environment in real time | |
| TW202240530A (en) | Neural blending for novel view synthesis | |
| US9897806B2 (en) | Generation of three-dimensional imagery to supplement existing content | |
| Wang et al. | People as scene probes | |
| US11961190B2 (en) | Content distribution system, content distribution method, and content distribution program | |
| Kim et al. | 3-d virtual studio for natural inter-“acting” | |
| Mori et al. | An overview of augmented visualization: observing the real world as desired | |
| Iwadate et al. | VRML animation from multi-view images | |
| Lee et al. | Real-time 3D video avatar in mixed reality: An implementation for immersive telecommunication | |
| Huang et al. | 360° stereo image composition with depth adaption | |
| KR101399633B1 (en) | Method and apparatus of composing videos | |
| EP3716217A1 (en) | Techniques for detection of real-time occlusion | |
| EP4303817A1 (en) | A method and an apparatus for 360-degree immersive video | |
| US11295531B1 (en) | System and method for generating interactive virtual image frames in an augmented reality presentation | |
| Canessa et al. | Morpholo: a hologram generator algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |