CN111986296B - CG animation synthesis method for bullet time - Google Patents
CG animation synthesis method for bullet time Download PDFInfo
- Publication number
- CN111986296B CN111986296B CN202010843271.9A CN202010843271A CN111986296B CN 111986296 B CN111986296 B CN 111986296B CN 202010843271 A CN202010843271 A CN 202010843271A CN 111986296 B CN111986296 B CN 111986296B
- Authority
- CN
- China
- Prior art keywords
- plane
- model
- dimensional
- image
- coordinates
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001308 synthesis method Methods 0.000 title claims abstract description 10
- 238000000034 method Methods 0.000 claims abstract description 22
- 238000009877 rendering Methods 0.000 claims abstract description 22
- 230000000007 visual effect Effects 0.000 claims abstract description 19
- 230000015572 biosynthetic process Effects 0.000 claims abstract description 11
- 238000003786 synthesis reaction Methods 0.000 claims abstract description 11
- 239000011159 matrix material Substances 0.000 claims description 39
- 239000013598 vector Substances 0.000 claims description 36
- 238000013519 translation Methods 0.000 claims description 18
- 238000001514 detection method Methods 0.000 claims description 16
- 239000012634 fragment Substances 0.000 claims description 9
- 238000003780 insertion Methods 0.000 claims description 6
- 230000037431 insertion Effects 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 4
- 230000009466 transformation Effects 0.000 claims description 4
- 238000013178 mathematical model Methods 0.000 claims description 3
- 230000002194 synthesizing effect Effects 0.000 claims description 3
- 238000006073 displacement reaction Methods 0.000 claims description 2
- 230000003190 augmentative effect Effects 0.000 abstract description 4
- 230000009977 dual effect Effects 0.000 abstract description 3
- 230000000694 effects Effects 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 5
- 230000003068 static effect Effects 0.000 description 4
- 239000000203 mixture Substances 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000005728 strengthening Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/20—3D [Three Dimensional] animation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/005—General purpose rendering architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/50—Lighting effects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/757—Matching configurations of points or features
Abstract
The invention discloses a CG animation synthesis method for bullet time, which comprises the following steps: s1: detecting a designated plane, carrying out three-dimensional reconstruction of a scene in the step S1, acquiring a sparse three-dimensional point cloud of the scene, searching a plane in the sparse three-dimensional point cloud, and selecting the designated plane; s2: and (3) aligning and rendering the model, namely designating the position of the model placed on a plane, wherein the position is selected on an image with a certain visual angle by a user through a UI, calculating the corresponding three-dimensional point coordinates on the ground plane by the position, and finally aligning the model with the scene to finish the model rendering and picture synthesis. The invention relates to a method for combining three-dimensional CG element and real scene under multi-view angle by applying motion matching in bullet time, which generates high-precision CG synthetic animation in real time. The method can obtain bullet time and dual visual experience brought by augmented reality in the video.
Description
Technical Field
The invention relates to the field of three-dimensional reconstruction and image processing of computer vision and computer graphics, in particular to a CG animation synthesis method for bullet time.
Background
Bullet time (Bullet time) is used as a computer-aided photography special effect technology and is often applied to televisions, advertisements or computer games so as to realize the visual effects of strengthening slow shots, time stills and the like, and enable spectators or users to obtain brand new experience. Bullet times can be categorized into static bullet times and dynamic bullet times. In static bullet time, the camera system photographs the object from different angles and presents the picture at the same time to achieve the effect of time-setting. In the dynamic bullet time, the camera system presents pictures with different angles at different moments, and can watch dynamic live-action pictures from multiple angles.
Meanwhile, with the development of stereoscopic vision technology and the gradual popularization of virtual reality/augmented reality (VR/AR) technology, how to insert three-dimensional CG elements into shooting scenes has become a hotspot and difficulty of commercial applications.
Therefore, those skilled in the art have been dedicated to developing a CG animation synthesis method for bullet time, and the present invention relates to the fields of three-dimensional reconstruction, image editing, and graphic rendering for computer vision and computer graphics, and is a process and a method capable of performing motion matching in video bullet time to achieve the combination of three-dimensional CG (computer generated) elements and live-action.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and aims to combine motion matching and bullet time in a simple and effective way so as to realize the insertion and picture synthesis of three-dimensional CG elements in real scenes under multiple visual angles.
In order to achieve the above object, the present invention provides a CG animation synthesis method for bullet time, comprising the steps of:
S1: detecting a designated plane, carrying out three-dimensional reconstruction of a live-action scene, obtaining a sparse three-dimensional point cloud of the scene, searching a plane in the sparse three-dimensional point cloud, and selecting the designated plane;
S2: and (3) aligning and rendering the model, namely designating the position where the model to be inserted is placed on a plane, wherein the position is selected on an image with a certain visual angle by a user through a UI, calculating the corresponding three-dimensional point coordinates on the plane by the position, and finally aligning the model with the scene to finish the model rendering and picture synthesis.
Further, the step S1 further includes:
s11, algorithm input: synchronously acquiring a group of images from a multi-view camera system, wherein each camera only needs to acquire one frame of picture; calibrating the camera by using the group of images to obtain internal and external parameters of the camera; the internal and external parameters of the camera and the group of images are used as the input of a plane detection algorithm;
s12, feature matching: performing feature matching on the group of images to obtain matching feature points of two adjacent images;
S13, triangulation: performing triangulation by using the internal and external parameters of the camera and the corresponding image coordinates of the characteristic points, and calculating the three-dimensional coordinates of the characteristic points in the scene to obtain sparse three-dimensional point clouds of the scene;
S14, plane fitting: fitting all planes existing in the three-dimensional point cloud by utilizing Ransac algorithm;
s15, selecting a designated plane: selecting an image anchor point from the images with the appointed visual angles; and determining a parameter equation of the appointed plane according to the anchor point.
Further, the step S2 further includes:
s21, algorithm input: collecting a video sequence needing to be inserted with CG elements by utilizing a multi-camera system; the camera internal and external parameters calibrated in the step S11; the plane equation fitted in the step S15; designating an image anchor point to be inserted with CG element; a three-dimensional CG model sequence and a three-dimensional anchor point of the CG model;
s22, calculating three-dimensional coordinates of the anchor points inserted into the CG elements: calculating three-dimensional point coordinates of a model to be placed according to the internal and external parameters of the camera and the plane equation;
S23, aligning the model with a plane: calculating a rotation matrix and a translation vector required by the alignment of the model and the ground surface according to the model anchor point, the model normal vector, the plane normal vector and the three-dimensional position where the model is placed; setting a transformation matrix of which the size and the orientation need to be adjusted;
S24, model rendering: rendering an image under a bullet time appointed view angle by using OpenGL according to the camera internal and external parameters, the model rotation matrix and displacement vector, model vertex coordinates, texture coordinates and texture images;
s25, image synthesis: and synthesizing the OpenGL rendered image and the corresponding live-action image to generate CG synthesized animation under the bullet time appointed visual angle.
Further, the step S12 further includes:
S121, feature detection: performing feature detection on the image, and detecting image feature points and descriptors by using AKAZE detection algorithm;
s122, feature matching: carrying out violent matching on the characteristics of two adjacent images, and determining the matched characteristic points of the two images;
S123, interior point detection: the feature after violent matching possibly has an outer point, and error matching is removed by using a Cross-Ratio-Check method, namely feature descriptors in the two images are subjected to violent matching once, and matching meeting a Ratio threshold in both sets of matching results is taken as an inner point. Further, the Ransac algorithm in the step S14 is:
Estimating a mathematical model of a plane from a group of point clouds containing outer points in an iterative mode; in the model, after each iteration finds a plane, removing the inner points belonging to the plane from the point cloud; performing the Ransac algorithm iteration again on the rest point clouds, and searching a plane; and stopping executing the Ransac algorithm until the number of points in the residual point cloud is smaller than a threshold value, and obtaining all possible plane sets in the three-dimensional point cloud.
Further, the step S15 further includes:
S151: selecting an image anchor point: selecting a designated view angle picture, drawing the detected characteristic points in the view angle picture, and picking up N characteristic points on a certain plane; after the characteristic points are picked up, the three-dimensional coordinates of the characteristic points can be determined;
S152: ground plane determination: when the ground plane characteristic points are determined, the corresponding three-dimensional coordinates are brought into all plane equations, and the loss is calculated according to the following formula: Where N is the number of anchor points picked up, and (a, B, C, D) is a parameter of a plane in the set S, and a plane with the smallest loss is selected as the designated plane.
Further, in the step S21:
The plane equation is the plane determined in the step S15, and the equation is a general parameter equation a·x+b·y+c·z+d=0 of the plane, where the plane unit normal vector n= (a, B, C);
the image anchor point is the position of the model, and a user selects an image coordinate point to be inserted into the model in a specified visual angle image through a UI; when the model insertion position needs to be changed, the coordinates of the image points are picked up again;
The three-dimensional model sequence displays continuous motion information of the model; using a model file in an OBJ format, and converting the model file and the texture map into a binary file in a special format; the model is required to be placed on an XY plane of a coordinate axis and is forward along a Z axis; the three-dimensional model anchor point is used for selecting and modeling the position where the first model in the model sequence needs to be aligned.
Further, the calculation formula of the three-dimensional coordinate is shown as follows, wherein R, K, (u, v), t, (A, B, C, D) are known parameters:
(x, y, z) is the CG element anchor three-dimensional coordinates to be calculated in step S22, K is the camera internal reference, R c is the camera rotation matrix, t c is the camera translation vector, (u, v) anchor image plane coordinates, and (a, B, C, D) is the plane equation parameters.
Further, the rotation matrix and the translation vector need to have a unit normal vector of a scene plane of n p = (a, B, C), a plane normal vector of a model of n m = (0, 1), a plane anchor point P p＝(xp,yp,zp), and a model anchor point of P m＝(xm,ym,zm);
The rotation matrix is The translation vector is t= -r·p m+pp, where c=n p·nm,v＝nm×np,[v]× is an antisymmetric matrix of v;
a scaling matrix s and a rotation matrix R z about the z-axis of the model, wherein Wherein (s x,s y,s z) is a scaling factor of the model on the xyz axis, and α is a rotation angle; when the model and plane are aligned, the total rotation matrix/>Translation vector/>
Further, the OpenGL rendering is an OpenGL standard flow, and vertex positions and texture coordinates of the model are sent to a vertex shader through a vertex buffer object, and a texture map is bound to a texture object; the vertex position, the normal vector and the texture coordinates are transmitted to the fragment shader through the vertex shader, and the texture object adds the texture to the fragment shader through the sampler; the fragment shader outputs the color of the triangular patches of the three-dimensional model.
The invention relates to a method for combining three-dimensional CG element and real scene under multi-view angle by applying motion matching in bullet time, which generates high-precision CG synthetic animation in real time.
Motion matching (MatchMoving) is used as a technique for tracking camera trajectories, which can be used to insert CG elements into a real shot and to ensure that the element is in the correct position, scale, orientation and relative motion with respect to other objects in the real shot. The motion matching technology and the bullet time technology are combined, visual dual gain can be obtained, visual special effects brought by bullet time can be experienced, and a virtual model (such as a dynamic character or a static object) can be implanted into a real scene to be combined with the real scene, so that the effect of augmented reality is achieved.
The method can obtain bullet time and dual visual experience brought by augmented reality in the video. The invention has the specific characteristics that:
(1) The operation is simple, and the manual intervention is less;
(2) Supporting three-dimensional reconstruction of a scene;
(3) Supporting alignment of the three-dimensional model and the reconstructed scene, namely correctly placing the model at a specified position of the scene;
(4) Supporting the adjustment of the size and orientation of the interactive model so as to meet the personalized requirements;
(5) Support static or dynamic model sequences;
(6) High-precision CG animation synthesis under multiple visual angles is supported;
(7) And the real-time playing requirement of high resolution is met.
The conception, specific structure, and technical effects of the present invention will be further described with reference to the accompanying drawings to fully understand the objects, features, and effects of the present invention.
Drawings
FIG. 1 is a flow chart of a method of a preferred embodiment of the present invention;
FIG. 2 is a schematic representation of a bullet time camera array of a preferred embodiment of the present invention;
FIG. 3 is a schematic view of image anchor picking up according to a preferred embodiment of the present invention;
Fig. 4 is a schematic diagram of CG synthetic animation for each view angle according to a preferred embodiment of the present invention.
Detailed Description
The following description of the preferred embodiments of the present invention refers to the accompanying drawings, which make the technical contents thereof more clear and easy to understand. The present invention may be embodied in many different forms of embodiments and the scope of the present invention is not limited to only the embodiments described herein.
In the drawings, like structural elements are referred to by like reference numerals and components having similar structure or function are referred to by like reference numerals. The dimensions and thickness of each component shown in the drawings are arbitrarily shown, and the present invention is not limited to the dimensions and thickness of each component. The thickness of the components is exaggerated in some places in the drawings for clarity of illustration.
As shown in fig. 1, the method of the present invention comprises two stages, each of which comprises the following steps:
Stage 1: designated plane detection
At this stage, a three-dimensional reconstruction of the scene is performed, a sparse three-dimensional point cloud of the scene is obtained, planes possibly present are searched in the point cloud, and a specified plane (typically a ground plane) is selected.
1. Algorithm input: synchronously acquiring a group of color (RGB) images from a multi-view camera system, wherein each camera only needs to acquire one frame of picture; calibrating the camera by using the group of images to obtain internal and external parameters of the camera; the internal and external parameters of the camera and the group of images are used as the input of a plane detection algorithm;
2. Feature matching: performing feature matching on the group of images to obtain matching feature points of two adjacent images;
3. Triangulation: performing triangulation by using the internal and external parameters of the camera and the corresponding image coordinates of the characteristic points, and calculating the three-dimensional coordinates of the characteristic points in the scene to obtain sparse three-dimensional point clouds of the scene;
4. Plane fitting: fitting all planes existing in the three-dimensional point cloud by utilizing Ransac algorithm
5. Selecting a designated plane (ground plane): selecting an image anchor point from the images with the appointed visual angles by a user; from the anchor point, a parametric equation specifying the plane is determined.
Stage 2: model alignment and rendering
1. Algorithm input: collecting a video sequence needing to be inserted with CG elements by utilizing a multi-camera system; the camera internal and external parameters calibrated in the stage 1; the (ground) plane equation fitted in stage 1; designating an image anchor point to be inserted with CG element; a three-dimensional CG model sequence and a three-dimensional anchor point of the CG model;
2. calculating three-dimensional coordinates of an anchor point inserted with the CG element: calculating three-dimensional point coordinates of a model to be placed according to the internal and external parameters of the camera and a plane equation;
3. The mold is aligned with the plane: calculating a rotation matrix and a translation vector required by the alignment of the model and the ground surface according to the model anchor point, the model normal vector, the plane normal vector and the three-dimensional position where the model is placed; in addition, a transformation matrix which needs to be adjusted in size and orientation of the model is set;
4. Model rendering: rendering an image under a bullet time appointed view angle by using OpenGL according to the camera internal parameters, the camera external parameters, the model transformation matrix (rotation matrix and translation vector), the model vertex coordinates, the texture coordinates and the texture image;
5. And (3) image synthesis: and synthesizing the OpenGL rendered image and the corresponding live-action image (only considering the non-shielding condition), and generating the CG synthetic animation under the bullet time appointed visual angle.
Example 1
The patent realizes the insertion and synthesis of three-dimensional CG element in bullet time based on motion matching; performing camera internal and external parameter calibration by utilizing RGB images synchronously collected from a multi-camera system, performing three-dimensional reconstruction on a scene, calculating a three-dimensional point cloud and a plane equation set of the scene, and selecting a (ground) plane equation; and transforming the CG three-dimensional model according to the image position specified by the user, and performing rendering synthesis. The method mainly comprises two parts of (ground) plane determination and model rendering.
Stage one: (ground) plane determination
The stage is a preprocessing stage, and after a bullet time camera system is built, a group of data is required to be synchronously acquired for a scene in advance and used for determining a parameter equation of a designated plane (ground plane); in the case where the same scene, the camera pose, remains unchanged, this phase is performed only once and only once.
1. Construction, calibration and data acquisition of a multi-camera system
Building a multi-camera system: the method has no special requirement on the model of the camera, the number of the cameras can be selected according to the scene size or the actual requirement of bullet time, and a group of RGB images are synchronously acquired from different angles by using the built cameras; the system built by using 40 IoI Industry cameras in the patent is used for data acquisition. Fig. 2 shows a common camera array pattern for bullet time.
Calibration (calibration) of a multi-camera system: calibration, i.e. the calculation of the internal parameters (INTRINSIC PARAMETERS) and the external parameters (extrinsic parameters) of the camera. The camera parameters used in this patent are calibrated by the AGI software.
2. Feature matching
And (3) feature detection: performing feature detection on an input image, and selecting SIFT, SURF, ORB, AKAZE algorithms; the AKAZE detection algorithm is used in the patent, and feature points and descriptors are detected on the image;
Feature matching: violent matching (Brute Force Matching) is carried out on the characteristics of two adjacent images, and the matched characteristic points of the two images are determined
Interior point detection: the violently matched features may have outliers (Outlier); the patent uses a Cross-Ratio-Check method to remove the error matching, namely, feature descriptors in two images are respectively used as QueryDescriptor and TrainDescriptor for carrying out one-time violent matching, and the matching meeting the Ratio threshold value in the two sets of matching results is used as an inner point (Inlier)
3. Triangulation
Triangulation: according to the internal and external parameters of the camera and the matched coordinates of the characteristic points, the corresponding three-dimensional coordinates of the points can be determined; and calculating the characteristic three-dimensional points of all the images to form a sparse three-dimensional point cloud of the scene.
4. Plane fitting
Random sample consensus algorithm (Ransac): from a set of point clouds containing outliers, a mathematical model of the plane is estimated in an iterative manner. In order to find all existing planes, after each iteration finds a plane, the interior points belonging to the plane are removed from the point cloud; performing Ransac iteration again on the residual point cloud, and searching a plane; until the number of points in the remaining point cloud is less than a certain threshold (e.g., 10), execution of the Ransac algorithm is stopped. In this step, a set S of planes of all possible existence in the three-dimensional point cloud can be obtained.
5. Ground plane selection
Selecting an image anchor point: in the step, a designated view angle picture is selected, the detected characteristic points are drawn in the view angle picture, and N characteristic points on a certain plane are picked up; this process is done by the user through the UI (typically three points are selectable); after the feature point is picked up, the three-dimensional coordinates of the feature point (the triangulation result in step 3) can be determined simultaneously
Ground plane determination: when the ground plane characteristic points are determined, the corresponding three-dimensional coordinates are brought into all plane equations, and the loss is calculated according to the following formula: where N is the number of anchor points picked up and (A, B, C, D) is a parameter of the plane in set S. The plane with the smallest loss is selected as the designated (ground) plane.
Stage two: model alignment and rendering
In stage one, a parametric equation specifying a (ground) plane can be determined using feature matching, plane fitting, and user interaction; this process may be completed in a pre-processing stage. In the second stage, a position where the model is placed on the plane needs to be designated, the position is selected by a user on an image of a certain visual angle through the UI, and corresponding three-dimensional point coordinates on the plane are calculated according to the position; finally, the alignment of the model and the scene is carried out, and the model rendering and the picture synthesis work are carried out.
1. Data preparation
Plane equation: the plane equation is the (ground) plane determined in the stage one, and the equation is the general parameter equation a·x+b·y+c·z+d=0 of the plane, wherein the plane unit normal vector n= (a, B, C).
Picking up the model position: the user selects an image coordinate point to be inserted into the model through the UI in the specified view angle image (see fig. 3); when the model insertion position needs to be changed, the coordinates of the image points are picked up again;
three-dimensional model sequence: the sequence shows continuous motion information of the model; each model includes model vertex three-dimensional coordinates, model vertex texture coordinates, triangle patch indices, texture maps, and the like. In the method, an OBJ format model file is used, and the model file and the texture map are converted into a special format binary file, so that the purpose of quickly loading the model is realized. The model is placed on the XY plane of the coordinate axes and is forward along the Z axis.
Three-dimensional model anchor points: selecting a position to be aligned of a first model in a model sequence; for example, for a character model, its anchor point may be a three-dimensional point between two feet.
2. Computing model insertion points
Under the condition that the internal and external parameters of the camera, the plane equation and the coordinates of the anchor point image are known, the corresponding three-dimensional coordinates of the point on the plane can be calculated. The calculation formula is shown as follows, wherein R, K, (u, v), t, (A, B, C, D) are known parameters:
(x, y, z) is the CG element anchor three-dimensional coordinates to be calculated, K is the camera internal reference, R c is the camera rotation matrix, t c is the camera translation vector, (u, v) anchor image plane coordinates, (a, B, C, D) is the plane equation parameters.
3. Alignment of the mould with the plane
In this step, the rotation and translation matrices required for the alignment of the three-dimensional model to the scene need to be calculated, which matrices are applicable to the entire model sequence. Calculating the rotation and translation matrix requires a unit normal vector n p = (a, B, C) of the scene plane, a plane normal vector n m = (0, 1) of the model location, a plane anchor point P p＝(xp,yp,zp), and a model anchor point: p m＝(xm,ym,zm)
Rotation matrix for three-dimensional model and scene alignmentThe translation vector t= -r·p m+pp, where c=n p·nm,v＝nm×np,[v]× is an antisymmetric matrix of v.
To facilitate the adjustment of the model by the user, a scaling matrix s and a rotation matrix R z around the z-axis of the model are required to be set, whereinWherein (s x,sy,sz) is a scaling factor of the model on the xyz axis, and alpha is a rotation angle, which are all adjusted by a user according to actual requirements through a UI.
For model and plane alignment, the total rotation matrix thereofTranslation vector/>
This step is debugged by the user before the final composition of the CG animation, and after the appropriate dimensions and orientations of the model are obtained, it is not necessary to continue to perform this step.
4. Model rendering
The OpenGL rendering process is an OpenGL standard flow. Vertex positions and texture coordinates of the three-dimensional model are sent to a Vertex Shader (Vertex Shader) through a Vertex buffer object (Vertex Buffer Objects, VBO), while the texture map is bound to the texture object through gl_ TEXTURE _2d; vertex positions, normal vectors and texture coordinates are transmitted to the fragment shader through the vertex shader, and texture objects add textures to the fragment shader through a Sampler (Sampler); finally, the fragment shader outputs the color of the triangular patches of the three-dimensional model.
Matrix setting: the rotation matrix and the translation vector in the Model alignment step are used as a Model matrix (Model matrix), camera external parameters are converted into a View matrix (View matrix), and an OpenGL projection matrix (Projection matrix) is calculated by camera internal parameters.
5. Image composition
At present, the method only considers the situation that the CG element and the picture foreground are not blocked, so that when the image is synthesized, only the corresponding pixels in the live-action are replaced by the pixels of the rendered image. Fig. 4 is a comparison of a live view with a composite animation.
The foregoing describes in detail preferred embodiments of the present invention. It should be understood that numerous modifications and variations can be made in accordance with the concepts of the invention without requiring creative effort by one of ordinary skill in the art. Therefore, all technical solutions which can be obtained by logic analysis, reasoning or limited experiments based on the prior art by the person skilled in the art according to the inventive concept shall be within the scope of protection defined by the claims.
Claims (4)
1. A CG animation synthesis method for bullet time, comprising the steps of:
s1: detecting a designated plane, carrying out three-dimensional reconstruction of a live-action scene, obtaining a sparse three-dimensional point cloud of the scene, searching the plane in the sparse three-dimensional point cloud, and selecting the designated plane, wherein the method specifically comprises the following steps of:
s11, algorithm input: synchronously acquiring a group of images from a multi-view camera system, wherein each camera only needs to acquire one frame of picture; calibrating the camera by using the group of images to obtain internal and external parameters of the camera; the internal and external parameters of the camera and the group of images are used as the input of a plane detection algorithm;
s12, feature matching: performing feature matching on the group of images to obtain matching feature points of two adjacent images;
S13, triangulation: performing triangulation by using the internal and external parameters of the camera and the corresponding image coordinates of the characteristic points, and calculating the three-dimensional coordinates of the characteristic points in the scene to obtain sparse three-dimensional point clouds of the scene;
S14, plane fitting: fitting all planes existing in the three-dimensional point cloud by utilizing Ransac algorithm;
S15, selecting a designated plane: selecting an image anchor point from the images with the appointed visual angles; selecting an image anchor point: selecting a designated view angle picture, drawing the detected characteristic points in the view angle picture, and picking up N characteristic points on a certain plane; after the characteristic points are picked up, the three-dimensional coordinates of the characteristic points can be determined;
Determining a parameter equation of a designated plane according to the anchor point; ground plane determination: when the ground plane characteristic points are determined, the corresponding three-dimensional coordinates are brought into all plane equations, and the loss is calculated according to the following formula: n is the number of picked anchor points, A, B, C and D are parameters of a plane equation, and a plane with the minimum loss is selected as a designated plane;
S2: the method comprises the steps of aligning and rendering a model, designating a position where the model to be inserted is placed on a plane, selecting the position on an image with a certain visual angle by a user through a UI, calculating corresponding three-dimensional point coordinates on the plane by the position, and finally aligning the model with a scene to finish the model rendering and picture synthesis, wherein the method comprises the following steps of:
S21, algorithm input: collecting a video sequence needing to be inserted with CG elements by utilizing a multi-camera system; the camera internal and external parameters calibrated in the step S11; the plane equation fitted in the step S15; designating an image anchor point to be inserted with CG element; a three-dimensional CG model sequence and a three-dimensional anchor point of the CG model;
The plane equation is the plane determined in the step S15, and the equation is a general parameter equation a·x+b·y+c·z+d=0 of the plane, where the plane unit normal vector n= (a, B, C);
the image anchor point is the position of the model, and a user selects an image coordinate point to be inserted into the model in a specified visual angle image through a UI; when the model insertion position needs to be changed, the coordinates of the image points are picked up again;
the three-dimensional model sequence displays continuous motion information of the model; using a model file in an OBJ format, and converting the model file and the texture map into a binary file in a special format; the model is required to be placed on an XY plane of a coordinate axis and is forward along a Z axis; the anchor point of the three-dimensional model is used for selecting a position which is required to be aligned by a first model in the model sequence;
S22, calculating three-dimensional coordinates of the anchor points inserted with the CG elements: calculating three-dimensional point coordinates of a model to be placed according to the internal and external parameters of the camera and a plane equation; the calculation formula of the three-dimensional coordinates is shown as follows, wherein R c,K,(u,v),tc, A, B, C and D are known parameters:
(x, y, z) is the CG element anchor three-dimensional coordinates to be calculated, K is the camera internal reference, R c is the camera rotation matrix, t c is the camera translation vector, (u, v) is the anchor image plane coordinates, a, B, C, D is the parameters of the plane equation;
the rotation matrix and translation vector need the unit normal vector of the scene plane to be n p = (A, B, C), the normal vector of the plane where the model is located to be n m = (0, 1), and the plane anchor point P p＝(xp,yp,zp), and the model anchor point to be P m＝(xm,ym,zm);
The rotation matrix is The translation vector is t= -r·p m+pp, where c=n p·nm,v＝nm×np,[v]× is an antisymmetric matrix of v;
a scaling matrix s and a rotation matrix R z about the z-axis of the model, wherein Wherein (s x,sy,sz) is a scaling factor of the model on the xyz axis, and α is a rotation angle; when the model and plane are aligned, the total rotation matrix/>Translation vector/>
S23, aligning the model with a plane: calculating a rotation matrix and a translation vector required by the alignment of the model and the ground surface according to the model anchor point, the model normal vector, the plane normal vector and the three-dimensional position where the model is placed; setting a transformation matrix of which the size and the orientation need to be adjusted;
S24, model rendering: rendering an image under a bullet time appointed view angle by using OpenGL according to the camera internal and external parameters, the model rotation matrix and displacement vector, model vertex coordinates, texture coordinates and texture images;
s25, image synthesis: and synthesizing the OpenGL rendered image and the corresponding live-action image to generate CG synthesized animation under the bullet time appointed visual angle.
2. The CG animation synthesis method for bullet time as claimed in claim 1 wherein said step S12 further comprises:
S121, feature detection: performing feature detection on the image, and detecting image feature points and descriptors by using AKAZE detection algorithm;
s122, feature matching: carrying out violent matching on the characteristics of two adjacent images, and determining the matched characteristic points of the two images;
S123, interior point detection: the feature after violent matching possibly has an outer point, and error matching is removed by using a Cross-Ratio-Check method, namely feature descriptors in the two images are subjected to violent matching once, and matching meeting a Ratio threshold in both sets of matching results is taken as an inner point.
3. The CG animation synthesis method for bullet time as claimed in claim 1 wherein the Ransac algorithm in step S14 is:
Estimating a mathematical model of a plane from a group of point clouds containing outer points in an iterative mode; in the model, after each iteration finds a plane, removing the inner points belonging to the plane from the point cloud; performing the Ransac algorithm iteration again on the rest point clouds, and searching a plane; and stopping executing the Ransac algorithm until the number of points in the residual point cloud is smaller than a threshold value, and obtaining all possible plane sets in the three-dimensional point cloud.
4. The CG animation synthesis method for bullet time of claim 1 wherein the OpenGL rendering is an OpenGL standard flow, the vertex positions and texture coordinates of the model are sent to a vertex shader through a vertex buffer object while a texture map is bound to a texture object; the vertex position, the normal vector and the texture coordinates are transmitted to the fragment shader through the vertex shader, and the texture object adds the texture to the fragment shader through the sampler; the fragment shader outputs the color of the triangular patches of the three-dimensional model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010843271.9A CN111986296B (en) | 2020-08-20 | 2020-08-20 | CG animation synthesis method for bullet time |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010843271.9A CN111986296B (en) | 2020-08-20 | 2020-08-20 | CG animation synthesis method for bullet time |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111986296A CN111986296A (en) | 2020-11-24 |
| CN111986296B true CN111986296B (en) | 2024-05-03 |
Family
ID=73443383
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010843271.9A Active CN111986296B (en) | 2020-08-20 | 2020-08-20 | CG animation synthesis method for bullet time |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111986296B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111986133B (en) * | 2020-08-20 | 2024-05-03 | 叠境数字科技（上海）有限公司 | Virtual advertisement implantation method applied to bullet time |
| CN112468728B (en) * | 2020-11-27 | 2022-09-30 | 湖南信达通信息技术有限公司 | Video synthesis method, video synthesis equipment and computer-readable storage medium |
| CN113763481B (en) * | 2021-08-16 | 2024-04-05 | 北京易航远智科技有限公司 | Multi-camera visual three-dimensional map construction and self-calibration method in mobile scene |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101908231A (en) * | 2010-07-27 | 2010-12-08 | 清华大学 | Reconstruction method and system for processing three-dimensional point cloud containing main plane scene |
| CN105205858A (en) * | 2015-09-18 | 2015-12-30 | 天津理工大学 | Indoor scene three-dimensional reconstruction method based on single depth vision sensor |
| CN105913489A (en) * | 2016-04-19 | 2016-08-31 | 东北大学 | Indoor three-dimensional scene reconstruction method employing plane characteristics |
| CN106920276A (en) * | 2017-02-23 | 2017-07-04 | 华中科技大学 | A kind of three-dimensional rebuilding method and system |
| CN107123161A (en) * | 2017-06-14 | 2017-09-01 | 西南交通大学 | A kind of the whole network three-dimensional rebuilding method of contact net zero based on NARF and FPFH |
| CN107610176A (en) * | 2017-09-15 | 2018-01-19 | 斯坦德机器人（深圳）有限公司 | A kind of pallet Dynamic Recognition based on Kinect and localization method, system and medium |
| WO2018019272A1 (en) * | 2016-07-29 | 2018-02-01 | 成都理想境界科技有限公司 | Method and apparatus for realizing augmented reality on the basis of plane detection |
| CN107918948A (en) * | 2017-11-02 | 2018-04-17 | 深圳市自由视像科技有限公司 | 4D Video Rendering methods |
| CN108632538A (en) * | 2018-05-22 | 2018-10-09 | 长沙全度影像科技有限公司 | The bullet time camera system and method that a kind of CG animations and camera array are combined |
| CN109427007A (en) * | 2018-09-17 | 2019-03-05 | 叠境数字科技（上海）有限公司 | Virtual fit method based on multi-angle of view |
| CN110766738A (en) * | 2019-05-08 | 2020-02-07 | 叠境数字科技（上海）有限公司 | Virtual shoe fitting method based on multi-view depth sensor |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8106924B2 (en) * | 2008-07-31 | 2012-01-31 | Stmicroelectronics S.R.L. | Method and system for video rendering, computer program product therefor |
-
2020
- 2020-08-20 CN CN202010843271.9A patent/CN111986296B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101908231A (en) * | 2010-07-27 | 2010-12-08 | 清华大学 | Reconstruction method and system for processing three-dimensional point cloud containing main plane scene |
| CN105205858A (en) * | 2015-09-18 | 2015-12-30 | 天津理工大学 | Indoor scene three-dimensional reconstruction method based on single depth vision sensor |
| CN105913489A (en) * | 2016-04-19 | 2016-08-31 | 东北大学 | Indoor three-dimensional scene reconstruction method employing plane characteristics |
| WO2018019272A1 (en) * | 2016-07-29 | 2018-02-01 | 成都理想境界科技有限公司 | Method and apparatus for realizing augmented reality on the basis of plane detection |
| CN106920276A (en) * | 2017-02-23 | 2017-07-04 | 华中科技大学 | A kind of three-dimensional rebuilding method and system |
| CN107123161A (en) * | 2017-06-14 | 2017-09-01 | 西南交通大学 | A kind of the whole network three-dimensional rebuilding method of contact net zero based on NARF and FPFH |
| CN107610176A (en) * | 2017-09-15 | 2018-01-19 | 斯坦德机器人（深圳）有限公司 | A kind of pallet Dynamic Recognition based on Kinect and localization method, system and medium |
| CN107918948A (en) * | 2017-11-02 | 2018-04-17 | 深圳市自由视像科技有限公司 | 4D Video Rendering methods |
| CN108632538A (en) * | 2018-05-22 | 2018-10-09 | 长沙全度影像科技有限公司 | The bullet time camera system and method that a kind of CG animations and camera array are combined |
| CN109427007A (en) * | 2018-09-17 | 2019-03-05 | 叠境数字科技（上海）有限公司 | Virtual fit method based on multi-angle of view |
| CN110766738A (en) * | 2019-05-08 | 2020-02-07 | 叠境数字科技（上海）有限公司 | Virtual shoe fitting method based on multi-view depth sensor |
Non-Patent Citations (3)
| Title |
|---|
| Xth Person View Video for Observation from Diverse Perspectives;Shimura, N;2018 JOINT 7TH INTERNATIONAL CONFERENCE ON INFORMATICS, ELECTRONICS & VISION (ICIEV) AND 2018 2ND INTERNATIONAL CONFERENCE ON IMAGING, VISION & PATTERN RECOGNITION (ICIVPR);39-43 * |
| 一种多模式融合的激光点云配准算法;彭蹦;杨耀权;江鹏宇;;激光与红外(第04期);全文 * |
| 基于虚拟现实技术的三维动画建模设计;谢娟;;现代电子技术(第16期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111986296A (en) | 2020-11-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111986296B (en) | CG animation synthesis method for bullet time | |
| Pagés et al. | Affordable content creation for free-viewpoint video and VR/AR applications | |
| CN111243071A (en) | Texture rendering method, system, chip, device and medium for real-time three-dimensional human body reconstruction | |
| Meilland et al. | A unified rolling shutter and motion blur model for 3D visual registration | |
| Maier et al. | Super-resolution keyframe fusion for 3D modeling with high-quality textures | |
| Zhang et al. | Personal photograph enhancement using internet photo collections | |
| Mori et al. | Efficient use of textured 3D model for pre-observation-based diminished reality | |
| CN111080776B (en) | Human body action three-dimensional data acquisition and reproduction processing method and system | |
| CN110458964B (en) | Real-time calculation method for dynamic illumination of real environment | |
| WO2022098606A1 (en) | Deferred neural rendering for view extrapolation | |
| Inamoto et al. | Intermediate view generation of soccer scene from multiple videos | |
| JP2004234350A (en) | Image processing device, image processing method, and image processing program | |
| Park et al. | Instant panoramic texture mapping with semantic object matching for large-scale urban scene reproduction | |
| Wang et al. | Neural opacity point cloud | |
| JP2001274973A (en) | Device and method for synthesizing microscopic image and computer-readable recording medium with microscopic image synthesizing processing program recorded thereon | |
| DuVall et al. | Compositing light field video using multiplane images | |
| CN113763544A (en) | Image determination method, image determination device, electronic equipment and computer-readable storage medium | |
| Nicolet et al. | Repurposing a relighting network for realistic compositions of captured scenes | |
| WO2023225891A1 (en) | Neural rendering method based on multi-resolution network structure | |
| CN115834800A (en) | Method for controlling shooting content synthesis algorithm | |
| CN111192308A (en) | Image processing method and device, electronic equipment and computer storage medium | |
| CN111986133B (en) | Virtual advertisement implantation method applied to bullet time | |
| JP5926626B2 (en) | Image processing apparatus, control method therefor, and program | |
| Inamoto et al. | Fly through view video generation of soccer scene | |
| CN111260544B (en) | Data processing method and device, electronic equipment and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |