CN111476716B - Real-time video stitching method and device - Google Patents
Real-time video stitching method and device Download PDFInfo
- Publication number
- CN111476716B CN111476716B CN202010260631.2A CN202010260631A CN111476716B CN 111476716 B CN111476716 B CN 111476716B CN 202010260631 A CN202010260631 A CN 202010260631A CN 111476716 B CN111476716 B CN 111476716B
- Authority
- CN
- China
- Prior art keywords
- video
- grid
- pixel point
- spliced
- grid model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4038—Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
- G06T17/205—Re-meshing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/44016—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving splicing one content stream with another content stream, e.g. for substituting a video clip
Abstract
A real-time video stitching method and device includes: intercepting a first video and a second video to be spliced, splicing a frame of picture corresponding to the first video and the second video, and acquiring a pixel point coordinate set of a picture splicing position; establishing a three-dimensional grid model, respectively importing the first video and the second video, and converting a pixel point coordinate set of a picture splicing position into a grid vertex coordinate set of the grid model; determining grids to be blanked, moving the grid model, removing the grids to be blanked, and completing splicing. According to the application, only one frame of picture is selected at will for splicing, and for the rest multi-frame pictures, only the transparency of the top points of the three-dimensional grid model is utilized to blank the public area of two sections of videos to be spliced, so that the two sections of real-time videos are spliced directly, the calculation amount and the resource consumption of a system are reduced, and the real-time performance of the videos can be better ensured.
Description
Technical Field
The application relates to the technical field of videos, in particular to a real-time video splicing method and device.
Background
With the development and popularization of network and multimedia technologies, more and more video and image application technologies are coming into the lives of people, wherein the video technologies occupy important components. In a three-dimensional scene, the access and processing of real-time video are research hot spots in the field in recent years, and in order to realize a wider visual range and solve the problems of serious picture deformation and the like caused by wide-angle equipment such as fish-eye lenses, the splicing technology of gun camera video is generated. The video stitching technology can well solve the problems, and has wide application in the fields of monitoring, fire protection, traffic, public security, medicine and the like.
The video stitching technology is developed on the basis of the picture stitching technology, each frame of picture of the video to be stitched is stitched by using the picture stitching technology, and the video is combined into a video by a common video stitching method. The method for splicing and recombining the pictures into the video is large in consumption of system resources and calculation amount, so that the real-time requirement of the video cannot be met.
Disclosure of Invention
The application provides a real-time video stitching method and device capable of reducing system resource consumption.
According to a first aspect of the present application, there is provided a real-time video stitching method comprising:
intercepting a first video and a second video to be spliced, splicing a frame of picture corresponding to the first video and the second video, and acquiring a pixel point coordinate set of a picture splicing position;
establishing a three-dimensional grid model, respectively importing the first video and the second video, and converting a pixel point coordinate set of a picture splicing position into a grid vertex coordinate set of the grid model;
determining grids to be blanked, removing the grids to be blanked by using the transparency of the vertexes of the three-dimensional grid model, and moving the three-dimensional grid model to finish splicing.
Further, the converting the pixel point coordinate set of the picture stitching position into the grid vertex coordinate set of the grid model includes:
and through coordinate mapping twice, the pixel point of the picture splicing position is represented by four vertex coordinates of the pixel point in the grid model.
Further, the representing, by two coordinate mappings, the pixel point of the image stitching location with four vertex coordinates of the pixel point in the grid model includes:
storing pixel points Px in the picture in a one-dimensional array P [ mn ] from left to right and from bottom to top, wherein the number of the pixel points Px is n, and m is the number of columns;
converting the pixel point Px into a point Pij in two-dimensional coordinates, wherein the position where the point Pij is located is represented as P [ x/m+1, x%m+1] through the two-dimensional coordinates;
positioning the pixel point Pij in the two-dimensional coordinates to the coordinate V of the pixel point corresponding to four vertexes of the grid on the grid model 1 [x/m+1,x％m+1，0]、V 2 [x/m+1,x％m，0]、V 3 [x/m,x％m，0]And V 4 [x/m,x％m+1，0]。
Further, the representing, by two coordinate mappings, the pixel point of the image stitching location with four vertex coordinates of the pixel point in the grid model, further includes:
adding an offset in the x-axis direction, wherein the offset is m;
the first video comprises a grid model to be spliced on the left side, and the coordinates of grid vertexes of pixel points to be spliced on the left side are V 11 [m,x％m+1，0]、V 12 [m,x％m，0]、V 13 [x/m,x％m，0]And V 14 [x/m,x％m+1，0]；
The second video is a grid model to be spliced on the right side, and coordinates of grid vertexes of pixel points to be spliced on the right side are obtained to be V 21 [x/m+m+1,x％m+1，0]、V 22 [x/m+1+m,x％m，0]、V 23 [m,x％m，0]And V 24 [m,x％m+1，0]。
Further, the determining the grid to be blanked, removing the grid to be blanked by using the transparency of the vertices of the three-dimensional grid model, and moving the three-dimensional grid model to complete the stitching includes:
the corresponding grid to be blanked is set as V 13 [x/m,x％m，0]And V 11 [m,x％m+1，0]A defined grid;
the corresponding grid to be blanked is set as V 23 [m,x％m，0]And V 21 [x/m+m+1,x％m+1，0]A defined grid;
the transparency of the top points of the blanking grids in the pixel point coordinate sets to be spliced on the left side and the right side are set to be 0 respectively;
and translating the grid model to be spliced on the right side leftwards by a translation amount which is the difference value of the horizontal coordinates of the pixel point to be spliced on the right side and the pixel point to be spliced on the left side corresponding to the pixel point to be spliced on the right side.
According to a second aspect of the present application, there is provided a real-time video stitching apparatus comprising:
the video processing module is used for intercepting a first video and a second video to be spliced, splicing a frame of picture corresponding to the first video and the second video, and acquiring a pixel point coordinate set of a picture splicing position;
the model building module is used for building a three-dimensional grid model, respectively importing the first video and the second video, and converting a pixel point coordinate set of the picture splicing position into a grid vertex coordinate set of the grid model;
and the video stitching module is used for determining grids to be blanked, removing the grids to be blanked by utilizing the transparency of the vertexes of the three-dimensional grid model, and moving the three-dimensional grid model to finish stitching.
Further, the model building module is further configured to represent, by two coordinate mappings, a pixel point of the picture stitching location with four vertex coordinates of the pixel point in the mesh model.
Further, the model building module includes:
the first conversion unit is used for storing pixel points Px in the picture in a one-dimensional array P [ mn ] from left to right and from bottom to top, wherein n rows are n rows, and m is a column number;
the second conversion unit is used for converting the pixel point Px of the picture splicing position into a point Pij in two-dimensional coordinates, and the position where the Pij is located is represented as P [ x/m+1, x%m+1] through the two-dimensional coordinates;
a third conversion unit for positioning the pixel Pij in the two-dimensional coordinates to the coordinate V of the pixel corresponding to four vertexes of the grid on the grid model 1 [x/m+1,x％m+1，0]、V 2 [x/m+1,x％m，0]、V 3 [x/m,x％m，0]And V 4 [x/m,x％m+1，0]；
The third conversion unit is further used for adding an offset in the x-axis direction, and the size of the offset is m; the first video comprises a grid model to be spliced on the left side, and the coordinates of grid vertexes of pixel points to be spliced on the left side are V 11 [m,x％m+1，0]、V 12 [m,x％m，0]、V 13 [x/m,x％m，0]And V 14 [x/m,x％m+1，0]The method comprises the steps of carrying out a first treatment on the surface of the The second video is a grid model to be spliced on the right side, and coordinates of grid vertexes of pixel points to be spliced on the right side are obtained to be V 21 [x/m+m+1,x％m+1，0]、V 22 [x/m+1+m,x％m，0]、V 23 [m,x％m，0]And V 24 [m,x％m+1，0]。
Further, the video stitching module includes:
a first processing unit, configured to set the point in the pixel point coordinate set to be spliced on the left side to be set as a grid to be blanked by V 13 [x/m,x％m，0]And V 11 [m,x％m+1，0]A defined grid;
the second processing unit is used for setting the corresponding grid to be blanked as V, wherein the grid to be blanked corresponds to the point in the pixel point coordinate set to be spliced on the right side 23 [m,x％m，0]And V 21 [x/m+m+1,x％m+1，0]A defined grid;
the third processing unit is used for setting the transparency of the top points of the blanking grids in the pixel point coordinate sets to be spliced on the left side and the right side to be 0 respectively;
and the splicing unit is used for translating the grid model to be spliced on the right side leftwards, wherein the translation amount is the difference value of the horizontal coordinates of the pixel point to be spliced on the right side and the pixel point to be spliced on the left side corresponding to the pixel point to be spliced.
According to a third aspect of the present application, there is provided a real-time video stitching apparatus comprising:
a memory for storing a program;
and the processor is used for executing the program stored in the memory to realize the method.
The method and the device for splicing the real-time video comprise the steps of firstly splicing a frame of picture corresponding to a first video and a second video, and obtaining a pixel point coordinate set of a picture splicing position; establishing a three-dimensional grid model, respectively importing a first video and a second video, and converting a pixel point coordinate set of a picture splicing position into a grid vertex coordinate set of the grid model; determining grids to be blanked, moving a grid model, removing the grids to be blanked, and completing splicing. According to the application, only one frame of picture is selected at will for splicing, and for the rest multi-frame pictures, only the transparency of the top points of the three-dimensional grid model is utilized to blank the public area of two sections of videos to be spliced, so that the two sections of real-time videos are spliced directly, the calculation amount and the resource consumption of a system are reduced, and the real-time performance of the videos can be better ensured.
Drawings
FIG. 1 is a flow chart of a method according to an embodiment of the application in one implementation;
fig. 2 is a schematic diagram of a pixel Px in a picture in a one-dimensional array according to a first embodiment of the present application;
FIG. 3 is a schematic diagram of the map of Px in FIG. 2 to pixel points Pij in a two-dimensional coordinate system;
fig. 4 shows the mapping of the pixel points Pij in fig. 3 to the pixel points P (v 1 ,v 2 ,v 3 ,v 4 ) Schematic of (2);
FIG. 5 is a schematic diagram of a blanking area of a three-dimensional mesh model according to a first embodiment of the present application;
FIG. 6 is a schematic view of three-dimensional mesh model stitching in accordance with a first embodiment of the present application;
FIG. 7 is a flow chart of a method according to an embodiment of the present application in another implementation;
FIG. 8 is a schematic diagram of a program module of an apparatus according to a second embodiment of the present application;
fig. 9 is a schematic diagram of a program module of an apparatus according to another embodiment of the application.
Detailed Description
The application will be described in further detail below with reference to the drawings by means of specific embodiments. This application may be embodied in many different forms and is not limited to the implementations described in this example. The following detailed description is provided to facilitate a more thorough understanding of the present disclosure, in which words of upper, lower, left, right, etc., indicating orientations are used solely for the illustrated structure in the corresponding figures.
However, one skilled in the relevant art will recognize that the detailed description of one or more of the specific details may be omitted, or that other methods, components, or materials may be used. In some instances, some embodiments are not described or described in detail.
The numbering of the components itself, e.g. "first", "second", etc., is used herein merely to distinguish between the described objects and does not have any sequential or technical meaning.
Furthermore, the features and aspects described herein may be combined in any suitable manner in one or more embodiments. It will be readily understood by those skilled in the art that the steps or order of operation of the methods associated with the embodiments provided herein may also be varied. Thus, any order in the figures and examples is for illustrative purposes only and does not imply that a certain order is required unless explicitly stated that a certain order is required.
The application can be used for splicing a plurality of gun camera videos with common pictures, wherein the gun camera positions are at the same horizontal height and the inclination angles are the same. If the pictures shot by the two gun cameras have the height difference, the pictures can be cut, and the common pictures can be spliced.
Embodiment one:
the embodiment of the application provides a real-time video stitching method, as shown in fig. 1, comprising the following steps:
step 102: and intercepting the first video and the second video to be spliced, splicing a frame of picture corresponding to the first video and the second video, and acquiring a pixel point coordinate set of a picture splicing position.
And intercepting a frame of picture Texture1 and a frame of picture Texture2 of the two real-time Video1 and Video2 which need to be spliced at the same time, and splicing by using a picture splicing algorithm, wherein the picture splicing algorithm can be determined according to actual conditions. Assuming that the sizes of two pictures are m×n (m, n represent the pixel sizes of the length and the width of the picture respectively), finally obtaining pixel Point coordinate sets Point1[ n ] and Point2[ n ] of the splicing positions on the two pictures.
Step 104: and establishing a three-dimensional grid model, respectively importing the first video and the second video, and converting a pixel point coordinate set of the picture splicing position into a grid vertex coordinate set of the grid model.
The coordinates of the picture pixels are stored in a one-dimensional array P [ mn ] from left to right and from bottom to top, each element Px in the array represents the color attribute value of a corresponding pixel in the picture, the size of the array is m x n, wherein m and n are natural numbers respectively, and x is {0,1,2, … … mn-1}, and the arrangement mode is shown in figure 2. Because of the two-dimensional characteristic of the picture, as shown in fig. 3, the position of a certain pixel Px in the picture can be expressed in the form of Pij, where i represents the number of rows where the pixel is located, and j represents the number of columns where the pixel is located, so Px can be expressed as P [ x/m+1, x%m+1].
Further, in order to facilitate calculation and more intuitively represent the relationship between the pixel image and the grid Model, two Plane models Model1 and Model2 having m unit grids in the horizontal direction and n unit grids in the vertical direction are respectively manufactured according to the image size. The model is manufacturedAnd then sequentially placing the two models at the original points of the three-dimensional coordinates, wherein the placing positions of the models are shown in fig. 4, and simultaneously, respectively accessing two real-time Video1 and Video2 to be spliced to the two models. In order to conveniently express the relation between the picture and the grid model, the number of unit grids of the established grid model is consistent with the size of pixels of the picture, so that the pixel point can be directly positioned to the corresponding grid vertex coordinate on the grid model according to Px, one pixel point in the picture is expressed as a square grid consisting of two triangles on the grid model, and the upper right corner three-dimensional coordinate P (V 1 ，V 2 ，V 3 ，V 4 ) Can be expressed as: v (V) 1 [x/m+1,x％m+1，0]The remaining three points in the clockwise direction are respectively expressed as: v (V) 2 [x/m+1,x％m，0]、V 3 [x/m,x％m，0]And V 4 [x/m,x％m+1，0]. Through the coordinate mapping twice, the pixel points in the left model surface picture in the scene can be represented by four vertex coordinates in the grid model. In this embodiment, left and right stitching is taken as an example to describe that the coordinates of the pixels in the left model surface picture can directly correspond to the coordinates of the three-dimensional scene, while the coordinates of the pixel points in the right model surface picture and the stitching points Point2[ n ] in the picture]It is necessary to add an offset in the x-axis direction, the offset being m, the resulting splice Point being Point 2' n]. Therefore, the vertex coordinates of the square grid corresponding to the pixel point Px of the right model surface picture are expressed as V 1 [x/m+m+1,x％m+1，0]、V 2 [x/m+1+m,x％m，0]、V 3 [m,x％m，0]And V 4 [m,x％m+1，0]。
Step 106: determining grids to be blanked, removing the grids to be blanked by using the transparency of the vertexes of the three-dimensional grid model, and moving the three-dimensional grid model to finish splicing.
Pixel Point coordinate set Point1[ n ] of splicing position available from two pictures]And Point2[ n ]]. As shown in fig. 5 and 6, for Point1[ n ]]The coordinates of the lower left corner of the square grid in the grid model corresponding to the point Px in (1) can be expressed as V [ x/m, x% m,0]At the same time because of Point1[ n ]]Splice points representing surface pictures of grid models positioned on leftCoordinates so that the blanking grid corresponding to the point is defined by V [ x/m, x% m,0]And V [ m, x% m+1,0]A defined grid. Point1[ n ]]The transparency of the defined vertex of the blanking grid corresponding to each pixel point is set to 0, namely, the common area of the coordinate model surface picture is set to be transparent. For Point 2' [ n ]]The coordinates of the upper right corner of the square grid in the corresponding point grid model can be expressed as V [ x/m+m+1, x%m+1,0]. At the same time because of Point 2' n]The coordinates of the stitching points representing the surface pictures of the grid model positioned on the right side, so that the blanking grid corresponding to the points is formed by V 23 [m,x％m，0]And V 21 [x/m+m+1,x％m+1，0]A defined grid. Point 2' n]And setting the transparency of the defined vertexes of the blanking grid corresponding to each pixel point to be 0, and setting the common area of the coordinate model surface picture to be transparent.
After the common pixel Point is eliminated, the right side model needs to be shifted leftwards to complete the splicing, and the shift size is Point 2' 0, x-Point1[0] x. The common part of the left and right patches is blanked, so that the splicing of two groups of videos is realized, and a three-dimensional grid splicing schematic diagram is shown in fig. 5.
As shown in fig. 7, another embodiment of the present application provides a real-time video stitching method, which includes the following steps:
step 702: and intercepting a frame of picture Texture1 and a frame of picture Texture2 of two segments of real-time Video1 and Video2 to be spliced at the same moment, and acquiring a pixel Point coordinate set Point1[ n ] and a pixel Point coordinate set Point2[ n ] of a picture splicing position.
The sizes of the two pictures are set to be m x n (m, n respectively represent the pixel sizes of the length and the width of the pictures), and finally, a pixel Point coordinate set (Point 1 n) and a pixel Point2 n) of the splicing position on the two pictures are obtained.
Step 704: and storing the pixel points Px in the picture in a one-dimensional array P [ mn ] from left to right and from bottom to top, wherein the number is n, and m is the number of columns.
Step 706: the pixel point Px is converted into a point Pij in two-dimensional coordinates, and the position where Pij is located is represented as P [ x/m+1, x%m+1] by the two-dimensional coordinates.
Step 708: establishing a three-dimensional grid model, and positioning a pixel Pij in two-dimensional coordinates to a coordinate V of four vertexes of a grid corresponding to the pixel Pij on the grid model 1 [x/m+1,x％m+1，0]、V 2 [x/m+1,x％m，0]、V 3 [x/m,x％m，0]And V 4 [x/m,x％m+1，0]。
And through coordinate mapping twice, the pixel point of the picture splicing position is represented by four vertex coordinates of the pixel point in the grid model.
Step 710: an offset is added in the x-axis direction, and the size of the offset is m.
Step 712: and acquiring coordinates of grid vertexes of the pixel points to be spliced of the Texture 1. Texture1 is a left grid model to be spliced, and the coordinates of grid vertices of pixel points to be spliced on the left side are V 11 [m,x％m+1，0]、V 12 [m,x％m，0]、V 13 [x/m,x％m，0]And V 14 [x/m,x％m+1，0]；
Step 714: and acquiring coordinates of grid vertexes of the pixel points to be spliced of the Texture 2. Texture2 is a grid model to be spliced on the right side, and coordinates of grid vertexes of pixel points to be spliced on the right side are obtained to be V 21 [x/m+m+1,x％m+1，0]、V 22 [x/m+1+m,x％m，0]、V 23 [m,x％m，0]And V 24 [m,x％m+1，0]。
Step 716: the corresponding grid to be blanked is set as V 13 [x/m,x％m，0]And V 11 [m,x％m+1，0]A defined grid.
Step 718: the corresponding grid to be blanked is set as V 23 [m,x％m，0]And V 21 [x/m+m+1,x％m+1，0]A defined grid.
Step 720: and respectively setting the transparency of the vertexes of the blanking grids in the pixel point coordinate sets to be spliced on the left side and the right side to be 0.
Step 722: and translating the grid model to be spliced on the right side leftwards by a translation amount which is the difference value between the pixel point to be spliced on the right side and the abscissa of the pixel point to be spliced on the left side corresponding to the pixel point to be spliced on the right side.
The real-time video stitching method comprises the steps of stitching a frame of picture corresponding to a first video and a second video, and obtaining a pixel point coordinate set of a picture stitching position; establishing a three-dimensional grid model, respectively importing a first video and a second video, and converting a pixel point coordinate set of a picture splicing position into a grid vertex coordinate set of the grid model; determining grids to be blanked, moving a grid model, removing the grids to be blanked, and completing splicing. According to the application, only one frame of picture is selected at will for splicing, and for the rest multi-frame pictures, only the transparency of the top points of the three-dimensional grid model is utilized to blank the public area of two sections of videos to be spliced, so that the two sections of real-time videos are spliced directly, the calculation amount and the resource consumption of a system are reduced, and the real-time performance of the videos can be better ensured.
Embodiment two:
the real-time video stitching device provided by the present application, as shown in fig. 8, in one embodiment thereof, may include a video processing module 810, a model building module 820, and a video stitching module 830.
The video processing module 810 may be configured to intercept a first video and a second video to be stitched, stitch a frame of picture corresponding to the first video and the second video, and obtain a pixel coordinate set of a stitching position of the picture.
And intercepting a frame of picture Texture1 and a frame of picture Texture2 of the two real-time Video1 and Video2 which need to be spliced at the same time, and splicing by using a picture splicing algorithm, wherein the picture splicing algorithm can be determined according to actual conditions. Assuming that the sizes of two pictures are m×n (m, n represent the pixel sizes of the length and the width of the picture respectively), finally obtaining pixel Point coordinate sets Point1[ n ] and Point2[ n ] of the splicing positions on the two pictures.
The model building module 820 may be configured to build a three-dimensional mesh model, respectively import videos, and convert the pixel coordinate set of the image stitching location into a mesh vertex coordinate set of the mesh model.
The coordinates of the picture pixels are stored in a one-dimensional array P [ mn ] from left to right and from bottom to top, each element Px in the array represents the color attribute value of a corresponding pixel in the picture, the size of the array is m x n, wherein m and n are natural numbers respectively, and x is {0,1,2, … … mn-1}, and the arrangement mode is shown in figure 2. Because of the two-dimensional characteristic of the picture, as shown in fig. 3, the position of a certain pixel Px in the picture can be expressed in the form of Pij, where i represents the number of rows where the pixel is located, and j represents the number of columns where the pixel is located, so Px can be expressed as P [ x/m+1, x%m+1].
Further, in order to facilitate calculation and more intuitively represent the relationship between the pixel image and the grid Model, two Plane models Model1 and Model2 having m unit grids in the horizontal direction and n unit grids in the vertical direction are respectively manufactured according to the image size. After the model is manufactured, the model is placed at the original point of the three-dimensional coordinate in sequence, the placing position of the model is shown in fig. 4, and simultaneously two real-time Video1 and Video2 which need to be spliced are respectively connected to the two models. In order to conveniently express the relation between the picture and the grid model, the number of unit grids of the established grid model is consistent with the size of pixels of the picture, so that the pixel point can be directly positioned to the corresponding grid vertex coordinate on the grid model according to Px, one pixel point in the picture is expressed as a square grid consisting of two triangles on the grid model, and the upper right corner three-dimensional coordinate P (V 1 ，V 2 ，V 3 ，V 4 ) Can be expressed as: v (V) 1 [x/m+1,x％m+1，0]The remaining three points in the clockwise direction are respectively expressed as: v (V) 2 [x/m+1,x％m，0]、V 3 [x/m,x％m，0]And V 4 [x/m,x％m+1，0]. Through the coordinate mapping twice, the pixel points in the left model surface picture in the scene can be represented by four vertex coordinates in the grid model. In this embodiment, left and right stitching is taken as an example to describe that the coordinates of the pixels in the left model surface picture can directly correspond to the coordinates of the three-dimensional scene, while the coordinates of the pixel points in the right model surface picture and the stitching points Point2[ n ] in the picture]It is necessary to add an offset in the x-axis direction, the offset being m, the resulting splice Point being Point 2' n]. Therefore, the square grid vertex coordinates corresponding to the pixel point Px of the right model surface pictureDenoted as V 1 [x/m+m+1,x％m+1，0]、V 2 [x/m+1+m,x％m，0]、V 3 [m,x％m，0]And V 4 [m,x％m+1，0]。
The video stitching module 830 may be configured to determine a grid to be blanked, move the grid model, remove the grid to be blanked, and complete stitching.
Pixel Point coordinate set Point1[ n ] of splicing position available from two pictures]And Point2[ n ]]. As shown in fig. 5 and 6, for Point1[ n ]]The coordinates of the lower left corner of the square grid in the grid model corresponding to the point Px in (1) can be expressed as V [ x/m, x% m,0]At the same time because of Point1[ n ]]The coordinates of the stitching points representing the surface pictures of the grid model located on the left, so that the corresponding blanking grid of the points is defined by V [ x/m, x% m,0]And V [ m, x% m+1,0]A defined grid. Point1[ n ]]The transparency of the defined vertex of the blanking grid corresponding to each pixel point is set to 0, namely, the common area of the coordinate model surface picture is set to be transparent. For Point 2' [ n ]]The coordinates of the upper right corner of the square grid in the corresponding point grid model can be expressed as V [ x/m+m+1, x%m+1,0]. At the same time because of Point 2' n]The coordinates of the stitching points representing the surface pictures of the grid model positioned on the right side, so that the blanking grid corresponding to the points is formed by V 23 [m,x％m，0]And V 21 [x/m+m+1,x％m+1，0]A defined grid. Point 2' n]And setting the transparency of the defined vertexes of the blanking grid corresponding to each pixel point to be 0, and setting the common area of the coordinate model surface picture to be transparent.
After the common pixel Point is eliminated, the right side model needs to be shifted leftwards to complete the splicing, and the shift size is Point 2' 0, x-Point1[0] x. The common part of the left and right patches is blanked, so that the splicing of two groups of videos is realized, and a three-dimensional grid splicing schematic diagram is shown in fig. 5.
The real-time video stitching device provided by the present application, as shown in fig. 9, in another embodiment thereof, includes a video processing module 910, a model building module 920, and a video stitching module 930.
The video processing module 910 is configured to intercept a first video and a second video to be spliced, splice a frame of picture corresponding to the first video and the second video, and obtain a pixel point coordinate set of a picture splicing position.
The model building module 920 is configured to build a three-dimensional mesh model, respectively import videos, and convert the pixel point coordinate set of the picture stitching location into a mesh vertex coordinate set of the mesh model.
The video stitching module 930 determines the grid to be blanked, moves the grid model, removes the grid to be blanked, and completes the stitching.
Further, the model building module 920 may be further configured to represent, by two coordinate mappings, a pixel point of the image stitching location with four vertex coordinates of the pixel point in the mesh model.
Further, the model building module 920 may include a first conversion unit 921, a second conversion unit 922, and a third conversion unit 923.
The first conversion unit 921 is configured to store the pixel points Px of the picture in a one-dimensional array P [ mn ] in order from left to right and from bottom to top, where n is the number of rows and m is the number of columns;
the second converting unit 922 may be configured to convert a pixel Px of the picture stitching location into a point Pij in two-dimensional coordinates, where the Pij is located, denoted by the two-dimensional coordinates as P [ x/m+1, x%m+1];
a third conversion unit 923, configured to position a pixel Pi j in two-dimensional coordinates to a coordinate V of the pixel corresponding to four vertices of the grid on the grid model 1 [x/m+1,x％m+1，0]、V 2 [x/m+1,x％m，0]、V 3 [x/m,x％m，0]And V 4 [x/m,x％m+1，0]；
The third conversion unit is further used for adding an offset in the x-axis direction, and the size of the offset is m; the first video comprises a grid model to be spliced on the left side, and the coordinates of grid vertexes of pixel points to be spliced on the left side are V 11 [m,x％m+1，0]、V 12 [m,x％m，0]、V 13 [x/m,x％m，0]And V 14 [x/m,x％m+1，0]The method comprises the steps of carrying out a first treatment on the surface of the The second video is a grid model to be spliced on the right side, and grid vertexes of pixel points to be spliced on the right side are obtainedIs V in the coordinate of 21 [x/m+m+1,x％m+1，0]、V 22 [x/m+1+m,x％m，0]、V 23 [m,x％m，0]And V 24 [m,x％m+1，0]。
Further, the video stitching module 930 may include a first processing unit 931, a second processing unit 932, a third processing unit 933, and a stitching unit 934.
A first processing unit 931 configured to set the corresponding grid to be blanked to be set as V 13 [x/m,x％m，0]And V 11 [m,x％m+1，0]A defined grid;
the second processing unit 932 may set the point in the coordinate set of the pixel point to be stitched on the right side to the corresponding grid to be blanked as a grid set by V 23 [m,x％m，0]And V 21 [x/m+m+1,x％m+1，0]A defined grid;
a third processing unit 933, configured to set the transparency of the vertices of the blanking grid in the pixel point coordinate sets to be stitched on the left side and the right side to 0;
and a stitching unit 934, configured to translate the grid model to be stitched on the right side to the left, where the translation amount is a difference value between the right pixel to be stitched and the abscissa of the corresponding left pixel to be stitched.
The real-time video stitching device comprises the steps of stitching a frame of picture corresponding to a first video and a second video, and acquiring a pixel point coordinate set of a picture stitching position; establishing a three-dimensional grid model, respectively importing a first video and a second video, and converting a pixel point coordinate set of a picture splicing position into a grid vertex coordinate set of the grid model; determining grids to be blanked, moving a grid model, removing the grids to be blanked, and completing splicing. According to the application, only one frame of picture is selected at will for splicing, and for the rest multi-frame pictures, only the transparency of the top points of the three-dimensional grid model is utilized to blank the public area of two sections of videos to be spliced, so that the two sections of real-time videos are spliced directly, the calculation amount and the resource consumption of a system are reduced, and the real-time performance of the videos can be better ensured.
Embodiment III:
the real-time video stitching device provided by the embodiment of the application comprises a memory and a processor.
A memory for storing a program;
a processor configured to implement the method in the first embodiment by executing a program stored in the memory.
Those skilled in the art will appreciate that all or part of the steps of the various methods in the above embodiments may be implemented by a program to instruct related hardware, and the program may be stored in a computer readable storage medium, where the storage medium may include: read-only memory, random access memory, magnetic or optical disk, etc. The foregoing is a further detailed description of the application in connection with specific embodiments, and it is not intended that the application be limited to such description. It will be apparent to those skilled in the art that several simple deductions or substitutions can be made without departing from the spirit of the application.
Claims (3)
1. A method of real-time video stitching, comprising:
intercepting a first video and a second video to be spliced, splicing a frame of picture corresponding to the first video and the second video, and acquiring a pixel point coordinate set of a picture splicing position;
establishing a three-dimensional grid model, respectively importing the first video and the second video, and converting a pixel point coordinate set of a picture splicing position into a grid vertex coordinate set of the grid model;
determining grids to be blanked, removing the grids to be blanked by using the transparency of the vertexes of the three-dimensional grid model, and moving the three-dimensional grid model to finish splicing;
the converting the pixel point coordinate set of the picture splicing position into the grid vertex coordinate set of the grid model comprises the following steps:
through coordinate mapping, the pixel point of the picture splicing position is represented by four vertex coordinates of the pixel point in the grid model;
the step of representing the pixel point of the picture splicing position by four vertex coordinates of the pixel point in the grid model through coordinate mapping comprises the following steps:
storing all pixel points of pictures in a first video and a second video in a one-dimensional array Px from left to right and from bottom to top respectively, wherein x is an integer which is more than or equal to 0 and less than or equal to (mn-1), n is the number of lines of the pictures, and m is the number of columns of the pictures;
converting the one-dimensional array Px into a point Pij in two-dimensional coordinates, wherein the position of the point Pij is represented as Px% m+1, x/m+1 by the two-dimensional coordinates;
positioning the pixel point Pij in the two-dimensional coordinates to the coordinate V of the pixel point corresponding to four vertexes of the grid on the grid model 1 [x％m+1,x/m+1，0]、V 2 [x％m+1,x/m，0]、V 3 [x％m,x/m，0]And V 4 [x％m,x/m+1，0]The method comprises the steps of carrying out a first treatment on the surface of the The pixels of the first video picture are represented by four vertexes corresponding to the pixels of the grid model to be used as a grid model to be spliced on the left side, the pixels of the second video picture are represented by four vertexes corresponding to the pixels of the grid model, and the pixels are shifted to the right by m to be used as a grid model to be spliced on the right side; setting grids corresponding to the pixel point coordinate set of the first video picture splicing position as left-side grids to be blanked, and setting corresponding pixels as left-side pixels to be spliced;
the vertex coordinates of grids corresponding to the pixel point coordinate set of the second video picture splicing position are set as right-side grids to be blanked after being shifted rightwards by m, and the corresponding pixel points are set as right-side pixels to be spliced after being shifted rightwards by m; the transparency of the vertexes of the left grid to be blanked and the right grid to be blanked are respectively set to 0;
and translating the grid model to be spliced on the right side leftwards by a translation amount which is the difference value of the horizontal coordinates of the pixel point to be spliced on the right side and the pixel point to be spliced on the left side corresponding to the pixel point to be spliced on the right side.
2. A real-time video stitching device, comprising:
the video processing module is used for intercepting a first video and a second video to be spliced, splicing a frame of picture corresponding to the first video and the second video, and acquiring a pixel point coordinate set of a picture splicing position;
the model building module is used for building a three-dimensional grid model, respectively importing the first video and the second video, and converting a pixel point coordinate set of the picture splicing position into a grid vertex coordinate set of the grid model;
the video stitching module is used for determining grids to be blanked, removing the grids to be blanked by utilizing the transparency of the vertexes of the three-dimensional grid model, and moving the three-dimensional grid model to finish stitching;
the converting the pixel point coordinate set of the picture splicing position into the grid vertex coordinate set of the grid model comprises the following steps:
through coordinate mapping, the pixel point of the picture splicing position is represented by four vertex coordinates of the pixel point in the grid model;
the model building module comprises:
the first conversion unit is used for storing all pixel points of pictures in the first video and the second video in a one-dimensional array Px from left to right and from bottom to top respectively, wherein x is an integer which is more than or equal to 0 and less than or equal to x (mn-1); wherein n is the number of rows and m is the number of columns;
the second conversion unit is used for converting the one-dimensional array Px into a point Pij in two-dimensional coordinates, and the position where the point Pij is located is represented as Px% m+1, x/m+1 by the two-dimensional coordinates;
a third conversion unit for positioning the pixel Pij in the two-dimensional coordinates to the coordinate V of the pixel corresponding to four vertexes of the grid on the grid model 1 [x％m+1,x/m+1，0]、V 2 [x％m+1,x/m，0]、V 3 [x％m,x/m，0]And V 4 [x％m,x/m+1，0]；
The third conversion unit is further configured to use four vertices corresponding to each pixel of the grid model of the first video picture as a grid model to be spliced on the left side, and use four vertices corresponding to each pixel of the grid model of the second video picture as a grid model to be spliced on the right side after the pixels of the first video picture are represented by four vertices corresponding to each pixel of the grid model and are shifted to the right by m;
the video stitching module comprises:
the first processing unit is used for setting grids corresponding to the pixel point coordinate set of the first video picture splicing position as left-side grids to be blanked and setting corresponding pixel points as left-side pixels to be spliced;
the second processing unit is used for setting the vertex coordinates of the grids corresponding to the pixel point coordinate set of the second video picture splicing position as a grid to be blanked on the right after shifting rightwards by m and setting the corresponding pixel points as pixel points to be spliced on the right after shifting rightwards by m;
the third processing unit is used for setting the transparency of the vertexes of the left grid to be blanked and the right grid to be blanked to 0 respectively;
and the splicing unit is used for translating the grid model to be spliced on the right side leftwards, wherein the translation amount is the difference value of the horizontal coordinates of the pixel point to be spliced on the right side and the pixel point to be spliced on the left side corresponding to the pixel point to be spliced.
3. A real-time video stitching device, comprising:
a memory for storing a program;
a processor for implementing the method as claimed in claim 1 by executing the program stored in the memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010260631.2A CN111476716B (en) | 2020-04-03 | 2020-04-03 | Real-time video stitching method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010260631.2A CN111476716B (en) | 2020-04-03 | 2020-04-03 | Real-time video stitching method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111476716A CN111476716A (en) | 2020-07-31 |
| CN111476716B true CN111476716B (en) | 2023-09-26 |
Family
ID=71750467
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010260631.2A Active CN111476716B (en) | 2020-04-03 | 2020-04-03 | Real-time video stitching method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111476716B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115086575B (en) * | 2022-08-16 | 2022-11-29 | 之江实验室 | Video picture splicing method and device based on unmanned vehicle remote driving |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103096043A (en) * | 2013-02-21 | 2013-05-08 | 安徽大学 | Mine safety monitoring method based on parallel video splicing technology |
| CN107392848A (en) * | 2017-06-14 | 2017-11-24 | 江西科技师范大学 | Panoramic image display method and device |
| CN107734268A (en) * | 2017-09-18 | 2018-02-23 | 北京航空航天大学 | A kind of structure-preserved wide baseline video joining method |
| CN108700798A (en) * | 2016-01-03 | 2018-10-23 | 人眼技术有限公司 | Frame during creating panoramic frame adapts to splicing |
| CN110136064A (en) * | 2019-05-14 | 2019-08-16 | 武汉高德红外股份有限公司 | A kind of image split-joint method and system based on transparent figure layer technology |
-
2020
- 2020-04-03 CN CN202010260631.2A patent/CN111476716B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103096043A (en) * | 2013-02-21 | 2013-05-08 | 安徽大学 | Mine safety monitoring method based on parallel video splicing technology |
| CN108700798A (en) * | 2016-01-03 | 2018-10-23 | 人眼技术有限公司 | Frame during creating panoramic frame adapts to splicing |
| CN107392848A (en) * | 2017-06-14 | 2017-11-24 | 江西科技师范大学 | Panoramic image display method and device |
| CN107734268A (en) * | 2017-09-18 | 2018-02-23 | 北京航空航天大学 | A kind of structure-preserved wide baseline video joining method |
| CN110136064A (en) * | 2019-05-14 | 2019-08-16 | 武汉高德红外股份有限公司 | A kind of image split-joint method and system based on transparent figure layer technology |
Non-Patent Citations (3)
| Title |
|---|
| Wei LYU等.A survey on image and video stitching.《Virtual Reality & Intelligent Hardware》.2019,1-30. * |
| 具有直线结构保护的网格化图像拼接;何川等;《中国图象图形学报》;20180716(第07期);973-983 * |
| 宽基线视差图像的拼接算法;左森等;《计算机工程》;20070520(第10期);175-177 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111476716A (en) | 2020-07-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109547766B (en) | Panoramic image generation method and device | |
| Zhang et al. | Framebreak: Dramatic image extrapolation by guided shift-maps | |
| CN107993276B (en) | Panoramic image generation method and device | |
| JPH0879622A (en) | Method and apparatus for removing distortion in wide-angle image | |
| CN107924556B (en) | Image generation device and image display control device | |
| WO2010051147A2 (en) | Method and apparatus for transforming a non-linear lens-distorted image | |
| JP2005339313A (en) | Method and apparatus for presenting image | |
| JP6683307B2 (en) | Optimal spherical image acquisition method using multiple cameras | |
| Nielsen | Surround video: a multihead camera approach | |
| CN110458960B (en) | Polarization-based three-dimensional reconstruction method for colored object | |
| CN111866523B (en) | Panoramic video synthesis method and device, electronic equipment and computer storage medium | |
| CN113808261B (en) | Panorama-based self-supervised learning scene point cloud completion data set generation method | |
| CN110807413B (en) | Target display method and related device | |
| CN114782648A (en) | Image processing method, image processing device, electronic equipment and storage medium | |
| CN111476716B (en) | Real-time video stitching method and device | |
| CN114926612A (en) | Aerial panoramic image processing and immersive display system | |
| CN111161130A (en) | Video correction method based on three-dimensional geographic information | |
| US11528412B2 (en) | Apparatus and method for stitching together multiple images | |
| CN111161138B (en) | Target detection method, device, equipment and medium for two-dimensional panoramic image | |
| CN111091117B (en) | Target detection method, device, equipment and medium for two-dimensional panoramic image | |
| CN114332356A (en) | Virtual and real picture combining method and device | |
| TWI756956B (en) | Image processing method and device for panorama image | |
| CN112866507B (en) | Intelligent panoramic video synthesis method and system, electronic device and medium | |
| CN114004773A (en) | Monocular multi-view video synthesis method based on deep learning and reverse mapping | |
| CN109801211B (en) | Panoramic camera-based target object elimination method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |