CN111476716A - Real-time video splicing method and device - Google Patents

Real-time video splicing method and device Download PDF

Info

Publication number
CN111476716A
CN111476716A CN202010260631.2A CN202010260631A CN111476716A CN 111476716 A CN111476716 A CN 111476716A CN 202010260631 A CN202010260631 A CN 202010260631A CN 111476716 A CN111476716 A CN 111476716A
Authority
CN
China
Prior art keywords
video
grid
spliced
pixel point
model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010260631.2A
Other languages
Chinese (zh)
Other versions
CN111476716B (en
Inventor
俞俊程
陈思
吕长宝
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen ZNV Technology Co Ltd
Nanjing ZNV Software Co Ltd
Original Assignee
Shenzhen ZNV Technology Co Ltd
Nanjing ZNV Software Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen ZNV Technology Co Ltd, Nanjing ZNV Software Co Ltd filed Critical Shenzhen ZNV Technology Co Ltd
Priority to CN202010260631.2A priority Critical patent/CN111476716B/en
Publication of CN111476716A publication Critical patent/CN111476716A/en
Application granted granted Critical
Publication of CN111476716B publication Critical patent/CN111476716B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G06T17/205—Re-meshing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44016—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving splicing one content stream with another content stream, e.g. for substituting a video clip

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Computer Graphics (AREA)
  • Architecture (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Geometry (AREA)
  • Image Generation (AREA)
  • Processing Or Creating Images (AREA)

Abstract

A real-time video splicing method and device comprises the following steps: intercepting a first video and a second video to be spliced, splicing a frame of picture corresponding to the first video and the second video, and acquiring a pixel point coordinate set of a picture splicing position; establishing a three-dimensional grid model, respectively importing the first video and the second video, and converting a pixel point coordinate set of a picture splicing position into a grid vertex coordinate set of the grid model; and determining the grids to be blanked, moving the grid model, removing the grids to be blanked, and completing splicing. According to the method and the device, only one frame of picture needs to be selected randomly for splicing, for the rest multiple frames of pictures, only the vertex transparency of the three-dimensional grid model is used, the public areas of two sections of videos needing to be spliced are blanked, the two sections of real-time videos are directly spliced, the calculated amount and the resource consumption of the system are reduced, and the real-time performance of the videos can be better guaranteed.

Description

Real-time video splicing method and device
Technical Field
The application relates to the technical field of videos, in particular to a real-time video splicing method and device.
Background
With the development and popularization of network and multimedia technologies, more and more video and image application technologies enter people's lives, wherein the video technologies occupy important components. In a three-dimensional scene, the access and processing of real-time videos are research hotspots in the field in recent years, and a splicing technology of the gunlock videos is developed in order to achieve a wider visual range and solve the problems of serious picture deformation and the like caused by wide-angle equipment such as a fisheye lens and the like. The video splicing technology can well solve the problems and is widely applied to the fields of monitoring, fire fighting, traffic, public security, medicine and the like.
The video splicing technology is developed on the basis of the picture splicing technology, and the common video splicing method is that each frame of picture of the video to be spliced is spliced by using the picture splicing technology and then combined into the video. The three-dimensional virtual scene is a new application field of the technology, and due to the particularity of the three-dimensional scene and the continuous improvement of the definition of the video at present, the method for splicing and recombining the pictures into the video has large consumption of system resources and large calculation amount, so that the real-time requirement of the video cannot be met.
Disclosure of Invention
The application provides a real-time video splicing method and device capable of reducing system resource consumption.
According to a first aspect of the present application, there is provided a real-time video stitching method, including:
intercepting a first video and a second video to be spliced, splicing a frame of picture corresponding to the first video and the second video, and acquiring a pixel point coordinate set of a picture splicing position;
establishing a three-dimensional grid model, respectively importing the first video and the second video, and converting a pixel point coordinate set of a picture splicing position into a grid vertex coordinate set of the grid model;
and determining a mesh to be blanked, removing the mesh to be blanked by using the transparency of the vertex of the three-dimensional mesh model, and moving the three-dimensional mesh model to finish splicing.
Further, the converting the pixel point coordinate set of the picture stitching position into a mesh vertex coordinate set of the mesh model includes:
and expressing the pixel point of the picture splicing position by four vertex coordinates of the pixel point in the grid model through two times of coordinate mapping.
Further, the representing the pixel point of the picture stitching position by four vertex coordinates of the pixel point in the mesh model through two times of coordinate mapping includes:
storing pixel points Px in the picture in a one-dimensional array P [ mn ] according to the sequence from left to right and from bottom to top, wherein n is the number of rows, and m is the number of columns;
converting the pixel point Px into a point Pij in a two-dimensional coordinate, wherein the position of the Pij is represented as P [ x/m +1, x% m +1] through the two-dimensional coordinate;
positioning a pixel point Pij in the two-dimensional coordinates to the coordinates V of the pixel point on the grid model corresponding to four vertexes of the grid1[x/m+1,x％m+1，0]、V2[x/m+1,x％m，0]、V3[x/m,x％m，0]And V4[x/m,x％m+1，0]。
Further, the representing the pixel point of the picture stitching position by the coordinates of the four vertexes of the pixel point in the mesh model through two times of coordinate mapping further includes:
adding an offset in the x-axis direction, wherein the offset is m;
the first video comprises a left mesh model to be spliced and left pixels to be splicedThe grid vertex of a point has a coordinate V11[m,x％m+1，0]、V12[m,x％m，0]、V13[x/m,x％m，0]And V14[x/m,x％m+1，0]；
The second video is a grid model to be spliced on the right side, and the coordinate of the grid vertex of the pixel point to be spliced on the right side is V21[x/m+m+1,x％m+1，0]、V22[x/m+1+m,x％m，0]、V23[m,x％m，0]And V24[m,x％m+1，0]。
Further, the determining the mesh to be blanked, removing the mesh to be blanked by using the transparency of the vertex of the three-dimensional mesh model, and moving the three-dimensional mesh model to complete the stitching includes:
setting the point in the left pixel point coordinate set to be spliced and the corresponding grid to be blanked as V13[x/m,x％m，0]And V11[m,x％m+1，0]A defined grid;
setting the point in the right pixel point coordinate set to be spliced as the grid to be blanked23[m,x％m，0]And V21[x/m+m+1,x％m+1，0]A defined grid;
respectively setting the transparency of the top points of the blanking grids in the left side and the right side pixel point coordinate sets to be spliced as 0;
and translating the grid model to be spliced on the right side leftwards, wherein the translation amount is the difference value of the horizontal coordinates of the pixel points to be spliced on the right side and the pixel points to be spliced on the left side corresponding to the pixel points to be spliced on the right side.
According to a second aspect of the present application, there is provided a real-time video stitching device, comprising:
the video processing module is used for intercepting a first video and a second video to be spliced, splicing a frame of picture corresponding to the first video and the second video and acquiring a pixel point coordinate set of a picture splicing position;
the model establishing module is used for establishing a three-dimensional grid model, respectively importing the first video and the second video, and converting a pixel point coordinate set of a picture splicing position into a grid vertex coordinate set of the grid model;
and the video splicing module is used for determining the mesh to be blanked, removing the mesh to be blanked by utilizing the transparency of the vertex of the three-dimensional mesh model, and moving the three-dimensional mesh model to complete splicing.
Further, the model building module is further configured to represent the pixel point of the picture stitching position by four vertex coordinates of the pixel point in the mesh model through two times of coordinate mapping.
Further, the model building module includes:
the first conversion unit is used for storing pixel points Px in the picture in a one-dimensional array P [ mn ] according to the sequence from left to right and from bottom to top, wherein n is the number of rows, and m is the number of columns;
the second conversion unit is used for converting the pixel point Px of the picture splicing position into a point Pij in a two-dimensional coordinate, and the position of the Pij is represented as P [ x/m +1, x% m +1] through the two-dimensional coordinate;
a third conversion unit, configured to position a pixel point Pij in the two-dimensional coordinates to a coordinate V of the pixel point at four vertices of the corresponding mesh on the mesh model1[x/m+1,x％m+1，0]、V2[x/m+1,x％m，0]、V3[x/m,x％m，0]And V4[x/m,x％m+1，0]；
The third conversion unit is further configured to add an offset in the x-axis direction, where the offset is m; the first video comprises a left-side to-be-spliced grid model, and the coordinates of grid vertexes of left-side to-be-spliced pixel points are V11[m,x％m+1，0]、V12[m,x％m，0]、V13[x/m,x％m，0]And V14[x/m,x％m+1，0](ii) a The second video is a grid model to be spliced on the right side, and the coordinate of the grid vertex of the pixel point to be spliced on the right side is V21[x/m+m+1,x％m+1，0]、V22[x/m+1+m,x％m，0]、V23[m,x％m，0]And V24[m,x％m+1，0]。
Further, the video stitching module includes:
a first processing unit for splicing the left sideThe point in the pixel point coordinate set, the corresponding grid to be blanked is set as V13[x/m,x％m，0]And V11[m,x％m+1，0]A defined grid;
the second processing unit sets the point in the coordinate set of the pixel points to be spliced on the right side and the corresponding grid to be blanked as V23[m,x％m，0]And V21[x/m+m+1,x％m+1，0]A defined grid;
the third processing unit is used for respectively setting the transparency of the vertexes of the blanking grids in the pixel point coordinate sets to be spliced on the left side and the right side to be 0;
and the splicing unit is used for translating the grid model to be spliced on the right side leftwards, and the translation amount is the difference value of the horizontal coordinates of the pixel points to be spliced on the right side and the pixel points to be spliced on the left side corresponding to the pixel points to be spliced on the right side.
According to a third aspect of the present application, there is provided a real-time video stitching device, comprising:
a memory for storing a program;
a processor for implementing the above method by executing the program stored in the memory.
The method and the device for real-time video splicing comprise the steps of splicing a frame of picture corresponding to a first video and a second video, and acquiring a pixel point coordinate set of a picture splicing position; establishing a three-dimensional grid model, respectively importing the first video and the second video, and converting a pixel point coordinate set of a picture splicing position into a grid vertex coordinate set of the grid model; and determining the grids to be blanked, moving the grid model, removing the grids to be blanked, and completing splicing. According to the method and the device, only one frame of picture needs to be selected randomly for splicing, for the rest multiple frames of pictures, only the vertex transparency of the three-dimensional grid model is used, the public areas of two sections of videos needing to be spliced are blanked, the two sections of real-time videos are directly spliced, the calculated amount and the resource consumption of the system are reduced, and the real-time performance of the videos can be better guaranteed.
Drawings
FIG. 1 is a flow chart of a method in one embodiment of the present application;
fig. 2 is a schematic diagram of a pixel point Px in a picture in a one-dimensional array according to a first embodiment of the present application;
FIG. 3 is a schematic diagram of Px in FIG. 2 mapped as a pixel point Pij in a two-dimensional coordinate system;
FIG. 4 is a diagram illustrating the mapping of the pixel point Pij in FIG. 3 to the pixel point P (v) disposed in the mesh model1,v2,v3,v4) A schematic diagram of (a);
FIG. 5 is a schematic diagram of a blanking region of a three-dimensional mesh model according to an embodiment of the present application;
FIG. 6 is a schematic diagram of three-dimensional mesh model stitching according to an embodiment of the present application;
FIG. 7 is a flow chart of a method in another embodiment according to the first embodiment of the present application;
FIG. 8 is a schematic diagram of program modules of an apparatus according to a second embodiment of the present application;
fig. 9 is a schematic diagram of program modules of an apparatus according to another embodiment of the second embodiment of the present application.
Detailed Description
The present invention will be described in further detail with reference to the following detailed description and accompanying drawings. The present application may be embodied in many different forms and is not limited to the embodiments described in the present embodiment. The following detailed description is provided to facilitate a more thorough understanding of the present disclosure, and the words used to indicate orientation, top, bottom, left, right, etc. are used solely to describe the illustrated structure in connection with the accompanying figures.
One skilled in the relevant art will recognize, however, that one or more of the specific details can be omitted, or other methods, components, or materials can be used. In some instances, some embodiments are not described or not described in detail.
The numbering of the components as such, e.g., "first", "second", etc., is used herein only to distinguish the objects as described, and does not have any sequential or technical meaning.
Furthermore, the technical features, aspects or characteristics described herein may be combined in any suitable manner in one or more embodiments. It will be readily appreciated by those of skill in the art that the order of the steps or operations of the methods associated with the embodiments provided herein may be varied. Thus, any sequence in the figures and examples is for illustrative purposes only and does not imply a requirement in a certain order unless explicitly stated to require a certain order.
This application can be used to the rifle bolt position and be in same level, and inclination is the same, has a plurality of rifle bolt video concatenations of public picture. If the pictures shot by the two gunlocks have height difference, the pictures can be cut, and the public pictures can be taken for splicing.
The first embodiment is as follows:
an implementation manner of the real-time video stitching method provided by the present application, as shown in fig. 1, includes the following steps:
step 102: intercepting a first video and a second video to be spliced, splicing a frame of picture corresponding to the first video and the second video, and acquiring a pixel point coordinate set of a picture splicing position.
Two frames of pictures Texture1 and Texture2 of real-time videos Video1 and Video2 needing to be spliced at the same time are captured, and a picture splicing algorithm is used for splicing and can be determined according to actual conditions. Assuming that the sizes of the two pictures are m × n (m and n respectively represent the pixel sizes of the length and the width of the pictures), the pixel Point coordinate sets Point1[ n ] and Point2[ n ] of the splicing positions of the two pictures are finally obtained.
Step 104: and establishing a three-dimensional grid model, respectively importing the first video and the second video, and converting a pixel point coordinate set of a picture splicing position into a grid vertex coordinate set of the grid model.
The coordinates of the picture pixels are stored in a one-dimensional array P [ mn ] from left to right and from bottom to top, each element Px in the array represents a color attribute value of a corresponding pixel in the picture, the size of the array is m × n, wherein m and n are respectively natural numbers, x ∈ {0,1,2, … … mn-1}, and the arrangement mode is shown in FIG. 2. due to the two-dimensional characteristic of the picture, as shown in FIG. 3, the position of a certain pixel Px in the picture can be represented in the form of Pij, wherein i represents the number of rows where the pixel is located, and j represents the number of columns where the pixel is located, so that Px can be represented as P [ x/m +1, x% m +1 ].
Furthermore, in order to facilitate calculation and more intuitively represent the relationship between the pixel picture and the grid Model, two Plane models 1 and 2 with m unit grids in the transverse direction and n unit grids in the longitudinal direction are respectively manufactured according to the picture size. After the model is manufactured, the three-dimensional Video images are sequentially placed at the three-dimensional coordinate origin, the model is placed at the position shown in fig. 4, and two real-time videos Video1 and Video2 which need to be spliced are simultaneously and respectively connected to the two models. In order to conveniently represent the relation between the picture and the grid model, the number of unit grids of the established grid model is consistent with the size of the picture pixel, so that the grid vertex coordinates of the pixel point on the grid model can be directly positioned according to Px, one pixel point in the picture is represented as a square grid consisting of two triangles on the grid model, and the upper right three-dimensional coordinate P (V) of the square grid1，V2，V3，V4) Can be expressed as: v1[x/m+1,x％m+1，0]The other three points in the clockwise direction are respectively represented as: v2[x/m+1,x％m，0]、V3[x/m,x％m，0]And V4[x/m,x％m+1，0]. So far, through two times of coordinate mapping, pixel points in the left model surface picture in the scene can be represented by four vertex coordinates in the mesh model. In this embodiment, a left and a right stitching are taken as an example for explanation, coordinates of pixels in the left model surface picture may directly correspond to coordinates of a three-dimensional scene, and coordinates of pixels in the right model surface picture and a stitching Point2[ n ] in the picture]An offset is added in the x-axis direction, the offset being m, and the resulting splice Point being Point 2' [ n ]]. Therefore, the vertex coordinates of the square grid corresponding to the pixel point Px of the right model surface picture are represented as V1[x/m+m+1,x％m+1，0]、V2[x/m+1+m,x％m，0]、V3[m,x％m，0]And V4[m,x％m+1，0]。
Step 106: and determining the mesh to be blanked, removing the mesh to be blanked by using the transparency of the vertex of the three-dimensional mesh model, and moving the three-dimensional mesh model to complete splicing.
Pixel Point coordinate set Point1 n of stitching position available from two pictures]And Point2[ n ]]. As shown in fig. 5 and 6, for Point1 n]The coordinate of the point Px in (1), which corresponds to the lower left corner of the square grid in the grid model, can be represented as V [ x/m, x% m, 0]And at the same time because of Point1[ n ]]The coordinates of the splice points representing the surface picture of the mesh model on the left side, so the blanked mesh corresponding to this point is represented by Vx/m, x% m, 0]And V [ m, x% m +1, 0]A defined grid. Point1[ n ]]The transparency of the defined vertex of the blanking grid corresponding to each pixel point in the coordinate model surface picture is set to be 0, that is, the common area of the coordinate model surface picture is set to be transparent. For Point 2' [ n]The coordinates of the point Px in (1), which corresponds to the upper right corner of the square grid in the point grid model, can be expressed as V [ x/m + m +1, x% m +1, 0]. At the same time, because of Point 2' [ n ]]The coordinates of the splicing points representing the surface picture of the mesh model positioned at the right side, so that the corresponding blanked mesh is formed by V23[m,x％m，0]And V21[x/m+m+1,x％m+1，0]A defined grid. Point 2' n]The transparency of the defined vertex of the blanking grid corresponding to each pixel point is set to be 0, and then the common area of the coordinate model surface picture is set to be transparent.
After eliminating the common pixel Point, the right model needs to be translated to the left to complete the splicing, the translation size is Point 2' 0. x-Point 10. x. At this point, the common part of the left and right patches is blanked, and the splicing of the two groups of videos is realized, and a three-dimensional mesh splicing schematic diagram is shown in fig. 5.
As shown in fig. 7, another embodiment of a real-time video splicing method provided in the present application includes the following steps:
step 702: intercepting one frame of picture Texture1 and Texture2 of two segments of real-time videos to be spliced, namely Video1 and Video2, at the same time, and acquiring pixel Point coordinate sets Point1[ n ] and Point2[ n ] of picture splicing positions.
Wherein, the sizes of the two pictures are set as m × n (m, n respectively represent the pixel sizes of the length and width of the pictures), and finally, the pixel Point coordinate sets Point1[ n ] and Point2[ n ] of the splicing positions of the two pictures are obtained.
Step 704: and storing pixel points Px in the picture in a one-dimensional array P [ mn ] according to the sequence from left to right and from bottom to top, wherein n is the number of rows, and m is the number of columns.
Step 706: and converting the pixel point Px into a point Pij in a two-dimensional coordinate, wherein the position of the Pij is represented as P [ x/m +1, x% m +1] through the two-dimensional coordinate.
Step 708: establishing a three-dimensional grid model, and positioning a pixel point Pi j in a two-dimensional coordinate to a coordinate V of the pixel point on the grid model corresponding to four vertexes of a grid1[x/m+1,x％m+1，0]、V2[x/m+1,x％m，0]、V3[x/m,x％m，0]And V4[x/m,x％m+1，0]。
And expressing the pixel point of the picture splicing position by four vertex coordinates of the pixel point in the grid model through two times of coordinate mapping.
Step 710: an offset is added in the x-axis direction, and the offset is m.
Step 712: and acquiring the coordinates of the grid vertexes of the pixels to be spliced of the Texture 1. Texture1 is a left-side to-be-spliced grid model, and the coordinates of grid vertexes of left-side to-be-spliced pixel points are V11[m,x％m+1，0]、V12[m,x％m，0]、V13[x/m,x％m，0]And V14[x/m,x％m+1，0]；
Step 714: and acquiring the coordinates of the grid vertexes of the pixels to be spliced of the Texture 2. Texture2 is a right-side to-be-spliced grid model, and the coordinate of the grid vertex of the right-side to-be-spliced pixel point is obtained as V21[x/m+m+1,x％m+1，0]、V22[x/m+1+m,x％m，0]、V23[m,x％m，0]And V24[m,x％m+1，0]。
Step 716: setting the point in the left pixel point coordinate set to be spliced and the corresponding grid to be blanked as V13[x/m,x％m，0]And V11[m,x％m+1，0]A defined grid.
Step 718: the right side is splicedThe point in the pixel point coordinate set, the corresponding grid to be blanked is set as V23[m,x％m，0]And V21[x/m+m+1,x％m+1，0]A defined grid.
Step 720: and respectively setting the transparency of the top points of the blanking grids in the pixel point coordinate sets to be spliced on the left side and the right side as 0.
Step 722: and translating the grid model to be spliced on the right side leftwards, wherein the translation amount is the difference value of the horizontal coordinates of the pixel points to be spliced on the right side and the pixel points to be spliced on the left side corresponding to the pixel points to be spliced on the right side.
The real-time video splicing method comprises the steps of splicing a frame of picture corresponding to a first video and a second video, and obtaining a pixel point coordinate set of a picture splicing position; establishing a three-dimensional grid model, respectively importing the first video and the second video, and converting a pixel point coordinate set of a picture splicing position into a grid vertex coordinate set of the grid model; and determining the grids to be blanked, moving the grid model, removing the grids to be blanked, and completing splicing. According to the method and the device, only one frame of picture needs to be selected randomly for splicing, for the rest multiple frames of pictures, only the vertex transparency of the three-dimensional grid model is used, the public areas of two sections of videos needing to be spliced are blanked, the two sections of real-time videos are directly spliced, the calculated amount and the resource consumption of the system are reduced, and the real-time performance of the videos can be better guaranteed.
Example two:
as shown in fig. 8, an embodiment of a real-time video splicing apparatus provided in the present application may include a video processing module 810, a model building module 820, and a video splicing module 830.
The video processing module 810 may be configured to intercept a first video and a second video to be spliced, splice a frame of picture corresponding to the first video and the second video, and obtain a pixel coordinate set of a picture splicing position.
Two frames of pictures Texture1 and Texture2 of real-time videos Video1 and Video2 needing to be spliced at the same time are captured, and a picture splicing algorithm is used for splicing and can be determined according to actual conditions. Assuming that the sizes of the two pictures are m × n (m and n respectively represent the pixel sizes of the length and the width of the pictures), the pixel Point coordinate sets Point1[ n ] and Point2[ n ] of the splicing positions of the two pictures are finally obtained.
The model building module 820 may be configured to build a three-dimensional mesh model, import a video, and convert a pixel point coordinate set of a picture splicing position into a mesh vertex coordinate set of the mesh model.
The coordinates of the picture pixels are stored in a one-dimensional array P [ mn ] from left to right and from bottom to top, each element Px in the array represents a color attribute value of a corresponding pixel in the picture, the size of the array is m × n, wherein m and n are respectively natural numbers, x ∈ {0,1,2, … … mn-1}, and the arrangement mode is shown in FIG. 2. due to the two-dimensional characteristic of the picture, as shown in FIG. 3, the position of a certain pixel Px in the picture can be represented in the form of Pij, wherein i represents the number of rows where the pixel is located, and j represents the number of columns where the pixel is located, so that Px can be represented as P [ x/m +1, x% m +1 ].
Furthermore, in order to facilitate calculation and more intuitively represent the relationship between the pixel picture and the grid Model, two Plane models 1 and 2 with m unit grids in the transverse direction and n unit grids in the longitudinal direction are respectively manufactured according to the picture size. After the model is manufactured, the three-dimensional Video images are sequentially placed at the three-dimensional coordinate origin, the model is placed at the position shown in fig. 4, and two real-time videos Video1 and Video2 which need to be spliced are simultaneously and respectively connected to the two models. In order to conveniently represent the relation between the picture and the grid model, the number of unit grids of the established grid model is consistent with the size of the picture pixel, so that the grid vertex coordinates of the pixel point on the grid model can be directly positioned according to Px, one pixel point in the picture is represented as a square grid consisting of two triangles on the grid model, and the upper right three-dimensional coordinate P (V) of the square grid1，V2，V3，V4) Can be expressed as: v1[x/m+1,x％m+1，0]The other three points in the clockwise direction are respectively represented as: v2[x/m+1,x％m，0]、V3[x/m,x％m，0]And V4[x/m,x％m+1，0]. To this end, the field is mapped by two times of coordinatesThe pixel points in the surface picture of the left model in the scene can be represented by four vertex coordinates in the mesh model. In this embodiment, a left and a right stitching are taken as an example for explanation, coordinates of pixels in the left model surface picture may directly correspond to coordinates of a three-dimensional scene, and coordinates of pixels in the right model surface picture and a stitching Point2[ n ] in the picture]An offset is added in the x-axis direction, the offset being m, and the resulting splice Point being Point 2' [ n ]]. Therefore, the vertex coordinates of the square grid corresponding to the pixel point Px of the right model surface picture are represented as V1[x/m+m+1,x％m+1，0]、V2[x/m+1+m,x％m，0]、V3[m,x％m，0]And V4[m,x％m+1，0]。
And the video splicing module 830 is configured to determine a mesh to be blanked, move the mesh model, remove the mesh to be blanked, and complete the splicing.
Pixel Point coordinate set Point1 n of stitching position available from two pictures]And Point2[ n ]]. As shown in fig. 5 and 6, for Point1 n]The coordinate of the point Px in (1), which corresponds to the lower left corner of the square grid in the grid model, can be represented as V [ x/m, x% m, 0]And at the same time because of Point1[ n ]]The coordinates of the splice points representing the surface picture of the mesh model on the left side, so the blanked mesh corresponding to this point is represented by Vx/m, x% m, 0]And V [ m, x% m +1, 0]A defined grid. Point1[ n ]]The transparency of the defined vertex of the blanking grid corresponding to each pixel point in the coordinate model surface picture is set to be 0, that is, the common area of the coordinate model surface picture is set to be transparent. For Point 2' [ n]The coordinates of the point Px in (1), which corresponds to the upper right corner of the square grid in the point grid model, can be expressed as V [ x/m + m +1, x% m +1, 0]. At the same time, because of Point 2' [ n ]]The coordinates of the splicing points representing the surface picture of the mesh model positioned at the right side, so that the corresponding blanked mesh is formed by V23[m,x％m，0]And V21[x/m+m+1,x％m+1，0]A defined grid. Point 2' n]The transparency of the defined vertex of the blanking grid corresponding to each pixel point is set to be 0, and then the common area of the coordinate model surface picture is set to be transparent.
After eliminating the common pixel Point, the right model needs to be translated to the left to complete the splicing, the translation size is Point 2' 0. x-Point 10. x. At this point, the common part of the left and right patches is blanked, and the splicing of the two groups of videos is realized, and a three-dimensional mesh splicing schematic diagram is shown in fig. 5.
As shown in fig. 9, another embodiment of the real-time video stitching apparatus provided in the present application includes a video processing module 910, a model building module 920, and a video stitching module 930.
The video processing module 910 is configured to intercept a first video and a second video to be spliced, splice a frame of picture corresponding to the first video and the second video, and acquire a pixel coordinate set of a picture splicing position.
The model building module 920 is configured to build a three-dimensional mesh model, respectively import a video, and convert a pixel point coordinate set of a picture splicing position into a mesh vertex coordinate set of the mesh model.
And the video stitching module 930 determines the mesh to be blanked, moves the mesh model, removes the mesh to be blanked, and completes stitching.
Further, the model building module 920 may be further configured to represent the pixel point of the picture stitching location by four vertex coordinates of the pixel point in the mesh model through two times of coordinate mapping.
Further, the model building module 920 may include a first converting unit 921, a second converting unit 922, and a third converting unit 923.
The first conversion unit 921, configured to store pixel points Px of the picture in a one-dimensional array P [ mn ] in an order from left to right and from bottom to top, where n is a number of rows and m is a number of columns;
the second conversion unit 922 is configured to convert a pixel point Px at the picture stitching position into a point Pij in a two-dimensional coordinate, where a position of the Pij is represented as P [ x/m +1, x% m +1] through the two-dimensional coordinate;
a third converting unit 923, configured to position a pixel point Pi j in the two-dimensional coordinates to a coordinate V of the pixel point at four vertices of the corresponding mesh on the mesh model1[x/m+1,x％m+1，0]、V2[x/m+1,x％m，0]、V3[x/m,x％m，0]And V4[x/m,x％m+1，0]；
The third conversion unit can also be used for adding an offset in the x-axis direction, and the offset is m; the first video comprises a left-side to-be-spliced grid model, and the coordinates of grid vertexes of left-side to-be-spliced pixel points are V11[m,x％m+1，0]、V12[m,x％m，0]、V13[x/m,x％m，0]And V14[x/m,x％m+1，0](ii) a The second video is a grid model to be spliced on the right side, and the coordinate of the grid vertex of the pixel point to be spliced on the right side is V21[x/m+m+1,x％m+1，0]、V22[x/m+1+m,x％m，0]、V23[m,x％m，0]And V24[m,x％m+1，0]。
Further, the video splicing module 930 may include a first processing unit 931, a second processing unit 932, a third processing unit 933, and a splicing unit 934.
The first processing unit 931 may be configured to set a point in the left to-be-stitched pixel coordinate set, where a corresponding grid to be blanked is set to be V13[x/m,x％m，0]And V11[m,x％m+1，0]A defined grid;
the second processing unit 932 may set the point in the coordinate set of the pixel points to be stitched on the right side, corresponding to the grid to be blanked, to be V23[m,x％m，0]And V21[x/m+m+1,x％m+1，0]A defined grid;
the third processing unit 933 is configured to set the transparency of the vertex of the blanking grid in the left-side and right-side pixel point coordinate sets to be spliced to 0;
the splicing unit 934 is used for translating the grid model to be spliced on the right side leftwards, and the translation amount is the difference between the pixel points to be spliced on the right side and the abscissa of the pixel points to be spliced on the left side corresponding to the pixel points to be spliced on the right side.
The real-time video splicing device comprises a first video and a second video, wherein one frame of picture corresponding to the first video and the second video are spliced, and a pixel point coordinate set of a picture splicing position is obtained; establishing a three-dimensional grid model, respectively importing the first video and the second video, and converting a pixel point coordinate set of a picture splicing position into a grid vertex coordinate set of the grid model; and determining the grids to be blanked, moving the grid model, removing the grids to be blanked, and completing splicing. According to the method and the device, only one frame of picture needs to be selected randomly for splicing, for the rest multiple frames of pictures, only the vertex transparency of the three-dimensional grid model is used, the public areas of two sections of videos needing to be spliced are blanked, the two sections of real-time videos are directly spliced, the calculated amount and the resource consumption of the system are reduced, and the real-time performance of the videos can be better guaranteed.
Example three:
the real-time video splicing device provided by the embodiment of the application comprises a memory and a processor.
A memory for storing a program;
and the processor is used for executing the program stored in the memory to realize the method in the first embodiment.
Those skilled in the art will appreciate that all or part of the steps of the various methods in the above embodiments may be implemented by instructions associated with hardware via a program, which may be stored in a computer-readable storage medium, and the storage medium may include: read-only memory, random access memory, magnetic or optical disk, and the like. The foregoing is a more detailed description of the present application in connection with specific embodiments thereof, and it is not intended that the present application be limited to the specific embodiments thereof. It will be apparent to those skilled in the art from this disclosure that many more simple derivations or substitutions can be made without departing from the spirit of the disclosure.

Claims (10)

1. A method for real-time video stitching, comprising:
intercepting a first video and a second video to be spliced, splicing a frame of picture corresponding to the first video and the second video, and acquiring a pixel point coordinate set of a picture splicing position;
establishing a three-dimensional grid model, respectively importing the first video and the second video, and converting a pixel point coordinate set of a picture splicing position into a grid vertex coordinate set of the grid model;
and determining a mesh to be blanked, removing the mesh to be blanked by using the transparency of the vertex of the three-dimensional mesh model, and moving the three-dimensional mesh model to finish splicing.
2. The method of claim 1, wherein converting the set of pixel point coordinates of the picture stitching location to a set of mesh vertex coordinates of the mesh model comprises:
and expressing the pixel point of the picture splicing position by four vertex coordinates of the pixel point in the grid model through two times of coordinate mapping.
3. The method of claim 2, wherein said representing a pixel point of said picture stitching location by coordinates of four vertices of said pixel point in said mesh model through two coordinate mappings comprises:
storing pixel points Px in the picture in a one-dimensional array P [ mn ] according to the sequence from left to right and from bottom to top, wherein n is the number of rows, and m is the number of columns;
converting the pixel point Px into a point Pij in a two-dimensional coordinate, wherein the position of the Pij is represented as P [ x/m +1, x% m +1] through the two-dimensional coordinate;
positioning a pixel point Pij in the two-dimensional coordinates to the coordinates V of the pixel point on the grid model corresponding to four vertexes of the grid1[x/m+1,x％m+1，0]、V2[x/m+1,x％m，0]、V3[x/m,x％m，0]And V4[x/m,x％m+1，0]。
4. The method of claim 3, wherein said representing a pixel point of said picture stitching location by coordinates of four vertices of said pixel point in said mesh model through two coordinate mappings, further comprises:
adding an offset in the x-axis direction, wherein the offset is m;
the first video comprises a left-side to-be-spliced grid model, and the coordinates of grid vertexes of left-side to-be-spliced pixel points are V11[m,x％m+1，0]、V12[m,x％m，0]、V13[x/m,x％m，0]And V14[x/m,x％m+1，0]；
The second video is a grid model to be spliced on the right side, and the coordinate of the grid vertex of the pixel point to be spliced on the right side is V21[x/m+m+1,x％m+1，0]、V22[x/m+1+m,x％m，0]、V23[m,x％m，0]And V24[m,x％m+1，0]。
5. The method of claim 4, wherein the determining the mesh to be blanked, removing the mesh to be blanked with a vertex transparency of the three-dimensional mesh model, and moving the three-dimensional mesh model to complete the stitching comprises:
setting the point in the left pixel point coordinate set to be spliced and the corresponding grid to be blanked as V13[x/m,x％m，0]And V11[m,x％m+1，0]A defined grid;
setting the point in the right pixel point coordinate set to be spliced as the grid to be blanked23[m,x％m，0]And V21[x/m+m+1,x％m+1，0]A defined grid;
respectively setting the transparency of the top points of the blanking grids in the left side and the right side pixel point coordinate sets to be spliced as 0;
and translating the grid model to be spliced on the right side leftwards, wherein the translation amount is the difference value of the horizontal coordinates of the pixel points to be spliced on the right side and the pixel points to be spliced on the left side corresponding to the pixel points to be spliced on the right side.
6. A real-time video stitching device, comprising:
the video processing module is used for intercepting a first video and a second video to be spliced, splicing a frame of picture corresponding to the first video and the second video and acquiring a pixel point coordinate set of a picture splicing position;
the model establishing module is used for establishing a three-dimensional grid model, respectively importing the first video and the second video, and converting a pixel point coordinate set of a picture splicing position into a grid vertex coordinate set of the grid model;
and the video splicing module is used for determining the mesh to be blanked, removing the mesh to be blanked by utilizing the transparency of the vertex of the three-dimensional mesh model, and moving the three-dimensional mesh model to complete splicing.
7. The apparatus of claim 6, wherein the model building module is further configured to represent a pixel point of the picture stitching location by four vertex coordinates of the pixel point in the mesh model through two coordinate mappings.
8. The apparatus of claim 7, wherein the model building module comprises:
the first conversion unit is used for storing pixel points Px in the picture in a one-dimensional array P [ mn ] according to the sequence from left to right and from bottom to top, wherein n is the number of rows, and m is the number of columns;
the second conversion unit is used for converting the pixel point Px into a point Pij in a two-dimensional coordinate, and the position of the Pij is represented as P [ x/m +1, x% m +1] through the two-dimensional coordinate;
a third conversion unit, configured to position a pixel point Pij in the two-dimensional coordinates to a coordinate V of the pixel point at four vertices of the corresponding mesh on the mesh model1[x/m+1,x％m+1，0]、V2[x/m+1,x％m，0]、V3[x/m,x％m，0]And V4[x/m,x％m+1，0]；
The third conversion unit is further configured to add an offset in the x-axis direction, where the offset is m; the first video comprises a left-side to-be-spliced grid model, and the coordinates of grid vertexes of left-side to-be-spliced pixel points are V11[m,x％m+1，0]、V12[m,x％m，0]、V13[x/m,x％m，0]And V14[x/m,x％m+1，0](ii) a The second video is a grid model to be spliced on the right side, and the coordinate of the grid vertex of the pixel point to be spliced on the right side is V21[x/m+m+1,x％m+1，0]、V22[x/m+1+m,x％m，0]、V23[m,x％m，0]And V24[m,x％m+1，0]。
9. The apparatus of claim 8, wherein the video stitching module comprises:
a first processing unit, configured to set a point in the left pixel coordinate set to be stitched, where the corresponding grid to be blanked is set to be V13[x/m,x％m，0]And V11[m,x％m+1，0]A defined grid;
the second processing unit sets the point in the coordinate set of the pixel points to be spliced on the right side and the corresponding grid to be blanked as V23[m,x％m，0]And V21[x/m+m+1,x％m+1，0]A defined grid;
the third processing unit is used for respectively setting the transparency of the vertexes of the blanking grids in the pixel point coordinate sets to be spliced on the left side and the right side to be 0;
and the splicing unit is used for translating the grid model to be spliced on the right side leftwards, and the translation amount is the difference value of the horizontal coordinates of the pixel points to be spliced on the right side and the pixel points to be spliced on the left side corresponding to the pixel points to be spliced on the right side.
10. A real-time video stitching device, comprising:
a memory for storing a program;
a processor for implementing the method of any one of claims 1-5 by executing a program stored by the memory.
CN202010260631.2A 2020-04-03 2020-04-03 Real-time video stitching method and device Active CN111476716B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010260631.2A CN111476716B (en) 2020-04-03 2020-04-03 Real-time video stitching method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010260631.2A CN111476716B (en) 2020-04-03 2020-04-03 Real-time video stitching method and device

Publications (2)

Publication Number Publication Date
CN111476716A true CN111476716A (en) 2020-07-31
CN111476716B CN111476716B (en) 2023-09-26

Family

ID=71750467

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010260631.2A Active CN111476716B (en) 2020-04-03 2020-04-03 Real-time video stitching method and device

Country Status (1)

Country Link
CN (1) CN111476716B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115086575A (en) * 2022-08-16 2022-09-20 之江实验室 Video picture splicing method and device based on unmanned vehicle remote driving

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103096043A (en) * 2013-02-21 2013-05-08 安徽大学 Mine safety monitoring method based on parallel video splicing technology
CN107392848A (en) * 2017-06-14 2017-11-24 江西科技师范大学 Panoramic image display method and device
CN107734268A (en) * 2017-09-18 2018-02-23 北京航空航天大学 A kind of structure-preserved wide baseline video joining method
CN108700798A (en) * 2016-01-03 2018-10-23 人眼技术有限公司 Frame during creating panoramic frame adapts to splicing
CN110136064A (en) * 2019-05-14 2019-08-16 武汉高德红外股份有限公司 A kind of image split-joint method and system based on transparent figure layer technology

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103096043A (en) * 2013-02-21 2013-05-08 安徽大学 Mine safety monitoring method based on parallel video splicing technology
CN108700798A (en) * 2016-01-03 2018-10-23 人眼技术有限公司 Frame during creating panoramic frame adapts to splicing
CN107392848A (en) * 2017-06-14 2017-11-24 江西科技师范大学 Panoramic image display method and device
CN107734268A (en) * 2017-09-18 2018-02-23 北京航空航天大学 A kind of structure-preserved wide baseline video joining method
CN110136064A (en) * 2019-05-14 2019-08-16 武汉高德红外股份有限公司 A kind of image split-joint method and system based on transparent figure layer technology

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
WEI LYU等: "A survey on image and video stitching" *
何川等: "具有直线结构保护的网格化图像拼接", 《中国图象图形学报》 *
左森等: "宽基线视差图像的拼接算法", 《计算机工程》 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115086575A (en) * 2022-08-16 2022-09-20 之江实验室 Video picture splicing method and device based on unmanned vehicle remote driving

Also Published As

Publication number Publication date
CN111476716B (en) 2023-09-26

Similar Documents

Publication Publication Date Title
CN109547766B (en) Panoramic image generation method and device
CN110009561B (en) Method and system for mapping surveillance video target to three-dimensional geographic scene model
CN109242961B (en) Face modeling method and device, electronic equipment and computer readable medium
EP3534336B1 (en) Panoramic image generating method and apparatus
CN108648257B (en) Panoramic picture acquisition method and device, storage medium and electronic device
CN103198488B (en) PTZ surveillance camera realtime posture rapid estimation
CN112437276B (en) WebGL-based three-dimensional video fusion method and system
JP2005339313A (en) Method and apparatus for presenting image
JP7484055B2 (en) Method, device, storage medium, electronic device, and computer program for generating panoramic image with depth information
CN111866523B (en) Panoramic video synthesis method and device, electronic equipment and computer storage medium
CN109064533B (en) 3D roaming method and system
Boliek et al. Next generation image compression and manipulation using CREW
CN111161130B (en) Video correction method based on three-dimensional geographic information
CN113379901A (en) Method and system for establishing house live-action three-dimension by utilizing public self-photographing panoramic data
CN110458960A (en) A kind of color body three-dimensional rebuilding method based on polarization
CN106485781A (en) Three-dimensional scene construction method based on live video stream and its system
CN110807413B (en) Target display method and related device
CN111091491B (en) Panoramic video pixel redistribution method and system for equidistant cylindrical projection
CN112351266A (en) Three-dimensional visual processing method, device, equipment, display system and medium
CN107958489B (en) Curved surface reconstruction method and device
CN114926612A (en) Aerial panoramic image processing and immersive display system
CN111476716A (en) Real-time video splicing method and device
WO2018234817A1 (en) Apparatus and method for stitching together multiple images
CN113132708A (en) Method and apparatus for acquiring three-dimensional scene image using fisheye camera, device and medium
CN112866507B (en) Intelligent panoramic video synthesis method and system, electronic device and medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant