CN112584060A - Video fusion system - Google Patents
Video fusion system Download PDFInfo
- Publication number
- CN112584060A CN112584060A CN202011478934.8A CN202011478934A CN112584060A CN 112584060 A CN112584060 A CN 112584060A CN 202011478934 A CN202011478934 A CN 202011478934A CN 112584060 A CN112584060 A CN 112584060A
- Authority
- CN
- China
- Prior art keywords
- video
- unit
- dimensional model
- picture
- adjusting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 25
- 238000012544 monitoring process Methods 0.000 claims abstract description 71
- 230000000694 effects Effects 0.000 claims abstract description 19
- 238000012545 processing Methods 0.000 claims abstract description 15
- 238000012937 correction Methods 0.000 claims description 29
- 239000000463 material Substances 0.000 claims description 24
- 238000000034 method Methods 0.000 claims description 17
- 230000000007 visual effect Effects 0.000 claims description 8
- 238000006243 chemical reaction Methods 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 5
- 210000004534 cecum Anatomy 0.000 claims description 4
- 239000000126 substance Substances 0.000 claims description 3
- 241001464837 Viridiplantae Species 0.000 description 4
- 229910052792 caesium Inorganic materials 0.000 description 3
- TVFDJXOCXUVLDH-UHFFFAOYSA-N caesium atom Chemical compound [Cs] TVFDJXOCXUVLDH-UHFFFAOYSA-N 0.000 description 3
- 238000007500 overflow downdraw method Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- GZPBVLUEICLBOA-UHFFFAOYSA-N 4-(dimethylamino)-3,5-dimethylphenol Chemical compound CN(C)C1=C(C)C=C(O)C=C1C GZPBVLUEICLBOA-UHFFFAOYSA-N 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/2628—Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/05—Geographic models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/65—Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/75—Media network packet handling
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/275—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/327—Calibration thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/363—Image reproducers using image projection screens
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/268—Signal distribution or switching
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/08—Indexing scheme for image data processing or generation, in general involving all processing steps from image acquisition to 3D model generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2210/00—Indexing scheme for image generation or computer graphics
- G06T2210/32—Image data format
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Architecture (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Remote Sensing (AREA)
- Closed-Circuit Television Systems (AREA)
Abstract
The invention belongs to the technical field related to virtual and real fusion video monitoring, and particularly relates to a video fusion system, which comprises: the device comprises a three-dimensional modeling module, an effect adjusting module, a video stream processing module, a video deformity correcting module and a video projection module; according to the video fusion technology, the video frames are extracted from the monitoring video data stream and projected into the virtual scene, so that full-time-space three-dimensional fusion of the video data and the virtual scene data is realized, the traditional mode that map application can only be statically displayed is changed, the multi-channel real-time monitoring videos deployed at different geographic positions are registered and fused with the three-dimensional model of the monitoring area, a large-range three-dimensional panoramic dynamic monitoring picture is generated, and real-time global control of the whole safety situation of the monitoring area is realized.
Description
Technical Field
The invention belongs to the technical field related to virtual and real fusion video monitoring, and particularly relates to a video fusion system.
Background
At present, the application of the traditional video monitoring technology mainly takes a planar matrix type video monitoring picture as a main part, and with the development and popularization and application of a Web information system technology, a GIS geographic information system technology, an H5 video streaming technology and a three-dimensional modeling technology, users put higher requirements on the implementation means of video monitoring, and hope to realize an innovative application mode combining video monitoring and a virtual space scene.
Disclosure of Invention
Technical problem to be solved
The technical problem to be solved by the invention is as follows: how to put forward a video monitoring scheme based on the combination of video data of a Web information system and virtual scenes of a GIS (geographic information system) to fuse video monitoring pictures and the virtual scenes, can view the whole situation and reduce the complexity of monitoring operation.
(II) technical scheme
In order to solve the above technical problem, the present invention provides a video fusion system, including: the device comprises a three-dimensional modeling module, an effect adjusting module, a video stream processing module, a video deformity correcting module and a video projection module;
wherein the content of the first and second substances,
the three-dimensional modeling module is used for controlling the unmanned aerial vehicle to carry digital photos of a modeling area of a five-mesh digital camera shooting target by adopting an unmanned aerial vehicle aerial oblique photography modeling method, and converting the digital photos into a three-dimensional model by using a three-dimensional model generation tool;
the effect adjusting module is used for loading the three-dimensional model to a geographic information system, adjusting the three-dimensional model, and building a virtual scene similar to reality, wherein the three-dimensional model comprises model simplification, three-dimensional coordinate setting, position adjustment and scale adjustment;
the video Stream processing module is used for pulling an RTSP video Stream from a video monitoring camera and a video server device, and converting the RTSP video Stream into a video Stream supporting an Html5 protocol through an H5Stream service;
the video deformity correction module is used for adjusting a video monitoring picture of a video code stream with wide-angle parameters and barrel distortion by adopting a camera distortion correction tool in an OPENCV algorithm library and applying a checkerboard picture correction parameter method, cutting and intercepting an effective view of the video monitoring picture, and matching the video monitoring picture with a spatial position in a virtual scene;
the video projection module is used for constructing a polygonal plane as a video projection carrier on a corresponding coordinate point in a virtual scene according to an effective visual field of a video monitoring picture, and then loading a video code stream supporting an Html5 protocol as a material of the polygonal plane; when the video is projected, multiple paths of independent videos need to be spliced to form an integral scene, and adjacent polygon blocks are subjected to position and size adjustment until edge areas of the adjacent polygon blocks are continuous without gaps.
Wherein the three-dimensional modeling module comprises: the system comprises a flight path planning unit, an unmanned aerial vehicle control unit, a photo importing unit and a three-dimensional model correcting unit;
the flight path planning unit is used for planning a modeling area, collating coordinate information, planning a horizontal flight path of the unmanned aerial vehicle, designating a flight height and covering the modeling area by the flight area;
the unmanned aerial vehicle control unit is used for controlling an unmanned aerial vehicle to carry a five-mesh digital camera to fly according to the flight route planned by the flight route planning unit, and the five-mesh digital camera is used for periodically and continuously shooting ground photos;
the photo importing unit is used for uniformly importing ground photos shot by the unmanned aerial vehicle into a three-dimensional model generating tool to generate a three-dimensional model of oblique photography;
the three-dimensional model correction unit is used for checking the oblique photography three-dimensional model generated by the three-dimensional model generation tool and controlling the unmanned aerial vehicle to take the key area live-action photos to finely correct the material and texture of the model.
Wherein the three-dimensional model generation tool is a Smart3D software tool.
Wherein the modeling area comprises buildings, roads, green plants and fixed facilities.
Wherein, the ground photo is in a JPG format.
Wherein the effect adjusting module comprises: a tool selection unit, a data loading unit, a model loading and adjusting unit,
The tool selection unit is used for selecting a Cecum digital earth engine supporting WebGL and building a GIS geographic information system environment;
the data loading unit is used for loading the satellite forward-looking image data to the GIS in the step 21 to serve as a basic layer to form a digital earth scene;
the model loading and adjusting unit is used for converting the three-dimensional model data of the oblique photography output by the three-dimensional modeling module into an Obj file format and loading the data to the digital earth scene; the satellite orthographic image matching system is used for adjusting the space coordinate parameters of the three-dimensional model until the space position of the three-dimensional model is matched with the orthographic image of the satellite in a contrast mode; and adjusting the scale of the three-dimensional model to be scaled in an equal proportion until the size of the three-dimensional model is close to the real scene.
The GIS geographic information system adopts an open source Cesium platform.
Wherein the video stream processing module comprises: the device comprises a first address configuration unit and a video stream conversion unit;
the address configuration unit is used for configuring an RTSP main code stream address of each camera for the installed cameras; the system is used for configuring the RTSP main code Stream address of each camera into an H5Stream service configuration file;
the video Stream conversion unit is used for starting an H5Stream service to convert an RTSP video Stream into a video Stream supporting an Html5 protocol, generating an Html5 video Stream address of each camera, and simultaneously accessing a plurality of paths of video Stream preview pictures by using a browser.
Wherein the effect adjusting module comprises: a second address configuration unit, a checkerboard picture setting and calibrating unit, a picture correcting unit and a picture cutting unit;
the second address configuration unit is used for configuring the Html5 video stream address of the video camera to the video stream input end of the camera distortion correction tool in the OPENCV algorithm library;
the checkerboard picture setting and calibrating unit is used for placing the printed 8 x 16 black and white checkerboard A3 picture into a video picture to be corrected and calibrating the intersection point of each checkerboard;
the picture correcting unit is used for restoring the calibration points corresponding to the intersection points of the unit grids into a matrix with equal interval distribution by uniformly stretching the video monitoring picture according to the calibrated intersection points of the unit grids, so as to realize the correction of the barrel-type distortion of the video stream;
the picture cutting unit is used for cutting the video monitoring picture after distortion correction by using a rectangular tool to generate a video code stream output address of an effective view field.
Wherein the video projection module comprises: the system comprises a polygonal plane creating and adjusting unit, a skin material loading unit and a scene splicing unit;
the polygon plane creating and adjusting unit is used for creating a polygon plane in a digital earth scene loaded with a three-dimensional model, and adjusting the size and the position of the polygon plane by referring to a video monitoring picture of an effective view; adjusting the height of the polygon plane to match the spatial position of the three-dimensional model of the corresponding region;
the skin material loading unit is used for setting polygonal plane material parameters and loading the video monitoring picture of the effective vision field as the skin material of the polygonal plane;
the scene splicing unit is used for adjusting the position and the size of the adjacent polygonal blocks until the edge areas of the adjacent polygonal blocks are continuous without gaps, so that the polygons are spliced to form a continuous large-scene video monitoring picture.
(III) advantageous effects
Compared with the prior art, the invention provides a video monitoring scheme combining video data based on a Web information system and a GIS (geographic information system) geographic information system virtual scene, so that a video monitoring picture and the virtual scene are fused, the overall situation can be seen, and the monitoring operation complexity is reduced.
The technical scheme of the invention is that a three-dimensional modeling is carried out on buildings, roads, green plants, fixed facilities and the like in a management area based on a geographic information system to form a three-dimensional model; putting the three-dimensional model into a geographic information system to adjust the model effect to form a virtual scene; the method comprises the steps that RTSP video streams are pulled from devices such as a video monitoring camera and a video server, and the RTSP video streams are converted into video code streams supporting an Html5 protocol; correcting the video with wide-angle parameters and barrel distortion into a corrected video matched with a spatial position in a high-precision manner; and projecting the corrected video to a virtual scene to form a virtual-real fusion scene. Meanwhile, the video fusion technology adopted by the invention extracts the video frame from the monitoring video data stream and projects the video frame into the virtual scene, so that full-time-space three-dimensional fusion of the video data and the virtual scene data is realized, the traditional mode that map application can only be statically displayed is changed, the multi-channel real-time monitoring videos deployed at different geographic positions are registered and fused with the three-dimensional model of the monitoring area, a large-range three-dimensional panoramic dynamic monitoring picture is generated, and the real-time global control of the overall security situation of the monitoring area is realized.
Drawings
Fig. 1 is a technical processing flow chart in the technical scheme of the invention.
Detailed Description
In order to make the objects, contents, and advantages of the present invention clearer, the following detailed description of the embodiments of the present invention will be made in conjunction with the accompanying drawings and examples.
In order to solve the above technical problem, the present invention provides a video fusion system, including: the device comprises a three-dimensional modeling module, an effect adjusting module, a video stream processing module, a video deformity correcting module and a video projection module;
wherein the content of the first and second substances,
the three-dimensional modeling module is used for controlling the unmanned aerial vehicle to carry digital photos of a modeling area of a five-mesh digital camera shooting target by adopting an unmanned aerial vehicle aerial oblique photography modeling method, and converting the digital photos into a three-dimensional model by using a three-dimensional model generation tool;
the effect adjusting module is used for loading the three-dimensional model to a geographic information system, adjusting the three-dimensional model, and building a virtual scene similar to reality, wherein the three-dimensional model comprises model simplification, three-dimensional coordinate setting, position adjustment and scale adjustment;
the video Stream processing module is used for pulling an RTSP video Stream from a video monitoring camera and a video server device, and converting the RTSP video Stream into a video Stream supporting an Html5 protocol through an H5Stream service;
the video deformity correction module is used for adjusting a video monitoring picture of a video code stream with wide-angle parameters and barrel distortion by adopting a camera distortion correction tool in an OPENCV algorithm library and applying a checkerboard picture correction parameter method, cutting and intercepting an effective view of the video monitoring picture, and matching the video monitoring picture with a spatial position in a virtual scene;
the video projection module is used for constructing a polygonal plane as a video projection carrier on a corresponding coordinate point in a virtual scene according to an effective visual field of a video monitoring picture, and then loading a video code stream supporting an Html5 protocol as a material of the polygonal plane; when the video is projected, multiple paths of independent videos need to be spliced to form an integral scene, and adjacent polygon blocks are subjected to position and size adjustment until edge areas of the adjacent polygon blocks are continuous without gaps.
Wherein the three-dimensional modeling module comprises: the system comprises a flight path planning unit, an unmanned aerial vehicle control unit, a photo importing unit and a three-dimensional model correcting unit;
the flight path planning unit is used for planning a modeling area, collating coordinate information, planning a horizontal flight path of the unmanned aerial vehicle, designating a flight height and covering the modeling area by the flight area;
the unmanned aerial vehicle control unit is used for controlling an unmanned aerial vehicle to carry a five-mesh digital camera to fly according to the flight route planned by the flight route planning unit, and the five-mesh digital camera is used for periodically and continuously shooting ground photos;
the photo importing unit is used for uniformly importing ground photos shot by the unmanned aerial vehicle into a three-dimensional model generating tool to generate a three-dimensional model of oblique photography;
the three-dimensional model correction unit is used for checking the oblique photography three-dimensional model generated by the three-dimensional model generation tool and controlling the unmanned aerial vehicle to take the key area live-action photos to finely correct the material and texture of the model.
Wherein the three-dimensional model generation tool is a Smart3D software tool.
Wherein the modeling area comprises buildings, roads, green plants and fixed facilities.
Wherein, the ground photo is in a JPG format.
Wherein the effect adjusting module comprises: a tool selection unit, a data loading unit, a model loading and adjusting unit,
The tool selection unit is used for selecting a Cecum digital earth engine supporting WebGL and building a GIS geographic information system environment;
the data loading unit is used for loading the satellite forward-looking image data to the GIS in the step 21 to serve as a basic layer to form a digital earth scene;
the model loading and adjusting unit is used for converting the three-dimensional model data of the oblique photography output by the three-dimensional modeling module into an Obj file format and loading the data to the digital earth scene; the satellite orthographic image matching system is used for adjusting the space coordinate parameters of the three-dimensional model until the space position of the three-dimensional model is matched with the orthographic image of the satellite in a contrast mode; and adjusting the scale of the three-dimensional model to be scaled in an equal proportion until the size of the three-dimensional model is close to the real scene.
The GIS geographic information system adopts an open source Cesium platform.
Wherein the video stream processing module comprises: the device comprises a first address configuration unit and a video stream conversion unit;
the address configuration unit is used for configuring an RTSP main code stream address of each camera for the installed cameras; the system is used for configuring the RTSP main code Stream address of each camera into an H5Stream service configuration file;
the video Stream conversion unit is used for starting an H5Stream service to convert an RTSP video Stream into a video Stream supporting an Html5 protocol, generating an Html5 video Stream address of each camera, and simultaneously accessing a plurality of paths of video Stream preview pictures by using a browser.
Wherein the effect adjusting module comprises: a second address configuration unit, a checkerboard picture setting and calibrating unit, a picture correcting unit and a picture cutting unit;
the second address configuration unit is used for configuring the Html5 video stream address of the video camera to the video stream input end of the camera distortion correction tool in the OPENCV algorithm library;
the checkerboard picture setting and calibrating unit is used for placing the printed 8 x 16 black and white checkerboard A3 picture into a video picture to be corrected and calibrating the intersection point of each checkerboard;
the picture correcting unit is used for restoring the calibration points corresponding to the intersection points of the unit grids into a matrix with equal interval distribution by uniformly stretching the video monitoring picture according to the calibrated intersection points of the unit grids, so as to realize the correction of the barrel-type distortion of the video stream;
the picture cutting unit is used for cutting the video monitoring picture after distortion correction by using a rectangular tool to generate a video code stream output address of an effective view field.
Wherein the video projection module comprises: the system comprises a polygonal plane creating and adjusting unit, a skin material loading unit and a scene splicing unit;
the polygon plane creating and adjusting unit is used for creating a polygon plane in a digital earth scene loaded with a three-dimensional model, and adjusting the size and the position of the polygon plane by referring to a video monitoring picture of an effective view; adjusting the height of the polygon plane to match the spatial position of the three-dimensional model of the corresponding region;
the skin material loading unit is used for setting polygonal plane material parameters and loading the video monitoring picture of the effective vision field as the skin material of the polygonal plane;
the scene splicing unit is used for adjusting the position and the size of the adjacent polygonal blocks until the edge areas of the adjacent polygonal blocks are continuous without gaps, so that the polygons are spliced to form a continuous large-scene video monitoring picture.
In addition, the invention also provides a video fusion method, which is applied to users with higher requirements on video monitoring safety and protection, such as military colleges, army bases, scientific research institutes and the like, and as shown in fig. 1, the video fusion method comprises the following steps:
step 1: three-dimensional modeling;
the method for modeling by aerial photography oblique photography of the unmanned aerial vehicle is adopted, the digital photo of a modeling area of a target is shot by carrying a five-mesh digital camera through the unmanned aerial vehicle, and the digital photo is converted into a three-dimensional model by using a three-dimensional model generation tool;
step 2: adjusting the effect;
loading the three-dimensional model to a geographic information system, and adjusting the three-dimensional model, wherein the three-dimensional model comprises model simplification, three-dimensional coordinate setting, position adjustment and scale adjustment, so as to build a virtual scene similar to reality;
and step 3: processing a video stream;
the method comprises the steps that RTSP video streams are pulled from a video monitoring camera and a video server device, and are converted into video streams supporting an Html5 protocol through an H5Stream service;
and 4, step 4: correcting video deformity;
adjusting a video monitoring picture of a video code stream with wide-angle parameters and barrel distortion by using a camera distortion correction tool in an OPENCV algorithm library and a checkerboard picture correction parameter method, cutting and intercepting an effective visual field of the video monitoring picture, and performing high-precision matching on the video monitoring picture and a spatial position in a virtual scene;
and 5: video projection;
constructing a polygonal plane as a video projection carrier on a corresponding coordinate point in a virtual scene by referring to an effective visual field of a video monitoring picture, and then loading a video code stream supporting an Html5 protocol as a material of the polygonal plane; when the video is projected, multiple paths of independent videos need to be spliced to form an integral scene, and adjacent polygon blocks are subjected to position and size adjustment until edge areas of the adjacent polygon blocks are continuous without gaps.
Wherein, the three-dimensional modeling process of the step 1 comprises the following steps:
step 11: planning a modeling area, sorting coordinate information, planning a horizontal flight route of the unmanned aerial vehicle, designating flight height, and covering the modeling area by the flight area;
step 12: controlling an unmanned aerial vehicle to carry a five-mesh digital camera to fly according to the flying route in the step 11, and periodically and continuously shooting ground photos by using the five-mesh digital camera;
step 13: uniformly importing the ground photos shot in the step 12 into a three-dimensional model generation tool to generate a three-dimensional model of oblique photography;
step 14: and (4) checking the three-dimensional model of the oblique photography in the step (13), and performing complementary shooting on the live-action photos in the key area to perform fine correction on the material and texture of the model.
Wherein the three-dimensional model generation tool is a Smart3D software tool.
Wherein the modeling area comprises buildings, roads, green plants and fixed facilities.
Wherein, the ground photo is in a JPG format.
The source file of the three-dimensional model of oblique photography is in osgb format, and the target file is in obj format.
Wherein, the effect adjusting process of step 2 comprises the following steps:
step 21: selecting a Cecum digital earth engine supporting WebGL, and building a GIS geographic information system environment;
step 22: loading the satellite front-view image data to the GIS (geographic information system) in the step 21 to serve as a basic layer to form a digital earth scene;
step 23: converting the three-dimensional model data of oblique photography output in the step 14 into an Obj file format, and loading the three-dimensional model data into the digital earth scene in the step 22; adjusting the space coordinate parameters of the three-dimensional model until the space position of the three-dimensional model is matched with the satellite front-view image in the step 22 in a comparison manner; and adjusting the scale of the three-dimensional model to be scaled in an equal proportion until the size of the three-dimensional model is close to the real scene.
The GIS geographic information system adopts an open source Cesium platform.
Wherein, the video stream processing procedure of step 3 includes the following steps:
step 31: installing cameras, and configuring an RTSP main code stream address of each camera;
step 32: configuring the RTSP main code Stream address of each camera in the step 31 into an H5Stream service configuration file;
step 33: starting an H5Stream service to convert the RTSP video Stream into a video Stream supporting an Html5 protocol, generating an Html5 video Stream address of each camera, and simultaneously accessing a plurality of paths of video Stream preview pictures by using a browser.
Wherein, the effect adjusting process of the step 4 comprises the following steps:
step 41: configuring the Html5 video stream address of the video camera of step 33 to the video stream input end of the camera distortion correction tool in the OPENCV algorithm library;
step 42: placing the printed 8X 16 black-white checkerboard A3 picture into a video picture to be corrected, and calibrating intersection points of each unit cell of the checkerboard;
step 43: according to the cell intersection points calibrated in the step 42, through uniformly stretching the video monitoring picture, the calibration points corresponding to the cell intersection points are restored to be a matrix distributed at equal intervals, and correction of barrel distortion of the video stream is realized;
step 44: and a rectangular tool clipping step 43 is used for clipping the distorted and corrected video monitoring picture to generate a video code stream output address of the effective view.
Wherein, the video projection process of step 5 comprises the following steps:
step 51: creating a polygonal plane in the digital earth scene loaded with the three-dimensional model in step 23, and adjusting the size and position of the polygonal plane with reference to the video monitoring picture of the effective field in step 44; adjusting the height of the polygon plane to match the spatial position of the three-dimensional model of the corresponding region;
step 52: setting the material parameters of the polygonal plane, and loading the video monitoring picture of the effective visual field in the step 44 as the skin material of the polygonal plane;
step 53: and adjusting the positions and the sizes of the adjacent polygonal blocks until the edge areas of the adjacent polygonal blocks are continuous without gaps, and splicing the polygons to form a continuous large-scene video monitoring picture.
In summary, the present invention provides a video fusion method, which is applied to users with high requirements for video monitoring security in military colleges, army bases, scientific research institutions, etc., and provides a design scheme for the requirements for video monitoring of security in large public areas, closed places, campus camps, etc. A GIS system based on a WebGL rendering engine is adopted as a bottom layer to construct a virtual-real combined video fusion monitoring system, and the system design method has high advancement and feasibility.
Example 1
The present embodiment includes:
(1) three-dimensional modeling
Planning the flight route and the flight height of the unmanned aerial vehicle, covering a modeling target area, controlling the unmanned aerial vehicle to fly along the planning route and shooting pictures by adopting an onboard five-eye camera according to the set shooting frequency. After the flight is finished, the photos are completely imported into SMART3D software for data processing and model generation. And (4) previewing the scene after the data generation is successful, performing on-site rephotography on the key area by using a digital camera, and performing material correction on the model by adopting 3 DMAX. And exporting to an obj format file.
(2) Effect adjustment
And loading the obj file of the three-dimensional model to a geographic information system, adjusting the parameter information of the three-dimensional model, including three-dimensional coordinate setting, position adjustment, scale adjustment, texture adjustment, model simplification and the like, wherein the model is attached to the position of the front view image layer of the satellite, and the height of the model is attached to the real terrain so as to build a real virtual scene.
(3) Video stream processing
The RTSP video stream is pulled from equipment such as a video monitoring camera and a video server, and video stream configuration information including an IP address, a login user password, a channel number and the like is recorded; and filling the video Stream information into a conf configuration file of the H5Stream service according to a specified format, starting the H5Stream service to access a video picture through a browser, converting the RTSP video Stream into a video Stream supporting the Html5 protocol, and recording the address of the converted H5 video Stream.
(4) Video deformity correction
Configuring an H5 video stream address into an OPENCV camera distortion correction program, adjusting a monitoring picture with wide-angle parameters and barrel distortion by using a checkerboard picture correction parameter method, cutting and intercepting a video monitoring effective visual field, and performing high-precision matching on the video picture and a space position in a virtual scene.
(5) Video projection
And constructing a polygonal plane as a video projection carrier on a corresponding coordinate point in a virtual scene by referring to the effective visual field of the video monitoring picture, and then loading a video code stream supporting an Html5 protocol as a polygonal material. When the video is projected, multiple paths of independent videos need to be spliced to form an integral scene, and splicing overlapping areas of adjacent polygonal blocks need to be flexibly configured and adjusted.
The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.
Claims (10)
1. A video fusion system, the video fusion system comprising: the device comprises a three-dimensional modeling module, an effect adjusting module, a video stream processing module, a video deformity correcting module and a video projection module;
wherein the content of the first and second substances,
the three-dimensional modeling module is used for controlling the unmanned aerial vehicle to carry digital photos of a modeling area of a five-mesh digital camera shooting target by adopting an unmanned aerial vehicle aerial oblique photography modeling method, and converting the digital photos into a three-dimensional model by using a three-dimensional model generation tool;
the effect adjusting module is used for loading the three-dimensional model to a geographic information system, adjusting the three-dimensional model, and building a virtual scene similar to reality, wherein the three-dimensional model comprises model simplification, three-dimensional coordinate setting, position adjustment and scale adjustment;
the video Stream processing module is used for pulling an RTSP video Stream from a video monitoring camera and a video server device, and converting the RTSP video Stream into a video Stream supporting an Html5 protocol through an H5Stream service;
the video deformity correction module is used for adjusting a video monitoring picture of a video code stream with wide-angle parameters and barrel distortion by adopting a camera distortion correction tool in an OPENCV algorithm library and applying a checkerboard picture correction parameter method, cutting and intercepting an effective view of the video monitoring picture, and matching the video monitoring picture with a spatial position in a virtual scene;
the video projection module is used for constructing a polygonal plane as a video projection carrier on a corresponding coordinate point in a virtual scene according to an effective visual field of a video monitoring picture, and then loading a video code stream supporting an Html5 protocol as a material of the polygonal plane; when the video is projected, multiple paths of independent videos need to be spliced to form an integral scene, and adjacent polygon blocks are subjected to position and size adjustment until edge areas of the adjacent polygon blocks are continuous without gaps.
2. The video fusion system of claim 1 wherein the three-dimensional modeling module comprises: the system comprises a flight path planning unit, an unmanned aerial vehicle control unit, a photo importing unit and a three-dimensional model correcting unit;
the flight path planning unit is used for planning a modeling area, collating coordinate information, planning a horizontal flight path of the unmanned aerial vehicle, designating a flight height and covering the modeling area by the flight area;
the unmanned aerial vehicle control unit is used for controlling an unmanned aerial vehicle to carry a five-mesh digital camera to fly according to the flight route planned by the flight route planning unit, and the five-mesh digital camera is used for periodically and continuously shooting ground photos;
the photo importing unit is used for uniformly importing ground photos shot by the unmanned aerial vehicle into a three-dimensional model generating tool to generate a three-dimensional model of oblique photography;
the three-dimensional model correction unit is used for checking the oblique photography three-dimensional model generated by the three-dimensional model generation tool and controlling the unmanned aerial vehicle to take the key area live-action photos to finely correct the material and texture of the model.
3. The video fusion system of claim 2 wherein the three-dimensional model generation tool is a Smart3D software tool.
4. The video fusion system of claim 2 wherein the modeled area includes buildings, roads, greenery, stationary facilities.
5. The video fusion system of claim 2 wherein the ground photographs taken are in JPG format.
6. The video fusion system of claim 2 wherein the effect adjustment module comprises: a tool selection unit, a data loading unit, a model loading and adjusting unit,
The tool selection unit is used for selecting a Cecum digital earth engine supporting WebGL and building a GIS geographic information system environment;
the data loading unit is used for loading the satellite forward-looking image data to the GIS in the step 21 to serve as a basic layer to form a digital earth scene;
the model loading and adjusting unit is used for converting the three-dimensional model data of the oblique photography output by the three-dimensional modeling module into an Obj file format and loading the data to the digital earth scene; the satellite orthographic image matching system is used for adjusting the space coordinate parameters of the three-dimensional model until the space position of the three-dimensional model is matched with the orthographic image of the satellite in a contrast mode; and adjusting the scale of the three-dimensional model to be scaled in an equal proportion until the size of the three-dimensional model is close to the real scene.
7. The video fusion system of claim 6 wherein the GIS geographic information system employs an open source cesum platform.
8. The video fusion system of claim 3 wherein the video stream processing module comprises: the device comprises a first address configuration unit and a video stream conversion unit;
the address configuration unit is used for configuring an RTSP main code stream address of each camera for the installed cameras; the system is used for configuring the RTSP main code Stream address of each camera into an H5Stream service configuration file;
the video Stream conversion unit is used for starting an H5Stream service to convert an RTSP video Stream into a video Stream supporting an Html5 protocol, generating an Html5 video Stream address of each camera, and simultaneously accessing a plurality of paths of video Stream preview pictures by using a browser.
9. The video fusion system of claim 8 wherein the effect adjustment module comprises: a second address configuration unit, a checkerboard picture setting and calibrating unit, a picture correcting unit and a picture cutting unit;
the second address configuration unit is used for configuring the Html5 video stream address of the video camera to the video stream input end of the camera distortion correction tool in the OPENCV algorithm library;
the checkerboard picture setting and calibrating unit is used for placing the printed 8 x 16 black and white checkerboard A3 picture into a video picture to be corrected and calibrating the intersection point of each checkerboard;
the picture correcting unit is used for restoring the calibration points corresponding to the intersection points of the unit grids into a matrix with equal interval distribution by uniformly stretching the video monitoring picture according to the calibrated intersection points of the unit grids, so as to realize the correction of the barrel-type distortion of the video stream;
the picture cutting unit is used for cutting the video monitoring picture after distortion correction by using a rectangular tool to generate a video code stream output address of an effective view field.
10. The video fusion system of claim 9, wherein the video projection module comprises: the system comprises a polygonal plane creating and adjusting unit, a skin material loading unit and a scene splicing unit;
the polygon plane creating and adjusting unit is used for creating a polygon plane in a digital earth scene loaded with a three-dimensional model, and adjusting the size and the position of the polygon plane by referring to a video monitoring picture of an effective view; adjusting the height of the polygon plane to match the spatial position of the three-dimensional model of the corresponding region;
the skin material loading unit is used for setting polygonal plane material parameters and loading the video monitoring picture of the effective vision field as the skin material of the polygonal plane;
the scene splicing unit is used for adjusting the position and the size of the adjacent polygonal blocks until the edge areas of the adjacent polygonal blocks are continuous without gaps, so that the polygons are spliced to form a continuous large-scene video monitoring picture.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011478934.8A CN112584060A (en) | 2020-12-15 | 2020-12-15 | Video fusion system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011478934.8A CN112584060A (en) | 2020-12-15 | 2020-12-15 | Video fusion system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112584060A true CN112584060A (en) | 2021-03-30 |
Family
ID=75135069
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011478934.8A Pending CN112584060A (en) | 2020-12-15 | 2020-12-15 | Video fusion system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112584060A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113538681A (en) * | 2021-07-14 | 2021-10-22 | 云南电网有限责任公司曲靖供电局 | Three-dimensional reconstruction method and device for dynamic real scene of transformer substation, electronic equipment and medium |
| CN113592326A (en) * | 2021-08-05 | 2021-11-02 | 合肥永信科翔智能技术有限公司 | Remote intelligent monitoring system suitable for wisdom mill |
| CN114140593A (en) * | 2021-12-02 | 2022-03-04 | 北京清晨动力科技有限公司 | Digital earth and panorama fusion display method and device |
| CN115546377A (en) * | 2022-12-01 | 2022-12-30 | 杭州靖安科技有限公司 | Video fusion method and device, electronic equipment and storage medium |
| CN117152400A (en) * | 2023-10-30 | 2023-12-01 | 武汉苍穹融新科技有限公司 | Method and system for fusing multiple paths of continuous videos and three-dimensional twin scenes on traffic road |
| CN117560578A (en) * | 2024-01-12 | 2024-02-13 | 北京睿呈时代信息科技有限公司 | Multi-channel video fusion method and system based on three-dimensional scene rendering and irrelevant to view points |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2001067749A2 (en) * | 2000-03-07 | 2001-09-13 | Sarnoff Corporation | Camera pose estimation |
| US20030085992A1 (en) * | 2000-03-07 | 2003-05-08 | Sarnoff Corporation | Method and apparatus for providing immersive surveillance |
| CN101951502A (en) * | 2010-10-19 | 2011-01-19 | 北京硅盾安全技术有限公司 | Three-dimensional intelligent video monitoring method |
| CN103226830A (en) * | 2013-04-25 | 2013-07-31 | 北京大学 | Automatic matching correction method of video texture projection in three-dimensional virtual-real fusion environment |
| CN103716586A (en) * | 2013-12-12 | 2014-04-09 | 中国科学院深圳先进技术研究院 | Monitoring video fusion system and monitoring video fusion method based on three-dimension space scene |
| US9240069B1 (en) * | 2015-06-30 | 2016-01-19 | Ariadne's Thread (Usa), Inc. | Low-latency virtual reality display system |
| CN106373148A (en) * | 2016-08-31 | 2017-02-01 | 中国科学院遥感与数字地球研究所 | Equipment and method for realizing registration and fusion of multipath video images to three-dimensional digital earth system |
| CN107835436A (en) * | 2017-09-25 | 2018-03-23 | 北京航空航天大学 | A kind of real-time virtual reality fusion live broadcast system and method based on WebGL |
| US20200126299A1 (en) * | 2018-10-18 | 2020-04-23 | Trimble Inc. | Virtual video projection system to synch animation sequences |
| CN111415416A (en) * | 2020-03-31 | 2020-07-14 | 武汉大学 | Method and system for fusing monitoring real-time video and scene three-dimensional model |
| CN112053446A (en) * | 2020-07-11 | 2020-12-08 | 南京国图信息产业有限公司 | Real-time monitoring video and three-dimensional scene fusion method based on three-dimensional GIS |
-
2020
- 2020-12-15 CN CN202011478934.8A patent/CN112584060A/en active Pending
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2001067749A2 (en) * | 2000-03-07 | 2001-09-13 | Sarnoff Corporation | Camera pose estimation |
| US20030085992A1 (en) * | 2000-03-07 | 2003-05-08 | Sarnoff Corporation | Method and apparatus for providing immersive surveillance |
| CN101951502A (en) * | 2010-10-19 | 2011-01-19 | 北京硅盾安全技术有限公司 | Three-dimensional intelligent video monitoring method |
| CN103226830A (en) * | 2013-04-25 | 2013-07-31 | 北京大学 | Automatic matching correction method of video texture projection in three-dimensional virtual-real fusion environment |
| CN103716586A (en) * | 2013-12-12 | 2014-04-09 | 中国科学院深圳先进技术研究院 | Monitoring video fusion system and monitoring video fusion method based on three-dimension space scene |
| US9240069B1 (en) * | 2015-06-30 | 2016-01-19 | Ariadne's Thread (Usa), Inc. | Low-latency virtual reality display system |
| CN106373148A (en) * | 2016-08-31 | 2017-02-01 | 中国科学院遥感与数字地球研究所 | Equipment and method for realizing registration and fusion of multipath video images to three-dimensional digital earth system |
| CN107835436A (en) * | 2017-09-25 | 2018-03-23 | 北京航空航天大学 | A kind of real-time virtual reality fusion live broadcast system and method based on WebGL |
| US20200126299A1 (en) * | 2018-10-18 | 2020-04-23 | Trimble Inc. | Virtual video projection system to synch animation sequences |
| CN111415416A (en) * | 2020-03-31 | 2020-07-14 | 武汉大学 | Method and system for fusing monitoring real-time video and scene three-dimensional model |
| CN112053446A (en) * | 2020-07-11 | 2020-12-08 | 南京国图信息产业有限公司 | Real-time monitoring video and three-dimensional scene fusion method based on three-dimensional GIS |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113538681A (en) * | 2021-07-14 | 2021-10-22 | 云南电网有限责任公司曲靖供电局 | Three-dimensional reconstruction method and device for dynamic real scene of transformer substation, electronic equipment and medium |
| CN113592326A (en) * | 2021-08-05 | 2021-11-02 | 合肥永信科翔智能技术有限公司 | Remote intelligent monitoring system suitable for wisdom mill |
| CN114140593A (en) * | 2021-12-02 | 2022-03-04 | 北京清晨动力科技有限公司 | Digital earth and panorama fusion display method and device |
| CN114140593B (en) * | 2021-12-02 | 2022-06-14 | 北京清晨动力科技有限公司 | Digital earth and panorama fusion display method and device |
| CN115546377A (en) * | 2022-12-01 | 2022-12-30 | 杭州靖安科技有限公司 | Video fusion method and device, electronic equipment and storage medium |
| CN117152400A (en) * | 2023-10-30 | 2023-12-01 | 武汉苍穹融新科技有限公司 | Method and system for fusing multiple paths of continuous videos and three-dimensional twin scenes on traffic road |
| CN117152400B (en) * | 2023-10-30 | 2024-03-19 | 武汉苍穹融新科技有限公司 | Method and system for fusing multiple paths of continuous videos and three-dimensional twin scenes on traffic road |
| CN117560578A (en) * | 2024-01-12 | 2024-02-13 | 北京睿呈时代信息科技有限公司 | Multi-channel video fusion method and system based on three-dimensional scene rendering and irrelevant to view points |
| CN117560578B (en) * | 2024-01-12 | 2024-04-16 | 北京睿呈时代信息科技有限公司 | Multi-channel video fusion method and system based on three-dimensional scene rendering and irrelevant to view points |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112584120A (en) | Video fusion method | |
| CN112584060A (en) | Video fusion system | |
| CN109872401B (en) | Unmanned aerial vehicle video augmented reality implementation method | |
| US7961983B2 (en) | Generating gigapixel images | |
| US10748243B2 (en) | Image distortion transformation method and apparatus | |
| JP5054971B2 (en) | Digital 3D / 360 degree camera system | |
| US20050271299A1 (en) | Image transforming method, image transforming device and multiprojection system | |
| US8963943B2 (en) | Three-dimensional urban modeling apparatus and method | |
| CN107067447B (en) | Integrated video monitoring method for large spatial region | |
| US20060165310A1 (en) | Method and apparatus for a virtual scene previewing system | |
| JPH07319091A (en) | Method and equipment for controlling distortion of projectedpicture | |
| CN112437276A (en) | WebGL-based three-dimensional video fusion method and system | |
| KR20180130911A (en) | Smart augmented reality service system and method based on virtual studio | |
| CN105005960A (en) | Method, apparatus and system for obtaining watermarking picture | |
| CN108322625B (en) | Panoramic video production method based on panoramic image | |
| CN115641401A (en) | Construction method and related device of three-dimensional live-action model | |
| US11176716B2 (en) | Multi-source image data synchronization | |
| WO2023029551A1 (en) | Image stitching method and system based on multiple unmanned aerial vehicles | |
| KR101725024B1 (en) | System for real time making of 360 degree VR video base on lookup table and Method for using the same | |
| CN108108396B (en) | Aerial photography picture splicing management system for aircraft | |
| See et al. | Virtual reality 360 interactive panorama reproduction obstacles and issues | |
| CN114332383B (en) | Scene three-dimensional modeling method and device based on panoramic video | |
| KR101697163B1 (en) | system and method for chroma-key composing for providing three-dimensional structure effect | |
| TWI681365B (en) | Building information model and digital terrain model integrating system and integrating method thereof | |
| Whitley | Unmanned aerial vehicles (UAVs) for documenting and interpreting historical archaeological Sites: Part II—return of the drones |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210330 |
|
| RJ01 | Rejection of invention patent application after publication |