CN112584060A - Video fusion system - Google Patents

Video fusion system Download PDF

Info

Publication number
CN112584060A
CN112584060A CN202011478934.8A CN202011478934A CN112584060A CN 112584060 A CN112584060 A CN 112584060A CN 202011478934 A CN202011478934 A CN 202011478934A CN 112584060 A CN112584060 A CN 112584060A
Authority
CN
China
Prior art keywords
video
unit
dimensional model
picture
adjusting
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011478934.8A
Other languages
Chinese (zh)
Inventor
邓子超
尹严研
张明敏
李梦
徐鑫
张巧霞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jinghang Computing Communication Research Institute
Original Assignee
Beijing Jinghang Computing Communication Research Institute
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jinghang Computing Communication Research Institute filed Critical Beijing Jinghang Computing Communication Research Institute
Priority to CN202011478934.8A priority Critical patent/CN112584060A/en
Publication of CN112584060A publication Critical patent/CN112584060A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment
    • H04N5/262Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/2628Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/05Geographic models
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00Manipulating 3D models or images for computer graphics
    • G06T19/20Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformations in the plane of the image
    • G06T3/40Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038Image mosaicing, e.g. composing plane images from plane sub-images
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/80Geometric correction
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60Network streaming of media packets
    • H04L65/65Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60Network streaming of media packets
    • H04L65/75Media network packet handling
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/275Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/327Calibration thereof
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/363Image reproducers using image projection screens
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment
    • H04N5/262Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/268Signal distribution or switching
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/18Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00Indexing scheme for image data processing or generation, in general
    • G06T2200/08Indexing scheme for image data processing or generation, in general involving all processing steps from image acquisition to 3D model generation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10016Video; Image sequence
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00Indexing scheme for image generation or computer graphics
    • G06T2210/32Image data format

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Architecture (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Remote Sensing (AREA)
  • Closed-Circuit Television Systems (AREA)

Abstract

The invention belongs to the technical field related to virtual and real fusion video monitoring, and particularly relates to a video fusion system, which comprises: the device comprises a three-dimensional modeling module, an effect adjusting module, a video stream processing module, a video deformity correcting module and a video projection module; according to the video fusion technology, the video frames are extracted from the monitoring video data stream and projected into the virtual scene, so that full-time-space three-dimensional fusion of the video data and the virtual scene data is realized, the traditional mode that map application can only be statically displayed is changed, the multi-channel real-time monitoring videos deployed at different geographic positions are registered and fused with the three-dimensional model of the monitoring area, a large-range three-dimensional panoramic dynamic monitoring picture is generated, and real-time global control of the whole safety situation of the monitoring area is realized.

Description

Video fusion system
Technical Field
The invention belongs to the technical field related to virtual and real fusion video monitoring, and particularly relates to a video fusion system.
Background
At present, the application of the traditional video monitoring technology mainly takes a planar matrix type video monitoring picture as a main part, and with the development and popularization and application of a Web information system technology, a GIS geographic information system technology, an H5 video streaming technology and a three-dimensional modeling technology, users put higher requirements on the implementation means of video monitoring, and hope to realize an innovative application mode combining video monitoring and a virtual space scene.
Disclosure of Invention
Technical problem to be solved
The technical problem to be solved by the invention is as follows: how to put forward a video monitoring scheme based on the combination of video data of a Web information system and virtual scenes of a GIS (geographic information system) to fuse video monitoring pictures and the virtual scenes, can view the whole situation and reduce the complexity of monitoring operation.
(II) technical scheme
In order to solve the above technical problem, the present invention provides a video fusion system, including: the device comprises a three-dimensional modeling module, an effect adjusting module, a video stream processing module, a video deformity correcting module and a video projection module;
wherein the content of the first and second substances,
the three-dimensional modeling module is used for controlling the unmanned aerial vehicle to carry digital photos of a modeling area of a five-mesh digital camera shooting target by adopting an unmanned aerial vehicle aerial oblique photography modeling method, and converting the digital photos into a three-dimensional model by using a three-dimensional model generation tool;
the effect adjusting module is used for loading the three-dimensional model to a geographic information system, adjusting the three-dimensional model, and building a virtual scene similar to reality, wherein the three-dimensional model comprises model simplification, three-dimensional coordinate setting, position adjustment and scale adjustment;
the video Stream processing module is used for pulling an RTSP video Stream from a video monitoring camera and a video server device, and converting the RTSP video Stream into a video Stream supporting an Html5 protocol through an H5Stream service;
the video deformity correction module is used for adjusting a video monitoring picture of a video code stream with wide-angle parameters and barrel distortion by adopting a camera distortion correction tool in an OPENCV algorithm library and applying a checkerboard picture correction parameter method, cutting and intercepting an effective view of the video monitoring picture, and matching the video monitoring picture with a spatial position in a virtual scene;
the video projection module is used for constructing a polygonal plane as a video projection carrier on a corresponding coordinate point in a virtual scene according to an effective visual field of a video monitoring picture, and then loading a video code stream supporting an Html5 protocol as a material of the polygonal plane; when the video is projected, multiple paths of independent videos need to be spliced to form an integral scene, and adjacent polygon blocks are subjected to position and size adjustment until edge areas of the adjacent polygon blocks are continuous without gaps.
Wherein the three-dimensional modeling module comprises: the system comprises a flight path planning unit, an unmanned aerial vehicle control unit, a photo importing unit and a three-dimensional model correcting unit;
the flight path planning unit is used for planning a modeling area, collating coordinate information, planning a horizontal flight path of the unmanned aerial vehicle, designating a flight height and covering the modeling area by the flight area;
the unmanned aerial vehicle control unit is used for controlling an unmanned aerial vehicle to carry a five-mesh digital camera to fly according to the flight route planned by the flight route planning unit, and the five-mesh digital camera is used for periodically and continuously shooting ground photos;
the photo importing unit is used for uniformly importing ground photos shot by the unmanned aerial vehicle into a three-dimensional model generating tool to generate a three-dimensional model of oblique photography;
the three-dimensional model correction unit is used for checking the oblique photography three-dimensional model generated by the three-dimensional model generation tool and controlling the unmanned aerial vehicle to take the key area live-action photos to finely correct the material and texture of the model.
Wherein the three-dimensional model generation tool is a Smart3D software tool.
Wherein the modeling area comprises buildings, roads, green plants and fixed facilities.
Wherein, the ground photo is in a JPG format.
Wherein the effect adjusting module comprises: a tool selection unit, a data loading unit, a model loading and adjusting unit,
The tool selection unit is used for selecting a Cecum digital earth engine supporting WebGL and building a GIS geographic information system environment;
the data loading unit is used for loading the satellite forward-looking image data to the GIS in the step 21 to serve as a basic layer to form a digital earth scene;
the model loading and adjusting unit is used for converting the three-dimensional model data of the oblique photography output by the three-dimensional modeling module into an Obj file format and loading the data to the digital earth scene; the satellite orthographic image matching system is used for adjusting the space coordinate parameters of the three-dimensional model until the space position of the three-dimensional model is matched with the orthographic image of the satellite in a contrast mode; and adjusting the scale of the three-dimensional model to be scaled in an equal proportion until the size of the three-dimensional model is close to the real scene.
The GIS geographic information system adopts an open source Cesium platform.
Wherein the video stream processing module comprises: the device comprises a first address configuration unit and a video stream conversion unit;
the address configuration unit is used for configuring an RTSP main code stream address of each camera for the installed cameras; the system is used for configuring the RTSP main code Stream address of each camera into an H5Stream service configuration file;
the video Stream conversion unit is used for starting an H5Stream service to convert an RTSP video Stream into a video Stream supporting an Html5 protocol, generating an Html5 video Stream address of each camera, and simultaneously accessing a plurality of paths of video Stream preview pictures by using a browser.
Wherein the effect adjusting module comprises: a second address configuration unit, a checkerboard picture setting and calibrating unit, a picture correcting unit and a picture cutting unit;
the second address configuration unit is used for configuring the Html5 video stream address of the video camera to the video stream input end of the camera distortion correction tool in the OPENCV algorithm library;
the checkerboard picture setting and calibrating unit is used for placing the printed 8 x 16 black and white checkerboard A3 picture into a video picture to be corrected and calibrating the intersection point of each checkerboard;
the picture correcting unit is used for restoring the calibration points corresponding to the intersection points of the unit grids into a matrix with equal interval distribution by uniformly stretching the video monitoring picture according to the calibrated intersection points of the unit grids, so as to realize the correction of the barrel-type distortion of the video stream;
the picture cutting unit is used for cutting the video monitoring picture after distortion correction by using a rectangular tool to generate a video code stream output address of an effective view field.
Wherein the video projection module comprises: the system comprises a polygonal plane creating and adjusting unit, a skin material loading unit and a scene splicing unit;
the polygon plane creating and adjusting unit is used for creating a polygon plane in a digital earth scene loaded with a three-dimensional model, and adjusting the size and the position of the polygon plane by referring to a video monitoring picture of an effective view; adjusting the height of the polygon plane to match the spatial position of the three-dimensional model of the corresponding region;
the skin material loading unit is used for setting polygonal plane material parameters and loading the video monitoring picture of the effective vision field as the skin material of the polygonal plane;
the scene splicing unit is used for adjusting the position and the size of the adjacent polygonal blocks until the edge areas of the adjacent polygonal blocks are continuous without gaps, so that the polygons are spliced to form a continuous large-scene video monitoring picture.
(III) advantageous effects
Compared with the prior art, the invention provides a video monitoring scheme combining video data based on a Web information system and a GIS (geographic information system) geographic information system virtual scene, so that a video monitoring picture and the virtual scene are fused, the overall situation can be seen, and the monitoring operation complexity is reduced.
The technical scheme of the invention is that a three-dimensional modeling is carried out on buildings, roads, green plants, fixed facilities and the like in a management area based on a geographic information system to form a three-dimensional model; putting the three-dimensional model into a geographic information system to adjust the model effect to form a virtual scene; the method comprises the steps that RTSP video streams are pulled from devices such as a video monitoring camera and a video server, and the RTSP video streams are converted into video code streams supporting an Html5 protocol; correcting the video with wide-angle parameters and barrel distortion into a corrected video matched with a spatial position in a high-precision manner; and projecting the corrected video to a virtual scene to form a virtual-real fusion scene. Meanwhile, the video fusion technology adopted by the invention extracts the video frame from the monitoring video data stream and projects the video frame into the virtual scene, so that full-time-space three-dimensional fusion of the video data and the virtual scene data is realized, the traditional mode that map application can only be statically displayed is changed, the multi-channel real-time monitoring videos deployed at different geographic positions are registered and fused with the three-dimensional model of the monitoring area, a large-range three-dimensional panoramic dynamic monitoring picture is generated, and the real-time global control of the overall security situation of the monitoring area is realized.
Drawings
Fig. 1 is a technical processing flow chart in the technical scheme of the invention.
Detailed Description
In order to make the objects, contents, and advantages of the present invention clearer, the following detailed description of the embodiments of the present invention will be made in conjunction with the accompanying drawings and examples.
In order to solve the above technical problem, the present invention provides a video fusion system, including: the device comprises a three-dimensional modeling module, an effect adjusting module, a video stream processing module, a video deformity correcting module and a video projection module;
wherein the content of the first and second substances,
the three-dimensional modeling module is used for controlling the unmanned aerial vehicle to carry digital photos of a modeling area of a five-mesh digital camera shooting target by adopting an unmanned aerial vehicle aerial oblique photography modeling method, and converting the digital photos into a three-dimensional model by using a three-dimensional model generation tool;
the effect adjusting module is used for loading the three-dimensional model to a geographic information system, adjusting the three-dimensional model, and building a virtual scene similar to reality, wherein the three-dimensional model comprises model simplification, three-dimensional coordinate setting, position adjustment and scale adjustment;
the video Stream processing module is used for pulling an RTSP video Stream from a video monitoring camera and a video server device, and converting the RTSP video Stream into a video Stream supporting an Html5 protocol through an H5Stream service;
the video deformity correction module is used for adjusting a video monitoring picture of a video code stream with wide-angle parameters and barrel distortion by adopting a camera distortion correction tool in an OPENCV algorithm library and applying a checkerboard picture correction parameter method, cutting and intercepting an effective view of the video monitoring picture, and matching the video monitoring picture with a spatial position in a virtual scene;
the video projection module is used for constructing a polygonal plane as a video projection carrier on a corresponding coordinate point in a virtual scene according to an effective visual field of a video monitoring picture, and then loading a video code stream supporting an Html5 protocol as a material of the polygonal plane; when the video is projected, multiple paths of independent videos need to be spliced to form an integral scene, and adjacent polygon blocks are subjected to position and size adjustment until edge areas of the adjacent polygon blocks are continuous without gaps.
Wherein the three-dimensional modeling module comprises: the system comprises a flight path planning unit, an unmanned aerial vehicle control unit, a photo importing unit and a three-dimensional model correcting unit;
the flight path planning unit is used for planning a modeling area, collating coordinate information, planning a horizontal flight path of the unmanned aerial vehicle, designating a flight height and covering the modeling area by the flight area;
the unmanned aerial vehicle control unit is used for controlling an unmanned aerial vehicle to carry a five-mesh digital camera to fly according to the flight route planned by the flight route planning unit, and the five-mesh digital camera is used for periodically and continuously shooting ground photos;
the photo importing unit is used for uniformly importing ground photos shot by the unmanned aerial vehicle into a three-dimensional model generating tool to generate a three-dimensional model of oblique photography;
the three-dimensional model correction unit is used for checking the oblique photography three-dimensional model generated by the three-dimensional model generation tool and controlling the unmanned aerial vehicle to take the key area live-action photos to finely correct the material and texture of the model.
Wherein the three-dimensional model generation tool is a Smart3D software tool.
Wherein the modeling area comprises buildings, roads, green plants and fixed facilities.
Wherein, the ground photo is in a JPG format.
Wherein the effect adjusting module comprises: a tool selection unit, a data loading unit, a model loading and adjusting unit,
The tool selection unit is used for selecting a Cecum digital earth engine supporting WebGL and building a GIS geographic information system environment;
the data loading unit is used for loading the satellite forward-looking image data to the GIS in the step 21 to serve as a basic layer to form a digital earth scene;
the model loading and adjusting unit is used for converting the three-dimensional model data of the oblique photography output by the three-dimensional modeling module into an Obj file format and loading the data to the digital earth scene; the satellite orthographic image matching system is used for adjusting the space coordinate parameters of the three-dimensional model until the space position of the three-dimensional model is matched with the orthographic image of the satellite in a contrast mode; and adjusting the scale of the three-dimensional model to be scaled in an equal proportion until the size of the three-dimensional model is close to the real scene.
The GIS geographic information system adopts an open source Cesium platform.
Wherein the video stream processing module comprises: the device comprises a first address configuration unit and a video stream conversion unit;
the address configuration unit is used for configuring an RTSP main code stream address of each camera for the installed cameras; the system is used for configuring the RTSP main code Stream address of each camera into an H5Stream service configuration file;
the video Stream conversion unit is used for starting an H5Stream service to convert an RTSP video Stream into a video Stream supporting an Html5 protocol, generating an Html5 video Stream address of each camera, and simultaneously accessing a plurality of paths of video Stream preview pictures by using a browser.
Wherein the effect adjusting module comprises: a second address configuration unit, a checkerboard picture setting and calibrating unit, a picture correcting unit and a picture cutting unit;
the second address configuration unit is used for configuring the Html5 video stream address of the video camera to the video stream input end of the camera distortion correction tool in the OPENCV algorithm library;
the checkerboard picture setting and calibrating unit is used for placing the printed 8 x 16 black and white checkerboard A3 picture into a video picture to be corrected and calibrating the intersection point of each checkerboard;
the picture correcting unit is used for restoring the calibration points corresponding to the intersection points of the unit grids into a matrix with equal interval distribution by uniformly stretching the video monitoring picture according to the calibrated intersection points of the unit grids, so as to realize the correction of the barrel-type distortion of the video stream;
the picture cutting unit is used for cutting the video monitoring picture after distortion correction by using a rectangular tool to generate a video code stream output address of an effective view field.
Wherein the video projection module comprises: the system comprises a polygonal plane creating and adjusting unit, a skin material loading unit and a scene splicing unit;
the polygon plane creating and adjusting unit is used for creating a polygon plane in a digital earth scene loaded with a three-dimensional model, and adjusting the size and the position of the polygon plane by referring to a video monitoring picture of an effective view; adjusting the height of the polygon plane to match the spatial position of the three-dimensional model of the corresponding region;
the skin material loading unit is used for setting polygonal plane material parameters and loading the video monitoring picture of the effective vision field as the skin material of the polygonal plane;
the scene splicing unit is used for adjusting the position and the size of the adjacent polygonal blocks until the edge areas of the adjacent polygonal blocks are continuous without gaps, so that the polygons are spliced to form a continuous large-scene video monitoring picture.
In addition, the invention also provides a video fusion method, which is applied to users with higher requirements on video monitoring safety and protection, such as military colleges, army bases, scientific research institutes and the like, and as shown in fig. 1, the video fusion method comprises the following steps:
step 1: three-dimensional modeling;
the method for modeling by aerial photography oblique photography of the unmanned aerial vehicle is adopted, the digital photo of a modeling area of a target is shot by carrying a five-mesh digital camera through the unmanned aerial vehicle, and the digital photo is converted into a three-dimensional model by using a three-dimensional model generation tool;
step 2: adjusting the effect;
loading the three-dimensional model to a geographic information system, and adjusting the three-dimensional model, wherein the three-dimensional model comprises model simplification, three-dimensional coordinate setting, position adjustment and scale adjustment, so as to build a virtual scene similar to reality;
and step 3: processing a video stream;
the method comprises the steps that RTSP video streams are pulled from a video monitoring camera and a video server device, and are converted into video streams supporting an Html5 protocol through an H5Stream service;
and 4, step 4: correcting video deformity;
adjusting a video monitoring picture of a video code stream with wide-angle parameters and barrel distortion by using a camera distortion correction tool in an OPENCV algorithm library and a checkerboard picture correction parameter method, cutting and intercepting an effective visual field of the video monitoring picture, and performing high-precision matching on the video monitoring picture and a spatial position in a virtual scene;
and 5: video projection;
constructing a polygonal plane as a video projection carrier on a corresponding coordinate point in a virtual scene by referring to an effective visual field of a video monitoring picture, and then loading a video code stream supporting an Html5 protocol as a material of the polygonal plane; when the video is projected, multiple paths of independent videos need to be spliced to form an integral scene, and adjacent polygon blocks are subjected to position and size adjustment until edge areas of the adjacent polygon blocks are continuous without gaps.
Wherein, the three-dimensional modeling process of the step 1 comprises the following steps:
step 11: planning a modeling area, sorting coordinate information, planning a horizontal flight route of the unmanned aerial vehicle, designating flight height, and covering the modeling area by the flight area;
step 12: controlling an unmanned aerial vehicle to carry a five-mesh digital camera to fly according to the flying route in the step 11, and periodically and continuously shooting ground photos by using the five-mesh digital camera;
step 13: uniformly importing the ground photos shot in the step 12 into a three-dimensional model generation tool to generate a three-dimensional model of oblique photography;
step 14: and (4) checking the three-dimensional model of the oblique photography in the step (13), and performing complementary shooting on the live-action photos in the key area to perform fine correction on the material and texture of the model.
Wherein the three-dimensional model generation tool is a Smart3D software tool.
Wherein the modeling area comprises buildings, roads, green plants and fixed facilities.
Wherein, the ground photo is in a JPG format.
The source file of the three-dimensional model of oblique photography is in osgb format, and the target file is in obj format.
Wherein, the effect adjusting process of step 2 comprises the following steps:
step 21: selecting a Cecum digital earth engine supporting WebGL, and building a GIS geographic information system environment;
step 22: loading the satellite front-view image data to the GIS (geographic information system) in the step 21 to serve as a basic layer to form a digital earth scene;
step 23: converting the three-dimensional model data of oblique photography output in the step 14 into an Obj file format, and loading the three-dimensional model data into the digital earth scene in the step 22; adjusting the space coordinate parameters of the three-dimensional model until the space position of the three-dimensional model is matched with the satellite front-view image in the step 22 in a comparison manner; and adjusting the scale of the three-dimensional model to be scaled in an equal proportion until the size of the three-dimensional model is close to the real scene.
The GIS geographic information system adopts an open source Cesium platform.
Wherein, the video stream processing procedure of step 3 includes the following steps:
step 31: installing cameras, and configuring an RTSP main code stream address of each camera;
step 32: configuring the RTSP main code Stream address of each camera in the step 31 into an H5Stream service configuration file;
step 33: starting an H5Stream service to convert the RTSP video Stream into a video Stream supporting an Html5 protocol, generating an Html5 video Stream address of each camera, and simultaneously accessing a plurality of paths of video Stream preview pictures by using a browser.
Wherein, the effect adjusting process of the step 4 comprises the following steps:
step 41: configuring the Html5 video stream address of the video camera of step 33 to the video stream input end of the camera distortion correction tool in the OPENCV algorithm library;
step 42: placing the printed 8X 16 black-white checkerboard A3 picture into a video picture to be corrected, and calibrating intersection points of each unit cell of the checkerboard;
step 43: according to the cell intersection points calibrated in the step 42, through uniformly stretching the video monitoring picture, the calibration points corresponding to the cell intersection points are restored to be a matrix distributed at equal intervals, and correction of barrel distortion of the video stream is realized;
step 44: and a rectangular tool clipping step 43 is used for clipping the distorted and corrected video monitoring picture to generate a video code stream output address of the effective view.
Wherein, the video projection process of step 5 comprises the following steps:
step 51: creating a polygonal plane in the digital earth scene loaded with the three-dimensional model in step 23, and adjusting the size and position of the polygonal plane with reference to the video monitoring picture of the effective field in step 44; adjusting the height of the polygon plane to match the spatial position of the three-dimensional model of the corresponding region;
step 52: setting the material parameters of the polygonal plane, and loading the video monitoring picture of the effective visual field in the step 44 as the skin material of the polygonal plane;
step 53: and adjusting the positions and the sizes of the adjacent polygonal blocks until the edge areas of the adjacent polygonal blocks are continuous without gaps, and splicing the polygons to form a continuous large-scene video monitoring picture.
In summary, the present invention provides a video fusion method, which is applied to users with high requirements for video monitoring security in military colleges, army bases, scientific research institutions, etc., and provides a design scheme for the requirements for video monitoring of security in large public areas, closed places, campus camps, etc. A GIS system based on a WebGL rendering engine is adopted as a bottom layer to construct a virtual-real combined video fusion monitoring system, and the system design method has high advancement and feasibility.
Example 1
The present embodiment includes:
(1) three-dimensional modeling
Planning the flight route and the flight height of the unmanned aerial vehicle, covering a modeling target area, controlling the unmanned aerial vehicle to fly along the planning route and shooting pictures by adopting an onboard five-eye camera according to the set shooting frequency. After the flight is finished, the photos are completely imported into SMART3D software for data processing and model generation. And (4) previewing the scene after the data generation is successful, performing on-site rephotography on the key area by using a digital camera, and performing material correction on the model by adopting 3 DMAX. And exporting to an obj format file.
(2) Effect adjustment
And loading the obj file of the three-dimensional model to a geographic information system, adjusting the parameter information of the three-dimensional model, including three-dimensional coordinate setting, position adjustment, scale adjustment, texture adjustment, model simplification and the like, wherein the model is attached to the position of the front view image layer of the satellite, and the height of the model is attached to the real terrain so as to build a real virtual scene.
(3) Video stream processing
The RTSP video stream is pulled from equipment such as a video monitoring camera and a video server, and video stream configuration information including an IP address, a login user password, a channel number and the like is recorded; and filling the video Stream information into a conf configuration file of the H5Stream service according to a specified format, starting the H5Stream service to access a video picture through a browser, converting the RTSP video Stream into a video Stream supporting the Html5 protocol, and recording the address of the converted H5 video Stream.
(4) Video deformity correction
Configuring an H5 video stream address into an OPENCV camera distortion correction program, adjusting a monitoring picture with wide-angle parameters and barrel distortion by using a checkerboard picture correction parameter method, cutting and intercepting a video monitoring effective visual field, and performing high-precision matching on the video picture and a space position in a virtual scene.
(5) Video projection
And constructing a polygonal plane as a video projection carrier on a corresponding coordinate point in a virtual scene by referring to the effective visual field of the video monitoring picture, and then loading a video code stream supporting an Html5 protocol as a polygonal material. When the video is projected, multiple paths of independent videos need to be spliced to form an integral scene, and splicing overlapping areas of adjacent polygonal blocks need to be flexibly configured and adjusted.
The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.

Claims (10)

1. A video fusion system, the video fusion system comprising: the device comprises a three-dimensional modeling module, an effect adjusting module, a video stream processing module, a video deformity correcting module and a video projection module;
wherein the content of the first and second substances,
the three-dimensional modeling module is used for controlling the unmanned aerial vehicle to carry digital photos of a modeling area of a five-mesh digital camera shooting target by adopting an unmanned aerial vehicle aerial oblique photography modeling method, and converting the digital photos into a three-dimensional model by using a three-dimensional model generation tool;
the effect adjusting module is used for loading the three-dimensional model to a geographic information system, adjusting the three-dimensional model, and building a virtual scene similar to reality, wherein the three-dimensional model comprises model simplification, three-dimensional coordinate setting, position adjustment and scale adjustment;
the video Stream processing module is used for pulling an RTSP video Stream from a video monitoring camera and a video server device, and converting the RTSP video Stream into a video Stream supporting an Html5 protocol through an H5Stream service;
the video deformity correction module is used for adjusting a video monitoring picture of a video code stream with wide-angle parameters and barrel distortion by adopting a camera distortion correction tool in an OPENCV algorithm library and applying a checkerboard picture correction parameter method, cutting and intercepting an effective view of the video monitoring picture, and matching the video monitoring picture with a spatial position in a virtual scene;
the video projection module is used for constructing a polygonal plane as a video projection carrier on a corresponding coordinate point in a virtual scene according to an effective visual field of a video monitoring picture, and then loading a video code stream supporting an Html5 protocol as a material of the polygonal plane; when the video is projected, multiple paths of independent videos need to be spliced to form an integral scene, and adjacent polygon blocks are subjected to position and size adjustment until edge areas of the adjacent polygon blocks are continuous without gaps.
2. The video fusion system of claim 1 wherein the three-dimensional modeling module comprises: the system comprises a flight path planning unit, an unmanned aerial vehicle control unit, a photo importing unit and a three-dimensional model correcting unit;
the flight path planning unit is used for planning a modeling area, collating coordinate information, planning a horizontal flight path of the unmanned aerial vehicle, designating a flight height and covering the modeling area by the flight area;
the unmanned aerial vehicle control unit is used for controlling an unmanned aerial vehicle to carry a five-mesh digital camera to fly according to the flight route planned by the flight route planning unit, and the five-mesh digital camera is used for periodically and continuously shooting ground photos;
the photo importing unit is used for uniformly importing ground photos shot by the unmanned aerial vehicle into a three-dimensional model generating tool to generate a three-dimensional model of oblique photography;
the three-dimensional model correction unit is used for checking the oblique photography three-dimensional model generated by the three-dimensional model generation tool and controlling the unmanned aerial vehicle to take the key area live-action photos to finely correct the material and texture of the model.
3. The video fusion system of claim 2 wherein the three-dimensional model generation tool is a Smart3D software tool.
4. The video fusion system of claim 2 wherein the modeled area includes buildings, roads, greenery, stationary facilities.
5. The video fusion system of claim 2 wherein the ground photographs taken are in JPG format.
6. The video fusion system of claim 2 wherein the effect adjustment module comprises: a tool selection unit, a data loading unit, a model loading and adjusting unit,
The tool selection unit is used for selecting a Cecum digital earth engine supporting WebGL and building a GIS geographic information system environment;
the data loading unit is used for loading the satellite forward-looking image data to the GIS in the step 21 to serve as a basic layer to form a digital earth scene;
the model loading and adjusting unit is used for converting the three-dimensional model data of the oblique photography output by the three-dimensional modeling module into an Obj file format and loading the data to the digital earth scene; the satellite orthographic image matching system is used for adjusting the space coordinate parameters of the three-dimensional model until the space position of the three-dimensional model is matched with the orthographic image of the satellite in a contrast mode; and adjusting the scale of the three-dimensional model to be scaled in an equal proportion until the size of the three-dimensional model is close to the real scene.
7. The video fusion system of claim 6 wherein the GIS geographic information system employs an open source cesum platform.
8. The video fusion system of claim 3 wherein the video stream processing module comprises: the device comprises a first address configuration unit and a video stream conversion unit;
the address configuration unit is used for configuring an RTSP main code stream address of each camera for the installed cameras; the system is used for configuring the RTSP main code Stream address of each camera into an H5Stream service configuration file;
the video Stream conversion unit is used for starting an H5Stream service to convert an RTSP video Stream into a video Stream supporting an Html5 protocol, generating an Html5 video Stream address of each camera, and simultaneously accessing a plurality of paths of video Stream preview pictures by using a browser.
9. The video fusion system of claim 8 wherein the effect adjustment module comprises: a second address configuration unit, a checkerboard picture setting and calibrating unit, a picture correcting unit and a picture cutting unit;
the second address configuration unit is used for configuring the Html5 video stream address of the video camera to the video stream input end of the camera distortion correction tool in the OPENCV algorithm library;
the checkerboard picture setting and calibrating unit is used for placing the printed 8 x 16 black and white checkerboard A3 picture into a video picture to be corrected and calibrating the intersection point of each checkerboard;
the picture correcting unit is used for restoring the calibration points corresponding to the intersection points of the unit grids into a matrix with equal interval distribution by uniformly stretching the video monitoring picture according to the calibrated intersection points of the unit grids, so as to realize the correction of the barrel-type distortion of the video stream;
the picture cutting unit is used for cutting the video monitoring picture after distortion correction by using a rectangular tool to generate a video code stream output address of an effective view field.
10. The video fusion system of claim 9, wherein the video projection module comprises: the system comprises a polygonal plane creating and adjusting unit, a skin material loading unit and a scene splicing unit;
the polygon plane creating and adjusting unit is used for creating a polygon plane in a digital earth scene loaded with a three-dimensional model, and adjusting the size and the position of the polygon plane by referring to a video monitoring picture of an effective view; adjusting the height of the polygon plane to match the spatial position of the three-dimensional model of the corresponding region;
the skin material loading unit is used for setting polygonal plane material parameters and loading the video monitoring picture of the effective vision field as the skin material of the polygonal plane;
the scene splicing unit is used for adjusting the position and the size of the adjacent polygonal blocks until the edge areas of the adjacent polygonal blocks are continuous without gaps, so that the polygons are spliced to form a continuous large-scene video monitoring picture.
CN202011478934.8A 2020-12-15 2020-12-15 Video fusion system Pending CN112584060A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011478934.8A CN112584060A (en) 2020-12-15 2020-12-15 Video fusion system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011478934.8A CN112584060A (en) 2020-12-15 2020-12-15 Video fusion system

Publications (1)

Publication Number Publication Date
CN112584060A true CN112584060A (en) 2021-03-30

Family

ID=75135069

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011478934.8A Pending CN112584060A (en) 2020-12-15 2020-12-15 Video fusion system

Country Status (1)

Country Link
CN (1) CN112584060A (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113538681A (en) * 2021-07-14 2021-10-22 云南电网有限责任公司曲靖供电局 Three-dimensional reconstruction method and device for dynamic real scene of transformer substation, electronic equipment and medium
CN113592326A (en) * 2021-08-05 2021-11-02 合肥永信科翔智能技术有限公司 Remote intelligent monitoring system suitable for wisdom mill
CN114140593A (en) * 2021-12-02 2022-03-04 北京清晨动力科技有限公司 Digital earth and panorama fusion display method and device
CN115546377A (en) * 2022-12-01 2022-12-30 杭州靖安科技有限公司 Video fusion method and device, electronic equipment and storage medium
CN117152400A (en) * 2023-10-30 2023-12-01 武汉苍穹融新科技有限公司 Method and system for fusing multiple paths of continuous videos and three-dimensional twin scenes on traffic road
CN117560578A (en) * 2024-01-12 2024-02-13 北京睿呈时代信息科技有限公司 Multi-channel video fusion method and system based on three-dimensional scene rendering and irrelevant to view points

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2001067749A2 (en) * 2000-03-07 2001-09-13 Sarnoff Corporation Camera pose estimation
US20030085992A1 (en) * 2000-03-07 2003-05-08 Sarnoff Corporation Method and apparatus for providing immersive surveillance
CN101951502A (en) * 2010-10-19 2011-01-19 北京硅盾安全技术有限公司 Three-dimensional intelligent video monitoring method
CN103226830A (en) * 2013-04-25 2013-07-31 北京大学 Automatic matching correction method of video texture projection in three-dimensional virtual-real fusion environment
CN103716586A (en) * 2013-12-12 2014-04-09 中国科学院深圳先进技术研究院 Monitoring video fusion system and monitoring video fusion method based on three-dimension space scene
US9240069B1 (en) * 2015-06-30 2016-01-19 Ariadne's Thread (Usa), Inc. Low-latency virtual reality display system
CN106373148A (en) * 2016-08-31 2017-02-01 中国科学院遥感与数字地球研究所 Equipment and method for realizing registration and fusion of multipath video images to three-dimensional digital earth system
CN107835436A (en) * 2017-09-25 2018-03-23 北京航空航天大学 A kind of real-time virtual reality fusion live broadcast system and method based on WebGL
US20200126299A1 (en) * 2018-10-18 2020-04-23 Trimble Inc. Virtual video projection system to synch animation sequences
CN111415416A (en) * 2020-03-31 2020-07-14 武汉大学 Method and system for fusing monitoring real-time video and scene three-dimensional model
CN112053446A (en) * 2020-07-11 2020-12-08 南京国图信息产业有限公司 Real-time monitoring video and three-dimensional scene fusion method based on three-dimensional GIS

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2001067749A2 (en) * 2000-03-07 2001-09-13 Sarnoff Corporation Camera pose estimation
US20030085992A1 (en) * 2000-03-07 2003-05-08 Sarnoff Corporation Method and apparatus for providing immersive surveillance
CN101951502A (en) * 2010-10-19 2011-01-19 北京硅盾安全技术有限公司 Three-dimensional intelligent video monitoring method
CN103226830A (en) * 2013-04-25 2013-07-31 北京大学 Automatic matching correction method of video texture projection in three-dimensional virtual-real fusion environment
CN103716586A (en) * 2013-12-12 2014-04-09 中国科学院深圳先进技术研究院 Monitoring video fusion system and monitoring video fusion method based on three-dimension space scene
US9240069B1 (en) * 2015-06-30 2016-01-19 Ariadne's Thread (Usa), Inc. Low-latency virtual reality display system
CN106373148A (en) * 2016-08-31 2017-02-01 中国科学院遥感与数字地球研究所 Equipment and method for realizing registration and fusion of multipath video images to three-dimensional digital earth system
CN107835436A (en) * 2017-09-25 2018-03-23 北京航空航天大学 A kind of real-time virtual reality fusion live broadcast system and method based on WebGL
US20200126299A1 (en) * 2018-10-18 2020-04-23 Trimble Inc. Virtual video projection system to synch animation sequences
CN111415416A (en) * 2020-03-31 2020-07-14 武汉大学 Method and system for fusing monitoring real-time video and scene three-dimensional model
CN112053446A (en) * 2020-07-11 2020-12-08 南京国图信息产业有限公司 Real-time monitoring video and three-dimensional scene fusion method based on three-dimensional GIS

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113538681A (en) * 2021-07-14 2021-10-22 云南电网有限责任公司曲靖供电局 Three-dimensional reconstruction method and device for dynamic real scene of transformer substation, electronic equipment and medium
CN113592326A (en) * 2021-08-05 2021-11-02 合肥永信科翔智能技术有限公司 Remote intelligent monitoring system suitable for wisdom mill
CN114140593A (en) * 2021-12-02 2022-03-04 北京清晨动力科技有限公司 Digital earth and panorama fusion display method and device
CN114140593B (en) * 2021-12-02 2022-06-14 北京清晨动力科技有限公司 Digital earth and panorama fusion display method and device
CN115546377A (en) * 2022-12-01 2022-12-30 杭州靖安科技有限公司 Video fusion method and device, electronic equipment and storage medium
CN117152400A (en) * 2023-10-30 2023-12-01 武汉苍穹融新科技有限公司 Method and system for fusing multiple paths of continuous videos and three-dimensional twin scenes on traffic road
CN117152400B (en) * 2023-10-30 2024-03-19 武汉苍穹融新科技有限公司 Method and system for fusing multiple paths of continuous videos and three-dimensional twin scenes on traffic road
CN117560578A (en) * 2024-01-12 2024-02-13 北京睿呈时代信息科技有限公司 Multi-channel video fusion method and system based on three-dimensional scene rendering and irrelevant to view points
CN117560578B (en) * 2024-01-12 2024-04-16 北京睿呈时代信息科技有限公司 Multi-channel video fusion method and system based on three-dimensional scene rendering and irrelevant to view points

Similar Documents

Publication Publication Date Title
CN112584120A (en) Video fusion method
CN112584060A (en) Video fusion system
CN109872401B (en) Unmanned aerial vehicle video augmented reality implementation method
US7961983B2 (en) Generating gigapixel images
US10748243B2 (en) Image distortion transformation method and apparatus
JP5054971B2 (en) Digital 3D / 360 degree camera system
US20050271299A1 (en) Image transforming method, image transforming device and multiprojection system
US8963943B2 (en) Three-dimensional urban modeling apparatus and method
CN107067447B (en) Integrated video monitoring method for large spatial region
US20060165310A1 (en) Method and apparatus for a virtual scene previewing system
JPH07319091A (en) Method and equipment for controlling distortion of projectedpicture
CN112437276A (en) WebGL-based three-dimensional video fusion method and system
KR20180130911A (en) Smart augmented reality service system and method based on virtual studio
CN105005960A (en) Method, apparatus and system for obtaining watermarking picture
CN108322625B (en) Panoramic video production method based on panoramic image
CN115641401A (en) Construction method and related device of three-dimensional live-action model
US11176716B2 (en) Multi-source image data synchronization
WO2023029551A1 (en) Image stitching method and system based on multiple unmanned aerial vehicles
KR101725024B1 (en) System for real time making of 360 degree VR video base on lookup table and Method for using the same
CN108108396B (en) Aerial photography picture splicing management system for aircraft
See et al. Virtual reality 360 interactive panorama reproduction obstacles and issues
CN114332383B (en) Scene three-dimensional modeling method and device based on panoramic video
KR101697163B1 (en) system and method for chroma-key composing for providing three-dimensional structure effect
TWI681365B (en) Building information model and digital terrain model integrating system and integrating method thereof
Whitley Unmanned aerial vehicles (UAVs) for documenting and interpreting historical archaeological Sites: Part II—return of the drones

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20210330

RJ01 Rejection of invention patent application after publication