CN111836012B - Video fusion and video linkage method based on three-dimensional scene and electronic equipment - Google Patents

Video fusion and video linkage method based on three-dimensional scene and electronic equipment Download PDF

Info

Publication number
CN111836012B
CN111836012B CN202010598667.1A CN202010598667A CN111836012B CN 111836012 B CN111836012 B CN 111836012B CN 202010598667 A CN202010598667 A CN 202010598667A CN 111836012 B CN111836012 B CN 111836012B
Authority
CN
China
Prior art keywords
camera
video
scene
target
dimensional
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010598667.1A
Other languages
Chinese (zh)
Other versions
CN111836012A (en
Inventor
王永威
刘金龙
牟风涛
张耀华
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Space Geodata Beijing Co ltd
Original Assignee
Space Geodata Beijing Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Space Geodata Beijing Co ltd filed Critical Space Geodata Beijing Co ltd
Priority to CN202010598667.1A priority Critical patent/CN111836012B/en
Publication of CN111836012A publication Critical patent/CN111836012A/en
Application granted granted Critical
Publication of CN111836012B publication Critical patent/CN111836012B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/40—Image enhancement or restoration using histogram techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/61—Control of cameras or camera modules based on recognised objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/67—Focus control based on electronic image sensor signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/695—Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/188—Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computational Linguistics (AREA)
  • Computing Systems (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Human Computer Interaction (AREA)
  • Closed-Circuit Television Systems (AREA)

Abstract

The invention relates to the technical field of comprehensive situation monitoring and commanding, in particular to a video fusion and video linkage method and electronic equipment based on a three-dimensional scene, which comprises the steps of three-dimensional modeling of a monitoring area, planning and deploying of the scene and a target camera, enhanced shake elimination and correction processing of the scene video, recognition and analysis of the scene video, scene video and thermodynamic diagram thereof, three-dimensional matching and fusion of a hot target, three-dimensional matching linkage of the target camera, monitoring and correction of position and attitude deviation of the scene camera and the like; the scene camera video image and the thermodynamic diagram thereof, the accurate space matching and fusion display of the hot target and the scene three-dimensional model are realized, the intelligent scheduling target camera obtains the fine image of the hot target, the accurate space matching and fusion display of other situation perception information, equipment facility information and task management and control information and the scene three-dimensional model are realized, the full-dimensional situation fusion monitoring based on the three-dimensional scene and the intelligent space positioning analysis and associated information fusion display based on the hot spot inspection, the plan task and the detection alarm are realized, and the intelligent level and the application efficiency of the comprehensive situation monitoring and commanding system are remarkably improved.

Description

Video fusion and video linkage method based on three-dimensional scene and electronic equipment
Technical Field
The invention relates to the technical field of comprehensive situation monitoring and commanding, in particular to a video fusion and video linkage method based on a three-dimensional scene and electronic equipment.
Background
The video monitoring is a main subsystem of a regional comprehensive situation monitoring and commanding system, cameras such as a fixed gun camera, a PTZ ball machine and the like are installed in a monitoring site in a type-selecting mode, a monitoring center adopts a display mode of one camera and one window, when a two-dimensional map or a three-dimensional model of a monitoring region is used for comprehensive display, icons are added at the installation positions of the cameras, the icons are manually clicked or the periphery is selected to call out related camera windows, and because video images are only schematically matched with the three-dimensional model, in the comprehensive situation monitoring and commanding application of scenes such as cities, parks, stations and the like, the high-efficiency positioning and scheduling of massive videos and related information are difficult, and the technical upgrading and the practical efficiency of the comprehensive situation monitoring and commanding system are restricted.
Disclosure of Invention
Aiming at the defects of the prior art, the invention discloses a video fusion and video linkage method and electronic equipment based on a three-dimensional scene, which are used for realizing the accurate spatial matching and fusion display of a scene camera video image and a thermodynamic diagram thereof, a hotspot target and a scene three-dimensional model, acquiring a fine image of the hotspot target by an intelligent scheduling target camera, realizing the accurate spatial matching and fusion display of other situation perception information, equipment facility information and task control information and the scene three-dimensional model, realizing the full-dimensional situation fusion monitoring based on the three-dimensional scene and the intelligent spatial positioning analysis and associated information fusion display based on hotspot patrol, planning tasks and detection alarm, and obviously improving the intelligent level and application efficiency of a comprehensive situation monitoring and commanding system.
The invention is realized by the following technical scheme:
in a first aspect, the invention discloses a video fusion and video linkage method based on a three-dimensional scene, which comprises the following steps:
s1, collecting and manufacturing a three-dimensional model of the monitoring area with the aim of establishing space matching fusion;
s2 planning and deploying scene cameras and target cameras with the purpose of obtaining global situation images;
s3, enhancing, eliminating shake and correcting the obtained scene video image;
s4, identifying, analyzing and processing the obtained scene video image to obtain a man-vehicle dynamic density thermodynamic diagram and a hot spot target in the monitored area;
s5 matching and fusing the scene video image, the thermodynamic diagram and the hot spot target thereof with the three-dimensional model;
s6, matching and linking the target camera with the three-dimensional model;
s7 monitors for a corrected scene camera pose offset.
Furthermore, in S1, a three-dimensional model with spatial accuracy, scale and texture meeting the demand of monitoring and commanding the comprehensive situation is manufactured by aerial photography, laser radar scanning, on-site photographing and architectural engineering design data.
Further, in S2, the principle of camera planning deployment is: generally, a high-definition gun camera is selected and installed at a high point position, and a monitoring area is statically covered to obtain the overall situation of the monitoring area; and a PTZ ball machine is selected as the target camera, and the monitoring area is dynamically covered to track and acquire a fine image of the hot target.
Further, in S3, the enhancing process includes:
s3a histogram equalizes the original image:
Figure GDA0003546802470000021
s3b histogram equalizes the target image:
Figure GDA0003546802470000022
s3c determining from rk→zmThe mapping relationship between: let sk＝vkObtaining:
Figure GDA0003546802470000023
will r iskAnd zmThe mapping table is formed by the corresponding relation of the original image and the pixels on the original image are sequentially updated according to the mapping table;
the method for eliminating the jitters comprises the following steps:
s3d determines the conversion relationship from the previous frame to the current frame, and is divided into three parameters: dx，dy，dα；
S3e, calculating the motion trail of the current frame through the transformation relation and the motion trail of the previous frame, wherein the solving relation is as follows:
xcur＝xpre+dx
ycur＝ypre+dy
αcur＝αpre+dα；
s3f, smoothing the motion trail of the current frame, wherein the smoothing method comprises the following steps:
Figure GDA0003546802470000031
s3g calculates a new frame-to-frame conversion relationship H', which is calculated by the following relationship:
dx'＝dx+x-xcur
dy'＝dy+y-ycur
dα'＝dα+α-αcur
the transformation matrix H' is obtained as:
Figure GDA0003546802470000032
and S3h, transforming the video by using the transformation matrix to obtain the de-jittered video.
In S3d, the conversion relationship from the previous frame to the current frame is determined and divided into three parameters dx，dy，dαMethod (2)Comprises the following steps:
s3d1 extracting feature point sets of a previous frame and a current frame;
s3d2 calculates a transformation matrix H between two frames from the change of the feature point set between the two frames:
Figure GDA0003546802470000033
s3d3 extracting transformation parameter d between two frames from transformation matrixx，dy，dα；
Wherein:
dx＝b1
dy＝b2
Figure GDA0003546802470000034
the correction treatment comprises the following steps:
s3i, detecting contour points in the image by using an edge detection algorithm;
s3j, detecting a distortion straight line in the image by a straight line detection method, and initially estimating and initializing a distortion model;
s3k iterative estimation of a parameter distortion model and a distortion center through the energy optimization process according to the distance between the correction point and the relevant straight line;
s3l performs distortion correction on the image using the calculated distortion model.
In S3j, detecting a distortion line in the image by a line detection method, and initially estimating and initializing the distortion model includes:
s3j1, because the contour which is supposed to be a straight line in the original image is changed into a discontinuous curve after being distorted, a new dimension is needed to be added to describe the discontinuous curve, whether the discontinuous curve belongs to the same straight line or not is determined by voting through an approximate approximation method, and the straight line is subjected to fitting correction;
s3j2 estimating an initialization distortion model through the corrected straight line group, selecting N longest straight lines, calculating the reliability of the straight lines, selecting the most reliable straight line parameters as the initialization model, and adopting the formula as follows:
Figure GDA0003546802470000041
Figure GDA0003546802470000042
the reliability of the ith line compared to the jth line.
In S3k, iteratively estimating the parameter distortion model and the distortion center through the energy optimization process according to the distance between the correction point and the relevant straight line includes:
s3k1, adding a constraint parameter item to the initial distortion model, and further constraining the distortion model;
and S3k2, optimizing the optimal parameters after iterative estimation constraint through an energy formula, and obtaining the optimal parameters through loop traversal of the pixels on the straight line and continuous iterative calculation. The energy formula is as follows:
Figure GDA0003546802470000043
nl denotes the number of lines, N (j) is the number of points on line j,
Figure GDA0003546802470000044
points corrected using the distortion model d are shown.
Furthermore, in S5, the spatial position of the camera in the three-dimensional model is determined through parameter analysis of the scene camera, the view cone of the projector corresponding to the camera is constructed, matching, positioning and fusion display of the scene video image and the thermodynamic diagram thereof and the hot spot target are performed, and the conflict and redundant parts in the video image are cut to improve the three-dimensional fusion visual effect.
The scene camera parameter analysis comprises the following steps:
s5a, extracting and matching feature points of the corrected image of the scene camera and the three-dimensional model of the corresponding area, selecting a plurality of feature point pairs and calculating pixel coordinates and three-dimensional coordinates of the feature point pairs;
s5b calculates camera parameters according to the correspondence between the pixel coordinates and the three-dimensional coordinates of the matching points, and the principle is as follows:
the corresponding relation between the pixel coordinate and the three-dimensional coordinate is as follows:
Figure GDA0003546802470000051
wherein R is a rotation matrix, T is a translation matrix, and the middle part on the right of the equal sign is a camera internal reference matrix; substituting the corresponding points selected in the step S5a into the formula, solving by using a least square method and repeatedly iterating to obtain an optimal solution, so as to obtain the positions and postures of the internal and external parameters of the camera;
in S5c, the view angle of the camera is calculated from the camera reference obtained in S5b, and the formula is:
FOVx＝2×tan(α)
FOVy＝2×tan(β)
wherein:
Figure GDA0003546802470000052
the matching and fusing of the scene video image and the three-dimensional model comprises the following steps:
s5d determining the position of the camera in the three-dimensional model space and constructing a visual cone of the projector corresponding to the camera according to the camera parameters calculated in S5b and S5 c;
s5e, carrying out frame processing on the real-time scene video stream, rendering the real-time scene video stream into dynamic video texture, and projecting and mapping the dynamic video texture to a three-dimensional model through a texture mapping technology;
s5f, fusing static textures and dynamic video textures on the surface of the three-dimensional model, and enabling the dynamic video textures to be covered as the uppermost layer of textures, so that the three-dimensional model is positioned and real-time scene video images are displayed;
s5g, according to the monitoring requirement, cutting the non-hot spot area and the part which affects the fusion visual effect in the scene video image, and cutting the overlapping area of the video images of the adjacent scene cameras, so that the multi-channel video images are seamlessly connected;
and S5h, obtaining a three-dimensional model static image under the view angle of the projector according to the parameters of the projector obtained in the step S5d by using a reverse imaging principle, and storing the three-dimensional model static image, the feature point pairs selected in the step S5a, the coordinates of the feature point pairs and the internal and external parameters calculated in the step S5b together for monitoring and correcting the state of the scene camera.
The matching and fusing of the density thermodynamic diagram and the three-dimensional model comprises the following steps:
s5i, rendering the real-time density thermodynamic diagrams analyzed by the video of each scene camera into dynamic thermodynamic textures, and realizing three-dimensional model positioning and real-time density thermodynamic diagram display according to S5e and S5 f;
s5j, cutting non-hot-spot areas and parts influencing the fusion visual effect in the density thermodynamic diagram according to the monitoring requirement; carrying out color processing on the overlapping areas of the adjacent thermodynamic diagrams by adopting a weighted average algorithm so as to optimize the fusion visual effect;
and S5k human-vehicle density thermodynamic diagrams can be further converted into human-vehicle three-dimensional models with corresponding density, direction and speed for fusion display.
The hot spot target and the three-dimensional model are matched and fused, and the hot spot target and the three-dimensional model are matched and fused by the following steps:
s5l extracting the pixel coordinates of the hot spot target central point in real time;
s5m, solving the three-dimensional coordinates of the hotspot target according to the camera parameters obtained in S5 b;
s5n, positioning the target label in real time on the three-dimensional model according to the three-dimensional coordinates of the hot target.
Further, in S6, based on the three-dimensional scene, the visual field of each target camera is calculated and the relay tracking interface of the adjacent overlapping visual fields is determined, and according to the position coordinates of the hot spot target, the designated target camera is scheduled and the cloud mirror is controlled by calculation, so as to obtain the fine image of the hot spot target, and the dynamic target can be continuously tracked in a relay manner.
The step of calculating the visual field of the target camera comprises the following steps:
s6a fixing the position and posture of the target camera, and solving the position of each target camera in the three-dimensional model according to S5 a-S5 d;
s6b calculating the visible range of each target camera within 360 degrees horizontally according to the camera position obtained in the step S6a and the given visible radius and the vertical visual angle range;
and S6c, optimizing the relay tracking interfaces of adjacent overlapped visible areas according to the monitoring requirements, and determining and storing the monitoring area of each target camera.
The step of resolving control of the target camera cloud mirror comprises the following steps:
s6d, when the position point of the hot spot target in the target camera monitoring area is given, firstly, the attitude of the control holder is calculated, and the rotation angles in the horizontal direction and the vertical direction are calculated according to the position of the camera and the target position;
and (3) determining the horizontal rotation direction and angle of the camera by using trigonometric function calculation, wherein the formula is as follows:
Figure GDA0003546802470000061
wherein d1 is calculated from the latitude of two location points and d2 is calculated from the longitude of two location points;
and (3) determining the vertical rotation direction and angle of the camera by using trigonometric function calculation, wherein the formula is as follows:
Figure GDA0003546802470000071
wherein the height difference d3 is calculated according to the elevations of the two position points, and the horizontal distance d4 is calculated according to the longitude and latitude of the two position points;
s6e, resolving the controlled holder attitude according to S6d, resolving the zoom factor of the focal length of the control lens, and calculating according to two strategies of a fixed space scale and a fixed target ratio;
wherein the zoom factor of the lens focal length of the fixed spatial scale mode is calculated as follows:
setting the width of a camera monitoring space as W, the installation height of the camera as H, the current field angle as theta, and the pitch angle after the vertical rotation control of the pan-tilt head as theta
Figure GDA0003546802470000076
After rotationThe field angle ω is:
Figure GDA0003546802470000072
the calculation formula of the focal length scaling factor after the pan-tilt rotates is as follows:
Figure GDA0003546802470000073
will ZnewThe focal distance of the camera is calculated and controlled in real time according to the pitch angle, and the set monitoring space scale can be kept;
and calculating a lens focal length scaling factor of the fixed target ratio mode:
in the 16:9 frame mode, the height pixel of the video image is HimgWidth pixel is WimgSetting the frame ratio of the target as P, identifying the height pixel as H, the width pixel as W, the current focal length scaling factor as Z,
when the 16H is larger than or equal to 9W, the calculation formula of the focal length scaling factor after the pan-tilt rotates is as follows:
Figure GDA0003546802470000074
when 16H <9W, the calculation formula of the focal length scaling factor after the pan-tilt rotation is as follows:
Figure GDA0003546802470000075
will ZnewAnd calculating and controlling the focal distance of the camera in real time according to the pitch angle, and keeping the set target frame ratio.
Further, in S7, the camera parameter change is monitored manually or periodically, the projection parameters are automatically corrected and updated or the alarm is given to prompt the field to adjust and maintain, and the scene video image and the thermodynamic diagram thereof, the three-dimensional fusion precision of the hot spot target, and the established monitoring coverage area are maintained, including the steps of:
s7a, automatically acquiring a frame of image of the video stream of the real-time scene camera manually or regularly at any time, and performing feature point extraction matching with the static image of the three-dimensional model under the visual angle of the camera projector stored in S5 h;
s7b, matching the feature point pairs extracted and matched in the step S7a with the feature point pairs stored in the step S5h, selecting a plurality of feature point pairs and analyzing pixel coordinates of the feature point pairs;
s7c, calculating the monitoring parameters of the camera by the method of the step S5b according to the characteristic points selected in the step S7 b;
s7d, comparing the camera monitoring parameters obtained in the step S7c with the storage parameters obtained in the step S5b, if the change is smaller than a set threshold value, updating the three-dimensional model positioning and displaying of the scene video images through the steps S5c to S5h according to the monitoring parameters, and updating and storing the selected feature point pairs and the coordinates thereof, the camera monitoring parameters and the three-dimensional model static images under the view angle of the projector; if the threshold value is larger than the set threshold value, an alarm is given to prompt field adjustment and maintenance.
Furthermore, the method is based on image depth processing of the regional monitoring video and accurate matching and fusion of the regional monitoring video and the three-dimensional model, and other situation perception information, equipment facility information and task management and control information are displayed in an accurate space matching and fusion mode.
In a second aspect, the present invention discloses an electronic device, which includes a processor and a memory storing execution instructions, wherein when the processor executes the execution instructions stored in the memory, the processor executes the method for video fusion and video linkage based on three-dimensional scene according to the first aspect.
The invention has the beneficial effects that:
according to the invention, based on the image depth processing of the regional monitoring video and the accurate matching and fusion with the three-dimensional model, the accurate spatial matching and fusion display of other situation perception information, equipment facility information and task management and control information are further realized, so that the full-dimensional situation fusion monitoring based on the three-dimensional scene, the intelligent spatial positioning analysis and the associated information fusion display based on the hot spot inspection, the planned task and the detection alarm are realized, and the intelligent level and the application efficiency of the comprehensive situation monitoring and commanding system are remarkably improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a schematic block diagram of a method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a camera deployment according to an embodiment of the invention;
FIG. 3 is a flow chart of video rectification according to an embodiment of the present invention;
FIG. 4 is a three-dimensional matching fusion effect diagram of a scene video according to an embodiment of the present invention;
FIG. 5 is a three-dimensional matching fusion effect diagram of scene video according to an embodiment of the present invention;
FIG. 6 is a three-dimensional matching fusion effect diagram of a people flow density thermodynamic diagram according to an embodiment of the invention;
FIG. 7 is a three-dimensional matching fusion effect diagram of a traffic density thermodynamic diagram according to an embodiment of the invention;
FIG. 8 is a three-dimensional matching fusion effect diagram of a human current density thermodynamic diagram target model according to an embodiment of the invention;
FIG. 9 is a diagram illustrating the fusion effect of three-dimensional matching of objects according to an embodiment of the present invention;
fig. 10 is a schematic view of calculation of the horizontal rotation angle of the target camera according to the embodiment of the present invention;
FIG. 11 is a schematic diagram of a target camera vertical pitch angle calculation according to an embodiment of the present invention;
FIG. 12 is a diagram illustrating a relay tracking of a target camera for a dynamic target according to an embodiment of the present invention;
FIG. 13 is an architectural diagram of a city/park full-dimensional situation fusion monitoring and commanding system according to an embodiment of the present invention;
FIG. 14 is a schematic diagram of an application interface of a city/park full-dimensional situation fusion monitoring and commanding system architecture according to an embodiment of the present invention;
FIG. 15 is a diagram of a station/venue full-dimensional situation fusion monitoring and commanding system architecture according to an embodiment of the present invention;
FIG. 16 is a schematic diagram of an application interface of a station/venue full-dimensional situation fusion monitoring and commanding system architecture according to an embodiment of the present invention;
FIG. 17 is a schematic illustration of a shortcut key for a three-dimensional model according to an embodiment of the present invention;
FIG. 18 is a navigation chart illustrating a three-dimensional model of a hot spot area according to an embodiment of the present invention;
FIG. 19 is a schematic view of a three-dimensional fusion window of a hot spot region according to an embodiment of the present invention;
FIG. 20 is a schematic view of a hotspot patrol pattern in accordance with an embodiment of the present invention;
FIG. 21 is a schematic view of a manual control mode according to an embodiment of the present invention;
FIG. 22 is a schematic diagram of the positioning and linkage of tasks according to an embodiment of the invention;
FIG. 23 is a schematic diagram of alarm positioning and linkage according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1
The embodiment discloses a video fusion and video linkage method based on a three-dimensional model, which comprises the steps of three-dimensional modeling of a monitoring area, planning and deployment of a scene and a target camera, enhanced shake elimination and correction processing of the scene video, identification and analysis of the scene video, the scene video and thermodynamic diagram thereof, three-dimensional matching fusion of a hot target, three-dimensional matching linkage of the target camera, monitoring and correction of position and attitude offset of the scene camera and the like, wherein the system architecture is shown in figure 1 and specifically comprises the following steps:
step 1, collecting and manufacturing a three-dimensional model of a monitoring area;
step 2, planning and deploying the scene camera and the target camera;
step 3, enhancing, eliminating and correcting the scene video image;
step 4, identifying and analyzing scene videos;
step 5, matching and fusing the scene video image, the thermodynamic diagram of the scene video image, the hot spot target and the three-dimensional model;
step 6, matching and linking the target camera and the three-dimensional model;
and 7, monitoring and correcting the position and posture offset of the scene camera.
The monitoring area three-dimensional model in the step 1 is collected and manufactured, a three-dimensional model with space precision, dimension and texture meeting the requirement of comprehensive situation monitoring and commanding can be processed and manufactured through aerial photography, laser radar scanning, field photographing, building engineering design data and the like, and space matching and fusion display of monitoring videos and other situation information are supported.
The principle of the camera planning deployment in the step 2 is as follows: generally, a high-definition gun camera is selected and installed at a high point position, and a monitoring area is statically covered to obtain the overall situation of the monitoring area; and a PTZ ball machine is selected as the target camera, and the monitoring area is dynamically covered to track and acquire a fine image of the hot target. The specific deployment scenario is shown in fig. 2.
And 3, enhancing, eliminating and correcting the scene video images to realize the uniform and beautiful color and accurate and stable image of the three-dimensional fusion display of the multi-path scene video.
The enhancing treatment comprises the following steps:
step 3 a: histogram equalization is carried out on the original image:
Figure GDA0003546802470000111
and step 3 b: histogram equalization is carried out on the target image:
Figure GDA0003546802470000112
and step 3 c: determine from rk→zmThe mapping relationship between: let sk＝vkObtaining:
Figure GDA0003546802470000113
will r iskAnd zmThe mapping table is formed by the corresponding relations, and the pixels on the original image are sequentially updated according to the mapping table.
The method for eliminating the jitters comprises the following steps:
and step 3 d: determining the conversion relation from the previous frame to the current frame, and dividing the conversion relation into three parameters: dx，dy，dα；
Step 3 e: calculating the motion trail of the current frame through the transformation relation and the motion trail of the previous frame, wherein the solving relation is as follows:
xcur＝xpre+dx
ycur＝ypre+dy
αcur＝αpre+dα；
and step 3 f: smoothing the motion trail of the current frame, wherein the smoothing method comprises the following steps:
Figure GDA0003546802470000121
step 3 g: calculating a new frame-to-frame conversion relationship H':
by calculating the relationship:
dx'＝dx+x-xcur
dy'＝dy+y-ycur
dα'＝dα+α-αcur；
the transformation matrix H' is obtained as:
Figure GDA0003546802470000122
step 3 h: and transforming the video by using the transformation matrix to obtain the video after the jittering is eliminated.
The method for determining the conversion relationship from the previous frame to the current frame and dividing the conversion relationship into three parameters dx, dy and d α in the step 3d comprises the following steps:
step 3d 1: extracting feature point sets of a previous frame and a current frame;
step 3d 2: calculating a transformation matrix H between two frames through the change of the feature point set between the two frames:
Figure GDA0003546802470000123
step 3d 3: extracting transformation parameter d between two frames from transformation matrixx，dy，dα. Wherein:
dx＝b1
dy＝b2
Figure GDA0003546802470000124
the flow chart of the correction process is shown in fig. 3, and the steps are as follows:
step 3 i: detecting contour points in the image using an edge detection algorithm;
step 3 j: detecting a distortion straight line in the image by a straight line detection method, and initially estimating and initializing a distortion model;
step 3k, iteratively estimating a parameter distortion model and a distortion center through an energy optimization process according to the distance between the correction point and the relevant straight line;
step 3 l: and performing distortion correction on the image by using the calculated distortion model.
The step 3j of detecting a distorted straight line in the image by a straight line detection method, and the step of preliminarily estimating and initializing the distorted model comprises the following steps:
step 3j1, because in the original image, the contour which should be a straight line is changed into a discontinuous curve after being distorted, a new dimension is needed to be added to describe the discontinuous curve, whether the discontinuous curve belongs to the same straight line or not is determined by voting by an approximate approximation method, and the straight line is subjected to fitting correction;
step 3j2, estimating an initialization distortion model through the corrected straight line group, selecting N longest straight lines, calculating the reliability of the straight lines, selecting the most reliable straight line parameters as the initialization model, and adopting the formula as follows:
Figure GDA0003546802470000131
Figure GDA0003546802470000132
the reliability of the ith line compared to the jth line.
The step of iteratively estimating the parameter distortion model and the distortion center through the distance between the correction point and the relevant straight line in the step 3k through an energy optimization process comprises the following steps:
step 3k 1: adding a constraint parameter item to the initial distortion model, and further constraining the distortion model;
step 3k 2: and optimizing the optimal parameters after iterative estimation constraint through an energy formula, and obtaining the optimal parameters through loop traversal of the pixels on the straight line and continuous iterative calculation. The energy formula is as follows:
Figure GDA0003546802470000133
nl denotes the number of lines, N (j) is the number of points on line j,
Figure GDA0003546802470000134
points corrected using the distortion model d are shown.
And 4, scene video identification and analysis, which is mainly used for acquiring dynamic density thermodynamic diagrams such as people and vehicles in a monitoring area, hot spot targets and the like required by situation monitoring so as to supplement and optimize situation information and visual effects displayed by three-dimensional fusion of video images of a specific scene.
And 5, matching and fusing the scene video image, the thermodynamic diagram of the scene video image, the hot spot target and the three-dimensional model, determining the spatial position of the camera in the three-dimensional model and constructing a view cone of the projector corresponding to the camera through parameter analysis of the scene camera, performing matching positioning and fusion display on the scene video image, the thermodynamic diagram of the scene video image and the hot spot target, and cutting conflict and redundant parts in the video image to improve the three-dimensional fusion visual effect.
The method for analyzing the scene camera parameters comprises the following steps:
step 5 a: extracting and matching feature points of a corrected image of a scene camera with a three-dimensional model of a corresponding area, selecting a plurality of feature point pairs and calculating pixel coordinates and three-dimensional coordinates of the feature point pairs;
and step 5 b: calculating camera parameters according to the corresponding relation between the pixel coordinates and the three-dimensional coordinates of the matching points, wherein the principle is as follows:
the corresponding relation between the pixel coordinate and the three-dimensional coordinate is as follows:
Figure GDA0003546802470000141
wherein R is a rotation matrix, T is a translation matrix, and the middle part on the right of the equal sign is a camera internal reference matrix. Substituting the corresponding points selected in the step 5a into the formula, solving by using a least square method and repeatedly iterating to obtain an optimal solution, so as to obtain the positions and postures of the internal and external parameters of the camera;
and step 5 c: calculating the view angle of the camera by the internal reference of the camera obtained in the step 5b, wherein the formula is as follows:
FOVx＝2×tan(α)
FOVy＝2×tan(β)
wherein:
Figure GDA0003546802470000142
the matching and fusing of the scene video image and the three-dimensional model comprises the following steps:
and step 5 d: determining the position of the camera in the three-dimensional model space and constructing a visual cone of the projector corresponding to the camera according to the camera parameters calculated in the steps 5b and 5 c;
step 5e, performing frame processing on the real-time scene video stream, rendering the real-time scene video stream into dynamic video textures, and projecting and mapping the dynamic video textures to the three-dimensional model through a texture mapping technology;
step 5f, fusing the static texture and the dynamic video texture on the surface of the three-dimensional model, and enabling the dynamic video texture to cover the uppermost texture, so as to realize the positioning of the three-dimensional model and the display of a real-time scene video image;
step 5 g: according to the monitoring requirements, cutting non-hot spot areas and parts influencing the fusion visual effect in the scene video images, such as trees, buildings and the like around dynamic monitoring areas such as roads, squares and the like; and cutting the overlapped area of the video images of the cameras in the adjacent scenes so as to enable the multi-channel video images to be connected seamlessly. The specific effect is shown in fig. 4 and 5.
Step 5 h: and (4) obtaining a three-dimensional model static image under the visual angle of the projector according to the parameters of the projector obtained in the step (5 d) by a reverse imaging principle, storing the static image, the feature point pairs and the coordinates thereof selected in the step (5 a) and the internal and external parameters calculated in the step (5 b) together, and monitoring and correcting the state of the scene camera.
The matching and fusing of the density thermodynamic diagram and the three-dimensional model comprises the following steps:
step 5 i: rendering the real-time density thermodynamic diagrams analyzed by the video of each scene camera into dynamic thermodynamic textures, and realizing three-dimensional model positioning and real-time density thermodynamic diagram display according to the steps 5e and 5 f;
step 5 j: according to the monitoring requirement, cutting non-hot spot areas and parts influencing the fusion visual effect in the density thermodynamic diagram; and carrying out color processing on the overlapped areas of the adjacent thermodynamic diagrams by adopting a weighted average algorithm so as to optimize the fusion visual effect. The specific effects are shown in fig. 6 and 7.
Step 5 k: the man-vehicle equal-density thermodynamic diagram can be converted into a man-vehicle three-dimensional model with corresponding density, direction and speed for fusion display. The specific effect is shown in figure 8.
The hot spot target and the three-dimensional model are matched and fused in the following steps:
step 5 l: extracting the pixel coordinates of the central point of the hot spot target in real time;
step 5 m: resolving the three-dimensional coordinates of the hot spot target according to the camera parameters obtained in the step 5 b;
and 5 n: and positioning the target label in real time on the three-dimensional model according to the three-dimensional coordinates of the hot target. The specific effect is shown in figure 9.
And 6, matching and linking the target cameras and the three-dimensional model, calculating the visual field of each target camera and determining a relay tracking interface of adjacent overlapped visual fields through space matching of the target cameras and the three-dimensional model, scheduling the specified target cameras and resolving and controlling the cloud mirror according to the position coordinates of the hot spot target, and realizing automatic scheduling and acquisition of fine images of the manually specified position, the patrolling hot spot region and the task/alarm related region and automatic scheduling and tracking of the specified or identified alarm people, vehicles and other dynamic targets to acquire fine images.
The step of calculating the visual field of the target camera comprises the following steps:
step 6 a: fixing the target cameras, and solving the position of each target camera in the three-dimensional model according to the steps 5a to 5 d;
step 6 b: calculating the visible range of each target camera within 360 degrees horizontally according to the camera position obtained in the step 6a and the given visible radius and the vertical visual angle range;
step 6 c: and optimizing the relay tracking interfaces of adjacent overlapped visible areas according to the monitoring requirement, and determining and storing the monitoring area of each target camera.
The step of resolving control of the target camera cloud mirror comprises the following steps:
and 6 d: when the position point of a hot spot target in a given target camera monitoring area is determined, firstly, the attitude of a control holder is calculated, and the rotation angles in the horizontal direction and the vertical direction are calculated according to the position of the camera and the target position;
as shown in fig. 10, the trigonometric function is used to determine the horizontal rotation direction and angle of the camera, and the formula is as follows:
Figure GDA0003546802470000161
where d1 is calculated from the latitude of two location points and d2 is calculated from the longitude of two location points.
As shown in fig. 11, the camera vertical rotation direction and angle are determined by trigonometric function calculation, and the formula is as follows:
Figure GDA0003546802470000162
wherein the height difference d3 is calculated according to the elevations of the two position points, and the horizontal distance d4 is calculated according to the longitude and latitude of the two position points.
Step 6 e: resolving the controlled holder attitude according to the step 6d, resolving the zoom factor of the focal length of the control lens, and calculating according to two strategies of determining the space scale and the target ratio;
and (3) calculating a lens focal length scaling factor of the fixed space scale mode:
setting the width of a camera monitoring space as W, the installation height of the camera as H, the current field angle as theta, and the pitch angle after the vertical rotation control of the pan-tilt head as theta
Figure GDA0003546802470000171
The rotated field angle ω is:
Figure GDA0003546802470000172
the formula for calculating the focal length scaling factor after the pan-tilt rotates is as follows:
Figure GDA0003546802470000173
will ZnewThe focal distance of the camera is calculated and controlled in real time according to the pitch angle, and the set monitoring space scale can be kept.
And calculating a lens focal length scaling factor of the fixed target ratio mode:
in the 16:9 frame mode, the height pixel of the video image is HimgWidth pixel is WimgSetting the frame ratio of the targetP, the height pixel of the recognition target is H, the width pixel is W, the current focal length scaling factor is Z,
when the 16H is larger than or equal to 9W, the calculation formula of the focal length scaling factor after the pan-tilt rotates is as follows:
Figure GDA0003546802470000174
when 16H <9W, the calculation formula of the focal length scaling factor after the pan-tilt rotation is as follows:
Figure GDA0003546802470000175
will ZnewAnd calculating and controlling the focal distance of the camera in real time according to the pitch angle, and keeping the set target frame ratio.
Manually designating the position, patrolling the hot spot area and linking the target cameras in the task/alarm related area:
step 6 f: analyzing and acquiring three-dimensional coordinates of a manually-specified position, a patrol hotspot area and a task/alarm related area;
step 6 g: and 6 d-6 e, calculating the cloud mirror attitude of the control target camera to obtain a fine image of the area.
Target camera linkage of dynamic targets such as appointed or intelligent identification people and vehicles:
step 6 h: continuously analyzing and obtaining three-dimensional coordinates of appointed or intelligently identified dynamic targets such as people, vehicles and the like;
step 6 i: continuously calculating and controlling the cloud mirror attitude of the appointed initial target camera through the steps 6 d-6 e, and continuously acquiring a fine image of the target;
step 6 j: and when the dynamic target crosses the relay tracking interface of the adjacent target camera, selecting the relay target camera according to the three-dimensional coordinates of the dynamic target obtained in the step 6h, and continuously calculating and controlling the cloud mirror posture of the relay target camera through the steps 6d to 6e to continuously obtain the fine image of the target. The process is schematically shown in FIG. 12.
The position and posture deviation monitoring and correction of the scene camera in the step 7, the parameter change of the camera is monitored manually or periodically, the projection parameters are automatically corrected and updated or the scene is prompted to be adjusted and maintained in an alarm mode, the three-dimensional fusion precision of the scene video image and the set monitoring coverage area are maintained, and the steps are as follows:
step 7 a: automatically acquiring a frame of image of the video stream of the camera in the real-time scene at any time manually or periodically, and performing feature point extraction matching with the static image of the three-dimensional model under the visual angle of the projector of the camera stored in the step 5 h;
and 7 b: matching the feature point pairs extracted and matched in the step 7a with the plurality of feature point pairs stored in the step 5h, selecting the plurality of feature point pairs and analyzing pixel coordinates of the feature point pairs;
and 7 c: calculating the monitoring parameters of the camera by the method in the step 5b according to the characteristic point pairs selected in the step 7 b;
and 7 d: comparing the camera monitoring parameters obtained in the step 7c with the storage parameters obtained in the step 5b, if the change is smaller than a set threshold value, updating the three-dimensional model positioning and displaying of the scene video image through the steps 5 c-5 h according to the monitoring parameters, and updating and storing the selected feature point pairs and the coordinates thereof, the camera monitoring parameters and the three-dimensional model static image under the projector view angle; if the threshold value is larger than the set threshold value, an alarm is given to prompt field adjustment and maintenance.
Example 2
In the embodiment, the system is applied to a full-dimensional situation fusion monitoring command scheme of an open management and control environment such as a city and a garden, high-definition and array scene cameras, regional scene cameras and target cameras matched with streets and communities are deployed in high-rise buildings in key areas such as squares and main roads, information such as monitoring facilities for license plate recognition and face recognition and alarm receiving centers is fused, the system is structured as shown in fig. 13, and the application interface is applied as shown in fig. 14.
The full-dimensional situation fusion monitoring and commanding application interface comprises comprehensive business chart split screens, three-dimensional operation situation split screens and three-dimensional hotspot video split screens.
Example 3
In this embodiment, the system is applied to a full-dimensional situation fusion monitoring command scheme of closed management and control environments such as stations and venues, high-altitude high-definition and array scene cameras are deployed in high-rise buildings in key regions such as squares, the scene cameras and the target cameras in internal regions of the building are matched, information such as equipment facilities and operation management of a barrier gate, fire fighting, lighting and the like is fused, the system architecture is shown in fig. 15, and the application interface is shown in fig. 16.
The full-dimensional situation fusion monitoring and commanding application interface comprises comprehensive business chart split screens, three-dimensional operation situation split screens and three-dimensional hotspot video split screens.
And a shortcut key is configured on the three-dimensional model in the three-dimensional operation situation split screen, and the display of the universe/region and the display of the designated view angle are supported, as shown in fig. 17.
When the hot spot region is shown in an enlarged manner, a navigation map is automatically added to the lower right corner of the interface, the position and the view angle of the hot spot region in the whole domain are displayed, and the overall spatial orientation is kept, as shown in fig. 18.
And on the basis of the global three-dimensional model, integrating and displaying the whole operation situation information of the station, wherein the information comprises equipment facility state information, routine task state information, alarm handling state information, vehicle in-out state information, personnel in-out state information, staff on duty state information and the like.
The three-dimensional fusion window of the hot spot area takes a three-dimensional model as a background, and a scene video image occupies 2/3 frames so as to enhance the spatial scene and the orientation of the scene video image; the lower right corner automatically adds a navigation map to enhance the spatial orientation and perspective of the hotspot area, as in fig. 19.
And configuring a hot spot area of the target camera, wherein the hot spot video is in a picture-in-picture mode, and the three-dimensional fusion scene video and the fine video of the target camera are switched.
The hot video split screen is configured into a directory window and a main display window.
The control mode and the priority thereof can be configured, and the general manual control is more than alarm linkage more than task linkage more than hot spot patrol.
1. Hotspot patrol mode
The sequencing, switching and duration of the hot videos displayed in the directory window and the main display window in turn can be set, and the patrol monitoring of all hot areas is supported, as shown in fig. 20.
2. Manual control mode
Clicking any hot video of the directory window, and switching the hot video into the main display window.
Clicking any point position of the global three-dimensional model, switching the hot video of the region into a main display window, and automatically following the point position by the target camera, as shown in fig. 21.
The target camera can set an automatic tour mode, automatically surrounds 360 degrees, can pause zooming at any position and captures a fine video of the target.
3. Positioning and linking tasks
After the routine task is started, the task tag (yellow) of the relevant area and the starting state of the associated equipment facility are displayed in a three-dimensional situation split screen mode, the hotspot video of the relevant area is automatically displayed in a split screen mode, and the hotspot video is overlaid with task information (yellow) as shown in fig. 22.
When a plurality of tasks are parallel, the hot video is split and automatically displayed in a screen-sharing manner, and the hot video of the relevant area of each task is displayed in a circulating manner; when a certain task of the integrated chart is clicked and split-screen-displayed, the task label of the relevant area of the task and the starting state of the relevant equipment facility are split-screen-displayed in a three-dimensional situation, the hot video of the relevant area of the task is automatically displayed in a split-screen mode, and task information is overlapped in a flashing mode.
4. Alarm positioning and linkage
After the equipment facilities and the videos are detected and alarmed, the three-dimensional situation screen-division flickering positioning display device displays an information label (red) of the alarm, the hot video of the alarm is automatically displayed in a screen-division mode, and the hot video is overlaid with the alarm information (red), as shown in fig. 23.
When a plurality of alarms are parallel, hot video screens are split, and hot videos of all the alarms are automatically displayed in a turn mode; when a certain alarm is clicked, the comprehensive diagram is split and the screen is split, the three-dimensional situation split screen flickers an information label of the alarm, the hot video split screen automatically displays the hot video of the alarm, and the hot video flickers and overlaps the alarm information.
Example 4
The embodiment discloses an electronic device which comprises a processor and a memory, wherein the memory stores execution instructions, and when the processor executes the execution instructions stored in the memory, the processor executes a video fusion and video linkage method based on a three-dimensional scene.
In conclusion, the invention is based on the image depth processing of the regional monitoring video and the accurate matching and fusion of the regional monitoring video and the three-dimensional model, and further realizes the accurate spatial matching and fusion display of other situation perception information, equipment facility information and task control information, thereby realizing the full-dimensional situation fusion monitoring command based on the whole regional three-dimensional model and the local three-dimensional model positioning and associated information intelligent scheduling fusion display based on the plan task, the hot spot region and the detection alarm, and obviously improving the intelligent level and the application efficiency of the system.
The above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (9)

1. A video fusion and video linkage method based on a three-dimensional scene is characterized by comprising the following steps:
s1, collecting and manufacturing a three-dimensional model of the monitoring area with the aim of establishing space matching fusion;
s2 planning and deploying scene cameras and target cameras with the purpose of obtaining global situation images;
s3, enhancing, eliminating shake and correcting the obtained scene video image;
s4, identifying, analyzing and processing the obtained scene video image to obtain a man-vehicle dynamic density thermodynamic diagram and a hot spot target in the monitored area;
s5 matching and fusing the scene video image, the thermodynamic diagram and the hot spot target thereof with the three-dimensional model;
s6, matching and linking the target camera with the three-dimensional model;
s7, monitoring and correcting the pose offset of the scene camera;
in the step S5, matching and fusing the scene video image, the thermodynamic diagram thereof, the hot spot target and the three-dimensional model, determining the spatial position of the camera in the three-dimensional model and constructing a view cone of the projector corresponding to the camera through parameter analysis of the scene camera, performing matching, positioning and fusion display of the scene video image, the thermodynamic diagram thereof and the hot spot target, and cutting out a conflict part and a redundant part in the video image to improve the three-dimensional fusion visual effect;
the scene camera parameter analysis comprises the following steps:
s5a, extracting and matching feature points of the corrected image of the scene camera and the three-dimensional model of the corresponding area, selecting a plurality of feature point pairs and calculating pixel coordinates and three-dimensional coordinates of the feature point pairs;
s5b calculates camera parameters according to the correspondence between the pixel coordinates and the three-dimensional coordinates of the matching points, and the principle is as follows:
the corresponding relation between the pixel coordinate and the three-dimensional coordinate is as follows:
Figure FDA0003546802460000011
wherein R is a rotation matrix, T is a translation matrix, and the middle part on the right of the equal sign is a camera internal reference matrix; substituting the corresponding points selected in the step S5a into the formula, solving by using a least square method and repeatedly iterating to obtain an optimal solution, so as to obtain the positions and postures of the internal and external parameters of the camera;
in S5c, the view angle of the camera is calculated from the camera reference obtained in S5b, and the formula is:
FOVx＝2×tan(α)
FOVy＝2×tan(β)
wherein:
Figure FDA0003546802460000021
the matching and fusing of the scene video image and the three-dimensional model comprises the following steps:
s5d determining the position of the camera in the three-dimensional model space and constructing a visual cone of the projector corresponding to the camera according to the camera parameters calculated in S5b and S5 c;
s5e, carrying out frame processing on the real-time scene video stream, rendering the real-time scene video stream into dynamic video texture, and projecting and mapping the dynamic video texture to a three-dimensional model through a texture mapping technology;
s5f, fusing static textures and dynamic video textures on the surface of the three-dimensional model, and enabling the dynamic video textures to be covered as the uppermost layer of textures, so that the three-dimensional model is positioned and real-time scene video images are displayed;
s5g, according to the monitoring requirement, cutting the non-hot spot area and the part which affects the fusion visual effect in the scene video image, and cutting the overlapping area of the video images of the adjacent scene cameras, so that the multi-path video images are seamlessly connected;
s5h, obtaining a three-dimensional model static image under the view angle of the projector through a reverse imaging principle according to the projector parameters obtained in the step S5d, storing the three-dimensional model static image, the feature point pairs and the coordinates thereof selected in the step S5a and the internal and external parameters calculated in the step S5b, and monitoring and correcting the state of the scene camera;
the matching and fusing of the density thermodynamic diagram and the three-dimensional model comprises the following steps:
s5i, rendering the real-time density thermodynamic diagrams analyzed by the video of each scene camera into dynamic thermodynamic textures, and realizing three-dimensional model positioning and real-time density thermodynamic diagram display according to S5e and S5 f;
s5j, cutting non-hot-spot areas and parts influencing the fusion visual effect in the density thermodynamic diagram according to the monitoring requirement; carrying out color processing on the overlapping areas of the adjacent thermodynamic diagrams by adopting a weighted average algorithm so as to optimize the fusion visual effect;
s5k the man-vehicle density thermodynamic diagram can be converted into a man-vehicle three-dimensional model with corresponding density, direction and speed for fusion display;
the hot spot target and the three-dimensional model are matched and fused, and the hot spot target and the three-dimensional model are matched and fused by the following steps:
s5l, extracting the pixel coordinates of the hot spot target central point in real time;
s5m, solving the three-dimensional coordinates of the hotspot target according to the camera parameters acquired in the step S5 b;
s5n, positioning the target label in real time on the three-dimensional model according to the three-dimensional coordinates of the hot target.
2. The method for fusing and linking video based on three-dimensional scene according to claim 1, wherein in step S3, the enhancement processing comprises:
s3a histogram equalizes the original image:
Figure FDA0003546802460000031
s3b histogram equalizes the target image:
Figure FDA0003546802460000032
s3c determining from rk→zmThe mapping relationship between: let sk＝vkObtaining:
Figure FDA0003546802460000033
will r iskAnd zmThe mapping table is formed by the corresponding relation of the original image and the pixels on the original image are sequentially updated according to the mapping table;
the method for eliminating the jitters comprises the following steps:
s3d determines the conversion relationship from the previous frame to the current frame, and is divided into three parameters: dx，dy，dα；
S3e, calculating the motion trail of the current frame through the transformation relation and the motion trail of the previous frame, wherein the solving relation is as follows:
xcur＝xpre+dx
ycur＝ypre+dy
αcur＝αpre+dα；
s3f, smoothing the motion trail of the current frame, wherein the smoothing method comprises the following steps:
Figure FDA0003546802460000041
s3g calculates a new frame-to-frame conversion relationship H', which is calculated by the following relationship:
dx'＝dx+x-xcur
dy'＝dy+y-ycur
dα'＝dα+α-αcur；
the transformation matrix H' is obtained as:
Figure FDA0003546802460000042
s3h, transforming the video by using the transformation matrix to obtain the video after shaking elimination;
the correction treatment comprises the following steps:
s3i, detecting contour points in the image by using an edge detection algorithm;
s3j, detecting a distortion straight line in the image by a straight line detection method, and initially estimating and initializing a distortion model;
s3k iterative estimation of a parameter distortion model and a distortion center through the energy optimization process according to the distance between the correction point and the relevant straight line;
s3l performs distortion correction on the image using the calculated distortion model.
3. The method as claimed in claim 2, wherein in step S3d, the conversion relationship from the previous frame to the current frame is determined and divided into three parameters dx，dy，dαThe method comprises the following steps:
s3d1 extracting feature point sets of a previous frame and a current frame;
s3d2 calculates a transformation matrix H between two frames from the change of the feature point set between the two frames:
Figure FDA0003546802460000051
s3d3 extracting transformation parameter d between two frames from transformation matrixx，dy，dαWherein:
dx＝b1
dy＝b2
Figure FDA0003546802460000052
4. the method for video fusion and video linkage based on three-dimensional scene according to claim 2, wherein in S3j, the distorted straight line in the image is detected by a straight line detection method, and the step of initially estimating and initializing the distorted model comprises:
s3j1, voting by an approximate approximation method to determine whether discontinuous curves belong to the same straight line, and performing fitting correction on the straight line;
s3j2 estimates an initialization distortion model through the corrected straight line group, selects N longest straight lines, calculates the reliability of the straight lines, selects the most reliable straight line parameters as the initialization model, and has the formula as follows:
Figure FDA0003546802460000053
Figure FDA0003546802460000054
the reliability of the ith line compared to the jth line.
5. The method for video fusion and video linkage based on three-dimensional scene according to claim 2, wherein in the step S3k, the step of iteratively estimating the parameter distortion model and the distortion center through the energy optimization process according to the distance between the correction point and the relevant straight line comprises:
s3k1, adding a constraint parameter item to the initial distortion model, and further constraining the distortion model;
s3k2, optimizing the optimal parameters after iterative estimation constraint through an energy formula, and obtaining the optimal parameters through loop traversal of the pixels on the straight line and continuous iterative calculation, wherein the energy formula is as follows:
Figure FDA0003546802460000061
nl denotes the number of lines, N (j) is the number of points on line j,
Figure FDA0003546802460000062
points corrected using the distortion model d are shown.
6. The video fusion and video linkage method based on three-dimensional scene according to claim 1, wherein in S6, the target cameras are linked with the three-dimensional model in a matching manner, the designated target cameras are scheduled and the cloud mirrors are controlled by resolving according to the position coordinates of the hot target by calculating the visual field of each target camera based on the three-dimensional scene and determining the relay tracking interface of the adjacent overlapped visual fields, so as to obtain the fine image of the hot target, and the dynamic target can be continuously tracked in a relay manner;
the step of calculating the visual field of the target camera comprises the following steps:
s6a fixing the position and posture of the target camera, and solving the position of each target camera in the three-dimensional model according to the steps S5 a-S5 d;
s6b calculating the visible range of each target camera within 360 degrees horizontally according to the camera position obtained in the step S6a and the given visible radius and the vertical visual angle range;
s6c, optimizing the relay tracking interface of adjacent overlapped visible areas according to the monitoring requirement, and determining and storing the monitoring area of each target camera;
the step of resolving control of the target camera cloud mirror comprises the following steps:
s6d, when the position point of the hot spot target in the target camera monitoring area is given, firstly, the attitude of the control holder is calculated, and the rotation angles in the horizontal direction and the vertical direction are calculated according to the position of the camera and the target position;
and (3) determining the horizontal rotation direction and angle of the camera by using trigonometric function calculation, wherein the formula is as follows:
Figure FDA0003546802460000071
wherein d1 is calculated from the latitude of two location points and d2 is calculated from the longitude of two location points;
and (3) determining the vertical rotation direction and angle of the camera by using trigonometric function calculation, wherein the formula is as follows:
Figure FDA0003546802460000072
wherein the height difference d3 is calculated according to the elevations of the two position points, and the horizontal distance d4 is calculated according to the longitude and latitude of the two position points;
s6e, resolving the posture of the controlled holder according to the step S6d, resolving the zoom factor of the focal length of the controlled lens, and calculating according to two strategies of fixed space scale and fixed target ratio;
wherein the zoom factor of the lens focal length of the fixed spatial scale mode is calculated as follows:
setting the width of a camera monitoring space as W, the installation height of the camera as H, the current angle of view as theta, the pitch angle after the vertical rotation control of the holder as phi, and the angle of view omega after the rotation as:
Figure FDA0003546802460000073
the calculation formula of the focal length scaling factor after the pan-tilt rotates is as follows:
Figure FDA0003546802460000074
will ZnewThe focal distance of the camera is calculated and controlled in real time according to the pitch angle, and the set monitoring space scale can be kept;
and calculating a lens focal length scaling factor of the fixed target ratio mode:
in 16:9 frame mode, video imagesIs HimgWidth pixel is WimgSetting the frame ratio of the target as P, identifying the height pixel of the target as H, the width pixel as W and the current focal length scaling factor as Z;
when the 16H is larger than or equal to 9W, the calculation formula of the focal length scaling factor after the pan-tilt rotates is as follows:
Figure FDA0003546802460000081
when 16H <9W, the calculation formula of the focal length scaling factor after the pan-tilt rotation is as follows:
Figure FDA0003546802460000082
will ZnewAnd calculating and controlling the focal distance of the camera in real time according to the pitch angle, and keeping the set target frame ratio.
7. The method as claimed in claim 1, wherein in step S7, the scene camera is monitored and corrected for attitude and attitude offset, and the camera parameters are monitored manually or periodically to automatically correct and update projection parameters or to alert and prompt field adjustment and maintenance, so as to maintain the scene video image and thermodynamic diagram thereof, the three-dimensional fusion accuracy of the hot spot target, and the established monitoring coverage area, the steps are as follows:
s7a, automatically acquiring a frame of image of the video stream of the real-time scene camera manually or periodically at any time, and extracting and matching feature points with the static image of the three-dimensional model under the visual angle of the camera projector, which is stored in the step S5 h;
s7b, matching the feature point pairs extracted and matched in the step S7a with the feature point pairs stored in the step S5h, selecting a plurality of feature point pairs and analyzing pixel coordinates of the feature point pairs;
s7c, calculating the monitoring parameters of the camera by the method of the step S5b according to the characteristic points selected in the step S7 b;
s7d comparing the camera monitoring parameters obtained in the step S7c with the storage parameters obtained in the step S5b, if the change is smaller than a set threshold, updating the three-dimensional model positioning and displaying of the scene video images through the steps S5c to S5h according to the monitoring parameters, and updating and storing the selected feature point pairs and the coordinates thereof, the camera monitoring parameters and the three-dimensional model static images under the projector view angle; if the threshold value is larger than the set threshold value, an alarm is given to prompt field adjustment and maintenance.
8. The three-dimensional scene-based video fusion and video linkage method according to claim 1, wherein the method is based on image depth processing of regional surveillance videos and precise matching and fusion with three-dimensional models, and further realizes precise spatial matching and fusion display of other situation awareness information, equipment facility information and task management and control information, so that full-dimensional situation fusion monitoring based on three-dimensional scenes and intelligent spatial positioning analysis and associated information fusion display based on hotspot tours, planning tasks and detection alarms are realized.
9. An electronic device comprising a processor and a memory storing execution instructions, wherein when the processor executes the execution instructions stored in the memory, the processor executes the three-dimensional scene-based video fusion and video linkage method according to any one of claims 1 to 8.
CN202010598667.1A 2020-06-28 2020-06-28 Video fusion and video linkage method based on three-dimensional scene and electronic equipment Active CN111836012B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010598667.1A CN111836012B (en) 2020-06-28 2020-06-28 Video fusion and video linkage method based on three-dimensional scene and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010598667.1A CN111836012B (en) 2020-06-28 2020-06-28 Video fusion and video linkage method based on three-dimensional scene and electronic equipment

Publications (2)

Publication Number Publication Date
CN111836012A CN111836012A (en) 2020-10-27
CN111836012B true CN111836012B (en) 2022-05-13

Family

ID=72898987

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010598667.1A Active CN111836012B (en) 2020-06-28 2020-06-28 Video fusion and video linkage method based on three-dimensional scene and electronic equipment

Country Status (1)

Country Link
CN (1) CN111836012B (en)

Families Citing this family (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112437276B (en) * 2020-11-20 2023-04-07 埃洛克航空科技（北京）有限公司 WebGL-based three-dimensional video fusion method and system
CN112445995B (en) * 2020-11-30 2024-02-13 北京邮电大学 Scene fusion display method and device under WebGL
CN112506465B (en) * 2020-12-01 2023-03-21 建信金融科技有限责任公司 Method and device for switching scenes in panoramic roaming
CN114630040B (en) * 2020-12-10 2023-07-14 中国人民解放军陆军工程大学 Self-adaptive tracking system and method for linkage of laser radar and dome camera
CN113259624A (en) * 2021-03-24 2021-08-13 北京潞电电气设备有限公司 Monitoring equipment and method thereof
CN112802208B (en) * 2021-03-30 2021-06-22 中国民用航空总局第二研究所 Three-dimensional visualization method and device in terminal building
CN113532390B (en) * 2021-06-18 2023-07-07 广州领汇信息科技有限公司 Target positioning method, device and storage medium based on artificial intelligence technology
CN113542722A (en) * 2021-06-30 2021-10-22 石家庄科林电气设备有限公司 Real-time data, three-dimensional model and video combined display system in electric power operation and maintenance
CN113452984B (en) * 2021-06-30 2023-02-03 石家庄科林电气设备有限公司 Real-time data, three-dimensional model and video combined display method in electric power operation and maintenance
CN113643788B (en) * 2021-07-15 2024-04-05 北京复数健康科技有限公司 Method and system for determining feature points based on multiple image acquisition devices
CN113703703A (en) * 2021-08-23 2021-11-26 深圳市道通智能航空技术股份有限公司 Unmanned aerial vehicle data display method, device, equipment and storage medium
CN114255285B (en) * 2021-12-23 2023-07-18 奥格科技股份有限公司 Video and urban information model three-dimensional scene fusion method, system and storage medium
CN114442805A (en) * 2022-01-06 2022-05-06 上海安维尔信息科技股份有限公司 Monitoring scene display method and system, electronic equipment and storage medium
CN115243091B (en) * 2022-06-14 2024-02-02 北京箩筐时空数据技术有限公司 Map track dynamic display method and device
CN114818992B (en) * 2022-06-23 2022-09-23 成都索贝数码科技股份有限公司 Image data analysis method, scene estimation method and 3D fusion method
CN115376313B (en) * 2022-07-26 2024-07-16 四川智慧高速科技有限公司 Image fusion and distortion correction implementation method based on monitoring camera group
CN115908706B (en) * 2022-11-15 2023-08-08 中国铁路设计集团有限公司 High-speed railway completion acceptance method with fusion of live three-dimensional model and image
CN117237438B (en) * 2023-09-18 2024-06-28 共享数据(福建)科技有限公司 Range matching method and terminal for three-dimensional model and unmanned aerial vehicle video data
CN117336459B (en) * 2023-10-10 2024-04-30 雄安雄创数字技术有限公司 Three-dimensional video fusion method and device, electronic equipment and storage medium
CN117152400B (en) * 2023-10-30 2024-03-19 武汉苍穹融新科技有限公司 Method and system for fusing multiple paths of continuous videos and three-dimensional twin scenes on traffic road
CN117237512B (en) * 2023-11-10 2024-03-12 深圳市易图资讯股份有限公司 Three-dimensional scene mapping method and system for video image
CN117278731B (en) * 2023-11-21 2024-05-28 启迪数字科技(深圳)有限公司 Multi-video and three-dimensional scene fusion method, device, equipment and storage medium
CN118411500B (en) * 2024-06-27 2024-08-27 杭州海康威视数字技术股份有限公司 Portable imaging device-based operation scene detection method and portable imaging device

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103716586A (en) * 2013-12-12 2014-04-09 中国科学院深圳先进技术研究院 Monitoring video fusion system and monitoring video fusion method based on three-dimension space scene
US8854282B1 (en) * 2011-09-06 2014-10-07 Google Inc. Measurement method
CN105096269A (en) * 2015-07-21 2015-11-25 北京交通大学 Radial image distortion rectifying method and system based on distorted linear structure detection
CN105956554A (en) * 2016-04-29 2016-09-21 广西科技大学 Face identification method
CN106204656A (en) * 2016-07-21 2016-12-07 中国科学院遥感与数字地球研究所 Target based on video and three-dimensional spatial information location and tracking system and method
CN106210643A (en) * 2016-07-29 2016-12-07 林玉峰 A kind of video camera viewing area call method
CN110737989A (en) * 2019-10-18 2020-01-31 中国科学院深圳先进技术研究院 parallel intelligent emergency cooperation method, system and electronic equipment

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9264689B2 (en) * 2011-08-04 2016-02-16 Semiconductor Components Industries, Llc Systems and methods for color compensation in multi-view video
CN104376542B (en) * 2014-10-25 2019-04-23 深圳市金立通信设备有限公司 A kind of image enchancing method
US10572982B2 (en) * 2017-10-04 2020-02-25 Intel Corporation Method and system of image distortion correction for images captured by using a wide-angle lens
CN110070624B (en) * 2019-04-26 2020-05-08 厦门大学 Urban geomorphology feature identification method based on VR combined with eye movement tracking

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8854282B1 (en) * 2011-09-06 2014-10-07 Google Inc. Measurement method
CN103716586A (en) * 2013-12-12 2014-04-09 中国科学院深圳先进技术研究院 Monitoring video fusion system and monitoring video fusion method based on three-dimension space scene
CN105096269A (en) * 2015-07-21 2015-11-25 北京交通大学 Radial image distortion rectifying method and system based on distorted linear structure detection
CN105956554A (en) * 2016-04-29 2016-09-21 广西科技大学 Face identification method
CN106204656A (en) * 2016-07-21 2016-12-07 中国科学院遥感与数字地球研究所 Target based on video and three-dimensional spatial information location and tracking system and method
CN106210643A (en) * 2016-07-29 2016-12-07 林玉峰 A kind of video camera viewing area call method
CN110737989A (en) * 2019-10-18 2020-01-31 中国科学院深圳先进技术研究院 parallel intelligent emergency cooperation method, system and electronic equipment

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
High-Accuracy Angle Detection for the Ultrawide FOV Acquisition Scheme in Free-Space Optical Links;L. Liu;《Lightwave Technology》;20141002;全文 *
Smooth adaptive fitting of 3D models using hierarchical triangular splines;A. Yvart;《International Conference on Shape Modeling and Applications 2005》;20051227;全文 *
增强现实(AR)技术在高速公路视频监控中的应用;曾祥平;《公路交通科技(应用技术版)》;20200115(第1(2020)期);全文 *
面向复杂三维场景的高质量纹理映射;姜翰青;《计算机学报》;20151215;第38卷;全文 *

Also Published As

Publication number Publication date
CN111836012A (en) 2020-10-27

Similar Documents

Publication Publication Date Title
CN111836012B (en) Video fusion and video linkage method based on three-dimensional scene and electronic equipment
EP3573024B1 (en) Building radar-camera surveillance system
US20210073573A1 (en) Ship identity recognition method based on fusion of ais data and video data
CN103024350B (en) A kind of principal and subordinate&#39;s tracking of binocular PTZ vision system and the system of application the method
US7583815B2 (en) Wide-area site-based video surveillance system
CN102148965B (en) Video monitoring system for multi-target tracking close-up shooting
EP2553924B1 (en) Effortless navigation across cameras and cooperative control of cameras
CN103513295B (en) A kind of weather monitoring system based on polyphaser captured in real-time and image procossing and method
CN105516654B (en) A kind of supervision of the cities video fusion method based on scene structure analysis
CN112053391B (en) Monitoring and early warning method and system based on dynamic three-dimensional model and storage medium
US20080291278A1 (en) Wide-area site-based video surveillance system
CN114693746B (en) Intelligent monitoring system and method based on identity recognition and cross-camera target tracking
Ciampa Pictometry Digital Video Mapping
CN112449093A (en) Three-dimensional panoramic video fusion monitoring platform
CN114827570A (en) Video situation perception and information fusion method based on three-dimensional scene and electronic equipment
CN103198488A (en) PTZ surveillance camera realtime posture rapid estimation method
CN206611521U (en) A kind of vehicle environment identifying system and omni-directional visual module based on multisensor
KR20190059395A (en) Gis 3 cctv 3 cctv
CN103414872A (en) Method for driving PTZ cameras through target position
US11703820B2 (en) Monitoring management and control system based on panoramic big data
Sankaranarayanan et al. A fast linear registration framework for multi-camera GIS coordination
KR101996907B1 (en) Apparatus for tracking object
CN112637509A (en) High-altitude parabolic monitoring and early warning system and method
Chen et al. Immersive surveillance for total situational awareness
CN114241126A (en) Method for extracting object position information in monocular video based on live-action model

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant