CN106713741B - Panoramic video quality diagnosis method and device - Google Patents
Panoramic video quality diagnosis method and device Download PDFInfo
- Publication number
- CN106713741B CN106713741B CN201611034166.0A CN201611034166A CN106713741B CN 106713741 B CN106713741 B CN 106713741B CN 201611034166 A CN201611034166 A CN 201611034166A CN 106713741 B CN106713741 B CN 106713741B
- Authority
- CN
- China
- Prior art keywords
- video image
- camera
- video
- quality diagnosis
- panoramic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N17/00—Diagnosis, testing or measuring for television systems or their details
- H04N17/002—Diagnosis, testing or measuring for television systems or their details for television cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/12—Picture reproducers
- H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
- H04N9/3141—Constructional details thereof
- H04N9/3147—Multi-projection systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/12—Picture reproducers
- H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
- H04N9/3191—Testing thereof
Abstract
The invention discloses a panoramic video quality diagnosis method, which comprises the following steps: acquiring a plurality of paths of video images through a plurality of cameras preset by a panoramic camera; establishing a mathematical model for video image quality diagnosis, and respectively performing quality diagnosis on the video image acquired by each camera according to the mathematical model to obtain a quality diagnosis result of the video image of each camera; and fusing the quality diagnosis result of the video image of each camera to obtain a panoramic video image quality diagnosis result. The invention also discloses a panoramic video quality diagnosis device. The invention improves the convenience of quality diagnosis of the panoramic video image so as to ensure the quality of the panoramic video image.
Description
Technical Field
The invention relates to the technical field of panoramic camera parameter processing, in particular to a panoramic video quality diagnosis method and a panoramic video quality diagnosis device.
Background
Reliable availability of video quality is the most fundamental requirement for video in the field of video surveillance. The video quality diagnosis may be classified into subjective video quality diagnosis and objective video quality diagnosis in type. The subjective video quality diagnosis method has high result accuracy, but is easily limited by experimental environment, poor in operability, too high in cost and the like. Although the objective video quality diagnosis method has a certain distance with the subjective evaluation of human eyes, the method has the advantages of good repeatability, high calculation speed, low evaluation cost, high portability and the like.
However, the current video quality diagnosis is only limited to the video quality diagnosis of a single camera, and is not applied to the quality diagnosis of panoramic video.
Disclosure of Invention
The invention mainly aims to provide a panoramic video quality diagnosis method and a panoramic video quality diagnosis device, and aims to improve the convenience of quality diagnosis of panoramic video images.
In order to achieve the above object, the present invention provides a quality diagnosis method for panoramic video, comprising:
acquiring a plurality of paths of video images through a plurality of cameras preset by a panoramic camera;
establishing a mathematical model for video image quality diagnosis, and respectively performing quality diagnosis on the video image acquired by each camera according to the mathematical model to obtain a quality diagnosis result of the video image of each camera;
and fusing the quality diagnosis result of the video image of each camera to obtain a panoramic video image quality diagnosis result.
Preferably, the fusing the quality diagnosis result of each camera video image to obtain a panoramic video image quality diagnosis result includes:
setting a weight value for the video image of each camera respectively;
respectively calculating the diagnosis result of the video image of each camera with the corresponding set weight value to obtain the diagnosis contribution value of the video image of each camera in the panoramic video image;
and accumulating the diagnosis contribution values of the video images of each camera to obtain a quality diagnosis result of the panoramic video image.
Preferably, the establishing a mathematical model of video image quality diagnosis, and performing quality diagnosis on the video image acquired by each camera according to the mathematical model respectively to obtain a quality diagnosis result of the video image of each camera includes:
determining a video image quality diagnostic function;
setting a corresponding video image quality evaluation function according to the determined image quality diagnosis function, and establishing a mathematical model of video image quality diagnosis according to the video image quality evaluation function;
and respectively carrying out quality diagnosis on the video images acquired by each camera according to the mathematical model to obtain the quality diagnosis result of the video image of each camera.
Preferably, the image quality diagnosis function includes sharpness detection, video noise detection, brightness anomaly detection, video snowflake detection, color cast detection, video freeze detection, video loss detection, and video stitching effect detection.
Preferably, the fusing the quality diagnosis result of the video image of each camera to obtain the panoramic video image quality diagnosis result includes:
establishing a projection model, and projecting the video image of each camera according to the projection model;
extracting feature information of the video image of each camera after projection;
according to the feature information, performing feature matching on every two video images with intersecting areas;
and performing video image fusion on the video image of each camera according to the feature matching result, and generating a panoramic video image according to the video image fusion result.
In addition, to achieve the above object, the present invention provides a quality diagnosis apparatus for panoramic video, including:
the acquisition module is used for acquiring a plurality of paths of video images through a plurality of cameras preset by the panoramic camera;
the diagnosis module is used for establishing a mathematical model for video image quality diagnosis, and performing quality diagnosis on the video image acquired by each camera according to the mathematical model to obtain a quality diagnosis result of the video image of each camera;
and the fusion module is used for fusing the quality diagnosis result of the video image of each camera to obtain a panoramic video image quality diagnosis result.
Preferably, the fusion module is further configured to set a weight value for each video image of each camera;
respectively calculating the diagnosis result of the video image of each camera with the corresponding set weight value to obtain the diagnosis contribution value of the video image of each camera in the panoramic video image;
and accumulating the diagnosis contribution values of the video images of each camera to obtain a quality diagnosis result of the panoramic video image.
Preferably, the diagnostic module is further configured to determine a video image quality diagnostic function;
setting a corresponding video image quality evaluation function according to the determined image quality diagnosis function, and establishing a mathematical model of video image quality diagnosis according to the video image quality evaluation function;
and respectively carrying out quality diagnosis on the video images acquired by each camera according to the mathematical model to obtain the quality diagnosis result of the video image of each camera.
Preferably, the image quality diagnosis function includes sharpness detection, video noise detection, brightness anomaly detection, video snowflake detection, color cast detection, video freeze detection, video loss detection, and video stitching effect detection.
Preferably, the apparatus for diagnosing quality of panoramic video further comprises:
the projection module is used for establishing a projection model and projecting the video image of each camera according to the projection model;
the extraction module is used for extracting the feature information of the video image of each camera after projection;
the matching module is used for performing feature matching on every two video images with the intersection region according to the feature information;
and the generating module is used for carrying out video image fusion on the video image of each camera according to the feature matching result and generating a panoramic video image according to the video image fusion result.
According to the embodiment of the invention, a plurality of paths of video images are obtained through a plurality of cameras preset by the panoramic camera, and a mathematical model for video image quality diagnosis is established to carry out quality diagnosis on the video images obtained by each camera, so that a quality diagnosis result of the video image of each camera is obtained. And then fusing the quality diagnosis results of the video images of each camera to obtain a panoramic video image quality diagnosis result. The quality diagnosis of the panoramic video image is realized, and the convenience of the quality diagnosis of the panoramic video image is improved, so that the quality of the panoramic video image is ensured.
Drawings
Fig. 1 is a schematic flow chart of a panoramic video quality diagnosis method according to a first embodiment of the present invention;
FIG. 2 is a schematic diagram of the position relationship of the viewing planes in the coordinate system of the panoramic camera according to the present invention;
FIG. 3 is a schematic view of a cylindrical image projection in a panoramic camera coordinate system in accordance with the present invention;
fig. 4 is a functional block diagram of a panoramic video quality diagnosis apparatus according to a first embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
As shown in fig. 1, a first embodiment of the method for diagnosing quality of a panoramic video according to the present invention is shown. The quality diagnosis method of the panoramic video of the embodiment includes:
s10, acquiring a plurality of paths of video images through a plurality of cameras preset by the panoramic camera;
in this embodiment, the panoramic video quality diagnosis method is applied to a panoramic camera for performing quality diagnosis on a collected panoramic video image, the panoramic camera may include a plurality of cameras, and before performing quality diagnosis on a panoramic video, a plurality of paths of video images need to be acquired through a plurality of cameras preset by the panoramic camera, so as to perform quality diagnosis on the paths of video images respectively.
Step S20, establishing a mathematical model for video image quality diagnosis, and performing quality diagnosis on the video images acquired by each camera according to the mathematical model to obtain a quality diagnosis result of the video images of each camera;
after obtaining the multi-channel video images, establishing a mathematical model for video image quality diagnosis, and applying the mathematical model to the video images acquired by each camera for quality diagnosis.
Preferably, the step S20 includes: determining a video image quality diagnostic function;
setting a corresponding video image quality evaluation function according to the determined image quality diagnosis function, and establishing a mathematical model of video image quality diagnosis according to the video image quality evaluation function;
and respectively carrying out quality diagnosis on the video images acquired by each camera according to the mathematical model to obtain the quality diagnosis result of the video image of each camera.
Specifically, an image quality diagnosis function is first determined, which includes sharpness detection, video noise detection, luminance abnormality detection, video snowflake detection, color cast detection, video freeze detection, video loss detection, video stitching effect detection, and the like.
And then selecting a proper video image quality evaluation function according to the selected image quality diagnosis function, wherein the evaluation function is related to the determined image quality diagnosis function, and different image quality diagnosis functions need to establish corresponding evaluation functions so as to establish a mathematical model of the diagnosis function.
applying the mathematical model to the video images acquired by each camera for quality diagnosis to obtain the quality diagnosis result of each video image, and assuming that the diagnosis value corresponding to the video images of the N cameras is αi(i∈[1,N])。
Taking sharpness detection as an example, when a video image to be detected is a color image, graying the color image, and then adopting a Tenengrad function as a sharpness evaluation function.
The Tenengrad function is to extract the gradient values in the horizontal and vertical directions using Sobel operators. In gradient detection, filtering the image by using a first order differential form of a gaussian function to obtain the following formula (1):
the gradient is a vector and the direction of the gradient of the function gives the direction in which the directional derivative takes the maximum, as shown in equation (2) below:
and the directional derivative of this direction is equal to the modulus of the gradient, as shown in equation (3) below:
the derivative of equation (1) for digital images can be approximated by a difference, the simplest gradient expression being:
the evaluation function f (I) is defined as the sum of the squares of the gradients, the gradient G (x, y) being above a threshold value T, i.e.
In the formula
The method is the convolution of a Sobel operator on a point (x, y), and T is an empirical value, so that the quality diagnosis of performing definition detection on each video image according to the established mathematical model is realized.
And step S30, fusing the quality diagnosis results of the video images of each camera to obtain a panoramic video image quality diagnosis result.
After the video images collected by each camera are subjected to quality diagnosis according to the mathematical model, the diagnosis results of each video image are subjected to weighted fusion to generate panoramic video image diagnosis results. The fusion method includes direct average fusion, median filtering, feathering, weighted/linear fusion, multiband fusion or pyramid fusion, etc. For example, a weight is distributed to each video image, then a mathematical operation is performed on the diagnosis result of each video image and the distributed weight of the video image to obtain a diagnosis contribution value of the video image in the panoramic video image, then the diagnosis contribution values of each video image are accumulated to obtain a quality diagnosis result of the panoramic video image, and finally the quality diagnosis result of the panoramic video image is output. The following examples will be described in detail.
According to the embodiment of the invention, a plurality of paths of video images are obtained through a plurality of cameras preset by the panoramic camera, and a mathematical model for video image quality diagnosis is established to carry out quality diagnosis on the video images obtained by each camera, so that a quality diagnosis result of the video image of each camera is obtained. And then fusing the quality diagnosis results of the video images of each camera to obtain a panoramic video image quality diagnosis result. The quality diagnosis of the panoramic video image is realized, and the convenience of the quality diagnosis of the panoramic video image is improved, so that the quality of the panoramic video image is ensured.
Further, based on the first embodiment of the quality diagnosis method for panoramic video, a second embodiment of the quality diagnosis method for panoramic video according to the present invention is provided, in which the step S30 includes:
setting a weight value for the video image of each camera respectively;
respectively calculating the diagnosis result of the video image of each camera with the corresponding set weight value to obtain the diagnosis contribution value of the video image of each camera in the panoramic video image;
and accumulating the diagnosis contribution values of the video images of each camera to obtain a quality diagnosis result of the panoramic video image.
In this embodiment, a weight is first distributed to the video images collected by each camera in the panoramic camera, and it is assumed that the weights of the N cameras are ω1,ω2,…,ωNWherein, ω isiThe value can be simply taken as
The weight is set in this way without taking into account the contribution ratio of the area size. OmegaiThe value can also be taken as a weight value according to the ratio of the pixel of the video image of each camera to the pixel of the video images of all cameras, and the total pixel of N cameras is assumed to be NpThe video image of each camera comprises M pixelsi(i is 1,2, …, N), then
Performing mathematical operation on the quality diagnosis result of the video image of each camera corresponding to the weight value of the video image distribution of the camera to obtain the diagnosis contribution value of the video image of the camera in the panoramic video image, and the contribution value x of the quality diagnosis of the video image of each camerai(i∈[1,N]) The calculation method may be, but is not limited to, the following method: x is the number ofi＝αi·ωi(i∈[1,N])。
Accumulating the diagnosis contribution values of the video images of all the cameras to obtain a quality diagnosis result of the panoramic video image, wherein the quality diagnosis result is y, and the y calculation mode is
In the embodiment, the video images of the cameras are respectively and correspondingly set with a weight, the diagnosis result of the video image of each camera is combined for calculation, so that the diagnosis contribution value of the video image of each camera in the panoramic video image is obtained, the diagnosis contribution values are accumulated to obtain the quality diagnosis result of the panoramic video image, and the reliability and the flexibility of quality diagnosis of the panoramic video image are improved.
Further, based on the first or second embodiment of the quality diagnosis method for panoramic video, a third embodiment of the quality diagnosis method for panoramic video according to the present invention is provided, where the step S30 includes:
establishing a projection model, and projecting the video image of each camera according to the projection model;
extracting feature information of the video image of each camera after projection;
according to the feature information, performing feature matching on every two video images with intersecting areas;
and performing video image fusion on the video image of each camera according to the feature matching result, and generating a panoramic video image according to the video image fusion result.
In this embodiment, after the quality diagnosis is performed on the video images acquired by each camera and the quality diagnosis results of each video image are fused to obtain the quality diagnosis result of the panoramic video image, multiple paths of video images can be spliced into one path of panoramic video image.
Firstly, a projection model is established, wherein the projection model comprises a plane, a cylindrical surface, a spherical surface or a polyhedron and the like. Then, projecting each path of video image according to the projection model, wherein the projection model comprises the following steps: by establishing a projection model, multiple paths of video images are drawn on the projection model, and the projection positions among the video images can be expressed according to the mapping relation among the matched feature points, so that the video images to be spliced are mapped to a specified coordinate space.
And extracting image features of the projected video images to obtain feature information of the video images, for example, extracting the feature information of the images through a feature detection operator and a descriptor. The feature information includes physical features of the video image, content description features of the video image, and the like, for example, Harris operators, Sift features, and the like.
After extracting the feature information of the video image, performing feature matching on each two paths of video images with intersecting areas, including: by extracting features (points, lines, faces, etc.) of two or more images respectively, feature descriptors are generated, and then matching is performed by the feature descriptors. The feature matching method includes a stream-based feature matching method, a phase-based feature matching method, a feature-based feature matching method, or the like.
Then, video image fusion is carried out according to the feature matching result, and the method comprises the following steps: and splicing the video images of the cameras, and smoothing the spliced boundary to ensure that the gap is in natural transition. The fusion method of the video image features further comprises direct average fusion, median filtering, feathering, weighted/linear fusion, multiband fusion or pyramid fusion and the like. And generating a panoramic video according to the video image fusion result.
The following is an example of cylindrical projection:
assuming that the coordinate system of the camera in the panoramic camera is xyz, the positional relationship of the view plane is as shown in fig. 2, where Z ═ -f is the view plane, the coordinate value of any point on the live-action image on the Z axis is — f, and assuming that there is no deviation in the lens center of the camera, the center of the live-action image is the intersection point between the optical axis of the camera (Z axis in the camera coordinate system) and the view plane. In the camera coordinate system, the X axis and the Y axis are parallel to the horizontal axis and the vertical axis of the image coordinate system, respectively, so that the pixel coordinate of any pixel point (X, Y) on the live-action image in the camera coordinate system oyx is (X-W/2, Y-H, -f), where W and H represent the width and height of the live-action image, respectively.
As shown in fig. 3, J is a real-scene image captured by the camera, and P (x, y) is a pixel point on the real-scene image J, the coordinates of the pixel point in the camera coordinate system can be represented as:
wherein, W and H are the width and height of the real image, the origin O in the camera coordinate system is the center of the cylinder, the radius of the cylinder is the focal length f of the camera, and the projection target of the cylinder surface is to find the projection point Q (x ', y') of any pixel point P (x, y) in the real image J on the cylinder surface.
The linear equation of the origin O and the pixel point P in the camera coordinate system can be expressed in the form of a parametric equation, as shown in the following equation (11):
where t represents a parameter, then the equation for the cylinder can be expressed as:
u2+v2＝f2(12)
simultaneous formula (11) and formula (12) can be obtained:
where (u, v, w) is the projected coordinates of P (x, y) on the cylindrical surface, the conversion of the three-dimensional parameter coordinates into two-dimensional image coordinates can be achieved by equation (14):
wherein the content of the first and second substances,
hfov is the horizontal viewing angle of the camera.
The projection transformation formula of any point P (x, y) on the live-action image J projected to the corresponding point Q (x ', y') on the cylindrical coordinate system can be obtained by combining the formula (13) and the formula (14):
where the camera focal length f is W/(2tan (hfov/2)), and hfov is the horizontal viewing angle of the camera.
Accordingly, the cylindrical back-projection formula can be derived from the cylindrical projection formula (15):
after the multi-channel video images are projected, the feature information can be extracted, feature matching is carried out, and the multi-channel video images are fused to generate a panoramic video image according to a matching result.
In the embodiment, the projection model is established to project the multiple paths of video images, the feature information and the feature matching are performed on the projected video images, the multiple paths of video images are fused to generate the panoramic video image according to the feature matching result, and the accuracy and the convenience of the fusion of the panoramic video image are improved.
Correspondingly, as shown in fig. 4, a first embodiment of the panoramic video quality diagnosis apparatus of the present invention is proposed. The quality diagnosis apparatus of a panoramic video of the embodiment includes:
the acquisition module 100 is configured to acquire multiple paths of video images through multiple cameras preset by the panoramic camera;
in this embodiment, the panoramic video quality diagnosis apparatus is applied to a panoramic camera to perform quality diagnosis on a collected panoramic video image, the panoramic camera may include a plurality of cameras, and before performing quality diagnosis on a panoramic video, the obtaining module 100 first needs to obtain a plurality of paths of video images through a plurality of cameras preset by the panoramic camera, so as to perform quality diagnosis on the plurality of paths of video images respectively.
The diagnosis module 200 is used for establishing a mathematical model for video image quality diagnosis, and performing quality diagnosis on the video images acquired by each camera according to the mathematical model to obtain a quality diagnosis result of the video images of each camera;
after obtaining the multiple paths of video images, the diagnosis module 200 establishes a mathematical model for quality diagnosis of the video images, and applies the mathematical model to the video images acquired by each camera for quality diagnosis.
Preferably, the diagnosis module 200 is further configured to determine a video image quality diagnosis function;
setting a corresponding video image quality evaluation function according to the determined image quality diagnosis function, and establishing a mathematical model of video image quality diagnosis according to the video image quality evaluation function;
and respectively carrying out quality diagnosis on the video images acquired by each camera according to the mathematical model to obtain the quality diagnosis result of the video image of each camera.
Specifically, an image quality diagnosis function is first determined, which includes sharpness detection, video noise detection, luminance abnormality detection, video snowflake detection, color cast detection, video freeze detection, video loss detection, video stitching effect detection, and the like.
And then selecting a proper video image quality evaluation function according to the selected image quality diagnosis function, wherein the evaluation function is related to the determined image quality diagnosis function, and different image quality diagnosis functions need to establish corresponding evaluation functions so as to establish a mathematical model of the diagnosis function.
applying the mathematical model to the video images acquired by each camera for quality diagnosis to obtain the quality diagnosis result of each video image, and assuming that the diagnosis value corresponding to the video images of the N cameras is αi(i∈[1,N])。
Taking sharpness detection as an example, when a video image to be detected is a color image, graying the color image, and then adopting a Tenengrad function as a sharpness evaluation function.
The Tenengrad function is to extract the gradient values in the horizontal and vertical directions using Sobel operators. In gradient detection, the image is filtered using the first order differential form of the gaussian function, resulting in the following equation (10):
the gradient is a vector and the direction of the gradient of the function gives the direction in which the directional derivative takes the maximum, as shown in equation (20) below:
and the directional derivative of this direction is equal to the modulus of the gradient, as shown in equation (30) below:
the derivative of equation (1) for digital images can be approximated by a difference, the simplest gradient expression being:
the evaluation function f (I) is defined as the sum of the squares of the gradients, the gradient G (x, y) being above a threshold value T, i.e.
In the formula
The method is the convolution of a Sobel operator on a point (x, y), and T is an empirical value, so that the quality diagnosis of performing definition detection on each video image according to the established mathematical model is realized.
And a fusion module 300, configured to fuse the quality diagnosis result of the video image of each camera to obtain a panoramic video image quality diagnosis result.
After the video images acquired by each camera are subjected to quality diagnosis according to the mathematical model, the fusion module 300 performs weighted fusion on the diagnosis result of each video image to generate a panoramic video image diagnosis result. The fusion method includes direct average fusion, median filtering, feathering, weighted/linear fusion, multiband fusion or pyramid fusion, etc. For example, a weight is distributed to each video image, then a mathematical operation is performed on the diagnosis result of each video image and the distributed weight of the video image to obtain a diagnosis contribution value of the video image in the panoramic video image, then the diagnosis contribution values of each video image are accumulated to obtain a quality diagnosis result of the panoramic video image, and finally the quality diagnosis result of the panoramic video image is output. The following examples will be described in detail.
According to the embodiment of the invention, a plurality of paths of video images are obtained through a plurality of cameras preset by the panoramic camera, and a mathematical model for video image quality diagnosis is established to carry out quality diagnosis on the video images obtained by each camera, so that a quality diagnosis result of the video image of each camera is obtained. And then fusing the quality diagnosis results of the video images of each camera to obtain a panoramic video image quality diagnosis result. The quality diagnosis of the panoramic video image is realized, and the convenience of the quality diagnosis of the panoramic video image is improved, so that the quality of the panoramic video image is ensured.
Further, based on the first embodiment of the panoramic video quality diagnosis apparatus, a second embodiment of the panoramic video quality diagnosis apparatus of the present invention is provided, in which the fusion module 300 is further configured to set a weight value for each video image of each camera;
respectively calculating the diagnosis result of the video image of each camera with the corresponding set weight value to obtain the diagnosis contribution value of the video image of each camera in the panoramic video image;
and accumulating the diagnosis contribution values of the video images of each camera to obtain a quality diagnosis result of the panoramic video image.
In this embodiment, the fusion module 300 first distributes a weight to the video image collected by each camera in the panoramic camera, assuming that the weights of the N cameras are ω respectively1,ω2,…,ωNWherein, ω isiThe value can be simply taken as
The weight is set in this way without taking into account the contribution ratio of the area size. OmegaiThe value can also be taken as a weight value according to the ratio of the pixel of the video image of each camera to the pixel of the video images of all cameras, and the total pixel of N cameras is assumed to be NpThe video image of each camera comprises M pixelsi(i is 1,2, …, N), then
Corresponding the quality diagnosis result of the video image of each cameraPerforming mathematical operation with the weight value of the video image distribution of the camera to obtain the diagnosis contribution value of the video image of the camera in the panoramic video image, and the contribution value x of the video image quality diagnosis of each camerai(i∈[1,N]) The calculation method may be, but is not limited to, the following method: x is the number ofi＝αi·ωi(i∈[1,N])。
Accumulating the diagnosis contribution values of the video images of all the cameras to obtain a quality diagnosis result of the panoramic video image, wherein the quality diagnosis result is y, and the y calculation mode is
In the embodiment, the video images of the cameras are respectively and correspondingly set with a weight, the diagnosis result of the video image of each camera is combined for calculation, so that the diagnosis contribution value of the video image of each camera in the panoramic video image is obtained, the diagnosis contribution values are accumulated to obtain the quality diagnosis result of the panoramic video image, and the reliability and the flexibility of quality diagnosis of the panoramic video image are improved.
Further, a third embodiment of the panoramic video quality diagnosis apparatus according to the present invention is proposed based on the first or second embodiment of the panoramic video quality diagnosis apparatus, and the panoramic video quality diagnosis apparatus further includes:
the projection module is used for establishing a projection model and projecting the video image of each camera according to the projection model;
the extraction module is used for extracting the feature information of the video image of each camera after projection;
the matching module is used for performing feature matching on every two video images with the intersection region according to the feature information;
and the generating module is used for carrying out video image fusion on the video image of each camera according to the feature matching result and generating a panoramic video image according to the video image fusion result.
In this embodiment, after the quality diagnosis is performed on the video images acquired by each camera and the quality diagnosis results of each video image are fused to obtain the quality diagnosis result of the panoramic video image, multiple paths of video images can be spliced into one path of panoramic video image.
Firstly, a projection model is established by a projection module, and the projection model comprises a plane, a cylindrical surface, a spherical surface or a polyhedron and the like. Then, projecting each path of video image according to the projection model, wherein the projection model comprises the following steps: by establishing a projection model, multiple paths of video images are drawn on the projection model, and the projection positions among the video images can be expressed according to the mapping relation among the matched feature points, so that the video images to be spliced are mapped to a specified coordinate space.
The extraction module extracts image features of the projected video images to obtain feature information of the video images, for example, feature information of the images is extracted through a feature detection operator and a descriptor. The feature information includes physical features of the video image, content description features of the video image, and the like, for example, Harris operators, Sift features, and the like.
After extracting the feature information of the video image, the matching module performs feature matching on every two paths of video images with intersecting areas, and the feature matching comprises the following steps: by extracting features (points, lines, faces, etc.) of two or more images respectively, feature descriptors are generated, and then matching is performed by the feature descriptors. The feature matching method includes a stream-based feature matching method, a phase-based feature matching method, a feature-based feature matching method, or the like.
Then the generation module performs video image fusion according to the feature matching result, and the method comprises the following steps: and splicing the video images of the cameras, and smoothing the spliced boundary to ensure that the gap is in natural transition. The fusion method of the video image characteristics further comprises direct average fusion, median filtering, feathering, weighting/linear fusion, multiband fusion or pyramid fusion and the like, and generates the panoramic video according to the video image fusion result.
The following is an example of cylindrical projection:
assuming that the coordinate system of the camera in the panoramic camera is xyz, the positional relationship of the view plane is as shown in fig. 2, where Z ═ -f is the view plane, the coordinate value of any point on the live-action image on the Z axis is — f, and assuming that there is no deviation in the lens center of the camera, the center of the live-action image is the intersection point between the optical axis of the camera (Z axis in the camera coordinate system) and the view plane. In the camera coordinate system, the X axis and the Y axis are parallel to the horizontal axis and the vertical axis of the image coordinate system, respectively, so that the pixel coordinate of any pixel point (X, Y) on the live-action image in the camera coordinate system oyx is (X-W/2, Y-H, -f), where W and H represent the width and height of the live-action image, respectively.
As shown in fig. 3, J is a real-scene image captured by the camera, and P (x, y) is a pixel point on the real-scene image J, the coordinates of the pixel point in the camera coordinate system can be represented as:
wherein, W and H are the width and height of the real image, the origin O in the camera coordinate system is the center of the cylinder, the radius of the cylinder is the focal length f of the camera, and the projection target of the cylinder surface is to find the projection point Q (x ', y') of any pixel point P (x, y) in the real image J on the cylinder surface.
The equation of the straight line between the origin O and the pixel point P in the camera coordinate system can be expressed in the form of a parametric equation, as shown in the following formula (110):
where t represents a parameter, then the equation for the cylinder can be expressed as:
u2+v2＝f2(120)
simultaneous equations (110) and (120) can be obtained:
where (u, v, w) is the projected coordinates of P (x, y) on the cylindrical surface, the conversion of the three-dimensional parameter coordinates into two-dimensional image coordinates can be achieved using equation (140):
wherein the content of the first and second substances,
hfov is the horizontal viewing angle of the camera.
The projection transformation formula of any point P (x, y) on the live-action image J projected to the corresponding point Q (x ', y') on the cylindrical coordinate system can be obtained by combining the formula (130) and the formula (140):
where the camera focal length f is W/(2tan (hfov/2)), and hfov is the horizontal viewing angle of the camera.
Accordingly, a cylindrical back-projection equation can be derived from the cylindrical projection equation (150):
after the multi-channel video images are projected, the feature information can be extracted, feature matching is carried out, and the multi-channel video images are fused to generate a panoramic video image according to a matching result.
In the embodiment, the projection model is established to project the multiple paths of video images, the feature information and the feature matching are performed on the projected video images, the multiple paths of video images are fused to generate the panoramic video image according to the feature matching result, and the accuracy and the convenience of the fusion of the panoramic video image are improved.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (4)
1. A method for diagnosing the quality of a panoramic video is characterized by comprising the following steps:
acquiring a plurality of paths of video images through a plurality of cameras preset by a panoramic camera;
establishing a mathematical model for video image quality diagnosis, and respectively performing quality diagnosis on the video image acquired by each camera according to the mathematical model to obtain a quality diagnosis result of the video image of each camera;
fusing the quality diagnosis result of the video image of each camera to obtain a panoramic video image quality diagnosis result;
establishing a projection model, and projecting the video image of each camera according to the projection model;
extracting feature information of the video image of each camera after projection;
according to the feature information, performing feature matching on every two video images with intersecting areas;
performing video image fusion on the video image of each camera according to the feature matching result, and generating a panoramic video image according to the video image fusion result;
the establishing of the mathematical model for video image quality diagnosis, and the quality diagnosis of the video image acquired by each camera according to the mathematical model, to obtain the quality diagnosis result of the video image of each camera, includes:
determining a video image quality diagnostic function;
setting a corresponding video image quality evaluation function according to the determined image quality diagnosis function, and establishing a mathematical model of video image quality diagnosis according to the video image quality evaluation function;
respectively performing quality diagnosis on the video images acquired by each camera according to the mathematical model to obtain the quality diagnosis result of the video image of each camera;
the image quality diagnosis function comprises definition detection, video noise detection, brightness abnormity detection, video snowflake detection, color cast detection, video freezing detection, video loss detection and video splicing effect detection.
2. The method for quality diagnosis of panoramic video according to claim 1, wherein the fusing the quality diagnosis result of each camera video image to obtain the quality diagnosis result of panoramic video image comprises:
setting a weight value for the video image of each camera respectively;
respectively calculating the diagnosis result of the video image of each camera with the corresponding set weight value to obtain the diagnosis contribution value of the video image of each camera in the panoramic video image;
and accumulating the diagnosis contribution values of the video images of each camera to obtain a quality diagnosis result of the panoramic video image.
3. A quality diagnosis apparatus for panoramic video, comprising:
the acquisition module is used for acquiring a plurality of paths of video images through a plurality of cameras preset by the panoramic camera;
the diagnosis module is used for establishing a mathematical model for video image quality diagnosis, and performing quality diagnosis on the video image acquired by each camera according to the mathematical model to obtain a quality diagnosis result of the video image of each camera;
the fusion module is used for fusing the quality diagnosis result of the video image of each camera to obtain a panoramic video image quality diagnosis result;
the projection module is used for establishing a projection model and projecting the video image of each camera according to the projection model;
the extraction module is used for extracting the feature information of the video image of each camera after projection;
the matching module is used for performing feature matching on every two video images with the intersection region according to the feature information;
the generating module is used for carrying out video image fusion on the video image of each camera according to the feature matching result and generating a panoramic video image according to the video image fusion result;
wherein the diagnostic module is further configured to determine a video image quality diagnostic function;
setting a corresponding video image quality evaluation function according to the determined image quality diagnosis function, and establishing a mathematical model of video image quality diagnosis according to the video image quality evaluation function;
respectively performing quality diagnosis on the video images acquired by each camera according to the mathematical model to obtain the quality diagnosis result of the video image of each camera;
the image quality diagnosis function comprises definition detection, video noise detection, brightness abnormity detection, video snowflake detection, color cast detection, video freezing detection, video loss detection and video splicing effect detection.
4. The panoramic video quality diagnosis device of claim 3, wherein the fusion module is further configured to set a weight value for each video image of each camera;
respectively calculating the diagnosis result of the video image of each camera with the corresponding set weight value to obtain the diagnosis contribution value of the video image of each camera in the panoramic video image;
and accumulating the diagnosis contribution values of the video images of each camera to obtain a quality diagnosis result of the panoramic video image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611034166.0A CN106713741B (en) | 2016-11-16 | 2016-11-16 | Panoramic video quality diagnosis method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611034166.0A CN106713741B (en) | 2016-11-16 | 2016-11-16 | Panoramic video quality diagnosis method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106713741A CN106713741A (en) | 2017-05-24 |
| CN106713741B true CN106713741B (en) | 2020-09-25 |
Family
ID=58941278
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611034166.0A Active CN106713741B (en) | 2016-11-16 | 2016-11-16 | Panoramic video quality diagnosis method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106713741B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108337505B (en) * | 2018-01-30 | 2020-11-10 | 百度在线网络技术（北京）有限公司 | Information acquisition method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102169576A (en) * | 2011-04-02 | 2011-08-31 | 北京理工大学 | Quantified evaluation method of image mosaic algorithms |
| CN103413298A (en) * | 2013-07-17 | 2013-11-27 | 宁波大学 | Three-dimensional image objective evaluation method based on visual characteristics |
| CN103780870A (en) * | 2012-10-17 | 2014-05-07 | 杭州海康威视数字技术股份有限公司 | Video image quality diagnosis system and method thereof |
| CN104243973A (en) * | 2014-08-28 | 2014-12-24 | 北京邮电大学 | Video perceived quality non-reference objective evaluation method based on areas of interest |
| CN105447865A (en) * | 2015-11-23 | 2016-03-30 | 深圳进化动力数码科技有限公司 | Method and device for evaluating static splicing quality of panoramic splicing algorithm |
| CN105915892A (en) * | 2016-05-06 | 2016-08-31 | 乐视控股（北京）有限公司 | Panoramic video quality determination method and system |
Family Cites Families (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4370726B2 (en) * | 2001-02-15 | 2009-11-25 | 株式会社ニコン | Electronic camera and image processing program |
| US7860343B2 (en) * | 2006-04-10 | 2010-12-28 | Nokia Corporation | Constructing image panorama using frame selection |
| CN101394485B (en) * | 2007-09-20 | 2011-05-04 | 华为技术有限公司 | Image generating method, apparatus and image composition equipment |
| CN101256275B (en) * | 2008-04-08 | 2011-08-31 | 天津大学 | Microminiaturization co-image face panoramic imagery technique |
| CN102231844B (en) * | 2011-07-21 | 2013-04-03 | 西安电子科技大学 | Video image fusion performance evaluation method based on structure similarity and human vision |
| CN103049893B (en) * | 2011-10-14 | 2015-12-16 | 深圳信息职业技术学院 | A kind of method of image fusion quality assessment and device |
| CN102881006A (en) * | 2012-08-03 | 2013-01-16 | 吉林禹硕动漫游戏科技股份有限公司 | Method for splicing and fusing image in multi-projection display system |
| CN102946548B (en) * | 2012-11-27 | 2015-02-18 | 西安电子科技大学 | Video image fusion performance evaluation method based on three-dimensional Log-Gabor conversion |
| CN103152600B (en) * | 2013-03-08 | 2015-04-08 | 天津大学 | Three-dimensional video quality evaluation method |
| CN103856727B (en) * | 2014-03-24 | 2017-01-25 | 北京工业大学 | Multichannel real-time video splicing processing system |
| CN104008543A (en) * | 2014-05-12 | 2014-08-27 | 河海大学 | Image fusion quality evaluation method |
-
2016
- 2016-11-16 CN CN201611034166.0A patent/CN106713741B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102169576A (en) * | 2011-04-02 | 2011-08-31 | 北京理工大学 | Quantified evaluation method of image mosaic algorithms |
| CN103780870A (en) * | 2012-10-17 | 2014-05-07 | 杭州海康威视数字技术股份有限公司 | Video image quality diagnosis system and method thereof |
| CN103413298A (en) * | 2013-07-17 | 2013-11-27 | 宁波大学 | Three-dimensional image objective evaluation method based on visual characteristics |
| CN104243973A (en) * | 2014-08-28 | 2014-12-24 | 北京邮电大学 | Video perceived quality non-reference objective evaluation method based on areas of interest |
| CN105447865A (en) * | 2015-11-23 | 2016-03-30 | 深圳进化动力数码科技有限公司 | Method and device for evaluating static splicing quality of panoramic splicing algorithm |
| CN105915892A (en) * | 2016-05-06 | 2016-08-31 | 乐视控股（北京）有限公司 | Panoramic video quality determination method and system |
Non-Patent Citations (1)
| Title |
|---|
| 视频监控中图像预处理技术研究;岳建坡;《中国优秀硕士学位论文全文数据库信息科技辑》;20140615(第06期);第23-25页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106713741A (en) | 2017-05-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11006104B2 (en) | Collaborative sighting | |
| JP5739584B2 (en) | 3D image synthesizing apparatus and method for visualizing vehicle periphery | |
| CN107169924B (en) | Method and system for establishing three-dimensional panoramic image | |
| CN110009561B (en) | Method and system for mapping surveillance video target to three-dimensional geographic scene model | |
| CN108694741B (en) | Three-dimensional reconstruction method and device | |
| EP1946567A2 (en) | Device for generating three dimensional surface models of moving objects | |
| CN110827392A (en) | Monocular image three-dimensional reconstruction method, system and device with good scene usability | |
| JP2016194895A (en) | Method, device, and system for creating indoor two-dimentional (2d) floor plan | |
| Possegger et al. | Unsupervised calibration of camera networks and virtual PTZ cameras | |
| CN109448105B (en) | Three-dimensional human body skeleton generation method and system based on multi-depth image sensor | |
| CN106780613B (en) | Visualization method and system for multi-camera calibration result | |
| CN111465937A (en) | Face detection and recognition method using light field camera system | |
| CN106713741B (en) | Panoramic video quality diagnosis method and device | |
| Huang et al. | Image registration among UAV image sequence and Google satellite image under quality mismatch | |
| WO2020196520A1 (en) | Method, system and computer readable media for object detection coverage estimation | |
| CN106791800B (en) | The quality diagnosis method and device of panoramic video | |
| CN107146195B (en) | Spherical image splicing method and device | |
| KR101673144B1 (en) | Stereoscopic image registration method based on a partial linear method | |
| Imre et al. | Calibration of nodal and free-moving cameras in dynamic scenes for post-production | |
| Aliakbarpour et al. | Multi-sensor 3D volumetric reconstruction using CUDA | |
| US10198617B2 (en) | Image-acquisition apparatus | |
| KR101718309B1 (en) | The method of auto stitching and panoramic image genertation using color histogram | |
| KR20160039447A (en) | Spatial analysis system using stereo camera. | |
| US9992398B2 (en) | Optical sighting | |
| JP2013200840A (en) | Video processing device, video processing method, video processing program, and video display device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |