CN113873099A - Power transmission channel video image stabilization method, equipment and medium - Google Patents
Power transmission channel video image stabilization method, equipment and medium Download PDFInfo
- Publication number
- CN113873099A CN113873099A CN202110993062.7A CN202110993062A CN113873099A CN 113873099 A CN113873099 A CN 113873099A CN 202110993062 A CN202110993062 A CN 202110993062A CN 113873099 A CN113873099 A CN 113873099A
- Authority
- CN
- China
- Prior art keywords
- motion
- power transmission
- transmission channel
- grid intersection
- motion vector
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 121
- 238000000034 method Methods 0.000 title claims abstract description 46
- 230000006641 stabilisation Effects 0.000 title claims abstract description 45
- 238000011105 stabilization Methods 0.000 title claims abstract description 45
- 230000033001 locomotion Effects 0.000 claims abstract description 261
- 239000013598 vector Substances 0.000 claims abstract description 133
- 238000004364 calculation method Methods 0.000 claims description 19
- 238000009499 grossing Methods 0.000 claims description 18
- 238000005457 optimization Methods 0.000 claims description 10
- 230000002159 abnormal effect Effects 0.000 claims description 6
- 230000004931 aggregating effect Effects 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 5
- 239000000126 substance Substances 0.000 claims description 5
- 230000006870 function Effects 0.000 claims description 4
- 238000005070 sampling Methods 0.000 claims description 4
- 238000012216 screening Methods 0.000 claims description 4
- 230000002123 temporal effect Effects 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 11
- 238000012545 processing Methods 0.000 abstract description 6
- 238000010586 diagram Methods 0.000 description 12
- 230000008859 change Effects 0.000 description 4
- 230000001788 irregular Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000000087 stabilizing effect Effects 0.000 description 3
- 238000006073 displacement reaction Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 206010034719 Personality change Diseases 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000009191 jumping Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/682—Vibration or motion blur correction
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/682—Vibration or motion blur correction
- H04N23/683—Vibration or motion blur correction performed by a processor, e.g. controlling the readout of an image memory
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/81—Camera processing pipelines; Components thereof for suppressing or minimising disturbance in the image signal generation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
Abstract
The embodiment of the application discloses a power transmission channel video image stabilization method, equipment and medium. Carrying out feature point matching on adjacent frame video images of a power transmission channel shot by a high-voltage power tower camera, and tracking feature points to determine motion vectors of the feature points, wherein a plurality of feature points are uniformly distributed in the power transmission channel video images; determining a corresponding geometric area by taking the feature point as a center; obtaining a motion vector corresponding to a grid intersection point in a geometric area according to the motion vector of the feature point; wherein any grid intersection receives one or more motion vectors; and calculating to obtain a smooth path corresponding to each grid intersection point according to the corresponding motion vector of each grid intersection point in the different frames of power transmission channel images, obtaining a motion track of the video image of the power transmission channel according to the smooth path corresponding to each grid intersection point, and realizing video image stabilization of the power transmission channel according to the motion track. By the method, the image stabilization processing effect is improved.
Description
Technical Field
The present application relates to the field of image processing technologies, and in particular, to a method, an apparatus, and a medium for video image stabilization for a power transmission channel.
Background
When the camera is used, the camera body is often shaken due to extreme weather, accidental collision and the like, so that the imaging effect is influenced, the problems of instability and jitter of the recorded video are caused, and particularly when a specific target needs to be tracked in a scene, a user cannot accurately position or estimate the position of the moving target, so that the position of the target in the video is unstable, and the subjective effect of the video is not ideal.
In the case that the physical anti-shake device is high in cost and vulnerable, in order to solve the problem, a video image stabilization algorithm needs to be designed to recognize the meaningless motion and try to realize the stable state of the video scene position by a compensation mode.
In a traditional image stabilization system based on a characteristic track, because a camera is rapidly translated to cause motion blur, a long track is difficult to obtain in a video with irregular large-amplitude jitter, the characteristic track is generally sparse in space and uneven in distribution, and the image stabilization processing effect is poor at last when the characteristic track is ended or started in any frame of the video.
Disclosure of Invention
The embodiment of the application provides a power transmission channel video image stabilization method, equipment and medium, which are used for solving the following technical problems: the traditional image stabilizing system based on the characteristic track has poor image stabilizing processing effect.
The embodiment of the application adopts the following technical scheme:
the embodiment of the application provides a power transmission channel video image stabilizing method. The method comprises the steps of matching characteristic points of adjacent frame video images of a power transmission channel shot by a high-voltage power tower camera, and tracking the characteristic points to determine motion vectors of the characteristic points, wherein a plurality of characteristic points are uniformly distributed in the power transmission channel video images; determining a corresponding geometric area by taking the feature point as a center; obtaining a motion vector corresponding to a grid intersection point in a geometric area according to the motion vector of the feature point; wherein any grid intersection receives one or more motion vectors; and calculating to obtain a smooth path corresponding to each grid intersection point according to the corresponding motion vector of each grid intersection point in the different frames of power transmission channel images, obtaining a motion track of the video image of the power transmission channel according to the smooth path corresponding to each grid intersection point, and realizing video image stabilization of the power transmission channel according to the motion track.
According to the embodiment of the application, the geometric area corresponding to the feature point is determined, and the motion vector of the feature point is sent to the grid intersection point where the geometric area is overlapped with the preset regular grid. So that each grid intersection point can receive the motion vectors of a plurality of surrounding feature points, thereby determining the motion vector of the accurately obtained grid intersection point. In addition, the grid intersection points are only subjected to smooth path calculation, and compared with a dense pixel path smoothing method, the method and the device for optimizing the grid intersection points can reduce the calculation amount while ensuring the accuracy, and further improve the path optimization speed.
In an implementation manner of the present application, calculating a smooth path of each grid intersection in a motion field according to a corresponding motion vector of each grid intersection in a video image of a power transmission channel of different frames, and obtaining a motion trajectory of the video image of the power transmission channel according to the smooth path of each grid intersection, specifically includes: according to the time sequence of shooting the video images of the power transmission channels, connecting the corresponding motion vectors of any grid intersection point in different power transmission channel images to obtain a local motion path corresponding to any grid intersection point, and determining local motion paths of other grid intersection points; and respectively carrying out smooth calculation on the local motion path of any grid intersection and the local motion paths of the rest grid intersections, and aggregating the calculated results into a motion track of the power transmission channel video image.
In an implementation manner of the present application, performing smoothing calculation on a local motion path of any grid intersection and local motion paths of other grid intersections respectively includes: aggregating the motion paths corresponding to all grid intersections in the power transmission channel video image of the current frame into a motion path of the whole frame image; according to the formula
Performing smooth calculation on the local motion path; wherein, C represents the motion path of the whole frame image, C (t) is the motion path of the camera in the t frame, P represents the optimal path of the camera, and P (t) represents the optimal path of the camera in the t frame; | P (t) -C (t) | non-woven hair2Representing the similarity of the optimal path and the original path; lambda [ alpha ]tIs a balance parameter; lambda [ alpha ]tThe larger the value of P, the smoother it is when λtWhen 0 is taken, P is equal to the original path C; r represents the frame number within the time smoothing window, P (r) represents the smoothing path P at the location of the r-th frame, ΩtRepresenting the time-smoothed radius, ωt,rRepresenting the gaussian weight, O (p (t)) represents the optimization objective function.
In an implementation manner of the present application, feature point matching is performed on adjacent frame video images of a power transmission channel shot by a high voltage tower camera, and the feature points are tracked to determine a motion vector of the feature points, which specifically includes: according to the formula
Calculating coordinates corresponding to the feature points between the t frame and the t-1 frame, and determining motion vectors of the feature points; wherein, p is the coordinate of the characteristic point in the t frame image;
coordinates of the characteristic points in the t-1 frame image are obtained; v. ofpMotion vectors corresponding to the feature points; t is a positive integer.
In one implementation of the present application, before calculating a smooth path of each grid intersection in the motion field, the method further includes: filtering the motion vectors received by the grid intersections through a first median filter to obtain a sparse motion field; the first median filter covers the area range corresponding to all the grid intersections; denoising the sparse motion field through a second median filter to obtain a smooth sparse motion field; the second median filter covers the area range corresponding to the geometric area; in the smoothed sparse motion field, a smoothed path of each mesh intersection is calculated.
In an implementation manner of the present application, before performing feature point matching, the method further includes: dividing a power transmission channel video image into a plurality of small areas with preset sizes; setting different local thresholds for each small area according to the image texture characteristics corresponding to the small areas respectively; and screening a plurality of characteristic points in each small area according to the local threshold corresponding to each small area so as to enable the plurality of characteristic points to be uniformly distributed in the power transmission channel video image.
According to the embodiment of the application, the video image of the power transmission channel is divided into the plurality of small areas, and the image can be divided according to different texture characteristics of the image. Secondly, the texture characteristics of the image corresponding to each region are different, different threshold values are set for each region, and corresponding feature points are determined in each small region according to the different threshold values, so that the determined feature points are uniformly distributed in the whole image.
In one implementation manner of the present application, before determining the motion vector of the feature point, the method further includes: dividing the power transmission channel image into a plurality of sub-images; removing abnormal values in the sub-images by a random sampling consistency algorithm; wherein the abnormal value at least comprises one or more of error matching and motion deviation caused by the dynamic object.
In one implementation manner of the present application, before determining the motion vector of the feature point, the method further includes: establishing a global motion vector field according to all grid intersections in the current power transmission channel image;
according to the formula
Calculating the corresponding local motion of the grid intersection point which does not receive the motion vector of the feature point; wherein the content of the first and second substances,
is a local motion vector; p is a radical of1The coordinates of key points of the t frame images;
the coordinates of the key points of the t-1 frame image; ftA global motion vector field established for all intersections;
according to the formula
When the grid cross points do not receive the motion vectors of the corresponding feature points, estimating the motion vectors corresponding to the feature points; wherein v isp1Local motion vectors of the feature points corresponding to the grid intersections; vtIs FtProduct of the global motion of all points.
The embodiment of the application provides a transmission channel video image stabilization equipment, includes: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to: carrying out feature point matching on adjacent frame video images of a power transmission channel shot by a high-voltage power tower camera, and tracking feature points to determine motion vectors of the feature points, wherein a plurality of feature points are uniformly distributed in the power transmission channel video images; determining a corresponding geometric area by taking the feature point as a center; obtaining a motion vector corresponding to a grid intersection point in a geometric area according to the motion vector of the feature point; wherein any grid intersection receives one or more motion vectors; and calculating to obtain a smooth path corresponding to each grid intersection point according to the corresponding motion vector of each grid intersection point in the different frames of power transmission channel images, obtaining a motion track of the video image of the power transmission channel according to the smooth path corresponding to each grid intersection point, and realizing video image stabilization of the power transmission channel according to the motion track.
A non-volatile computer storage medium provided in an embodiment of the present application stores computer-executable instructions, and the computer-executable instructions are configured to: carrying out feature point matching on adjacent frame video images of a power transmission channel shot by a high-voltage power tower camera, and tracking feature points to determine motion vectors of the feature points, wherein a plurality of feature points are uniformly distributed in the power transmission channel video images; determining a corresponding geometric area by taking the feature point as a center; obtaining a motion vector corresponding to a grid intersection point in a geometric area according to the motion vector of the feature point; wherein any grid intersection receives one or more motion vectors; and calculating to obtain a smooth path corresponding to each grid intersection point according to the corresponding motion vector of each grid intersection point in the different frames of power transmission channel images, obtaining a motion track of the video image of the power transmission channel according to the smooth path corresponding to each grid intersection point, and realizing video image stabilization of the power transmission channel according to the motion track.
The embodiment of the application adopts at least one technical scheme which can achieve the following beneficial effects: according to the embodiment of the application, the geometric area corresponding to the feature point is determined, and the motion vector of the feature point is sent to the grid intersection point where the geometric area is overlapped with the preset regular grid. So that each grid intersection point can receive the motion vectors of a plurality of surrounding feature points, thereby determining the motion vector of the accurately obtained grid intersection point. In addition, the grid intersection points are only subjected to smooth path calculation, and compared with a dense pixel path smoothing method, the method and the device for optimizing the grid intersection points can reduce the calculation amount while ensuring the accuracy, and further improve the path optimization speed.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without any creative effort. In the drawings:
fig. 1 is a flowchart of a power transmission channel video image stabilization method according to an embodiment of the present disclosure;
fig. 2 is a schematic diagram of a motion trajectory of a power transmission channel in a horizontal direction according to an embodiment of the present disclosure;
fig. 3 is a schematic diagram of a smooth trajectory of a power transmission channel video according to an embodiment of the present disclosure;
fig. 4 is a schematic diagram of sending motion vectors according to an embodiment of the present application;
fig. 5 is a schematic diagram of filtering performed by a first median filter according to an embodiment of the present application;
fig. 6 is a schematic diagram of denoising with a second median filter according to an embodiment of the present disclosure;
fig. 7 is a flowchart of power transmission channel video image stabilization motion estimation and path smoothing according to an embodiment of the present disclosure;
fig. 8 is a schematic structural diagram of a power transmission channel video image stabilization device according to an embodiment of the present application.
Detailed Description
The embodiment of the application provides a power transmission channel video image stabilization method, equipment and medium.
In order to make those skilled in the art better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any inventive step based on the embodiments of the present disclosure, shall fall within the scope of protection of the present application.
When the high-voltage electric tower camera is used, the camera body is frequently shaken due to reasons such as extreme weather and accidental collision, so that the imaging effect is influenced, the instability and jumping problems of the recorded video are caused, and particularly when a specific target needs to be tracked in a scene, a user cannot accurately position or estimate the position of the moving target, so that the position of the target in the video is unstable, and the subjective effect of the video is not ideal.
In the case of a high cost and a vulnerable physical anti-shake device, in order to solve the problem, a video image stabilization algorithm needs to be designed to recognize such meaningless motion and try to stabilize the position of the video scene by means of compensation.
The traditional image stabilization system based on the characteristic track is mostly applied to complex irregular jitter of a handheld camera, but due to rapid camera translation or motion blur, a long track is difficult to obtain in a video with irregular large-amplitude jitter, the characteristic track is generally sparse in space and uneven in distribution, and can end or start at any frame of the video, so that the problem of poor image stabilization processing effect of the characteristic track on the camera is finally caused.
In order to solve the above problem, embodiments of the present application provide a method, an apparatus, and a medium for power transmission channel video image stabilization. And determining a geometric area corresponding to the feature point, and sending the motion vector of the feature point to a grid intersection point where the geometric area is overlapped with the preset regular grid. So that each grid intersection point can receive the motion vectors of a plurality of surrounding feature points, thereby determining the motion vector of the accurately obtained grid intersection point. In addition, the grid intersection points are only subjected to smooth path calculation, and compared with a dense pixel path smoothing method, the method and the device for optimizing the grid intersection points can reduce the calculation amount while ensuring the accuracy, and further improve the path optimization speed.
The technical solutions proposed in the embodiments of the present application are described in detail below with reference to the accompanying drawings.
Fig. 1 is a flowchart of a power transmission channel video image stabilization method according to an embodiment of the present application. As shown in fig. 1, the power transmission channel video image stabilization method includes the following steps:
s101, the power transmission channel video image stabilization equipment performs feature point matching on adjacent frame video images of the power transmission channel shot by the high-voltage tower camera, and tracks the feature points to determine motion vectors of the feature points.
In one embodiment of the application, power transmission passes through a video image stabilization device, and a power transmission channel video image is divided into a plurality of small areas with preset sizes. And setting different local threshold values for each small region according to the image texture characteristics corresponding to the small regions respectively. And screening a plurality of characteristic points in each small area according to the local threshold corresponding to each small area, so that the plurality of characteristic points are uniformly distributed in the power transmission channel video image.
Specifically, in the power channel video image, there may be regions with different texture characteristics. For example, the texture characteristics of sky regions may be relatively similar, while the texture characteristics of ground regions may be more different. Therefore, the corresponding feature points need to be determined according to different texture characteristics. The image is divided into a plurality of small areas, for example, a sky area and a ground area. The texture characteristics of the sky area are relatively approximate, so that a lower texture characteristic threshold value can be set, and a larger number of characteristic points can be selected. Because the difference of the texture characteristics of the ground area is large, a high threshold value of the texture characteristics can be set to reduce the number of the selected feature points. Therefore, the number of the characteristic points of the sky area and the ground area is uniformly distributed in the whole power transmission channel video image.
According to the embodiment of the application, the accuracy of the calculation result can be improved by extracting the feature points in a balanced manner. In a power transmission scene, the space area occupancy ratio and the depth of field change in a picture shot by the camera are large. The feature points proposed by the traditional method are severely unevenly distributed, and the sky area often has no effective key point to calculate the stable image. In the embodiment of the application, the final image stabilization result is improved by 2.7% in stability compared with the traditional method through balanced feature distribution.
In one embodiment of the present application, the formula is based on
And calculating coordinates corresponding to the feature points between the t frame and the t-1 frame, and determining the motion vectors of the feature points. Wherein, p is the coordinate of the characteristic point in the t frame image;
coordinates of the characteristic points in the t-1 frame image are obtained; v. ofpIs corresponding to the characteristic pointThe motion vector of (2); t is a positive integer.
Specifically, feature points between two adjacent frames are tracked and matched. Therein, the FAST feature can be used and tracked by the KLT optical flow algorithm. And the obtained position change of the same characteristic point in the two adjacent frames of images is the motion vector of the characteristic point.
In the embodiment of the present application, a coordinate system is set in advance for a power transmission channel video image, and coordinates of each feature point are determined from the coordinate system, so that a motion vector of each feature point is obtained. For example, the embodiment of the present application may use an intersection of a left edge and an upper edge of an image as an origin of a coordinate system, a direction extending rightward as an abscissa axis, and a direction extending downward as an ordinate axis.
Fig. 2 is a schematic diagram of a motion trajectory of a power transmission channel in a horizontal direction according to an embodiment of the present application. After the motion direction between the image frames is detected, matching the adjacent frame images to obtain a transformation matrix between the two images, and extracting corresponding horizontal displacement, vertical displacement and rotation angle from the matrix to obtain the motion trail of the video image. Fig. 3 is a schematic diagram of a smooth trajectory of a power transmission channel video according to an embodiment of the present application. Planning a virtual camera smooth path on the obtained motion track of the video image, wherein the smooth path can be a stable camera attitude change during shooting and shows that a shot picture is smooth and has no scene change of severe jitter, and a curve is generally smoothed by using methods such as filtering, fitting or optimization.
And S102, determining a corresponding geometric area by using the characteristic point as a center through the power transmission channel video image stabilization equipment.
In an embodiment of the application, a uniform regular grid is set for the power transmission channel video image, and then a corresponding geometric area is set for each feature point. And determining the area of the geometrical area coinciding with the regular grid so as to determine the motion vector of the grid intersection point in the coinciding area part through the motion vector of the characteristic point.
Fig. 4 is a transmission motion vector provided in an embodiment of the present applicationQuantitative schematic. As shown in fig. 4, the motion vector v at the feature point ppCan be calculated as
The grid intersection points near the feature point p should be related to νpWith similar motion. Thus, a circle centered at p is defined, which covers 3 × 3 meshes.
It should be noted that, in the embodiment of the present application, the geometric area corresponding to the feature point is preferably set to be a circle covering 3 × 3 grids. In application, the shape and size of the geometric area may be set according to actual conditions, which is not limited in the present application.
S103, obtaining a motion vector corresponding to the grid intersection point in the geometric area according to the motion vector of the feature point.
In one embodiment of the present application, a power transmission channel image is divided into a plurality of sub-images before determining a motion vector of a feature point. And eliminating abnormal values in the sub-images by a random sampling consistency algorithm. Wherein the abnormal value at least comprises one or more of error matching and motion deviation caused by the dynamic object.
Specifically, the image may be divided into 4 × 4 sub-images, and a random sampling consensus algorithm is used to perform local homography fitting to remove abnormal values. Large motion deviations due to mismatch or dynamic objects can be filtered out and only changes due to depth changes or translational jitter are calculated.
In one embodiment of the present application, a preset regular grid of 16 × 16 may be used for the power channel video image of each frame. The geometric area corresponding to the feature point comprises a plurality of grid cross points, and the grid cross points near the feature point have similar movement with the feature point. Therefore, the motion vector of the feature point is sent to the mesh intersection in the geometric region. And determining the motion vector of each grid intersection point through the motion vectors received by the grid intersection points, and further obtaining the motion field of the power transmission channel video image according to the motion vector of each grid intersection point.
As shown in the diagram b in fig. 4, the circle center of the circular area formed by the dashed lines is the position of the feature point, and the direction indicated by the arrow is the motion vector of the feature point. The grid intersections within the circular area are determined and the motion vectors are sent to the grid intersections within the circular area. As can be seen from the lower image in fig. b, each grid intersection point in the circular area receives the motion vector of the feature point, and each grid intersection point has the same arrow direction as the feature point.
In one embodiment of the present application, the motion vectors received at the grid intersections are filtered by a first median filter to obtain a sparse motion field. And the first median filter covers the area range corresponding to all the grid intersections. And denoising the sparse motion field through a second median filter to obtain a smooth sparse motion field. And the second median filter covers the area range corresponding to the geometric area. In the smoothed sparse motion field, a smoothed path of each mesh intersection is calculated.
In particular, a sparse motion field is obtained by applying the first median filter to all intersections. Fig. 5 is a schematic diagram of filtering performed by a first median filter according to an embodiment of the present disclosure, as shown in fig. 5. One grid intersection point can obtain motion vectors of a plurality of feature points, and partial differences may exist among the motion vectors. A sparse motion field is obtained by a first median filter. This sparsity makes the motion estimation calculation of spatial variation lightweight.
In particular, noise may be present on the motion field due to feature matching errors, object dynamics, and the like. Fig. 6 is a schematic diagram of denoising by using a second median filter according to an embodiment of the present disclosure. As shown in the left image of fig. 6, white arrows in the figure represent noise caused by a matching error or other reasons. As shown in the right image of fig. 6, it is denoised by the second median filter to produce a spatially smooth sparse motion field. Wherein each second median filter covers 3 × 3 cells.
In one embodiment of the present application, a global motion vector field is established based on all grid intersections in the current power transmission channel image before determining the motion vectors of the feature points.
According to the formula
And calculating the corresponding local motion of the grid intersection point which does not receive the motion vector of the feature point. Wherein the content of the first and second substances,
is a local motion vector; p is a radical of1The coordinates of key points of the t frame images;
the coordinates of the key points of the t-1 frame image; ftA global motion vector field is established for all intersections.
According to the formula
So as to estimate the motion vector corresponding to the grid intersection point when the motion vector of the corresponding feature point is not received by the grid intersection point. Wherein v isp1Local motion vectors of the feature points corresponding to the grid intersections; vtIs FtProduct of the global motion of all points.
Specifically, there arises a problem that the individual mesh intersections do not have corresponding feature points when images are matched, that is, the motion vector having no feature point transmits a motion vector to the mesh intersection. Therefore, the motion vectors cannot be determined for a small number of grid intersections. In order to ensure that the motion fields involved in the computation of a grid intersection point are uniformly distributed, it is necessary to estimate the motion fields of the grid intersection point. In this case, the local motion field can be derived from the globally established motion vector field, and the motion vectors for the grid intersections are estimated from the local motion field. Thereby ensuring that all grid intersections have corresponding motion fields.
S104, the power transmission channel video image stabilization equipment calculates to obtain a smooth path corresponding to each grid intersection point according to the corresponding motion vector of each grid intersection point in different frames of power transmission channel images, obtains a motion track of the power transmission channel video image according to the smooth path corresponding to each grid intersection point, and realizes power transmission channel video image stabilization according to the motion track.
In an embodiment of the application, according to the time sequence of shooting the power transmission channel video images, any grid intersection point is connected with the corresponding motion vector in different power transmission channel images, a local motion path corresponding to any grid intersection point is obtained, and local motion paths of other grid intersection points are determined. And respectively carrying out smooth calculation on the local motion path of any grid intersection and the local motion paths of the rest grid intersections, and aggregating the calculated results into a motion track of the power transmission channel video image.
Specifically, a single mesh intersection point motion path is first used as a local camera motion path, and then all mesh intersection point motion paths of the current frame are aggregated into a camera motion path of the whole frame image. In order to realize the image stabilization effect, the paths are smoothed by balancing the smoothness of the paths and the similarity with the original paths to obtain the optimal paths, and finally, smoothing calculation is respectively carried out on the motion paths of each grid intersection point.
In an embodiment of the application, motion paths corresponding to all grid intersections in a power transmission channel video image of a current frame are aggregated into a camera motion path of the whole frame image.
According to the formula
Performing smooth calculation on the local motion path;
wherein, C represents the camera motion path of the whole frame image, C (t) is the camera motion path of the t frame, P represents the optimal path of the camera, and P (t) represents the optimal path of the camera when the t frame;||P(t)-C(t)||2representing the similarity of the optimal path and the original path; lambda [ alpha ]tIs a balance parameter; lambda [ alpha ]tThe larger the value of P, the smoother it is when λtWhen 0 is taken, P is equal to the original path C; r represents the frame number within the time smoothing window, P (r) represents the smoothing path P at the location of the r-th frame, ΩtRepresenting the time-smoothed radius, ωt,rRepresenting the gaussian weight, O (p (t)) represents the optimization objective function.
Specifically, a single cross point motion path is used as the local camera motion path, and all cross point motion paths of the current frame can be aggregated into a camera motion path C of the entire frame image. In order to achieve the image stabilization effect, the path C is smoothed and the optimal path P is obtained by balancing the path smoothness and the similarity with the original path.
It should be noted that, in the embodiment of the present application, regarding the minimization problem of the O (p (t)) function, a linear solver based on a jacobian iteration method may be used for solving the problem.
Since the motion estimation has strong spatial correlation, the embodiment of the application does not need additional spatial constraint in the smoothing process of the motion. In the embodiment of the application, in the process of respectively smoothing the motion path of each grid intersection, the motion path of each grid intersection is smoothed through multi-thread parallel computation, so that the computation time is further shortened. The traditional algorithm has low adjustment flexibility of balance parameters and is difficult to satisfy the smoothing effect in different application scenes. Experiments show that the path optimization speed of the independent grid intersection point path optimization parallelization method is improved by 7-10 times compared with the path optimization speed of the traditional method.
Fig. 7 is a flowchart of power transmission channel video image stabilization motion estimation and path smoothing according to an embodiment of the present application. As shown in fig. 7, the video image-stabilized motion estimation and path smoothing includes the following steps:
in one embodiment of the present application, feature points are matched, and motion vectors of the feature points are determined by optical flow tracking.
Specifically, a power transmission channel video image is divided into a plurality of small areas of a preset size. And setting different local threshold values for each small region according to the image texture characteristics corresponding to the small regions respectively. And screening a plurality of characteristic points in each small area according to the local threshold corresponding to each small area, so that the plurality of characteristic points are uniformly distributed in the power transmission channel video image. And tracking and matching the feature points between two adjacent frames. Therein, the FAST feature can be used and tracked by the KLT optical flow algorithm. The obtained position change of the same feature point in two adjacent frames of images is the motion vector of the feature point
In one embodiment of the present application, the motion vectors of the feature points are transmitted to the intersections of the preset mesh.
Specifically, a 16 × 16 grid may be used for the power channel video image of each frame. The geometric area corresponding to the feature point comprises a plurality of grid cross points, and the grid cross points near the feature point have similar movement with the feature point. Therefore, the motion vector of the feature point is sent to the mesh intersection in the geometric region. And determining the motion vector of each grid intersection point through the motion vectors received by the grid intersection points, and further obtaining the motion field of the power transmission channel video image according to the motion vector of each grid intersection point.
In one embodiment of the application, the motion vectors of the grid intersections are filtered by a termination filter.
Specifically, the motion vectors received by the grid intersections are filtered through a first median filter, so that a sparse motion field is obtained. And the first median filter covers the area range corresponding to all the grid intersections. And denoising the sparse motion field through a second median filter to obtain a smooth sparse motion field. And the second median filter covers the area range corresponding to the geometric area.
In one embodiment of the present application, the filtered grid intersections are formed into a smooth sparse motion field.
In one embodiment of the present application, the smooth path is solved independently for each grid intersection to achieve power channel video stabilization.
According to the time sequence of shooting the power transmission channel video images, connecting the corresponding motion vectors of any grid intersection point in different power transmission channel images to obtain a local motion path corresponding to any grid intersection point, and determining local motion paths of other grid intersection points. And respectively carrying out smooth calculation on the local motion path of any grid intersection and the local motion paths of the rest grid intersections, and aggregating the calculated results into a motion track of the power transmission channel video image.
Fig. 8 is a schematic structural diagram of a power transmission channel video image stabilization device according to an embodiment of the present application. As shown in fig. 8, the power transmission channel video image stabilization apparatus includes:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to:
carrying out feature point matching on adjacent frame video images of a power transmission channel shot by a high-voltage power tower camera, and tracking the feature points to determine motion vectors of the feature points, wherein a plurality of the feature points are uniformly distributed in the power transmission channel video images;
determining a corresponding geometric area by taking the characteristic point as a center;
obtaining a motion vector corresponding to the grid intersection point in the geometric area according to the motion vector of the feature point; wherein any grid intersection receives one or more of the motion vectors;
calculating to obtain a smooth path corresponding to each grid intersection point according to the corresponding motion vector of each grid intersection point in different frames of power transmission channel images, obtaining a motion track of the power transmission channel video images according to the smooth path corresponding to each grid intersection point, and realizing power transmission channel video image stabilization according to the motion track.
Embodiments of the present application also include a non-volatile computer storage medium storing computer-executable instructions configured to:
carrying out feature point matching on adjacent frame video images of a power transmission channel shot by a high-voltage power tower camera, and tracking the feature points to determine motion vectors of the feature points, wherein a plurality of the feature points are uniformly distributed in the power transmission channel video images;
determining a corresponding geometric area by taking the characteristic point as a center;
obtaining a motion vector corresponding to the grid intersection point in the geometric area according to the motion vector of the feature point; wherein any grid intersection receives one or more of the motion vectors;
calculating to obtain a smooth path corresponding to each grid intersection point according to the corresponding motion vector of each grid intersection point in different frames of power transmission channel images, obtaining a motion track of the power transmission channel video images according to the smooth path corresponding to each grid intersection point, and realizing power transmission channel video image stabilization according to the motion track.
The embodiments in the present application are described in a progressive manner, and the same and similar parts among the embodiments can be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the embodiments of the apparatus, the device, and the nonvolatile computer storage medium, since they are substantially similar to the embodiments of the method, the description is simple, and for the relevant points, reference may be made to the partial description of the embodiments of the method.
The foregoing description of specific embodiments of the present application has been presented. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art to which the embodiments of the present application pertain. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. A power transmission channel video image stabilization method is characterized by comprising the following steps:
carrying out feature point matching on adjacent frame video images of a power transmission channel shot by a high-voltage power tower camera, and tracking the feature points to determine motion vectors of the feature points, wherein a plurality of the feature points are uniformly distributed in the power transmission channel video images;
determining a corresponding geometric area by taking the characteristic point as a center;
obtaining a motion vector corresponding to the grid intersection point in the geometric area according to the motion vector of the feature point; wherein any grid intersection receives one or more of the motion vectors;
calculating to obtain a smooth path corresponding to each grid intersection point according to the corresponding motion vector of each grid intersection point in different frames of power transmission channel images, obtaining a motion track of the power transmission channel video images according to the smooth path corresponding to each grid intersection point, and realizing power transmission channel video image stabilization according to the motion track.
2. The method according to claim 1, wherein a smooth path corresponding to each grid intersection point is obtained by calculation according to a corresponding motion vector of each grid intersection point in a video image of a different frame power transmission channel, and a motion trajectory of the video image of the power transmission channel is obtained according to the smooth path corresponding to each grid intersection point, specifically comprising:
according to the time sequence of shooting the power transmission channel video images, connecting corresponding motion vectors of any grid intersection point in different power transmission channel images to obtain a local motion path corresponding to any grid intersection point, and determining local motion paths of other grid intersection points;
and respectively carrying out smooth calculation on the local motion path of any grid intersection and the local motion paths of the rest grid intersections, and aggregating the calculated results into the motion track of the power transmission channel video image.
3. The power transmission channel video image stabilization method according to claim 2, wherein the performing of the smoothing calculation on the local motion path of any one grid intersection and the local motion paths of the other grid intersections respectively specifically includes:
aggregating the motion paths corresponding to all grid intersections in the power transmission channel video image of the current frame into a motion path of the whole frame image;
according to the formula
Performing smooth calculation on the local motion path;
wherein, C represents the motion path of the whole frame image, C (t) is the motion path of the camera in the t frame, P represents the optimal path of the camera, and P (t) represents the optimal path of the camera in the t frame; | P (t) -C (t) | non-woven hair2Representing the similarity of the optimal path and the original path;
represents temporal smoothness; lambda [ alpha ]tIs a balance parameter; lambda [ alpha ]tThe larger the value of P, the smoother it is when λtWhen 0 is taken, P is equal to the original path C; r represents the frame number within the time smoothing window, P (r) represents the smoothing path P at the location of the r-th frame, ΩtRepresenting the time-smoothed radius, ωt,Representing the gaussian weight, O (p (t)) represents the optimization objective function.
4. The method according to claim 1, wherein the matching of feature points is performed on adjacent frame video images of the power transmission channel shot by the high-voltage tower camera, and the feature points are tracked to determine the motion vectors of the feature points, specifically comprising:
according to the formula
Calculating coordinates corresponding to the feature points between the t frame and the t-1 frame, and determining motion vectors of the feature points;
wherein p is the coordinate of the characteristic point in the t frame image;
the coordinates of the characteristic points in the t-1 frame image are obtained; v. ofpThe motion vector corresponding to the characteristic point is obtained; t is a positive integer.
5. The method according to claim 1, wherein before the calculating a smooth path of each grid intersection in the motion field, the method further comprises:
filtering the motion vectors received by the grid intersections through a first median filter to obtain a sparse motion field; wherein, the first median filter covers the area range corresponding to all the grid intersections;
denoising the sparse motion field through a second median filter to obtain a smooth sparse motion field; wherein the second median filter covers an area range corresponding to the geometric area;
and in the smooth sparse motion field, calculating a smooth path of each grid intersection.
6. The power transmission channel video image stabilization method according to claim 1, wherein before the feature point matching, the method further comprises:
dividing the power transmission channel video image into a plurality of small areas with preset sizes;
setting different local thresholds for the small areas according to the image texture features respectively corresponding to the small areas;
and screening a plurality of characteristic points in each small area according to the local threshold corresponding to each small area, so that the plurality of characteristic points are uniformly distributed in the power transmission channel video image.
7. The method according to claim 1, wherein before determining the motion vector of the feature point, the method further comprises:
dividing the power transmission channel image into a plurality of sub-images;
removing abnormal values in the sub-images through a random sampling consistency algorithm; wherein the outliers include at least one or more of a mismatch, a motion deviation caused by a dynamic object.
8. The method according to claim 1, wherein before determining the motion vector of the feature point, the method further comprises:
establishing a global motion vector field according to all grid intersections in the current power transmission channel image;
according to the formula
Calculating a local motion vector corresponding to a grid intersection point which does not receive the motion vector of the feature point; wherein the content of the first and second substances,
is a local motion vector; p is a radical of1The coordinates of key points of the t frame images;
frame imageKey point coordinates of the image; ftA global motion vector field established for all intersections;
according to the formula
When the grid intersection point does not receive the motion vector of the corresponding characteristic point, estimating the motion vector corresponding to the grid intersection point; wherein v isp1Local motion vectors of the feature points corresponding to the grid intersection points; vtIs FtProduct of the global motion of all points.
9. A power transmission channel video image stabilization apparatus comprising:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to:
carrying out feature point matching on adjacent frame video images of a power transmission channel shot by a high-voltage power tower camera, and tracking the feature points to determine motion vectors of the feature points, wherein a plurality of the feature points are uniformly distributed in the power transmission channel video images;
determining a corresponding geometric area by taking the characteristic point as a center;
obtaining a motion vector corresponding to the grid intersection point in the geometric area according to the motion vector of the feature point; wherein any grid intersection receives one or more of the motion vectors;
calculating to obtain a smooth path corresponding to each grid intersection point according to the corresponding motion vector of each grid intersection point in different frames of power transmission channel images, obtaining a motion track of the power transmission channel video images according to the smooth path corresponding to each grid intersection point, and realizing power transmission channel video image stabilization according to the motion track.
10. A non-transitory computer storage medium storing computer-executable instructions configured to:
carrying out feature point matching on adjacent frame video images of a power transmission channel shot by a high-voltage power tower camera, and tracking the feature points to determine motion vectors of the feature points, wherein a plurality of the feature points are uniformly distributed in the power transmission channel video images;
determining a corresponding geometric area by taking the characteristic point as a center;
obtaining a motion vector corresponding to the grid intersection point in the geometric area according to the motion vector of each feature point; wherein any grid intersection receives one or more of the motion vectors;
calculating to obtain a smooth path corresponding to each grid intersection point according to the corresponding motion vector of the grid intersection point in different frames of power transmission channel images, obtaining a motion track of the power transmission channel video image according to the smooth path corresponding to each grid intersection point, and realizing power transmission channel video image stabilization according to the motion track.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110993062.7A CN113873099B (en) | 2021-08-27 | 2021-08-27 | Video image stabilizing method, device and medium for power transmission channel |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110993062.7A CN113873099B (en) | 2021-08-27 | 2021-08-27 | Video image stabilizing method, device and medium for power transmission channel |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113873099A true CN113873099A (en) | 2021-12-31 |
| CN113873099B CN113873099B (en) | 2024-04-12 |
Family
ID=78988578
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110993062.7A Active CN113873099B (en) | 2021-08-27 | 2021-08-27 | Video image stabilizing method, device and medium for power transmission channel |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113873099B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060002632A1 (en) * | 2004-06-30 | 2006-01-05 | Accuray, Inc. | Motion field generation for non-rigid image registration |
| WO2006075394A1 (en) * | 2005-01-14 | 2006-07-20 | Morhpo, Inc. | Motion vector calculation method, hand-movement correction device using the method, imaging device, and motion picture generation device |
| US8923400B1 (en) * | 2007-02-16 | 2014-12-30 | Geo Semiconductor Inc | Method and/or apparatus for multiple pass digital image stabilization |
| CN106101535A (en) * | 2016-06-21 | 2016-11-09 | 北京理工大学 | A kind of based on local and the video stabilizing method of mass motion disparity compensation |
| CN108805908A (en) * | 2018-06-08 | 2018-11-13 | 浙江大学 | A kind of real time video image stabilization based on the superposition of sequential grid stream |
| CN109361880A (en) * | 2018-11-30 | 2019-02-19 | 三星电子（中国）研发中心 | A kind of method and system showing the corresponding dynamic picture of static images or video |
| CN110753181A (en) * | 2019-09-29 | 2020-02-04 | 湖北工业大学 | Video image stabilization method based on feature tracking and grid path motion |
| CN110781903A (en) * | 2019-10-12 | 2020-02-11 | 中国地质大学（武汉） | Unmanned aerial vehicle image splicing method based on grid optimization and global similarity constraint |
| CN111819837A (en) * | 2018-03-11 | 2020-10-23 | 谷歌有限责任公司 | Static video identification |
-
2021
- 2021-08-27 CN CN202110993062.7A patent/CN113873099B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060002632A1 (en) * | 2004-06-30 | 2006-01-05 | Accuray, Inc. | Motion field generation for non-rigid image registration |
| WO2006075394A1 (en) * | 2005-01-14 | 2006-07-20 | Morhpo, Inc. | Motion vector calculation method, hand-movement correction device using the method, imaging device, and motion picture generation device |
| US8923400B1 (en) * | 2007-02-16 | 2014-12-30 | Geo Semiconductor Inc | Method and/or apparatus for multiple pass digital image stabilization |
| CN106101535A (en) * | 2016-06-21 | 2016-11-09 | 北京理工大学 | A kind of based on local and the video stabilizing method of mass motion disparity compensation |
| CN111819837A (en) * | 2018-03-11 | 2020-10-23 | 谷歌有限责任公司 | Static video identification |
| CN108805908A (en) * | 2018-06-08 | 2018-11-13 | 浙江大学 | A kind of real time video image stabilization based on the superposition of sequential grid stream |
| CN109361880A (en) * | 2018-11-30 | 2019-02-19 | 三星电子（中国）研发中心 | A kind of method and system showing the corresponding dynamic picture of static images or video |
| CN110753181A (en) * | 2019-09-29 | 2020-02-04 | 湖北工业大学 | Video image stabilization method based on feature tracking and grid path motion |
| CN110781903A (en) * | 2019-10-12 | 2020-02-11 | 中国地质大学（武汉） | Unmanned aerial vehicle image splicing method based on grid optimization and global similarity constraint |
Non-Patent Citations (1)
| Title |
|---|
| 熊炜;王传胜;管来福;童磊;刘敏;曾春艳;: "基于特征跟踪和网格路径运动的视频稳像算法", 计算机工程与科学, no. 05 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113873099B (en) | 2024-04-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9854168B2 (en) | One-pass video stabilization | |
| US8611602B2 (en) | Robust video stabilization | |
| KR100985805B1 (en) | Apparatus and method for image stabilization using adaptive Kalman filter | |
| JP7078139B2 (en) | Video stabilization methods and equipment, as well as non-temporary computer-readable media | |
| US9406108B2 (en) | Deblurring of an image from a sequence of images | |
| CN110753181A (en) | Video image stabilization method based on feature tracking and grid path motion | |
| US9036695B2 (en) | Motion-compensated temporal filtering based on variable filter parameters | |
| US9253415B2 (en) | Simulating tracking shots from image sequences | |
| CN106991650A (en) | A kind of method and apparatus of image deblurring | |
| CN112288628B (en) | Aerial image splicing acceleration method and system based on optical flow tracking and frame extraction mapping | |
| CN103139568A (en) | Video image stabilizing method based on sparseness and fidelity restraining | |
| US20170111585A1 (en) | Method and system for stabilizing video frames | |
| CN107360377B (en) | Vehicle-mounted video image stabilization method | |
| CN114429191B (en) | Electronic anti-shake method, system and storage medium based on deep learning | |
| Chen et al. | Pixstabnet: Fast multi-scale deep online video stabilization with pixel-based warping | |
| CN113873099A (en) | Power transmission channel video image stabilization method, equipment and medium | |
| CN107135331B (en) | The UAV Video antihunt means and device of low-latitude flying scene | |
| US20130286289A1 (en) | Image processing apparatus, image display apparatus, and image processing method | |
| CN104796721A (en) | Method of subjecting changing light scenes of images to motion estimate by MEMC (motion estimation and moment compensation) technique | |
| Ito et al. | Deep homography-based video stabilization | |
| Senshiki et al. | Blind restoration of blurred images using local patches | |
| CN116664625A (en) | High-real-time image stabilization method for mobile camera equipment | |
| Shang et al. | Video stabilization based on low‐rank constraint and trajectory optimization | |
| Zhou et al. | A joint spatial-temporal 3D video stabilization algorithm | |
| Sreegeethi et al. | Online Video Stabilization using Mesh Flow with Minimum Latency |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |