CN113099217B - Video frame continuity detection method, device, equipment and storage medium - Google Patents
Video frame continuity detection method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113099217B CN113099217B CN202110352170.6A CN202110352170A CN113099217B CN 113099217 B CN113099217 B CN 113099217B CN 202110352170 A CN202110352170 A CN 202110352170A CN 113099217 B CN113099217 B CN 113099217B
- Authority
- CN
- China
- Prior art keywords
- image
- video image
- frame
- frame video
- binary
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N17/00—Diagnosis, testing or measuring for television systems or their details
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a video frame continuity detection method, a device, equipment and a storage medium, which are used for detecting the continuity of a video to be detected, and the method comprises the following steps: analyzing a video to be detected to generate a multi-frame video image; sequentially carrying out image edge calculation on each pixel contained in each frame of video image to obtain a corresponding edge image, and transforming the edge image to obtain a binary image corresponding to each frame of video image; determining whether the M-1 frame video image and the M frame video image are continuous or not according to the binary image of the M-N, M-N +1, \8230 \8230andthe M + N frame video image; wherein, N is the frame number before and after the selected Mth frame, and M is larger than N. According to the scheme, the video continuity detection efficiency and the accuracy are effectively improved.
Description
Technical Field
The present invention relates to the field of video continuity detection technologies, and in particular, to a method, an apparatus, a device, and a storage medium for detecting video frame continuity.
Background
In the judicial field, videos serving as evidences or certificates need to be original videos, and whether the videos are edited or not needs to be detected based on the possibility that recorded videos are edited and modified by people at a later stage.
Since the conventional editing method for video as evidence mainly cuts a video in a local segment, such an operation inevitably causes degradation of continuity of video frames before and after the cut, it is a conventional use means for those skilled in the art to determine the continuity of the video by determining the continuity of the video.
However, at present, it is mainly to adopt a manual mode to judge whether a video is continuous, and adopt a manual mode to judge that video continuity is inefficient, needs to consume a large amount of manpower and material resources, when a large amount of videos need to be detected, the mode of judging through manual work hardly satisfies the demand.
Disclosure of Invention
Therefore, the technical problem to be solved by the present invention is to overcome the problems of low efficiency and large consumption of manpower and material resources caused by manual video continuity detection, thereby providing a technical scheme capable of efficiently detecting video frame continuity.
In a first aspect, a method for detecting continuity of video frames according to an embodiment of the present invention is used to detect continuity of a video to be detected, and includes:
analyzing a video to be detected to generate a multi-frame video image;
sequentially carrying out image edge calculation on each pixel contained in each frame of video image to obtain a corresponding edge image, and transforming the edge image to obtain a binary image corresponding to each frame of video image;
determining whether the M-1 frame video image and the M frame video image are continuous or not according to the binary image of the M-N, M-N +1, \8230 \ 8230and the M + N frame video image;
wherein N is the number of frames before and after the selected Mth frame, and M is greater than N;
the step of determining whether the M-1 frame video image and the M frame video image are continuous or not according to the binary image of the M-N, M-N +1, \8230 \ 8230: (M + N) frame video image comprises the following steps:
and determining whether the M-1 frame video image and the M-1 frame video image are continuous or not according to the binary image of the M-1 frame video image, the accumulated image of the M-N, M-N +1, \8230, the accumulated image of the binary image of the M-1 frame video image, the M, M +1, \8230andthe accumulated image of the binary image of the M + N frame video image.
Preferably, the determining whether the M-1 frame video image and the M + N frame video image are consecutive according to the binary image of the M-N, M-N +1, 8230 \8230; (M + N frame video image) comprises:
determining a first change matrix and a first change coefficient according to a first matrix formed by binary images of an M-th frame of video image and a second matrix formed by an accumulative graph of binary images of the M-th frame of video image, wherein the first matrix is M-N, M-N +1, \ 8230 \8230;
determining a second change matrix and a second change coefficient according to a third matrix formed by binary images of the M-1 frame video image and an M, M +1, \ 8230, and a fourth matrix formed by a cumulative picture of the binary images of the M + N frame video image;
if the first variation coefficient is not larger than a first preset value or the second variation coefficient is not larger than a first preset value, determining that the M frame video image is continuous with the M-1 frame video image;
the first variation coefficient is obtained according to the first variation matrix, and the second variation coefficient is obtained according to the second variation matrix.
Preferably, the method further comprises the following steps:
if the first change coefficient and the second change coefficient are both larger than a first preset value, determining whether the M frame video image and the M-1 video image are continuous according to the intersection ratio IOU of the first change matrix and the second change matrix;
and if the intersection ratio IOU of the first change matrix and the second change matrix is smaller than a second preset value, determining that the video image of the M frame is discontinuous with the video image of the M-1 frame.
Preferably, the method further comprises the following steps:
and if the M frame video image is determined to be discontinuous with the M-1 frame video image according to the binary image of the M frame video image, the binary image of the M-1 frame video image, the M-N, M-N +1, \8230 \ 8230;, the cumulative image of the binary image of the M-1 frame video image and the M, M +1, \8230;, and the cumulative image of the binary image of the M + N frame video image, determining whether the M frame video image is continuous with the M-1 frame video image according to the gray level image of the M frame video image and the gray level image of the M-1 frame video image.
Preferably, the determining whether the mth frame video image and the M-1 frame video image are continuous according to the grayscale images of the mth frame video image and the M-1 frame video image includes:
obtaining a gray scale image G of the Mth frame video image m And gray-scale image G of the M-1 frame video image m-1 ；
Gray scale G based on Mth frame video image m Grayscale image G of the M-1 th frame video image m-1 Determining a grayscale map G m Gray scale image G m-1 An optical flow graph of (a);
according to the gray scale map G m Gray scale image G m-1 Determines whether the M-th frame video image and the M-1 th frame video image are continuous.
Preferably, the gray-scale map G m Gray scale image G m-1 Determining whether the M-th frame video image and the M-1 th frame video image are consecutive, including:
selecting a target pixel point with displacement larger than a preset value according to the displacement of the corresponding pixel point recorded in the light flow graph;
taking a nine-square image P1 and a nine-square image P2 with the target pixel point as the center of the nine-square in the M frame video image and the M-1 frame video image respectively;
and calculating the similarity between the nine-square image P1 and the nine-square image P2, and when the similarity between the nine-square image P1 and the nine-square image P2 is smaller than a set threshold, determining that the M frame video image is discontinuous with the M-1 frame video image.
Preferably, the method further comprises the following steps:
if the M frame video image is determined to be discontinuous with the M-1 frame video image according to the gray level images of the M frame video image and the M-1 frame video image, determining whether the M frame video image and the M-1 frame video image are continuous based on a deep learning neural network;
if the M frame video image is determined to be discontinuous with the M-1 frame video image based on the deep learning neural network, determining that the M frame video image is discontinuous with the M-1 frame video image; if not, then,
and determining that the M frame video image is continuous with the M-1 frame video image.
In a second aspect, a video frame continuity detecting apparatus is provided according to an embodiment of the present invention, for detecting continuity of a video to be detected, including:
the analysis module is used for analyzing the video to be detected to generate a multi-frame video image;
the binary image generation module is used for sequentially carrying out image edge calculation on each pixel contained in each frame of video image to obtain a corresponding edge image, and transforming the edge image to obtain a binary image corresponding to each frame of video image;
the determining module is used for determining whether the M-1 frame video image and the M frame video image are continuous or not according to the binary image of the M-N, M-N +1, \8230 \ 8230and M + N frame video images;
wherein N is the number of frames before and after the selected Mth frame, and M is greater than N;
the determining module is further used for determining whether the Mth frame video image and the M-1 frame video image are continuous or not according to the binary image of the Mth frame video image, the binary image of the M-1 frame video image, the M-N, M-N +1, \8230, the cumulative image of the binary image of the M-1 frame video image, the M, M +1, \8230, and the cumulative image of the binary image of the M + N frame video image.
In a third aspect, there is provided a video frame continuity detecting device according to an embodiment of the present invention, which includes a memory and a processor, where the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform any one of the video frame continuity detecting methods described above.
In a fourth aspect, a computer-readable storage medium is provided according to an embodiment of the present invention, where the computer-readable storage medium stores computer instructions for causing a computer to execute any one of the video frame continuity detection methods described above.
The video frame continuity detection method, the device, the equipment and the storage medium provided by the embodiment of the invention at least have the following beneficial effects:
according to the video frame continuity detection method, the video frame continuity detection device, the video frame continuity detection equipment and the storage medium, a video to be detected can be analyzed into a plurality of frames of video images through a computer and other equipment, then image edge calculation is sequentially carried out on each pixel contained in each frame of video image to obtain a corresponding edge image, and the edge image is converted to obtain a binary image corresponding to each frame of video image; the continuity detection method of the video frame comprises the steps of determining whether the M-1 frame video image and the M-1 frame video image are continuous or not according to the binary image of the M-1 frame video image, the accumulated image of the M-N, M-N +1, \8230, the accumulated image of the binary image of the M-1 frame video image and the M, M +1, \8230, and the accumulated image of the binary image of the M + N frame video image.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a flowchart of a video frame continuity detection method according to an embodiment of the present invention;
FIG. 2 is a flow chart of another method for detecting continuity of video frames according to an embodiment of the present invention;
FIG. 3 is a flowchart of another video frame continuity testing method according to an embodiment of the present invention;
FIG. 4 is a block diagram of a video continuity testing apparatus according to an embodiment of the present invention;
fig. 5 is a schematic diagram of a video continuity check apparatus according to an embodiment of the present invention.
Detailed Description
The technical solutions of the present invention will be described clearly and completely with reference to the accompanying drawings, and it should be understood that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
Furthermore, the technical features involved in the different embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
The video which we conventionally view is generated by one frame of video image, and the general 1s video comprises 25-30 frames of video images; in the judicial field, the requirement for video serving as evidence or evidence is high, and based on the fact that a party providing video evidence may crop the video to be beneficial to the video, whether the analyzed video images which are immediately adjacent in time are continuous or not needs to be detected, if the analyzed video images are continuous, the analyzed video images are not edited, and if the analyzed video images are not continuous, the analyzed video images are edited, and a plurality of video frames are cut among the analyzed video images which are immediately adjacent in time.
Example 1
An embodiment of the present invention provides a method for detecting continuity of video frames, for detecting continuity of a video to be detected, as shown in fig. 1, including:
s12, analyzing a video to be detected to generate a multi-frame video image;
step S14, image edge calculation is sequentially carried out on each pixel contained in each frame of video image to obtain a corresponding edge image, and the edge image is converted to obtain a binary image corresponding to each frame of video image;
s16, determining whether the M-1 frame video image and the M frame video image are continuous or not according to the binary image of the M-N, M-N +1, \8230 \8230andthe M + N frame video image;
wherein N is the number of frames before and after the selected Mth frame, and M is greater than N;
the determining whether the M-1 frame video image and the M frame video image are continuous according to the binary image of the M-N, M-N +1, \8230 \8230andthe M + N frame video image comprises the following steps:
and determining whether the M frame video image and the M-1 frame video image are continuous or not according to the binary image of the M-1 frame video image, the accumulated image of the M-N, M-N +1, \8230, the accumulated image of the binary image of the M-1 frame video image and the accumulated image of the M, M +1, \8230.
In the embodiment of the invention, in the process of detecting the continuity of the video to be detected, the video to be detected is firstly analyzed into the video images of one frame and one frame, the continuity of the video to be detected is determined by detecting the continuity between two adjacent video images one by one, namely, if the video images of every two frames which are adjacent to each other in frame time are continuous after the detection is finished one by one, the video to be detected is continuous, and if the continuous images of certain two frames which are continuous in frame time are found to be discontinuous in the detection process, the video to be detected is edited, and the video to be detected is discontinuous.
In the embodiment of the invention, the adopted detection method comprises the steps of firstly carrying out image edge calculation on pixels in each frame of video image to obtain a corresponding edge image, then carrying out conversion on the edge image of the pixels in the video image aiming at each frame of video image, if the pixels with edges are defined as 1, the pixels without edges are defined as 0, and arranging the binary values of each pixel according to the positions of the pixels in the video image to obtain a binary image corresponding to each frame of video image.
After obtaining the binary image corresponding to each frame of video object, when determining whether the Mth frame and the M-1 frame are continuous, calculating the matrix M formed by the accumulation graph of the binary image of the M-1 frame of video image, wherein M-N, M-N +1, \ 8230, and M-1 frame of video image -1 And a matrix M consisting of the cumulative picture of the binary pictures of the M, M +1, \ 8230n, M + N frame video images 1 Then according to the matrix M -1 Matrix M 1 Binary image C of Mth frame video image M And binary image C of the M-1 frame video image M-1 It is determined whether the M-th frame video image is consecutive to the M-1 th frame video image. And for the video to be detected, if all the adjacent two frames of video images are detected to be continuous, determining that the video to be detected is continuous and is not edited or cut. Otherwise, the video to be detected is determined to be edited and cut.
In the embodiment of the present invention, referring to fig. 2, in step S16, determining whether the M-1 th frame video image and the M + N th frame video image are consecutive according to the binary image of the M-N, M-N +1, \8230; M + N frame video image includes:
step S161, a first matrix C formed by binary images of the Mth frame video image M And (M-N, M-N + 1) \ 8230, a second matrix M formed by accumulation graphs of binary images of the M-1 frame video image -1 Determining a first change matrix and a first change coefficient;
it is pointed out here that the second matrix M -1 Is a matrix obtained by accumulating 0 and 1 values of the edge map transformation results of pixels at the same position in the video image of the M-1 frame;
and the first change matrix isThe change matrix of the two matrices relative to the first matrix, here the first change matrix Diff before (i, j) using the first mathematical model (2-1) to obtain: diff before (i,j)＝M -1 -C M *M -1 (2-1)；
After obtaining the first change matrix Diff before (i, j) then, obtaining a first change coefficient alpha, specifically, obtaining by summing up the elements in the ith row and the jth column in the first change matrix, wherein i =0,1, \8230;, G-1; j =0,1, \ 8230;, H-1, g, H are respectively the height and width of the video frame image, and specifically, the first coefficient of variation α is calculated using a second mathematical model (2-2):
step S162, forming a third matrix M according to the binary image of the M-1 frame video image 1 And a fourth matrix M composed of accumulated graphs of binary images of the M, M +1, \ 8230 `, the M + N frame video image 1 Determining a second change matrix and a second change coefficient;
note that the third matrix M 1 The method is a matrix obtained by accumulating 0 and 1 values of edge map transformation results of pixels at the same position in an Mth frame, M +1, \ 8230;, an Mth frame and an N th frame;
the second change matrix is a change matrix of the fourth matrix relative to the third matrix and is a second sub-matrix Diff after (i, j) using a third mathematical model (2-3):
Diff after (i,j)＝M 1 -C M-1 *M 1 (2-3)；
after obtaining the first change matrix Diff after (i, j) thereafter, a second coefficient of variation beta is determined, in particular, a second matrix of variation Diff is determined after (i, j) wherein i =0,1, \ 8230;, G-1; j =0,1, \ 8230;, H-1, g, H are respectively the height and width of the video frame image, and specifically, the second coefficient of variation β is calculated using a fourth mathematical model (2-4):
step S163, if the first variation coefficient alpha is not greater than a first preset value or the second variation coefficient beta is not greater than a first preset value, determining that the M frame video image is continuous with the M-1 frame video image;
in the embodiment of the invention, when the first variation coefficient alpha or the second variation coefficient beta is not larger than the first preset value, the M frame video image and the M-1 frame video image are judged to be continuous; and when the first change coefficient alpha and the second change coefficient beta are both larger than a first preset value, preliminarily judging whether the M frame video image and the M-1 frame video image are possibly discontinuous, and in order to further determine that the M frame video image and the M-1 frame video image are possibly discontinuous, continuously determining whether the M frame video image and the M-1 frame video image are continuous according to the intersection ratio of the first change matrix and the second change matrix.
Wherein M is -1 Is the second matrix formed by the accumulated image of the binary image of the M-N, M-N +1, \ 8230;, the M-1 frame video image 1 Is the fourth matrix formed by the accumulated graph of the binary image of the M, M +1, \ 8230;, the M + N frame video image, C M A first matrix formed of binary images of the Mth frame of video images, C M-1 And G is the height of the video image, and H is the width of the video image.
Further, referring to fig. 3, the method for detecting continuity of video frames according to an embodiment of the present invention further includes:
and S18, if the first change coefficient and the second change coefficient are both larger than a first preset value, determining whether the M frame video image and the M-1 video image are continuous or not according to the intersection ratio of the first change matrix and the second change matrix.
In the embodiment of the invention, based on the fact that when the first change coefficient and the second change coefficient are both greater than the first preset value, it is preliminarily judged that the M-th frame video image and the M-1-th frame video image may be discontinuous, and in order to further determine whether the M-th frame video image and the M-1-th frame video image are continuous, a method according to the intersection ratio of the first change matrix and the second change matrix is continuously adopted to determine whether the M-th frame video image and the M-1-th frame video image are continuous.
Further, in step S18, the determining whether the mth frame video image and the M-1 th video image are consecutive according to the intersection ratio of the first variation matrix and the second variation matrix includes:
1) Solving the intersection ratio IOU of the first change matrix and the second change matrix by adopting a fifth mathematical model (2-5);
2) And if the intersection ratio IOU of the first change matrix and the second change matrix is smaller than a second preset value, determining that the video image of the M frame is discontinuous with the video image of the M-1 frame.
As for the second preset value, it can be set according to actual requirements, and as a preferable practical implementation manner, the second preset value can be set to be 0.75.
If the intersection ratio IOU of the first change matrix and the second change matrix is not smaller than a second preset value, determining that the M frame video image is continuous with the M-1 frame video image; and if the intersection ratio IOU of the first change matrix and the second change matrix is smaller than a second preset value, determining that the video image of the M frame and the video image of the M-1 frame are possibly discontinuous again.
In an embodiment of the present invention, the method further includes:
and S19, if the M frame video image is determined to be discontinuous with the M-1 frame video image according to the binary image of the M frame video image, the binary image of the M-1 frame video image, the M-N, M-N +1, \8230; \ 8230;, the cumulative image of the binary image of the M-1 frame video image and the M, M +1, \8230;, and the cumulative image of the binary image of the M + N frame video image, determining whether the M frame video image is continuous with the M-1 frame video image according to the gray level image of the M frame video image and the gray level image of the M-1 frame video image.
In the embodiment of the invention, after determining that the M frame video image is not continuous with the M-1 frame video image according to the binary image of the M frame video image, the binary image of the M-1 frame video image, the M-N, M-N +1, \8230 \ 8230;, the cumulative image of the binary image of the M-1 frame video image and the M, M +1, \8230;, and the cumulative image of the binary image of the M + N frame video image, the grayscale image of the M frame video image and the grayscale image of the M-1 frame video image are continuously adopted, thereby improving the accuracy of detecting the video continuity.
Further, in step S19, determining whether the mth frame video image and the M-1 th frame video image are continuous according to the grayscale images of the mth frame video image and the M-1 th frame video image includes:
1) Obtaining a gray scale image G of the Mth frame video image m And a grayscale map G of the M-1 frame video image m-1 ；
In this embodiment, a mathematical model G is used K ＝0.30R K +0.59G K +0.11B K Obtaining a gray scale image G of the Mth frame video image m And gray-scale image G of the M-1 frame video image m-1 Wherein R is K 、G K And B K Three channels representing the K frame video image.
2) Gray scale G based on Mth frame video image m Grayscale image G of the M-1 th frame video image m-1 Determining a grayscale map G m Gray scale image G m-1 An optical flow map of (a);
wherein, according to the gray-scale image G of the Mth frame video image m Determining a grayscale map G m The steps of the light flow diagram of (a) are:
constructing a coordinate system, and determining a gray scale G of the Mth frame video image in the coordinate system m Coordinates of the pixel points and a gray scale image G of the M-1 frame video image m-1 Coordinates of the corresponding pixel points are obtained;
calculating G on the grayscale map m-1 Pixel point-to-gray level map G m The distance between the upper corresponding pixel points is calculated as a gray image G m The optical flow of the pixel point is obtained;
finally, theAccording to the gray scale map G m Generating gray level graph G by using optical flow of each pixel point m The light flow diagram of (a).
Similarly, based on the constructed coordinate system and the determined gray-scale image G of the Mth frame video image m Coordinates of the pixel points on the frame and a gray scale G of the M-1 frame video image m-1 Coordinates of the corresponding pixel points are obtained;
calculating a gray-scale map G m From the pixel point on to the gray scale image G m-1 The distance of the corresponding pixel point is counted as a gray level graph G m-1 The optical flow of the pixel point is obtained;
finally, according to the gray-scale image G m-1 Generating gray level graph G by using optical flow of each pixel point m-1 A light flow diagram of (a).
3) Selecting a target pixel point with displacement larger than a preset value according to the displacement of the pixel point recorded in the light flow graph;
in this embodiment, after the light flow graph is calculated, the displacement of the pixel point is recorded in the light flow graph, and when it is detected that the displacement of the pixel point in the light flow graph is greater than a preset value, the preset value may be 5, or may be another value, or may be set according to an actual requirement.
4) Taking a nine-square image P1 and a nine-square image P2 with a target pixel point as the center of a nine-square in the Mth frame video image and the M-1 th frame video image respectively;
in this embodiment, when a pixel with a displacement greater than a preset value is selected from the optical flow graph, the pixel with the displacement greater than the preset value is determined as a target pixel; after the target pixel point is determined, selecting a Sudoku image P1 and a Sudoku image P2 which take the pixel point as the center on the Mth frame video image and the M-1 th frame video image respectively.
5) And calculating the similarity between the nine-grid image P1 and the nine-grid image P2, and when the similarity between the nine-grid image P1 and the nine-grid image P2 is smaller than a set threshold value, determining that the M frame video image is not continuous with the M-1 frame video image.
After P1 and P2 are selected, the similarity between the image P1 and the image P2 is calculated by the following calculation formula:
wherein, P 1 (x, y) are pixels of the image P1, P 2 (x, y) are pixel points on the image P2.
When the similarity between the image P1 and the image P2 is smaller than a set threshold, for example, the threshold may be 0.8, 0.9, or 0.85, the result of determining whether the M-th frame video image and the M-1-th frame video image are continuous according to the gray-scale map is discontinuous.
Meanwhile, the method for detecting the continuity of the video frames provided by the embodiment of the invention further comprises the following steps:
step S110, if the M frame video image and the M-1 frame video image are determined to be discontinuous according to the gray scale image of the M frame video image and the gray scale image of the M-1 frame video image, determining whether the M frame video image and the M-1 frame video image are continuous based on a deep learning neural network;
step S111, if the M frame video image is determined to be discontinuous with the M-1 frame video image based on the deep learning neural network, determining that the M frame video image is discontinuous with the M-1 frame video image;
and S112, if the M frame video image is determined to be continuous with the M-1 frame video image based on the deep learning neural network, determining that the M frame video image is continuous with the M-1 frame video image.
The specific process of determining whether the M-1 frame video image and the M frame video image are continuous or not based on the deep learning neural network can be as follows:
the deep learning neural network training platform is adopted as follows: a pyroch; the deep learning network structure is as follows: resnet18; the loss function is: crossEntropyLoss (); the output of the deep learning neural network is classified into two categories, and 0 is two frames of continuous data; 1 means that two frames of data are discontinuous;
training process: adopting an optimizer: SGD; initial learning rate: 0.01; learning rate decay strategy: cos; size of each batch: 1024; number of iteration cycles (epoch): 100, respectively; training data are divided into two types and marked as 0,1, wherein the label 0 represents two frames of data are continuous, and the label 1 represents two frames of data are discontinuous. The method for obtaining the data with the label value of 0 comprises the following steps:
D 0 ＝I M-1 -I M
the method for obtaining the data with the label value of 1 comprises the following steps:
D 1 ＝I M-1 -I M+randint(a,b)
the data marked as 0 is the difference between two consecutive frames of video images, and the data marked as 1 is the difference image between two consecutive frames of video images: where randint (a, b) is the generation of a random integer between a (= 1) and b (= 10). Using data D 0 、D 1 Training network, wherein D 0 Data representing a tag value of 0, D 1 Indicating data with a tag value of 1.
And (3) deducing: network input: d = I M-1 -I M When the network outputs 0, it is illustrated that the video frames M-1 and M may be consecutive video frames, and when the network outputs 1. In the embodiment of the invention, when the M-1 frame video image and the M frame video image are determined to be still possibly not continuous video images based on the deep learning neural network, the M-1 frame video image and the M frame video image are determined to be discontinuous, namely, the video to be detected is edited and cut, and a video frame is lacked between the M-1 frame video image and the M frame video image.
In the embodiment of the invention, when the result of determining whether the Mth frame video image and the M-1 frame video image are continuous or not is discontinuous by continuously adopting the neural network based on deep learning, the M frame video image and the M-1 frame video image are determined to be discontinuous; and if the result of determining whether the Mth frame video image and the M-1 frame video image are continuous by adopting the neural network based on deep learning is continuous, determining that the Mth frame video image and the M-1 frame video image are continuous.
Example 2
An embodiment of the present invention further provides a video frame continuity detection apparatus, configured to detect continuity of a video to be detected, as shown in fig. 4, the video frame continuity detection apparatus includes:
the analysis module 42 is configured to analyze the video to be detected to generate a multi-frame video image;
the binary image generation module 44 is configured to perform image edge calculation on each pixel included in each frame of video image in sequence to obtain a corresponding edge map, and transform the edge map to obtain a binary image corresponding to each frame of video image;
the determining module 46 is used for determining whether the M-1 frame video image and the M frame video image are continuous or not according to the binary image of the M-N, M-N +1, \8230; \ 8230and the M + N frame video image;
wherein N is the number of frames before and after the selected Mth frame, and M is greater than N;
the determining module 46 is further configured to determine whether the mth frame video image and the M-1 frame video image are consecutive according to the binary image of the mth frame video image, the binary image of the M-1 frame video image, the M-N, M-N +1, \ 8230;, the cumulative image of the binary image of the M-1 frame video image and the cumulative image of the binary image of the M, M +1, \ 8230;, and the cumulative image of the M + N frame video image.
In an embodiment of the present invention, the determining module includes:
the first calculation unit is used for determining a first change matrix and a first change coefficient according to a first matrix formed by binary images of the Mth frame of video image and an Mth-N, M-N +1, \ 8230, and a second matrix formed by an accumulative image of the binary images of the M-1 th frame of video image;
the second calculation unit is used for determining a second change matrix and a second change coefficient according to a third matrix formed by binary images of the M-1 frame video image and a fourth matrix formed by an accumulative graph of the binary images of the M + N frame video image;
the continuity determining unit is used for determining that the M frame video image is continuous with the M-1 frame video image if the first variation coefficient is not larger than a first preset value or the second variation coefficient is not larger than a first preset value;
in the first computing unit:
first change matrix Diff before Is obtained by adopting a first mathematical model (2-1):
Diff before (i,j)＝M -1 -C M *M -1 (2-1)；
the first coefficient of variation is calculated using a second mathematical model (2-2):
in the second calculation unit:
second change matrix Diff after Is solved by adopting a third mathematical model (2-3):
Diff after (i,j)＝M 1 -C M-1 *M 1 (2-3)；
the second coefficient of variation is calculated by a fourth mathematical model (2-4):
wherein M is -1 Is M-N, M-N +1, \ 8230, a second matrix formed by the accumulation graph of the binary image of the M-1 frame video image, M 1 Is the fourth matrix formed by the accumulated graph of the binary image of the M, M +1, \ 8230;, the M + N frame video image, C M A first matrix formed of binary images of the Mth frame of video images, C M-1 And G is the height of the video image, and H is the width of the video image.
In this embodiment of the present invention, the determining module is further configured to:
and if the first change coefficient and the second change coefficient are both larger than a first preset value, determining whether the Mth frame video image and the M-1 video image are continuous or not according to the intersection ratio of the first change matrix and the second change matrix.
In this embodiment of the present invention, the determining module, according to an intersection ratio of the first variation matrix and the second variation matrix, determines whether the mth frame video image and the M-1 th video image are consecutive, including:
solving the intersection ratio IOU of the first change matrix and the second change matrix by adopting a fifth mathematical model (2-5);
and if the intersection ratio IOU of the first change matrix and the second change matrix is smaller than a second preset value, determining that the video image of the M frame is discontinuous with the video image of the M-1 frame.
After the embodiment of the present invention, the determining module is further configured to:
if the M frame video image is determined to be discontinuous with the M-1 frame video image according to the binary image of the M frame video image, the binary image of the M-1 frame video image, the M-N, M-N +1, \8230;, the cumulative image of the binary image of the M-1 frame video image and the M, M +1, \8230;, and the cumulative image of the binary image of the M + N frame video image, the M frame video image is determined to be discontinuous with the M-1 frame video image
And determining whether the Mth frame video image and the M-1 frame video image are continuous or not according to the gray-scale image of the Mth frame video image and the gray-scale image of the M-1 frame video image.
Further, the determining module determines whether the mth frame video image and the M-1 frame video image are continuous according to the grayscale images of the mth frame video image and the M-1 frame video image, and includes:
obtaining a gray scale image G of the Mth frame video image m And a grayscale map G of the M-1 frame video image m-1 ；
Gray scale G based on Mth frame video image m Grayscale image G of the M-1 th frame video image m-1 Determining a grayscale map G m Gray scale image G m-1 An optical flow map of (a);
selecting the pixel points with the displacement larger than a preset value according to the displacement of the corresponding pixel points recorded in the light flow graph;
taking a nine-square image P1 and a nine-square image P2 which take the pixel point as the center of the nine-square in the M frame video image and the M-1 frame video image respectively;
and calculating the similarity between the nine-square image P1 and the nine-square image P2, and when the similarity between the nine-square image P1 and the nine-square image P2 is smaller than a set threshold, determining that the M frame video image is discontinuous with the M-1 frame video image.
In this embodiment of the present invention, the determining module is further configured to:
if the M frame video image is determined to be discontinuous with the M-1 frame video image according to the gray level images of the M frame video image and the M-1 frame video image, determining whether the M frame video image and the M-1 frame video image are continuous based on a deep learning neural network;
if the M frame video image is determined to be discontinuous with the M-1 frame video image based on the deep learning neural network, determining that the M frame video image is discontinuous with the M-1 frame video image; if not, then,
and determining that the M frame video image is continuous with the M-1 frame video image.
Example 3
The present embodiment provides a video frame continuity detecting device, as shown in fig. 5, the video frame continuity detecting device includes a processor 501 and a memory 502, where the processor 501 and the memory 502 may be connected by a bus or by other means, and fig. 5 takes the example of being connected by a bus as an example.
The Processor 501 may be a Central Processing Unit (CPU), other general-purpose Processor, a Digital Signal Processor (DSP), a Graphics Processing Unit (GPU), an embedded Neural Network Processor (NPU), or other dedicated deep learning coprocessor, an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, or the like, or a combination thereof.
The memory 502, which is a non-transitory computer-readable storage medium, may be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions/modules (e.g., the parsing module 42, the binary image generation module 44, and the determination module 46 shown in fig. 4) corresponding to the video frame continuity detection method in the embodiment of the present invention. The processor 501 executes various functional applications and data processing of the processor by running non-transitory software programs, instructions and modules stored in the memory 502, that is, implements the video frame continuity detection method in the above-described method embodiment 1.
The memory 502 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created by the processor 501, and the like. Further, the memory 502 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 502 optionally includes memory located remotely from processor 501, which may be connected to processor 501 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The one or more modules are stored in the memory 502 and, when executed by the processor 501, perform a video frame continuity detection method as shown in fig. 1.
In this embodiment, the memory 502 stores a program instruction or a module of the video frame continuity detection method, and when the processor 501 executes the program instruction or the module stored in the memory 502, the processor analyzes a video to be detected to generate a multi-frame video image; sequentially carrying out image edge calculation on pixels contained in each frame of video image to obtain a corresponding edge image, and transforming the edge image to obtain a binary image corresponding to each frame of video image; determining whether the Mth frame video image and the M-1 frame video image are continuous or not according to the binary image of the Mth frame video image, the binary image of the M-1 frame video image, the Mth-N, M-N +1, \8230, the accumulated image of the binary image of the M-1 frame video image, the Mth, M +1, \8230, the accumulated image of the binary image of the M-1 frame video image and the accumulated image of the binary image of the M + N frame video image; wherein, the M is larger than N according to the number of the selected frames before and after the Mth frame. Therefore, the video continuity detection of the video to be detected is replaced by the traditional manual detection to the intelligent device (such as a computer) detection, the accuracy is high, and the video continuity detection efficiency is effectively improved.
The embodiment of the invention also provides a non-transitory computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the standard dynamic monitoring method in any method embodiment. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk Drive (Hard Disk Drive, abbreviated as HDD), or a Solid State Drive (SSD); the storage medium may also comprise a combination of memories of the kind described above.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, apparatus, device or computer readable storage medium all relating to or including a computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
It should be understood that the above examples are only for clarity of illustration and are not intended to limit the embodiments. Other variations and modifications will be apparent to persons skilled in the art in light of the above description. And are neither required nor exhaustive of all embodiments. And obvious variations or modifications therefrom are within the scope of the invention.
Claims (9)
1. A video frame continuity detection method is used for detecting continuity of a video to be detected, and is characterized by comprising the following steps:
analyzing a video to be detected to generate a multi-frame video image;
sequentially carrying out image edge calculation on each pixel contained in each frame of video image to obtain a corresponding edge image, and transforming the edge image to obtain a binary image corresponding to each frame of video image;
determining whether the M-1 frame video image and the M frame video image are continuous or not according to the binary image of the M-N, M-N +1, \8230 \8230andthe M + N frame video image;
wherein N is the number of frames before and after the selected Mth frame, and M is greater than N;
the determining whether the M-1 frame video image and the M frame video image are continuous according to the binary image of the M-N, M-N +1, \8230 \8230andthe M + N frame video image comprises the following steps:
determining whether the M-1 frame video image and the M-1 frame video image are continuous or not according to the binary image of the M-1 frame video image, the M-N, M-N +1, \8230;, the cumulative image of the binary image of the M-1 frame video image and the M, M +1, \8230;, and the cumulative image of the binary image of the M + N frame video image;
the step of determining whether the M-1 frame video image and the M frame video image are continuous or not according to the binary image of the M-N, M-N +1, \8230 \ 8230: (M + N) frame video image comprises the following steps:
determining a first change matrix and a first change coefficient according to a first matrix formed by binary images of an M-th frame of video image and a second matrix formed by an accumulative graph of binary images of the M-th frame of video image, wherein the first matrix is M-N, M-N +1, \ 8230 \8230;
determining a second change matrix and a second change coefficient according to a third matrix formed by binary images of the M-1 frame video image and an M, M +1, \ 8230, and a fourth matrix formed by a cumulative picture of the binary images of the M + N frame video image;
if the first variation coefficient is not larger than a first preset value or the second variation coefficient is not larger than the first preset value, determining that the Mth frame video image is continuous with the M-1 th frame video image;
the first change coefficient is obtained according to the first change matrix, and the second change coefficient is obtained according to the second change matrix;
said first change matrix Diff before (i, j) using the first mathematical model (2-1) to obtain:
Diff before (i,j)＝M -1 -C M *M -1 (2-1)；
the first coefficient of variation alpha is calculated by using a second mathematical model (2-2):
second submatrix Diff after (i, j) using a third mathematical model (2-3):
Diff after (i,j)＝M 1 -C M-1 *M 1 (2-3)；
the second coefficient of variation beta is obtained by adopting a fourth mathematical model (2-4):
wherein M is -1 Is M-N, M-N +1, \ 8230, a second matrix formed by the accumulation graph of the binary image of the M-1 frame video image, M 1 Is the fourth matrix formed by the accumulated graph of the binary image of the M, M +1, \ 8230;, the M + N frame video image, C M A first matrix formed of binary images of the Mth frame of video images, C M-1 And G is the height of the video image, and H is the width of the video image.
2. The method of claim 1, further comprising:
if the first change coefficient and the second change coefficient are both larger than a first preset value, determining whether the M frame video image and the M-1 video image are continuous according to the intersection ratio IOU of the first change matrix and the second change matrix;
and if the intersection ratio IOU of the first change matrix and the second change matrix is smaller than a second preset value, determining that the video image of the M frame is discontinuous with the video image of the M-1 frame.
3. The method for detecting continuity of video frames according to claim 1 or 2, further comprising:
and if the M frame video image is determined to be discontinuous with the M-1 frame video image according to the binary image of the M frame video image, the binary image of the M-1 frame video image, the M-N, M-N +1, \8230 \ 8230;, the cumulative image of the binary image of the M-1 frame video image and the M, M +1, \8230;, and the cumulative image of the binary image of the M + N frame video image, determining whether the M frame video image is continuous with the M-1 frame video image according to the gray level image of the M frame video image and the gray level image of the M-1 frame video image.
4. The method according to claim 3, wherein the determining whether the Mth frame video image and the M-1 frame video image are consecutive according to the grayscale images of the Mth frame video image and the M-1 frame video image comprises:
obtaining a gray scale image G of the Mth frame video image m And gray-scale image G of the M-1 frame video image m-1 ；
Gray scale G based on Mth frame video image m Grayscale image G of the M-1 th frame video image m-1 Determining a grayscale map G m Gray scale image G m-1 An optical flow graph of (a);
according to the gray scale map G m Gray scale image G m-1 Determines whether the M-th frame video image and the M-1 th frame video image are continuous.
5. The method according to claim 4, wherein the video frame continuity check is based on a gray-scale map G m Gray scale image G m-1 Determining whether the Mth frame video image and the M-1 frame video image are continuous or not, including:
selecting a target pixel point with displacement larger than a preset value according to the displacement of the corresponding pixel point recorded in the light flow graph;
taking a nine-grid image P1 and a nine-grid image P2 with the target pixel point as the center of a nine-grid from the Mth frame video image and the M-1 frame video image respectively;
and calculating the similarity between the nine-grid image P1 and the nine-grid image P2, and when the similarity between the nine-grid image P1 and the nine-grid image P2 is smaller than a set threshold value, determining that the M frame video image is not continuous with the M-1 frame video image.
6. The method of claim 3, further comprising:
if the M frame video image is determined to be discontinuous with the M-1 frame video image according to the gray level images of the M frame video image and the M-1 frame video image, determining whether the M frame video image and the M-1 frame video image are continuous based on a deep learning neural network;
if the M frame video image is determined to be discontinuous with the M-1 frame video image based on the deep learning neural network, determining that the M frame video image is discontinuous with the M-1 frame video image; if not, then,
and determining that the M frame video image is continuous with the M-1 frame video image.
7. A video frame continuity check device for checking continuity of a video to be checked, comprising:
the analysis module is used for analyzing the video to be detected to generate a multi-frame video image;
the binary image generation module is used for sequentially carrying out image edge calculation on each pixel contained in each frame of video image to obtain a corresponding edge image, and transforming the edge image to obtain a binary image corresponding to each frame of video image;
the determining module is used for determining whether the M-1 frame video image and the M frame video image are continuous or not according to the binary image of the M-N, M-N +1, \8230 \ 8230and the M + N frame video image;
wherein N is the number of frames before and after the selected Mth frame, and M is greater than N;
the determining module is further used for determining whether the M frame video image and the M-1 frame video image are continuous or not according to the binary image of the M frame video image, the binary image of the M-1 frame video image, the M-N, M-N +1, \8230;, the cumulative image of the binary image of the M-1 frame video image and the M, M +1, \8230;, and the cumulative image of the binary image of the M + N frame video image;
the determining module includes:
the first calculation unit is used for determining a first change matrix and a first change coefficient according to a first matrix formed by binary images of the Mth frame of video image and an Mth-N, M-N +1, \ 8230, and a second matrix formed by an accumulative image of the binary images of the M-1 th frame of video image;
the second calculation unit is used for determining a second change matrix and a second change coefficient according to a third matrix formed by binary images of an M-1 frame video image and a fourth matrix formed by an accumulative graph of binary images of an M + N frame video image, wherein the fourth matrix is M, M +1, \8230;
the continuity determining unit is used for determining that the M frame video image is continuous with the M-1 frame video image if the first variation coefficient is not larger than a first preset value or the second variation coefficient is not larger than a first preset value;
in the first computing unit:
first change matrix Diff before Is solved by adopting a first mathematical model (2-1):
Diff before (i,j)＝M -1 -C M *M -1 (2-1)；
the first coefficient of variation is determined using a second mathematical model (2-2):
in the second computing unit:
second change matrix Diff after Is solved by adopting a third mathematical model (2-3):
Diff after (i,j)＝M 1 -C M-1 *M 1 (2-3)；
the second coefficient of variation is calculated by a fourth mathematical model (2-4):
wherein, M -1 Is the second matrix formed by the accumulated image of the binary image of the M-N, M-N +1, \ 8230;, the M-1 frame video image 1 Is the fourth matrix formed by the accumulated graph of the binary image of the M, M +1, \ 8230;, the M + N frame video image, C M A first matrix formed of binary images of the Mth frame of video images, C M-1 And G is the height of the video image, and H is the width of the video image.
8. A video frame continuity check device, comprising a memory and a processor, wherein the memory and the processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to execute the video frame continuity check method according to any one of claims 1 to 6.
9. A computer-readable storage medium storing computer instructions for causing a computer to execute the video frame continuity detection method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110352170.6A CN113099217B (en) | 2021-03-31 | 2021-03-31 | Video frame continuity detection method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110352170.6A CN113099217B (en) | 2021-03-31 | 2021-03-31 | Video frame continuity detection method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113099217A CN113099217A (en) | 2021-07-09 |
| CN113099217B true CN113099217B (en) | 2022-11-25 |
Family
ID=76672194
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110352170.6A Active CN113099217B (en) | 2021-03-31 | 2021-03-31 | Video frame continuity detection method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113099217B (en) |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108769776B (en) * | 2018-05-31 | 2021-03-19 | 北京奇艺世纪科技有限公司 | Title subtitle detection method and device and electronic equipment |
| CN110708568B (en) * | 2019-10-30 | 2021-12-10 | 北京奇艺世纪科技有限公司 | Video content mutation detection method and device |
| CN110909712B (en) * | 2019-12-05 | 2021-04-16 | 重庆紫光华山智安科技有限公司 | Moving object detection method and device, electronic equipment and storage medium |
| CN111935506B (en) * | 2020-08-19 | 2023-03-28 | 百度时代网络技术（北京）有限公司 | Method and apparatus for determining repeating video frames |
-
2021
- 2021-03-31 CN CN202110352170.6A patent/CN113099217B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN113099217A (en) | 2021-07-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110738207B (en) | Character detection method for fusing character area edge information in character image | |
| CN112132156B (en) | Image saliency target detection method and system based on multi-depth feature fusion | |
| CN111681273B (en) | Image segmentation method and device, electronic equipment and readable storage medium | |
| CN111008600B (en) | Lane line detection method | |
| CN114187311A (en) | Image semantic segmentation method, device, equipment and storage medium | |
| US10373022B1 (en) | Text image processing using stroke-aware max-min pooling for OCR system employing artificial neural network | |
| CN114359563B (en) | Model training method, device, computer equipment and storage medium | |
| CN111783712A (en) | Video processing method, device, equipment and medium | |
| WO2021164280A1 (en) | Three-dimensional edge detection method and apparatus, storage medium and computer device | |
| CN111639230B (en) | Similar video screening method, device, equipment and storage medium | |
| CN114241388A (en) | Video instance segmentation method and segmentation device based on space-time memory information | |
| CN111368634A (en) | Human head detection method, system and storage medium based on neural network | |
| CN111444807A (en) | Target detection method, device, electronic equipment and computer readable medium | |
| CN111242066A (en) | Large-size image target detection method and device and computer readable storage medium | |
| CN116994721B (en) | Quick processing system of digital pathological section graph | |
| CN113963333A (en) | Traffic sign board detection method based on improved YOLOF model | |
| CN113435370A (en) | Method and device for obtaining vehicle queuing length based on image feature fusion | |
| CN116385466B (en) | Method and system for dividing targets in image based on boundary box weak annotation | |
| CN113099217B (en) | Video frame continuity detection method, device, equipment and storage medium | |
| CN116664829A (en) | RGB-T semantic segmentation method, system, device and storage medium | |
| CN114511862B (en) | Form identification method and device and electronic equipment | |
| CN114724175A (en) | Pedestrian image detection network, detection method, training method, electronic device, and medium | |
| CN112487911A (en) | Real-time pedestrian detection method and device based on improved yolov3 in intelligent monitoring environment | |
| CN111161250B (en) | Method and device for detecting dense houses by using multi-scale remote sensing images | |
| CN115550684B (en) | Improved video content filtering method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |