CN107657228A - Video scene similarity analysis method and system, video coding-decoding method and system - Google Patents
Video scene similarity analysis method and system, video coding-decoding method and system Download PDFInfo
- Publication number
- CN107657228A CN107657228A CN201710873784.2A CN201710873784A CN107657228A CN 107657228 A CN107657228 A CN 107657228A CN 201710873784 A CN201710873784 A CN 201710873784A CN 107657228 A CN107657228 A CN 107657228A
- Authority
- CN
- China
- Prior art keywords
- mrow
- msub
- key frame
- mtr
- mtd
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 39
- 238000000034 method Methods 0.000 title claims abstract description 30
- 239000011159 matrix material Substances 0.000 claims abstract description 131
- 239000013598 vector Substances 0.000 claims abstract description 60
- 238000010276 construction Methods 0.000 claims description 15
- 230000006835 compression Effects 0.000 claims description 14
- 238000007906 compression Methods 0.000 claims description 14
- 238000000605 extraction Methods 0.000 claims description 12
- 230000001186 cumulative effect Effects 0.000 claims description 11
- 238000013016 damping Methods 0.000 claims description 6
- 238000012216 screening Methods 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 4
- 230000014509 gene expression Effects 0.000 claims description 4
- 239000003086 colorant Substances 0.000 claims description 2
- 102100037812 Medium-wave-sensitive opsin 1 Human genes 0.000 claims 4
- 230000000903 blocking effect Effects 0.000 claims 1
- 238000000638 solvent extraction Methods 0.000 claims 1
- 238000005516 engineering process Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 239000000284 extract Substances 0.000 description 3
- 238000013139 quantization Methods 0.000 description 3
- 238000009825 accumulation Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 230000002457 bidirectional effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000000969 carrier Substances 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004043 responsiveness Effects 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/13—Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/186—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a colour or a chrominance component
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Signal Processing (AREA)
- Image Analysis (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The present invention, which provides video scene similarity analysis method and system, video coding-decoding method and system, analysis method, to be included:A two field picture is chosen respectively as key frame from video each camera lens;The characteristic vector of each key frame is extracted, builds similarity matrix;According to similarity matrix, using Attraction Degree and degree of membership and as linear kernel each key frame is clustered, set the cluster labels of each key frame.Video coding-decoding method includes above-mentioned video scene similarity analysis method;The key frame gathered for one kind is compressed as a GOP;Rebuild after key frame is compressed in coding side and be placed on frame buffer, remaining GOP B frames and P frames find key frame corresponding to frame buffer by respective crucial frame index and carry out inter prediction encoding;During decoding, all key frames are first decoded, B frames and P frames are decoded further according to cluster labels.The above method and system can excavate the correlation between discontinuous frame and discontinuous GOP.
Description
Technical Field
The present invention relates to the field of video processing technologies, and in particular, to a video scene similarity analysis method, a video encoding and decoding method, a video scene similarity analysis system, and a video encoding and decoding system.
Background
With the rapid development of internet industry in recent years, carriers of various multimedia contents such as digital image video and audio are also increasing explosively, so that bandwidth and storage are subject to great pressure if image video data is not compressed redundantly. Conventional redundancy compression methods are intra prediction coding of key frames and inter prediction coding of consecutive frames, and do not consider redundancy between non-consecutive frames and non-consecutive GOPs (groups of Pictures).
In the field of image video compression coding research, the mainstream idea at present is expanded and improved based on 'prediction-transformation-entropy coding'. Although the development of applications has been successful, the compression efficiency has reached a bottleneck. Therefore, people begin to analyze the content of the image video and combine with the video compression technology to break through the traditional image video compression technology bottle.
In 2007, Liu et al, proposed an image coding method based on image inpainting technology, and the idea of the method is to divide an image into a structural region and a texture region, and use analysis tools such as image edge extraction technology and texture detection. Each region is further divided into three categories: essential content, partial essential content and redundant content. Necessary structural information and texture information are necessary contents, and necessary contents are referred to determine whether the gradient change is large or not, and the necessary contents are partial necessary contents; the portions that need to be restored from the first two are then redundant portions. For the different content areas which are separated out, different coding methods are adopted, and the redundancy does not need to be coded, so that the coding efficiency can be greatly improved. On the basis of the work, Liu et al combines image content analysis with a traditional compression coding method, provides an intra-frame prediction method based on edges, utilizes image edge structure information and image local continuity described by Laplace equation to enable the intra-frame prediction direction to be more self-adaptive, and can be compatible with the existing image and video coding standards based on blocks.
Conventional video coding systems generally use the correlation between consecutive frames to remove temporal redundancy, and only search for frames within a GOP for reference even with the multi-reference frame technique, so that the number of I frames (key frames) is large and the correlation between I frames is not considered.
Disclosure of Invention
In view of the foregoing problems, it is an object of the present invention to provide a video scene similarity analysis method, a video encoding and decoding method, a video scene similarity analysis system, and a video encoding and decoding system for mining the correlation between discontinuous frames and discontinuous GOPs.
According to an aspect of the present invention, there is provided a video scene similarity analysis method, including: respectively selecting a frame of image from each shot of a video as a key frame of each shot; extracting the feature vector of each key frame, measuring the similarity between the key frames according to the feature vector, and constructing a similarity matrix; and according to the similarity matrix, clustering each key frame by taking the sum of the attraction degree and the attribution degree as a linear core, and setting a clustering label of each key frame, wherein the clustering label is represented by a clustering center corresponding to the key frame, the attraction degree represents information transmitted from the key frame to the clustering center, and the attribution degree represents information transmitted from the clustering center to the key frame.
According to another aspect of the present invention, there is provided a video encoding and decoding method, including: the video scene similarity analysis method is provided; compressing the key frames grouped into one class as a GOP; compressing the key frame, reconstructing the compressed key frame at an encoding end, and placing the compressed key frame in a frame buffer area, and finding the key frame corresponding to the frame buffer area by the B frame (bidirectional difference frame) and the P frame (difference frame with the previous frame) of the rest GOPs through respective key frame indexes to perform inter-frame prediction encoding; when decoding, all key frames are decoded first, and then B frames and P frames are decoded according to the clustering labels.
According to a third aspect of the present invention, there is provided a video scene similarity analysis system, comprising: the key frame extraction module is used for respectively selecting a frame of image from each shot of the video as a key frame of each shot and sending each key frame to the similarity matrix construction module; the similarity matrix construction module is used for extracting the feature vector of each key frame, measuring the similarity between the key frames according to the feature vector, constructing a similarity matrix and sending the similarity matrix to the clustering module; and the clustering module is used for clustering each key frame by taking the sum of the attraction degree and the attribution degree as a linear check according to the similarity matrix, setting a clustering label of each key frame, wherein the clustering label is represented by a clustering center corresponding to the key frame, the attraction degree represents information transmitted from the key frame to the clustering center, and the attribution degree represents information transmitted from the clustering center to the key frame.
According to a fourth aspect of the present invention, there is provided a video coding and decoding system, comprising:
the video scene similarity analysis system; a compression unit for compressing the key frames grouped into one group as a GOP; the encoding part is used for reconstructing the compressed key frames at an encoding end and placing the compressed key frames in a frame buffer area, and finding the key frames corresponding to the frame buffer area by the B frames and the P frames of the rest GOPs through respective key frame indexes to perform inter-frame prediction encoding; and the decoding part decodes all the key frames and then decodes the B frames and the P frames according to the clustering labels.
The video scene similarity analysis method and the video scene similarity analysis system cluster I frames (key frames), and compress the I frames which are clustered into one class as a GOP, so that the number of the I frames which are independently predicted in the frame is reduced, and the data volume of the I frames is further compressed.
The video coding and decoding method and the video coding and decoding system provided by the invention utilize the similarity of video scenes to cluster the I frames occupying the most code rate in each GOP, excavate the correlation between discontinuous frames and discontinuous GOPs, are combined with the video compression technology to greatly improve the video compression efficiency, and can improve the compression efficiency by 3% -4% compared with the traditional coding when the scene similarity analysis technology is applied to the compression of the I frames under the condition that the PSNR is basically maintained.
Drawings
Other objects and results of the present invention will become more apparent and more readily appreciated as the same becomes better understood by reference to the following description and appended claims, taken in conjunction with the accompanying drawings. In the drawings:
FIG. 1 is a flow chart of a video scene similarity analysis method according to the present invention;
FIG. 2 is a flow chart of constructing a similarity matrix according to the present invention;
FIG. 3 is a flow chart of clustering key frames according to the present invention using the sum of the attraction degree and the attribution degree as a linear core;
FIG. 4 is a schematic diagram of the clustering compression of key frames according to the present invention;
FIG. 5 is a block diagram of a video scene similarity analysis system according to the present invention;
fig. 6 is a block diagram of the video codec system according to the present invention.
The same reference numbers in all figures indicate similar or corresponding features or functions.
Detailed Description
In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more embodiments. It may be evident, however, that such embodiment(s) may be practiced without these specific details. Specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
Specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
Fig. 1 is a flowchart of a video scene similarity analysis method according to the present invention, and as shown in fig. 1, the video scene similarity analysis method includes:
step 1, respectively selecting a frame of image from each shot of a video as an I frame of each shot;
step 2, extracting the feature vector of each I frame, measuring the similarity between the I frames according to the feature vector, and constructing a similarity matrix;
and 3, according to the similarity matrix, taking the sum of the attraction degree and the attribution degree as a linear check to cluster the I frames, and setting the clustering labels of the I frames, wherein the clustering labels are represented by the clustering centers corresponding to the I frames, the attraction degree represents the information transmitted from the I frames to the clustering centers, and the attribution degree represents the information transmitted from the clustering centers to the I frames.
For videos with high content regression rate, the relevance between non-continuous frames and non-continuous GOPs is still very large, the video scene similarity analysis method clusters the I frames occupying the most code rate in each GOP, excavates the relevance between the non-continuous frames and the non-continuous GOPs, reduces the number of the I frames independently subjected to intra-frame prediction, and further compresses the data volume of the I frames.
In a preferred embodiment of the present invention, the step 1 includes: detecting the boundaries of the shots, and determining two adjacent boundaries as a shot; selecting a frame at a set position in each shot as an I frame, for example, determining each set number of frames (15 frames) as a shot, then using the first frame in each set number of frames as a key frame of the shot, or, for example, directly selecting the first frame, the last frame, or an internal frame in the shot as an I frame of the shot according to a requirement.
In a preferred embodiment of the present invention, as shown in fig. 2, the step 2 includes:
step 21, the image of the key frame is blocked to form sub-blocks with the same size, a histogram of the color of the sub-blocks is obtained, a cumulative histogram of the key frame is obtained through accumulation, the histogram represents the distribution situation of all pixel chromaticity space values on one image, it reflects the distribution of the image color space and the basic tones, but the histogram does not include the location features, therefore, different images may obtain the same histogram feature through calculation, and aiming at the problem, the invention adopts a representation method of cumulative histograms, that is, instead of directly calculating the image histogram, the image histogram is preprocessed first to divide the image into blocks with the same size, and then the color histogram of the original image sub-blocks is calculated, then, a cumulative histogram is calculated, and the cumulative histogram is a method with higher robustness compared with a color histogram;
step 22, extracting the color feature vector of the cumulative histogram of each I frame by using a color space, for example, using an HSV color space, performing non-uniform quantization on HSV values to reduce the calculated amount, generally dividing hue H into eight quantization regions, dividing saturation S and brightness V into three quantization regions, and according to the difference of the value ranges thereof, specifically quantizing the hue H as follows:
the invention quantizes the chroma H into 8 parts, the saturation S into 3 parts and the brightness V into 3 parts, thus obtaining a 72-dimensional characteristic vector;
step 23, extracting the texture feature vector of each I frame by using a gray level co-occurrence matrix, specifically: constructing a gray level co-occurrence matrix of each key frame; constructing a texture feature vector by adopting one or more indexes of energy, inertia, correlation, entropy, local uniformity and maximum probability; normalizing the texture feature vector by adopting Gaussian normalization;
step 24, constructing a similarity matrix according to the following formula (1) by using the color feature vector and the texture feature vector of each I frame,
where m, n are two I-frames, sm,nRepresents the similarity of I frame m and I frame n, (x)1,x2,...,xd) Is the feature vector of I frame m, (y)1,y2,...,yd) Is an I-frame n feature vector.
Preferably, in step 23, a co-occurrence matrix in 4 directions (0, pi/4, pi/2, 3 pi/4) is constructed, gaussian normalization is applied to the co-occurrence matrix, and the mean and standard deviation of the 4 parameters of inertia, energy entropy, correlation and entropy are taken as components to describe the texture feature vector, so as to obtain an 8-dimensional texture feature vector.
In addition, preferably, a similarity matrix, for example, (x) may be constructed according to the above formula (1) from the color texture synthesis feature vector1,x2,...,xd) In the feature vector of I frame m (x)1,x2,...,xj) Is a color feature vector, (x)j+1,xj+2,...,xd) the color texture comprehensive characteristic vector is (α x) if the color texture comprehensive characteristic vector is the texture characteristic vector1,αx2,...,αxj,βxj+1,...,βxd) α and β are characteristic coefficients, α + β being 1.
In a preferred embodiment of the present invention, as shown in fig. 3, the step 3 includes:
step 31, initializing the similarity matrix, using the average value (reference P) of each similarity in the similarity matrix as the self-similarity of each I frame, setting the initial attribution degree and the initial attraction degree between each I frame in the attribution matrix and the attraction matrix to be 0,
wherein S is a similarity matrix, and the average value of all similarities in the similarity matrix S is used as S1,1,s2,2,…,sm,mA value of (d); a is a attribution degree matrix, am,nRepresenting the attribution degree of the I frame m to the I frame n; r is an attraction matrix, Rm,nRepresenting the attraction of the I frame n to the I frame m;
step 32, updating the attraction degree matrix according to the similarity degree matrix by adopting the following formula (2),
wherein r'm,nAdopting the attraction degree of the key frame n to the key frame m after the similarity matrix is updated for the initial attribution degree and the initial attraction degree which are both 0;
step 33, updating the attribution degree matrix according to the following formulas (3) and (4) according to the updated attraction degree matrix,
wherein, a'm,nAnd a'm,mRespectively carrying out the attribution degree of the key frame m by the key frame n updated according to the updated attraction degree matrix and the autocorrelation attribution degree of the key frame m;
step 34, setting a damping factor λ (for example, 0.5) and the number of iterations T, iteratively updating the attraction degree matrix and the attribution degree matrix by using the following equations (5) and (6),
wherein,for the attraction of the keyframe n to the keyframe m after being updated by T iterations,attribution degree of the key frame n to the key frame m after the iteration update for T times;
step 35, after the iterative update, filtering out the I frame whose sum of the autocorrelation attribution degree and the autocorrelation attraction degree is not more than 0, namely judging whether the autocorrelation attribution degree and the autocorrelation attraction degree of each key frame accord with the following formula (7), filtering out the key frames which do not accord with the following formula (7), wherein,
step 36, the maximum value of the sum of the attribution degree and the attraction degree of the I frame meeting the condition that the autocorrelation attribution degree and the autocorrelation attraction degree are more than 0 and other I framesAnd the corresponding other I frame is used as the clustering center of the I frame, so that the clustering center of each I frame meeting the condition is obtained, and the I frames belonging to the same clustering center are clustered into one class.
According to the method for clustering each key frame by using the sum of the attraction degree and the attribution degree as the linear check according to the similarity matrix, the defect that a traditional clustering algorithm is sensitive when an initial value is selected is overcome, all data are used as candidate clustering centers, each data needs to be continuously calculated, iterated and updated through messages transmitted among the data, the data competing to become the clustering centers or other clustering centers are finally determined, and finally a plurality of high-quality clustering centers are obtained after iteration is finished.
In addition, the above-mentioned clustering method "message passing mechanism" performs the passing of two kinds of information, i.e., the attraction (responsiveness) and the attribution (availabilitity), between the data point and the candidate clustering center, and the attraction isIndicating information transferred from the data to the candidate cluster center k, degree of attributionRepresenting the information passed from candidate cluster center k to data point i; the attraction degree represents the appropriate degree of the k point becoming the clustering center of the data point i, and the attribution degree represents the appropriate degree of the i point taking the k point as the clustering center;andthe larger the probability is, the higher the probability that the data point i has k points as its own cluster center is, and the higher the probability that the k points become the cluster centers is.
Furthermore, a Reference (Reference) is associated with the attraction and the attribution, and the Reference is not information to be transmitted but a criterion. I.e. using the value S on the diagonal of the similarity matrix Sk,kTo judge whether the k point can become a clustering center, sk,kThe larger the k point becomes the cluster center, the higher the probability that the k point becomes the cluster center, and in the initialization process of the cluster, any data point i does not belong to any class, and any data k is not the cluster center, so the mean value of the matrix S is assigned to the reference P of all data points.
The video scene similarity analysis method can be applied to a video coding and decoding method, which includes the video analysis method, and further includes:
compressing I frames grouped into one class as a GOP, wherein as shown in FIG. 4, the I frames of F01-F20 respectively have respective clustering centers which can be represented by a code number, such as S1-S4, and compressing the I frames grouped into one class as a GOP, such as GOP1-GOP 4;
compressing the I frame, reconstructing the I frame at an encoding end, and placing the I frame in a frame buffer area, wherein B frames and P frames of other GOPs find the I frame corresponding to the frame buffer area through respective I frame indexes to perform inter-frame prediction encoding;
when decoding, all I frames are decoded, and then B frames and P frames are decoded according to the clustering labels.
Fig. 5 is a block diagram of a video scene similarity analysis system according to the present invention, and as shown in fig. 5, the video scene similarity analysis system 100 includes:
the key frame extraction module 110 is configured to select a frame of image from each shot of the video as an I frame of each shot, and send each I frame to the similarity matrix construction module 120;
the similarity matrix construction module 120 extracts the feature vector of each frame I, measures the similarity between frames I according to the feature vector, constructs a similarity matrix, and sends the similarity matrix to the clustering module 130;
the clustering module 130 is configured to perform clustering on each I frame by using the sum of the attraction degree and the attribution degree as a linear check according to the similarity matrix, and set a clustering label of each I frame, wherein the clustering label is represented by a clustering center corresponding to the I frame, the attraction degree represents information transmitted from the I frame to the clustering center, and the attribution degree represents information transmitted from the clustering center to the I frame.
In a preferred embodiment of the present invention, the key frame extracting module 11 includes:
a shot determining unit 111 that detects a boundary of a shot and determines a space between two adjacent boundaries as a shot;
the key frame extracting unit 112 selects a frame at a predetermined position in each shot as an I frame.
In a preferred embodiment of the present invention, the similarity matrix structure modeling 120 block includes:
the cumulative histogram construction unit 121 divides the image of the I frame into blocks to form sub-blocks with the same size, obtains a histogram of colors of the sub-blocks, obtains the cumulative histogram of the I frame through accumulation, and sends the cumulative histogram to the color feature vector extraction unit 122;
the color feature vector extraction unit 122 extracts the color feature vector of the cumulative histogram of each I frame by using a color space, and sends the color feature vector of each I frame to the similarity matrix construction unit 124;
the texture feature vector extraction unit 123 extracts the texture feature vector of each I frame by using the gray level co-occurrence matrix, and sends the texture feature vector of each I frame to the similarity matrix construction unit 124;
a similarity matrix construction unit 124 constructing a similarity matrix according to the following equation (1) using the color feature vector and texture feature vector of each I frame,
where m, n are two I-frames, sm,nRepresents the similarity of I frame m and I frame n, (x)1,x2,…,xd) Is the feature vector of I frame m, (y)1,y2,…,yd) Is an I-frame n feature vector.
In a preferred embodiment of the present invention, the clustering module 130 includes:
the initializing unit 131 initializes the similarity matrix, uses an average value of each similarity in the similarity matrix as a self-similarity of each I frame, and sends the initialized similarity matrix to the first updating unit 133, wherein,
wherein S is a similarity matrix, and the average value P of all similarities in the similarity matrix S is used as S1,1,s2,2,…,sm,mA value of (d);
the setting unit 132 sets the initial attribution degree and the initial attraction degree between the I frames in the attribution degree matrix and the attraction degree matrix to be 0, sends the initial attribution degree and the initial attraction degree to the first updating unit 133, sets the damping factor λ and the iteration number T, sends the damping factor λ and the iteration number T to the second updating unit 134, wherein,
wherein A is a attribution degree matrix, am,nRepresenting the attribution degree of the I frame m to the I frame n; r is an attraction matrix, Rm,nRepresenting the attraction of the I frame n to the I frame m;
the first updating unit 133 updates the affinity matrix according to the similarity matrix by using the following expression (2), updates the attribution matrix according to the following expressions (3) and (4) according to the updated affinity matrix, transmits the updated affinity matrix and attribution matrix to the second updating unit 134,
wherein r'm,nThe attraction degree a 'of the key frame n to the key frame m updated by the similarity matrix is adopted for both the initial attribution degree and the initial attraction degree being 0'm,nAnd a'm,mRespectively carrying out the attribution degree of the key frame m by the key frame n updated according to the updated attraction degree matrix and the autocorrelation attribution degree of the key frame m;
the second updating unit 134 iteratively updates the attraction degree matrix and the attribution degree matrix updated by the first updating unit by using the following formulas (5) and (6), and sends the attraction degree matrix and the attribution degree matrix after T iterations to the screening unit 135, wherein,
wherein,for the attraction of the keyframe n to the keyframe m after being updated by T iterations,attribution degree of the key frame n to the key frame m after the iteration update for T times;
the screening unit 135 filters out the I frame whose sum of the autocorrelation attribution degree and the autocorrelation attraction degree is not greater than 0, and sends the attraction degree matrix and the attribution degree matrix, which are used for filtering out the vectors corresponding to the I frame that do not meet the above conditions, to the clustering unit 136;
the clustering unit 136 uses the attraction degree matrix and the other I frame corresponding to the maximum value of the sum of the attraction degree and the attraction degree of the I frame and the other I frame in the attraction degree matrix filtered by the screening unit as the clustering center of the I frame, so as to obtain the clustering center of each I frame meeting the conditions, and cluster the I frames belonging to the same clustering center into one class.
The video scene similarity analysis system can be applied to a video coding and decoding system, as shown in fig. 6, the video coding and decoding system 1000 includes, in addition to the video scene similarity analysis system 100, the following:
a compression unit 200 for compressing I frames grouped into one group as one GOP;
an encoding part 300, which reconstructs the I frame after compressing the I frame at an encoding end and places the I frame in a frame buffer area, and B frames and P frames of other GOPs find the I frame corresponding to the frame buffer area through respective I frame indexes to perform inter-frame prediction encoding;
the decoding unit 400 decodes all I frames first, and then decodes B frames and P frames based on the cluster labels.
In summary, the video scene similarity analysis method, the video encoding and decoding method, the video scene similarity analysis system and the video encoding and decoding system proposed by the present invention are described by way of example with reference to the accompanying drawings. However, it will be appreciated by those skilled in the art that various modifications could be made to the system and method of the present invention described above without departing from the spirit of the invention. Therefore, the scope of the present invention should be determined by the contents of the appended claims.
Claims (10)
1. A video scene similarity analysis method is characterized by comprising the following steps:
respectively selecting a frame of image from each shot of a video as a key frame of each shot;
extracting the feature vector of each key frame, measuring the similarity between the key frames according to the feature vector, and constructing a similarity matrix;
and according to the similarity matrix, clustering each key frame by taking the sum of the attraction degree and the attribution degree as a linear core, and setting a clustering label of each key frame, wherein the clustering label is represented by a clustering center corresponding to the key frame, the attraction degree represents information transmitted from the key frame to the clustering center, and the attribution degree represents information transmitted from the clustering center to the key frame.
2. The method for analyzing similarity of video scenes according to claim 1, wherein the method for constructing the similarity matrix comprises:
the method comprises the steps of partitioning an image of a key frame into sub-blocks with the same size to obtain a histogram of colors of the sub-blocks, and accumulating to obtain an accumulated histogram of the key frame;
extracting a color feature vector of the cumulative histogram of each key frame by adopting a color space;
extracting a texture feature vector of each key frame by adopting a gray level co-occurrence matrix;
constructing a similarity matrix according to the following formula (1) by using the color feature vector and the texture feature vector of each key frame,
<mrow> <msub> <mi>s</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>n</mi> </mrow> </msub> <mo>=</mo> <mfrac> <mrow> <munderover> <mo>&Sigma;</mo> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>d</mi> </munderover> <msub> <mi>x</mi> <mi>i</mi> </msub> <msub> <mi>y</mi> <mi>i</mi> </msub> </mrow> <msqrt> <mrow> <mo>(</mo> <munderover> <mo>&Sigma;</mo> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>d</mi> </munderover> <msub> <msup> <mi>x</mi> <mn>2</mn> </msup> <mi>i</mi> </msub> <mo>)</mo> <mo>(</mo> <munderover> <mo>&Sigma;</mo> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>d</mi> </munderover> <msub> <msup> <mi>y</mi> <mn>2</mn> </msup> <mi>i</mi> </msub> <mo>)</mo> </mrow> </msqrt> </mfrac> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>1</mn> <mo>)</mo> </mrow> </mrow>
where m, n are two key frames, sm,nRepresenting the similarity of key frame m and key frame n, (x)1,x2,...,xd) Is the feature vector of the key frame m, (y)1,y2,...,yd) The keyframe n feature vectors.
3. The method of claim 1, wherein the selecting a frame of image from each shot of the video as the key frame of each shot comprises:
detecting the boundaries of the shots, and determining two adjacent boundaries as a shot;
and selecting a frame at a set position in each lens as a key frame.
4. The video scene similarity analysis method according to claim 1, wherein the method for clustering the key frames by using the sum of the attraction degree and the attribution degree as a linear kernel according to the similarity matrix, and setting the clustering label of each key frame comprises:
initializing the similarity matrix, taking the average value of each similarity in the similarity matrix as the self-similarity of each key frame,
<mrow> <mi>S</mi> <mo>=</mo> <mfenced open = "(" close = ")"> <mtable> <mtr> <mtd> <mrow> <msub> <mi>s</mi> <mrow> <mn>1</mn> <mo>,</mo> <mn>1</mn> </mrow> </msub> <mo>,</mo> <msub> <mi>s</mi> <mrow> <mn>1</mn> <mo>,</mo> <mn>2</mn> </mrow> </msub> <mo>,</mo> <mo>...</mo> <msub> <mi>s</mi> <mrow> <mn>1</mn> <mo>,</mo> <mi>n</mi> </mrow> </msub> </mrow> </mtd> </mtr> <mtr> <mtd> <mo>.</mo> </mtd> </mtr> <mtr> <mtd> <mo>.</mo> </mtd> </mtr> <mtr> <mtd> <mo>.</mo> </mtd> </mtr> <mtr> <mtd> <mrow> <msub> <mi>s</mi> <mrow> <mi>m</mi> <mo>,</mo> <mn>1</mn> </mrow> </msub> <mo>,</mo> <msub> <mi>s</mi> <mrow> <mi>m</mi> <mo>,</mo> <mn>2</mn> </mrow> </msub> <mo>,</mo> <mo>...</mo> <msub> <mi>s</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>n</mi> </mrow> </msub> </mrow> </mtd> </mtr> </mtable> </mfenced> </mrow>
wherein S is a similarity matrix, and the average value P of all similarities in the similarity matrix S is used as S1,1,s2,2,…,sm,mA value of (d);
setting the initial attribution degree and the initial attraction degree between the key frames in the attribution degree matrix and the attraction degree matrix to be 0, wherein,
<mrow> <mi>A</mi> <mo>=</mo> <mfenced open = "(" close = ")"> <mtable> <mtr> <mtd> <mrow> <msub> <mi>a</mi> <mrow> <mn>1</mn> <mo>,</mo> <mn>1</mn> </mrow> </msub> <mo>,</mo> <msub> <mi>a</mi> <mrow> <mn>1</mn> <mo>,</mo> <mn>2</mn> </mrow> </msub> <mo>,</mo> <mo>...</mo> <msub> <mi>a</mi> <mrow> <mn>1</mn> <mo>,</mo> <mi>n</mi> </mrow> </msub> </mrow> </mtd> </mtr> <mtr> <mtd> <mo>.</mo> </mtd> </mtr> <mtr> <mtd> <mo>.</mo> </mtd> </mtr> <mtr> <mtd> <mo>.</mo> </mtd> </mtr> <mtr> <mtd> <mrow> <msub> <mi>a</mi> <mrow> <mi>m</mi> <mo>,</mo> <mn>1</mn> </mrow> </msub> <mo>,</mo> <msub> <mi>a</mi> <mrow> <mi>m</mi> <mo>,</mo> <mn>2</mn> </mrow> </msub> <mo>,</mo> <mo>...</mo> <msub> <mi>a</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>n</mi> </mrow> </msub> </mrow> </mtd> </mtr> </mtable> </mfenced> </mrow>
<mrow> <mi>R</mi> <mo>=</mo> <mfenced open = "(" close = ")"> <mtable> <mtr> <mtd> <mrow> <msub> <mi>r</mi> <mrow> <mn>1</mn> <mo>,</mo> <mn>1</mn> </mrow> </msub> <mo>,</mo> <msub> <mi>r</mi> <mrow> <mn>1</mn> <mo>,</mo> <mn>2</mn> </mrow> </msub> <mo>,</mo> <mo>...</mo> <msub> <mi>r</mi> <mrow> <mn>1</mn> <mo>,</mo> <mi>n</mi> </mrow> </msub> </mrow> </mtd> </mtr> <mtr> <mtd> <mo>.</mo> </mtd> </mtr> <mtr> <mtd> <mo>.</mo> </mtd> </mtr> <mtr> <mtd> <mo>.</mo> </mtd> </mtr> <mtr> <mtd> <mrow> <msub> <mi>r</mi> <mrow> <mi>m</mi> <mo>,</mo> <mn>1</mn> </mrow> </msub> <mo>,</mo> <msub> <mi>r</mi> <mrow> <mi>m</mi> <mo>,</mo> <mn>2</mn> </mrow> </msub> <mo>,</mo> <mo>...</mo> <msub> <mi>r</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>n</mi> </mrow> </msub> </mrow> </mtd> </mtr> </mtable> </mfenced> </mrow>
wherein A is a attribution degree matrix, am,nRepresenting the attribution degree of the key frame m to the key frame n; r is an attraction matrix, Rm,nRepresenting the attraction of the key frame n to the key frame m;
updating by adopting the following formula (2) attraction degree matrix according to the similarity degree matrix,
<mrow> <msubsup> <mi>r</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>n</mi> </mrow> <mo>&prime;</mo> </msubsup> <mo>=</mo> <msub> <mi>s</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>n</mi> </mrow> </msub> <mo>-</mo> <munder> <mrow> <mi>m</mi> <mi>a</mi> <mi>x</mi> </mrow> <mrow> <mi>t</mi> <mo>&NotEqual;</mo> <mi>n</mi> </mrow> </munder> <mrow> <mo>(</mo> <msub> <mi>s</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>t</mi> </mrow> </msub> <mo>)</mo> </mrow> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>2</mn> <mo>)</mo> </mrow> </mrow>
wherein r'm,nAdopting the attraction degree of the key frame n to the key frame m after the similarity matrix is updated for the initial attribution degree and the initial attraction degree which are both 0;
updating the attribution degree matrix according to the following formulas (3) and (4) according to the updated attraction degree matrix,
wherein, a'm,nAnd a'm,mRespectively carrying out the attribution degree of the key frame m by the key frame n updated according to the updated attraction degree matrix and the autocorrelation attribution degree of the key frame m;
setting a damping factor lambda and an iteration number T, and performing iterative update on the attraction degree matrix and the attribution degree matrix by using the following formulas (5) and (6),
<mrow> <msubsup> <mi>r</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>n</mi> </mrow> <mrow> <mo>&prime;</mo> <mi>T</mi> </mrow> </msubsup> <mo>=</mo> <mi>&lambda;</mi> <mo>&times;</mo> <mo>&lsqb;</mo> <msub> <mi>s</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>n</mi> </mrow> </msub> <mo>-</mo> <munder> <mrow> <mi>m</mi> <mi>a</mi> <mi>x</mi> </mrow> <mrow> <mi>t</mi> <mo>&NotEqual;</mo> <mi>n</mi> </mrow> </munder> <mrow> <mo>(</mo> <msubsup> <mi>a</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>t</mi> </mrow> <mrow> <mo>&prime;</mo> <mi>T</mi> <mo>-</mo> <mn>1</mn> </mrow> </msubsup> <mo>+</mo> <msub> <mi>s</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>t</mi> </mrow> </msub> <mo>)</mo> </mrow> <mo>&rsqb;</mo> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>5</mn> <mo>)</mo> </mrow> </mrow>
wherein,For the attraction of the keyframe n to the keyframe m after being updated by T iterations,attribution degree of the key frame n to the key frame m after the iteration update for T times;
filtering the key frames with the sum of the autocorrelation attribution degree and the autocorrelation attraction degree not more than 0 after the iterative updating;
and taking another key frame corresponding to the maximum value of the sum of the attribution degree and the attraction degree of the key frame and other key frames which meet the condition that the autocorrelation attribution degree and the autocorrelation attraction degree are greater than 0 as the clustering center of the key frame, thereby obtaining the clustering center of each key frame which meets the condition, and clustering the key frames belonging to the same clustering center into one class.
5. A video encoding and decoding method, comprising:
the video scene similarity analysis method of any one of claims 1-4;
compressing the key frames grouped into one class as a GOP;
compressing the key frame, reconstructing the key frame at an encoding end, and placing the key frame in a frame buffer area, wherein the B frames and the P frames of the rest GOPs find the key frame corresponding to the frame buffer area through respective key frame indexes to perform inter-frame prediction encoding;
when decoding, all key frames are decoded first, and then B frames and P frames are decoded according to the clustering labels.
6. A video scene similarity analysis system, comprising:
the key frame extraction module is used for respectively selecting a frame of image from each shot of the video as a key frame of each shot and sending each key frame to the similarity matrix construction module;
the similarity matrix construction module is used for extracting the feature vector of each key frame, measuring the similarity between the key frames according to the feature vector, constructing a similarity matrix and sending the similarity matrix to the clustering module;
and the clustering module is used for clustering each key frame by taking the sum of the attraction degree and the attribution degree as a linear check according to the similarity matrix, setting a clustering label of each key frame, wherein the clustering label is represented by a clustering center corresponding to the key frame, the attraction degree represents information transmitted from the key frame to the clustering center, and the attribution degree represents information transmitted from the clustering center to the key frame.
7. The video scene similarity analysis system according to claim 6, wherein the key frame extraction module comprises:
the shot determining unit detects the boundaries of the shots and determines the interval between two adjacent boundaries as a shot;
the key frame extraction unit selects a frame at a set position in each lens as a key frame.
8. The video scene similarity analysis system according to claim 6, wherein the similarity matrix construction module comprises:
the accumulated histogram construction unit is used for blocking the image of the key frame to form sub-blocks with the same size to obtain a color histogram of the sub-blocks, accumulating to obtain the accumulated histogram of the key frame, and sending the accumulated histogram to the color feature vector extraction unit;
the color feature vector extraction unit is used for extracting the color feature vector of the cumulative histogram of each key frame by adopting a color space and sending the color feature vector of each key frame to the similarity matrix construction unit;
the texture feature vector extraction unit is used for extracting the texture feature vector of each key frame by adopting a gray level co-occurrence matrix and sending the texture feature vector of each key frame to the similarity matrix construction unit;
a similarity matrix construction unit for constructing a similarity matrix according to the following formula (1) using the color feature vector and texture feature vector of each key frame,
<mrow> <msub> <mi>s</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>n</mi> </mrow> </msub> <mo>=</mo> <mfrac> <mrow> <munderover> <mo>&Sigma;</mo> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>d</mi> </munderover> <msub> <mi>x</mi> <mi>i</mi> </msub> <msub> <mi>y</mi> <mi>i</mi> </msub> </mrow> <msqrt> <mrow> <mo>(</mo> <munderover> <mo>&Sigma;</mo> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>d</mi> </munderover> <msub> <msup> <mi>x</mi> <mn>2</mn> </msup> <mi>i</mi> </msub> <mo>)</mo> <mo>(</mo> <munderover> <mo>&Sigma;</mo> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>d</mi> </munderover> <msub> <msup> <mi>y</mi> <mn>2</mn> </msup> <mi>i</mi> </msub> <mo>)</mo> </mrow> </msqrt> </mfrac> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>1</mn> <mo>)</mo> </mrow> </mrow>
where m, n are two key frames, sm,nRepresenting the similarity of key frame m and key frame n, (x)1,x2,...,xd) Is the feature vector of the key frame m, (y)1,y2,...,yd) The keyframe n feature vectors.
9. The video scene similarity analysis system according to claim 6, wherein the clustering module comprises:
an initialization unit for initializing the similarity matrix, taking the average value of each similarity in the similarity matrix as the self-similarity of each key frame, and sending the initialized similarity matrix to a first updating unit,
<mrow> <mi>S</mi> <mo>=</mo> <mfenced open = "(" close = ")"> <mtable> <mtr> <mtd> <mrow> <msub> <mi>s</mi> <mrow> <mn>1</mn> <mo>,</mo> <mn>1</mn> </mrow> </msub> <mo>,</mo> <msub> <mi>s</mi> <mrow> <mn>1</mn> <mo>,</mo> <mn>2</mn> </mrow> </msub> <mo>,</mo> <mo>...</mo> <msub> <mi>s</mi> <mrow> <mn>1</mn> <mo>,</mo> <mi>n</mi> </mrow> </msub> </mrow> </mtd> </mtr> <mtr> <mtd> <mo>.</mo> </mtd> </mtr> <mtr> <mtd> <mo>.</mo> </mtd> </mtr> <mtr> <mtd> <mo>.</mo> </mtd> </mtr> <mtr> <mtd> <mrow> <msub> <mi>s</mi> <mrow> <mi>m</mi> <mo>,</mo> <mn>1</mn> </mrow> </msub> <mo>,</mo> <msub> <mi>s</mi> <mrow> <mi>m</mi> <mo>,</mo> <mn>2</mn> </mrow> </msub> <mo>,</mo> <mo>...</mo> <msub> <mi>s</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>n</mi> </mrow> </msub> </mrow> </mtd> </mtr> </mtable> </mfenced> </mrow>
wherein S is a similarity matrix, and the average value P of all similarities in the similarity matrix S is used as S1,1,s2,2,…,sm,mA value of (d);
a setting unit for setting the initial attribution degree and the initial attraction degree between the key frames in the attribution degree matrix and the attraction degree matrix to be 0, sending the initial attribution degree and the initial attraction degree to the first updating unit, setting the damping factor lambda and the iteration number T, sending the damping factor lambda and the iteration number T to the second updating unit, wherein,
<mrow> <mi>A</mi> <mo>=</mo> <mfenced open = "(" close = ")"> <mtable> <mtr> <mtd> <mrow> <msub> <mi>a</mi> <mrow> <mn>1</mn> <mo>,</mo> <mn>1</mn> </mrow> </msub> <mo>,</mo> <msub> <mi>a</mi> <mrow> <mn>1</mn> <mo>,</mo> <mn>2</mn> </mrow> </msub> <mo>,</mo> <mo>...</mo> <msub> <mi>a</mi> <mrow> <mn>1</mn> <mo>,</mo> <mi>n</mi> </mrow> </msub> </mrow> </mtd> </mtr> <mtr> <mtd> <mo>.</mo> </mtd> </mtr> <mtr> <mtd> <mo>.</mo> </mtd> </mtr> <mtr> <mtd> <mo>.</mo> </mtd> </mtr> <mtr> <mtd> <mrow> <msub> <mi>a</mi> <mrow> <mi>m</mi> <mo>,</mo> <mn>1</mn> </mrow> </msub> <mo>,</mo> <msub> <mi>a</mi> <mrow> <mi>m</mi> <mo>,</mo> <mn>2</mn> </mrow> </msub> <mo>,</mo> <mo>...</mo> <msub> <mi>a</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>n</mi> </mrow> </msub> </mrow> </mtd> </mtr> </mtable> </mfenced> </mrow>
<mrow> <mi>R</mi> <mo>=</mo> <mfenced open = "(" close = ")"> <mtable> <mtr> <mtd> <mrow> <msub> <mi>r</mi> <mrow> <mn>1</mn> <mo>,</mo> <mn>1</mn> </mrow> </msub> <mo>,</mo> <msub> <mi>r</mi> <mrow> <mn>1</mn> <mo>,</mo> <mn>2</mn> </mrow> </msub> <mo>,</mo> <mo>...</mo> <msub> <mi>r</mi> <mrow> <mn>1</mn> <mo>,</mo> <mi>n</mi> </mrow> </msub> </mrow> </mtd> </mtr> <mtr> <mtd> <mo>.</mo> </mtd> </mtr> <mtr> <mtd> <mo>.</mo> </mtd> </mtr> <mtr> <mtd> <mo>.</mo> </mtd> </mtr> <mtr> <mtd> <mrow> <msub> <mi>r</mi> <mrow> <mi>m</mi> <mo>,</mo> <mn>1</mn> </mrow> </msub> <mo>,</mo> <msub> <mi>r</mi> <mrow> <mi>m</mi> <mo>,</mo> <mn>2</mn> </mrow> </msub> <mo>,</mo> <mo>...</mo> <msub> <mi>r</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>n</mi> </mrow> </msub> </mrow> </mtd> </mtr> </mtable> </mfenced> </mrow>
wherein A is a attribution degree matrix, am,nRepresenting the attribution degree of the key frame m to the key frame n, R is an attraction degree matrix, Rm,nRepresenting the attraction of the key frame n to the key frame m;
a first updating unit for updating the attraction degree matrix according to the similarity matrix by using the following expression (2), updating the attribution degree matrix according to the following expressions (3) and (4) according to the updated attraction degree matrix, and sending the updated attraction degree matrix and the attribution degree matrix to a second updating unit,
<mrow> <msubsup> <mi>r</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>n</mi> </mrow> <mo>&prime;</mo> </msubsup> <mo>=</mo> <msub> <mi>s</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>n</mi> </mrow> </msub> <mo>-</mo> <munder> <mrow> <mi>m</mi> <mi>a</mi> <mi>x</mi> </mrow> <mrow> <mi>t</mi> <mo>&NotEqual;</mo> <mi>n</mi> </mrow> </munder> <mrow> <mo>(</mo> <msub> <mi>s</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>t</mi> </mrow> </msub> <mo>)</mo> </mrow> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>2</mn> <mo>)</mo> </mrow> </mrow>
wherein r'm,nThe attraction degree a 'of the key frame n to the key frame m updated by the similarity matrix is adopted for both the initial attribution degree and the initial attraction degree being 0'm,nAnd a'm,mRespectively carrying out the attribution degree of the key frame m by the key frame n updated according to the updated attraction degree matrix and the autocorrelation attribution degree of the key frame m;
a second updating unit which iteratively updates the attraction degree matrix and the attribution degree matrix updated by the first updating unit by using the following formulas (5) and (6) and sends the attraction degree matrix and the attribution degree matrix after T iterations to the screening unit, wherein,
<mrow> <msubsup> <mi>r</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>n</mi> </mrow> <mrow> <mo>&prime;</mo> <mi>T</mi> </mrow> </msubsup> <mo>=</mo> <mi>&lambda;</mi> <mo>&times;</mo> <mo>&lsqb;</mo> <msub> <mi>s</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>n</mi> </mrow> </msub> <mo>-</mo> <munder> <mrow> <mi>m</mi> <mi>a</mi> <mi>x</mi> </mrow> <mrow> <mi>t</mi> <mo>&NotEqual;</mo> <mi>n</mi> </mrow> </munder> <mrow> <mo>(</mo> <msubsup> <mi>a</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>t</mi> </mrow> <mrow> <mo>&prime;</mo> <mi>T</mi> <mo>-</mo> <mn>1</mn> </mrow> </msubsup> <mo>+</mo> <msub> <mi>s</mi> <mrow> <mi>m</mi> <mo>,</mo> <mi>t</mi> </mrow> </msub> <mo>)</mo> </mrow> <mo>&rsqb;</mo> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>5</mn> <mo>)</mo> </mrow> </mrow>
wherein,for the attraction of the keyframe n to the keyframe m after being updated by T iterations,attribution degree of the key frame n to the key frame m after the iteration update for T times;
the screening unit filters the key frames with the sum of the autocorrelation attribution degree and the autocorrelation attraction degree not greater than 0, and sends the attraction degree matrix and the attribution degree matrix of the vectors corresponding to the key frames which do not meet the conditions to the clustering unit;
and the clustering unit is used for taking another key frame corresponding to the maximum value of the sum of the attribution degree and the attraction degree of the key frame and other key frames in the attraction degree matrix and the attribution degree matrix filtered by the screening unit as a clustering center of the key frame, so that the clustering center of each key frame meeting the conditions is obtained, and the key frames belonging to the same clustering center are clustered into one class.
10. A video coding/decoding system, comprising:
the video scene similarity analysis system of any one of claims 6-9;
a compression unit for compressing the key frames grouped into one group as a GOP;
the encoding part is used for reconstructing the compressed key frames at an encoding end and placing the compressed key frames in a frame buffer area, and finding the key frames corresponding to the frame buffer area by the B frames and the P frames of the rest GOPs through respective key frame indexes to perform inter-frame prediction encoding;
and the decoding part decodes all the key frames and then decodes the B frames and the P frames according to the clustering labels.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201710873784.2A CN107657228B (en) | 2017-09-25 | 2017-09-25 | Video scene similarity analysis method and system, and video encoding and decoding method and system |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201710873784.2A CN107657228B (en) | 2017-09-25 | 2017-09-25 | Video scene similarity analysis method and system, and video encoding and decoding method and system |
Publications (2)
Publication Number | Publication Date |
---|---|
CN107657228A true CN107657228A (en) | 2018-02-02 |
CN107657228B CN107657228B (en) | 2020-08-04 |
Family
ID=61130165
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201710873784.2A Active CN107657228B (en) | 2017-09-25 | 2017-09-25 | Video scene similarity analysis method and system, and video encoding and decoding method and system |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN107657228B (en) |
Cited By (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108509917A (en) * | 2018-03-30 | 2018-09-07 | 北京影谱科技股份有限公司 | Video scene dividing method and device based on shot cluster correlation analysis |
CN109688429A (en) * | 2018-12-18 | 2019-04-26 | 广州励丰文化科技股份有限公司 | A kind of method for previewing and service equipment based on non-key video frame |
CN110062163A (en) * | 2019-04-22 | 2019-07-26 | 珠海格力电器股份有限公司 | Multimedia data processing method and device |
CN110188625A (en) * | 2019-05-13 | 2019-08-30 | 浙江大学 | A kind of video fine structure method based on multi-feature fusion |
CN110377587A (en) * | 2019-07-15 | 2019-10-25 | 腾讯科技(深圳)有限公司 | Method, apparatus, equipment and medium are determined based on the migrating data of machine learning |
CN110796088A (en) * | 2019-10-30 | 2020-02-14 | 行吟信息科技(上海)有限公司 | Video similarity determination method and device |
CN110853033A (en) * | 2019-11-22 | 2020-02-28 | 腾讯科技(深圳)有限公司 | Video detection method and device based on inter-frame similarity |
CN110879952A (en) * | 2018-09-06 | 2020-03-13 | 阿里巴巴集团控股有限公司 | Method and device for processing video frame sequence |
CN111126126A (en) * | 2019-10-21 | 2020-05-08 | 武汉大学 | Intelligent video strip splitting method based on graph convolution neural network |
WO2023050431A1 (en) * | 2021-09-30 | 2023-04-06 | 浙江大学 | Encoding method, decoding method, decoder, encoder and computer-readable storage medium |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101404030A (en) * | 2008-11-05 | 2009-04-08 | 中国科学院计算技术研究所 | Method and system for periodic structure fragment detection in video |
CN102572435A (en) * | 2012-01-16 | 2012-07-11 | 中南民族大学 | Compressive sampling-based (CS-based) video coding/decoding system and method thereof |
US20130114703A1 (en) * | 2005-03-31 | 2013-05-09 | Euclid Discoveries, Llc | Context Based Video Encoding and Decoding |
CN104732545A (en) * | 2015-04-02 | 2015-06-24 | 西安电子科技大学 | Texture image segmentation method combined with sparse neighbor propagation and rapid spectral clustering |
CN105469383A (en) * | 2014-12-30 | 2016-04-06 | 北京大学深圳研究生院 | Wireless capsule endoscopy redundant image screening method based on multi-feature fusion |
CN106202256A (en) * | 2016-06-29 | 2016-12-07 | 西安电子科技大学 | Propagate based on semanteme and mix the Web graph of multi-instance learning as search method |
CN107194312A (en) * | 2017-04-12 | 2017-09-22 | 广东银禧科技股份有限公司 | A kind of model recommendation method based on 3D printing model content |
-
2017
- 2017-09-25 CN CN201710873784.2A patent/CN107657228B/en active Active
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130114703A1 (en) * | 2005-03-31 | 2013-05-09 | Euclid Discoveries, Llc | Context Based Video Encoding and Decoding |
CN101404030A (en) * | 2008-11-05 | 2009-04-08 | 中国科学院计算技术研究所 | Method and system for periodic structure fragment detection in video |
CN102572435A (en) * | 2012-01-16 | 2012-07-11 | 中南民族大学 | Compressive sampling-based (CS-based) video coding/decoding system and method thereof |
CN105469383A (en) * | 2014-12-30 | 2016-04-06 | 北京大学深圳研究生院 | Wireless capsule endoscopy redundant image screening method based on multi-feature fusion |
CN104732545A (en) * | 2015-04-02 | 2015-06-24 | 西安电子科技大学 | Texture image segmentation method combined with sparse neighbor propagation and rapid spectral clustering |
CN106202256A (en) * | 2016-06-29 | 2016-12-07 | 西安电子科技大学 | Propagate based on semanteme and mix the Web graph of multi-instance learning as search method |
CN107194312A (en) * | 2017-04-12 | 2017-09-22 | 广东银禧科技股份有限公司 | A kind of model recommendation method based on 3D printing model content |
Non-Patent Citations (1)
Title |
---|
钱丽丽,施鹏飞: "近邻传播算法在非监督图像聚类中的应用", 《微型电脑应用》 * |
Cited By (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108509917A (en) * | 2018-03-30 | 2018-09-07 | 北京影谱科技股份有限公司 | Video scene dividing method and device based on shot cluster correlation analysis |
CN108509917B (en) * | 2018-03-30 | 2020-03-03 | 北京影谱科技股份有限公司 | Video scene segmentation method and device based on lens class correlation analysis |
CN110879952B (en) * | 2018-09-06 | 2023-06-16 | 阿里巴巴集团控股有限公司 | Video frame sequence processing method and device |
CN110879952A (en) * | 2018-09-06 | 2020-03-13 | 阿里巴巴集团控股有限公司 | Method and device for processing video frame sequence |
CN109688429A (en) * | 2018-12-18 | 2019-04-26 | 广州励丰文化科技股份有限公司 | A kind of method for previewing and service equipment based on non-key video frame |
CN110062163B (en) * | 2019-04-22 | 2020-10-20 | 珠海格力电器股份有限公司 | Multimedia data processing method and device |
CN110062163A (en) * | 2019-04-22 | 2019-07-26 | 珠海格力电器股份有限公司 | Multimedia data processing method and device |
CN110188625A (en) * | 2019-05-13 | 2019-08-30 | 浙江大学 | A kind of video fine structure method based on multi-feature fusion |
CN110188625B (en) * | 2019-05-13 | 2021-07-02 | 浙江大学 | Video fine structuring method based on multi-feature fusion |
CN110377587A (en) * | 2019-07-15 | 2019-10-25 | 腾讯科技(深圳)有限公司 | Method, apparatus, equipment and medium are determined based on the migrating data of machine learning |
CN111126126A (en) * | 2019-10-21 | 2020-05-08 | 武汉大学 | Intelligent video strip splitting method based on graph convolution neural network |
CN111126126B (en) * | 2019-10-21 | 2022-02-01 | 武汉大学 | Intelligent video strip splitting method based on graph convolution neural network |
CN110796088A (en) * | 2019-10-30 | 2020-02-14 | 行吟信息科技(上海)有限公司 | Video similarity determination method and device |
CN110853033A (en) * | 2019-11-22 | 2020-02-28 | 腾讯科技(深圳)有限公司 | Video detection method and device based on inter-frame similarity |
CN110853033B (en) * | 2019-11-22 | 2022-02-22 | 腾讯科技(深圳)有限公司 | Video detection method and device based on inter-frame similarity |
WO2023050431A1 (en) * | 2021-09-30 | 2023-04-06 | 浙江大学 | Encoding method, decoding method, decoder, encoder and computer-readable storage medium |
Also Published As
Publication number | Publication date |
---|---|
CN107657228B (en) | 2020-08-04 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN107657228B (en) | Video scene similarity analysis method and system, and video encoding and decoding method and system | |
CN108632625B (en) | Video coding method, video decoding method and related equipment | |
US6618507B1 (en) | Methods of feature extraction of video sequences | |
Ravi et al. | Compression noise based video forgery detection | |
JP2000217117A (en) | Processing method for digital image expression video data in compression form | |
CN108347612B (en) | Monitoring video compression and reconstruction method based on visual attention mechanism | |
EP2131594B1 (en) | Method and device for image compression | |
JP2002541738A (en) | Image compression | |
Wang et al. | Scalable facial image compression with deep feature reconstruction | |
KR102261669B1 (en) | Artificial Neural Network Based Object Region Detection Method, Device and Computer Program Thereof | |
CN109982071B (en) | HEVC (high efficiency video coding) dual-compression video detection method based on space-time complexity measurement and local prediction residual distribution | |
US7068720B2 (en) | Coding of digital video with high motion content | |
KR101149522B1 (en) | Apparatus and method for detecting scene change | |
Hong et al. | Detection of frame deletion in HEVC-Coded video in the compressed domain | |
CN103020138A (en) | Method and device for video retrieval | |
EP1089280A2 (en) | Method of indexing and searching moving picture data using a motion activity description method | |
CN103067713A (en) | Method and system of bitmap joint photographic experts group (JPEG) compression detection | |
CN108366295A (en) | Visual classification feature extracting method, transcoding weight contracting detection method and storage medium | |
Uddin et al. | Double compression detection in HEVC-coded video with the same coding parameters using picture partitioning information | |
CN108833928B (en) | Traffic monitoring video coding method | |
CN106331730B (en) | H.264 video same quantization factor double compression detection method | |
CN112770116B (en) | Method for extracting video key frame by using video compression coding information | |
KR101163774B1 (en) | Device and process for video compression | |
US20060109902A1 (en) | Compressed domain temporal segmentation of video sequences | |
Dai et al. | HEVC video steganalysis based on PU maps and multi-scale convolutional residual network |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |