CN102222101A - Method for video semantic mining - Google Patents

Abstract

The invention relates to a video semantic mining method. The method first performs Chinese continuous speech recognition, video target recognition and video text recognition on the video to be processed, and then performs Chinese word segmentation and part-of-speech tagging on the recognition results, and retains nouns and verbs as graph models. The vertices, the edge weights between the vertices are set to the Chinese semantic distance of the words represented by the two vertices, and finally the semantic information of the video is mined according to the dense subgraph discovery algorithm. The present invention is characterized in that it realizes the semantic mining of video through the fusion of three recognition results of Chinese continuous speech recognition, video target recognition and video text recognition; the video is expressed as a graph model, the vertices are the words in the video, and the weights of the edges are set is the semantic distance between two vertices; further transform the video semantic mining algorithm into a dense subgraph discovery algorithm of the graph model. The invention solves the problems of high error rate of single recognition results and ineffective fusion of multiple recognition results in the process of Chinese continuous speech recognition, video target recognition and video text recognition; it solves the problem of structured expression of video and the algorithm realization of video semantic mining question. The invention can be used for automatic labeling, classification and semantic mining of batch videos.

Description

A Video Semantic Mining Method

technical field

The invention relates to the fields of digital media and machine learning. It performs semantic analysis on video input by users, and performs semantic annotation on the video by fusing voice, text and image information.

Background technique

With the development of online video sharing sites and video processing technology, a large number of content in video formats has emerged. Because video is unformatted data and lacks necessary descriptive information, it cannot be processed as easily as text. Manual semantic annotation of videos is time-consuming and labor-intensive, and cannot meet the requirements of batch video processing. Content-based video processing technology is a current research hotspot, but the existing technology has a high error rate for video content labeling, and does not comprehensively consider the effective integration of image, text and voice content. At present, the image target recognition technology is gradually mature. In the visual target category classification challenge, the image target recognition has reached a practical level. Continuous speech recognition technology enables speech signals to be transcribed into text. Video text recognition can recognize the embedded text in the video and process it as text. Combining the above three recognition techniques, video semantic analysis requires an effective fusion method. "HowNet" is a Chinese semantic dictionary, which can calculate the semantic distance between two words by using the conceptual hierarchical relationship in "HowNet". According to the semantic distance, the semantic measurement of the three kinds of recognition results can be carried out. According to the high correlation of the three modal information of video image, text and voice, different modal information can be effectively fused to remove recognition error information. The graph model is composed of vertices and edges, which can express the relationship of concepts in the whole video. The dense subgraph discovery algorithm can realize the discovery of semantic aggregation relationship in the video graph model, and achieve the purpose of video semantic annotation.

Contents of the invention

The existing content-based video processing technology does not fully utilize the three high-level semantic information of image, voice and text, and cannot perform video classification and mining on high-level semantics. In order to solve the deficiencies of the existing technical problems, the present invention proposes a method for semantic mining of videos.

In order to achieve the stated purpose, the present invention provides a method for video expression and mining, and its technical solution includes the following steps:

Step S1: For the video to be processed, respectively perform Chinese continuous speech recognition, video object recognition and video text recognition;

Step S2: For the three kinds of recognition results described in step S1, each is expressed as a text vector, which together form a tensor to express the video;

Step S3: For the three text vectors in step S2, respectively perform Chinese word segmentation and part-of-speech tagging, and retain nouns and verbs;

Step S4: Construct a graph model to express the video, wherein the vertices of the graph are the nouns and verbs obtained in S3, and the edge weights of the graph are set to the semantic distance of the Chinese words represented by the two vertices;

Step S5: For the graph model constructed in step S4, use the dense subgraph discovery algorithm to mine the semantics in the graph model.

The beneficial effect of the present invention is that automatic semantic annotation, automatic classification and video similarity measurement can be realized for videos. For massive video data, with the help of this technology, the boring and tedious labor caused by manual labeling can be avoided. The present invention effectively integrates the results of Chinese continuous speech recognition, Chinese text recognition and image target recognition, and shows the semantic distance relationship of each semantic concept in the video by expressing the video as a graphical model. "Semantic distance measurement to achieve; Finally, the dense subgraph discovery algorithm can realize the labeling and mining of semantic concepts in videos.

Description of drawings

FIG. 1 is an overall flow chart of video processing in the present invention.

Fig. 2 is the Chinese continuous speech recognition flowchart of the present invention.

Fig. 3 is a flow chart of video text recognition in the present invention.

Fig. 4 is a flow chart of image target recognition in the present invention.

Fig. 5 is a hierarchical relationship diagram of semantic distance measurement in the present invention.

Figure 6 is a video dense subgraph mining representation of the present invention.

Fig. 7 is the video labeling result of the present invention.

Detailed ways

Various details involved in the technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings. It should be pointed out that the described embodiments are only intended to facilitate the understanding of the present invention, rather than limiting it in any way.

The present invention proposes a method for video semantic mining. As shown in FIG. 1 , the method is divided into four layers in the processing flow. The bottom layer is the video library layer, which stores various forms of video resources; the upper layer of the video library is the multi-modal fusion layer, where the structural analysis of the video and the recognition and effective fusion of images, text, and voice are completed; The next upper layer is the video mining layer, which realizes the graphical model representation of video and the video mining algorithm based on dense subgraph discovery. In addition, video classification and mining can be realized based on the support vector machine model; the top layer is provided to users. The transparent intelligent video service layer; the far right is the semantic computing support layer based on "HowNet". According to the above process, the specific implementation steps are as follows:

1. Video preprocessing

Carry out shot segmentation for the video to be processed, then extract key frames for each shot, and save these key frames for subsequent image target recognition; for the audio signal in the video, sample according to the requirements of 16KHZ, 16bit, and save as The wav format is used for subsequent speech recognition.

2. Image target recognition

First download the Visual Object Classification Challenge picture library (PASCAL VOC Challenge 2010 Database, http://pascallin.ecs.soton.ac.uk/challenges/VOC/voc2010/index.html), and extract the Local gradient features (HOG), using mean value clustering algorithm (k-means) to cluster the features in the picture library, the number of categories can be set to 1000, thus forming 1000 visual words, and then using these 1000 visual words Words describe each picture. At this time, each picture constitutes a bag of words (bag of words) as an intermediate feature. Finally, the support vector machine (SVM) method is used to train the classification model of 20 visual categories on the bag of words feature of the picture. The 20 categories are: people, birds, cats, cows, dogs, horses, sheep, airplanes, boats, bicycles, motorcycles, trains, cars, buses, bottles, chairs, dining tables, flower pots, sofas, TV sets . Finally, these classification models are used to perform target recognition on video key frame images, and the recognition results are saved as a text, which is recorded as Text _OBJECT . Refer to Figure 2 for the processing flow of this step.

3. Chinese continuous speech recognition

First download the open source code of the Sphinx continuous speech recognition system and the supporting Chinese language model, Chinese acoustic model and Chinese vocabulary files (Sphinx, http://sphinx.sourceforge.net). Continuous speech recognition is performed on the audio signal obtained by video preprocessing, and the audio signal is transcribed into text, which is recorded as Text _ASR . Refer to Figure 3 for the processing flow of this step.

1. Video text recognition

For text positioning in video images, firstly, candidate text regions are obtained based on the double-edge model of character strokes, and then the candidate text regions are decomposed to obtain precisely positioned text blocks. The text extraction algorithm in the video takes a frame every several video frames to perform image-based text positioning to obtain the text object, then tracks the text object forward and backward in the video frame sequence, and finally recognizes the text object to obtain the text Extract the result, and save the recognition result as a text, denoted as Text _VOCR . Refer to Figure 4 for the processing flow of this step.

其中Ψ表示视频的语义张量特征，

表示由三个向量Word_OBJECT、Word_ASR和Word_VOCR形成的张量空间。5. For Text _OBJECT , Text _ASR , and Text _VOCR , perform Chinese word segmentation based on the Hidden Marko Large Model, remove meaningless "stop words" and Chinese part-of-speech tagging, and retain verbs and nouns for the next step of analysis. After processing, respectively Recorded as Word _OBJECT , Word _ASR and Word _VOCR . Then the entire video can be semantically expressed as a tensor, namely

where Ψ represents the semantic tensor feature of the video,

Represents the tensor space formed by the three vectors Word _OBJECT , Word _ASR , and Word _VOCR .

6. For the nouns and verbs obtained in the previous step, calculate the semantic similarity between two words with the same part of speech. The calculation method adopts the hierarchical distance measurement method based on "HowNet", and the similarity is defined between 0 and 1. For example, the similarity between a table and a chair is 0.8, and the similarity between a landscape and a ship is 0.1. Refer to Figure 5 for the processing flow of this step.

有一个非负的权重该权重的定义就是上一步中确定的词语之间的语义相似度。7. Take the words in Word _OBJECT , Word _ASR and Word _VOCR as the vertex V of the graph model, define the similarity between the words defined in the previous step as the weight w of the edge between the vertices, and construct a graph model to express the video. This is a weighted undirected graph, G=(V, E), V represents the set of vertices, E represents the set of edges, |V|=n represents the number of vertices, each edge

has a non-negative weight The definition of this weight is the semantic similarity between words determined in the previous step.

其中

表示两个词语，其中m，n∈{Word_OBJECT，Word_ASR，Word_VOCR}，表示的视频张量中的一个向量；0≤i≤|m|，0≤j≤|n|，表示的词语编号，该编号最大为视频张量中一个向量的最大维数。f(·)表示的词语的相似度距离值，定义在0和1之间。8. Since the image, text and audio in the video jointly express a theme, they are consistent in semantics. Therefore, the semantic similarity of the three vectors of the video tensor should be minimized, and the words that do not meet the minimization principle are caused by recognition errors and should be removed. This principle can be expressed as:

in

Represents two words, where m, n∈{Word _OBJECT , Word _ASR , Word _VOCR }, represents a vector in the video tensor; 0≤i≤|m|, 0≤j≤|n|, represents the word number, This number is up to the largest dimension of a vector in the video tensor. The similarity distance value of words represented by f(·), defined between 0 and 1.

即子图中各个边的平均权重之和最大化，其中|X|表示子图顶点个数，

表示子图边的集合，w(l)表示边的权重，边的权重计算方法同上一步中的f(·)。视频语义特征张量空间包含三个向量，即

每个向量构造一个图模型社区(community)，从而整个视频表达为由三个社区组成的一个图模型。稠密子图发现算法在上述图模型上进行，如图6所示。9. For the above optimization problem, it is transformed into a dense subgraph discovery problem of the graphical model. That is, in G=(V, E), find the subgraph H=(X, F), H is the subgraph, x is the vertex set of the subgraph, and F is the edge set of the subgraph. The dense subgraph discovery algorithm can be expressed as

That is, the sum of the average weights of each edge in the subgraph is maximized, where |X| represents the number of subgraph vertices,

Represents the set of subgraph edges, w(l) represents the weight of the edge, and the calculation method of the weight of the edge is the same as f(·) in the previous step. The video semantic feature tensor space contains three vectors, namely

Each vector constructs a graph model community (community), so that the whole video is expressed as a graph model composed of three communities. The dense subgraph discovery algorithm is performed on the above graph model, as shown in Figure 6.

10. For the dense subgraph in the graphical model found in the previous step, record the words represented by the vertices in the dense subgraph as effective annotations of the video, which embodies the semantic information of the video. The video annotation results are shown in Figure 7.

Claims (5)

1. A video semantic mining method is characterized in that the steps of the method are as follows: Step S1: For the video to be processed, respectively perform Chinese continuous speech recognition, video object recognition and video text recognition; Step S2: For the three kinds of recognition results described in step S1, each is expressed as a text vector, which together form a tensor to express the video; Step S3: For the three text vectors in step S2, respectively perform Chinese word segmentation and part-of-speech tagging, and retain nouns and verbs; Step S4: Construct a graph model to express the video, wherein the vertices of the graph are the nouns and verbs obtained in S3, and the edge weights of the graph are set to the semantic distance of the Chinese words represented by the two vertices; Step S5: For the graph model constructed in step S4, use the dense subgraph discovery algorithm to mine the semantics in the graph model. 2. video semantics mining method according to claim 1, is characterized in that, described video object recognition first extracts the gradient feature (HOG) of picture and scale difference on Visual Object Classification Challenge picture storehouse (PASCAL VOC Challenge 2010). Variable features (SIFT), and use the mean value clustering (K-means) algorithm clustering for these features, these classes are called visual words, and then use these visual words to construct a bag of words (bag of words) for the pictures in the picture library To describe, use the bag of words as the image feature to train the support vector machine model (SVM), and use the support vector machine model to perform target recognition on the key frame image of the video lens. 3. video semantic mining method according to claim 1, is characterized in that, for the processing of video, has merged Chinese continuous speech recognition, video object recognition and video character recognition, and three kinds of recognition results are unified as character feature and are processed, Word processing includes Chinese word segmentation and part-of-speech tagging. 4. video semantic mining method according to claim 1, is characterized in that, for the structure of graph model, apex represents noun and verb in three kinds of recognition results of video, and the weight of edge represents the semantic distance between apex, The calculation of edge weight is based on the semantic measurement method of "HowNet", and the semantic distance between two words is calculated by querying the hierarchy and affiliation relationship between words in the semantic dictionary of "HowNet". 5. The video semantic mining method according to claim 1, wherein the discovery algorithm of the dense subgraph is realized by constantly removing isolated vertices in the graph model, and the mining process of the video semantics is expressed as a dense subgraph in the graph model. Subgraph discovery problem. the

Images