TWI722327B - Audio-visual content and user interaction sequence analysis system and method - Google Patents

Abstract

The present invention provides a system and method for analyzing audio-visual content and user interaction sequence, which includes user-defined interactive components, user analysis modules, audio-visual content analysis modules, message analysis modules, time-series interactive modules, and integrated multi-sequence analysis A module that integrates a multi-sequence analysis module to collect the user grouping data, image object sequence, audio sequence, message sequence, and time-series interaction sequence provided by the aforementioned module, to generate any interaction time point and interaction on the multimedia information timeline Quantified results of factor-related learning behavior. And to generate a concentration assessment, predict the current point of time when the user's concentration in the learning process is reduced and improve, so that the user can really watch the content of the multimedia information and increase the interest in learning.

Description

Audio-visual content and user interaction sequence analysis system and method

The present invention is a learning system and method, in particular, a system and method that achieves adaptive evaluation of learning information and analysis of user interaction sequence by collecting user interaction information.

With the advancement of the Internet and personal electronic equipment, the old teaching methods such as cram school or home tutoring have gradually been replaced by multimedia teaching. Among them, in the field of personal training or corporate education and training, online multimedia teaching is not limited to a fixed teaching time, not limited to a fixed teaching location, and the ability to stop at any time, replay, fast forward and other features are also the main reasons for its popularity.

Existing multimedia teaching often sends learning information from the teaching system to the user. However, the real participation and learning effectiveness of the user cannot be measured or received, and it is impossible to rely on the user’s learning. The degree is adjusted to the appropriate learning content, resulting in a unidirectional transmission of learning information. Although data such as the number of times and hours of playing learning information are currently used as a measure of learning effectiveness, it is still impossible to obtain feedback from users to further adjust appropriate learning content. Taking corporate education and training as an example, users usually use idle computers to broadcast learning information to lengthen the viewing time, but there is no real response to participating in learning or making learning behaviors. In addition, learning effectiveness and measurement. Under the learning information of different fields, types and even different lecturers, the meaning of user interaction is different, so there must be a weighted measurement. Calculation.

It can be seen that how to use the user's learning history and collect and analyze the feedback information in the learning process to achieve the appropriateness of the learning information is the focus that all parties urgently want to improve.

The present invention provides a sequence analysis system for audio-visual content and user interaction, which mainly includes user-defined interactive components, user analysis modules, audio-visual content analysis modules, message analysis modules, time-series interactive modules, and integrated multi-sequence analysis modules . The user-defined interactive elements are set in the multimedia information interface. There are multiple interactive factors, such as questions, answer questions, notes, key marks, emoticons, fast forward, reverse, etc. Each interactive element provides input interactive data. The user analysis module calculates the distance and grouping the historical learning behavior data of all users in the database based on the historical learning behavior data, and classifies the current user into the corresponding user group, and generates related results of the grouping User grouping data. The audio-visual content analysis module inputs the objects in the image data belonging to the multimedia information to mark, generate a sequence of image objects related to each time point in the multimedia information, and perform audio analysis on the audio data belonging to the input multimedia information, and Calculate the pitch of each sound frame, and generate an audio sequence related to each time point in the multimedia information. The message analysis module uses the text extracted from the database to classify the interactive data according to the purpose of the interactive data, and generates a message sequence related to the purpose of the interactive data. The time-series interactive module inputs the interactive data into the time and the corresponding information in the multimedia information timeline. The time point is connected to generate an interactive time point, and then the interactive time points in the multimedia information timeline are combined to generate a time-series interactive sequence. Integrate multi-sequence analysis module to collect the aforementioned user grouping data, image object sequence, and audio Sequences, message sequences, and time-series interactive sequences are generated at any interactive time point on the multimedia information timeline and the quantitative results of learning behaviors related to the interactive factors.

Among them, the weight of the interaction factor is further set, and the quantified results of learning behavior corresponding to each user's grouping and interaction factor are generated by integrating the association between the multi-sequence analysis module and the user grouping data.

The present invention provides a method for analyzing audio-visual content and user interaction sequence. The steps include: inputting interactive data in the interaction factor in the user-defined interactive element set in the multimedia information interface; and compiling the history of all users in the database Learning behavior data. After calculating the distance and grouping based on the historical learning behavior data by the user analysis module, the current user is classified into corresponding user groups, and the user grouping data is generated; the image data in the multimedia information, Mark the objects in the image data through the audio-visual content analysis module to generate a sequence of image objects related to each time point in the multimedia information; the audio data belonging to the multimedia information is calculated through the audio-visual content analysis module for each audio frame of the audio data Pitch audio analysis generates audio sequences related to each time point in multimedia information; extracts text in the database, and classifies interactive data through the message analysis module to generate a message sequence related to the purpose of the interactive data; The input time of each interactive data and the corresponding time points in the multimedia information are generated through the time-series interactive module to generate the interaction time points, and then the interactive time points in the multimedia information timeline are combined to generate a time-series interactive sequence; and the user is grouped into data and image objects Sequence, audio sequence, message sequence and timing interaction sequence Through the integration of multi-sequence analysis modules and interactive data associations, the quantitative results of learning behaviors generated at any point in the multimedia information.

Among them, the present invention provides a time-series interaction module in a method for analyzing audio-visual content and user interaction sequence to collect the quantified results of each interaction time point on the multimedia information time axis and corresponding learning behaviors to generate a long- and short-term memory model.

Among them, the current users are compared with the long-term short-term memory model, and the current users are classified into groups of users with similar learning types in the long-term short-term memory model based on the quantitative results of learning behavior.

It uses the long- and short-term memory model to determine the user group to which the current user belongs, and based on the distribution of each interaction time point on the time axis of the user grouping in the multimedia information and the quantified results of the corresponding learning behavior at each interaction time point. Assess concentration index.

Among them, the long and short-term memory model is used to determine the user group to which the current user belongs, and the interaction factor on the user-defined interactive component is activated according to the time interval of the user group's lower concentration index on the time axis in the multimedia information , Provide input interactive information.

The above detailed description is a specific description of a possible embodiment of the present invention, but this embodiment is not intended to limit the scope of the patent of the present invention. Any equivalent implementation or modification that does not deviate from the technical spirit of the present invention should be included in In the scope of the patent in this case.

To sum up, this case is not only innovative in terms of spatial form, but also can enhance the above-mentioned multiple functions compared with conventional articles. It should fully meet the requirements of novel and progressive statutory invention patents. An application is filed in accordance with the law, and you are kindly requested to approve it. This invention patent application is to encourage invention, and it is easy to feel the virtue.

100‧‧‧User-defined interactive components

110‧‧‧Interaction factor

200‧‧‧User Analysis Module

300‧‧‧Message Analysis Module

400‧‧‧Video and audio content analysis module

500‧‧‧Timing Interactive Module

600‧‧‧Integrated multi-sequence analysis module

S301~S308‧‧‧Step Process

Figure 1 is a schematic diagram of a user-defined interactive component of the present invention.

Fig. 2 is a schematic diagram of the system for analyzing the sequence of video and audio content and user interaction of the present invention.

Fig. 3 is a flow chart of the method for analyzing the sequence of video and audio content and user interaction of the present invention.

Figure 4 is a schematic diagram of the long short-term memory model of the present invention.

Figure 5 is a schematic diagram of the concentration prediction and incentive mechanism of the present invention.

In order to facilitate the reviewers to understand the technical features, content and advantages of the present invention and its achievable effects, the present invention is described in detail with the accompanying drawings and in the form of embodiment expressions as follows, and the diagrams used therein are as follows: The subject matter is only for the purpose of illustration and auxiliary description, and may not be the true proportions and precise configuration after the implementation of the invention. Therefore, it should not be interpreted in terms of the proportions and configuration relationships of the accompanying drawings, and should not limit the scope of rights of the invention in actual implementation. Hexian stated.

Please refer to FIG. 1, which is a schematic diagram of the user-defined interactive element of the present invention. As shown in the figure, the user-defined interactive element 100 is set in a software interface or a multimedia information interface, and the user-defined interactive element 100 has multiple interactions. The factor 110 provides input interactive data. The interaction factor 110 includes asking questions, answering questions, notes, key marks, emoticons, fast forwarding, rewinding, etc. When the user turns on the interactive factor 110, you can open the dialog box or Type text content, symbols, mouse clicks, etc. in the fields as input interactive data. The system administrator can adjust the weight of the interaction factor 100 and the meaning of interaction, as shown in Figure 1: cooperative behavior 30%, communication behavior 40%, and conflict behavior 30%.

訊息分析模組300利用資料庫中提取對應之文本，針對當下使用者所輸入之互動資料進行目的分類，如使用意圖分類(Intent classification)方法，並產生訊息序列，其中意圖分類是以使用者輸入的文字訊息，進一步判斷使用者輸入該文字訊息的目的是什麼，如於資料庫中事先定義多項類別，例如：飛機、火車、捷運等類別的交通工具後，當使用長短期記憶模型(long-short term memory，LSTM)時，則依據使用者所輸入之文字訊息的內容，判斷輸出使用者最有可能需要搭乘的交通工具類別，例如：使用者輸 入“我想從台灣去日本”，則LSTM會輸出“飛機”，藉此意圖分析以判斷訊息序列之目的內容，如使用者的提問、抱怨、閒聊。影音內容分析模組400將輸入之多媒體資訊中所屬的影像資料內之物件作標記，如辨識影像資料內之物件為何物，產生與多媒體資訊中各時間點相關的該影像物件序列，以及將輸入之多媒體資訊中所屬的音訊資料進行音頻分析，如先將整個音訊資料切成多個音框，再以基音檢測算法(Harmonic product spectrum method)對原始音訊做多次向下取樣(Down sampling)，將取樣後對原音訊的壓縮的音訊合併，以凸顯基頻的高點，並計算各音框的音高、排除不穩定的音高及平滑化，產生與多媒體資訊中各時間點相關的音訊序列，如各時間點之音高及頻率。時序互動模組500將輸入互動資料之時間點，與輸入互動資料之時多媒體資訊時間軸中所對應的時間點作為一互動時間點，並產生包含了多個互動時間點的時序互動序列。整合多序列分析模組600將上述使用者分群資料、影像物件序列、音訊序列、訊息序列及時序互動序列結合，產生一學習行為量化結果，由此學習行為量化結果可得知，在多媒體資訊時間軸中任一互動時間點上，當下使用者及其所述的使用者分群中之使用者，與多媒體資訊中的影像或聲音產生之互動結果。亦可由多媒體資訊時間軸中上互動時間點之分布，以建立一長短期記憶模型。 Please refer to FIG. 2, which is a schematic diagram of the video content and user interaction sequence analysis system of the present invention, which includes a user analysis module 200, a message analysis module 300, a video content analysis module 400, a time sequence interaction module 500, and integrated multiple Sequence analysis module 600. The user analysis module 200 collects the historical learning behavior data of all users in the database, such as using data clustering techniques, such as K-means and Hierarchical clustering, according to history The learning behavior data is calculated by distance and grouped into groups, and the current users are classified into corresponding user groups. The result of this grouping is used as the user grouping data, where the user grouping data is related to the user's past learning history or use of the platform Statistical information items of, such as: correct answer rate, average viewing time, number of friends, etc. Take the K-average algorithm as an example. Assuming that the user’s question answer rate and the average viewing time are used as the variables for grouping, the relationship can be expressed in a two-dimensional coordinate system. First, k is selected randomly. Calculate the distance between each data point χ _j and each center μ _i (using Euclidean distance), and use the center with the closest distance to each point as the grouping, and then recalculate a new center point S according to the grouping result, with the following Formula, continue to iterate this process to get the minimum value J:

The message analysis module 300 extracts the corresponding text from the database, and classifies the interactive data currently input by the user, such as using the intent classification method, and generates a message sequence, where the intent classification is input by the user To further determine the purpose of the user’s input of the text message. For example, after defining multiple categories in the database in advance, such as airplanes, trains, and MRT vehicles, when using the long and short-term memory model (long -short term memory, LSTM), based on the content of the text message entered by the user, determine the type of transportation that the user is most likely to take. For example: the user enters "I want to go from Taiwan to Japan", then LSTM will output "aircraft" to analyze the intention to determine the purpose of the message sequence, such as user questions, complaints, and small chats. The audio-visual content analysis module 400 marks the object in the image data belonging to the input multimedia information, such as identifying the object in the image data, generating the sequence of the image object related to each time point in the multimedia information, and inputting Audio analysis is performed on the audio data belonging to the multimedia information. For example, the entire audio data is cut into multiple sound frames, and then the original audio is down-sampled multiple times by the Harmonic product spectrum method. Combine the compressed audio of the original audio after sampling to highlight the high point of the fundamental frequency, calculate the pitch of each sound frame, eliminate unstable pitch and smooth, and generate audio related to each time point in the multimedia information Sequence, such as pitch and frequency at each point in time. The time-series interaction module 500 regards the time point at which the interactive data is input, and the time point corresponding to the time axis of the multimedia information when the interactive data is input as an interaction time point, and generates a time-series interaction sequence including multiple interaction time points. The integrated multi-sequence analysis module 600 combines the above-mentioned user grouping data, image object sequence, audio sequence, message sequence, and time-series interaction sequence to generate a learning behavior quantification result. From the learning behavior quantification result, it can be known that in the multimedia information time At any interaction time point in the axis, the interaction result of the current user and the users in the said user group with the video or sound in the multimedia information. It is also possible to establish a long and short-term memory model based on the distribution of interactive time points in the multimedia information timeline.

Please refer to FIG. 3, which is a flowchart of a method for analyzing audio-visual content and user interaction sequence of the present invention. The steps are as follows: S301: the user plays multimedia information, wherein a user-defined interactive element is provided on the multimedia information interface; S302: The user inputs interactive data in the interactive factors in the user-defined interactive component; S303: The user analysis module calculates the distance and grouping based on the historical learning behavior data of all users in the database After that, group the corresponding users into which the current user is classified, and use the grouping result as the user grouping data; S304: use the video data in the multimedia information to use the video content analysis module to make the objects in the video data Mark, generate a sequence of image objects related to each time point in the multimedia information; S305: Audio data belonging to the multimedia information, calculate the audio analysis of the pitch of each audio frame of the audio data through the audio-visual content analysis module, and generate the multimedia information S306: extract the text in the database, and the message analysis module classifies the input interactive data to generate the message sequence related to the purpose of the interactive data; S307: interact The data input time point and the time point corresponding to the multimedia information time axis are used as the interaction time point, and multiple interaction time points on the time axis are collected to generate a time-series interaction sequence; and S308: integrated multi-sequence analysis module to group the above-mentioned user data, The image object sequence, audio sequence, message sequence, and time-series interactive sequence are combined to generate a quantitative result of the corresponding learning behavior at the interactive time point in the multimedia information timeline. When the user turns on the interaction factor in the user-defined interactive element again, the steps of S302 to S308 are repeated to generate a quantitative result of the learning behavior corresponding to the interactive time point in the multimedia information timeline.

The steps of S302 to S307 mentioned above are executed after receiving the interaction data entered by the current user in the interaction factor, and the execution sequence is not limited to this.

Building a long and short-term memory model

As shown in Figure 4, the time-series interactive module divides the multimedia information on the time axis. The interaction time point and the quantitative results of each learning behavior are collected and a long- and short-term memory model is collected and constructed. In other words, user grouping data, image object sequence, audio sequence, message sequence, and time-series interaction sequence are used as input parameters to create A long-term and short-term memory model capable of achieving self-learning, concentration assessment and adaptability.

Self-learning of long and short-term memory models

The long and short-term memory model compares the quantified learning behavior input by the current user with the long-term short-term memory model, and classifies the current users into groups of users with similar learning behaviors in the long-term short-term memory model based on the quantified learning behavior results In order to learn by yourself.

This long and short-term memory model can be learned as the amount of learning behavior data of users in the database increases, and the current users can be classified into corresponding user groups in real time to make subsequent predictions.

Concentration assessment of long-term short-term memory model

Through the long and short-term memory model, determine the user group to which the current user belongs, and evaluate the concentration based on the distribution of each interaction time point of the user group on the time axis of the multimedia information and the quantitative results of learning behavior at each interaction time point Power index.

With the current user or user grouping, the change of the concentration index and the time axis in the progress of multimedia information can provide further predictions of possible interactive behaviors or serve as a basis for system administrators to adjust the content of multimedia information.

Concentration prediction and incentive mechanism of long and short-term memory models

As shown in Figure 5, through the long and short-term memory model, determine the user group to which the current user belongs, and activate the user based on the user group in the time interval with a low concentration index on the time axis in the multimedia information Customize the interaction factor on the interactive component to improve It is used to input interactive data, and increase the user's motivation to input interactive data by increasing the weight of the interaction factor, and improve the quantitative results of learning behavior.

By using the focus evaluation of the long and short-term memory model to replace the existing viewing hours as the basis for evaluating learning effectiveness, it can predict the current user’s time point in the multimedia information at which the focus index may decrease, and at that time Click to activate the interactive factor to provide incentives, points and other incentives to obtain interactive feedback from current users and increase attention, effectively improve the problem of playing learning information on idle computers, and enable current users to truly watch the content of multimedia information, and Enhance the interest of current users.

To sum up, this case is not only innovative in terms of technical ideas, but also has the above-mentioned multiple functions that are not available in conventional existing methods. It has fully met the requirements of novel and progressive statutory invention patents. I file an application in accordance with the law. I implore your office. Approval of this invention patent application to encourage invention, so that it will be convenient.

100‧‧‧User-defined interactive components

110‧‧‧Interaction factor

200‧‧‧User Analysis Module

300‧‧‧Message Analysis Module

400‧‧‧Video and audio content analysis module

500‧‧‧Timing Interactive Module

600‧‧‧Integrated multi-sequence analysis module

Claims (10)

A video and audio content and user interaction sequence analysis system, comprising: an integrated multi-sequence analysis module, the integrated multi-sequence analysis module uses a user grouping data, a video object sequence, an audio sequence, a message sequence and A time-series interaction sequence that generates a quantitative result of a learning behavior related to an interaction factor at any interaction time point of the current user on the multimedia information timeline; a user-defined interactive element, the user-defined interactive element is set in the multimedia The information interface includes a plurality of the interactive factors, and each of the interactive factors provides input interactive data; a user analysis module that compiles the historical learning behavior data of all users in a database, Calculate the distance and group according to historical learning behavior data, and classify the current user into the corresponding user group to generate the user grouping data related to the grouping result; a video content analysis module, the video content analysis module Mark the objects in the image data belonging to the multimedia information to generate a sequence of the image objects related to each time point in the multimedia information, and the audio-visual content analysis module performs audio analysis on the audio data belonging to the multimedia information, And calculate the pitch of each sound frame to generate the audio sequence related to each time point in the multimedia information; a message analysis module, the message analysis module uses the text extracted from the database for the purpose of interactive data Classification to generate the message sequence related to the purpose of the interactive data; a time-series interactive module that relates the input time of the interactive data and the corresponding time point in the multimedia information timeline and generates the interactive time point, and then Combine the interactive time points in the multimedia information time axis to generate the time-series interactive sequence. For example, the audio-visual content and user interaction sequence analysis system described in item 1 of the scope of patent application, wherein the interaction factor includes asking questions, answering questions, notes, key marks, emoticons, fast forwarding, and rewinding. For example, the audio-visual content and user interaction sequence analysis system described in item 1 of the scope of patent application, wherein the weight of the interaction factor is further set, and the association between the integrated multi-sequence analysis module and the user grouping data is generated and used The quantified results of the learning behaviors corresponding to the interaction factor. A method for analyzing audio-visual content and user interaction sequence. The steps include: inputting interactive data in an interactive factor on a user-defined interactive element of a multimedia information interface; learning the history of all users in a database Behavior data. After calculating the distance and grouping based on historical learning behavior data by a user analysis module, the current user is classified into corresponding user groups, and a user grouping data is generated; the image data belonging to the multimedia information , Through an audio-visual content analysis module to mark the objects in the image data to generate an image object sequence related to each time point in the multimedia information; the audio data belonging to the multimedia information is calculated through the audio-visual content analysis module The audio analysis of the pitch of each sound frame generates an audio sequence related to each time point in the multimedia information; extracts the text in the database, and classifies the interactive data through a message analysis module to generate and interact data A message sequence related to the purpose of each interactive data; the input time of each interactive data and the corresponding time point in the multimedia information are generated through a time-series interactive module, and then the interactive time points in the multimedia information timeline are combined to produce Generate a time-series interactive sequence; and associate the user grouping data, the image object sequence, the audio sequence, the message sequence, and the time-series interactive sequence with the interactive data through an integrated multi-sequence analysis module, and generate any information in the multimedia information. A quantitative result of learning behavior at one point in time. For example, the method for analyzing the video content and user interaction sequence described in item 4 of the scope of patent application, wherein the interaction factor includes asking questions, answering questions, notes, key marks, emoticons, fast forwarding, and rewinding. For example, the method for analyzing audio-visual content and user interaction sequence described in item 4 of the scope of patent application, wherein the weight of the interaction factor is further set, and the association between the integrated multi-sequence analysis module and the user grouping data is used to generate and use The quantified results of the learning behaviors corresponding to the interaction factor. For example, the method for analyzing audio-visual content and user interaction sequence described in item 4 of the scope of patent application, wherein the time-series interaction module collects each interaction time point on the multimedia information time axis and the corresponding learning behavior quantification result to generate a long and short-term memory model. For example, the method for analyzing video content and user interaction sequence described in item 7 of the scope of patent application, in which the current user is compared with the long and short-term memory model, and the current user is classified into the long and short-term memory model based on the quantitative results of learning behavior Groups of users with similar learning styles. For example, the method for analyzing the video content and the user interaction sequence described in item 8 of the scope of patent application, which uses the long and short-term memory model to determine the user group to which the current user belongs, and based on the user group on the time axis in the multimedia information The distribution of each interaction time point and the quantitative results of the corresponding learning behavior of each interaction time point are evaluated as a concentration index. As described in item 9 of the scope of patent application, the method of analyzing the sequence of interaction between audio and video content and users, wherein Through the long and short-term memory model, determine the user group to which the current user belongs, and activate the user-defined interactive component based on the user grouping in the time interval of the time axis in the multimedia information when the concentration index is low The interactive factor provides input interactive data.

Images