TWI741550B - Method for bookmark frame generation, and video player device with automatic generation of bookmark and user interface thereof - Google Patents

Abstract

The present invention discloses a method for bookmark frame generation, a video player device with automatic generation bookmarks, and a user interface thereof. Based on the concept that "The presentation of important frames should allow enough time for careful reading", the method for bookmark frame generation includes: extracting a plurality of frames ordered by time from a movie; calculating the structural similarity between two adjacent frames in the frames; when the structural similarity of a first frame in the frames with an adjacent second frames is less than a preset value, and the structural similarity of all the frames corresponding to the first frame and within a preset time is greater than or equal to the preset value, the second frame is designated as a bookmark frame; and storing the bookmark frame and its occurrence time.

Description

Method for generating bookmark frame, video and audio playback device for automatically generating bookmark, and user interface thereof

The invention relates to a method for generating a bookmark frame, a video and audio playback device for automatically generating a bookmark, and a user interface thereof.

In recent years, digital learning has become more and more common in universities and colleges. The situation of class can be restored through video, and more detailed supplementary information can be added to help students understand the content of class. Digital learning can no longer be restricted by space and time, helping students to absorb knowledge at their own pace. Although there are still shortcomings in digital learning, it is clear that digital learning is the future learning trend.

Among the current teaching methods of digital learning, teaching videos are the largest. However, one problem will be encountered when watching videos, that is, it is very troublesome to review. A few teaching videos will sort out the hyperlinks (bookmarks) of each segment, but This function is often carried out manually, which is a time-consuming and laborious task, so normal teaching videos do not provide this function. Without this function, it will cause students to have a lot of trouble in reviewing, because the length of teaching videos is often very long, but the content that students want to review may only take one or two minutes, so there will be Kind of the feeling of finding a needle in a haystack.

In addition, audio-visual analysis technology has made great progress. It can analyze various data in the video, so that users can have better control of the content in the video, but today's video playback platform (such as YouTube or BiliBili) does not Significant progress. Players on most platforms still only have basic functions such as timeline, fast forward, rewind, and pause, and have not played the potential of audio-visual analysis technology.

The purpose of the present invention is to provide a method for generating a bookmark frame, an audio-visual playback device for automatically generating bookmarks, and a user interface thereof. Through the bookmark frame of the movie, the user can quickly jump to the location of the movie to be queried, so that the user can quickly Find the desired video content. In addition, the present invention allows users to browse videos in a smarter way, saving time for searching and viewing video clips.

Based on the core concept of "the presentation of important frames should allow people to have time to read carefully", the present invention proposes a method for generating bookmark frames through the presentation time of similar pictures, an audio-visual playback device that automatically generates bookmarks, and a user interface thereof. According to the research of the present invention, it is found that in order to convey important information, the film producer will stay the important picture for a longer time. Based on this characteristic, this research proposes a similar frame presentation time to extract the important picture in the film.

To achieve the above object, a method for generating bookmark frames according to the present invention includes: extracting a plurality of frames sorted by time from a movie; calculating the structural similarity between two adjacent frames in the frames; The structural similarity between a first frame and the next adjacent second frame in these frames is less than a predetermined value, and the structural similarity of all frames after the first frame and within a predetermined period of time is greater than When it is equal to the preset value, the second frame is designated as a bookmark frame; and the image of the bookmark frame and its appearance time are stored.

To achieve the above objective, an audio-visual playback device for automatically generating bookmarks according to the present invention includes one or more processing units and a memory unit, the memory unit is electrically connected to the one or more processing units, and the memory unit stores one or more A plurality of program instructions, when the one or more program instructions are executed by the one or more processing units, the one or more processing units at least perform: extract a plurality of frames sorted by time from a video; calculate the In some frames, the structural similarity between two adjacent frames; when the structural similarity between a first frame in the frames and a second adjacent frame after the frame is less than a preset value, and after the first frame, and When the structural similarity corresponding to all frames within a preset time is greater than or equal to the preset value, the second frame is designated as a bookmark frame, and the image of the bookmark frame and its appearance time are stored; and The bookmark frame performs image text recognition to generate a bookmark.

In order to achieve the above objective, according to the present invention, a user interface of an audio-visual playback device that automatically generates bookmarks includes a display unit that presents the user interface, the The user interface includes a bookmark presentation area and a video playback area. The bookmark presentation area displays at least one bookmark, wherein the bookmark is automatically generated by the audio-visual playback device according to the following steps: extracting a plurality of frames sorted by time from a video; calculating between two adjacent frames of the frames The structural similarity of; when the structural similarity of a first frame and a second adjacent frame in the frames is less than a predetermined value, and all frames after the first frame and within a predetermined time When the corresponding structural similarity is greater than or equal to the preset value, the second frame is designated as a bookmark frame; the image of the bookmark frame and its appearance time are stored; and the image text recognition is performed on the bookmark frame, To generate the bookmark. The video playback area is adjacent to the bookmark presentation area; wherein, when the user clicks on the bookmark, the video and audio playback device plays a video clip corresponding to the bookmark in the video playback area.

In one embodiment, the preset value is between 0.8 and 0.9.

In one embodiment, the predetermined time is between 1 second and 3 seconds.

In one embodiment, the bookmark contains text, or a combination of text and image.

In one embodiment, the one or more processing units further perform the following steps: storing the bookmark in the memory unit.

In one embodiment, the audio-visual playback device further includes a display unit which is electrically connected to the one or more processing units, wherein the one or more processing units further perform the following steps: make the display unit present a user interface , Wherein the user interface includes a bookmark presentation area; and the bookmark is displayed in the bookmark presentation area.

In one embodiment, the number of the bookmarks is multiple, and the multiple bookmarks are displayed in the bookmark presentation area in a row.

In one embodiment, the user interface further includes a bookmark search area, which is adjacent to the bookmark presentation area; wherein, when the user enters a keyword in the bookmark search area and searches, the video playback area displays corresponding to The bookmark frame of the keyword.

As mentioned above, in the method for generating bookmark frames, the video and audio playback device for automatically generating bookmarks, and the user interface of the present invention, the structure similarity between two adjacent frames can be calculated and the judgment conditions can be found in the movie. The bookmark frame and its corresponding bookmarks, and these bookmarks can be displayed in the user interface, and through these bookmarks, users can quickly jump to the position of the video to be queried, thereby finding The content of the video to be watched, therefore, the present invention allows the user to browse the video in a smarter way, saving time for searching and viewing video fragments, and is particularly suitable for teaching videos for digital learning.

1: Audio and video playback device

11: Processing unit

12: Memory unit

13: display unit

A1: Video playback area

A2: Bookmark presentation area

A3: Bookmark search area

BF1, BF2: Bookmark frame

BM1, BM2, BM3: bookmarks

F: Movie

F1, F2, F3: Frame

S1, S2, S3, S31, S32, S33, S4: steps

UI: User interface

FIG. 1A is a flowchart of a method for generating a bookmark frame according to a preferred embodiment of the present invention.

FIG. 1B is a functional block diagram of a video and audio playback device for automatically generating bookmarks according to a preferred embodiment of the present invention.

2A is a schematic diagram of continuous frames of a movie arranged in time according to an embodiment of the present invention.

2B is a schematic diagram of video clips arranged in time according to an embodiment of the present invention.

FIG. 3 is a schematic diagram of an audio-visual playback device according to an embodiment of the invention.

Hereinafter, referring to related drawings, the method for generating a bookmark frame, an audio-visual playback device for automatically generating bookmarks, and a user interface thereof according to the preferred embodiment of the present invention will be described. The same components will be described with the same reference symbols.

FIG. 1A is a flowchart of a method for generating a bookmark frame according to a preferred embodiment of the present invention, and FIG. 1B is a functional block diagram of an audio-visual playback device for automatically generating bookmarks according to a preferred embodiment of the present invention. The first thing to note is that in general teaching videos, the transitions between different video clips often have large changes in the picture, so the frames (or images, pictures) with large differences can be regarded as the beginning of different clips. This kind of frame with big difference is called "bookmark frame" in this creation.

1A, the method for generating bookmark frames of the present invention at least includes the following steps: extracting a plurality of frames sorted by time from a video (step S1), calculating the number of frames between two adjacent frames Structural Similarity (Structural Similarity, SSIM) (step S2), when the structural similarity between a first frame in the frames and a second adjacent frame next to it is less than a preset value, and after the first frame, and When the structural similarity corresponding to all frames within a preset time is greater than or equal to the preset value, the second frame is designated as a bookmark frame (step S3 includes steps S31~S33), and the bookmark frame is stored And its appearance time (step S4).

The structural similarity (SSIM) of the aforementioned step S2 can be defined as follows: the input frame will be converted to grayscale, and each pixel in the frame will have a value between 0 and 255.

Given the frame signals x and y of two one-dimensional arrays, the structural similarity between the two can be defined as: SSIM( x , y )=[ l ( x , y )] ^α [ c ( x , y )] ^β [ s ( x , y )] ^γ ,

Among them, l ( x , y ) is to compare the brightness of x and y , c ( x , y ) is to compare the contrast of x and y , and s ( x , y ) is to compare the structure of x and y. μ _x and μ _y , σ _x and σ _y are the mean and standard deviation of x and y _{, respectively, σ xy} is the cross-covariance of x and y , C1, C2, and C3 are constants, which are used to maintain l ( x , y ), c ( x , y ), s ( x , y ) are stable. α>0, β>0, γ>0, which are important parameters for adjusting l ( x , y ), c ( x , y ), and s ( x , y ). Among them, the larger the SSIM value, the higher the similarity between the two signals. If two congruent frames (that is, two identical pictures) are used to perform SSIM operations, that is, μ _x = μ _y , σ _x = σ _y , then

Based on the structure term affecting the calculation result very little, in order to facilitate the calculation of the formula, the above formula is set as α = β = γ =1, and C3=C2/2 is substituted, the formula can be simplified as follows:

In addition, the audio-visual playback device of the present invention can be a mobile phone, a computer (including a tablet computer, a notebook computer, or a desktop computer), or an audio-visual player, without limitation.

1B, the audio-visual playback device 1 for automatically generating bookmarks of this embodiment includes one or more processing units 11 and a memory unit 12. The memory unit 12 can be electrically connected to the one or more processing units through, for example, a bus. Unit 11. Here, FIG. 1B shows one processing unit 11 and one memory unit 12 as an example.

The processing unit 11 can access data stored in the memory unit 12. The processing unit 11 may include the core control components of the audio-visual playback device 1, for example, may include at least one central processing unit (CPU) and a memory, or include other control hardware, software, or firmware. The memory unit 12 can store at least one application software. The application software can include one or more program instructions. When the one or more program instructions of the application software are When one or more processing units 11 are executed, the one or more processing units 11 can perform at least steps S1 to S4 of the aforementioned bookmark frame generation method, and perform image and text recognition on the bookmark frame to generate the content of a bookmark .

The memory unit 12 may be a non-transitory computer readable storage medium (non-transitory computer readable storage medium), for example, may include at least one memory, a memory card, an optical disc, a video tape, a computer tape, or Any combination of it. The memory may include read-only memory (ROM), flash memory, or field-programmable gate array (FPGA), or other forms of memory, or a combination thereof. The memory unit 12 can be a built-in memory of the audio-visual playback device 1, or a cloud memory located in the cloud device, so the application software can also be stored in the cloud device, and the user can load the audio-visual playback device 1 from the cloud device It can be executed.

In addition, the audio-visual playback device 1 of this embodiment may further include a display unit 13 which is electrically connected to the processing unit 11. The display unit 13 can be a display screen, a display screen, or a monitor.

Hereinafter, please refer to FIG. 1A and FIG. 1B in conjunction with FIG. 2A, FIG. 2B and FIG. 3 to illustrate the foregoing bookmark frame generation method, the audio-visual playback device for automatically generating bookmarks and the detailed technical content of the user interface thereof. 2A is a schematic diagram of continuous frames of a film arranged in time according to an embodiment of the present invention, FIG. 2B is a schematic diagram of film fragments arranged in time according to an embodiment of the present invention, and FIG. 3 is a schematic diagram of an embodiment of the present invention Schematic diagram of an audio-visual playback device. Among them, the audio-visual playback device 1 in FIG. 3 is an example of a notebook computer.

First, after the audio-visual playback device 1 loads the video F, the processing unit 11 can first convert the video F into a format supported by the audio-visual playback device 1 (for example, from MKV format or MP4 format) through conversion software (such as ffmpeg software). FLV format), and at the same time identify the basic data of the video F, such as resolution, video time and other information for subsequent processing.

Then, the processing unit 11 then extracts a plurality of frames sorted by time from the movie F (ie, step S1), such as the frames F1 to F5 shown in FIG. 2A. Here, the processing unit 11 captures a plurality of frames sorted by time, and then uses image processing software (such as OpenCV software) to convert these frames into gray-scale frames to calculate the SSIM value in step S2. The frames before and after conversion are still collectively referred to as "frames" in the present invention.

After that, the processing unit 11 calculates the structural similarity between two adjacent frames in the frames (step S2). Specifically, the processing unit 11 will calculate the SSIM value between all frames (such as the SSIM value between frame F1 and frame F2, the SSIM value between frame F2 and frame F3...), until the SSIM value corresponding to all frames is obtained. . Among them, the SSIM value between frame F1 and frame F2 is the SSIM value corresponding to frame F1, the SSIM value between frame F2 and frame F3 is the SSIM value corresponding to frame F2, and so on. If there are 100 frames, there will be 99 corresponding SSIM values.

After that, the processing unit 11 checks each SSIM value in order and makes a judgment, namely, proceed to step S3: When the structural similarity between a first frame in the frames and a second adjacent frame after it is less than a preset value ( Step S31), then check whether the SSIM values corresponding to all the frames after the first frame and within the preset time are greater than or equal to the preset value (Step S32), that is, check the preset time, after the first frame The SSIM value corresponding to the second frame, the third frame, the fourth frame..., if the SSIM values corresponding to all the frames appearing within the preset time are all "Yes", then the second frame is designated as Bookmark frame (step S33), and store the image of the bookmark frame and its appearance time (step S4) in the memory unit 12. If the SSIM value corresponding to one frame in step S32 is less than the preset value, then step S32 is "No".

Specifically, if the SSIM value corresponding to frame F1 is less than the preset value, it indicates that the image difference between frame F1 and frame F2 is large, and then check the subsequent frames F2, F3... which appear within the preset time. If the SSIM values corresponding to frames F2, F3... that appear within the preset time are all greater than or equal to the preset value, it means that the difference of all frames that appear within the subsequent preset time is not large (Pictures belonging to the same video segment), the frame F2 is the bookmark frame, and the processing unit 11 stores the image of the bookmark frame (frame F2) and its appearance time in the memory unit 12.

After the SSIM value corresponding to frame F1 is checked, the processing unit 11 will continue to check the SSIM value corresponding to frame F2 appearing after frame F1, that is, return to step S31, until the SSIM values corresponding to all frames have been checked. In this way, all the bookmark frames in the movie F and their corresponding movie fragments can be found. It is reminded that the first frame, the second frame, and the third frame that appear in the above steps... are multiple frames that appear consecutively in the sequence of time in the film F, which can be the first frame that appears at the beginning of the film F, The second frame and the third frame... can also be multiple consecutive frames that appear at any time in the film F (such as the middle part).

In the above, the preset value of the teaching video may be between 0.8 and 0.9, such as 0.85; and the preset time may be between 1 second and 3 seconds, such as 2 seconds. In other words, in the teaching video, if the SSIM value corresponding to a certain frame (such as the first frame) is less than 0.85, for example, and in all the frames that appear within, for example, 2 seconds after the first frame (ie, 2 seconds after the first frame) When the second frame, third frame, fourth frame... etc. appearing in the frame, its corresponding SSIM value is greater than or equal to, for example, 0.85, then the second frame appearing after the first frame can be regarded as a bookmark frame (Bookmark frames BF1 and BF2 in Figure 2B), and the movie fragment between the two bookmark frames is the movie corresponding to the bookmark frame, for example, the movie fragment 1 between the bookmark frame BF1 and the bookmark frame BF2 is the bookmark frame The video corresponding to BF1. It is specially reminded that when the video is not a teaching video, the aforementioned preset value and preset time range may be different, depending on the characteristics of the video.

On the contrary, if step S31 is “No”, it means that the difference between the first frame and the second frame is not large enough (the two may be related frames of the same movie segment), then return to step S31 and continue to check the first frame. Whether the SSIM value corresponding to the two frames meets this condition. In addition, if step S31 is "Yes" but step S32 is "No", then go back to step S31 until the SSIM values corresponding to all frames in movie F are checked, and then all the frames in movie F can be found Bookmark frame.

After the bookmark frame is found, the processing unit 11 can recognize the image and text of the bookmark frame to generate a bookmark corresponding to the bookmark frame, and store the bookmark in the memory unit 12. In other words, after obtaining the bookmark frames BF1, BF2..., the processing unit 11 can use image recognition software (such as Google Cloud Vision) to perform image text recognition of the bookmark frames BF1, BF2..., and then identify the bookmark frames BF1, BF2... The text messages in BF2. These text messages can be used to find the representative narrative of the frame through the text emphasis weighting algorithm. The formula is as follows:

)用於計算逆向檔案頻率，去除書籤影格中重複性高的文字訊息，NLP為分析語音訊息後， 該文字訊息的TextRank權重。在選出書籤影格中權重最高的文字訊息後，將其作為各書籤影格BF1、BF2的代表敘述，代表敘述即為書籤的名稱。其他的文字訊息透過斷詞處理後會成為該書籤影格的關鍵字，方便使用者透過影片內容來搜尋對應的書籤影格，進而搜尋對應的影片片段。當然，書籤的名稱也可以是該書籤影格的關鍵字之一。除了文字之外，處理單元11也可擷取書籤影格中的部分或全部圖像作為該書籤影格的代表(書籤)。換句話說，書籤影格所對應的書籤可以包含代表書籤影格的文字，或是文字和圖像的組合。此外，處理單元11也可儲存各書籤、各書籤影格及各影片片段之間的關聯性。 F1 is the ratio of fonts to all fonts in the screen, C is the color weight, SZ is the font size, T is the total number of bookmark frames in the movie, CT is the number of frames containing the text in the movie, and NLP is the voice focus weight. Among them, F1 is used to detect special fonts in the frame, such as bold and italic. This special font occupies a small proportion, but it is mostly important content indication. C is the custom and universally recognized text color weight, such as: red is the highest and black is the lowest. SZ is the font size of the text message. The unit is pixel, log (

) Is used to calculate the frequency of the reverse file and remove the repetitive text messages in the bookmark frame. NLP is the TextRank weight of the text message after analyzing the voice message. After selecting the text message with the highest weight in the bookmark frame, it is used as the representative description of each bookmark frame BF1 and BF2, and the representative description is the name of the bookmark. Other text messages will become the keywords of the bookmark frame after word hyphenation, which is convenient for users to search for the corresponding bookmark frame through the video content, and then search for the corresponding video clip. Of course, the name of the bookmark can also be one of the keywords of the bookmark frame. In addition to text, the processing unit 11 can also capture part or all of the images in the bookmark frame as a representative (bookmark) of the bookmark frame. In other words, the bookmark corresponding to the bookmark frame can contain text representing the bookmark frame, or a combination of text and image. In addition, the processing unit 11 can also store the associations between bookmarks, bookmark frames, and video clips.

Finally, the processing unit 11 may further perform the following steps: cause the display unit 13 to present a user interface UI. As shown in FIG. 3, the processing unit 11 may cause the user interface UI presented by the display unit 13 to include a video playing area A1 and a bookmark presenting area A2. As the name implies, the movie playing area A1 is an area for playing movies, and the bookmark presentation area A2 is an area where at least one bookmark (name) is displayed. Therefore, after obtaining the bookmark, the processing unit 11 can control the bookmark presentation area A2 of the user interface UI to display the corresponding bookmark of the bookmark frame, that is, display the bookmark name corresponding to the bookmark frame in the bookmark presentation area A2. Of course, if the movie has multiple bookmarks, such as the bookmarks BM1, BM2, BM3... etc. in Figure 3 (a bookmark name corresponds to a bookmark frame and a movie clip), the bookmarks BM1, BM2, BM3... can be displayed in a row In the bookmark presentation area A2. Moreover, when the user clicks a bookmark in the bookmark presentation area A2, for example, clicks the bookmark BM1, the bookmark frame BF1 corresponding to the bookmark BM1 can be displayed in the movie playback area A1, and of course the corresponding movie fragment 1 can also be played. In addition, in some embodiments, the processing unit 11 can also automatically generate a text file of the bookmark frame and its verbatim manuscript as the user's study notes, which saves the user from copying notes and studying time.

In addition, in addition to the video playback area A1 and the bookmark presentation area A2, the user interface UI of this embodiment may also include a bookmark search area A3, and the bookmark search area A3 is adjacent to the bookmark presentation area A2. Here, the bookmark search area A3 can be located on the upper, lower, left, or right side of the bookmark presentation area A2, without limitation. When the user enters a keyword in the bookmark search area A3 and searches, the corresponding bookmark can be found. Wherein, the keyword can be the name of the bookmark or the text content of the bookmark frame corresponding to the bookmark. After finding the corresponding bookmark, when the user clicks on the bookmark, the video playback area A1 can display the bookmark frame corresponding to the keyword, and the user can press play to watch the video corresponding to the bookmark frame in the video playback area A1. Therefore, users can search for the content they want to watch without pulling the timeline from memory. Searching through bookmarks can browse the videos in a smarter way, saving users viewing time.

Therefore, the audio-visual playback device 1 of the present invention can obtain the text content of the bookmark frame by using the above-mentioned bookmark frame generation method and image text analysis technology, and then obtain the corresponding bookmark (and keyword), which is convenient for the user to display on the display unit. 13 Perform keyword query in the displayed user interface UI. Moreover, these bookmarks displayed through the user interface UI can allow users to quickly jump to their corresponding video locations, and can also fast forward and reverse video paragraphs through these bookmarks. Therefore, even long videos can be used The user can accurately find the desired video content in a short time, increase the user’s grasp of the video, and enhance the value of the video.

In summary, in the method for generating bookmark frames, the audio-visual playback device for automatically generating bookmarks, and the user interface of the present invention, the structure similarity between two adjacent frames can be calculated and the judgment conditions can be found in the movie. The bookmark frame and its corresponding bookmarks, and these bookmarks can be displayed in the user interface, and through these bookmarks, the user can quickly jump to the position of the video to be queried, thereby finding the content of the video to be watched. Therefore, the present invention can Allows users to browse videos in a smarter way, saving video clips search and viewing time, especially suitable for teaching videos for digital learning.

The above descriptions are merely illustrative and not restrictive. Any equivalent modifications or alterations that do not depart from the spirit and scope of the present invention should be included in the scope of the appended patent application.

S1, S2, S3, S31, S32, S33, S4: steps

Claims (14)

A method for generating bookmark frames includes: extracting multiple frames sorted by time from a movie; calculating the structural similarity between two adjacent frames in the frames; when a first frame in the frames is When the structural similarity of the next adjacent second frame is less than a preset value, and the structural similarity of all the frames after the first frame and within a preset time is greater than or equal to the preset value, then Designating the second frame as a bookmark frame; and storing the image of the bookmark frame and its appearance time. The generating method according to claim 1, wherein the preset value is between 0.8 and 0.9. The generating method according to claim 1, wherein the preset time is between 1 second and 3 seconds. An audio-visual playback device for automatically generating bookmarks, comprising: one or more processing units; and a memory unit electrically connected to the one or more processing units, the memory unit storing one or more program instructions, when the one or more When a program instruction is executed by the one or more processing units, the one or more processing units at least perform: extract a plurality of frames sorted by time from a video; calculate the difference between two adjacent frames in the frames Structural similarity; when the structural similarity of a first frame and a second adjacent frame in the frames is less than a preset value, and all frames after the first frame and within a preset time correspond to When the structural similarity of is greater than or equal to the preset value, the second frame is designated as a bookmark frame, and the image of the bookmark frame and its appearance time are stored; and the image and text recognition of the bookmark frame is performed to Generate a bookmark. The audio-visual playback device according to claim 4, wherein the preset value is between 0.8 and 0.9. The audio-visual playback device according to claim 4, wherein the preset time is between 1 second and 3 seconds. The audio-visual playback device according to claim 4, wherein the bookmark contains text, or a combination of text and image. The audio-visual playback device according to claim 4, wherein the one or more processing units further perform the following steps: storing the bookmark in the memory unit. The audio-visual playback device according to claim 4, wherein the audio-visual playback device further includes: a display unit electrically connected to the one or more processing units, wherein the one or more processing units further perform the following steps: The display unit presents a user interface, wherein the user interface includes a bookmark presentation area; and displays the bookmark in the bookmark presentation area. The audio-visual playback device according to claim 9, wherein the number of the bookmarks is multiple, and the multiple bookmarks are displayed in the bookmark presentation area in a row. A user interface of an audio-visual playback device. The audio-visual playback device includes a display unit for presenting the user interface. The user interface includes: a bookmark presentation area displaying at least one bookmark, wherein the bookmark is created by the audio The playback device automatically generates according to the following steps: extracting multiple frames sorted by time from a video; calculating the structural similarity between two adjacent frames in the frames; when a first frame in the frames is When the structural similarity of the next adjacent second frame is less than a preset value, and the structural similarity of all the frames after the first frame and within a preset time is greater than or equal to the preset value, then Designate the second frame as a bookmark frame; store the image of the bookmark frame and its appearance time; and perform image and text recognition on the bookmark frame to generate the bookmark; and a video playback area adjacent to the bookmark Presentation area; wherein, when the user clicks on the bookmark, the video playback device plays the video clip corresponding to the bookmark in the video playback area. The user interface according to claim 11, wherein the bookmark includes text, or a combination of text and image. The user interface according to claim 11, wherein the number of the bookmarks is multiple, and the multiple bookmarks are displayed in the bookmark presentation area in a bar. The user interface described in claim 11 further includes: a bookmark search area adjacent to the bookmark display area; Wherein, when the user enters a keyword in the bookmark search area and searches, the video playback area displays the bookmark frame corresponding to the keyword.

Images