TW202143743A - Method of video graph developing - Google Patents

Abstract

The present disclosure provides a method of video graph developing. The method of video graph developing includes: at least according to a feature type, extracting video frames from a to-be-processed video as graph images; and developing a position index of the graph images and the video frames in the to-be-processed video, and each of the graph images is related to a feature information of the feature type, to develop a video graph of the to-be-processed video. The feature information is configured to search the graph images. The present disclosure is convenient for users to find videos and segments in the videos through high-dimension feature, and improve inefficiency of video searching information developing at the same time.

Description

Method of creating video atlas

This case is about a computer application field, especially a method for establishing a film atlas.

With the ever-increasing number of personal video information, it often takes a lot of manpower and time to find out the content of the films that have been browsed and filmed in the past, or to find the content you want in the film library. Find the content you need by manually watching the video recording and replaying. The efficiency is very low, and sometimes even impossible tasks. At the same time, the storage identification (mark) of the movie can only be summarized by the file name, which is inconvenient to input, and the information that can be stored is limited. It is not easy to provide a more comprehensive general description of the movie, which makes it difficult to obtain when searching. At the same time, fast-forward playback when searching, often dragging back and forth, and screen skipping search, which tends to give people a kind of nervousness, easy to get tired, and it is easy to lose the screen that needs attention and miss the content. In addition to personal video information, the above-mentioned problems may also exist in the retrieval and search of various video platforms.

It can be seen that there are two main defects in movie retrieval: 1) There are too many movie materials, and it is difficult to accurately find the desired movie; 2) When the movie is long, it is difficult to find the required movie segment in the movie.

In order to solve the above-mentioned problems, in some prior art, screenshots of the movie are taken at fixed time intervals or manually selected time points of the movie to extract the screenshots of the movie to determine whether it is a target movie. However, in this way, on the one hand, it is necessary to view each movie through the movie playback tool, or manually click and slide to view without interruption, which is very inefficient; on the other hand, the extracted movie screenshots are extracted at fixed intervals according to time, or manually randomly extracted Yes, the amount of screenshots is often very large, but it fails to catch the target feature that needs to be found, it is still difficult to find, and the efficiency is still not high. In some other existing technologies, based on the number of times a movie or a video segment is viewed, a "excellent library" is created for the movies with a high frequency. However, in this way, it is still impossible to retrieve the target video based on the information of the human memory segment.

Therefore, how to improve the efficiency of constructing movie retrieval information while facilitating users to find movies and clips in movies through multi-dimensional features is a problem to be solved urgently in this field.

In order to overcome the above-mentioned drawbacks of the prior art, the present disclosure provides a method for creating a video atlas, which can improve the efficiency of video retrieval information construction and at the same time facilitate users to find videos and video clips through multi-dimensional features.

An aspect of the present disclosure provides a method for creating a video atlas, which includes the following steps. According to at least one feature type, a film frame is extracted from the film to be processed as an atlas picture. Establishing the position index of each of the atlas pictures and the video frame in the to-be-processed video, and each of the atlas pictures is associated with the feature information of the feature type to create a video atlas of the to-be-processed video, the The feature information is used to retrieve the atlas picture.

In some embodiments of the present disclosure, when a film frame is extracted as an atlas picture in each of the sub-film segments according to multiple feature types, for each feature type: establish each of the atlas extracted according to the feature type The position index of the picture and the video frame in the video to be processed, and each of the atlas pictures extracted according to the feature type is associated with the feature information of the feature type, so as to establish the corresponding to the video to be processed Feature type movie atlas.

In some embodiments of the present disclosure, extracting a film frame as an atlas picture from the film to be processed according to at least one feature type includes the following steps. The movie to be processed is divided into a plurality of sub-movie segments. Determine the number of film frames extracted from each sub-film segment. For each sub-film segment, a film frame is extracted as an atlas picture according to the feature type and the parameters of the film frame.

In some embodiments of the present disclosure, the determining the number of film frames extracted from each sub-film segment includes the following steps. According to the feature type, the sub-film segments to be extracted are selected from the plurality of sub-film segments. The number of film frames extracted from each sub-film segment to be extracted is determined according to the ratio of the duration of the sub-film segment to be extracted to the total duration of the sub-film segment to be extracted.

In some embodiments of the present disclosure, after creating the film atlas of the film to be processed, the method further includes: displaying the film atlas of the film to be processed.

In some embodiments of the present disclosure, the displaying the film atlas of the film to be processed includes: according to the length of time occupied in the film to be processed by the feature information of the feature type associated with each of the atlas pictures, The atlas pictures are sorted.

In some embodiments of the present disclosure, the displaying the film atlas of the film to be processed includes: according to the time sequence in which the feature information of the feature type associated with each of the atlas pictures appears in the film to be processed, The atlas pictures are sorted.

In some embodiments of the present disclosure, the feature type includes a scene feature, an object feature, a text feature, an interest feature, or a combination thereof.

In some embodiments of the present disclosure, the method further includes: constructing associated content for one or more of the to-be-processed movie and the atlas picture as the object to be associated, and the associated content is related to the to-be-associated object according to at least one attribute feature. The associated object is associated, and the associated content includes a movie, a movie segment, an atlas picture, or a combination thereof.

In some embodiments of the present disclosure, the attribute characteristics include object attributes, gesture attributes, expression attributes, scene attributes, text attributes, interest attributes, predetermined behavior attributes, or a combination thereof.

Compared with the prior art, the method provided by the present disclosure has the following advantages: automatically extract atlas pictures from the video to be processed through feature types, which is convenient for users to quickly understand the content of the video; and the atlas pictures are associated with the feature information of the feature type used for retrieval, with Realize dimensionality search, which improves the construction efficiency of movie retrieval information, and at the same time, it is convenient for users to find atlas pictures through feature information to locate them in the movie.

In order to solve the disadvantages of the prior art, the present disclosure provides a method for creating a video atlas, which can improve the efficiency of constructing video retrieval information and at the same time facilitate users to find videos and video clips through multi-dimensional features.

First, referring to Figure 1, Figure 1 shows a flowchart of a method for establishing a film atlas according to an embodiment of the present disclosure. Figure 1 shows steps S110 and S120 in total, as described below.

Step S110: Extract a film frame from the film to be processed as an atlas picture according to at least one feature type; Step S120: Establish a position index of each of the atlas picture and the film frame in the film to be processed, and each of the The atlas picture is associated with the feature information of the feature type to create a film atlas of the to-be-processed video, and the feature information is used to retrieve the atlas picture.

In the method for creating a video atlas provided in this disclosure, the atlas pictures are automatically extracted from the video to be processed through the feature type, which is convenient for users to quickly understand the content of the video; and the atlas pictures are associated with the feature information of the feature type used for retrieval to achieve reduction Dimensional search, while improving the efficiency of video retrieval information construction, it is convenient for users to search for atlas pictures through feature information to locate them in the video.

In the various embodiments of the present disclosure, the video to be processed may be a local video of the user equipment, an online video, or a video of each video platform.

In the various embodiments of the present disclosure, the feature information may include text, image, feature code, feature matrix, and so on.

For the specific implementation of the foregoing step S110, refer to FIG. 2, which shows a flowchart of extracting atlas pictures according to a specific embodiment of the present disclosure. Figure 2 shows steps S111 to S114, as described below.

Step S111: Divide the film to be processed into multiple sub-film segments; Step S112: Filter the sub-film segments to be extracted from a plurality of sub-film segments according to the feature type; Step S113: According to the sub-film segments to be extracted The ratio of the duration to the total duration of the sub-film segments to be extracted determines the number of film frames extracted from each sub-film segment to be extracted; Step S114: For each sub-film segment, according to the feature type and the film The parameters of the frame extract the film frame as the atlas picture.

In some embodiments, in step S111, the sub-film segment can be divided by switching the film shots. In some other embodiments, step S111 may implement the division of sub-movie segments through different backgrounds in the movie (for example, indoors, parks, roads, etc.). In still other embodiments, step S111 may be combined with the switching of the film shots to determine the positions that can be divided, and then through different background recognition, select and divide from the positions that can be divided, so as to obtain sub-movie segments.

Specifically, in step S112, according to the feature type, it is determined whether each sub-film segment conforms to the feature type, so that only the sub-film segment conforming to the film type is used as the video terminal to be extracted. In some embodiments, the feature type may include one or more of scene feature, object feature, text feature, and interest feature. When the feature type is a scene feature, the atlas pictures are not extracted for the sub-film segment that does not have a scene (the sub-film segment without a scene may be, for example, a sub-film segment that only includes text). When the feature type is an object feature, the atlas picture is not extracted for the sub-movie segment that does not have an object. When the feature type is a text feature, the atlas picture is not extracted for the sub-movie segment that does not have text. When the feature type is an interest feature, no atlas pictures are extracted from the sub-film segments that do not have interest operations.

In some embodiments, the same number of atlas pictures can be set for each movie to be processed. Therefore, in step S113, the film extracted from each sub-film segment to be extracted can be determined according to the product of the ratio of the duration of the sub-film segment to be extracted to the total duration of the sub-film segment to be extracted and the set number. The number of frames.

In some embodiments of the present disclosure, step S114 may extract a plurality of film frames matching the feature type from the sub-film segment, and then analyze the image quality basic parameters of the extracted film frames (the image quality basic parameters may include, for example: One or more of definition, sharpness, brightness, noise, and focus), and multiple film frames with higher basic image quality parameters are used as map pictures in each extracted film frame. Further, step S114 may also include determining the duration of the corresponding feature information in the movie to be processed for each atlas picture. Specifically, the duration of the feature information corresponding to the atlas picture in the to-be-processed video may refer to the continuous duration of the atlas picture and the feature information included in the to-be-processed video.

Next, referring to FIG. 3, FIG. 3 shows a schematic diagram of a film atlas according to a specific embodiment of the present disclosure. The film atlas 200 includes a plurality of atlas pictures 212, and each atlas picture 212 is extracted from the to-be-processed film 211 according to a feature type. Specifically, in this embodiment, the feature type may include one or more of scene features, object features, text features, and interest features.

For scene features, multiple atlas pictures 212 of different scenes can be extracted from the movie to be processed 211, each atlas picture 212 corresponds to a time point on the time axis T of the movie to be processed 211, and each atlas picture 212 is associated with The feature information is scene information (the atlas picture 212 contains the scene described by the scene information). The scene information can be, for example, indoor/outdoor, animals/plants, mountains/water, weather (rainy/sunny/snow), urban/rural, etc.

For object features, multiple atlas pictures 212 of different objects can be extracted from the video to be processed 211, and the feature information associated with each atlas picture 212 is object information (the atlas picture 212 contains the object described by the object information). The object information can be, for example, different person avatars, different objects, different animals, and so on.

For text features, multiple atlas pictures 212 of different texts can be extracted from the video to be processed 211, and the feature information associated with each atlas picture 212 is text information (the atlas picture 212 contains the text information). The text information can be, for example, text data such as subtitles, sounds, and bullet screens.

For the interest feature, multiple atlas pictures 212 of different interest operations can be extracted from the to-be-processed video 211, and the feature information associated with each atlas picture 212 is the interest operation information (the user performed the interest operation on the atlas picture 212 for the video to be processed) . Interest operation information can be, for example, the bullet screen issued, the mark made, etc.

Further, in some embodiments of the present disclosure, when displaying the film atlas of the to-be-processed film, the characteristic information of the feature type associated with each of the atlas pictures may be occupied in the to-be-processed film. Long, sort the atlas pictures. For example, for the feature of the object, in step S114, the step of determining the duration of the corresponding object in the movie to be processed for each atlas picture has been performed. The atlas pictures are sorted. As a result, it is convenient for the user to get to know the longer-playing objects in the movie more quickly. The above sorting method can also be applied to scene features, text features, and interest features.

In other embodiments of the present disclosure, when displaying the film atlas of the to-be-processed film, the time sequence in which the feature information of the feature type associated with each of the atlas pictures appears in the to-be-processed film may also be used. To sort the atlas pictures. That is, the atlas pictures are sorted according to the time axis as shown in Figure 3, so that the user can understand the content of the movie according to the time sequence. The above sorting method can also be applied to object features, scene features, text features, and interest features.

In some embodiments of the present disclosure, the present disclosure may also construct related content for the generated film atlas. Specifically, the present disclosure includes the step of constructing associated content for one or more of the to-be-processed movie, the sub-movie segment, and the atlas picture as the object to be associated, and the associated content is based on at least one attribute feature and The object to be associated is associated, and the associated content includes one or more of a movie, a movie segment, and an atlas picture. The display of related content is as shown in FIG. 4, which shows a schematic diagram of related content according to a specific embodiment of the present disclosure. For the to-be-processed movie 211, construct the associated content 220. The associated content 220 and the to-be-processed movie 211 can contain the same objects (object attributes), the objects contained have the same poses (posture attributes), and the objects contained have the same expressions (emojis). Attributes), including the same scene (scene attributes, for example, the attributes of the scene can also include similar tones), including the same text (text attributes), and the user performs the same interest operations on them (sending bullet screens, setting flags and other interest attributes ) Or contains objects that perform the same predetermined behavior (predetermined behavior attributes, which can include handshake, running, etc.). The “same” in the above description can also be understood as “similar”, and whether “same” can be determined according to the recognition algorithm and similarity calculation, which will not be repeated here. After constructing the related content, when the user retrieves the video, the related content can be provided to the user when the user selects the video or atlas picture (such as the interface provided in Figure 4), so that the user can view the related content. To further improve user retrieval efficiency.

The following describes the movie retrieval of the embodiment of the disclosure in conjunction with FIG. 5 to FIG. 7. Figure 5 shows a schematic diagram of a movie list according to a specific embodiment of the present disclosure. Fig. 5 shows thumbnails of a plurality of movies 310. When entering the movie search, the interface shown in Figure 6 can be displayed.

FIG. 6 shows a list of film atlases according to a specific embodiment of the present disclosure, and the atlas pictures 321 of the film atlas 320 for each film 310 are shown. The number of the displayed atlas pictures 321 is merely illustrative, and the present disclosure may only display part of the atlas pictures, and when the user receives further operations, all the atlas pictures can be displayed. The displayed map picture can be a thumbnail of the map picture. The number of atlas pictures displayed can be determined by the screen size of the device displayed. When the user operates the atlas picture 321, he can directly jump to the corresponding position of the movie through the position index.

Figure 6 also shows a feature type switching virtual icon 330. In this embodiment, the position index of each of the atlas pictures extracted according to the feature type and the film frame in the film to be processed is established, and the atlas pictures extracted according to the feature type are associated with each other. The feature information of the feature type is used to create a movie map of the to-be-processed movie corresponding to the feature type. Therefore, for each movie 310, there may be multiple feature types of movie atlases, and by operating the feature type switching icon 330, the movie atlases of different feature types are switched for display.

Further, FIG. 6 also shows a retrieval icon 340, and the retrieval of the atlas pictures is realized by operating the retrieval icon 330. In some specific embodiments, different retrieval interfaces may be provided for different feature types. For example, for the feature of an object, the retrieval interface can provide multiple object pictures, so that the atlas pictures can be retrieved according to the matching of the object pictures. Further, other information (such as gestures, expressions, specific behaviors) can also be combined to search for atlas pictures. For another example, for text features, the search interface can provide a search input field, so as to search for atlas pictures based on the text information input by the user. After the retrieval, the retrieved atlas pictures can be marked, as shown in FIG. 7, which shows a schematic diagram of the retrieval result of the movie picture list according to the specific embodiment of the present disclosure.

Next, referring to FIG. 8, FIG. 8 shows a schematic diagram of a system for establishing a film atlas according to a specific embodiment of the present disclosure. FIG. 8 shows multiple shooting devices 410, a cloud server 420, and a user device 430. The video shot by the shooting device 410 is uploaded to the cloud server 420. After the video is decoded or the video stream 421 is obtained in real time, the cloud server 420 performs the atlas analysis 422 of different feature types to extract the atlas pictures, and at the same time, performs the association analysis 423 of different feature attributes to associate the content of the created atlas. The cloud server 420 creates a video atlas index 424 after performing the atlas analysis 422 and the association analysis 423. Then, the user equipment 430 is used to display the video atlas 431 and display the related content 432.

Figure 8 only schematically shows an implementation manner of the present disclosure. For example, the video uploaded to the cloud server 420 can be implemented through the user device 430 or other cloud platforms. For another example, the steps executed by the cloud server 420 may be executed locally by the user equipment 430. These changes are within the scope of protection of this disclosure.

Next, referring to Fig. 9, Fig. 9 shows a flowchart of a method for establishing a film atlas according to a specific embodiment of the present disclosure.

First, step S501 is executed to analyze the format of the movie. Perform step S502: decode the movie and audio. Step S503 is executed to classify and cut molecular movie segments according to the background. According to the multiple different feature types (object feature, interest feature, text feature, scene feature) according to step S504 to step S507, the atlas picture is analyzed. Step S508: Perform parameter analysis of the film frame to determine the atlas picture to be extracted. Step S509: Determine the duration of the feature information of the feature type of the video frame to be extracted in the video (for sorting during display). Step S510: Determine the extraction position of the film frame. Step S511: Determine the number of atlas pictures of each sub-film segment. In step S512, a film frame is extracted from each sub-film segment as an atlas picture. In step S513, the position index of each atlas picture in the movie is established and added to the atlas library 500. Step S514: Perform correlation analysis on the atlas pictures/movies. Step S515-Step S518 determine the associated content according to different feature attributes (different feature attributes have been explained in the description of Figure 4, and will not be repeated here). Step S519 constructs an association relationship between the associated content and the atlas picture/movie, and adds it to the association library 590.

Figure 9 merely schematically illustrates an implementation manner of the present disclosure.

This disclosure also provides a method for movie retrieval. Fig. 10 shows a flowchart of a method for film retrieval according to an embodiment of the present disclosure. The method of movie retrieval includes step S610 and step S620, as described below.

Step S610: Receive search information, and match feature information from a movie atlas library based on the search information. The movie atlas library includes a plurality of movie atlases created according to the method for creating a movie atlas as described above; Step S620: Output the matched The map picture corresponding to the feature information of.

In some embodiments of the present disclosure, the method for movie retrieval further includes: in response to the selected atlas picture and/or movie, providing associated content of the selected atlas picture and/or movie, and the associated content is based on at least one attribute feature Associated with the atlas pictures and/or movies, the associated content includes one or more of a movie, a movie segment, and atlas pictures. The specific content of the associated content has been described in the method of creating a movie map, and will not be repeated here.

In the method for movie retrieval provided in the present disclosure, the atlas pictures are automatically extracted from the movies to be processed by the feature type, which is convenient for users to quickly understand the content of the movie; and the atlas pictures are associated with the feature information of the feature type used for retrieval to achieve dimensionality reduction search , While improving the efficiency of constructing video retrieval information, it is convenient for users to find atlas pictures through feature information to locate them in the video. In addition, the video retrieval method can also provide related content, so that the user can directly provide related videos when viewing a selected video, so as to avoid the user from re-retrieving.

Compared with the prior art, the method and device provided by the present disclosure have the following advantages: automatically extract atlas pictures from the movie to be processed by feature types, and the atlas pictures are associated with the feature information of the feature type used for retrieval, so as to achieve dimensionality reduction search. While improving the construction efficiency of video retrieval information, it is convenient for users to find atlas pictures through feature information to locate them in the video.

Those skilled in the art will easily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptive changes of this disclosure. These variations, uses, or adaptive changes follow the general principles of this disclosure and include common knowledge or conventional technical means in the technical field that are not disclosed in this disclosure. . The description and the embodiments are only regarded as exemplary, and the true scope and spirit of the disclosure are pointed out by the appended claims.

In order to make the above and other objectives, features, advantages and embodiments of the present disclosure more obvious and understandable, the description of the attached symbols is as follows: S110, S120, S111~S114, S501~S519, S610, S620: steps 200: Film Atlas 211: Pending Video 212: Atlas picture T: Timeline 220: Associated Content 310: Movie 320: Film Atlas 321: Atlas Picture 330: Switch icon 340: Search icon 410: shooting equipment 420: Cloud Server 421: Video decoding or real-time acquisition of video stream 422: Atlas Analysis 423: Association Analysis 424: Create a video atlas index 430: user equipment 431: Show Video Atlas 432: Show related content 500: Atlas Library 590: Associated Library

In order to make the above and other objectives, features, advantages and embodiments of the present disclosure more obvious and understandable, the description of the accompanying drawings is as follows: Figure 1 shows a flowchart of a method for establishing a film atlas according to an embodiment of the present disclosure; Figure 2 shows a flowchart of extracting atlas pictures according to a specific embodiment of the present disclosure; Figure 3 shows a schematic diagram of a film atlas according to a specific embodiment of the present disclosure; Figure 4 shows a schematic diagram of related content according to a specific embodiment of the present disclosure; Figure 5 shows a schematic diagram of a movie list according to a specific embodiment of the present disclosure; FIG. 6 shows a schematic diagram of a film atlas list according to a specific embodiment of the present disclosure; FIG. 7 shows a schematic diagram of a retrieval result of a film picture list according to a specific embodiment of the present disclosure; Figure 8 shows a schematic diagram of a system for establishing a film atlas according to a specific embodiment of the present disclosure; Figure 9 shows a flowchart of a method for establishing a film atlas according to a specific embodiment of the present disclosure; and Fig. 10 shows a flowchart of a method for film retrieval according to an embodiment of the present disclosure.

S110, S120: steps

Claims (10)

A method of creating a video atlas, including: Extracting at least one film frame from a to-be-processed film as at least one atlas picture according to at least one feature type; and Establishing each of the at least one atlas picture and the position index of the at least one video frame in the to-be-processed video, and each of the at least one at least one atlas picture is associated with a feature information of the feature type to create a video atlas of the to-be-processed video, The at least one feature information is used to retrieve the at least one atlas picture. The method for creating a film atlas according to claim 1, wherein when the at least one film frame is extracted as the at least one at least one atlas picture from each of the plurality of sub-film segments in the to-be-processed film according to a plurality of the feature types, Each of this feature type: Establish the position index of each of the at least one atlas image extracted according to the feature type and the at least one video frame in the to-be-processed video, and each of the at least one atlas image extracted according to the feature type is associated with the feature information of the feature type , To create the movie map corresponding to the feature type of the movie to be processed. The method for creating a film atlas according to claim 1, wherein extracting the at least one film frame from the to-be-processed film as the at least one atlas picture at least according to the feature type includes: Divide the to-be-processed movie into multiple sub-movie segments; Determining the number of the at least one film frame extracted from each of the at least one sub-film segment; and For each of the at least one sub-film segment, extract the at least one film frame as the at least one atlas picture according to the feature type and the parameters of the at least one film frame. The method for creating a film atlas according to claim 3, wherein determining the number of the at least one film frame extracted from each of the plurality of sub-film segments includes: According to the feature type, filter at least one sub-film segment to be extracted from the plurality of sub-film segments; and According to the ratio of the duration of the at least one sub-film segment to be extracted to the total duration of the at least one sub-film segment to be extracted, determine the amount of the at least one film frame extracted from each of the at least one sub-film segment to be extracted quantity. The method for creating a movie atlas according to claim 1, wherein after creating the movie atlas of the to-be-processed movie, the method further includes: Show the movie map of the movie to be processed. The method for creating a movie atlas according to claim 5, wherein the movie atlas for displaying the to-be-processed movie includes: The at least one at least one atlas picture is sorted according to the length of time the at least one at least one atlas picture is associated with the characteristic information of the characteristic type in the to-be-processed video. The method for creating a movie atlas according to claim 5, wherein the movie atlas for displaying the to-be-processed movie includes: The at least one atlas picture is sorted according to the time sequence in which the feature information of the feature type associated with each at least one atlas picture appears in the to-be-processed video. The method for creating a movie atlas according to any one of claims 1 to 7, wherein the feature type includes scene feature, object feature, text feature, interest feature, or a combination thereof. The method for creating a video atlas as described in any one of Claims 1 to 7, further includes: Construct an associated content for one or more of the to-be-processed video and the at least one atlas picture as a to-be-associated object, the associated content is associated with the to-be-associated object according to at least one attribute feature, and the associated content includes a movie and a movie segment , Atlas picture or a combination thereof. The method for establishing a movie atlas according to claim 9, wherein the at least one attribute feature includes object attributes, gesture attributes, expression attributes, scene attributes, text attributes, interest attributes, predetermined behavior attributes, or a combination thereof.

Images