JP2021048506A - Video scene information management apparatus - Google Patents

Abstract

To provide a video scene management apparatus capable of selectively reproducing each scene in a video file by a form of displaying a scene identifier of each scene at a proper place within a screen.SOLUTION: A scene information acquisition unit 101 acquires, as scene information, a data set including a scene identifier, a scene interval, and positional information related with a display position of the scene identifier in reproduction of a scene, for each scene of a video file. A scene information registration unit 102 registers a data set of each scene information for each video file to a scene information management server 3. A scene information retrieval unit 104 selects the scene information by keyword retrieval of the scene identifier. A video file reproduction unit 103 reproduces each scene of the video file. A scene information reproduction unit 105 displays the scene identifier in a reproduced scene.SELECTED DRAWING: Figure 1

Description

The present invention relates to a video scene information management device that extracts scenes from a video file, generates and manages scene information, and selectively reproduces the scenes based on the scene information. In particular, the present invention relates to a video scene information management device that selectively reproduces each scene in the video file. The present invention relates to a video scene information management device that reproduces the scene identifier in a format in which the scene identifier is displayed in an appropriate place on the screen.

Patent Document 1 discloses a technique in which a user extracts a program or a scene included in the program from subtitle information and displays it on a display.

JP-A-2015-52897

In the prior art, the viewing user can extract the display scene of the subtitle as a desired scene by performing a keyword search for the subtitle, but cannot extract the scene from the video file that does not include the subtitle.

In addition, by automatically detecting and identifying an object from a video file and registering the identification result as text, it is possible to automatically extract a scene in which a desired object appears by a keyword search targeting the text. it can.

However, if the object detection / identification process is black-boxed, the viewing user cannot grasp the object detection / identification process. Therefore, even if the scene extracted based on the identification result of the object is reproduced, it may not be possible to recognize the meaning and content of the scene, such as what the scene was selected based on.

Furthermore, when a video file of a pet, etc. taken with a home video is published on the network and used for viewing by a large number of net users, general-purpose object detection / identification identifies that the pet is a dog or a cat. Even if it can be done, unique information such as a name cannot be identified. Therefore, it is not possible to provide a scene search using a name or the like as a search keyword to a net user.

Furthermore, if the video file is a 360 ° video, it is difficult to comprehensively search for the playback position to be extracted as a scene due to the presence of blind spots.

An object of the present invention is to solve the above technical problems and to provide a video scene management device capable of reproducing each scene in a video file in a format in which the scene identifier is displayed at an appropriate position on the screen.

In order to achieve the above object, the present invention is characterized in that the video scene management device for managing the scene of the video file has the following configuration.

(1) For each scene in the video file, the scene information acquisition means for acquiring the data set including the scene identifier, the scene section, and the position information of the scene identifier to be displayed in the scene as the scene information, and the data set of each scene information are video. It is equipped with a scene information registration means for registering each file.

(2) A means for selecting scene information by keyword search for the scene identifier, a means for reproducing each scene of the video file based on the selected scene information, and a means for reproducing each scene based on the selected scene information. It was provided with a means for displaying the scene identifier.

(3) The scene information acquisition means includes a means for detecting an object from a video file and a means for identifying the object, and the scene information registration means sets the object identification result as a scene identifier and the object detection section as a scene. Changed to register the scene information that uses the position of the section and the object as the position information of the scene identifier.

(4) Further provided with an input operation means for accepting a user's input operation, the scene information acquisition means acquires a means for acquiring a scene section input from the input operation means and a scene identifier input from the input operation means. It is provided with means and means for acquiring the display position of the scene identifier input from the input operation means as position information.

(5) When the video file is a 360 ° video, the position information of the scene identifier is used as the coordinate value on the coordinates obtained by expanding the 360 ° video in two dimensions.

According to the present invention, the following effects are achieved.

(1) For each scene in the video file, a data set including the scene identifier, the scene section, and the position information of the scene identifier is acquired and registered as the scene information. Therefore, if the scene information is referred to, the scene identifier of the video file can be used. The corresponding scene can be played back in a format in which the scene identifier is displayed in place.

(2) Since the scene information of the video file is searched and each scene of the video file is played back based on this scene information, the scene corresponding to the scene identifier of the video file can be played back together with the scene identifier. Therefore, by referring to the scene identifier of each scene, it becomes possible to visually recognize the content of each scene and the meaning and name of the information reflected in each screen of each scene.

(3) The scene information acquisition means detects and identifies an object from a video file, and registers scene information in which the identification result of the object is the scene identifier, the detection section of the object is the scene section, and the position of the object is the position information of the scene identifier. Therefore, not only can the scene be objectively extracted from the video file and played back, but also the scene identifier can be displayed at an appropriate place in the scene.

(4) An input operation means for accepting the user's input operation is provided, and the position information regarding the scene section, the scene identifier, and the display position of the scene identifier input from the input operation means is registered as the scene information. Can be extracted and the scene identifier and its display position can be registered arbitrarily. Therefore, if the registration of the scene information is permitted for a large number of users, the scene information can be registered and displayed from various viewpoints depending on the subjectivity of each user.

(5) If the video file is a 360 ° video, the position information of the scene identifier is expressed by the coordinate values when the 360 ° video is expanded in two dimensions, so even if the video file is a 360 ° video, If scene information is registered in a characteristic scene, the scene can be played back with the coordinates of each scene identifier selectively displayed on the screen.

It is a functional block diagram of the video scene information management system to which this invention is applied. It is a figure which showed the registration format of the scene information in the scene information management server. It is a functional block diagram of the 1st Embodiment of a scene information management apparatus. It is a figure which showed typically the acquisition and registration method of a scene information. It is a flowchart which showed the reproduction method of the desired scene of a video file. It is a figure which showed an example of the scene search screen. It is a figure which showed the example of the scene reproduction when a keyword is input. It is a figure which showed the scene reproduction when the keyword "pet dog △△" is input. It is a figure which showed the scene reproduction when the keyword "love cat 〇〇" is input. It is a functional block diagram of the 2nd Embodiment of a scene information management apparatus. It is a figure which showed an example of the registration screen at the time of manual registration of a scene information. It is a figure which showed typically the input and registration method of a scene information.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing a configuration of a video scene information management system to which the present invention is applied, and has a video scene information management device 1, a video file server 2, and a scene information management server 3 as main configurations.

A large number of video files are stored in the video file server 2. Each video file includes a video signal and an audio signal, and may also include subtitle information. The video scene information management device 1 sets scene information for each characteristic scene of the video file and registers it in the scene information management server 3, while when playing back the video file, the corresponding scene information is transmitted from the scene information management server 3 from the scene information management server 3. Read out and selectively play the scene.

In the video scene information management device 1, the scene information acquisition unit 101 displays a text text as a scene identifier unique to each scene of the video file, a start time ts and an end time te of the scene, and the text text on the screen during scene playback. Acquire a data set including position information for specifying a position to be performed as scene information. The scene information registration unit 102 registers the scene information data set in the scene information management server 3 in a table format in association with the video file identifier.

FIG. 2 is a diagram showing a registration format of each scene information in the scene information management server 3. In the present embodiment, a text text as a scene identifier, a start time ts and an end time te of the scene, and a text text are displayed. A type that indicates whether the position information p1 and p2 for superimposed display during scene playback are registered, and whether the scene information is acquired objectively (auto) or subjectively (user1, user2 ...). Is registered. The scene information management server 3 manages one or more of such data sets in a table format for each video file.

Returning to FIG. 1, the video file reproduction unit 103 reproduces the video file specified by the user from the input operation unit 107 on the display panel 106. The scene information search unit 104 searches for and acquires scene information corresponding to the search keyword specified by the user from the input operation unit 107 from the scene information management server 3. The scene information reproduction unit 105 displays the text text associated with the scene at the coordinate positions specified by the position information p1 and p2 in accordance with the reproduction of the scene specified by the scene information of the video file.

As described above, according to the present embodiment, when the scene of the video file based on the scene information is reproduced, the text text associated with the scene is displayed at the position related to the scene, so that the scene reproduction becomes easy. , The viewer can easily recognize the main part, interpretation, purpose, etc. of the scene.

FIG. 3 is a functional block diagram showing the configuration of the first embodiment of the scene information management device 1, and the same reference numerals as those above represent the same or equivalent parts, and thus the description thereof will be omitted. The present embodiment is characterized in that the scene information acquisition unit 101 includes the object detection unit 101a, the object identification unit 101b, and the object identification model 101c, and automatically, that is, objectively acquires the scene information from the video file. ..

The object detection unit 101a detects objects such as people, objects, and animals from the video file. If the video file is a 360 ° video, the video is expanded in two dimensions and object detection is executed on the two-dimensional video.

The object identification unit 101b identifies the object by applying the detected object to the object identification model 101c that has learned the image feature amount for object identification in advance, and outputs the identification result in a text format.

The scene information registration unit 102 sets the object identification result (text), the object appearance period (start time ts and end time te), the position information p1 and p2 on the screen of the object, and the type of these information (auto). A data set consisting of the above is registered in the scene information management server 3 as scene information.

FIG. 4 is a diagram schematically showing a method of acquiring and registering scene information in the present embodiment. At playback time t1, the object detection unit 101a detects a dog as an object as shown in FIG. 4A. Then, the rectangular frame k1 circumscribing the object is set, and the upper right coordinate and the lower left coordinate of the rectangular frame k1 are acquired as the position information p1_1 and p2_1 of the object, respectively. The object identification unit 101b identifies the object as a "dog" using the object identification model 101c, and object tracking is started. Further, if a specific model, for example, "pet dog Δ△" exists by the object identification model 101c, it is identified as "pet dog Δ△" and object tracking is started.

When the object detection unit 101a newly detects a cat as an object at the playback time t2 as shown in Fig. (B), a rectangular frame k2 circumscribing the object is set, and its position information p1_2 and p2_2 are acquired. .. The object identification unit 101b identifies the object as a "cat" and starts object tracking. Further, if a specific model, for example, "love cat △△" exists by the object identification model 101c, it is identified as "love cat 〇〇" and object tracking is started.

At time t3, when the dog frames out as shown in Fig. (C), the scene information is registered. Here, the text text is "pet dog △△", the start time ts is "t1", the end time te is "t3", the first coordinate is "p1_1", the second coordinate is "p2_1", and the type is based on object identification. A data set consisting of "auto" indicating that it is an objective automatic registration is registered in the scene information management server 3 as one of the scene information related to the video file.

Further, at time t4, when the cat frames out as shown in Fig. (D), the scene information is registered. Here, the text text is "love cat 〇〇", the start time ts is "t2", the end time te is "t4", the first coordinate is "p1_2", the second coordinate is "p2_2", and the type is "auto". Data set is additionally registered in the scene information management server 3 as another scene information related to the video file.

FIG. 5 is a flowchart showing a method of reproducing a desired scene of a video file using the scene information registered in the scene information management server 3 as described above, and mainly includes the scene information search unit 104 and the scene. The operation of the information reproduction unit 105 is shown.

In step S1, the scene search screen is displayed on the display panel 106. FIG. 6 is a diagram showing an example of a scene search screen, in which a search box 12 for inputting a search keyword, a “back” button 13 for returning to the previous scene, and a “next” for advancing to the next scene are shown together with the video playback frame 11. A "to" button 14 is provided. Further, on the right side of the video reproduction frame 11, the total number of search results "4" and the ordinal number "1" of the search results currently being reproduced are displayed in fractions, and a message display space 15 is secured below the total number of search results. ..

When the user confirms the keyword input to the search box 12 in step S2, the process proceeds to step S3, and the scene information table of the scene information management server 3 is referred to. The user can input the above-mentioned "pet dog △△" and "pet cat 〇〇" as keywords.

In step S4, it is determined whether or not a data set whose text text matches the keyword is registered. If it is not registered, the process proceeds to step S12, a message indicating that the scene matching the keyword is not registered is displayed in the message display space 15, and the process is terminated.

On the other hand, if a data set whose text text matches the keyword is registered, the process proceeds to step S5, and it is determined whether or not there are a plurality of such data sets. If there is only one, the process proceeds to step S13, and scene reproduction from the position specified by the start time ts of the data set to the position specified by the end time te is started. At this time, the text text is superimposed and displayed at the positions corresponding to the first and second coordinates p1 and p2 of the scene.

FIG. 7 is a diagram showing an example of scene reproduction when "pet dog Δ△" is input as a keyword. In the present embodiment, since there is a data set in which "pet dog △△" is registered as a text text, the start time ts_1, the end time te_1, and the first and second coordinates p1 and p2 are acquired. Then, the scene playback is started from the start time ts_1 when the "pet dog △△" appears, and the text "pet dog △△" is displayed at or near the positions defined by the first and second coordinates p1 and p2. ..

FIG. 8 is a diagram showing an example of scene reproduction when "love cat 〇〇" is input as a keyword. In the present embodiment, since there is a data set in which "love cat 〇〇" is registered as a text text, the start time, end time, first and second coordinates thereof are acquired. Then, the scene playback is started from the start time ts_1 when "love cat 〇〇" appears, and the text "love cat 〇〇" is displayed at or near the positions defined by the first and second coordinates p1 and p2. Will be done.

If the logical product of "pet dog △△" and "pet cat 〇〇" is input as the keyword, both "pet dog △△" and "pet cat 〇〇" appear as shown in FIG. Scene playback including the keyword is started from the time. Also, when the logical sum is input, all the scenes in which either "Pet dog △△" or "Pet cat 〇〇tama" appears are played.

Returning to FIG. 5, in step S5, if there are a plurality of data sets containing text matching the keyword, the process proceeds to step S6, and based on the data set having the earliest start time ts, from the start time ts to the end time te. Scene is played back with the text. In step S7, it is determined whether or not there is an operation for the "Next" button, and when such an operation is detected, the process proceeds to step S8. In step S8, a scene including the superimposed display of the text is reproduced from the start time to the end time based on the next data set having a start time ts.

In step S9, it is determined whether or not there is an operation for the "back" button, and when such an operation is detected, the process proceeds to step S10. In step S10, the scene including the superimposed display of the text is reproduced from the start time to the end time based on the data set immediately before the start time. In step S11, it is determined whether or not there is an operation to stop the scene reproduction, and until such an operation is detected, the process returns to step S7 and each of the above processes is repeated.

According to this embodiment, not only can the scene be objectively extracted from the video file and played back, but also the scene identifier can be displayed at an appropriate position in the scene, so that the content of each scene and the information reflected in each screen of each scene can be displayed. You will be able to visually recognize the meaning and name of.

FIG. 10 is a functional block diagram showing the configuration of the second embodiment of the scene information management device 1, and the same reference numerals as those above represent the same or equivalent parts, and thus the description thereof will be omitted. In the present embodiment, the scene information acquisition unit 101 includes the text acquisition unit 101d, the scene section acquisition unit 101e, and the display position acquisition unit 101f, and manually acquires the scene information of the video file, that is, subjectively. It is characterized by points.

The text acquisition unit 101d acquires the text text input by the user from the input operation unit 107 in each scene of the video file. The scene section acquisition unit 101e acquires the start time ts and the end time te that the user inputs from the input operation unit 107 as the scene section corresponding to the text. The display position acquisition unit 101f acquires the position information p1 and p2 input by the user from the input operation unit 107 as the display position of the text. The scene information registration unit 102 registers the input text text, start time ts, end time te, and data sets of position information p1 and p2 as scene information in the scene information management server 3.

FIG. 11 is a diagram showing an example of a registration screen when manually registering scene information in the second embodiment, which is displayed on the display panel 106 and operated by the user.

The video reproduction frame 21 reproduces a video file and a part of the scene. If the pet dog △△ appears at time t1 as shown in FIG. 12 (a) and the playback position is registered as the pet dog △△ scene, the user inputs the text “pet dog △△” in the text box 22. Enter and enter the time t1 in the Start Time box.

The user further inputs, for example, the upper right coordinate p1 of the "pet dog △△" in the first coordinate box 25, the lower left coordinate p2 in the second coordinate box 26, and the user name in the type box 27. To do.

After that, as shown in FIG. 6B, when the “pet dog △△” framed out at the time t2, the time t2 is input to the end time box 24. Finally, the input button 28 is checked and the above data set is registered in the server as scene information.

According to the present embodiment, a scene can be manually extracted from a video file and a scene identifier and its display position can be arbitrarily registered. Therefore, if registration of scene information is permitted for a large number of users, each user can register the scene. Depending on the subjectivity, scene information can be registered and displayed from various viewpoints.

In the first embodiment described above, the object detection and identification are performed on the image of the video file to acquire the text as the scene identifier, but the present invention is not limited to this. , A characteristic text may be acquired by voice recognition targeting an audio signal of a video file, or a characteristic text may be acquired from subtitle information.

1 ... video scene information management device, 2 ... video file server, 3 ... scene information management server, 101 ... scene information acquisition unit, 101a ... object detection unit, 101b ... object identification Unit, 101c ... Object identification model, 101d ... Text acquisition unit, 101e ... Scene section acquisition unit, 101f ... Display position acquisition unit, 102 ... Scene information registration unit, 103 ... Video File playback unit, 104 ... scene information search unit, 105 ... scene information playback unit, 106 ... display panel, 107 ... user input operation unit

Claims (9)

In the video scene management device that manages the scenes of the video file
A scene information acquisition means for acquiring a data set including a scene identifier, a scene section, and a position information of a scene identifier to be displayed in the scene as scene information for each scene of a video file.
A video scene information management device characterized in that it is provided with a scene information registration means for registering a data set of each scene information for each video file. A means of selecting scene information by keyword search for the scene identifier,
A means to play each scene in the video file based on the selected scene information,
The video scene information management device according to claim 1, wherein each scene to be reproduced is provided with a means for displaying a scene identifier based on the selected scene information. The scene information acquisition means is
A means of detecting objects from video files,
It has a means to identify the detected object,
Claim 1 or 2 is characterized in that the scene information registration means registers scene information in which the object identification result is a scene identifier, the object detection section is a scene section, and the object position is the position information of the scene identifier. The video scene information management device described in. The video file contains video and audio signals.
The video scene information management device according to claim 3, wherein the means for detecting the object is to extract the object from at least one of a video signal and an audio signal. The video file contains video signals, audio signals and subtitle information.
The video scene information management device according to claim 3, wherein the means for detecting the object is to extract the object from at least one of a video signal, an audio signal, and subtitle information. The video scene information management device according to any one of claims 3 to 5, wherein the data set includes type information indicating that the scene information is objectively generated. Further equipped with an input operation means for accepting a user's input operation,
The scene information acquisition means is
A means to acquire the scene section input from the input operation means, and
A means to acquire the scene identifier input from the input operation means, and
The video scene information management device according to claim 1 or 2, further comprising means for acquiring the position information of the scene identifier input from the input operation means. The video scene information management device according to claim 7, wherein the data set includes type information indicating that the scene information is subjectively generated by the user. According to any one of claims 1 to 8, the video file is a 360 ° video, and the position information of the scene identifier is a coordinate value on the coordinates obtained by expanding the 360 ° video in two dimensions. The described video scene information management device.

Images