JP6091551B2 - Display control apparatus, display control method, and program - Google Patents

Description

The present invention relates to a display control device , a display control method, and a program , and more particularly, to a technique suitable for use in order to easily find a scene important for a user in consideration of user preferences.

  In recent years, with the development of video analysis technology and voice recognition technology, videos with abundant metadata are increasing. Information that can be given as metadata includes, for example, information on who the person is in and information on where a child can laugh. There has been proposed a method that makes it possible to quickly find a desired scene by using metadata, or to view only a scene that matches an individual's preference (see Patent Documents 1 and 2).

  On the other hand, with the development of communication networks in recent years, a viewing mode in which a video stored in a video distribution device is streamed and played on a display device at hand by a plurality of users through the network is spreading. In the future, it is considered that the video distribution system will need to provide a user interface that can easily find a scene important for the user in consideration of each user's preference.

  In addition, it is convenient for the user to refer to a frame cut out from the video in order to search for an important scene for the user from the video. That is, the user moves the slider and a representative frame is displayed on the video display screen. In parallel with streaming reproduction, a method for acquiring a plurality of video frame data existing within a certain prefetch target period has been proposed (see Patent Document 3).

JP 2001-292402 A JP 2007-208651 A JP 2006-287875 A

  In an existing display device, a user interface called a seek bar is used. The seek bar shows the amount of playback of the entire video projected onto the length on the display screen. Further, when the user moves the slider on the seek bar to a certain position by some operation device, the video is reproduced from the time point corresponding to the position. That is, by moving the slider, the video can be reproduced from an arbitrary time point.

  Usually, since the passage of time of video and the position on the bar are associated with each other in a proportional relationship, it is possible that a scene important for the user is biased on the seek bar. Then, when moving the slider, it is necessary to move the slider largely in order to skip an area that is not important to the user. On the contrary, in order to finely seek an area where important scenes are gathered, a fine operation of the slider is required. In this case, there is a problem that convenience for video reproduction for the user is impaired.

  The present invention has been made in view of the above-described problems, and an object thereof is to improve convenience related to video reproduction for a user.

The display control apparatus according to the present invention includes a first acquisition unit that acquires video data, and feature information based on at least one of a feature relating to video content and an audio feature related to the video data acquired by the first acquisition unit. Display control means for displaying on the display section a bar for allowing the user to specify the playback position of the video data acquired by the first acquisition means, and the display data is equal in time to the video data. The lengths of the different portions of the bar corresponding to the different length periods differ depending on the feature information based on at least one of the features relating to the content of the video and the features of the audio acquired by the second acquisition means. The display control means for displaying the bar on the display unit .

  According to the present invention, it is possible to improve convenience related to video reproduction for a user.

It is a figure explaining the specific example of the seek bar generally used. It is a figure which shows embodiment of this invention and demonstrates the specific example of metadata. It is a figure which shows embodiment of this invention and demonstrates the specific example of user preference data. It is a block diagram which shows the example of 1 structure of the video delivery system of embodiment of this invention. It is a sequence diagram which shows embodiment of this invention and shows an example of the whole flow. It is a figure explaining the specific example of the user preference data described by XML. It is a flowchart explaining the detail of the delivery preparation in FIG. It is a figure which shows embodiment of this invention and demonstrates the specific example of frame data. It is a figure which shows embodiment of this invention and demonstrates the specific example of a non-linear seek bar. It is a figure explaining the specific example of user interface data. It is a flowchart explaining the detail of the delivery in FIG. It is a figure which shows a mode that frame data are dynamically created from the periphery of a frame.

First, a general seek bar will be described. In an existing display device, a user interface called a seek bar is often used.
FIG. 1 is a diagram illustrating an example of a general seek bar. In this user interface, the bars B are arranged in a straight line. This bar B shows the amount of playback of the entire video projected onto the length on the display screen. Usually, the left end corresponds to the beginning of the video, the right end corresponds to the end of the video, and the passage of time of the video and the position on the bar are often correlated in a proportional relationship.

  When the user requests video playback, the video is displayed on the video display screen D. During the reproduction of the video, the slider SL above the bar B moves on the bar B so as to correspond to the time point being reproduced. Further, when the user moves the slider SL to a certain position by some operation device, the video is reproduced from the time point corresponding to the position. That is, by moving the slider SL, it is possible to reproduce the video from an arbitrary time point.

  Such a general seek bar has the following problems. For example, it is assumed that the user has a preference to see a scene in which the user is shown. At this time, when the scene in which the user is shown is projected onto the seek bar, it is assumed that the scenes are arranged as I1 to I10 in FIG. Usually, since the correspondence between the bar B and the time point is in a proportional relationship, it is possible that scenes important for the user are biased on the seek bar.

  Then, when the slider SL is moved, the slider SL must be largely moved in order to skip an area that is not important to the user. Conversely, in order to finely seek an area where important scenes are gathered, a fine operation of the slider SL is required. In this case, there is a problem that it is impossible to reproduce a video that is convenient for the user.

Hereinafter, embodiments of the present invention will be described.
First, terms used in the following description are defined.
“Metadata” is data that accompanies the video, and is data that indicates the characteristics of the video, such as what scenes exist at which time points in the video. Specifically, among face and object recognition information, landscape recognition information, zoom operation information applied to the video camera that captured the video, volume change information, audio recognition information, tagging information by the user himself Including at least one. For example, the metadata includes information that a child's face starts to appear at a time point of 0 minute 14 seconds 16 of the video. There are both cases where the metadata is embedded in the video and cases where the metadata is managed separately from the video. In the present embodiment, the latter case will be described. FIG. 2 shows an example of metadata.

  “User preference data” is a score given to various information given as metadata. This score represents the degree of preference the user has for each piece of metadata. FIG. 3 shows an example of user preference data.

  An “important scene” represents a time point of a video corresponding to a type of metadata having a high score in user preference data.

  FIG. 4 shows a configuration of a typical video distribution system in the embodiment of the present invention. In the present embodiment, the video distribution device 100 and one or more display devices 200 are used. The video distribution device 100 and the display device 200 are connected via a network 300. The user views the video stored in the video distribution device 100 via the network by operating the display device 200.

  The components of the video distribution device 100 will be described. The receiving unit 101 and the transmitting unit 109 exchange information with the display device 200. The video database 103 stores a video corresponding to the video ID. Here, the video ID is an ID for identifying a video. The metadata database 104 stores video metadata corresponding to the video ID.

  The user preference data database 105 stores user preference data transmitted and acquired from the display device 200. The database management unit 102 manages the video database 103, the metadata database 104, and the user preference data database 105. In the present embodiment, a method using user preference data transmitted from the display device 200 is used. However, the database management unit 102 may estimate user preferences and create user preference data.

  The important scene determination unit 106 creates important scene information of the video based on the metadata weighted by the user preference data. “Important scene information” is a collection of information on how important scenes there are at which time points in a video. The user interface data creation unit 107 creates user interface data customized so as to be suitable for searching for an important scene for the user based on the obtained important scene information. The frame data creation unit 108 creates frame data from the video based on the important scene information, video, and seek information. Here, the “frame data” includes a frame extracted from the video and data related to the frame.

Next, components of the display device 200 will be described.
The operation unit 201 is connected to the operation device 400 and receives an operation from the user. The operation unit 201 also creates seek information that is information indicating which frame in the video the user has selected. The display unit 203 outputs the video received from the video distribution apparatus 100 to the display device 500 for display.

  In addition, the display unit 203 receives user interface data from the video distribution apparatus 100, and creates and displays a user interface based on the user interface data. Further, the display unit 203 receives frame data from the video distribution device 100 and appropriately displays a frame when the user performs a seek operation. The transmission unit 202 and the reception unit 204 exchange information with the video distribution device.

Next, the flow of processing in this embodiment will be described with reference to FIG.
First, the display device 200 transmits user preference data of the user to the video distribution device 100 (S100). The transmitted user preference data is stored in the user preference data database 105 of FIG. 4 (S200). A specific example of user preference data to be transmitted will be described later.

  Next, the display device 200 transmits the ID of the desired video to the video distribution device 100 (S300). The video distribution apparatus 100 performs distribution preparation (S400) based on this, and transmits the created frame data and user interface data to the display apparatus 200. Details of the distribution preparation (S400) will be described later.

  Subsequently, the display apparatus 200 notifies the video distribution apparatus 100 of a distribution start request (S500). Based on this, the video distribution apparatus 100 distributes the video (S600) and transmits the video to the display apparatus 200. Details of distribution (S600) will be described later. The above is the flow of processing in this embodiment.

Next, a specific example of user preference data transmitted from the display device 200 to the video distribution device 100 will be described with reference to FIG. Here, the user preference data is expressed in the XML format (extensible markup language format) as a data format when received from the outside, or it is binarized, and the following contents are described. ing.
(1) ID for identifying the user, (2) metadata class and type, and (3) score given to the metadata.

  FIG. 6 shows an example in which a user having an ID “0000123” sets his / her preferences for four types of metadata. Here, the first description of the user preference data means that a score “5” is given to “night scene” belonging to the “landscape” class. The second description means that the score “4” is given to the ID “0000123” (that is, the user himself) belonging to the “face” class.

  The third description means that a score “4” is given to “favorites” belonging to the “user-defined” class. Here, the user-defined class is a class to which metadata tagged by the user himself belongs. The fourth description means that a score “−3” is given to “noise” belonging to the “sound” class. Here, a negative score means a preference of not wanting to watch this scene.

Next, a specific example of the delivery preparation (S400) method in FIG. 5 will be described in detail with reference to FIGS.
When the distribution preparation is started (S701), the video distribution apparatus 100 receives the video ID from the display apparatus 200 (S702). The database management unit 102 checks whether there is a video corresponding to this video ID and confirms the validity of the video ID (S703). If the video ID is not valid, an error is notified to the display device 200 (S704), and the preparation for distribution ends (S705). If the video ID is valid, the video corresponding to the video ID is read from the video database 103 (S706).

  Subsequently, the database management unit 102 checks whether or not the metadata given the score in the user preference data exists in the metadata database 104 (S707). If the corresponding metadata does not exist, it is checked whether the metadata can be created by the video distribution apparatus 100 (S708).

  If the metadata can be created, a metadata creation process for dynamically creating the metadata from the video is executed (S711), and the metadata database 104 is updated (S712). If the metadata cannot be created, an error is notified to the display device 200 (S709), and the reproduction preparation is terminated (S710). Of course, this method shown in FIG. 7 can be omitted, and when the metadata does not exist, the user preference data corresponding to the metadata can be ignored and the processing can be continued.

  In this way, after extracting the metadata (S713), the important scene determination unit 106 creates important scene information indicating which scene is important to whom at which time point of the video based on the extracted metadata. (S714). At this time, according to the score given in user preference data, the height of the importance of a scene can be weighted.

  Based on the important scene information, the frame data creation unit 108 creates frame data in an important scene (S715), and the user interface data creation unit 107 generates user interface data optimized for the display device 200. Create (S716). Then, the transmission unit 109 sends the frame data and the user interface data to the reception unit 204 of the external display device 200 connected via the network 300, the video distribution preparation is completed (S717), and the process is finished ( S718).

Next, a specific example of frame data transmitted from the video distribution device 100 to the display device 200 will be described.
As described above, the frame data includes a frame extracted from the video and data related to the frame. An example of the frame data is shown in FIG. The frame is extracted from the video and given an ID for identification. The data regarding the frame is expressed using, for example, the XML format of FIG. The XML in FIG. 8 describes, for each frame, information as to which time point of the video the frame was created from and the class and type of metadata that caused the frame to be extracted.

  In FIG. 8, a method of collecting all data related to a plurality of frames into one XML is used, but a method of preparing one XML for each frame may be used. By transmitting the frame data configured as described above to the display device 200, the frame can be arranged at a corresponding place on the seek bar on the display device 200 side. This frame data is transmitted when preparing for video distribution and when the user performs a seek operation (described later).

  FIG. 9 shows an example of a typical user interface in the present embodiment. The seek bar in FIG. 9 has a small time width corresponding to the moving distance of the slider in an area where many scenes important for the user are gathered. Conversely, for areas where there are few important scenes for the user, the time width corresponding to the moving distance of the slider is set large. Thus, by providing a user interface in the form of a seek bar that is non-linear with respect to time, it is possible to reproduce video according to user preferences.

Next, an example method for creating a non-linear seek bar will be described.
First, the total time of the video is equally divided by an appropriate number N. Next, for each divided section, an appropriate score is given with the number and types of important scenes existing in the section as parameters. Various methods can be used to give the score, for example, a method that gives a high score to scenes that are important to the user, and a method that gives a negative score to scenes that the user does not want to watch. Is possible.

Then, the seek bar is divided so as to be proportional to the score of each section and assigned to each section. In this way, a non-linear seek bar can be constructed.
When the division number N is 6 and the score calculation method is “add 1 to the number of important scenes existing in each section”, the seek bar shown in FIG. .

Next, a specific example of user interface data transmitted from the video distribution apparatus 100 to the display apparatus 200 will be described with reference to FIG.
In FIG. 10, the user interface data is expressed in the XML format, and the following contents are described.
(1) User interface class (here, seek bar), (2) total time of video, (3) correspondence between video time point and seek bar position. The display device 200 can receive such user interface data and create a user interface.

Next, the method of distribution (S600) in FIG. 5 will be described in detail with reference to FIGS.
As described above, the video distribution device 100 transmits frame data corresponding to an important scene to the display device 200 prior to video distribution.
When distribution is started (S1101), the video distribution apparatus 100 receives a distribution start request from the display apparatus 200 (S1102). Next, the video distribution apparatus 100 first sets the reproduction start position (S1103), and transmits the video to the display apparatus 200 (S1104).

  Here, it is assumed that the user enters a seek operation and selects one of the frames in the video (S1105). Information indicating which frame is selected is sent as seek information from the display device 200 to the frame data creation unit 108. The frame data creation unit 108 creates frame data at a time interval shorter than the time interval between the previously extracted frames within a certain range before and after the selected frame (S1106), and transmits the frame data to the display device 200. (S1107). It is determined whether the frame selection operation by the user has been completed (S1108). If completed, the playback start position is corrected (S1109), and the video is retransmitted to the display device 200 (S1110). If the user's frame selection operation is not completed in S1108, the process returns to S1105.

  In S1111, it is determined whether or not the reproduction is finished. If the reproduction is not completed, the process returns to S1105, and if the reproduction is completed, the process proceeds to S1112 to perform an end process.

Next, FIG. 12 shows an operation example when the user performs a seek operation.
The positions of frame data corresponding to important scenes sent to the display device 200 prior to video distribution are I1 to I10.

  FIG. 12 shows that when the user slides the slider SL to the position I6, frame data is dynamically created around the slider SL and arranged in J1 to J3 on the seek bar. As a result, the user can seek more finely.

  When the frame selection operation by the user is completed, the video distribution apparatus 100 corrects the reproduction start position (S1109) and redistributes the video (S1110). In this way, by transmitting frame data in stages, the user can easily seek scenes that are important to the user over the entire area of the video, and more finely without wasting communication bandwidth. Can seek.

Next, a method for extracting a frame to be sent when the user enters a seek operation will be described.
Several methods can be applied to create frame data to be sent when the user enters a seek operation. As a first method, there is a method of creating several pieces of frame data at a time point separated from a selected frame by a certain time. For example, when the selected frame is extracted from the 22.50 second of the video, from the time point separated by 2.00 seconds before and after 22.50 seconds and the time point separated by 1.00 seconds before and after. Create frame data.

  As a second method, there is a method of extracting a frame corresponding to the preference of a user other than the user who is currently viewing the video. Hereinafter, a user who wants to view a video is referred to as “viewing user”, and a user other than the viewing user is referred to as “other user”. As described above, the important scene determination unit 106 creates the important scene information of the video based on the user preference data and the metadata.

  At this time, not only important scene information for the viewing user is created from the viewing data of the viewing user, but also important scene information for the other user is created from the preference data of the other user. This makes it possible to create frame data corresponding to a scene important for other users. As a result, when the viewing user performs a seek operation, an effect of noticing a scene that the viewing user alone did not notice can be expected. However, from the viewpoint of personal information protection, it is desirable that the range of other users can be set in advance.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (computer program) that implements the functions of the above-described embodiments is supplied to a system or apparatus via a network or various computer-readable storage media. Then, the computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program.

  DESCRIPTION OF SYMBOLS 100 Video delivery apparatus, 101 Receiving part, 102 Database management part, 103 Video database, 104 Metadata database, 105 User preference data database, 106 Important scene determination part, 107 User interface data creation part, 108 Frame data creation part, 109 Transmitter, 200 Display Device, 300 Network, 400 Operation Device, 500 Display Device

Claims (18)

First acquisition means for acquiring video data;
Second acquisition means for acquiring feature information based on at least one of a feature related to video content and a feature of audio related to the video data acquired by the first acquisition means;
Display control means for causing a display unit to display a bar for allowing a user to specify a reproduction position of video data acquired by the first acquisition means , corresponding to different periods of the same time length of the video data; The different bar lengths are displayed on the display unit according to feature information based on at least one of the feature related to the content of the video and the feature of the audio acquired by the second acquisition unit. display control apparatus characterized by having said display control means for. The display control apparatus according to claim 1 , wherein the feature information acquired by the second acquisition unit is metadata created from video data acquired by the first acquisition unit . The characteristic information acquired by the second acquisition unit, the display control device according to claim 1 or 2, characterized in that it comprises information for specifying a predetermined scene in the video data. Wherein the predetermined scene, according to claim 3, wherein the characteristic information acquired by the second acquisition means, to be specified based on the data representing the importance of a scene which is specified based on the feature information The display control apparatus according to 1. The display control device according to claim 4 , wherein the data representing the importance represents an importance set for each user with respect to a scene specified based on the feature information . Wherein the predetermined scene, the out of the scene which is specified based on the feature information, according to claim 4 or 5 importance represented by the data representing the importance characterized in that it is a more scenes predetermined threshold The display control apparatus according to 1. In the display control means, the lengths of different portions of the bar corresponding to the different periods of the same time length of the video data differ depending on whether the predetermined scene is included in the different periods. The display control apparatus according to claim 3 , wherein the bar is displayed on the display unit. The display control means, wherein the lengths of the different portions of the bar corresponding to the different periods of the equal time length of the video data are different depending on the importance of the predetermined scene included in the different periods. The display control apparatus according to claim 3 , wherein a bar is displayed on the display unit. The display control means, when the predetermined scene is included in a first period of the video data and the predetermined scene is not included in a second period having a time length equal to the first period, any of claims 3 to 8, characterized in that to display a longer the bar than the length of the portion length of the portion corresponding to the first period corresponds to the second period of the bar on the display unit 1 The display control device according to item. The display control means is configured such that the total importance of the predetermined scene included in the first period of the video data is equal to or greater than a predetermined value , and is included in the second period having a time length equal to the first period . When the total importance of the scene is smaller than the predetermined value, the length of the part corresponding to the first period of the bar is longer than the length of the part corresponding to the second period of the bar. the display control device according to any one of claims 3 to 8, characterized in that on the display unit. The display control unit includes a case where a predetermined number or more of the predetermined scenes are included in the first period of the video data, and a predetermined number or more of the predetermined scenes are not included in a second period having a time length equal to the first period. to, claim 3, characterized in that to display a longer the bar than the length of the portion length of the portion corresponding to the first period of the bars corresponding to the second period of the bar on the display unit The display control apparatus according to any one of 1 to 8 . The display control apparatus according to claim 3 , wherein the predetermined scene is a scene in which a specific object is included in a video . The feature information acquired by the second acquisition unit includes recognition information of at least one of a face, an object, and a landscape included in the video, information about a zoom operation applied to the camera that captured the video, and information about a change in volume. and the display control device according to any one of claims 1 to 12, characterized in that it comprises at least one of identification information of the speech. 2. A reproduction control unit that reproduces video data acquired by the first acquisition unit from a reproduction position designated by a user based on a bar displayed on a display unit by the display control unit. 14. The display control device according to any one of items 13 to 13. A first acquisition step of acquiring video data;
A second acquisition step of acquiring feature information based on at least one of a feature related to video content and a feature of audio related to the video data acquired in the first acquisition step;
A display control step of displaying on the display unit a bar for allowing the user to specify the playback position of the video data acquired in the first acquisition step, corresponding to different periods of the same time length of the video data; The bars having different lengths according to the feature information based on at least one of the feature related to the content of the video and the feature of the sound acquired by the second acquisition step are displayed on the display unit. A display control method for a display control device . The display control method according to claim 15, wherein the feature information acquired in the second acquisition step is metadata created from the video data acquired in the first acquisition step. The display control method according to claim 15 or 16, wherein the characteristic information acquired in the second acquisition step includes information for specifying a predetermined scene in the video data. The program for functioning a computer as each means of the display control apparatus of any one of Claims 1 thru | or 14.

Images