JP2024024479A - Information processing device, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To facilitate metadata editing to make it easier for a viewer to focus on a particular portion of a video when viewing it.

SOLUTION: An image processing device acquires a position input that specifies a position on a video image. On the basis of the position input, the image processing device generates position information indicating the position on the video image and time information indicating the playback position of the video image. The image processing device generates at least one of the position information and the time information further on the basis of at least one of the content of the video image at the position on the video image specified by the position input and the sound associated with the video image corresponding to the playback position of the video image that is the subject of the position input. The image processing device records the position information and the time information.

SELECTED DRAWING: Figure 2

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present disclosure relates to an information processing device, an information processing method, and a program, and particularly relates to a method of recording metadata related to distributed video.

In recent years, video distribution services have been widely used. For example, in educational settings, due to the impact of the new coronavirus, real-time distribution and on-demand distribution of lecture videos are being introduced.

Additionally, various techniques related to metadata added to videos have been proposed. Such technology contributes to increasing the efficiency of video production and improving the learning effect for viewers of lecture videos. For example, Patent Document 1 discloses dividing video data based on "detection of switching of presentation slides" and "audio analysis of lecturer's utterance content". Further, Patent Document 1 discloses recording information indicating each divided section. By using the technology described in Patent Document 1, viewers can quickly reproduce sections related to a desired topic.

Japanese Patent Application Publication No. 2009-147538

Viewers may rewatch videos for the purpose of reviewing lectures or the like. At this time, it is convenient for the viewer to be able to easily focus on important parts. For example, when rewatching a lecture video, it becomes easier to focus on "important parts" and "parts you don't understand," which can further improve the learning effect.

The present disclosure provides a technology that facilitates metadata editing to make it easier for a viewer to focus on a specific part when viewing a video.

An information processing device according to an embodiment of the present disclosure includes the following configuration. That is,
an acquisition means for acquiring a position input specifying a position on the moving image;
A generating means for generating position information indicating a position on the moving image and time information indicating a playback position of the moving image based on the position input, the generating means generating at least one of the position information and the time information by the at least one of the content of the moving image at the position on the moving image specified by the position input, and the sound related to the moving image corresponding to the playback position of the moving image that is the target of the position input. further generating means for generating based on;
recording means for recording the location information and the time information;
Equipped with.

It is possible to easily edit metadata to make it easier for viewers to focus on specific parts when viewing a video.

FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing device according to an embodiment. FIG. 1 is a diagram illustrating an example of a functional configuration of an information processing device according to an embodiment. 1 is a flowchart of an information processing method according to an embodiment. FIG. 3 is a diagram illustrating an example of a method for inputting a position and generating position information. FIG. 2 is a diagram illustrating a method of generating location information according to an embodiment. FIG. 3 is a diagram illustrating an example of a method for inputting a position and generating position information. FIG. 2 is a diagram illustrating a method of generating location information according to an embodiment. FIG. 3 is a diagram illustrating an example of a method for inputting a position and generating time information. FIG. 2 is a diagram illustrating a time information generation method according to an embodiment. FIG. 2 is a diagram illustrating a time information generation method according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claims. Although a plurality of features are described in the embodiment, not all of these features are essential, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted.

FIG. 1 shows the overall configuration of an information processing device according to an embodiment of the present disclosure. The information processing device 101 is a computer device such as a PC (personal computer), a smartphone, or a tablet terminal device. The information processing device 101 includes a CPU 102, a ROM 103, a RAM 104, a storage section 105, a display section 106, and an operation section 107. Each of these configurations is connected to each other by a bus 108.

The CPU 102 can control the operation of each part within the information processing apparatus 101 according to the contents of the ROM 103 or RAM 104. Further, the CPU 102 can execute programs loaded into the RAM 104. ROM 103 is a read-only memory. The ROM 103 can store a boot program, firmware, various processing programs for implementing processes described below, and various data. RAM 104 is a work memory. The RAM 104 can temporarily store programs and data for the CPU 102 to process. Various processing programs and data are loaded into the RAM 104 by the CPU 102. The storage unit 105 is a recording medium for storing a large amount of changeable data. The storage unit 105 may be, for example, a hard disk drive or a solid state drive.

In this embodiment, the storage unit 105 can store metadata that will be described later. Furthermore, the storage unit 105 can store moving image data. A moving image consists of multiple frames. Further, a moving image can include multiple frame images. In this specification, a moving image may be referred to as a video. Furthermore, the storage unit 105 can store sound data related to moving images. For example, the storage unit 105 may store a video file including moving image data and sound data.

The display unit 106 is a screen such as a liquid crystal screen or a touch panel screen. The display unit 106 can display the processing result by the CPU 102 as an image or text. Furthermore, the display unit 106 can display moving images. The moving image displayed by the display unit 106 may be stored in the storage unit 105 or may be transmitted from an external device via a network. Furthermore, the display unit 106 can display information based on metadata as described below. Note that when the display unit 106 has a touch panel screen, the display unit 106 can notify the CPU 102 of an operation input input by the user operating the touch panel screen.

The operation unit 107 is a user interface. The operation unit 107 may be, for example, a keyboard, a mouse, a button, a touch panel screen, or the like. The user can input various instructions to the CPU 102 by operating the operation unit 107. In this embodiment, a user who is a viewer of a moving image can operate the operation unit 107 in order to input a position on the moving image.

FIG. 2 shows a logical configuration of the information processing device 101 according to an embodiment of the present disclosure. The information processing device 101 includes an acquisition section 201, a position determination section 202, a time determination section 203, and a recording section 204. Such an information processing device 101 can be realized by a computer including a processor and a memory as shown in FIG. That is, by a processor such as the CPU 102 executing a program stored in a memory such as the ROM 103, the RAM 104, or the storage unit 105, the functions of each unit shown in FIG. 2 can be realized. However, some or all of the functions of the information processing device 101 may be realized by dedicated hardware. Further, the image processing device according to an embodiment of the present disclosure may be configured by, for example, a plurality of information processing devices connected via a network.

The acquisition unit 201 acquires a position input specifying a position on a moving image. The user can specify a specific position of a specific frame image of a moving image when the specific frame image is displayed on the display unit 106. The position input by the user may be an input specifying one coordinate or may be an input specifying a specific area. The method of inputting the area will be described later.

The position determination unit 202 generates position information indicating the position on the moving image based on the position input acquired by the acquisition unit 201. This position information may indicate the position on the moving image designated by the position input. On the other hand, this position information may indicate another position determined by the position determination unit 202 based on the position on the moving image specified by the position input. For example, as will be described later with reference to FIGS. 5 and 7, the position determining unit 202 can generate position information based on the content of the moving image at a position on the moving image specified by the position input. Further, the position determination unit 202 may generate position information based on sound related to a moving image corresponding to the playback position of the moving image that is the target of the position input. More specifically, the position determining unit 202 determines the position on the moving image specified by the position input, the content of the moving image at this position, and the moving image corresponding to the playback position of the moving image that is the target of the position input. can be corrected based on at least one of the following:

The time determination unit 203 generates time information indicating the playback position of the moving image based on the position input acquired by the acquisition unit 201. This time information can indicate a specific time in a moving image. For example, the time information may represent the elapsed time from the start of the video to a specific point in time. Further, the time information may indicate the time of a specific frame in the moving image. For example, the time information may be information indicating the display timing of frame images in a moving image. The information indicating the display timing may be information indicating the time from the start of the moving image until the frame image is displayed, or may be information indicating the number of the frame image.

The time information generated by the time determination unit 203 may indicate the playback position of the moving image that is the target of the position input. For example, the time information may represent the elapsed time from the start of the video to the time when the user inputs the position. Further, this time information may indicate the time of the frame whose position is input by the user. On the other hand, this time information may indicate another time determined by the time determination unit 203 based on the playback position of the moving image that is the subject of position input. For example, as will be described later with reference to FIG. 9, the position determination unit 202 may generate time information based on the content of the moving image at a position on the moving image specified by the position input. Further, as will be described later with reference to FIG. 10, the position determining unit 202 generates time information based on the sound associated with the video corresponding to the playback position of the video that is the target of the position input. Good too. More specifically, the time determination unit 203 determines the playback position of the video that is the target of the position input based on the content of the video at the position on the video specified by the position input and the video corresponding to this playback position. The image-related sound can be corrected based on at least one of the following. Figures 9 and 10 illustrate such an example.

In this way, the position determining unit 202 and the time determining unit 203 generate position information indicating the position on the moving image and time information indicating the playback position of the moving image, based on the position input acquired by the acquiring unit 201. Here, at least one of the position information and the time information is the content of the video at the position on the video specified by the position input, and the video corresponding to the playback position of the video that is the target of the position input. and/or the associated sound. For example, the position determining unit 202 can determine the content of the moving image at the position on the moving image specified by the position input, and the sound related to the moving image corresponding to the playback position of the moving image that is the target of the position input. Location information can be generated further based on at least one. At this time, the time determining unit 203 may generate time information indicating the playback position of the moving image that is the target of the position input. In another example, the time determination unit 203 determines the content of the moving image at the position on the moving image specified by the position input, and the sound related to the moving image corresponding to the playback position of the moving image that is the target of the position input. The time information can be further generated based on at least one of the following. At this time, the position determination unit 202 may generate position information indicating the position on the video image specified by the position input.

The recording unit 204 records the position information generated by the position determining unit 202 and the time information generated by the time determining unit 203. The recording unit 204 can record this information as metadata for the moving image. Further, the recording unit 204 can record this information in an arbitrary storage medium such as the storage unit 105.

The playback unit 205 plays back the moving image. The playback unit 205 can display a moving image recorded in a storage medium such as the storage unit 105 on a screen such as the display unit 106, for example. The user can input the position of the moving image while viewing the screen displayed on the display unit 106. Furthermore, the playback unit 205 can control the display of moving images based on metadata such as position information and time information recorded by the recording unit 204. A specific display control method will be described later.

However, the configuration of the information processing device 101 is not limited to the above. For example, the information processing device 101 may be connected to a network. Further, moving images may be distributed to the information processing apparatus 101 from a streaming server on a network. In this case, the recording unit 204 is not essential. Further, the information processing apparatus 101 can record metadata such as location information and time information in the streaming server via the network. Further, the information processing apparatus 101 can acquire such metadata from the streaming server. Even with such a configuration, the information processing apparatus 101 can operate.

FIG. 3 shows a flowchart of an information processing method according to an embodiment. The processing shown in FIG. 3 can be performed by the information processing apparatus 101. Further, the processing shown in FIG. 3 can be performed while the playback unit 205 is playing back the moving image.

In S301, the acquisition unit 201 acquires a position input for a moving image from the user. Note that, as described above, the acquisition unit 201 may further acquire attribute information related to position input (for example, information indicating the type of mark).

In S302, the position determining unit 202 generates position information based on the position input acquired in S301. This location information is stored as metadata. A detailed method of generating position information will be described later.

In S303, the time determination unit 203 generates time information based on the position input acquired in S301. This time information is stored as metadata. A detailed time information generation method will be described later.

In S304, the recording unit 204 records the position information generated in S302 and the time information generated in S303 in the metadata storage area. The recording unit 204 can record this information as metadata of the moving image being played back.

In S305, the recording unit 204 increments the index n indicating the metadata identifier by one in preparation for a position input by the user.

In S306, the recording unit 204 determines whether the index n has reached the maximum value. This maximum value indicates the maximum number of metadata that can be stored. If the index n is the maximum value, the process according to the flowchart shown in FIG. 3 ends. Otherwise, the process returns to S301.

FIG. 4(D) shows an example of metadata recorded by the recording unit 204. When a user inputs a location, location information (e.g., coordinates), time information (e.g., time in a video image), and the type of mark entered are stored in a storage area for each index, which is a metadata identifier. .

(Method of generating location information based on area specification)
In the following, a method for generating position information when the user's position input specifies a rectangular area will be described. As already explained, in this case, the position determining unit 202 may generate position information indicating the rectangular area specified by the user. On the other hand, in the following example, the position determining unit 202 generates position information based on the content of the moving image at the position on the moving image specified by the position input. For example, the position determining unit 202 can generate position information so as to indicate an area that includes information on the moving image at a position on the moving image specified by the position input. For example, the position input may specify an area that includes at least part of the information in the moving image. In this case, the position determining unit 202 can generate position information indicating a region that includes the entire information (for example, the entire continuous description). In the example below, the position determination unit 202 calculates coordinates for identifying such an area. Here, the type of information on the moving image is not particularly limited. The information on the moving image may be, for example, a continuous description, illustration, diagram, or table. In the example below, the position determination unit 202 calculates coordinates for identifying such an area.

Such an embodiment will be described with reference to FIGS. 4(A) to 4(D). In this example, position information is generated to indicate an area that includes continuous descriptions on a moving image. As shown in FIG. 4(A), the user pays attention to the descriptive content "ABCDE" written on the blackboard 402 in the delivered video 401 of the lecture. Then, the user inputs a rectangular frame 403 that overlaps the description content "ABCDE" as a position input, and further inputs a mark 404 indicating that this description is important. In this manner, the user can input a position indicating the portion of interest in the moving image. Moreover, this position input may point to a part of a continuous description in a moving image.

Furthermore, the user can input attribute information (mark 404 in this example) indicating the type of the portion of interest in association with the position input. Note that the type of attribute information is not particularly limited. For example, the attribute information may be selectable from a plurality of types. For example, it may be possible to input an "I don't understand" mark indicating that the user does not understand the description as the attribute information.

In such an example, the position determining unit 202 can determine an area that includes information (for example, a continuous description) on the moving image by enlarging or reducing the area specified by the position input. An example of the process by which the position determining unit 202 generates position information in S302 will be described below with reference to the flowchart shown in FIG. 5. In the process shown in FIG. 5, when descriptive information exists in an area adjacent to the area indicated by the position input, the position determining unit 202 expands the area to include the adjacent area.

In S501, the acquisition unit 201 detects the vertex coordinates of the area (in this example, the rectangular frame 403) indicated by the position input. In this example, as shown in FIG. 4B, a left horizontal coordinate x1, a right horizontal coordinate x2, an upper vertical coordinate y1, and a lower vertical coordinate y2 are detected.

In S502, the position determining unit 202 analyzes the image area above the area indicated by the position input. Through this analysis, the position determining unit 202 can determine whether descriptive information exists in this upper image area. In FIG. 4B, this upper image region is a region having a width α above the region indicated by the position input, represented as region 405. Specifically, the area 405 is an area surrounded by coordinates (x1, y1), (x2, y1), (x1, y1+α), and (x2, y1+α). Here, α is an arbitrary constant value.

The specific analysis method is not particularly limited. For example, position determiner 202 can scan RGB values within region 405. If an RGB value different from the RGB value of the background (for example, the blackboard 402) is detected during scanning, the position determining unit 202 can determine that descriptive information exists. Further, the position determining unit 202 may perform the analysis using a method based on pattern recognition. By using pattern recognition, descriptive information can be detected with even higher accuracy. For example, if a character is detected in the area 405, the position determining unit 202 can determine that descriptive information exists in the area 405.

In S503, the position determination unit 202 determines whether descriptive information is present in the upper image area. If there is descriptive information, the process advances to S505; otherwise, the process advances to S504. In S504, the position determining unit 202 replaces the upper vertical coordinate y1 with y1+α, and then the process returns to S502.

In S505, the position determination unit 202 analyzes the image area below the area indicated by the position input. The analysis method is the same as S502. In S506, the position determining unit 202 determines whether descriptive information is present in the lower image area. If there is descriptive information, the process advances to S508; otherwise, the process advances to S507. In S507, the position determining unit 202 replaces the lower vertical coordinate y2 with y2-α, and then the process returns to S505.

In S508, the position determination unit 202 analyzes the image area on the right side of the area indicated by the position input. The analysis method is the same as S502. In S509, the position determination unit 202 determines whether descriptive information is present in the right image area. If there is descriptive information, the process advances to S511; otherwise, the process advances to S510. In S510, the position determining unit 202 replaces the right horizontal coordinate x2 with x2+α, and then the process returns to S508.

In S511, the position determination unit 202 analyzes the image area on the left side of the area indicated by the position input. The analysis method is the same as S502. In S512, the position determination unit 202 determines whether descriptive information is present in the left image area. If there is descriptive information, the process advances to S514; otherwise, the process advances to S513. In S513, the position determining unit 202 replaces the left horizontal coordinate x1 with x1-α, and then the process returns to S511.

In S514, the position determination unit 202 generates position information indicating x1 calculated in S504, x2 calculated in S507, y1 calculated in S510, and y2 calculated in S513. In this way, the processing according to the flowchart of FIG. 5 ends.

Up to this point, the case where the position input indicates a rectangular area has been described. For example, such a method can be used when the user specifies a rectangular area so as to overlap the area of interest by operating the mouse. On the other hand, as described above, the position input may specify one coordinate, that is, a point. A position input specifying one coordinate can be easily performed by, for example, line-of-sight input or a tap operation on a smartphone. In this case, in S501, the horizontal coordinates of the coordinates designated as x1 and x2 can be used, and the vertical coordinates of the coordinates designated as y1 and y2 can be used. Processing after S502 can be performed in the same manner.

Further, the shape of the area indicated by the position input is not particularly limited. For example, the position input may indicate a loop shape such as a circle. In this case, in S501, horizontal coordinates x1, x2 and vertical coordinates y1, y2 can be arbitrarily selected from coordinates inside the shape or on the boundary line of the shape. For example, among the coordinate values of each point on the boundary line of the shape indicated by the position input, the smallest horizontal coordinate and largest horizontal coordinate are x1 and x2, and the smallest vertical coordinate and largest vertical coordinate are The coordinates of can be set to y2 and y1, respectively. Processing after S502 can be performed in the same manner.

According to the above method, when a positional input indicating a part of continuous descriptions in a moving image is made, positional information indicating an area including the entire continuous descriptions can be generated. The position indicated by the position information obtained through such processing corresponds to the position specified by the position input, which has been corrected. According to such a method, position information indicating the entire portion of interest can be generated by the user's simple operation of specifying a part of the portion of interest. As will be described later, by showing an area according to such positional information on a moving image, it becomes easier for the user to grasp the part of interest.

Up to this point, a case has been described in which the area according to the position input is enlarged based on the moving image at the position indicated by the position input. On the other hand, the area according to the position input or the area enlarged according to the above method may be reduced based on the moving image at the position indicated by the position input. For example, if the area indicated by the position input is wider than the area of continuous description, the area may be reduced. Specifically, the position determination unit 202 can determine appropriate coordinates that indicate the area so as to eliminate blank spaces between the boundaries of the area and the description. In such processing, the position determination unit 202 determines whether descriptive information is present in the edge area of the area indicated by the position input, and reduces the area so as to exclude the edge area that does not include descriptive information. This can be achieved by Specifically, by performing the processing according to FIG. 5 using α having a negative value, the coordinates x1, x2, y1, and y2 can each be moved inward of the area.

(Method for generating location information based on area division)
The position input may specify a dividing position of an area on a moving image. In this case, the position determining unit 202 can generate position information to indicate one of the divided areas obtained by dividing according to the position input. In such an example, the user's position input may specify a line. A position input for specifying a line can be easily performed by, for example, line-of-sight input or a swipe operation on a smartphone.

In the following, a method for generating positional information when a line dividing a predetermined area shown in a moving image into two areas is input will be described. The type of the predetermined area is not particularly limited, but may be, for example, a blackboard or whiteboard area. In this case, the position determining unit 202 can detect a predetermined area from the moving image and generate position information so as to indicate one of the divided areas of the predetermined area divided according to the position input. Generating position information based on a predetermined area located at a position on a moving image specified by position input in this way is also an example of a method of generating position information based on the contents of a moving image.

FIG. 6A shows a case where the user focuses on the description “ABCDEVWXYZ” on the blackboard 602 in the distributed video 601. In this example, the user has drawn a line 603 to record the location of the area containing the description "ABCDEVWYZ" as metadata. As shown in FIG. 6C, the position determining unit 202 determines an area 604 that includes the description "ABCDEVWXYZ" and generates position information indicating the position of this area 604.

In such an example, a processing example for the position determination unit 202 to generate position information in S302 will be described with reference to the flowchart shown in FIG. In the process shown in FIG. 7, the position determining unit 202 selects one region from the plurality of regions based on whether information is included in each of the plurality of regions on the moving image divided according to the position input, Generate location information to indicate the selected area. In the following example, the position determining unit 202 selects an area containing descriptive information from areas divided by lines. Note that the method of dividing the area is not limited to two. Furthermore, the method of dividing the region is not limited to the method of drawing lines. In any case, a similar technique can be used to select a region containing descriptive information from among the divided regions obtained by division. In this way, generating position information based on the content of the moving image in each divided area specified by the position input specifying the dividing position is also an example of a method of generating position information based on the content of the moving image. It is.

In S701, the position determining unit 202 calculates four vertex coordinates (x1, x2, y1, y2) of the blackboard 602 by image analysis, as shown in FIG. 6(B). Note that the method of image analysis is not particularly limited, but for example, a region division method can be used. For example, the position determination unit 202 can determine a blackboard area based on the feature amount in the frame image, and calculate the vertex coordinates of the determined blackboard area. Furthermore, the position determining unit 202 calculates the coordinates x3 of the intersection between the line indicated by the position input and the upper side of the blackboard by image analysis.

In S702, the position determination unit 202 determines a portion of the blackboard 602 where descriptive information exists by analyzing the image area of the blackboard 602. Detection of descriptive information can be performed in the same manner as in S502.

In S703, the position determination unit 202 determines whether there is a description on both the right and left sides of the line indicated by the position input. If there are descriptions in both, the process advances to S704; otherwise, the process advances to S707.

In S704, the position determination unit 202 determines whether there is a description on the right side of the line indicated by the position input. If there is a description, the process advances to S709; otherwise, the process advances to S705.

In S705, the position determination unit 202 determines whether there is a description on the left side of the line indicated by the position input. If there is a description, the process advances to S710; otherwise, the process advances to S706.

In S706, the position determining unit 202 determines that there is no position information recorded as metadata. Then, the process according to FIG. 7 ends.

In step S707, the position determining unit 202 obtains a user input indicating which area, the right side or the left side, is to be recorded as metadata. The position determining unit 202 may output a prompt to the user for user input. For example, when position input is performed on a smartphone, the user can input by tapping either the right or left side. Further, when position input is performed using line of sight input, input can be performed by directing the line of sight to either the right side or the left side for a certain period of time or more.

In S708, the position determining unit 202 determines whether the user has selected the right side. If the right side is selected, the process advances to S709; otherwise, the process advances to S710.

In S709, the position determining unit 202 determines to record position information indicating the area on the right side of the line indicated by the position input as metadata. Then, the position determination unit 202 generates such position information according to the coordinates calculated in S701. This position information is, for example, information indicating vertex coordinates (x1, x3, y1, y2). Then, the process according to FIG. 7 ends.

In S710, the position determination unit 202 determines to record position information indicating the area to the left of the line indicated by the position input as metadata. Then, the position determination unit 202 generates such position information according to the coordinates calculated in S701. This position information is, for example, information indicating vertex coordinates (x2, x3, y1, y2). Then, the process according to FIG. 7 ends.

In the above example, the case where the user draws a line in the vertical direction has been described. On the other hand, the method of dividing the regions is not particularly limited. For example, the direction of the line is not particularly limited. Even when the user draws a line in the horizontal direction, the position determination unit 202 can similarly generate position information. In that case, the position determination unit 202 can divide the area into upper and lower areas according to the line indicated by the position input, and determine whether there is a description in each of the upper and lower areas.

According to such an example, position information indicating the portion of interest can be generated based on a simple input operation of dividing an area with lines or the like according to the portion of interest. As will be described later, by showing an area according to such positional information on a moving image, it becomes easier for the user to grasp the part of interest.

(Method for generating time information based on moving images)
Below, a method for generating time information by the time determination unit 203 will be explained. As described above, the time determination unit 203 may generate time information indicating the playback position of the moving image that is the subject of position input. On the other hand, the time determining unit 203 may generate time information based on the content of the moving image at the position on the moving image specified by the position input.

In this embodiment, the time determination unit 203 determines the time when information on the moving image at the position indicated by the position input starts to appear, and generates time information to indicate the determined time. For example, the time determination unit 203 identifies the timing at which descriptive information at the position indicated by the position input appears. Then, the time determination unit 203 generates time information indicating this timing. In the example below, the time determining unit 203 identifies the timing when the lecturer starts writing on the blackboard the part of interest in the distributed video of the lecture. According to such a configuration, when the user views the video again for review, it becomes easy to play the video from the timing at which the explanation on the board of interest starts.

FIG. 8(A) shows how a user inputs a position on a moving image. The user is inputting a position indicating a rectangular area 803. This area 803 includes the written content "ABCDE" on the blackboard 802 in the distributed video 801. Here, the time of the frame being displayed when the user inputs the position (that is, the frame that is the target of the position input) is assumed to be t1. At this time, the time determination unit 203 identifies the frame when the lecturer 804 started writing the written content surrounded by the area 803, as shown in FIG. 8(B). Then, the time determination unit 203 generates time information indicating the time t0 of the specified frame. The time information generated in this way is stored as metadata.

In such an example, a processing example for the time determination unit 203 to generate time information in S303 will be described with reference to the flowchart shown in FIG. In S901, the time determination unit 203 prepares a temporary variable t for searching for time t0. Further, the time determination unit 203 sets the time t1 of the frame whose position is input to the variable t.

In S902, the time determination unit 203 assigns to the variable t the time of the frame immediately before the frame at the time indicated by the variable t.

In S903, the time determination unit 203 extracts descriptive information from the frame image in the frame at the time indicated by the variable t. Here, the time determination unit 203 can extract descriptive information from the position (for example, a rectangular area) indicated by the position information generated in S302 from the frame image. In this way, the time determining unit 203 can acquire, as information on the moving image at the position indicated by the position input, all of the information that is continuous with the information in the area indicated by the position information. In another embodiment, the time determination unit 203 may extract descriptive information from a position (for example, a rectangular area) indicated by the position information acquired in S301 in the frame image. Extraction of descriptive information can be performed in the same manner as in S502.

In S904, the time determination unit 203 determines whether descriptive information has been extracted. If there is no extracted descriptive information, the process advances to S905; otherwise, the process returns to S902. In S905, the time determination unit 203 generates time information indicating the variable t. This time information is recorded as metadata. The playback position indicated by the time information obtained through such processing corresponds to a corrected playback position of the moving image that is the subject of position input.

According to such an embodiment, it is possible to record the time when the descriptive information of the portion of interest appears. Therefore, when the user views the moving image again, it becomes easy to reproduce the section of the moving image that is related to the portion of interest.

(Method for generating time information based on sound)
The time determining unit 203 may generate the time information based on the sound related to the moving image corresponding to the playback position of the moving image that is the target of the position input. In particular, the time determination unit 203 determines the content of the moving image at the position on the moving image specified by the position input and the sound related to the moving image corresponding to the playback position of the moving image that is the target of the position input. Time information can be generated based on both. In particular, the time determination unit 203 determines the descriptive information on the moving image at the position on the moving image specified by the position input and the sound related to the moving image corresponding to the playback position of the moving image that is the target of the position input. Time information can be generated based on the content shown.

For example, in the example of FIG. 8A, the lecturer may have started explaining the written content "ABCDE" in the rectangular area 803 at time t00, which is earlier than time t0. In this embodiment, the time determination unit 203 generates time information indicating the time at which an utterance indicating descriptive information such as "ABCDE" starts. Specifically, the time determination unit 203 can identify time t00 based on the spoken language indicated by the sound associated with the moving image. Then, the time determination unit 203 generates time information indicating the thus specified time t00. According to such a configuration, even if the explanation of the part of interest starts before writing the part of interest on the board, it is easy to play the video from the timing when the explanation of the part of interest starts written on the board. Become.

In such an example, a processing example for the time determination unit 203 to generate time information in S303 will be described with reference to the flowchart shown in FIG. In S1101, the time determination unit 203 extracts descriptive information from the frame image in the frame at time t1. Extraction of descriptive information can be performed in the same manner as in S903, and specifically, for example, pattern recognition can be used. In S1102, the time determination unit 203 identifies the time t0 when writing the description content starts, according to the method shown in FIG. Furthermore, the time determination unit 203 prepares a temporary variable t for searching for time t00. Further, the time determination unit 203 sets the specified time t0 to the variable t.

In S1103, the time determination unit 203 extracts sound data starting from the time indicated by the variable t and ending after a certain period of time. Furthermore, the time determination unit 203 extracts the spoken language indicated by the extracted sound data by voice analysis. The voice analysis method is not particularly limited.

In S1104, the time determination unit 203 determines whether the descriptive information extracted in S1101 and the spoken language extracted in S1103 match. Note that when determining whether the descriptive information and the spoken language match, the time determination unit 203 may determine whether the descriptive information is included in the spoken language. If they match, the process advances to S1105; otherwise, the process advances to S1106.

In S1105, the time determination unit 203 generates time information indicating the time indicated by the variable t. Then, the process according to FIG. 10 ends.

In S1106, the time determination unit 203 determines whether sound data at the end of the moving image has been extracted. Note that in the process shown in FIG. 10, sound data within a certain range from time t1 may be extracted as the processing target. In this case, the time determination unit 203 may determine whether the last sound data within this range has been extracted. If it is determined that the sound data at the end of the moving image has been extracted, the process advances to S1107. If not, the process returns to S1103. After that, in S1103, the time determination unit 203 updates the variable t by adding the length of the sound extracted in S1103, and extracts the sound data again.

In S1107, the time determination unit 203 assigns t0 to initialize the variable t. In S1108, the time determination unit 203 extracts sound data starting from a certain time before the time indicated by the variable t and ending at time t. Furthermore, the time determining unit 203 extracts the spoken language indicated by the extracted sound data in the same manner as in S1103. Further, the time determination unit 203 updates the variable t by subtracting the length of the extracted sound.

In S1109, the time determination unit 203 determines whether the descriptive information extracted in S1101 and the language extracted in S1108 match. If they match, the process advances to S1105; otherwise, the process advances to S1110.

In S1110, the time determination unit 203 determines whether the sound data at the beginning of the moving image has been extracted. If it is determined that the sound data at the beginning of the moving image has been extracted, the process advances to S1111. If not, the process returns to S1108.

In S1111, the time determination unit 203 determines that there is no time information recorded as metadata. Then, the process according to FIG. 10 ends. However, in such a case, the time determination unit 203 may generate time information indicating the playback position of the moving image that is the subject of position input.

According to the above method, the time determination unit 203 determines, based on the sound, the playback position (time t1) of the moving image that is the target of position input or the vicinity of the playback position after correction according to FIG. 9 (time t0). , it is possible to specify the time (t00) at which the utterance indicating the descriptive information starts. Then, the time determination unit 203 can generate time information indicating this time (t00). The playback position indicated by the time information obtained through such processing corresponds to a corrected playback position of the moving image that is the subject of position input. Note that after performing the processing in S1107 to S1110, the processing in S1103 to S1106 may be performed.

In the above example, time information was generated based on the spoken language shown in the sound data and the descriptive information extracted from the video image. However, the method of generating time information using sound data is not limited to this method. The time determination unit 203 can generate time information based on sounds associated with moving images using various methods. For example, the time determination unit 203 can determine the timing at which continuous speech is interrupted based on the sound data. Then, the time determination unit 203 determines the timing at which the continuous speech is interrupted, before the time in the video when the position input is performed, and the most at the time in the video when the position input is performed. Time information indicating a nearby timing may be generated. As another example, the time determination unit 203 may generate time information indicating a timing closest to time t0, which is before time t0 determined by the process according to FIG. 9, among the timings at which continuous speech is interrupted. good.

Further, the time determination unit 203 can generate time information based on the content indicated by the sound related to the moving image using various methods. For example, the time determination unit 203 can generate time information based on the playback time of a sound effect included in a sound related to a moving image. For example, sounds associated with moving images may include specific sound effects (eg, a sound effect played at the beginning of a section, a sound effect indicating a point of interest, etc.). In this case, the time determination unit 203 can determine whether the sound effect is being played back at the time corresponding to the playback position of the moving image that is the subject of the position input. If the sound effect is being played at this time, the time determining unit 203 can generate time information indicating the playback start time or playback end time of this sound effect. Furthermore, the time determining unit 203 may detect a sound effect that is played near the playback position of the moving image that is the target of the position input. In this case, the time determination unit 203 can generate time information indicating the playback start time or playback end time of the detected sound effect. By using the time information generated by such a configuration, it becomes possible to reproduce moving images and sounds related thereto from an appropriate position that matches the content of the sounds and the user's preferences. As described above, it is not essential to refer to descriptive information in order to generate time information.

According to such an embodiment, it is possible to record the time when the explanation regarding the descriptive information of the portion of interest is started. Therefore, when the user views the moving image again, it becomes easy to reproduce the section related to the portion of interest in the moving image.

(How to use metadata)
FIG. 4C shows an example of a method of using location information and time information, which are metadata recorded by the recording unit 204. As shown in FIG. 4C, the playback unit 205 can display the moving image on the display unit 106. At this time, the playback unit 205 can display, on the moving image, information indicating a portion according to the position information in the frame indicated by the time information. For example, in the example of FIG. 4C, an area 406 according to position information is specified on the video image. Furthermore, the playback unit 205 can indicate the attribute information of this part in association with the information indicating the part according to the position information. In the example of FIG. 4C, a mark indicating that this part is important is displayed in association with the area 406.

Furthermore, the playback unit 205 may display a marker indicating the playback position indicated by the time information on a seek bar used to determine the playback position of the moving image. When the user specifies this marker, the playback unit 205 may start playing the moving image from the playback position indicated by the time information. As a further example, the playback unit 205 displays a thumbnail image obtained by superimposing information indicating the position on the video image indicated by the position information on a frame image at the playback position of the video image that is the target of the position input. You may. When the user specifies this thumbnail image, the playback unit 205 may start playing the moving image from the playback position indicated by the time information.

(Other examples)
Up to this point, a method for generating position information based on the content of a moving image at a position on the moving image specified by position input has been described (FIGS. 5 and 7). Furthermore, a method for generating time information based on the content of a moving image at a position on the moving image specified by position input has also been described (FIGS. 9 and 10). Furthermore, a method for generating time information based on the sound associated with a moving image corresponding to the playback position of the moving image that is the target of position input has also been described (FIG. 10). On the other hand, the position information may be generated based on the sound related to the moving image corresponding to the playback position of the moving image that is the target of the position input. For example, the position determination unit 202 can detect the spoken language indicated by the sound related to the moving image that corresponds to the playback position of the moving image that is the target of the position input. Then, the position determining unit 202 can search for descriptive information corresponding to the detected spoken language in the vicinity of the position on the moving image designated by the position input, for example, using pattern matching or the like. When such descriptive information is retrieved, the position determination unit 202 can generate positional information to indicate a region that includes the retrieved descriptive information. These methods can be used in any combination to generate at least one of location information and time information.

The content of the present disclosure provides a system or device with a program that implements one or more functions of the embodiments described above via a network or a storage medium, and one or more processors in a computer of the system or device executes the program. This can also be realized by reading and executing processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The disclosure of this specification includes the following information processing device, information processing method, and program.

(Item 1)
an acquisition means for acquiring a position input specifying a position on the moving image;
A generating means for generating position information indicating a position on the moving image and time information indicating a playback position of the moving image based on the position input, the generating means generating at least one of the position information and the time information by the at least one of the content of the moving image at the position on the moving image specified by the position input, and the sound related to the moving image corresponding to the playback position of the moving image that is the target of the position input. further generating means for generating based on;
recording means for recording the location information and the time information;
An information processing device comprising:

(Item 2)
The generating means converts the position on the moving image specified by the position input into the contents of the moving image at the position on the moving image specified by the position input, and the moving image that is the target of the position input. generating the position information by correcting it based on at least one of: a sound associated with the moving image corresponding to a reproduction position of the image; and/or
The playback position of the video that is the target of the position input, the content of the video at the position on the video specified by the position input, and the reproduction of the video that is the target of the position input. The information processing device according to item 1, wherein the time information is generated by correcting the time information based on at least one of: a sound related to the moving image corresponding to a position;

(Item 3)
Item 1 or 2, wherein the generating means generates the position information so as to indicate an area including information on the moving image at a position on the moving image specified by the position input. 2. The information processing device according to 2.

(Item 4)
The information processing device according to item 3, wherein the information is a continuous description.

(Item 5)
The position input specifies an area including at least part of information on the video image,
According to any one of items 2 to 4, the generating means determines an area that includes information on the moving image by enlarging or reducing the area specified by the position input. information processing equipment.

(Item 6)
The information processing device according to any one of items 1 to 5, wherein the position input specifies a rectangle, a loop shape, or a point.

(Item 7)
The position input specifies a division position of a predetermined area on the video image,
The generating means detects the predetermined area from the moving image and generates the position information so as to indicate one of the divided areas of the predetermined area divided according to the position input. , the information processing device according to item 1.

(Item 8)
8. The information processing device according to item 7, wherein the predetermined area is an area of a blackboard or a whiteboard.

(Item 9)
The generating means selects one region from the plurality of regions based on whether information is included in each of the plurality of regions on the moving image divided according to the position input, and generates the selected region. 9. The information processing device according to item 7 or 8, characterized in that the position information is generated as shown.

(Item 10)
9. The information processing device according to any one of items 7 to 9, wherein the position input specifies a line.

(Item 11)
Item 1, wherein the generating means determines a time when information on the moving image at a position indicated by the position input starts to appear, and generates the time information to indicate the determined time. The information processing device according to any one of 10 to 10.

(Item 12)
The generation means is characterized in that it determines the time when the appearance of all information that is continuous with the information in the area indicated by the position information starts to appear, and generates the time information so as to indicate the determined time. The information processing device according to any one of items 1 to 10.

(Item 13)
The generating means generates the content of the moving image at the position on the moving image specified by the position input, and the sound related to the moving image corresponding to the playback position of the moving image that is the target of the position input. The information processing device according to any one of items 1 to 10, characterized in that the time information is generated based on both of and.

(Item 14)
The generating means is configured to generate descriptive information on the moving image at a position on the moving image specified by the position input, and information related to the moving image corresponding to a playback position of the moving image that is the target of the position input. The information processing device according to any one of items 1 to 10, wherein the information processing device generates the time information based on the content indicated by the sound.

(Item 15)
15. The information processing device according to item 14, wherein the generating means generates the time information indicating a time when an utterance indicating the descriptive information starts.

(Item 16)
The generating means identifies, based on the sound, a time at which an utterance indicating the descriptive information starts in the vicinity of a playback position of the moving image that is the target of the position input, and generates the time information indicating the time. The information processing device according to item 14 or 15, characterized in that the information processing device generates.

(Item 17)
An information processing method performed by an information processing device, the method comprising:
obtaining a position input specifying a position on the video image;
The step of generating position information indicating a position on the moving image and time information indicating a playback position of the moving image based on the position input, the step of generating at least one of the position information and the time information based on the position input. Further, at least one of the content of the moving image at the position on the moving image specified by the input, and the sound related to the moving image corresponding to the playback position of the moving image that is the target of the position input. a step of generating based on the
recording the location information and the time information;
An information processing method characterized by comprising:

(Item 18)
A program for causing a computer to function as the information processing device according to any one of items 1 to 16.

The scope of the present disclosure is not limited to the above embodiments, and various changes and modifications can be made without departing from the spirit and scope thereof. Accordingly, the claims are appended hereto in order to disclose the scope of the rights.

101: Information processing device, 201: Acquisition unit, 202: Position determination unit, 203: Time determination unit, 204: Storage unit, 205: Reproduction unit

Claims (18)

an acquisition means for acquiring a position input specifying a position on the moving image;
A generating means for generating position information indicating a position on the moving image and time information indicating a playback position of the moving image based on the position input, the generating means generating at least one of the position information and the time information by the at least one of the content of the moving image at the position on the moving image specified by the position input, and the sound related to the moving image corresponding to the playback position of the moving image that is the target of the position input. further generating means for generating based on;
recording means for recording the location information and the time information;
An information processing device comprising: The generating means converts the position on the moving image specified by the position input into the contents of the moving image at the position on the moving image specified by the position input, and the moving image that is the target of the position input. generating the position information by correcting it based on at least one of: a sound associated with the moving image corresponding to a reproduction position of the image; and/or
The playback position of the video that is the target of the position input, the content of the video at the position on the video specified by the position input, and the reproduction of the video that is the target of the position input. The information processing apparatus according to claim 1, wherein the time information is generated by correcting the time information based on at least one of: a sound related to the moving image corresponding to a position; 1 . The generating means generates the position information so as to indicate a region including information on the moving image located at a position on the moving image specified by the position input. The information processing device described in . The information processing device according to claim 3, wherein the information is a continuous description. The position input specifies an area including at least part of information on the video image,
4. The information processing apparatus according to claim 3, wherein the generating means determines an area that includes information on the moving image by enlarging or reducing an area specified by the position input. 4. The information processing apparatus according to claim 3, wherein the position input specifies a rectangle, a loop shape, or a point. The position input specifies a division position of a predetermined area on the video image,
The generating means detects the predetermined area from the moving image and generates the position information so as to indicate one of the divided areas of the predetermined area divided according to the position input. , The information processing device according to claim 1. 8. The information processing apparatus according to claim 7, wherein the predetermined area is an area of a blackboard or a whiteboard. The generating means selects one region from the plurality of regions based on whether information is included in each of the plurality of regions on the moving image divided according to the position input, and generates the selected region. 8. The information processing apparatus according to claim 7, wherein the position information is generated as shown in FIG. 8. The information processing apparatus according to claim 7, wherein the position input specifies a line. 3. The generating means determines a time when information on the moving image at a position indicated by the position input starts to appear, and generates the time information so as to indicate the determined time. 1. The information processing device according to 1. The generation means is characterized in that it determines the time when the appearance of all information that is continuous with the information in the area indicated by the position information starts to appear, and generates the time information so as to indicate the determined time. The information processing device according to claim 1. The generating means generates the content of the moving image at the position on the moving image specified by the position input, and the sound related to the moving image corresponding to the playback position of the moving image that is the target of the position input. The information processing device according to claim 1, wherein the time information is generated based on both of the following. The generating means is configured to generate descriptive information on the moving image at a position on the moving image specified by the position input, and information related to the moving image corresponding to a playback position of the moving image that is the target of the position input. The information processing apparatus according to claim 1, wherein the time information is generated based on the content indicated by the sound. 15. The information processing apparatus according to claim 14, wherein the generating means generates the time information indicating a time when an utterance indicating the descriptive information starts. The generating means identifies, based on the sound, a time at which an utterance indicating the descriptive information starts in the vicinity of a playback position of the moving image that is the target of the position input, and generates the time information indicating the time. 15. The information processing apparatus according to claim 14, wherein the information processing apparatus generates the information processing apparatus. An information processing method performed by an information processing device, the method comprising:
obtaining a position input specifying a position on the video image;
The step of generating position information indicating a position on the moving image and time information indicating a playback position of the moving image based on the position input, the step of generating at least one of the position information and the time information based on the position input. Further, at least one of the content of the moving image at the position on the moving image specified by the input, and the sound related to the moving image corresponding to the playback position of the moving image that is the target of the position input. a step of generating based on the
recording the location information and the time information;
An information processing method characterized by comprising: A program for causing a computer to function as the information processing device according to any one of claims 1 to 16.

Images