WO2021240652A1

WO2021240652A1 - Information processing device, control method, and storage medium

Info

Publication number: WO2021240652A1
Application number: PCT/JP2020/020772
Authority: WO
Inventors: はるな渡辺; 克菊池; 壮馬白石; 悠鍋藤
Original assignee: 日本電気株式会社
Priority date: 2020-05-26
Filing date: 2020-05-26
Publication date: 2021-12-02
Also published as: JP7420243B2; US20230205815A1; JPWO2021240652A1

Abstract

An information processing device 1X primarily comprises a pair determination means 15X and a relevance calculating means 16X. The pair determination means 15X determines a pair of pieces of data that will serve as a first digest candidate in which at least one of the pieces of data is a digest candidate, the pair of pieces of data including image data and/or sound data. The relevance calculating means 16X calculates a relevance indicating the level of likelihood that the pair determined by the pair determination means 15X will be simultaneously included in a digest.

Description

Information processing equipment, control method and storage medium

The present disclosure relates to technical fields of information processing devices, control methods, and storage media that perform processing related to digest generation.

There is a technology to generate a digest by editing the video data that is the material. For example, Patent Document 1 discloses a method of confirming and producing highlights from a video stream of a sporting event on the ground.

Special Table 2019-522948 Gazette

When the importance is calculated for the video as the material and the digest generation is automatically generated based on the importance, it is possible to generate a digest that makes it difficult to understand the story as a result of combining scenes with low relevance. There was sex.

An object of the present disclosure is to provide an information processing device, a control method, and a storage medium capable of generating information suitable for digest generation in consideration of the above problems.

One aspect of the information processing apparatus is a pair determining means for determining a pair of data including at least one of video data or sound data, wherein at least one of the data is a first digest candidate which is a digest candidate. An information processing apparatus comprising, and a relevance calculation means for calculating a relevance degree indicating the degree of probability that the pair is simultaneously included in the digest.

One aspect of the control method is to determine by a computer the pair of data including at least one of video data or sound data, and the pair in which at least one data is a first digest candidate, which is a digest candidate. It is a control method for calculating the degree of association indicating the degree of probability that the pair is included in the digest at the same time.

One aspect of the storage medium is a pair of data including at least one of video data or sound data, and a pair determining means for determining the pair in which at least one data is a first digest candidate which is a digest candidate. , A storage medium in which a program for operating a computer as a relevance calculation means for calculating a relevance indicating the degree of probability that the pair is simultaneously included in the digest is stored.

According to the present disclosure, it is possible to generate information suitable for digest generation.

The configuration of the digest candidate selection system in the first embodiment is shown. The hardware configuration of the information processing device is shown. This is an example of a functional block of an information processing device. It is a figure which shows the outline of the selection process of the 1st digest candidate common to the 1st selection example and the 2nd selection example. It is a figure which shows the outline of the selection process of the 2nd digest candidate which concerns on the 1st selection example after the selection of the 1st digest candidate. It is a figure which shows the outline of the selection process of the 2nd digest candidate which concerns on the 2nd selection example after the selection of the 1st digest candidate. It is a schematic block diagram of the learning system which generates the relevance inferior information. An example of the functional block configuration of the learning device is shown. This is an example of a flowchart showing a procedure of processing executed by the information processing apparatus in the first embodiment. This is an example of a flowchart showing a procedure of processing executed by the learning device in the first embodiment. It is an example of the functional block diagram of the information processing apparatus which concerns on modification 2. This is an example of a flowchart showing a procedure of processing executed by the information processing apparatus in the second modification. It is a functional block diagram of the information processing apparatus in 2nd Embodiment. This is an example of a flowchart executed by the information processing apparatus in the second embodiment.

Hereinafter, embodiments of an information processing device, a control method, and a storage medium will be described with reference to the drawings.

<First Embodiment>
(1) System Configuration Figure 1 shows the configuration of the digest candidate selection system 100 according to the first embodiment. The digest candidate selection system 100 preferably selects video data as a digest candidate from video data (also referred to as “material data”) as a material. The digest candidate selection system 100 mainly includes an information processing device 1, an input device 2, an output device 3, and a storage device 4.

The information processing device 1 performs data communication with the input device 2 and the output device 3 via a communication network or by direct communication by radio or wire. The information processing device 1 refers to the relevance inferior information D2 and the importance inferior information D3 also stored in the storage device 4 from the material data D1 stored in the storage device 4, and is a video data that is a candidate for a digest. To select. Then, the information processing device 1 generates an output signal "S1" related to the above selection result, and supplies the generated output signal S1 to the output device.

The input device 2 is an arbitrary user interface that accepts user input, and corresponds to, for example, a button, a keyboard, a mouse, a touch panel, a voice input device, and the like. The input device 2 supplies the input signal "S2" generated based on the user input to the information processing device 1. The output device 3 is, for example, a display device such as a display or a projector, and a sound output device such as a speaker, and performs a predetermined display or sound output based on the output signal S1 supplied from the information processing device 1.

The storage device 4 is a memory for storing various information necessary for processing of the information processing device 1. The storage device 4 stores, for example, the material data D1, the relevance inferior information D2, and the importance inferrer information D3.

The material data D1 is video data to be edited in the generation of the digest. Hereinafter, the video data corresponding to the section having a predetermined reproduction time length extracted from the material data D1 is also referred to as “section data”. Each section data includes a predetermined number of time-series images, which is one or more. In the first embodiment, the information processing apparatus 1 calculates the importance and generates a pair for calculating the relevance of the section data obtained by dividing the material data D1 for each unit interval.

Relevance inference device information D2 is information about an inference device (also referred to as "relevance inference device") that infers the relevance to a pair of interval data (also referred to as "inference target pair Ptag"). The degree of relevance is an index showing the relevance in terms of whether or not the inferred pair Ptag is simultaneously included in the digest, in other words, the degree of probability (or validity) that the inferred pair Ptag is simultaneously included in the digest. It is an index showing. The relevance inferior is learned in advance to infer these relevances when a predetermined number (one or more) of image pairs corresponding to the pair of interval data are input, and the relevance inferior information D2. Contains the parameters of the learned relevance inferior.

Importance inferrer information D3 is information about an inferior (also referred to as "importance inferior") that infers importance to interval data. The above-mentioned importance is an index as a reference for determining whether the section in the material data D1 corresponding to the video data input in the generation of the digest is an important section or a non-important section. The importance inferior is learned in advance so as to infer the importance of the target section when a predetermined number (one or more) of images constituting the section data are input, and the importance inferior information D3 is used. Includes learned importance inferrer parameters.

The learning model of the relevance inferior and the importance inferior may be a learning model based on arbitrary machine learning such as a neural network or a support vector machine, respectively. For example, when the above-mentioned model of the relevance inferior and the importance inferior is a neural network such as a convolutional neural network, the relevance inferior information D2 and the importance inferior information D3 are a layer structure, a neuron structure of each layer, and the like. Includes various parameters such as the number and size of filters in each layer and the weight of each element of each filter.

The storage device 4 may be an external storage device such as a hard disk connected to or built in the information processing device 1, or may be a storage medium such as a flash memory. Further, the storage device 4 may be a server device that performs data communication with the information processing device 1. Further, the storage device 4 may be composed of a plurality of devices. In this case, the storage device 4 may store the material data D1, the relevance inferior information D2, and the importance inferior information D3 in a distributed manner.

The configuration of the digest candidate selection system 100 described above is an example, and various changes may be made to the configuration. For example, the input device 2 and the output device 3 may be integrally configured. In this case, the input device 2 and the output device 3 may be configured as a tablet-type terminal integrated with the information processing device 1. Further, the information processing device 1 may be composed of a plurality of devices. In this case, the plurality of devices constituting the information processing device 1 exchange information necessary for executing the pre-assigned process among the plurality of devices.

(2) Hardware Configuration of Information Processing Device FIG. 2 shows the hardware configuration of the information processing device 1. The information processing apparatus 1 includes a processor 11, a memory 12, and an interface 13 as hardware. The processor 11, the memory 12, and the interface 13 are connected via the data bus 19.

The processor 11 executes a predetermined process by executing the program stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a quantum processor.

The memory 12 is composed of various volatile memories such as RAM (Random Access Memory) and ROM (Read Only Memory) and non-volatile memory. Further, the memory 12 stores a program executed by the information processing apparatus 1. Further, the memory 12 is used as a working memory and temporarily stores information and the like acquired from the storage device 4. The memory 12 may function as the storage device 4. Similarly, the storage device 4 may function as the memory 12 of the information processing device 1. The program executed by the information processing apparatus 1 may be stored in a storage medium other than the memory 12.

The interface 13 is an interface for electrically connecting the information processing device 1 and another device. For example, the interface for connecting the information processing device 1 and another device may be a communication interface such as a network adapter for transmitting / receiving data to / from another device based on the control of the processor 11 by wire or wirelessly. good. In another example, the information processing apparatus 1 and the other apparatus may be connected by a cable or the like. In this case, the interface 13 includes a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Atchment), etc. for exchanging data with other devices.

The hardware configuration of the information processing device 1 is not limited to the configuration shown in FIG. For example, the information processing device 1 may include at least one of an input device 2 and an output device 3.

(3) Functional block Next, the functional block of the information processing apparatus 1 will be described. Here, the information processing apparatus 1 selects a provisional digest candidate (also referred to as a “first digest candidate”) from the section data of the material data D1, and finally outputs the digest based on the first digest candidate. Select candidates (also called "second digest candidates").

FIG. 3 is an example of a functional block of the processor 11 of the information processing device 1. Functionally, the processor 11 of the information processing device 1 includes a first digest candidate selection unit 14, a pair determination unit 15, a relevance calculation unit 16, a second digest candidate selection unit 17, and an output control unit 18. Has. In FIG. 3, the blocks in which data is exchanged are connected by a solid line, but the combination of blocks in which data is exchanged is not limited to FIG. The same applies to the figures of other functional blocks described later.

The first digest candidate selection unit 14 calculates the importance of each of the section data constituting the material data D1, and selects the first digest candidate from the section data constituting the material data D1 based on the calculated importance. do. Here, the section data is data in which the material data D1 is divided by a section having a unit time length, and is data including one or more images for a predetermined number of sheets. Then, the first digest candidate selection unit 14 configures the importance inference device by referring to the importance inference device information D3, and sequentially inputs the section data extracted from the material data D1 into the importance inference device. Then, the importance corresponding to all the section data is acquired. Then, the first digest candidate selection unit 14 selects the section data whose importance is equal to or higher than the preset threshold value as the first digest candidate. Hereinafter, the section data that is not the first digest candidate is also referred to as a “non-first digest candidate”. The first digest candidate selection unit 14 supplies information regarding the first digest candidate (also referred to as “first digest candidate information Idc1”) to the pair determination unit 15.

The pair determination unit 15 determines a plurality of inference target pairs Ptag that combines two section data extracted from the material data D1 based on the first digest candidate information Idc1 determined by the first digest candidate selection unit 14. In this case, in the first example, the pair determination unit 15 determines an inference target pair Ptag that combines arbitrary two interval data that are candidates for the first digest. In the second example, the pair determination unit 15 determines an inference target pair Ptag that arbitrarily combines the section data that is the first digest candidate and the section data that is the non-first digest candidate. Then, the pair determination unit 15 supplies the determined inference target pair Ptag to the relevance calculation unit 16.

The pair determination unit 15 may limit the inference target pair Ptag by the method described below in order to reduce the overall processing load of the information processing apparatus 1. For example, the pair determination unit 15 may determine two section data in which the difference between the corresponding reproduction times is within a predetermined time difference as the inference target pair Ptag. In another example, the pair determination unit 15 may determine the inference target pair Ptag by targeting only the interval data extracted from the material data D1 at predetermined time intervals. In yet another example, the pair determination unit 15 applies an arbitrary clustering method to the interval data to classify the interval data, and determines the inference target pair Ptag with only the interval data belonging to a predetermined class as the target of combination. You may.

The relevance calculation unit 16 calculates the relevance degree for each of the inference target pair Ptag supplied from the pair determination unit 15. In this case, the relevance calculation unit 16 configures the relevance inference device by referring to the relevance inference device information D2, and sequentially inputs the inference target pair Ptag acquired from the pair determination unit 15 into the relevance inference device. Then, the degree of relevance to each of the inference target pair Ptag is calculated. Then, the relevance calculation unit 16 supplies the calculated relevance information (also referred to as “relevance information Ia”) to the second digest candidate selection unit 17.

The second digest candidate selection unit 17 selects the second digest candidate based on the relevance information Ia supplied from the relevance calculation unit 16. The second digest candidate selection unit 17 supplies information regarding the selected second digest candidate (also referred to as “second digest candidate information Idc2”) to the output control unit 18. Here, the second digest candidate information Idc2 may be the section data itself that is the second digest candidate, and is the time information (information indicating the reproduction time in the material data D1) of the section data that is the second digest candidate. You may.

Here, when the inference target pair Ptag is selected from the first digest candidate, the second digest candidate selection unit 17 selects the first digest candidate constituting the inference target pair Ptag whose relevance is equal to or higher than a predetermined threshold value. Select as a second digest candidate. As a result, the second digest candidate selection unit 17 can suitably determine the second digest candidate narrowed down from the first digest candidate based on the degree of relevance. On the other hand, when the combination of the first digest candidate and the non-first digest candidate is the inference target pair Ptag, the second digest candidate selection unit 17 is the non-first digest candidate of the inference target pair Ptag whose relevance is equal to or higher than the threshold value. Is added to the second digest candidate. In this case, the second digest candidate selection unit 17 can suitably incorporate the non-first digest candidate, which has a high degree of relevance to the first digest candidate, into the second digest candidate.

The output control unit 18 performs output control based on the second digest candidate information Idc2 supplied from the second digest candidate selection unit 17. In the first example, the output control unit 18 generates an output signal S1 related to the second digest candidate information Idc2, and transmits the generated output signal S1 to the output device 3 via the interface 13. In this case, the output control unit 18 transmits, for example, an output signal S1 for reproducing the section data as the second digest candidate to the output device 3, so that the section data as the second digest candidate is transmitted to the output device 3. You may regenerate it. As a result, the output control unit 18 can make the viewer confirm the suitability of the second digest candidate as a digest. In the second example, the output control unit 18 stores the second digest candidate information Idc2 in the storage device 4 via the interface 13. In the third example, the output control unit 18 transmits the second digest candidate information Idc2 to the external device that performs the final digest generation process via the interface 13.

The processor 11 stores, for example, each component of the first digest candidate selection unit 14, the pair determination unit 15, the relevance calculation unit 16, the second digest candidate selection unit 17, and the output control unit 18 described in FIG. This can be achieved by executing a program stored in the device 4 or the memory 12. Further, each component may be realized by recording a necessary program in an arbitrary non-volatile storage medium and installing it as needed. It should be noted that each of these components is not limited to being realized by software by a program, and may be realized by any combination of hardware, firmware, and software. Further, each of these components may be realized by using a user-programmable integrated circuit such as an FPGA (field-programmable gate array) or a microcomputer. In this case, this integrated circuit may be used to realize a program composed of each of the above components. As described above, each component may be realized by any controller including hardware other than the processor. The above is the same in other embodiments described later.

(4) Specific Examples Next, specific examples regarding the selection of the second digest candidate will be described. Hereinafter, the first selection example in which the second digest candidate is selected from the first digest candidate, and the non-first digest candidate which is highly related to the first digest candidate in addition to the first digest candidate are selected as the second digest candidate. 2 An example of selection will be described.

FIG. 4 is a diagram showing an outline of the selection process of the first digest candidate common to the first selection example and the second selection example.

First, the first digest candidate selection unit 14 extracts a plurality of section data having a unit time length from the material data D1 and refers to the importance inferrer information D3 for the extracted section. Enter the data sequentially. As a result, the first digest candidate selection unit 14 calculates the importance of each of the section data. Then, the first digest candidate selection unit 14 considers the section data having an importance of more than a predetermined threshold value as the first digest candidate, and the section data having an importance less than the threshold value as a non-first digest candidate. ..

FIG. 5 is a diagram showing an outline of the selection process of the second digest candidate according to the first selection example after the selection of the first digest candidate.

In the first selection example, the pair determination unit 15 determines the inference target pair Ptag that combines the section data that are the first digest candidates. In this case, the pair determination unit 15 may determine all combinations of pairs of the first digest candidates as the inference target pair Ptag, or determine a part of the pair of the first digest candidates as the inference target pair Ptag. May be good.

Then, the relevance calculation unit 16 inputs each of the inference target pairs Ptag determined by the pair determination unit 15 into the relevance inference device configured with reference to the relevance inference device information D2. As a result, the relevance calculation unit 16 calculates the relevance of each of the inference target pair Ptag. Then, the second digest candidate selection unit 17 selects the section data constituting the inference target pair Ptag whose calculated relevance degree is equal to or higher than a predetermined threshold value as the second digest candidate. As a result, the second digest candidate selection unit 17 can suitably narrow down only the section data having a high degree of relevance as the second digest candidate from the section data that becomes the first digest candidate.

Here, a supplementary explanation will be given on a method of determining a part of the pair of the first digest candidates as the inference target pair Ptag. In the first determination method, the pair determination unit 15 selects only the pair of the section data of the first digest candidate whose reproduction time is within a predetermined time difference as the inference target pair Ptag. In the second determination method, the pair determination unit 15 determines the inference target pair Ptag only for the first digest candidate corresponding to the section data selected from the material data D1 at regular time (for example, 2 seconds) intervals. In the third selection method, the pair determination unit 15 clusters the section data that is the first digest candidate, and determines the inference target pair Ptag only for the section data included in the specific class. In this case, the pair determination unit 15 performs, for example, a predetermined feature amount extraction process for each section data, and determines which of the preset classes belongs to (that is, the class) based on the extracted feature amount. Identification). In another example, the pair determination unit 15 may perform clustering based on the user input to the input device 2.

FIG. 6 is a diagram showing an outline of the selection process of the second digest candidate according to the second selection example after the selection of the first digest candidate.

In the second selection example, the pair determination unit 15 determines an inference target pair Ptag that is taken out from each of the first digest candidate and the non-first digest candidate and combined. In this case, the pair determination unit 15 may select all combinations of pairs of the first digest candidate and the non-first digest candidate as the inference target pair Ptag, and select some combinations thereof as the inference target pair Ptag. You may.

Then, the relevance calculation unit 16 inputs each of the inference target pairs Ptag determined by the pair determination unit 15 into the relevance inference device configured with reference to the relevance inference device information D2. As a result, the relevance calculation unit 16 calculates the relevance of each of the inference target pair Ptag. Then, the second digest candidate selection unit 17 sets the non-first digest candidate constituting the inference target pair Ptag whose calculated relevance is equal to or higher than a predetermined threshold value and all the first digest candidates as the second digest candidate. Select. In this case, the second digest candidate selection unit 17 can incorporate the non-first digest candidate having a high degree of relevance to the first digest candidate into the second digest candidate. As a result, the scenes around the important scenes necessary for understanding the story can be suitably incorporated into the digest candidates.

Here, a supplementary explanation will be given on a method of selecting a partial combination of a pair of a first digest candidate and a non-first digest candidate as an inference target pair Ptag. In the first selection method, the pair determination unit 15 selects only the pair of the section data between the first digest candidate and the non-first digest candidate whose reproduction time is within a predetermined time difference as the inference target pair Ptag. In the second selection method, the pair determination unit 15 targets only the first digest candidate and the non-first digest candidate corresponding to the section data selected from the material data D1 at regular time (for example, 2 seconds) intervals, and the pair is inferred. Select Ptag. In the third selection method, the pair determination unit 15 clusters the first digest candidate and the non-first digest candidate, and the inference target pair targets only the first digest candidate and the non-first digest candidate included in the specific class. Select Ptag.

(5) Learning of the relevance inferior device Next, the generation of the relevance inferior information D2 by learning the relevance inferior device will be described. FIG. 7 is a schematic configuration diagram of a learning system that generates the relevance inferior information D2. The learning system has a learning device 6 that can refer to the learning data D4.

The learning device 6 has the same configuration as that of the information processing device 1 shown in FIG. 2, for example, and mainly has a processor 21, a memory 22, and an interface 23. The learning device 6 may be an information processing device 1 or any device other than the information processing device 1.

The learning data D4 includes learning material data which is learning material data and a label indicating whether the learning material data is important or non-important for each unit interval. Here, a digest (learning digest) is generated in advance from the learning material data by manual work, and important labels are attached to the section data of the learning material data adopted as the digest, and the section data other than the digest is assigned. Is labeled as non-important. Hereinafter, the section data of the learning material data with the important label (that is, the learning digest) is referred to as "important data", and the section data of the learning material data with the non-important label is "non-important". Called "data".

FIG. 8 shows an example of the functional block configuration of the learning device 6. Functionally, the learning device 6 mainly has a pair determination unit 61 and a learning unit 62. The pair determination unit 61 and the learning unit 62 are realized by, for example, the processor 21.

The pair determination unit 61 refers to the learning data D4, determines the inference target pair Ptag for learning from the interval data of the learning material data, and generates a correct label for these pairs. A specific example of the processing of the pair determination unit 61 will be described later.

The learning unit 62 learns the relevance inference device based on the combination of the inference target pair Ptag for learning determined by the pair determination unit 61 and the correct answer label. In this case, the learning device 6 has the output of the relevance inference device when the inference target pair Ptag for learning is input to the relevance inference device, and the correct answer label corresponding to the input inference target pair Ptag for learning. Determine the parameters of the relevance inferior so that the error (loss) is minimized. The algorithm for determining the above parameters to minimize the loss may be any learning algorithm used in machine learning such as gradient descent or backpropagation. The learning unit 62 may further perform a process of learning the importance inference device with reference to the learning data D4 and generating the importance inference device information D3.

Next, a specific example of the processing of the pair determination unit 61 (first learning example and second learning example) will be described.

In the first learning example, the pair determination unit 61 determines all two combinations of the important data and the non-important data constituting the learning material data as the inference target pair Ptag for learning. Then, the pair determination unit 61 attaches a correct answer label indicating that it is a correct example to the inference target pair Ptag for learning that is a pair of important data, and the inference target pair Ptag for learning that is another pair. On the other hand, a correct answer label indicating that it is a negative example is attached. The "other pair" refers to a pair of non-important data and a pair of important data and non-important data. After that, the learning unit 62 considers that, for example, in the case of a correct answer label indicating that it is a negative example, the degree of relevance that becomes a correct answer is the lowest value, and in the case of a correct answer label indicating that it is a correct example, it is regarded as a correct answer. The degree of relevance is considered to be the maximum value, and the relevance inferior is learned.

According to the first learning example, the learning device 6 can suitably learn the relevance inferior so that the higher the probability that the pair of input interval data is included in the digest at the same time, the higher the relevance is output. ..

In the second learning example, the pair determination unit 61 determines arbitrary two interval data (including important data and non-important data) of the learning material data as the inference target pair Ptag for learning. Then, the learning unit 62 determines a value corresponding to the difference in reproduction time of the two section data constituting the inference target pair Ptag for learning as the correct label for the inference target pair Ptag for learning. The "value according to the difference in reproduction time" is, for example, a number closer to 1 (for example, the maximum value of the degree of relevance) as the reproduction time of the two section data is closer, and 0 (for example, the lowest value of the degree of relevance) as the distance is farther. It is a value normalized according to the range of the degree of relevance so that the numbers are close to each other.

According to the second learning example, the learning device 6 can suitably learn the relevance degree inferior so that the pair of section data having a deep connection as a story outputs a higher relevance degree.

The information processing apparatus 1 applies the relevance inferior learned in the second learning example to, for example, the second selection example shown in FIG. 7, so that the information processing apparatus 1 is non-first, which is close in time to the first digest candidate. The digest candidate can be suitably incorporated as the second digest candidate. As a result, the information processing apparatus 1 can suitably select the peripheral scenes of the important scenes as the second digest candidate, and can suitably support the generation of the digest in which the story is easy to understand.

(6) Processing Flow FIG. 9 is an example of a flowchart showing a procedure of processing executed by the information processing apparatus 1 in the first embodiment. The information processing apparatus 1 executes the processing of the flowchart shown in FIG. 9, for example, when a user input instructing the start of the processing is detected.

First, the first digest candidate selection unit 14 of the information processing device 1 acquires the material data D1 from the storage device 4 via the interface 13 (step S11). When the material data D1 corresponding to a plurality of contents is stored in the storage device 4, the first digest candidate selection unit 14 acquires the material data D1 corresponding to the contents specified by the user input or the like. ..

Next, the first digest candidate selection unit 14 selects the first digest candidate from the section data constituting the material data (step S12). In this case, the first digest candidate selection unit 14 calculates the importance of each section data by inputting the section data into the importance inferior configured with reference to the importance inferior information D3, and the calculated importance. The section data that is the first digest candidate is selected based on the degree.

Next, the pair determination unit 15 generates an inference target pair Ptag including the first digest candidate (step S13). In this case, the pair determination unit 15 may generate an inference target pair Ptag in which the first digest candidates are paired according to the first selection example described above, and the first digest candidate and the non-first digest candidate may be generated according to the second selection example. An inference target pair Ptag that pairs with one digest candidate may be generated.

Next, the relevance calculation unit 16 calculates the relevance of each of the inference target pair Ptag generated in step S13 (step S14). In this case, the relevance calculation unit 16 calculates the relevance of each of the inference target pair Ptag by sequentially inputting the inference target pair Ptag into the relevance inferior configured by referring to the relevance inferior information D2. do.

Next, the second digest candidate selection unit 17 selects the second digest candidate (step S15). In this case, the second digest candidate selection unit 17 selects, for example, the first digest candidate which is the inference target pair Ptag having a high degree of relevance among the first digest candidates as the second digest candidate according to the first selection example described above. .. In another example, the second digest candidate selection unit 17 selects a non-first digest candidate having a high degree of relevance to the first digest candidate as the second digest candidate together with the first digest candidate according to the second selection example.

Next, the output control unit 18 outputs information regarding the second digest candidate (step S16). In this case, as described above, the output control unit 18 may supply the information regarding the second digest candidate to an external device such as the storage device 4, or may output the information to the output device 3.

FIG. 10 is an example of a flowchart showing a procedure of processing executed by the learning device 6 in the first embodiment. The learning device 6 executes the processing of the flowchart shown in FIG. 10, for example, when a user input instructing the start of the processing is detected.

First, the pair determination unit 61 of the learning device 6 acquires learning material data from the learning data D4 (step S21). When the learning data D4 includes learning material data corresponding to a plurality of contents, the pair determination unit 61 acquires the learning material data corresponding to the contents specified by the user input or the like.

Next, the pair determination unit 61 generates an inference target pair Ptag for learning (step S22). In this case, the pair determination unit 61 generates, for example, a pair of interval data extracted from the learning material data as an inference target pair Ptag according to either the first learning example or the second learning example described above.

Further, the pair determination unit 61 determines the correct label for the inference target pair Ptag for learning generated in step S22 (step S23). In this case, the pair determination unit 61 may determine the correct answer label based on whether or not it is a pair of important data according to the first learning example, and according to the second learning example, the section that becomes the inference target pair Ptag for learning. A value corresponding to the difference in reproduction time between the data may be determined as the correct answer label.

Then, the learning unit 62 learns the relevance inference device based on the inference target pair Ptag for learning and the correct answer label (step S24). Then, the learning device 6 generates the parameters of the relevance inferior obtained by learning as the relevance inferior information D2. The generated relevance inferior information D2 may be immediately stored in the storage device 4 by data communication between the storage device 4 and the learning device 6, and is stored in the storage device 4 via a removable storage medium. You may.

(7) Modifications Next, each modification suitable for the above embodiment will be described. The following modifications may be applied to the above-described embodiment in any combination.

(Modification 1)
The pair determination unit 15 may combine the interval data extracted from different material data as the inference target pair Ptag instead of combining the two interval data extracted from the same material data as the inference target pair Ptag.

For example, in this case, the first digest candidate selection unit 14 selects the first digest candidate from the second material data different from the material data D1. In this case, the material data D1 and the second material data may be, for example, data taken at the same place (for example, a sports venue) by different cameras in the same time zone. The second material data may be labeled in advance to identify important sections and non-important sections. In this case, the first digest candidate selection unit 14 selects the section data labeled with the important section as the first digest candidate.

Then, the pair determination unit 15 determines a pair of the first digest candidate extracted from the second material data and the section data extracted from the material data D1 as the inference target pair Ptag. In this case, the second digest candidate selection unit 17 is, for example, the section data of the material data D1 constituting the inference target pair Ptag whose relevance degree is equal to or higher than a predetermined value, and the first digest candidate extracted from the second material data. Is selected as the second digest candidate. According to this aspect, the information processing apparatus 1 can suitably select a digest candidate from a plurality of material data.

(Modification 2)
The information processing apparatus 1 may calculate and output the degree of relevance to the inference target pair Ptag specified by the user input.

FIG. 11 is an example of a functional block diagram of the information processing apparatus 1A according to the modified example 2. The processor 11 of the information processing device 1A has a pair determination unit 15A, a relevance calculation unit 16A, and an output control unit 18A. Hereinafter, the same components as those in the above-described embodiment will be appropriately designated by the same reference numerals, and the description thereof will be omitted.

The pair determination unit 15A determines the inference target pair Ptag based on the input signal S2 received from the input device 2 via the interface 13. For example, the pair determination unit 15A determines two section data of the material data D1 designated based on the input signal S2 as the inference target pair Ptag.

In this case, for example, the user of the information processing apparatus 1A designates the section data to be the digest candidate as one section data of the inference target pair Ptag, and determines whether or not it is suitable as the digest candidate as the other section data. Specify the section data to be. The pair determination unit 15 may accept inputs for designating section data to be the inference target pair Ptag from different material data, and determine these section data as the inference target pair Ptag. Further, the pair determination unit 15A may determine a plurality of inference target pairs Ptag based on the input signal S2.

Then, the relevance calculation unit 16A calculates and calculates the relevance of the inference target pair Ptag determined by the pair determination unit 15A based on the relevance inferior configured by referring to the relevance inferior information D2. Relevance information Ia regarding the degree is supplied to the output control unit 18A. Then, the output control unit 18A outputs based on the relevance information Ia. In this case, for example, the output control unit 18A supplies the output device 3 with an output signal S1 for displaying the relevance of the inference target pair Ptag.

When the output control unit 18A has a plurality of inference target pair Ptags, the output control unit 18A may output only the information regarding the relevance of the inference target pair Ptags corresponding to the upper predetermined number having the highest relevance. May output only the information about the relevance of the inferred pair Ptag in which is equal to or greater than a predetermined threshold. The above-mentioned predetermined number may be set to any number of 1 or more.

FIG. 12 is an example of a flowchart executed by the information processing apparatus 1A in the modification 2. First, the pair determination unit 15A of the information processing apparatus 1A accepts a user input for designating the inference target pair Ptag (step S31). In this case, the pair determination unit 15A may display the reproduction screen of the material data D1 including the seek bar or the like, and specify the section data corresponding to an arbitrary reproduction time as the inference target pair Ptag. Next, the relevance calculation unit 16A calculates the relevance of the inference target pair Ptag specified in step S31 (step S32). Then, the output control unit 18A outputs based on the degree of relevance calculated in step S32 (step S33).

As described above, the information processing apparatus 1A according to the modification 2 can suitably calculate and output the degree of relevance to the inference target pair Ptag specified by the user input.

(Modification 3)
When the material data D1 includes sound data in addition to the video data, the relevance calculation unit 16 may calculate the relevance using the sound data.

In this case, in the first example, the relevance calculation unit 16 calculates the relevance with reference to the relevance inferior information D2 based on the video data and the sound data of the two section data constituting the inference target pair Ptag. do. In this case, the parameter of the relevance inferior learned in advance so that the relevance is output when a pair of interval data including video data and sound data is input is stored in advance as the relevance inferior information D2. Is remembered in. The feature amount of the sound data may be input to the relevance inferior instead of directly inputting the sound data. In this case, after a predetermined feature amount extraction process or the like is performed on the sound data, the extracted feature amount is input to the relevance inferior. Similarly, when calculating the importance of each section data of the material data D1, the first digest candidate selection unit 14 may calculate the importance of each section data using sound data in addition to the video data. good.

In the second example, the relevance calculation unit 16 may calculate the relevance by referring to the relevance inferior information D2 based only on the sound data included in the two section data constituting the inference target pair Ptag. good. In this case, the parameters of the relevance inferior learned in advance so that the relevance is output when the pair of sound data is input are stored in the storage device 4 in advance as the relevance inferior information D2.

As described above, the information processing apparatus 1 can suitably calculate the relevance of the inference target pair Ptag using at least one of the video data and the sound data.

(Modification example 4)
In the functional block of FIG. 3, the information processing apparatus 1 does not have to include at least one of the first digest candidate selection unit 14 and the second digest candidate selection unit 17.

For example, when a label that distinguishes an important section and a non-important section is added to the material data D1 in advance, the pair determination unit 15 considers the section data of the important section as the first digest candidate, and the pair Ptag to be inferred. May be determined. In another example, the output control unit 18 may perform a predetermined output based on the relevance information Ia output by the relevance calculation unit 16. In this case, the output control unit 18A may output information about the inference target pair Ptag having the highest relevance for the upper predetermined number, and output only the information about the inference target pair Ptag whose relevance is equal to or higher than the predetermined threshold. You may. The above-mentioned predetermined number may be set to any number of 1 or more. The above-mentioned "information about the inference target pair Ptag" may be the section data itself constituting the inference target pair Ptag, and the time information of the section data constituting the inference target pair Ptag (information indicating the reproduction time in the material data D1). ) May be.

<Second Embodiment>
FIG. 13 is a functional block diagram of the information processing apparatus 1X according to the second embodiment. The information processing apparatus 1X mainly includes a pair determination unit 15X and a relevance calculation unit 16X.

The pair determining means 15X determines a pair of data including at least one of video data or sound data, and at least one of the data is a candidate for the first digest. Here, the "video data" may be composed of one image or may include a plurality of images. The above-mentioned "data" and "pair" can be "interval data" and "inference target pair Ptag" in the first embodiment (including modification, the same applies hereinafter), respectively. Further, the pair determination means 15X can be the pair determination unit 15 or the pair determination unit 15A in the first embodiment.

The relevance calculation means 16X calculates the relevance degree indicating the degree of probability that the pair determined by the pair determination means 15X is included in the digest at the same time. The relevance calculation means 16X can be the relevance calculation unit 16 or the relevance calculation unit 16A in the first embodiment.

FIG. 14 is an example of a flowchart executed by the information processing apparatus 1X in the second embodiment. First, the pair determining means 15X determines a pair of data including at least one of video data or sound data, and at least one of the data is a candidate for the first digest (step S41). The relevance calculation means 16X calculates the relevance degree indicating the degree of probability that the pair determined by the pair determination means 15X is simultaneously included in the digest (step S42).

The information processing apparatus 1X according to the second embodiment can suitably calculate the degree of relevance as an index for determining whether or not the paired data should be included in the digest at the same time.

In each of the above-described embodiments, the program is stored using various types of non-transitory computer readable medium and can be supplied to a processor or the like which is a computer. Non-temporary computer-readable media include various types of tangible storage media. Examples of non-temporary computer-readable media include magnetic storage media (eg, flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (eg, magneto-optical disks), CD-ROMs (ReadOnlyMemory), CD-Rs, Includes CD-R / W, semiconductor memory (eg, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (RandomAccessMemory)). The program may also be supplied to the computer by various types of temporary computer readable medium. Examples of temporary computer-readable media include electrical, optical, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

Other than that, a part or all of each of the above embodiments may be described as in the following appendix, but is not limited to the following.

[Appendix 1]
A pair determining means for determining a pair of data including at least one of video data or sound data, wherein at least one of the data is a first digest candidate which is a digest candidate.
A relevance calculation means for calculating the relevance indicating the degree of probability that the pair is simultaneously included in the digest, and
Information processing device with.

[Appendix 2]
The information according to Appendix 1, wherein the pair determining means determines the pair based on the section data constituting the section of the material data when the material data used as the material in the generation of the digest is divided into a plurality of sections. Processing equipment.

[Appendix 3]
The information processing apparatus according to Appendix 2, further comprising a second digest candidate selection means for selecting the section data to be the second digest candidate based on the degree of relevance.

[Appendix 4]
The pair determination means determines the two section data as the first digest candidate as the pair.
The information processing apparatus according to Appendix 3, wherein the second digest candidate selection means selects the first digest candidate to be the second digest candidate based on the degree of relevance.

[Appendix 5]
The pair determination means determines the first digest candidate and the non-first digest candidate, which is the section data that is not the first digest candidate, as the pair.
The information processing apparatus according to Appendix 3, wherein the second digest candidate selection means selects the non-first digest candidate to be the second digest candidate based on the degree of relevance.

[Appendix 6]
The information according to any one of Supplementary Provisions 2 to 5, further comprising a first digest candidate selection means for selecting the first digest candidate from the section data based on the importance calculated for each of the section data. Processing equipment.

[Appendix 7]
The information processing apparatus according to any one of Supplementary note 2 to 6, wherein the pair determining means determines as the pair the two section data in which the difference in reproduction time is within a predetermined time difference.

[Appendix 8]
The information processing apparatus according to any one of Supplementary note 2 to 6, wherein the pair determining means determines the pair from the section data extracted from the material data at predetermined time intervals.

[Appendix 9]
The information processing apparatus according to any one of Supplementary note 2 to 6, wherein the pair determination means clusters the interval data and determines the pair from the interval data belonging to a predetermined class.

[Appendix 10]
The relevance calculation means uses a pair of section data of the learning digest created from the learning material data as a positive example, and a pair of section data of the learning material data other than the positive example. The information processing apparatus according to any one of Supplementary note 1 to 9, which calculates the degree of relevance based on the relevance inferior learned as a negative example.

[Appendix 11]
The relevance calculation means reproduces a pair of data when it is assumed that the pair of data is extracted from the same material data when a pair of data including at least one of video data or sound data is input. The information processing apparatus according to any one of Supplementary note 1 to 9, which calculates the degree of relevance based on a relevance inferior learned to output information corresponding to a time difference.

[Appendix 12]
The information processing apparatus according to any one of Supplementary note 1 to 11, further comprising an output control means for outputting information on the degree of relevance or information on a second digest candidate selected based on the degree of relevance.

[Appendix 13]
By computer
A pair of data including at least one of video data or sound data, and the pair in which at least one data is a first digest candidate, which is a digest candidate, is determined.
Calculate the degree of association that indicates the degree of probability that the pair will be included in the digest at the same time.
Control method.

[Appendix 14]
A pair determining means for determining a pair of data including at least one of video data or sound data, wherein at least one of the data is a first digest candidate which is a digest candidate.
A storage medium in which a program for operating a computer as a relevance calculation means for calculating a relevance indicating the degree of probability that the pair is simultaneously included in the digest is stored.

Although the invention of the present application has been described above with reference to the embodiment, the invention of the present application is not limited to the above embodiment. Various changes that can be understood by those skilled in the art can be made within the scope of the present invention in terms of the configuration and details of the present invention. That is, it goes without saying that the invention of the present application includes all disclosure including claims, various modifications and modifications that can be made by those skilled in the art in accordance with the technical idea. In addition, each disclosure of the above-mentioned patent documents cited shall be incorporated into this document by citation.

1, 1A, 1B, 1X Information processing device 2 Input device 3 Output device 4 Storage device 6 Learning device 8 Terminal device 100, 100B Digest candidate selection system

Claims

A pair determining means for determining a pair of data including at least one of video data or sound data, wherein at least one of the data is a first digest candidate which is a digest candidate.
A relevance calculation means for calculating the relevance indicating the degree of probability that the pair is simultaneously included in the digest, and
Information processing device with.
The pair-determining means according to claim 1, wherein the pair-determining means determines the pair based on the section data constituting the section of the material data when the material data as the material is divided into a plurality of sections in the generation of the digest. Information processing device.
The information processing apparatus according to claim 2, further comprising a second digest candidate selection means for selecting the section data to be the second digest candidate based on the degree of relevance.
The pair determination means determines the two section data as the first digest candidate as the pair.
The information processing apparatus according to claim 3, wherein the second digest candidate selection means selects the first digest candidate to be the second digest candidate based on the degree of relevance.
The pair determination means determines the first digest candidate and the non-first digest candidate, which is the section data that is not the first digest candidate, as the pair.
The information processing apparatus according to claim 3, wherein the second digest candidate selection means selects the non-first digest candidate to be the second digest candidate based on the degree of relevance.
The item according to any one of claims 2 to 5, further comprising a first digest candidate selection means for selecting the first digest candidate from the section data based on the importance calculated for each of the section data. Information processing device.
The information processing device according to any one of claims 2 to 6, wherein the pair determining means determines as the pair the two section data whose reproduction time difference is within a predetermined time difference.
The information processing apparatus according to any one of claims 2 to 6, wherein the pair determining means determines the pair from the section data extracted from the material data at predetermined time intervals.
The information processing apparatus according to any one of claims 2 to 6, wherein the pair determination means clusters the interval data and determines the pair from the interval data belonging to a predetermined class.
The relevance calculation means uses a pair of section data of the learning digest created from the learning material data as a positive example, and a pair of section data of the learning material data other than the positive example. The information processing apparatus according to any one of claims 1 to 9, which calculates the degree of relevance based on the relevance inferior learned as a negative example.
The relevance calculation means reproduces a pair of data when it is assumed that the pair of data is extracted from the same material data when a pair of data including at least one of video data or sound data is input. The information processing apparatus according to any one of claims 1 to 9, which calculates the degree of relevance based on a relevance inferior learned to output information corresponding to a time difference.
The information processing apparatus according to any one of claims 1 to 11, further comprising an output control means for outputting information on the degree of relevance or information on a second digest candidate selected based on the degree of relevance.
By computer
A pair of data including at least one of video data or sound data, and the pair in which at least one data is a first digest candidate, which is a digest candidate, is determined.
Calculate the degree of association that indicates the degree of probability that the pair will be included in the digest at the same time.
Control method.
A pair determining means for determining a pair of data including at least one of video data or sound data, wherein at least one of the data is a first digest candidate which is a digest candidate.
A storage medium in which a program for operating a computer as a relevance calculation means for calculating a relevance indicating the degree of probability that the pair is simultaneously included in the digest is stored.