JP5424306B2 - Information processing apparatus and method, program, and recording medium - Google Patents

Description

  The present invention relates to an information processing apparatus and method, a program, and a recording medium, and in particular, when performing predetermined processing on content according to content classification, identifies and identifies the optimal content classification for the processing The present invention relates to an information processing apparatus and method, a program, and a recording medium that can execute the processing by a method according to classification.

  With the spread of digital television broadcasting, the use of electronic program guides (EPG (Electric Program Guide)) has become common. In addition to the program title and broadcast date, the data constituting the EPG indicates program classification such as information indicating a genre such as whether the program is a news program or a soccer program (hereinafter referred to as genre information). Contains information.

  By the way, the program genre information obtained from the EPG is used for various functions in the recording / reproducing apparatus. For example, there is a so-called digest playback function that creates and plays back summary video from recorded video so that the contents of a large amount of recorded video can be easily grasped. The genre information of the program is used for the purpose of improvement (see, for example, Patent Document 1).

In Patent Document 1, a predetermined feature amount detected from a recorded video is weighted based on genre information of a program acquired from the EPG, and a video to be digest-reproduced is determined based on the result.
JP 2003-283993 A

  However, the EPG is mainly created on the broadcast station side, for example, so as to be convenient for the viewer to select a program. The genre information shown therein is also classified according to the viewer's program selection. Therefore, the classification may not necessarily be suitable for processing on the device side, for example, digest playback.

  The present invention has been made in view of such a situation, and makes it possible to identify the most suitable content classification for processing on the device side.

An information processing apparatus according to an aspect of the present invention extracts a predetermined number of frames from a plurality of first contents, extracts feature amounts from the extracted frames, and extracts the extracted first feature amounts. A storage means for storing a multi-dimensional vector composed of: an extraction means for extracting a predetermined number of frames from the second content and extracting a second feature quantity for each frame; and the multi-dimensional vector to the each of the plurality of the first feature amount, of the frames of the second predetermined number extracted from the content, the distance between the second feature amounts extracted from the frames processed a calculating means for calculating for, among the distances calculated for each amount the second feature by the calculation means, and holds only the minimum distance, and generates a feature vector composed of the minimum distance Comprising a vector generation unit, using the feature vector generated by said generating means performs a process based on a predetermined algorithm and a parameter generating means for generating parameters for classifying the content.

  The extraction unit may extract the second feature amount from a predetermined part of the second content.

The algorithm may be one of a steepest descent method, a support vector machine, and backpropagation .

According to an information processing method of one aspect of the present invention, a predetermined number of frames are extracted from a plurality of first contents, a feature amount is extracted from each of the extracted frames, and the extracted first feature amount Are stored in the storage means, a predetermined number of frames are extracted from the second content, a second feature amount is extracted for each frame, and a plurality of the plurality of vectors constituting the multi-dimensional vector Calculating a distance between each of the first feature values and the second feature value extracted from a frame to be processed among a predetermined number of frames extracted from the second content; the second among the distances calculated for each feature quantity, retains only the minimum distance, and generates a feature vector composed of the minimum distance, given with reference to the feature vector the generated Performs processing based on the algorithm, comprises the step of generating a parameter for classifying the content.

A program according to an aspect of the present invention is configured by extracting a predetermined number of frames from a plurality of first contents, extracting feature amounts from the extracted frames, and configuring the extracted first feature amounts. A plurality of dimensional vectors stored in the storage means, a predetermined number of frames are extracted from the second content, a second feature amount is extracted for each frame, and a plurality of the second dimensional vectors constituting the multi-dimensional vector are extracted. Calculating a distance between each of the one feature quantity and the second feature quantity extracted from a frame to be processed among a predetermined number of frames extracted from the second content; of the distances calculated every two feature amounts, only holds the minimum distance, and generates a feature vector composed of the minimum distance, given a using the feature vectors the generated It performs processing based on Gorizumu, computer readable program for executing a process including the step of generating a parameter for classifying the content.

  A recording medium according to one aspect of the present invention records the program.

In the information processing apparatus, method, and program according to one aspect of the present invention , a predetermined number of frames are extracted from a plurality of first contents, and feature amounts are extracted from the extracted frames. A multi-dimensional vector composed of the first feature amount is stored , a predetermined number of frames are extracted from the second content, and a second feature amount is extracted for each frame to form a multi-dimensional vector. A distance between each of the plurality of first feature values and a second feature value extracted from a frame to be processed among a predetermined number of frames extracted from the second content is calculated . Of the distances calculated for each of the two feature amounts , only the minimum distance is retained, and a feature vector composed of the minimum distance is generated.

  According to one aspect of the present invention, it is possible to identify a content classification that is optimal for a predetermined process to be executed, and to execute the predetermined process on the content by a method according to the classification.

It is a figure which shows the structure of one Embodiment of the recording / reproducing apparatus to which this invention is applied. It is a figure which shows the example of teacher data. It is a figure which shows the example of other teacher data. It is a figure which shows the structure of one Embodiment of the learning device to which this invention is applied. It is a flowchart explaining the process regarding acquisition of reference data. It is a flowchart explaining the process regarding the production | generation of an identification parameter. It is a figure explaining the production | generation of a feature vector. It is a flowchart explaining a classification identification process. It is a figure which shows the example of the program of chapter information detection object. It is a figure which shows the example of the other program of chapter information detection object. It is a figure which shows the example of the other program of chapter information detection object. And FIG. 16 is a block diagram illustrating a configuration example of a personal computer.

  Embodiments of the present invention will be described below with reference to the drawings.

[Configuration of recording / playback apparatus]
FIG. 1 shows a configuration example of a recording / reproducing apparatus 1 to which the present invention is applied. The recording / playback apparatus 1 has a function of recording a received digital television broadcast program and performing digest playback of the recorded program. When performing the digest playback, the recording / playback apparatus 1 detects a chapter break point and assigns a score indicating a priority for selecting a video to be played back in the digest playback.

  As will be described later, the chapter breakpoints and scores are assigned by identifying the program classification suitable for detecting chapter breakpoints and scores (hereinafter referred to as chapter information as appropriate), and according to the identified program classification. Chapter information is detected by the method described above.

  For example, digital data of a digital television broadcast wave received by the receiving unit supplied from a receiving unit (not shown) is input to the data separating unit 11. The data separation unit 11 separates the input digital data into EPG (electronic program guide) data, audio data, and video data. Hereinafter, audio data and video data are collectively referred to as AV data as appropriate.

  The data separation unit 11 supplies the separated EPG data to the holding unit 22 to hold it, and supplies the separated AV data to the input control unit 12. When the received broadcast program is recorded, the input control unit 12 supplies the AV data supplied from the data separation unit 11 to the holding unit 20 for holding. The input control unit 12 also supplies the AV data supplied from the data separation unit 11 to the decoder 13 as a chapter information detection target.

  The decoder 13 separates the AV data as the detection target of the chapter information supplied from the input control unit 12 or the AV data read out as the detection target of the chapter information from the holding unit 20 into audio data and video data. Are supplied to the audio feature amount extraction unit 14 and the video data are supplied to the video feature amount extraction unit 15, respectively.

  The audio feature amount extraction unit 14 extracts volume, frequency spectrum, left and right channel correlation values, and the like from the audio data supplied from the decoder 13 as audio feature amounts, and outputs them to the feature vector generation unit 16 and the chapter information detection unit 18. Supply.

  The video feature amount extraction unit 15 extracts a color histogram, a color moment, a difference image, a reduced image, and the like from the video data supplied from the decoder 13 as video feature amounts, and a feature vector generation unit 16 and a chapter information detection unit. 18 is supplied. When a frame image is used as a feature quantity extraction target, it is possible to divide the frame into small areas and connect the feature quantities of each area to obtain a feature quantity of one frame. . In particular, it is effective when using feature quantities that do not have position or shape information by themselves, such as color histograms and color moments.

  The feature vector generation unit 16 is used by the identification unit 17 to identify the classification of the program to which the chapter information is attached from the feature amounts supplied from the audio feature amount extraction unit 14 or the video feature amount extraction unit 15. A predetermined feature amount is selected, and a vector having the selected feature amount as an element (hereinafter referred to as a feature vector) is generated. The feature vector generation unit 16 supplies the generated feature vector to the identification unit 17.

  Based on the feature vector supplied from the feature vector generation unit 16, the identification unit 17 identifies a program classification (in the present example, a program classification suitable for detecting chapter information). For example, the discriminating unit 17 includes a discriminator such as a linear discriminator, a nonlinear discriminator, or a neural network, and generates each element constituting the feature vector based on the discriminating parameter set by the learning device (described later). The program classification is identified based on the divided area of the feature space to which the distribution of the arranged elements belongs.

  The identification unit 17 supplies information indicating the classification of the program (hereinafter referred to as classification information) to the chapter information detection unit 18 as an identification result. The chapter information detection unit 18 detects chapter information according to the program classification indicated by the classification information supplied from the identification unit 17, supplies it to the holding unit 19, and holds it.

  For example, the chapter information detection unit 18 selects a feature amount according to the program classification from the feature amounts supplied from the audio feature amount extraction unit 14 or the video feature amount extraction unit 15 and also according to the program classification. Perform arithmetic processing.

  That is, in this case, the chapter information detection unit 18 holds execution data (for example, a program including parameters and algorithms) for executing selection and calculation of feature amounts according to the program classification for each program classification. The chapter information is detected by selecting and executing the execution data corresponding to the classification of the program.

  The playback unit 21 reads the AV data held in the holding unit 20 and performs normal playback and digest playback. When performing digest reproduction, the reproduction unit 21 appropriately reads out and reproduces video sections in which the score assigned to the chapter is equal to or greater than a certain value based on the chapter information held in the holding unit 19. To do. That is, the video is thinned and reproduced based on the chapter information.

  In a program that broadcasts a soccer game, as shown in FIG. 2, many scenes including a video of the ground are normally broadcasted. Therefore, from a program that broadcasts a soccer game, for example, a color histogram with a high green frequency is displayed. Many (eg continuously) are obtained.

  Therefore, in learning performed by a learning device, which will be described in detail later, a program that broadcasts a soccer game composed of a scene including a ground image as shown in FIG. 2 is set as teacher data whose classification is “soccer program”. A feature vector of the color histogram for each frame obtained from the teacher data is extracted.

  Then, the learning device generates an identification parameter capable of identifying that the classification is “soccer program”. That is, for example, an identification parameter is generated for generating a straight line or the like that divides the feature space so that the distribution of the green frequency obtained from the color histogram in the feature space is within the region of “soccer program”. The identification parameter set in such a manner is set in the identification unit 17.

  In addition, as shown in Fig. 3, programs that report incidents and events broadcast many scenes that include images of people and studios. A large number of color histograms with high frequency are obtained.

  Therefore, in learning performed by a learning device, which will be described in detail later, a program that reports an incident or event composed of a scene including a person and a studio video as shown in FIG. Then, the feature vector of the color histogram for each frame obtained from the teacher data is extracted.

  Then, the learning device generates an identification parameter capable of identifying that the classification is “news program”. That is, for example, an identification parameter for generating a straight line or the like that divides the feature space so that the distribution of the frequency of a specific color on the feature space is within the area of the “news program” is generated. The identification parameter set in such a manner is set in the identification unit 17.

  Returning to the description of FIG. 1, the control unit 41 controls the entire recording / reproducing apparatus 1 to execute processing such as program recording, normal reproduction, and digest reproduction.

[About the configuration of the learning device]
FIG. 4 is a diagram showing a configuration of an embodiment of a learning device to which the present invention is applied. The learning device 100 illustrated in FIG. 4 includes an input control unit 111, a decoder 112, a frame extraction unit 113, a video feature amount extraction unit 114, a reference data storage unit 115, a distance calculation unit 116, a minimum distance holding unit 117, and a learning algorithm process. Unit 118, identification parameter holding unit 119, drive 120, and communication unit 121.

  The input control unit 111 controls input of video data input from the outside. Here, an example in which video data is input and an identification parameter is generated from the video data will be described. Therefore, the description will be continued assuming that video data is input to the learning device 100. Also, the configuration of the learning device 100 shown in FIG. 4 shows a configuration for processing a video stream. However, for example, audio data may be input to the learning device 100, and an identification parameter may be generated from the audio data. In such a case, the learning device 100 is configured to control input of audio data and generate an identification parameter from the audio data.

  The decoder 112 decodes the video data whose input is controlled by the input control unit 111. If the video data is encoded in some way, decoding corresponding to the encoding is performed in the decoding 112. The decoded video data is supplied to the frame extraction unit 113.

  The frame extraction unit 113 extracts frames from the decoded video data based on a predetermined condition. Although it is possible to treat all frames of the decoded video data as processing, doing so will lead to an increase in processing load and an increase in processing time, so in order to reduce the processing load and processing time, here The description will be continued assuming that a predetermined number of frames are extracted as processing targets based on a predetermined condition. Note that the predetermined conditions and the like will be clarified in the description with reference to the flowchart of FIG.

  The frame extracted by the frame extraction unit 113 is supplied to the video feature amount extraction unit 114 as a frame to be processed. The video feature amount extraction unit 114 extracts feature amounts from the supplied frames.

  The decoder 112 executes the same processing as that of the decoder 13 (FIG. 1) of the recording / playback apparatus 1, and the video feature amount extraction unit 114 performs the same processing as that of the video feature amount extraction unit 15 (FIG. 1) of the recording / playback device 1. Execute. Therefore, when the video feature quantity extraction unit 15 extracts a color histogram, a color moment, a difference image, a reduced image, and the like from the frame as described above as a video feature quantity, the video feature quantity extraction unit 114 also selects the color from the frame. Histograms, difference images, reduced images, and the like are extracted as video feature amounts.

  The video feature amount from the video feature amount extraction unit 114 is supplied to the reference data storage unit 115 or the distance calculation unit 116. When the video feature amount is used as reference data, it is supplied to and stored in the reference data storage unit 115. On the other hand, when the video feature amount is extracted from the video data as the teacher data and is the data to be compared with the reference data, the video feature amount is supplied to the distance calculation unit 116.

  The reference data storage unit 115 stores reference data. This reference data storage unit 115 is data stored in advance as data to be compared in order to generate an identification parameter.

  The reference data stored in the reference data storage unit 115 may be data created from video data whose input is controlled by the input control unit 111, or data created in advance by another device or the like. In the case of data created by another device, for example, it may be stored in the removable disk 141 and distributed. In such a case, the removable disk 141 is set in the drive 120, and the reference data is supplied to and stored in the reference data storage unit 115 by being read from the set removable disk 141.

  Further, reference data may be distributed via a network. When the reference data is distributed via the network, the distributed reference data is received by the communication unit 121, and the received reference data is supplied to the reference data storage unit 115 to be stored.

  In this way, the reference data may be generated and stored by the learning device 100, supplied via a recording medium such as the removable disk 141, and stored, or may be stored in a network. It is also possible to be supplied and stored via

  The learning device 100 can also be provided in the recording / reproducing apparatus 1. When the learning device 100 is provided in the recording / reproducing device 1 and the learning device 100 itself is configured to generate the reference data, the reference data can be generated from the video data input to the recording / reproducing device 1. Also, when configured to be distributed via a network, or configured to be distributed using a recording medium, the reference data can be easily updated.

  Returning to the description of the learning device 100 shown in FIG. 4, the distance calculation unit 116 calculates the reference data stored in the reference data storage unit 115 and the frame to be processed from the video feature amount extraction unit 114. Calculate the distance. Here, the distance is calculated using the feature quantity as reference data and the feature quantity extracted from the frame.

  The distance (distance data) calculated by the distance calculation unit 116 is supplied to the minimum distance holding unit 117. The minimum distance holding unit 117 holds a distance from a frame whose distance from the reference data is the minimum distance among a plurality of frames to be processed. For example, when the reference data is composed of feature amounts for 30 frames, the minimum distance to each frame is held in the minimum distance holding unit 117, and therefore 30 minimum distances are held. For example, the 30 minimum distances held in this way are supplied to the learning algorithm processing unit 118 as feature vectors.

  The learning algorithm processing unit 118 generates an identification parameter using the supplied feature vector based on a predetermined algorithm. The generated identification parameter is supplied to the identification parameter holding unit 119 and held.

  The identification parameter held in the identification parameter holding unit 119 is supplied to and held in the identification unit 17 (FIG. 1) of the recording / reproducing apparatus 1. For example, the removable disk 141 is set in the drive 120, and the identification parameter held in the identification parameter holding unit 119 is written in the set removable disk 141. Then, the removable disk 141 in which the identification parameter is written is set in the recording / reproducing apparatus 1, whereby the identification parameter is supplied to the identification unit 17.

  Further, the identification parameter may be distributed via a network. In this case, the identification parameter held in the identification parameter holding unit 119 is read by the communication unit 121 and supplied to the identification unit 17 of the recording / reproducing apparatus 1 under the control of the communication unit 121.

  Learning of the learning device 100 having such a configuration will be described below.

[About obtaining reference data]
First, processing related to acquisition of reference data stored in the reference data storage unit 115 will be described. The reference data needs to be stored in the reference data storage unit 115 before the identification parameter is generated. Therefore, before explaining the generation of the identification parameter, the processing when the learning device 100 creates reference data will be described with reference to the flowchart of FIG.

  In step S101, the input control unit 111 acquires a video stream. This acquired video stream is a video stream of a program whose category is known in advance. A category is, for example, a genre to which a program belongs, such as “news” or “variety”, and is information related to classification. In the above description, the classification is the “classification” when the identification unit 17 “identifies the classification of the program based on the feature vector supplied from the feature vector generation unit 16”.

  It should be noted that the information related to this classification, that is, the category, is preferably a category assigned based on a detailed classification. For example, the category of sports is also classified in detail, and is preferably a category such as “sports relay”, “sports news”, “sports variety”.

  In the EPG, information related to a category is described as information related to a program, and the described information is often rough information such as “sports”. For example, if chapter information is detected based on such information, it may not be possible to detect appropriate chapter information. In other words, when detecting chapter information for a “Sports Broadcast” program and a “Sports News” program, the chapter information that is more suitable for each program is detected if it is detected by a different algorithm rather than the same algorithm. it can.

  The learning device 100 performs learning so that chapter information in consideration of such a situation can be detected. Therefore, it is preferable that the category related to the video data input to the input control unit 111 is also a category (information thereof) as a result of detailed classification. In the following description, it is assumed that such information is input. Continue.

  In step S101, when a video stream is acquired by the input control unit 111, in step S102, the decode 112 decodes the video data and generates a frame. Further, the frame extraction unit 113 extracts a frame to be processed. In other words, the frame to be processed is a reference frame used as reference data.

  If all the frames generated from the video data are set as reference frames, the burden of processing when generating a feature vector described later and an identification parameter generated from the feature vector will be increased. Considering this, a plurality of frames are extracted from all frames included in the category based on a predetermined rule. The predetermined rule is, for example, a rule such as random extraction, extraction at a predetermined interval (predetermined time interval, predetermined number of frames), or extraction based on a clustering method.

  In addition, when extracting a reference frame by a clustering method, for example, clustering is performed using a feature vector of a frame, which will be described later, and after selecting a predetermined number of clusters in descending order of the number of components, a frame close to the center of gravity of each cluster A method such as selection is conceivable. Moreover, it is preferable that once a reference frame is selected, that frame is used and not changed.

  In step S102, the frame extracted by the frame extraction unit 113 is supplied to the video feature amount extraction unit 114. In step S103, the video feature amount extraction unit 114 extracts a feature amount from the supplied frame (image). Examples of the feature amount include a color histogram, a color moment, a difference image, and a reduced image. The feature amount extracted by the video feature amount extraction unit 114 is supplied to the reference data storage unit 115.

  In step S104, the reference data storage unit 115 stores the feature amount extracted by the video feature amount extraction unit 114 as reference data.

  In this way, reference data is extracted from a plurality of frames for one category. For example, when N1 frames are extracted from one category, N1 pieces of reference data (features) are stored in the reference data storage unit 115 as reference data of the category.

  Since reference data is similarly extracted from a plurality of categories, for example, when reference data is extracted from M categories, reference data for M categories is stored in the reference data storage unit 115. The

  Note that the reference data stored in the reference data storage unit 115 may be generated by another device and supplied via a network or recorded on a recording medium and supplied as described above. Also in such a case, in other apparatuses, the same processing as the processing described above is executed, so that the reference data is generated.

[Generation of identification parameters]
Thus, when the reference data is stored in the reference data storage unit 115, the learning device 100 performs learning, that is, in this case, creates an identification parameter. The generation of the identification parameter will be described with reference to the flowchart of FIG.

  In step S151, a video stream is acquired. This video stream is used as teacher data and belongs to a predetermined category, and the category is classified into fine categories as in the case of generating reference data. When a video stream is acquired, information on the category to which the video stream belongs is also acquired.

  Next, in step S152, a frame is extracted. The decoder 112 decodes the video stream whose input is controlled by the input control unit 111. A predetermined number of frames among the decoded frames are extracted by the frame extraction unit 113.

  All of the video streams of programs belonging to a predetermined category may be processed. For example, if it is a 60-minute program, a 60-minute video stream may be processed. However, if this is done, the number of frames to be processed increases, leading to an increase in processing load and an increase in processing time.

  Therefore, a video stream for a predetermined time of a program belonging to a predetermined category is set as a processing target. For example, a 10-minute video stream at the beginning of a program is a processing target. In such a case, the input control unit 111 controls the video stream to be input only for the first 10 minutes of the program.

  Next, all the frames included in the 10-minute video stream may be processed. However, as in the above case, the processing load and the processing time are increased, so that a predetermined number of frames are processed. When M frames are processed as the predetermined number, the frame extraction unit 113 extracts M frames from the video stream from the decoder 112 and outputs the M frames to the video feature amount extraction unit 114. For example, the M frames are extracted at predetermined time intervals, randomly extracted, and extracted for each predetermined number of frames.

  The frame extracted by the frame extraction unit 113 is output to the video feature amount extraction unit 114 as a processing target frame. In step S153, the video feature amount extraction unit 114 extracts a predetermined feature amount from the supplied frame (image). The predetermined feature amount is a color histogram, a color moment, a difference image, a reduced image, or the like. The predetermined feature amount is the same feature amount as the reference data. That is, for example, when the feature amount as the reference data is a feature amount in the color histogram, the feature amount extracted by the video feature amount extraction unit 114 in step S153 is also the feature amount in the color histogram.

In step S154, the reference data Ri is set to “1” which is an initial value. The reference data is composed of feature amounts extracted from a plurality of frames for a plurality of categories. For example, when feature values are extracted from n ₁ frames, n ₂ frames, n ₃ frames,..., N _n frames for each of M categories, (n ₁ + n ₂ + n ₃ +. .. n _n ) feature quantities are stored in the reference data storage unit 115 as reference data.

Assume that numbers are sequentially assigned to the (n ₁ + n ₂ + n ₃ +... N _n ) feature quantities. That is, numbers from _{1 to} (n ₁ + n ₂ + n ₃ +... N _n ) are assigned to each feature quantity. In step S154, as the initial setting, the first feature quantity of the feature quantity constituting the reference data is set as the reference data to be processed.

  In step S155, the distance calculation unit 116 calculates the distance using the feature amount supplied from the video feature amount extraction unit 114 and the reference data Ri. That is, the distance related to the similarity between the frame that has been processed by the video feature quantity extraction unit 114 and the frame from which the reference data Ri has been extracted is calculated. Here, the description is continued assuming that the shorter the distance, the more similar.

  In step S <b> 155, the distance calculated by the distance calculation unit 116 is supplied to the minimum distance holding unit 117. In step S156, the minimum distance holding unit 117 compares the supplied distance with the held distance, and determines whether the supplied distance is shorter. The minimum distance holding unit 117 holds a distance for each reference data Ri. The distance to be held is the shortest distance in the process.

  For example, when the reference data Ri is “1” (when the reference data R1 is a processing target), the distance associated with the reference data R1 is compared with the supplied distance, and the shorter distance is calculated. Is retained. Therefore, in step S156, it is determined whether or not the supplied distance is shorter than the held distance. If it is determined that the supplied distance is shorter, the process proceeds to step S157.

  In step S157, the distance determined to be short is associated with the reference data Ri that is to be processed at that time. That is, in this case, the distance associated with the reference data Ri at that time is replaced with a new distance. After the replacement is executed, the process proceeds to step S158.

  On the other hand, if it is determined in step S156 that the supplied distance is longer than the held distance, the process in step S157 is skipped and the process proceeds to step S158. That is, at that time, the state in which the distance associated with the reference data Ri is directly associated is maintained.

  In step S158, it is determined whether there is next reference data Ri. For example, when the reference data R1 is a processing target, it is determined whether or not there is reference data R2. If it is determined in step S158 that there is the next reference data Ri, the process proceeds to step S159.

  In step S159, the next reference data Ri is set as the new reference data Ri to be processed. And the process after step S155 is repeated with respect to the reference data Ri newly made into the process target.

  As described above, by repeating the processes of steps S155 to S159, the distance between all the reference data Ri stored in the reference data storage unit 115 and the feature amount extracted from one frame is calculated. In other words, the distances between the plurality of frames that are the basis of the reference data Ri and one frame in the video stream to be processed are calculated, and only the minimum distance is held.

  On the other hand, if it is determined in step S158 that there is no next reference data Ri, the process returns to step S152, and the next frame is set as a processing target. In this way, by repeating the processing of steps S152 to S159, the distance between the predetermined number of frames extracted from the video stream to be processed and the frame from which the reference data Ri is extracted is Each is calculated and only the minimum distance information is retained.

  An identification parameter is generated from a feature vector composed of such minimum distances, and the description thereof will be described later. The processing so far will be described again with reference to FIG.

  Referring to FIG. 7, category 1, category 2,..., Category N are set. These categories are categorized in detail. For example, Category 1 is “Sports News”, Category 2 is “Sports Broadcast”, Category 3 is “Sports Variety”, etc. It is a detailed category that indicates whether or not there is.

Category 1 includes a plurality of frames. N ₁ frames are extracted from the plurality of frames. This processing corresponds to the processing in steps S101 and S102 described with reference to the flowchart of FIG. Similarly, category 2 includes a plurality of frames, and n ₂ frames are extracted from the plurality of frames. Similarly, the category n includes a plurality of frames, and n _n frames are extracted from the plurality of frames.

  Each frame extracted from each category 1 to n is a reference frame. A feature amount is extracted from this reference frame. This process corresponds to step S103 described above, and is a process in which a feature amount is extracted from a reference frame and the feature amount is used as reference data.

Therefore, from the category 1, since n ₁ piece of reference frame is extracted, n ₁ pieces of reference data are generated. Similarly, from the category 2, since n ₂ sheets of reference frame is extracted, n ₂ pieces of reference data are generated. Similarly, from the category n, since n _n pieces of reference frame is extracted, n _n pieces of reference data are generated. Therefore, (n ₁ + n ₂ +... + N _n ) reference data are finally generated.

  In this way, reference data (in FIG. 7, described as a reference frame, and a quadrilateral resembling a frame is generated) is generated and stored in the reference data storage unit 115. Each reference data is assigned a number. Here, as shown in FIG. 7, the reference data extracted from category 1 is referred to as reference data R1, reference data R2, and reference data R3. Numbers are assigned to other reference data, which are not shown in FIG.

  In such a state, the video stream V1 is acquired. The range of the video stream V1 is limited. For example, as described above, sampling is performed only for the first 10 minutes of the program. As a result, M frames are extracted. A feature amount is extracted from each of these M frames. This process corresponds to steps S151 to S153 in the flowchart of FIG.

  In FIG. 7, feature amounts extracted from M frames are described as feature amount M1, feature amount M2, feature amount M3,..., Feature amount Mm.

First, the feature amount M1 and the reference data R1 are used to calculate the distance D1. Similarly, the feature amount M1 and the reference data R2 are used to calculate the distance D2. Similarly, the distance D3 is calculated using the feature amount M1 and the reference data R3. In this way, the distance between one feature amount M1 and all reference data Ri is calculated. Therefore, at this point, a (n ₁ + n ₂ +... + N _n ) -dimensional feature quantity vector is generated. This process corresponds to the processes of steps S154 to S159.

  Thus, when the distance between one feature value M and all the reference data Ri is obtained, the distance between the next feature value M and all the reference data Ri is obtained. After the feature value M1, the feature value M2 is processed, and the distance from the reference data Ri is calculated.

  In step S156, the distance D1 between the feature quantity M1 and the reference data R1 is compared with the distance D1 'between the feature quantity M2 and the reference data R1. As a result, when it is determined that the distance D1 'is shorter than the distance D1, the distance D1 associated with the reference data R1 at that time is replaced with the distance D1'. Further, when it is determined that the distance D1 ′ is longer than the distance D1, the state where the distance D1 associated with the reference data R1 at that time is directly associated with the reference data R1 is maintained. . In this way, each reference data Ri is associated with the shortest distance value among the feature amounts M1 to Mm.

Therefore, one (n ₁ + n ₂ +... + N _n ) -dimensional feature vector is finally generated from one video stream. Such a feature vector is data representing the feature of the video stream V1 to be processed.

  By generating such feature vectors for each of a plurality of video streams, the minimum distance holding unit 117 holds a plurality of feature vectors generated from a plurality of video streams. In other words, one or a plurality of feature vectors are held for each of a plurality of categories. Here, the reason why the number is one or more is that one feature vector may be generated in one category or a plurality of feature vectors may be generated in one category. .

  When the plurality of feature vectors generated in this way are held in the minimum distance holding unit 117 (FIG. 4), the learning algorithm processing unit 118 generates an identification parameter using the feature vectors. .

  The learning algorithm processing unit 118 generates an identification parameter based on a predetermined algorithm and using the feature vector held in the minimum distance holding unit 117. As the predetermined algorithm, for example, an algorithm such as a steepest descent method, a support vector machine, or back propagation can be used. The identification parameter calculated based on these algorithms is held in the identification parameter holding unit 119 as a parameter for identifying the category.

  The learning device 100 performs such model learning processing and generates, as a result of learning, an identification parameter used for identifying a program classification in the identification unit 17, that is, for example, a straight line or a curve for dividing a feature space. The identification parameters are supplied to the identification unit 17 and set. As described above, the setting is performed via a network or a recording medium or directly.

[Generation of classification information]
As described above, the identification parameter is generated and held in the identification unit 17 of the recording / reproducing apparatus 1, whereby the recording / reproducing apparatus 1 can identify the category of the program. With reference to the flowchart of FIG. 8, the identification process in which the recording / reproducing apparatus 1 (FIG. 1) classifies programs will be described.

  In step S201, a video stream is acquired. This video stream may be a video stream whose input is controlled by the input control unit 12 or a video stream stored in the storage unit 20. In step S202, a frame is extracted from the acquired video stream. When all the frames generated from the input video stream are to be processed, it is possible to omit the process of extracting the frames. The recording / reproducing apparatus 1 shown in FIG. 1 does not show a portion for extracting a frame (frame extraction unit).

  However, when a predetermined frame is extracted and processed, for example, a frame extraction unit is provided between the decoder 13 and the video feature amount extraction unit 15, and the frame extraction unit extracts the frame. It is considered as a configuration. Although not shown, the description will be continued here assuming that a frame is extracted. Further, here, it is assumed that the video feature amount extraction unit 15 selects a frame supplied from the decoder 13 to extract a frame.

  The frame extraction is performed in the same manner as the processing performed by the frame extraction unit 113 (FIG. 4) of the learning device 100, in other words, the processing performed in step S102 (FIG. 5) and step S152 (FIG. 6). Is called. That is, a frame for the first 10 minutes of a program is a processing target, or a frame extracted at a predetermined time interval is a processing target.

  When a frame is extracted in step S202, the frame is set as a processing target, and the process proceeds to step S203. In step S203, the video feature amount extraction unit 15 extracts a feature amount from the frame to be processed. This process is performed in the same manner as the video feature amount extraction unit 114 of the learning device 100. That is, video feature amounts such as a color histogram, a color moment, a difference image, and a reduced image are extracted from the frame.

  In step S204, the feature vector generation unit 16 generates a feature vector. The feature vector generation unit 16 selects, from the feature amounts supplied from the video feature amount extraction unit 15, a predetermined feature amount used for identifying the classification of the program to which the chapter information is attached in the identification unit 17. Then, a vector (feature vector) having the selected feature amount as an element is generated. The feature vector generation unit 16 supplies the generated feature vector to the identification unit 17.

  In step S205, the identification unit 17 identifies a category. The identification unit 17 identifies the category to which the program of the input video stream belongs using the feature vector supplied from the feature vector generation unit 16 and the retained identification pattern. For example, the discriminating unit 17 includes a discriminator such as a linear discriminator, a nonlinear discriminator, or a neural network, and each element constituting the feature vector is generated based on a straight line generated based on the discrimination parameter set by the learning device 100, Arranged in a predetermined feature space divided by a curve or the like, and identifies the category of the program based on the divided area of the feature space to which the distribution of the arranged elements belongs.

  This identified category is a category finely classified by learning in the learning device 100. And since it identifies using the identification parameter produced | generated based on the category classified so finely, the identification result can also be made into the result in a fine category.

  In step S206, the category as the identification result is output to the chapter information detection unit 18 as classification information.

  In this way, the chapter information detection unit 18 is supplied with the classification information related to the finely classified category to which the program belongs, so that detailed chapter information as described below can be detected. That is, it is possible to detect chapter information most suitable for a program for each program.

[Detection of chapter information]
The detection of chapter information will be described with reference to FIGS. A case will be described in which chapter information is detected from a program broadcasted by broadcasting a soccer game as shown in FIG. The upper part of FIG. 9 shows a video that constitutes a program that broadcasts a soccer game, and the lower part of FIG. 9 shows the volume of audio that is output simultaneously with the video.

  In this case, the audio feature amount extraction unit 14 converts the volume, frequency spectrum, left and right channel correlation values, and the like from the audio data of the sound whose volume varies as shown in FIG. And supplied to the feature vector generation unit 16 and the chapter information detection unit 18.

  The video feature quantity extraction unit 15 extracts a color histogram, a color moment, a difference image, a reduced image, and the like from the video data of the video shown in FIG. 9 supplied from the decoder 13 as a video feature quantity, and a feature vector generation unit 16 and the chapter information detection unit 18.

  In the present example, the feature vector generation unit 16 uses the color histogram supplied from the video feature amount extraction unit 15 among the feature amounts supplied from the audio feature amount extraction unit 14 or the video feature amount extraction unit 15. The feature vector is generated and supplied to the identification unit 17.

  From the video shown in FIG. 9, as described above with reference to FIG. 2, for example, a feature vector including many color histograms having a high green frequency is generated and supplied to the identification unit 17. When the feature vector including many color histograms having a high green frequency is supplied from the feature vector generation unit 16, the identification unit 17 determines that the program classification is “soccer program” from the feature vector as described above. Is identified so that it can be identified (identification parameters have been acquired), so that the classification of the program is identified as “soccer program”, and as a result of the identification, the classification is “soccer program” The classification information indicating this is supplied to the chapter information detection unit 18.

  A program that broadcasts a soccer game is a lively scene such as a kickoff, a goal chance, or a goal (in the example of FIG. 9, an upward arrow is attached in the figure) The image F42, F51, F53) has a feature that the volume is increased.

  That is, since this exciting scene is a point that the user is interested in, it is desirable that this scene is reproduced in the digest reproduction. Therefore, when the classification information indicating that the classification of the program is “soccer program” is supplied from the identification unit 17, the chapter information detection unit 18 is supplied from the audio feature amount extraction unit 14 and the video feature amount extraction unit 15. A color histogram, a difference image, and a volume are selected from the feature amount, and a position (for example, a frame) (hereinafter referred to as a cut point) where there is no video continuity is detected using them, and a cut point is detected. A chapter breakpoint is determined based on the result and the change in volume, and a calculation is performed with the degree of excitement based on the volume as the chapter score. The chapter information detection unit 18 supplies the chapter information detected as a result of the calculation to the holding unit 19 for holding.

  As described above, when the program classification is identified as “soccer program”, when chapter information based on the excitement is detected, in digest playback, at the time of kick-off, when there is a goal chance, or there is a goal. The images F42, F51, F53, etc. of the scene that is rising when the video is played are digest-reproduced.

  Next, a case where chapter information is detected from a program reporting an incident or event as shown in FIG. 10 will be described. The audio feature amount extraction unit 14 extracts volume, frequency spectrum, left and right channel correlation values, and the like from the audio data supplied from the decoder 13 as audio feature amounts, and outputs them to the feature vector generation unit 16 and the chapter information detection unit 18. Supply.

  The video feature amount extraction unit 15 extracts a color histogram, a color moment, a difference image, a reduced image, and the like as video feature amounts from the video data of the video shown in FIG. 10 supplied from the decoder 13, and a feature vector generation unit 16 and the chapter information detection unit 18.

  In the present example, the feature vector generation unit 16 uses the color histogram supplied from the video feature amount extraction unit 15 among the feature amounts supplied from the audio feature amount extraction unit 14 or the video feature amount extraction unit 15. The feature vector is generated and supplied to the identification unit 17.

  From the video shown in FIG. 10, as described above with reference to FIG. 3, a feature vector including a large number of color histograms with high frequency of specific colors of people and studios is generated and supplied to the identification unit 17. . When the feature vector including a large number of color histograms having a high frequency of colors peculiar to people and studios is supplied from the feature vector generation unit 16, the identification unit 17 obtains the program from the feature vector as described above. Since learning is performed so that the classification is “news program”, the classification of the program is identified as “news program”, and the classification result indicates that the classification is “news program”. The classification information is supplied to the chapter information detection unit 18.

  In a program reporting an incident or an event, an announcer's video explaining the content of the report and a video corresponding to the event are sequentially switched (in the example of FIG. 10, video F61, F63 with an upward arrow in the figure). , F71, F72).

  Since the viewer can grasp the outline of the news by viewing the video of the announcer explaining the contents of the report, it is desirable that this scene is reproduced in the digest reproduction. Therefore, when the classification information indicating that the classification of the program is “news program” is supplied from the identification unit 17, the chapter information detection unit 18 is supplied from the audio feature amount extraction unit 14 and the video feature amount extraction unit 15. A color histogram and a difference image are selected from the feature values, and the cut point is detected using them, and the position where the announcer's video explaining the content of the report from the similarity of the color histogram and the video corresponding to the report are switched. Is detected as a chapter break point, and an operation is performed to give a high score to the announcer's video. The chapter information detection unit 18 supplies the chapter information detected as a result of the calculation to the holding unit 19 for holding.

  In this way, when the program is identified as a “news program”, when chapter information based on the similarity between the cut point and the color histogram is detected, the video F61 of the announcer explaining the contents of the report in digest playback. , F62, and F71 are digest-reproduced.

  Next, a case where chapter information is detected from a program introducing a soccer game result (so-called digest) as shown in FIG. 11 will be described. The audio feature amount extraction unit 14 extracts volume, frequency spectrum, left and right channel correlation values, and the like from the audio data supplied from the decoder 13 as audio feature amounts, and outputs them to the feature vector generation unit 16 and the chapter information detection unit 18. Supply.

  The video feature amount extraction unit 15 extracts a color histogram, a color moment, a difference image, a reduced image, and the like from the video data of the video shown in FIG. 11 supplied from the decoder 13 as a video feature amount, and a feature vector generation unit 16 and the chapter information detection unit 18.

  In the present example, the feature vector generation unit 16 uses the color histogram supplied from the video feature amount extraction unit 15 among the feature amounts supplied from the audio feature amount extraction unit 14 or the video feature amount extraction unit 15. The feature vector is generated and supplied to the identification unit 17.

  From the video shown in FIG. 11, a color histogram with a high frequency of colors specific to the person and the studio and a color histogram with a high frequency of green are mixedly generated and supplied to the identification unit 17. As described above, when the feature vector in which the color histogram having a high frequency frequency specific to the person and the studio and the color histogram having a high green frequency are mixed is supplied from the feature vector generation unit 16, the identification unit 17 From the feature vector, learning is performed so that the classification of the program can be identified as “soccer digest program”. Therefore, the classification of the program is identified as “soccer digest program”. Supplies the chapter information detecting unit 18 with classification information indicating that “is a“ soccer digest program ”.

  When classification information indicating that the classification of the program is “soccer digest program” is supplied from the identification unit 17, the chapter information detection unit 18, as in the example of FIG. A color histogram and a difference image are selected from the feature amounts supplied from the video feature amount extraction unit 15, and using them, the chapter breakpoints based on the similarity between the cut points and the color histograms and the video during the match are high. Perform an operation that gives a score. The chapter information detection unit 18 supplies the chapter information detected as a result of the calculation to the holding unit 19 for holding.

  In the program introducing soccer game results (so-called digest), the video of the announcer explaining the content of the game results and the video of the game are sequentially switched (in the example of FIG. 11, the video with an upward arrow in the figure) F81, F82, F91, and F92).

  In a program introducing soccer game results, the video of the game is more important for the viewer than the video of the announcer. Therefore, it is desirable that the video of the game is preferentially reproduced in the digest reproduction.

  In this way, when the program is identified as a “soccer digest program” and the chapter information based on the similarity between the cut points and the color histogram is detected, it is optimal for the program introducing the soccer game result. Digest playback can be performed. In the case of the example of FIG. 11, the game scene videos F83, F92, etc. are digest-reproduced.

  For example, for the program shown in FIG. 11, if a climax point is detected as chapter information as in the case of the example of FIG. 9 according to the EPG classification, an appropriate video cannot be reproduced in digest reproduction.

  As described above, program classification suitable for detection of chapter information used in digest reproduction is performed, and chapter information is detected based on the classification.

  In the above description, the chapter information detection used for digest playback of a broadcast program has been described as an example. However, the same can be applied to chapter information detection for digest playback of other content. For example, the present invention can also be applied to personal contents taken with a camcorder. Although learning for personal content classification is required, the learning device 100 is built in the recording / reproducing apparatus 1 shown in FIG. 1 so that the learning can be performed easily.

  In the above description, the case of identifying a program category suitable for detecting chapter information used in digest playback has been described as an example. However, a category suitable for other processing may be identified.

  In the above description, the timing at which the classification identification process is executed is not mentioned, but it can be performed simultaneously with recording a program. That is, the input control unit 12 supplies AV data to the holding unit 20 for storage (that is, recording) and supplies the AV data to the decoder 13.

  Based on the AV data supplied from the input control unit 12, the decoder 13 to the identification unit 17 execute the classification identification process as described above. At this time, the chapter information detection unit 18 does not operate, and the feature amounts supplied from the audio feature amount extraction unit 14 and the video feature amount extraction unit 15 are held in, for example, the chapter information detection unit 18. It can also be discarded.

  In the chapter information detection, after the recording of the program is completed and the classification of the program is identified, the AV data held in the holding unit 20 is read out by the decoder 13, and the audio feature quantity extraction unit 14 and the video feature quantity extraction are performed. Each feature amount is extracted by the unit 15, and the chapter information is detected by the chapter information detection unit 18 by selecting a feature amount according to the classification of the previously identified program from the extracted feature amount.

  Further, the feature amount of the feature vector necessary for classification and identification can be extracted over the entire program, or can be extracted from, for example, a predetermined time (for example, 10 minutes) portion at the beginning of the program. When extracting feature values over the entire program, classification is performed after the recording of the program is completed as described above. When extracting feature values from a part of the program, an audio feature value extracting unit 14 and the video feature quantity extraction unit 15 and the chapter information detection unit 18 are provided with a buffer, and if the feature quantity from when the feature vector is generated until the classification is identified is buffered, the classification is identified. After that, chapter information detection can be started immediately.

  In the above description, the case where the program category is “news program” or “soccer program” has been described as an example. However, the program category can be classified into other categories such as “music program”.

The classification is not limited to what corresponds to a so-called genre, and may be other types. There are the following types of programs that broadcast songs and performances, but if they can be identified as classifications, chapter information detection in digest playback can be performed more appropriately.
・ The type of actual song and performance time is longer than the conversation between the moderator and the performer ・ The type of conversation between the moderator and the performer is long ・ The recording of the hall etc., and the cheering and applause of the audience Type that contains

  In the above description, EPG data is not used for classification identification. However, the EPG information held in the holding unit 22 can be further used to improve the classification identification accuracy of the program.

  FIG. 12 is a block diagram illustrating a hardware configuration example of a computer that executes the above-described series of processing by a program.

  In a computer, a central processing unit (CPU) 201, a read only memory (ROM) 202, and a random access memory (RAM) 203 are connected to each other by a bus 204.

  An input / output interface 205 is further connected to the bus 204. The input / output interface 205 includes an input unit 206 composed of a keyboard, mouse, microphone, etc., an output unit 207 composed of a display, a speaker, etc., a storage unit 208 composed of a hard disk or nonvolatile memory, and a communication unit 209 composed of a network interface. A drive 210 for driving a removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is connected.

  In the computer configured as described above, the CPU 201 loads, for example, the program stored in the storage unit 208 to the RAM 203 via the input / output interface 205 and the bus 204 and executes the program, and the series described above. Is performed.

  The program executed by the computer (CPU 201) is, for example, a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.), a magneto-optical disk, or a semiconductor. The program is recorded on a removable medium 211 that is a package medium composed of a memory or the like, or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

  The program can be installed in the storage unit 208 via the input / output interface 205 by attaching the removable medium 211 to the drive 210. The program can be received by the communication unit 209 via a wired or wireless transmission medium and installed in the storage unit 208. In addition, the program can be installed in advance in the ROM 202 or the storage unit 208.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

1 recording / reproducing apparatus, 11 data separation unit, 12 input control unit, 13 decoder,
14 audio feature quantity extraction units, 15 video feature quantity extraction units, 16 feature vector generation units, 17 identification units, 18 chapter information detection units, 19 holding units, 20 holding units, 21 playback units, 22 holding units, 41 control units, 100 learning unit, 111 input control unit, 112 decoder, 113 frame extraction unit, 114 video feature amount extraction unit, 115 reference data storage unit, 116 distance calculation unit, 117 minimum distance holding unit, 118 learning algorithm processing unit, 119 identification parameter Holding unit, 120 drive, 121 communication unit

Claims (6)

A predetermined number of frames are extracted from the plurality of first contents, a feature quantity is extracted from each of the extracted frames, and a multi-dimensional vector composed of the extracted first feature quantities is stored. Storage means;
Extracting means for extracting a predetermined number of frames from the second content and extracting a second feature amount for each frame;
Of each of the plurality of first feature amounts constituting the multi-dimensional vector and a predetermined number of frames extracted from the second content, the second extracted from a frame to be processed. Calculating means for calculating a distance from the feature amount of
Vector generation means for holding only the minimum distance among the distances calculated for each of the second feature amounts by the calculation means, and generating a feature vector composed of the minimum distance;
An information processing apparatus comprising: parameter generation means that performs processing based on a predetermined algorithm using the feature vector generated by the generation means, and generates parameters for classifying content. The information processing apparatus according to claim 1, wherein the extraction unit extracts the second feature amount from a predetermined part of the second content. The information processing apparatus according to claim 1, wherein the algorithm is one of a steepest descent method, a support vector machine, and backpropagation. A predetermined number of frames are extracted from the plurality of first contents, a feature quantity is extracted from each of the extracted frames, and a multi-dimensional vector composed of the extracted first feature quantities is stored. Remember
Extract a predetermined number of frames from the second content, extract a second feature amount for each frame,
Of each of the plurality of first feature amounts constituting the multi-dimensional vector and a predetermined number of frames extracted from the second content, the second extracted from a frame to be processed. Calculate the distance from the feature amount of
Of the distances calculated for each of the second feature amounts, only the minimum distance is retained, and a feature vector composed of the minimum distance is generated ,
An information processing method including a step of generating a parameter for classifying content by performing processing based on a predetermined algorithm using the generated feature vector . A predetermined number of frames are extracted from the plurality of first contents, a feature quantity is extracted from each of the extracted frames, and a multi-dimensional vector composed of the extracted first feature quantities is stored. Remember
Extract a predetermined number of frames from the second content, extract a second feature amount for each frame,
Of each of the plurality of first feature amounts constituting the multi-dimensional vector and a predetermined number of frames extracted from the second content, the second extracted from a frame to be processed. Calculate the distance from the feature amount of
Of the distances calculated for each of the second feature amounts, only the minimum distance is retained, and a feature vector composed of the minimum distance is generated ,
A computer-readable program for executing a process including a step of generating a parameter for classifying content by performing a process based on a predetermined algorithm using the generated feature vector . A recording medium on which the program according to claim 5 is recorded.

Images