JP2004260847A - Multimedia data processing apparatus, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a data processing apparatus for deciding and outputting a content category (such as drama or variety) of inputted multimedia data. <P>SOLUTION: While paying attention to video in multimedia data, the time in which a scene is switched, is successively stored. When the interval is a multiple of 15sec in several times of scene switching in the past, such a time block is decided to be a commercial. In a device connected to the post-stage of this multimedia data processing apparatus, multimedia data processing in response to the program class can be performed. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

  The present invention relates to a multimedia data processing device and method for determining and outputting the contents of multimedia data, and a recording medium.

  Due to the spread of satellite broadcasting and cable television broadcasting, the number of video channels that can be viewed at home is steadily increasing. In addition to conventional video recording media such as video tapes and laser disks, DVDs that can record and reproduce video and audio information as digital data have also been commercialized, and the environment in which ordinary users can access large amounts of video is being enhanced. You can say.

  On the other hand, the provision of technology that allows efficient access to a target video from among such a large amount of video has remained unchanged. For example, a television viewer can select a video (program) to watch by referring to a program column published in a newspaper or the like, and a consumer who wants to purchase a video software package can use a title or a program described in the package. Select the desired video by referring to the performers, synopsis, etc.

  These are textual descriptions of video information in characters, but not all of the video is described, and it is very difficult to describe all of them in such a way. Needless to say.

  Also, the demands on which part of the video to view should naturally be different for each user. Creating a text description that can be handled by all users leads to an increase in additional information, and is mainly due to broadcast media. Such a source is limited in the amount of information that can be supplied, and is difficult to use. For this reason, TV viewers who want to watch only a specific program in a news program must keep on the TV and wait for a topic to be heard, or watch news already seen on another program. It has been repetitively seen, and it has been a significant waste of time.

  In such a poor video access environment, even if a situation in which users can access a large amount of video information is realized in the near future, it cannot be said that the amount of information can be enjoyed smoothly. There is also concern that more users will be confused.

  In connection with this, the present applicant has already applied for a moving image processing method capable of listing videos with a small number of icons (Japanese Patent Application Laid-Open No. 9-270006). According to this method, it is possible to estimate a scene configuration in a video and present a list of video contents in units of a scene. According to the above-described invention, when a video of a drama is processed, a scene including a dialogue or the like can be displayed as a single icon.

  However, if the genre of a program such as a drama, a movie, or news is different, the scene composition is different. For example, in the case of news, similar shots rarely reappear. Instead, newscaster shots appear repeatedly, but are often the beginning of news items, and applying the same display method as above for drama as described above is not good visibility .

Conversely, in the case of the above invention, if it is known that the input video is news, a list display separated for each news item is presented using the same method of finding shots that appear repeatedly as it is. It becomes possible.
JP-A-9-270006

  The present invention is characterized by having means for inputting multimedia data, means for analyzing the data and estimating a category, and means for outputting the estimated category.

  That is, an object of the present invention is to automatically determine the program type of multimedia data and to facilitate media analysis with reference to the obtained program type.

  The multimedia data processing apparatus of the present invention analyzes multimedia data including an input video, and estimates content type to which the multimedia data belongs, and content estimated by the content type estimating means. Means for outputting seeds.

  The content type estimating means for estimating the content type to which the multimedia data belongs from the history of the time interval at which the screen is switched in the video of the multimedia data including the input video, and the content type estimating means. And means for outputting the content type.

  Also, a holding means for holding multimedia data of a known content type in advance, and comparing the known multimedia data possessed by the holding means with the multimedia data including the input video, The content type estimating means for estimating the content type to which the data belongs, and means for outputting the content type estimated by the content type estimating means are provided.

  The holding means may include a dictionary in which the known multimedia data is a language, and the content type estimating means may extract character information in the input multimedia data, and determine a content type to which the character information belongs in the dictionary. , The content type of the input multimedia data.

  The multimedia data including the input video is analyzed, the content type to which the multimedia data belongs is estimated, and the estimated content type is output.

  Further, a recording medium that records a program for processing the multimedia data in a computer-executable manner, analyzes the multimedia data including the input video, estimates the content type to which the multimedia data belongs, This is a recording medium on which a program for outputting the estimated content type is recorded so as to be executable by a computer.

  As described above, the multimedia data processing apparatus of the present invention having the capabilities of video recognition, voice recognition, character recognition, text recognition, and dictionary management estimates the type of input multimedia data (program). Since the data can be output, a device connected downstream of the multimedia data processing device of the present invention can perform multimedia data processing according to the program type.

  The multimedia data processing device of the present invention has its output terminal connected to the multimedia analyzing device, and the multimedia data analyzing device at the subsequent stage is optimally adapted to the program type information output by the multimedia data processing device of the present invention. It is assumed that the analysis method is automatically selected.

  An embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram showing a configuration of a multimedia processing device of the present invention. FIG. 2 is a flowchart showing an outline of the flow of the present invention. That is, the present invention reads multimedia data (S21), estimates the content type of the data (S22), determines whether or not the content type can be estimated (S23), and outputs the content type when the estimation is successful. (S24). The details will be described below.

  (Embodiment 1) Multimedia data enters an input terminal 101 from a recording medium such as a DVD, a CD, or a hard disk, or an upper information processing device such as a personal computer. In the case of the multimedia processing apparatus of the present invention, the cut detection unit 102 constantly checks a transition (cut) between scenes of the input multimedia data during operation. When a scene change occurs, the time is recorded in the temporary storage unit 103.

  In the present invention, the fact that the commercial (CM) of the current broadcast is usually 15 seconds, 30 seconds, and 1 minute is used. The content type estimating unit 106 observes the times of several to several dozens of recent cuts in the past cuts. For example, when comparing the times at which certain two cuts occur, they are determined to be intervals that are just multiples of 15 seconds. If the event has occurred in the above, it is determined that the time section surrounded by the two has been CM, and the result is output from the output terminal 107.

  FIG. 3 is a flowchart showing this procedure. That is, the frame of the multimedia data input from the input terminal 101 is read (S31), and whether or not the cut is detected by the cut detection unit 102 (S32). The time of the latest one is observed (S33), and it is determined whether or not they occurred at intervals of a multiple of 15 seconds (S34). If they occurred at intervals of a multiple of 15 seconds, It is estimated that there is a CM between the cuts (S35).

  FIG. 4 is a diagram for explaining this technique. In FIG. 4, a short vertical line crossing the time axis indicates a cut. The numbers shown between the cuts indicate the gap time (seconds) between the cuts. Now, since the interval between the cut 201 and the cut 202 is just 15 seconds, it is estimated that the time section from the cut 201 to the cut 202 would have been a CM. The content type estimating unit 106 in FIG. 1 outputs the time of the cut 201, the time of the cut 202, and the program type “CM” estimated in the section to the output terminal 107 using such a method.

  (Embodiment 2) Next, a second embodiment of the multimedia processing apparatus will be described with reference to FIGS. The multimedia information input from the input terminal 101 (S51) is sent to the content type estimating unit 106, where the type of the program is already known, and is compared with the multimedia information stored in the temporary storage unit 103. You. At this time, the temporary storage unit 103 stores multimedia information appearing in the past or its compression information, and also stores program type information of the multimedia information.

  The content type estimating unit 106 compares the similarity between the multimedia information in the temporary storage unit 103 and the newly input multimedia information (S53). If the similarity can be determined as similar (S53), the content information is accumulated. The same program type as that stored in correspondence with the multimedia information is output from the output terminal 107 (S54).

  For example, if there is a drama program that starts on the same opening screen every week, and if a program type called “drama” has already been output for the last broadcast, the temporary storage unit 104 stores the video and audio itself of the program. And the program type "Drama" is recorded. When the broadcast for this week is input, if the screen is compared with the opening screen of the drama for the last week, they match, so that "this broadcast is a drama" is output at that time.

  In this case, as an example of the means for similarity determination performed by the content type estimating unit 106, in the case of a video, the similarity of the shape of the hue histogram of the entire screen is observed, or a strict difference is obtained for each pixel, and the sum thereof is constant. It may be based on being equal to or less than the value. For the similarity comparison method, a method introduced in Nagasaka et al., “Automatic Indexing Method and Object Search Method in Color Video Images” (Information Processing Society of Japan, Vol. 33, No. 4, 1992) is applied. It can be realized by doing.

  In addition, the similarity of the audio waveform may be checked instead of the video. If the name of the program currently being broadcast is transmitted as text data as additional data of the broadcast, the text is stored in the stored text. An inspection may be performed to determine whether or not they match. Further, the similarity comparison of these video, audio, and text may be used together. The present invention is not limited to such a similar comparison technique.

  (Embodiment 3) Next, a third embodiment of the present invention will be described with reference to the drawings. The operation of the apparatus will be described with reference to FIG. FIG. 6 is a schematic diagram showing dictionary data stored in the program type dictionary 105 in FIG. This diagram is for the purpose of making the description easier to understand, and is not always the data of the dictionary in such a form. The multimedia data input from the input terminal 101 is input to the dictionary translation unit 104.

  The dictionary word is a thesaurus expression of the language expressed in the program type dictionary 105. For example, suppose that there is an utterance of "the XXth extraordinary Diet is convened today ..." in the audio data in the multimedia. At this time, the dictionary translation unit 104 divides the sentence into phrases, and extracts only those that are significant as dictionary words. Among the words “the XXth”, “temporary”, “diet”, “today”, and “convocation”, the characteristic word “diet” is selected in this sentence.

  Next, in correspondence with the thesaurus in FIG. 6, it is determined that the program type genre expression closest to “diet” 302 in the dictionary is “politics” 301. Here, the determination result is sent to the content type determination unit 106, and the content type determination unit 106 outputs “politics” from the output terminal 107 as a result.

  At this time, a method of waiting for a plurality of sentences to be input and searching for a program type genre expression in which characteristic words extracted from each sentence are close as a whole may be used. Further, the multimedia data used for comparison with the dictionary is not limited to the voice of the above example, but may be used if text information such as teletext is added. You may recognize the subtitle to be imposed and use that word.

  FIG. 7 is for explaining a case where the same dictionary comparison as described above is applied by another method. Personal names may be registered in the program type dictionary 105.

  For example, if there is a place in the program where the talents call each other, the cast may be estimated from the audio data, or the subtitles superimposed on the screen with the appearance of the talent may be recognized. In the case of a movie or the like, the character of the end roll that flows after the finale may be recognized. When information such as performers is obtained as text as additional information of a program, it may be used as it is. FIG. 7 shows an image of dictionary data in which, when the performer's name is obtained in this manner, a program type in which the person is likely to appear is recorded. For example, talent names that often appear in variety programs are arranged near the program type "variety" 401 (strong links are provided). Therefore, when many of the talents appearing are arranged near “variety” 401, the program is determined to be “variety”.

  However, when the above-mentioned talent is performed, but there are more performers belonging to the category "singer" than that, it is determined to be a "song program". This technique is not limited to real people. For example, if a dictionary is designed such that personal names such as "Iemitsu", "Tadasaku Ooka" and "Nishikaku" are placed near "History" and "Drama", multimedia data of such a genre will be correctly judged. It is expected to be.

  Further, the dictionary may store place names. When the dictionary translation unit 104 has a function of determining a shooting location from a video, words such as “Aizu Bandaisan”, “Oirase” and “Kirishima” are near “travel” and “travel”, for example. Should be placed at As a means for determining the shooting location, for example, a similarity comparison with a database (not shown) in which a characteristic scenery is recorded may be made, or the location may be extracted from a character telop or voice. Further, when information such as a photographing location is broadcast as text additional data, it may be used as it is.

  (Embodiment 4) Another program type determination method will be described. For example, in the case of a news program, a bust shot of one or two casters is often the head of a news item as indicated by 501 in FIG. 8 (A and B indicate caster shots. X indicates either A or B). X), each X is different. same as below). Therefore, the multimedia data is determined to be "news" for "a shot with a person in it", "1 to 3 patterns are present", "each appears twice or more", "appears with a maximum of 1 minute or more". May go.

  The determination of a news program has also been proposed in Sakauchi “Image and Multimedia Database” (Television Society Journal, Vol. 46, No. 11, pp. 1474) and the like. However, the method of the present invention is different in that a shot appearance pattern is used. By combining a plurality of pieces of information in this way, more accurate program type determination is realized.

  In addition, a program type determination method utilizing dialog scene detection used in Japanese Patent Application Laid-Open No. 9-270006, filed by the present applicant, is also conceivable. Each of 502, 503, and 504 in FIG. 8 has a dialogue scene (a section in which alphabets are repeated). However, a long dialogue scene as long as 5 minutes or more, such as 502, can be determined to be a "variety" program. This is because variety programs are often shot with three to four fixed cameras.

  In addition, it is possible to determine that an interactive scene having the same screen configuration continues over the entire program, such as 503, is a “talk” program. In a variety program, since the picture changes when the corner changes, the same scene of the dialogue scene is rarely reproduced, but in the case of a talk program, the same set is often performed throughout the program. Furthermore, when the dialogue scene of the same pattern is not reproduced as in 504, but the dialogue scenes of some patterns appear intermittently, it can be determined that it is “drama”. These can be expected to be more accurate when combined with human detection.

  In addition, it can be determined that a program in which many shots of the entire landscape are taken and the camera work is slow is a “journey”.

  Furthermore, even when the program type is determined to be "news" in the above procedure, if the content type estimating unit 106 has a music recognition function, a more accurate determination can be made. For example, in the case of a program having the structure of “news”, if the ratio of time during which music and sound effects are used is larger than the set ratio, it can be estimated that the program is “documentary”.

  The "sports" program can be estimated by the following method. For example, a program in which a similar angle pattern appears in the entire program, panning (a method of moving the camera left and right) is frequently used, and a scene in which a plurality of people appear in a small number of times appears. It can be estimated that it is "sports".

  In addition, the content type estimating unit 106 analyzes the audio pattern, and can estimate "sports" together with the phenomenon that cheers appear intermittently from the beginning to the end of the program.

  Furthermore, if subtitles that change with a cycle longer than 30 seconds frequently appear at a fixed position in one corner of the screen, it is estimated as a game score, and the program is estimated to be "sports". Can be.

  In the present invention, the procedures shown in FIGS. 2, 3, and 5 can be realized by software in which a program executable by a computer is stored in an FD or a CD-ROM.

1 is a block diagram according to a multimedia data processing device of the present invention. FIG. 3 is a diagram showing a procedure of a processing method of the multimedia data processing device of the present invention. FIG. 3 is a diagram showing a procedure of a processing method of the multimedia data processing device of the present invention. FIG. 4 is a diagram illustrating a method for detecting a CM of the multimedia data processing device of the present invention. FIG. 3 is a diagram showing a procedure of a processing method of the multimedia data processing device of the present invention. FIG. 3 is a diagram showing dictionary data of the multimedia data processing device of the present invention. FIG. 3 is a diagram showing dictionary data of the multimedia data processing device of the present invention. 6 is a flowchart showing an arrangement of shots when determining a program type in the multimedia data processing device of the present invention.

Explanation of reference numerals

101 multimedia data input terminal 102 cut detection unit 103 temporary storage unit 104 dictionary translation unit 105 program type dictionary 106 content type estimation unit 107 multimedia data output terminal

Claims (4)

A first input unit for inputting first multimedia data including video,
A second input unit for inputting second multimedia data whose content type is known,
A similarity comparing unit that compares the similarity of the entire screen between one or more video frames of each of the first multimedia data and the second multimedia data;
If the similarity comparison unit determines that the similarity is equal to or greater than a predetermined value, the content to be estimated that the content type of the first multimedia data is the same as the content type of the second multimedia data Species estimation means;
A content type output unit that outputs the content type estimated by the content type estimation unit,
A multimedia data processing device comprising: The second multimedia data is dictionary data in a language,
The similarity comparison unit includes:
Means for extracting language information by caption recognition from a video signal in the first multimedia data;
Means for comparing and matching the extracted linguistic information with the dictionary data;
With
The content type estimating means estimates a content type to which the linguistic information belongs in the dictionary data as a content type of the first multimedia data.
The multimedia data processing device according to claim 1. Computer
First input means for inputting first multimedia data including video,
Second input means for inputting second multimedia data whose content type is known,
Similarity comparing means for comparing the similarity of the entire screen between one or more video frames of each of the first multimedia data and the second multimedia data;
If the similarity comparison unit determines that the similarity is equal to or greater than a predetermined value, the content to be estimated that the content type of the first multimedia data is the same as the content type of the second multimedia data Species estimation means,
Content type output means for outputting the content type estimated by the content type estimation means,
A computer-readable recording medium that records a multimedia data processing program for functioning as a computer. The second multimedia data is dictionary data in a language,
The similarity comparison means includes:
Means for extracting language information by caption recognition from a video signal in the first multimedia data;
Means for comparing and matching the extracted linguistic information with the dictionary data;
With
The content type estimating means estimates a content type to which the linguistic information belongs in the dictionary data as a content type of the first multimedia data.
A computer-readable recording medium on which a multimedia data processing program is recorded.

Images