JP2006120018A - Image data processor, image processing method, program and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform video retrieval having the degree of freedom by reflecting the intent of a person inputting information at the time of describing contents of image data. <P>SOLUTION: An image data editing/retrieving device 1 for editing and retrieving image data is provided with: a scene change detecting part 308 having a characteristic analyzing part 309 inside for analyzing the amount of image characteristic and the voice characteristic in an image and for detecting scene change on the basis of its analysis result; a message preparing part 301 for outputting a question item from a device by character and voice; and an inputting part 312 for receiving a contents description input from a user by character and voice in the form of answering the question item, wherein in addition, a question item preparing part 310 attaches a free description item as a question item. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to video data processing, and more particularly to video data processing for structuring a video that enables a desired video to be selectively viewed from a large amount of video.

In recent years, with the development of communication technology and the increase in capacity of hard disks, a large amount of video data is accumulated, and a system capable of selectively viewing a desired video from the video data has been developed. That is, a system has been proposed in which content description is performed on video information in order to reuse the captured video, and video search is performed based on the description information. In general, in such a system, in order to make a description, a method is often used in which a video is played, stopped when the contents are gathered, and the portion is described. A video that is usually taken can be seen once with the people involved in the shooting, and at that time the situation at the time of shooting is talked between the people involved in the shooting, or a third party who does not know the situation An explanation is given.
In such a system, in order to select desired video data, it is necessary to delimit the structure of the video data in a semantic base in advance and to describe information on this. In this way, in order to describe the contents in the video data, it is necessary to first rewind and play back the video, detect a scene that is semantically separated, and input the content that matches the video scene. Therefore, it takes at least several times as much time as the actual time of the video data, which requires a lot of labor.
For the above problem, the time and effort of content description is reduced by automatically detecting the switching of the scene that is the unit of content description, and even at the time of content description, to a third party who does not know the video content, By letting the character appearing on the monitor take the setting of telling the situation where the video was shot, while watching a video taken by the family, even while enjoying the conversation with the character appearing on the monitor, even for children, A video data editing / retrieval device capable of describing content has been proposed (see Patent Document 1).
JP 2003-178076 A ITU-T G729 AnnexB A silence compressoion scheme for G.729 optimized for terminals conforming to Recommendation V.70 Examination of speaker indexing in drama, Nishida et al., Acoustical Society of Japan, 1999 spring Individual differences in fundamental frequency pattern parameters, Fujisaki, Ohno et al., Acoustical Society of Japan, 1999 spring Latest Trends in Image Processing Algorithms, New Technology Communications Practical image processing learned in C, Ohm Human Detection in Unstable Brightness, Proceedings of the 1996 IEICE Society Conference Toshiaki Yokoyama “Program Generation / Interactive Editing System Based on TVML Description Language TVML” 3rd Intelligent Information Media Symposium, December 1997 Toshiaki Yokoyama, “Development of Machine Interface for TV Program Description Language TVML” Electronic Information Society, 1997

However, the above-described conventional technique has a problem that the contents of a question are generated using the feature amount of video data, and the contents can be described only in the form of an answer to the question, and the intention of the person who inputs information is not reflected. It was.
SUMMARY OF THE INVENTION It is an object of the present invention to solve such a problem and to allow a video search with a high degree of freedom by reflecting the intention of a person who inputs information when describing the contents of video data.

In order to solve the above problems, the invention according to claim 1 is a video data processing apparatus for adding video content information for each scene to video data recorded together with audio. Item creation means and input means for accepting a content description input from a user by text or voice corresponding to the question item, the question item creation means adding a description item input as a free description to the question item It is characterized by doing.
According to a second aspect of the present invention, in the video data processing apparatus according to the first aspect, the question item creating unit deletes a question item for which information on the question item has not been input from the input unit.
According to a third aspect of the present invention, there is provided the video data processing apparatus according to the first or second aspect, further comprising search means for searching for the portion in which the contents are described, and reproducing the video data from the scene or the scene portion searched by the search means. It was made to do.
According to a fourth aspect of the present invention, there is provided the video data processing apparatus according to the third aspect, wherein the search means enables a search for a part in which the content of the video data is described in another video data processing apparatus on the network. Means are provided.
According to a fifth aspect of the present invention, there is provided a video data processing method for adding video content information for each scene to video data recorded together with sound. And an input step of accepting a content description input from a user by text or voice, wherein the question item creating step adds a description item input as a free description to the question item.
According to a sixth aspect of the present invention, in the video data processing method according to the fifth aspect, the question item creating step deletes a question item for which information on the question item has not been input from the input unit.
According to a seventh aspect of the present invention, in the video data processing method according to the fifth or sixth aspect, a search step for searching for the content-described portion is provided, and the video data is reproduced from the searched scene or scene portion. It is characterized by that.
Further, according to an eighth aspect of the present invention, in the video data processing method according to the seventh aspect, in the search step, communication for enabling a search of a part in which the content of the video data is described in another video data processing device on the network. It is characterized by having a function.
The ninth aspect is a program for causing a computer to function as the video data processing apparatus according to the first, second, third, or fourth aspect.
A tenth aspect of the present invention is a computer-readable recording medium on which the video data processing program according to the ninth aspect is recorded.

As explained above, in addition to content description items by text and audio questions displayed on the display every time a video scene changes, it is possible to freely describe content according to the editor's intention, and freely Since the described items are added as question items, the description items according to the editor's intention can be presented when editing next time.
In addition, since description items that have not been input for the question items are deleted, only necessary content description items can be input efficiently.
In addition, it is possible to process and search a single video data, and to search and reproduce a desired video from information registered in a video data processing apparatus connected on a network.

Hereinafter, embodiments of the present invention will be described in detail. First, the outline of the present invention will be described. As mentioned above, the video that was usually shot is seen once with the person concerned with the shooting, and at that time, the situation at the time of shooting is talked between the persons concerned with the shooting, or the situation An explanation is given to a third party who does not know.
The video data processing apparatus of the present invention allows a character (implemented by TVML or the like) appearing on a monitor to have a setting of speaking to a third party who does not know this situation, thereby allowing a video shot by a family to be captured. When watching, even children can describe content while enjoying dialogue with characters appearing on the monitor. In this embodiment, the video data processing apparatus of the present invention does not edit or process the video data itself, but adds and searches information in association with the video data, so that the video data editing / search system (apparatus) ).
First, in the case of content description, the content description mode is selected by the video data editing / retrieval apparatus. Then, a message notifying that the content description is to be started is displayed on the voice or the display. Then, while the tape is reproduced, the video data is subjected to scene change analysis by a scene change detection device. When the change point is detected, the tape is stopped, and a question about information (5W1H, etc.) for describing contents corresponding to the change point information is made.
In other words, the character asks “When did you go?” In response to this question, “December 20th, 20XX” is answered to Mike. In addition, the character asks questions such as “Where did you go?”, “Who?”, “What did you do?” And so on. When a question about a content description item for a scene is made, the character may finish inputting the content description for the current scene, such as “It was fun. That question is asked.

On the other hand, if a response to end such as “End” is returned, the next scene is played back, the above-mentioned question about the content description for the video is repeatedly performed, and the described result is recorded. The If there is more content that you want to input, return a response such as “I have more” and the character will ask you “What?” And a list of input items will be displayed on the monitor. The user is instructed to answer by selecting from the displayed items. If you select “What” from the list, the character will ask you “What did you do?” In response to this, it is confirmed whether or not the content description can be finished again, and in addition to the question item, the editor can freely input a matter to be described regarding the scene.
When the content description is performed while interacting in this way and the video is reproduced to the end, an end message is output and the content description menu is terminated.
In the video data search device, when a user inputs a search request for data described in this way, the search unit searches the content description data for a match with the request. If found, the information is displayed.

Next, details of the present invention will be described with reference to the drawings. FIG. 1 shows an example of a system configuration of an interactive video data editing / retrieval apparatus for editing and retrieving video data according to the present invention. FIG. 2 shows a processing flow in the video data editing / retrieval apparatus. FIG. 3 shows the configuration of the video data editing / retrieval apparatus. FIG. 4 shows an example of a content description interface screen displayed on the display when editing video with the video data editing / retrieval apparatus. FIG. 5 shows a scene change detection processing block. FIG. 6 shows a flow of video change and question item generation. FIG. 7 shows the question items and flag bit patterns. FIG. 8 shows an example of the TVML creation block. FIG. 9 shows the configuration of the meta information file creation unit. FIG. 10 shows an example of the TVML script. FIGS. 11 to 13 show examples of MPEG-7 content description files.
First, the configuration of this system will be described with reference to FIG. This system includes a video data editing / retrieval device 1, a display device 2 for displaying visual information (image information) among video data and created TVML information, and among video data and TVML information. A speaker 3 for presenting audio-related information (audio information), and a disk drive device (hereinafter referred to as a disk drive device) for driving a disk as a recording medium for storing structure information and content description information created for video data 4) and an input device 5 for inputting description contents and search conditions by voice.
The video data editing / retrieval device 1 and the display device 2 are connected by a video signal cable 9, and the video data editing / retrieval device 1 and the speaker 3 are connected by an audio signal cable 6. The video data editing / retrieval device 1 and the disk device 4 are connected by, for example, a SCSI (Small Computer System Interface) specification data cable 7. The input device 5 is connected to the video data editing / retrieval device 1 by an audio signal cable 8.

The internal configuration of the video data editing / retrieval apparatus 1 is as shown in FIG. For example, a 19-inch CRT (cathode ray tube) is used as the display device 2. As the disk device 4, for example, a device that records and reproduces data using a magnetic disk as a recording medium is used. The magnetic disk stores one or more meta information files associated with the moving image file. The meta information file will be described later.
Here, the recording medium is not limited to a magnetic disk, and may be another recording medium. For example, it may be an optical disc such as a CD-R (writable CD), CD-RW (rewritable CD), DVD (digital video disc), or a magneto-optical disc. The disk device 4 may be built in the main body.
The audio signal cable 8 for connecting the input device 5 to the video data editing / retrieval device 1 is directly connected to the audio signal input / output unit 312 in FIG. 3 inside the video data editing / retrieval device 1. The audio input from 5 is supplied to the video data editing / retrieval apparatus 1 as it is.

FIG. 3 shows the internal configuration of the video data editing / retrieval apparatus 1. As shown in this figure, the video data editing / retrieval apparatus 1 automatically uses a CPU 300 that controls the operation of the entire apparatus, a video deck 320 that plays back a videotape, images, and audio feature quantities to automatically generate scenes. An A / D converter 311 for converting a sound signal input from the scene change detection unit 308, the video drive control unit 314, the video signal input / output unit 313, and the audio signal input / output unit 312 to detect a change point; A video data search unit 315 that searches for desired data from the video data, a message creation unit 301 that creates a message output program using an interactive interface, and a video structure / content description information based on scene change points. A meta information creation unit 302 that creates a meta information file as information, a memory 306, an OS, and an application A hard disk drive (HDD) 307 which programs are stored, and a display connection portion 317 which functions as an interface between the display device 2.
Further, the OS stored in the HDD 307 has a so-called GUI function so that the user can operate interactively (interactively) mainly by the screen display of the input device 5 and the display device 2. It has become. The video drive control unit 314 inputs and outputs control signals related to video drive. The units constituting the video data editing / retrieval apparatus are connected to each other by a system bus 316.

The operation of the video data editing / retrieval apparatus configured as described above will be described with reference to the processing flow in the video data editing / retrieval apparatus shown in FIG. First, video is played back at 201 in FIG. 2, and the tape is played back on the video deck 320 shown in FIG. The video image is analyzed for audio / image feature values at 203 in FIG. As shown in FIG. 3, the video signal and the audio signal are separated, and the audio signal input / output unit 312 and the video signal input / output unit 313 output the respective signals. Next, a method for analyzing the separated audio signal and video signal will be described.
The audio signal output from the audio signal input / output unit 312 is converted from an analog signal to a digital signal by the A / D converter 311. At this time, since the normal voice band is considered to be about 4 kHz, sampling is performed at 8 kHz. The digitized audio signal is input to the scene change detection unit 308.
Next, the flow of video change and question item generation will be described with reference to FIG. At 603, the audio signal is divided into frames of approximately several ms to 40 ms. Next, using the digital audio signal divided into frames, as the detection of the video change, the voiced / silent section detection, the voice section detection, and the switching of the speaker are determined.
When sound is heard after a certain period of silence, the sound power (0th-order autocorrelation function) is calculated at 604, and the sound / silence interval is detected at 605 based on the sound power. When the time T-t between the time T when the silent section is detected and the time t when the voiced section is detected is larger than the threshold τ, the voice section flag 640 is set to 1.
In addition, the VAD method (spectrum distortion, energy difference, low-frequency energy difference, zero-crossing number difference) described in “ITU-T G729 Annex B A silence compression scheme for G.729 optimized for terminals conforming to Recommendation V.70” Can be applied to discriminate a voiced section by discriminating according to a preset threshold. In the detection of the voice section, a method as shown in “Study indexing of speakers in drama, Nishida et al., Acoustical Society 1999 Spring” can be applied.

In step 608, the variance value of the first-order cepstrum is calculated from the digital voice divided in frame 603 in FIG. 6. In the case of speech, since the variance of the first-order coefficient of the cepstrum is larger than that of music and noise, it is possible to detect a speech section using a threshold set in advance in 609. If a speech segment is detected, the speech segment flag is set to 1 in 611.
Further, the switching of the speaker is determined for this voice section. As a speaker discrimination method, the method described in “Individual differences in parameters of fundamental frequency pattern, Fujisaki, Ohno et al., Acoustical Society 1999 Spring” can be applied. This method takes advantage of the fact that the fundamental frequency of speech, the phrase control mechanism, and the time constant value of the accent control mechanism, which are closely related to the physical and physiological characteristics of the model, have almost constant values for the same speaker. Thus, the speaker is discriminated. 612 to 615 are analyzes using this method.
First, at 612, the fundamental frequency is calculated. The fundamental frequency can be calculated by an autocorrelation method, a modified correlation method, or a cepstrum. The cepstrum is defined as the inverse Fourier transform of the logarithm of the short-time amplitude spectrum of the waveform, and is calculated by approximately separating the spectral envelope and the fine structure. Details are described in Tokai University Press, Digital Audio Processing.
Further, in 613, successive approximation is performed by an analysis-by-synthesis method based on the model of the fundamental frequency pattern generation process, and each of the base frequency, the size of the phrase command and the time of occurrence, the size of the accent command and the time of occurrence, and the accent of each phrase. Find the time constant of the control mechanism. Based on the magnitude of these values, at 614, speaker switching is determined. If it is determined that the speaker has changed, the speaker change flag is set to 1 in 615.

Next, a video signal analysis method will be described. The video signal input from the video signal input / output unit 313 in FIG. 3 is input to the scene change detection unit 308, and frame division is performed in 617 shown in FIG. First, a scene change is detected for the frame-divided image. In detecting a scene change, there is a method of detecting a scene change from a luminance average value. Each pixel on the display has a luminance value that changes with time according to scanning, and in 618, by calculating the average value of the luminance values in one frame (arithmetic average in a certain frame), within one frame An arithmetic average is calculated at coordinates in all pixels (x, y).
In the video, if the scene suddenly changes, this average value can change greatly. Therefore, the luminance average value for each frame is calculated, and if the change is equal to or greater than a predetermined threshold α in 639, the scene It is determined that there has been a change, and the scene change flag is set to 1 at 638.
Further, edge extraction is performed at 619 in order to extract an object existing area candidate. As edge extraction methods, zero crossing method (method using the property that the zero crossing point in the second derivative coincides with the position of the new step edge), mask operator (considering variance and average in the mask by Jacobus et al.) A method of creating a threshold image that has been made) can be applied. Details of the edge extraction method are described in “Latest Trends in Image Processing Algorithms, New Technology Communications”.

Furthermore, the presence of an object can be detected by calculating curvature and RGB color distribution for the edge extracted region and cutting out an image using changes in these values. Details are described in "Practical Image Processing Learned in C Language, Ohmsha". In addition, when determining the presence of a person, the method described in “Person Inspection in a Brightness Unstable Environment, Proceedings of the 1996 IEICE Society Conference” can be applied.
In 621, the gradient direction difference is obtained as the amount of change in the direction vector at each coordinate position for the edge extracted region. First, by comparing the two images and extracting an area where the difference in brightness is a certain value or more, and applying a first-order differentiation process to the changed area by the Sobel operator, the direction of the edge is determined from the horizontal component and the vertical component. Create a gradient vector as Then, gradient vectors having the same coordinate position are compared to detect a gradient direction difference as a difference in direction, and a gradient direction difference histogram is created at 624.
In this histogram, when a person is present, the component change (625 in FIG. 6) becomes large in the region occupied by the person. At this time, an evaluation value is obtained by determining a straight line parallel to the horizontal axis that minimizes the sum of the distances from each point on the histogram. The presence of a person is determined based on the magnitude of the evaluation value. If it is confirmed that a person exists, the person presence flag is set to 1 in 634.
Further, the presence determination of another person is performed. The difference between the current frame component change value β in 625 and the component change value γ calculated in the previous frame is determined based on the magnitude of a predetermined threshold value κ to determine whether the person is different. If it is, the different person presence flag 631 is set to 1.
The method described in “Latest Trends in Image Processing Algorithms, New Technology Communications” can be applied to the determination of the movement of an object. In 617 of FIG. 6, an inter-frame difference method (a dark area and a bright area before and after the target are obtained by subtracting the gray value of each pixel of the previous frame from the gray value of the pixel of the current frame, and the target moving direction and moving amount are estimated. Furthermore, in order to detect the contour of the moving object from the difference image, it can be determined by using the continuous difference image. Only the positive (or negative) component of the difference is used, and a logical sum is taken in successive frames. This is in order to eliminate the effect of the next object entering the area occupied by one object in the previous frame. In this way, candidate moving objects are detected.

Next, feature point correlation 629 is performed. There is a method based on feature points as to where a part of a screen corresponds to the next screen. There is a method by Moravec's Interest operator and Nagel et al. That uses Gaussian curvature as a feature point to extract spots. The Moravec operator calculates the density variance between adjacent pixels in a small area in the vertical, horizontal, and diagonal directions, and gives the operator's initial value with the minimum value, and then the maximum point of these values. Is a feature point.
In the method of Nagel et al., A feature point is obtained from the property of Gaussian curvature, considering a surface in the xyz space whose concentration is the height in the z-axis direction. When the feature point is obtained, the feature point corresponding to the feature point in the current screen is searched in the next screen. In the search, if the maximum moving speed of the moving object is known, the search range can be narrowed.
In the method using the relaxation method by Barnard et al., A corresponding feature point candidate list of the second screen is created in the characteristic vicinity of the first screen. The probability that each candidate element is associated is determined by a correlation value by weighting considering the case where there is no corresponding feature point. Since the vector of each feature point is considered to have a similar tendency locally, the probability of whether the velocity vector has the same tendency in the vicinity is calculated. In this way, the tendency of the velocity vector in the vicinity is propagated, and the conversion is stopped when the probability for one label exceeds a certain threshold for all the feature points.
In this way, feature points are associated with each other to detect a moving object. If the moving object is detected, the moving object flag is set to 1 in 636. When these flags are changed, conditions as shown in FIG. 7 are determined based on the flag bit pattern 632 to determine each question item.

Next, a table of questions and flag bit patterns shown in FIG. 7 will be described. When the flag bit pattern when the table is viewed for each column is prepared, the question shown in the question item is asked. For example, as a question item of “Who”, the condition for asking “Who” is that both the voice section detection flag and the scene change flag are 1, the case where the separate person presence flag is 1, and the person presence flag. There are a case where the scene change flag is 1 and a case where the speaker change flag is 1.
The condition for making the “WhatObject” question is that the voice segment detection flag and the scene change flag are both 1, the speaker switching flag is 1, and the person presence flag and the scene change flag are 1. In this case, both the scene change flag and the object detection flag are 1.
The condition for the question “WhatAction” is that the speaker switching flag is 1, the moving object detection flag is 1, the different person presence flag is 1, the voice section detection flag and the scene. A case where both the change flags are 1 and a case where both the person presence flag and the scene change flag are 1.
The conditions for making the “When” and “Where” questions are when the speaker switching flag is 1, when the sound flag is 1, and when the scene change flag is 1.
The conditions for the “Why” and “How” questions are when the speaker change flag is 1, when the object detection flag is 1, and when both the voice segment detection flag and the scene change flag are 1. And the person presence flag and the scene change flag are both 1.
Based on the audio and video analysis described above, a feature amount change is detected at 204 in FIG. The CPU 300 in FIG. 3 sends a control signal related to video playback to the video drive control unit 314, and stops video playback on the video deck 320 (205 in FIG. 2). Further, information related to the change scene, for example, information related to the video data structure such as the first frame and the last frame number of the video block, time, and the question are stored in the structure storage unit 502 in the meta information file creation unit 302 shown in FIG. In FIG. 2, 206).

Next, the TVML creation 207 of FIG. 2 is executed to make the above-mentioned question. TVML is a text-based language designed to describe one TV program, and a program script created in the TVML language is interpreted by a TVML player to create a TV program in real time. . The TVML player is software that runs on a personal computer or graphic workstation. When a TVML script and various data (moving images, audio, etc.) are given to this, a TV program can be created by the following functions of the TVML player.
The functions of TVML are CG casters that speak with a synthetic voice function against the background of studio shots in real-time full CG, camera work, text superimpose, title display by HTML description, video file playback, audio file playback Voice synthesis narration, etc.
For details, see http://www.strl.nhk.or.jp/TVML/Japanese/Jsitemap.html, Toshiaki Yokoyama, “Program Generation / Interactive Editing System Based on TVML Description Language TVML”, 3rd Intelligent Information Media Symposium, December 1997, and Toshiaki Yokoyama, “Development of Machine Interface for TV Program Description Language TVML,” Electronic Information Society, 1997.

The TVML creation unit 319 shown in the question item display of FIG. When the scene change point is detected from the video data by the above processing, the CPU 300 further activates the TVML creation unit 319 in the message creation unit 301, and the TVML program creation unit 500 in FIG. Information about the structure and question items are inquired to the scene change detection unit 308, and a TVML script for inquiring the question items with respect to the video is created based on the information. FIG. 10 shows an example of the TVML script.
The script created by the TVML program creating unit 500 displays the question items in 208 of FIG. 2, so that the audio signal is input / output unit 312 and the video signal is input / output from the TVML player 501 shown in FIG. Through the part 313, the content is reproduced from the speaker 3 and the display device 2, and a question is made to the content writer.
According to the output result by TVML, the content writer issues information input to the input device 5 shown in FIG. 3 at 209 in FIG. In order to recognize input information in 210 of FIG. 2, the audio signal acquired by the input device 5 shown in FIG. 3 is converted into a digital signal of a predetermined sampling frequency by the A / D converter 311. The digital signal output from the A / D converter 311 is supplied to the digital speech processing circuit 304 in the speech recognition processing unit 303 in FIG.
The digital speech processing circuit 304 converts the digital speech signal into vector data by processing such as band division and filtering, and supplies this vector speech signal to the speech recognition circuit 305. A speech recognition data storage ROM is connected to the speech recognition circuit 305, and the speech recognition data stored in the ROM is compared with the vector data supplied from the digital speech processing circuit, and based on predetermined conditions. When a match is detected, the text data stored corresponding to the vector data is read out.
For this comparison, for example, a hidden Markov model (HMM) is used. For example, a DSP (digital signal processor) is used for these processes. Such speech recognition technology is described in Sadaaki Furui, “Digital Speech Processing”.

The read text data is input to the TVML creation unit 319 in the message creation unit 301 shown in FIG. 3 in order to display the input information in 211 of FIG. 2, and the TVML shown in the display of input information in FIG. A TVML script for confirming and displaying the information input by the program creation unit 500 is created. The TVML script is read by the TVML player 501, and the audio signal is output from the speaker 3 and the display device 2 through the audio signal input / output unit 312 and the video signal through the video signal input / output unit 313.
Further, in 212 of FIG. 2, as the confirmation of the input information, whether or not the output information is correct is confirmed by the content writer. When the input device 5 confirms the input information from the descriptor, the input information is read out by the speech recognition circuit in the speech recognition processing unit 303 in FIG. Meta information file creation 213 is performed. If there is a problem with the input information, the display returns to the question item display at 208 in FIG. 2, and the process is performed again.
A case where there is no problem in the input information will be described. In this case, the process proceeds to the meta information file creation 213 in FIG. The text data read by the speech recognition circuit 305 in the speech recognition processing unit 303 in FIG. 3 is stored in the content storage unit 503 in FIG.

Next, the structure information (number of frames, time, etc.) for each group of frames based on the detected scene change point, meta information of the question items and input information is converted into a meta information file having a predetermined data structure and stored. The information file creation unit (shown in FIG. 9) will be described.
The structure information stored in the structure storage unit 502 in the meta information file creation unit 302 and the question information and the input information stored in the content storage unit 503 follow the data structure information stored in the meta information schema storage unit 504. The data is converted into a data structure, and information is stored in the meta information storage unit 506. The information stored in the meta information storage unit 506 is further sent to and stored in the disk device 4.
As a meta information schema, MPEG-7, which is currently standardized as a standard for showing the structure and content description information for video, is known. FIGS. 11 to 13 show description examples of data structure information according to MPEG-7.
In this way, when a change point is detected in the video, the TVML creation program is started, and is repeated until the execution of the TVML program and the reproduction of the creation tape of the meta information file are completed. The content description interface by these TVML programs is shown in FIG.
Further, when no information is input for the question item, the question item creation unit 310 deletes the question item. When the meta information file of the content description information by the question item is created, the free description item input of FIG. 2 is displayed (214), and the content description item is input. Similar to the processing of the input information by the question items, the input information recognition 216, the input information display 217, and the input information confirmation 218 are performed. If there is no problem with the input information, a meta information file creation 219 is performed. Further, the question item is added by the question item creation unit 310 with the free description item as a question item.
When the meta information file is created, confirmation of the tape end 220 in FIG. 2 is performed. The CPU 300 detects the presence or absence of a synchronization signal from the received video signal. When the synchronization signal is interrupted for a certain period of time, the video is considered to have ended, and a video stop signal is sent to the video drive control unit 314. Stop video playback. Then, as the processing of the description end message 221 in FIG. 2, the processing is ended by displaying on the TVML a message for ending this apparatus (222 in FIG. 2).
The system further provides a search interface for the video data that has been structured and described as described above. When the user inputs a search condition from the input device 5 in FIG. 3, the video data search unit 315 searches for video data corresponding to the request from the disk device 4, and if there is a match, for example by the display 2 or the speaker 3. Presented. Although the addition / deletion of the question items is performed at the time of creating the meta information file, it may be performed at the end of the tape 220 or at the start time 200 when editing the next video data.

Although one embodiment of the present invention has been described above, the system of the present invention may be configured as follows. As a device for playing back images, for example, a video deck is recorded on a video tape, a CD-R drive is used for a CD-ROM, a DVD drive is used for a DVD, and a mpeg format is used on a hard disk. If video is stored, a PC or hard disk recorder may be used.
In this embodiment, the input device has been described using a microphone as an example. However, a keyboard, a touch panel, an electronic pen, or the like may be used. In this case, the audio signal input / output unit 312 in FIG. 3 is additionally changed to an appropriate device according to the device. The storage device may be a storage unit in the video reproduction device or a storage device such as a hard disk. The disk device may be local or may be placed on a network to manage the content description information of the video data. Furthermore, only the search interface may be placed on the network. FIG. 14 is a diagram illustrating an example of a network configuration. The video data editing device 1101, the video data editing / searching device 1102, the video data management device 1103, and the video data searching device 1104 having different functional configurations can access the disk devices of other devices via the network 1100. ing. Of course, information exchange between the same kind of video data editing / retrieval apparatuses 1102 is also possible.

In addition, as an option, it may be possible to set segmentation settings to assist in improving the accuracy of switching of video scenes, setting ON / OFF of addition / deletion of questionnaire items, setting of response status, etc. Conceivable. In that case, the response status settings for the message output for the content description are selected in advance as to whether or not a character appears, whether to output a message by voice or character display, to select a character to appear, and to respond to the character. If so, information (name, age, gender, etc.) for responders, content description items (when, where, who, etc.), and response time for questions from characters can be set.
As segmentation settings, numerical values can be set for video feature values (color, position, texture, time code shift, etc.). Also, information about the video can be input in advance here. When setting by option is not performed, processing is performed according to a mode set as a standard value.
A case where “8 years old, ○ man, man” is registered as an example of responder information set as an option, and “location: amusement park, who: father, mother, ○ man” is registered as video information will be described. Insert the video tape that the video has been recorded, and press the play button character appeared on the monitor, "Hello, I heard you ○ man-kun. ○ man-kun went to the amusement park with dad mom. I hope. I'll tell you what happened because I asked a question. "

1 is a configuration diagram of a video data editing / retrieval system. The figure which shows the processing flow in a video data editing / retrieval apparatus. 1 is a configuration diagram of a video data editing / retrieval apparatus according to an embodiment of the present invention. The figure which shows the example of the content description interface screen displayed on a display, when editing an image | video with an image data editing / retrieval apparatus. The scene change detection process block diagram. A flow chart of changes in video and question items. The figure which shows a question item and a flag bit pattern. TVML creation block diagram. Meta information file creation block diagram. The figure which shows the example of TVML script. The figure which shows the example of an MPEG-7 content description file. The figure which shows the example of an MPEG-7 content description file. The figure which shows the example of an MPEG-7 content description file. The figure which shows the example of a network structure.

Explanation of symbols

1 Video data editing / retrieval device, 2 Display device, 3 Speaker, 4 Disc device, 5 Input device, 6 Audio signal cable, 7 Data cable, 8 Audio signal cable, 9 Video signal cable, 300 CPU, 301 Message creation unit, 302 meta information file creation unit, 303 speech recognition processing unit, 304 digital speech processing circuit, 305 speech recognition circuit, 306 memory, 307 HDD, 308 scene change detection unit, 309 feature amount analysis unit, 310 question item creation unit, 311 A / D converter, 312 Audio signal input / output unit (input unit), 313 Video signal input / output unit, 314 Video drive control unit, 315 Video data search unit, 316 System bus, 317 Display connection unit, 319 TVML creation unit, 320 VCR, 5 00 TVML program creation unit, 501 TVML player, 502 structure storage unit, 503 content storage unit, 504 meta information schema storage unit, 506 meta information storage unit, 632 flag bit pattern, 640 audio section flag, 1100 network

Claims (10)

  In a video data processing apparatus for adding video content information for each scene to video data recorded together with sound, a question item creating means for creating a question item related to a scene, and a user by text or voice corresponding to the question item The video data processing apparatus is characterized in that the question item creating means adds a description item input as a free description to the question item.   The video data processing apparatus according to claim 1, wherein the question item creation unit deletes a question item for which information on the question item is not input from the input unit.   3. The video data processing apparatus according to claim 1, further comprising search means for searching for the portion in which the contents are described, and reproducing the video data from the scene or the scene portion searched by the search means. A video data processing device.   4. The video data processing apparatus according to claim 3, wherein the search means includes a communication means that enables searching for a part in which the content of the video data is described in another video data processing apparatus on the network. .   In a video data processing method for adding video content information for each scene to video data recorded together with audio, a step of creating a question item for creating a question item related to a scene, and a user by text or voice corresponding to the question item The video data processing method characterized in that the question item creating step adds a description item input as a free description to the question item.   6. The video data processing method according to claim 5, wherein the question item creating step deletes a question item for which information on the question item has not been input from the input unit.   7. The video data processing method according to claim 5, further comprising a search step for searching for the content-described portion so that the video data is reproduced from the searched scene or scene portion. Processing method.   8. The video data processing method according to claim 7, wherein the searching step comprises a communication function that enables searching for a portion in which the content of the video data is described in another video data processing device on the network. .   A program for causing a computer to function as the video data processing apparatus according to claim 1, 2, 3, or 4. A computer-readable recording medium on which the video data processing program according to claim 9 is recorded.

Images