JP4235635B2 - Data retrieval apparatus and control method thereof - Google Patents

Description

  The present invention relates to a data search apparatus capable of searching for desired data by specifying a search condition from search target data, and a control method therefor.

  In recent years, the recording method of the video camera has changed from an analog format to a digital format, and various types of video editing using a video camera body or a personal computer have been performed by a user. When editing a movie, a part of the movie data is frequently cut, copied, and pasted. In order to perform these operations, it is necessary to search for and mark the beginning of a cutout portion of a moving image. In addition, when editing a moving image, effects such as subtitle super or a speech balloon are added when the subject is a speaker.

  In this way, a desired scene is searched for a scene in the moving image data, and an editing operation is performed. As one of the methods, there is a method of searching using a speaker's dialogue as a keyword. For example, when searching for a greeting scene of “Good morning”, it is possible to search for a greeting scene in the video data by specifying “Good morning”.

  As such a scene search method, Patent Document 1 proposes a method of searching a subtitle text file by inputting a desired search keyword for a video content file created by a video content file generation system.

  Further, Patent Literature 2 proposes a method of searching for a keyword input by voice using a voice pattern.

JP 2002-374494 A Japanese Patent Laid-Open No. 2001-290496

  However, when searching with keywords in audio and subtitle texts in the prior art, it is not possible to identify the speaker, especially when many people speak the same line (keyword) in the video data. The result is output, and it takes time to identify the speaker, which is inefficient.

  Accordingly, an object of the present invention is to make it possible to efficiently search for data including contents sung by a designated person.

The data search device of the present invention uses identification data for identifying a person to identify the person from data related to the person of the person included in the search target data, and generates data indicating the person 1 data generation means, second data generation means for extracting voice data of the person from the data relating to the person, and generating data indicating the utterance content of the person from the extracted voice data, and the search target data Third data generating means for generating position data indicating the position of the data relating to the person in the image data, and generating existence data indicating whether or not the image data of the person is included in the data relating to the person 4 data generation means, third search condition input means for inputting data specifying whether or not image data of a person to be searched is included, and the third search condition input means When the search condition is input by the search condition input means, the input search is performed from the data set generated by the first data generation means, the second data generation means, and the fourth data generation means. Based on position data corresponding to the data set determined by the determining means for determining a data set that matches the condition, the position data indicated by the position data is searched from the search target data. And a data search means.
A control method for a data search apparatus according to the present invention is a control method for a data search apparatus for searching for data, and the person of the person included in the search target data using identification data for identifying a person. A first data generation step for identifying the person from the data related to the data and generating data indicating the person; extracting voice data of the person from the data related to the person; and uttering the person from the extracted voice data A second data generating step for generating data indicating content, a third data generating step for generating position data indicating the position of the data relating to the person in the search target data, and the data relating to the person A fourth data generation step for generating presence data indicating whether or not the image data of the person is included; and image data of the person to be searched A search condition input step for inputting data specifying whether or not as a search condition; and when the search condition is input by the search condition input step, the first data generation step, the second data generation step, and the first Among the data sets generated by the data generation step of 4, based on the determination step for determining the data set that matches the input search condition, and the position data corresponding to the data set determined by the determination step And a data search step for searching for data at a position indicated by the position data from the search target data.
A program according to the present invention causes a computer to execute the control method of the data search apparatus.
The computer-readable recording medium of the present invention is characterized in that the program is recorded.

  According to the present invention, by designating whether or not a person and the person's dialogue are included together with the image data of the speaker, the content of the specified person is included, and the speaker is reflected in the image. It is possible to efficiently search for existing image data.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments to which the invention is applied will be described in detail with reference to the accompanying drawings.

-First embodiment-
First, a first embodiment of the present invention will be described. 1 and 2 are block diagrams showing the configuration of the speaker specific search apparatus according to the first embodiment of the present invention. Reference numeral 100 denotes the internal structure of the moving image data. The moving image data 100 includes speaker data 109, image data 101, audio data 102, and caption data 103, and the image data 101, audio data 102, and caption data 103 are arranged in time series. Of the data included in the moving image data 100 shown in FIG. 1 or FIG. 2, the image data 101 and the audio data 102 are original data, and the speaker data 109 and the caption data 103 analyze the image data 101 and the audio data 102. As shown in FIG. 1 or FIG. 2, it is data that is generated later and embedded at a predetermined position. The internal structure of the moving image data shown here is an example and does not limit the present invention.

  The caption data 103 includes speaker information 106, caption text information 107, and utterance start information 108. In the speaker data 109, data related to a speaker in the original moving image data 100 is stored. The speaker data 109 includes, for example, data for identifying a speaker (for example, data indicating the name of the speaker), face identifying feature data, face image data, voice identifying feature data, and the like. Stored in association with each speaker. The speaker data 109 is read by the speaker data reading unit 120, and a list of speakers existing in the moving image data can be acquired. In the present embodiment, a configuration in which the speaker data 109 is included in the moving image data 100 is adopted. However, as another embodiment, the moving image data 100 is not included in the moving image data 100. It is also possible to use a configuration that is held in the memory and read as necessary and used in the corresponding processing.

  The caption data 103 includes speaker information 106, text information 107, and utterance start information 108. The speaker information 106 is generated when a speaker is specified from image data using the face identification feature value data or when a speaker is specified from voice data using the voice identification feature value data. Data for identifying a speaker. The text information 107 is text data obtained by analyzing the voice data of the speaker specified this time when the voice data is identified using the voice identification feature data, and converting it into text. The utterance start information 108 is information including time information when the voice data of the speaker specified this time is identified.

  Reference numeral 110 denotes a caption data reading unit that sequentially reads only the caption data 103 from the moving image data 100. The read subtitle data 103 is sent to the speaker identification acquisition unit 111, the text acquisition unit 112, and the utterance start information acquisition unit 113, and the subtitle speaker information 106, the subtitle text information 107, and the subtitle utterance start information, respectively. 108 is obtained. The acquired speaker information 106 and subtitle text information 107 are sent to the subtitle data comparison unit 115.

  FIG. 3 shows an example of caption data. A caption data reading unit 110 reads only the caption data 103 from the moving image data 100. Examples of read subtitle data 103 are 121 and 122. The following description is given by taking the caption data 122 as an example. The caption data 122 read by the caption data reading unit 110 is sent to the speaker identification acquisition unit 111, and the speaker information 106 is read. In the present embodiment, the speaker information 106 is a portion surrounded by Speaker tags. It is acquired from the caption data 122 that “B child” surrounded by <Speaker>... </ Speaker> is a speaker.

  The caption data 122 read by the caption data reading unit 110 is sent to the text acquisition unit 112, and the text information 107 of the caption is read. In the present embodiment, the subtitle text information 107 is a portion surrounded by SubTitle tags. From the subtitle data 122, “Good morning.” Enclosed in <SubTitle> .... </ SubTitle> is acquired as subtitle text.

  The caption data 122 read by the caption data reading unit 110 is sent to the utterance start information acquisition unit 113, and the utterance start information 108 is read. In the present embodiment, the utterance start information 108 is a portion surrounded by a StartTimeCode tag. From the subtitle data 122, “T01: 12: 03 11” surrounded by <StartTimeCode>... </ StartTimeCode> is acquired as the utterance start time code. In the present embodiment, subtitle data is created using tags, but may be in other formats.

  Reference numeral 114 denotes a search condition input unit, which inputs a speaker and the content uttered by the speaker as a search condition. The speaker input means may be selection of a person name, selection of a face, or selection from an image. The content input means spoken by the speaker may be text input or voice input. The input search condition is sent to the caption data comparison unit 115.

  The subtitle data comparison unit 115 compares the search condition, the speaker information 106, and the subtitle text information 107, and searches and specifies the subtitle data 116 that matches. In the search condition input unit 114, when the speaker is "B child" and the content of the speaker uttered is "Good morning", the caption data 121 is the mismatched caption data because the speaker information is "A child". And the subtitle data 122 is determined to be matched, and the matched subtitle data 116 is obtained. The matched subtitle data 116 is used by combining the utterance start information 108 acquired from the subtitle data 122 as the utterance start information 117 of the matched subtitle data.

  The utterance start information 117 of the matching subtitle data is sent to the voice data search unit 118 and the moving image data search unit 119 as the utterance start time code. FIG. 2 is a diagram for explaining operations of the audio data search unit 118 and the moving image data search unit 119.

  In the example of the caption data 122, “T01: 12: 03 11” is sent to the audio data search unit 118 and the moving image data search unit 119 as the time code for starting speech. The audio data search unit 118 reads only the audio data 102 from the moving image data 100 and searches for the position of the time code. In FIG. 1 or FIG. 2, the position indicated by (1) is the data position of the searched voice. The moving image data search unit 119 reads only the image data 101 from the moving image data 100 and searches for the position of the time code. In FIG. 1 or FIG. 2, the position shown in (2) is the data position of the searched image.

  FIG. 4 is an example of caption display. Reference numeral 200 denotes screen data in which a child B 201 is reflected and the child B is uttering “Good morning”. A subtitle displayed in accordance with the subtitle data 122 is 202.

  FIG. 5 is a diagram showing a screen configuration example displayed by the search software on the speaker specific search device. Reference numeral 300 denotes a main operation screen. Reference numeral 301 denotes an image data display screen. Reference numeral 302 denotes a time code of image data displayed on the image display screen 301. Reference numeral 305 denotes a dial whose operation is changed by a jog button 303 and a shuttle button 304. When the jog button 303 is pressed, the dial 305 functions as a jog dial, and frame advance is possible in the rotation direction, and the frame feed speed is changed according to the rotation speed. When the shuttle button 304 is pressed, the dial 305 functions as a shuttle dial, and changes the frame feed direction and speed according to the rotation direction and rotation angle. Reference numeral 306 denotes a move button to the front mark point, 307 denotes a rewind button, 308 denotes a play button, 309 denotes a fast forward button, 310 denotes a move button to the rear mark point, and 311 denotes a search button.

  FIG. 6 is a diagram showing a search condition input screen 320 displayed when the search button 311 is pressed together with the main operation screen. Reference numeral 321 denotes a pull-down menu for selecting a speaker, which selects the speaker. Reference numeral 322 denotes a screen for inputting dialogue to be searched. Reference numeral 323 denotes a forward search button, which executes a search retroactively from the current time code. Reference numeral 324 denotes a backward search button, which executes a search after the current time code. Reference numeral 325 denotes a cancel button, which stops the search condition input.

  The flow of processing by software in the first embodiment will be described with reference to the flowchart of FIG. Here, a procedure for searching for a scene where child B uttered “Good morning” from moving image data 100 including subtitle data 103 will be described as an example.

  When the search button 311 is pressed on the search software screen 300 in FIG. 5, a scene search step (step S100) is entered. When entering the scene search step (step S100), the search condition input screen 320 of FIG. 6 is displayed.

  Subsequently, a speaker specifying step (step S101) is executed. In the speaker specifying step (step S101), the speaker data 109 is read from the moving image data 100 by the speaker data reading unit 120, and a list of speakers in the moving image data 100 is acquired. A list of acquired speakers can be selected by a speaker selection pull-down 321. In the speaker pull-down 321 of this embodiment, the speaker name included in the speaker data 109 is displayed, and the speaker is selected by the speaker name. When selecting a speaker, it is also possible to display and select the speaker's face other than the speaker name. In the example of FIG. 6, “B child” is selected.

  After designating the speaker, the speaker's utterance content input step (step S102) is executed. In the speaker content input step (step S102), the speech uttered by the speaker is input from the speech input screen 322. In the example of FIG. 6, “Good morning” is specified. By these steps, “Good morning” is set as a search condition by “B child”, and a scene having the same condition as subtitle data is searched by pressing the forward search button 323 or the backward search button 324. . In this embodiment, the search conditions are set in the order of the speaker specifying step (step S101) and the utterance content input step (step S102), but the order may be changed.

  The search is started when the forward search button 323 or the backward search button 324 is pressed. In the search, first, a subtitle data reading step (step S103) is executed. In the caption data reading step (step S103), the caption data reading unit 110 sequentially reads only the caption data 103 from the moving image data 100. As for the read caption data 103, speaker information 106 is acquired by the speaker specification acquisition unit 111 in the speaker specification acquisition step (step S104). For example, “A child” is acquired as the speaker for the caption data 121 and “B child” is acquired as the speaker for the caption data 122. The acquired speaker is compared with the speaker to be searched specified in the speaker specifying step (step S101) in the speaker matching step (step S105). In the example of FIG. 6, since “B child” is designated as the speaker to be searched, it is determined that the caption data 121 does not match and the caption data 122 matches. If they do not match, the process returns to the next caption data reading step (step S103). If they match, the process proceeds to the next text acquisition step (step S106).

  In the text acquisition step (step S106), the text acquisition unit 112 acquires subtitle text information 107 from the subtitle data 103 read in the subtitle data read step (step S103). For example, “Good morning” is acquired as the caption text 121 and “Good morning” is acquired as the caption text for the caption data 122. The obtained subtitle text is compared with the search target dialogue specified in the utterance content input step (step S102) in the text matching step (step S107). In the example of FIG. 6, “Good morning” is specified as the search target dialogue, so that the caption data 122 is determined to match. If they do not match, the process returns to the next caption data reading step (step S103). In the case of coincidence, the caption data 122 is passed to the utterance start information acquisition step (step S108) as the matched caption data 116.

  In the utterance start information acquisition step (step S108), the utterance start information acquisition unit 113 acquires the utterance start information 108 from the caption data 103 read in the caption data read step (step S103). For example, “T01: 11: 50 03” is acquired as the utterance start information for the caption data 121 and “T01: 12: 03 11” is acquired for the caption data 122. Here, since the “B child” subtitle data 122 is sent as the matched subtitle data 116, “T01: 12: 03 11” is acquired as the matched utterance start information 117. The acquired utterance start information 117 is transferred to the voice data search step (step S109).

  In the audio data search step (step S109), only the audio data 102 is sequentially read from the moving image data 100 by the audio data search unit 118, and the position (1) designated by the matched utterance start information 117 is searched. Here, since “T01: 12: 03 11” is input as the matched utterance start information 117, the audio data position of the time code “01:12:03 11” is acquired.

  The utterance start information 117 acquired in the utterance start information acquisition step (step S108) is passed to the moving image data search step (step S110). In the utterance start information acquisition step (step S108), only the image data 101 is sequentially read out from the moving image data 100 by the moving image data search unit 119, and the position (2) designated by the matched utterance start information 117 is searched. Here, since “T01: 12: 03 11” is input as the matched utterance start information 117, the image data position of the time code “01:12:03 11” is acquired.

  In the search result OK step (step S111), the searched audio data and image data are displayed on the image display screen 301 and the time code display 302, and the search result is confirmed. If this search result is acceptable, the search ends. When further search is performed, the next subtitle data is read by pressing the button 306 for moving to the front mark point or the button 310 for moving to the rear mark point. It is possible to repeat the search under the same conditions.

  The flowchart shown here is one embodiment and does not limit the present invention. The order of the speaker matching step (step S105) and the text matching step (step S107) may be changed, and the order of the voice data searching step (step S109) and the moving image data searching step (step S110) may be changed. .

  In the present embodiment, the search is performed for each item. However, all subtitle data matching the search condition is searched from the moving image data 100, and the search result is displayed on the image display screen 301 as a plurality of index images. It is also possible to display.

  In the present embodiment, the number of speakers is limited to one in the speaker designation step (step S101), but it is also possible to designate and search for two or more speakers. In this embodiment, one line is specified in the utterance content input step (step S102). However, it is also possible to search by specifying a plurality of lines.

  Furthermore, in the present embodiment, it is determined in the text matching step (step S107) that the search condition and the caption text of the caption data match, but this can be extended to known search methods such as regular expressions and fuzzy searches. Is also possible.

  As described above, according to the present embodiment, it is possible to easily search for an image using a specific dialogue of a specific speaker, so that a scene search can be performed efficiently.

-Second Embodiment-
Next, a second embodiment of the present invention will be described. FIG. 9 is a diagram illustrating a configuration of an imaging apparatus according to the second embodiment of the present invention. Since the image pickup apparatus shown in FIG. 9 generally has no character input means such as a keyboard, it is necessary to specify a search condition by an image. In the present embodiment, a solution example in such a case is shown.

  In the image pickup apparatus of FIG. 9, reference numeral 400 denotes an image pickup apparatus body (operation surface). The imaging device has a lens unit (not shown), an imaging unit (not shown), a signal processing unit (not shown), and a recording medium (not shown), and images taken through the lens unit are captured by the imaging unit. The signal is captured, subjected to digital signal processing by the signal processing unit, and displayed on the liquid crystal display device 401 on the rear surface. Further, when the recording button 405 is pressed, an image captured through the lens unit is captured by the imaging unit, and the image data subjected to the digital signal processing by the signal processing unit is stored in the recording medium. Hereinafter, the configuration of the moving image data 100 shown in FIG. 1 or FIG.

  In the imaging device 400 of FIG. 9, 401 is a liquid crystal display device for displaying captured images and reproduced images and for displaying various setting screens. Reference numeral 402 denotes a time code of the displayed image data. Reference numeral 403 denotes an operation member for selection, which includes buttons in the up / down / left / right directions. A setting button 404 is pressed to determine the result selected by the selection operation member 403. Reference numeral 405 denotes a recording button which starts and stops recording. 406 is a move button to the front mark point, 407 is a rewind button, 408 is a play button, 409 is a fast forward button, 410 is a move button to the back mark point, 411 is a search button, and 412 is a menu button. Reference numeral 413 denotes a microphone for voice input.

  Also in this embodiment, the moving image data 100 includes image data 101, audio data 102, speaker data 109, and caption data 103. The data structure of the caption data 103 and the process for generating it are the same as in the first embodiment described above, and the speaker data 109 is data for identifying a speaker, for example, as in the first embodiment. (For example, data indicating the name of a speaker, etc.), face identifying feature data, voice identifying feature data, and the like are stored in association with each speaker. The speaker data 109 may be included in the moving image data 100 as shown in FIG. 1 or FIG. 2, or may be held in a recording medium inside or outside the image pickup apparatus and read as necessary. It may be configured to be used in processing.

  By the way, when the speaker is specified from the voice data using the voice identification feature quantity data and the caption data 103 is created, the speaker is specified from the voice data, but the speech of the voice data is spoken. There is a case where the speaker does not exist in the screen and the image data of the speaker is not included in the image data corresponding to the sound data.

  In the present embodiment, in view of the above case, it is determined whether or not the speaker for which the caption data 103 is to be created exists in the screen (whether or not the speaker can be identified from the corresponding image data). The in-screen presence information shown is generated. This is because the speaker can be specified only by the feature data for voice identification, and when the speaker cannot be specified by the feature data for face identification, the speaker who spoke the speech of the corresponding voice data In-screen presence information is generated to the effect that it does not exist in the screen. On the other hand, when subtitle text information 107 is generated (that is, when a speaker is identified from voice data by at least voice identification amount data, and the voice data is analyzed and converted into text), In other cases, in-screen presence information is generated to the effect that the speaker who has spoken the speech data is present in the screen.

  FIG. 8 shows an example of the caption data 103 created in this way. The part enclosed by the Speaker tag is the speaker name corresponding to the speaker information. The speaker name is generated by identification information for uniquely specifying the above-mentioned speaker in the speaker data 109, for example. It is also possible to update this to the speaker name later. The portion surrounded by the Existence tag indicates the in-screen presence information. In the subtitle data 131, the content enclosed by <Existence>... </ Existence> is “Y”, so the speaker is present on the screen. In the caption data 133, the content enclosed by <Existence>... </ Existence> is “N”, so that no speaker exists in the screen. The part enclosed by the SubTitle tag is the subtitle text spoken by the speaker. The portion enclosed by the StartTimeCode tag is the time code when the speaker starts speaking, and corresponds to the generation start information. In the caption data 131, the speaker is "A child", and the child A is reflected in the image of the frame number indicated by the time code "T01: 11: 50 03". From that position, say "Good morning." It shows that. In the caption data 132, the speaker is “B child”, and the child B is reflected in the image of the frame number indicated by the time code “T01: 12: 03 11”. From that position, say “Good morning.” It shows that. In the caption data 133, the speaker is “C child” and the frame number indicated by the time code “T01: 12: 23 10” does not show the child C. From that position, “Today is good weather.” Indicates that you are speaking.

  FIG. 10 is a diagram illustrating an example of a search condition designation screen in the imaging apparatus 400. Reference numeral 420 denotes a list of search target speakers. Reference numeral 421 denotes a speaker selection display frame indicating a speaker being selected. 422 is the face of child A, 423 is the face of child B, and 424 is the face of child C. In this screen, a speaker is selected using the operation member 403 for selection. Reference numeral 425 is a display of lines to be searched.

  FIG. 11 is a diagram illustrating another example of a search condition designation screen in the imaging apparatus 401. Reference numeral 430 denotes a child A, and reference numeral 431 denotes a registered speaker frame indicating that the child A is recognized as a speaker. Reference numeral 432 denotes a child B, and reference numeral 433 denotes a selected speaker frame indicating that the child B is recognized as a speaker and is selected as a speaker to be searched. Reference numeral 434 denotes a child C, and reference numeral 435 denotes a registered speaker frame indicating that the child C is recognized as a speaker. Reference numeral 436 is a display of dialogue to be searched.

  The search screens in FIGS. 10 and 11 can be selected and switched from the menu. 10 can be displayed once by pressing the search button 411, and the search screen in FIG. 11 can be displayed by further pressing the search button 411.

  FIG. 12 is a flowchart showing the flow of processing in this embodiment. When the search button 411 is pressed, a search mode step (step S201) is entered. In the search mode step (step S201), either a speaker list selection screen (FIG. 10) or a speaker selection screen from an image (FIG. 11) is displayed. In the present embodiment, the menu button 412 is operated to set a screen to be displayed first by a speaker selection method designation menu (not shown).

  First, the case where the speaker list selection screen (FIG. 10) is displayed in the search mode step (step S201) will be described. In the speaker selection step (step S202), the image data of the speaker's face is acquired from the speaker data, and a list of registered speakers is displayed. In this embodiment, three persons A, B, and C are registered as speakers. In FIG. 10, 422 is the face of child A, 423 is the face of child B, and 424 is the face of child C. When the speaker name is registered in the speaker data, the name may be displayed on the right side of the face. When the list of speakers is displayed, the speaker selection display frame 421 can be moved by the selection operation member 403 to select the speakers. When four or more speakers are registered, the selection operation member 403 moves the speaker selection display frame 421 and scrolls the speaker list in the search target speaker list display 420. A speaker to be searched is selected and set by the setting button 404. FIG. 10 shows a state where child B is selected as a speaker. When the speaker is determined, the process proceeds to the speaker specifying step (step S204), and the speaker data of the selected speaker is acquired.

  When the speaker data is acquired in the speaker specifying step (step S204), the utterance content input step (step S205) is performed. In the present embodiment, voice input is performed in the utterance content input step (step S205). When facing the speech input microphone 413 and speaking the speech to be searched, speech recognition is performed and the text is automatically converted into text and input to the speech display 425 for searching. In FIG. 10, “Good morning” is designated as the utterance content. If the input is not correct, the user can press the left button of the selection operation member 403 to delete the character in the dialogue display 425 to be searched and input again. When the utterance content input is correctly input, the setting button 404 is used for determination. Through these steps, "Good morning."

  Next, a case where a speaker selection screen (FIG. 11) from image data is displayed in the search mode step (step S201) will be described. In the speaker selection step (step S202), face feature amount data for identifying the speaker is acquired from the speaker data, and face detection is performed from the image data displayed on the liquid crystal display device 401 in the speaker search step (step S203). The registered speaker frame is displayed on the image data of the speaker's face registered in the speaker data.

  In the example of FIG. 11, in the speaker search step (step S203), three members A child 430, B child 432, and C child 434 are displayed on the liquid crystal display device 401, and face detection is performed from the image data of each face. The face feature amount is calculated and compared with the speaker face feature amount data registered in the speaker data. As a result of comparison, since each speaker is registered, a speaker registration frame is displayed in the face image data (A child registered speaker frame 431, B child selected speaker frame 432, C child registered speaker frame. 435), the face image data and the speaker data of each speaker are associated with each other. When the registered speaker frame and the selected speaker frame are displayed on the speaker's face, the selected speaker frame can be moved by the selection operation member 403. Since the selection range of the speakers is selected only from the speakers displayed on the liquid crystal display device 401, even if four or more speakers are registered in the case of FIG. Select a speaker. When the image data of the face of the speaker to be searched does not exist in the liquid crystal display device 401, the image data of the face of another speaker is reflected by changing the display image with the rewind button 407 and the fast forward button 409. In this state, if the speaker is registered in the speaker data, the registered speaker frame is also displayed in the face image data, and the same can be searched.

  The user selects a speaker to be searched and determines with the setting button 404. FIG. 11 shows a state where child B is selected as a speaker. When the speaker is determined, the process proceeds to a speaker specifying step (S204), and speaker data of the selected speaker is acquired as a search condition.

  When the speaker data is acquired in the speaker specifying step (step S204), the utterance content input step (step S205) is performed. In the present embodiment, voice input is performed in the utterance content input step (step S205). When facing the speech input microphone 413 and speaking the speech to be searched, speech recognition is performed and the text is automatically converted into text and input to the speech display 436 for searching. In FIG. 11, “Good morning” is designated as the utterance content. If the input is not correct, the user can press the left button of the selection operation member 403, delete the characters in the dialogue display 436 to be searched, and input again. When the utterance content input is correctly input, the setting button 404 is used for determination. Through these steps, "Good morning."

  When the search condition is determined in the utterance content input step (step S205), the following operation is the same as the operation after the subtitle data reading step (step S103) of the first embodiment.

  In the second embodiment, as shown in FIG. 8, the subtitle data 103 includes data indicating the presence of the speaker in the screen (in-screen presence information). By specifying the presence of the speaker in the screen as a search option, it is possible to search for a scene based on the speaker's and speaker's on-screen presence and the speaker's dialogue. In this case, in the speaker specifying step (step S204), the presence or absence of the speaker in the screen is designated. As an example of the operation, presence / absence in the screen is selected with the up and down buttons of the selection operation member 403, and determined with the setting button 404.

  The speaker's in-screen presence information is read by reading the caption data 103 in the caption data reading step (step S206) and acquiring the speaker information 106 in the speaker specifying acquisition step (step S207). The acquired speaker information 106 includes a speaker name and in-screen presence information. When the speaker information 106 is acquired in the speaker specifying acquisition step (step S207) for the caption data 131, information "Y" in which "A child" is present in the image as the speaker is acquired. When the speaker information 106 is acquired in the speaker identification acquisition step (step S207) with respect to the caption data 132, information "Y" in which "B child" exists in the image is acquired as a speaker. When the speaker information 106 is acquired in the speaker identification acquisition step (step S207) for the caption data 133, information “N” in which “C child” does not exist in the image is acquired as a speaker.

  The speaker name and in-screen presence information acquired from the caption data 103 in the speaker specifying acquisition step (step S207) are the search conditions set in the speaker specifying step (step S204) in the speaker matching step (step S208). Compared with

  The audio data 102 at the position (1) and the image data 101 at the position (2) shown in FIG. 1 or FIG. 2 retrieved by these steps are displayed on the liquid crystal display device 401, and the time code is displayed in the time code display area 402. Is displayed. Since the subtitle data 132 corresponds to the search result in the present embodiment, “01:12:13 11” is displayed as the time code. When the play button 408 is pressed, the video showing the child B is started, the subtitle super “Good morning” is displayed, and “Good morning” is played with the child B's voice.

  As described above, according to the present embodiment, it is possible to easily search for an image using a specific speech of a specific speaker even in a device having no text input means and selection means such as a keyboard such as an imaging device. The scene search can be performed efficiently.

  As described above, according to each of the embodiments described above, a scene in image data including the content spoken by the designated speaker can be efficiently searched by designating the speaker and searching with dialogue (keywords). It becomes possible to do.

  Also, by specifying the presence of the speaker in the screen and searching with dialogue (keywords), it is possible to efficiently search the contents of the specified person and scenes in the image data where the speaker is reflected in the image. It becomes possible to do.

  Another object of the present invention is to supply a storage medium storing software program codes for realizing the functions of the above-described embodiments to a system or apparatus, and the computer (or CPU or MPU) of the system or apparatus stores the storage medium. Needless to say, this can also be achieved by reading and executing the program code stored in.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium storing the program code constitute the present invention.

  As a storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (basic system or operating system) running on the computer based on the instruction of the program code. Needless to say, a case where the functions of the above-described embodiment are realized by performing part or all of the actual processing and the processing is included.

  Further, after the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function is determined based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion board or function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

It is a block diagram which shows the structure of the speaker specific search apparatus which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the structure of the speaker specific search apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows an example of caption data. It is a figure which shows an example of a caption display. It is a figure which shows the example of a screen structure displayed with the search software on a speaker specific search device. It is a figure which shows the search condition input screen at the time of pressing down a search button with a main operation screen. It is a flowchart which shows the flow of the process in the 1st Embodiment of this invention. It is a figure which shows an example of caption data. It is a figure which shows the structure of the imaging device which concerns on the 2nd Embodiment of this invention. It is a figure which shows an example of the search condition designation | designated screen in an imaging device. It is a figure which shows an example of the search condition designation | designated screen in an imaging device. It is a flowchart which shows the flow of the process in the 2nd Embodiment of this invention.

Explanation of symbols

  100: moving image data, 101: image data, 102: audio data, 103: subtitle data, 106: speaker information in subtitle data, 107: subtitle text information in subtitle data, 108: utterance start information in subtitle data, 109: Speaker data, 110: Subtitle data reading unit, 111: Speaker specification acquisition unit, 112: Text acquisition unit, 113: Speaking start information acquisition unit, 114: Search condition input unit, 115: Subtitle data comparison unit, 116 : Matched subtitle data, 117: utterance start information of matched subtitle data, 118: audio data search unit, 119: moving image data search unit, 120: speaker data read unit, 121: subtitle data of child A, 122: Subtitle data of child B, 131: Subtitle data of child A, 132: Subtitle data of child B, 133: Subtitle data of child C, 201: Video of child B 202: Subtitle, 300: Search software main screen, 301: Image display screen, 302: Time code display, 303: Jog button, 304: Shuttle button, 305: Jog, shuttle dial, 306: Move button to the front mark point 307: Rewind button, 308: Play button, 309: Fast forward button, 310: Move to back mark point, 311: Search button, 320: Search condition input screen, 321: Pull-down for selecting speaker, 322: Screen for inputting dialogue to search 323: Forward search button, 324: Back search button, 325: Cancel button, 400: Imaging device body (operation surface), 401: Liquid crystal display device, 402: Time code display, 403: Selection Operation member 404: setting button 405: recording button 406: Move button to the front mark point, 407: Rewind button, 408: Play button, 409: Fast forward button, 410: Move button to the rear mark point, 411: Search button, 412: Menu button, 413 : Voice input microphone 420: Search target speaker list display 421: Speaker selection display frame 422: Child A face 423: Child B face 424: Child C face 430: Child A face 431: A child registered speaker frame, 432: B child face, 433: B child selected speaker frame, 434: C child face, 435: C child registered speaker frame, 436: Dialog to search display

Claims (6)

First data generating means for identifying the person from data relating to the person of the person included in the search target data using identification data for identifying the person, and generating data indicating the person;
Second data generating means for extracting voice data of the person from the data related to the person, and generating data indicating the utterance content of the person from the extracted voice data;
Third data generation means for generating position data indicating the position of data relating to the person in the search target data;
Fourth data generation means for generating presence data indicating whether or not the image data of the person is included in the data relating to the person;
A third search condition input means for inputting data specifying whether or not image data of a person to be searched is included as a search condition;
When a search condition is input by the third search condition input unit, among a set of data generated by the first data generation unit, the second data generation unit, and the fourth data generation unit, Determination means for determining a set of data that matches the input search condition;
A data search device comprising: data search means for searching data of a position indicated by the position data from the search target data based on position data corresponding to the data set determined by the determination means. . Display control means for displaying the image data of each person on the display means;
A selection means capable of selecting arbitrary image data from the image data of each person;
2. The data search apparatus according to claim 1, further comprising first search condition input means for inputting data for specifying a person corresponding to the image data selected by the selection means as a search condition. . Identification means for identifying a person corresponding to the image data displayed on the display means using the identification data;
A selection means capable of selecting arbitrary image data from the image data of each person identified by the identification means;
The data search device according to claim 1, further comprising second search condition input means for inputting data for specifying a person corresponding to the image data selected by the selection means as a search condition. . A method for controlling a data search apparatus for searching data, comprising:
A first data generation step of identifying the person from data related to the person included in the search target data using identification data for identifying the person, and generating data indicating the person;
A second data generation step of extracting voice data of the person from the data relating to the person, and generating data indicating the utterance content of the person from the extracted voice data;
A third data generation step of generating position data indicating a position of data related to the person in the search target data;
A fourth data generation step of generating presence data indicating whether or not the image data of the person is included in the data related to the person;
A search condition input step for inputting, as a search condition, data specifying whether or not image data of a person to be searched is included;
When a search condition is input in the search condition input step, it is input out of the data set generated in the first data generation step, the second data generation step, and the fourth data generation step. A determination step of determining a set of data that matches the search condition;
And a data search step of searching from the search target data for data at a position indicated by the position data based on the position data corresponding to the set of data determined by the determination step. Control method.   The program for making a computer perform the control method of the data search device of Claim 4.   A computer-readable recording medium on which the program according to claim 5 is recorded.

Images