JP2006107109A - Information management device and information management method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform retrieval based on annotation information even for information without an annotation. <P>SOLUTION: Retrieval information is retrieved on the basis of an inputted retrieval condition and the annotation information related to the retrieval information, related information is extracted from a storage means storing the retrieval information for each retrieved retrieval information, and the extracted information and the related information are related and presented to a user. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an information management apparatus and an information management method for searching for information.

  In recent years, the spread of digital cameras and the like has been remarkable. In general, a user manages a digital image captured by a portable imaging device such as a digital camera using a PC or a server. For example, captured images can be organized in folders on a PC or server, or specific images can be printed and incorporated into New Year's cards or the like. In addition, when managed by the server, some images can be disclosed to other users.

  When performing such work, it is necessary to find a specific image intended by the user. When the number of images to be found is small, the images can be displayed as thumbnails, and can be found visually from the list. However, when the number of target images is hundreds, or when the target image group is divided and stored in a plurality of folders, it is difficult to find it visually.

  Therefore, an audio annotation (annotation by voice) is added to an image on the imaging apparatus, and the information is used at the time of retrieval. For example, an image of a mountain is picked up by a portable imaging device, and “Hakone no Yama” is uttered to the image. This audio data is stored in the imaging device as a pair with the previous image data, and then recognized in the imaging device or in the PC to which the image is uploaded, and converted to text information “Hakoneyama”. Is done. Once the voice annotation data is converted into text information, it can be processed with a general text search technique, and the image can be searched with text input such as “Yama” or “Hakone”. .

  Patent Document 1 is a prior art using such voice annotation. In Patent Document 1, a user inputs a voice to be an annotation at the time of image capture or after image capture, and the sound data is used for image search using an existing speech recognition technology.

  In a general search system using annotation information, an image to which no annotation is attached cannot be a search target. However, it is burdensome for the user to add annotation information to all search objects. Therefore, approaches for automatically adding annotation information and approaches for estimating annotation information of an object from annotation information attached before and after are proposed.

  For example, Patent Document 2 is a system that can add annotation information to an image on an imaging apparatus. Here, an embodiment is described in which the same annotation information as that attached to the previous image is attached unless the annotation information attached to the image is changed.

In Patent Document 3, there is a proposal to copy annotation information from the immediately preceding image only when the time lag with the image captured immediately before is small.
JP 2003-219327 A JP 2003-224750 A Japanese Patent Laid-Open No. 7-121561

  In these proposals, a copy of the annotation information and a link to the annotation information are given even to those that are not annotated. Therefore, it is effective in reducing the burden on the user. However, since these processes are performed in the image pickup apparatus at the time of image pickup, a copy of the annotation information and the link information is held in the memory in the image pickup apparatus, which causes a problem that the memory use efficiency is lowered. . Another problem is that images captured by different image capturing apparatuses cannot share annotation information.

  An information management apparatus according to the present invention includes a search unit that searches for a search target image based on input search conditions and annotation information associated with the search target information, and for each search target information searched by the search unit. , An extraction unit that extracts a related image from a storage unit that stores the search target information, and a presentation control unit that controls the image extracted by the search unit and the related image to be presented to the user in association with each other. It is characterized by.

  According to the present invention, it is possible to perform a search based on annotation information even for information without an annotation. In order to do these things, it is common to copy the annotation information attached to the previous image to the image without the annotation information. However, according to the present invention, this process is not necessary, so that it is possible to prevent an increase in memory consumption in the imaging apparatus.

  Hereinafter, an embodiment of an information management apparatus according to the present invention will be described with reference to the drawings, taking an example of managing images.

Example 1
FIG. 1 is a functional configuration diagram of the information management apparatus according to the present embodiment. The information management apparatus 101 of the present invention is connected to a control command issuing unit 102, a search character string input unit 103, and an image output unit 104. The control command issuing unit 102 is a part that transmits an event for voice recognition of voice annotation data and a command for executing an image search to the information management apparatus. As the control command issuing unit 102, for example, a GUI button or a physical button can be considered. Alternatively, it may be considered that a specific command is issued when a specific memory device (compact flash (registered trademark) card) is inserted into the slot. The search character string input unit 103 is a device for inputting a search query for performing an image search, and a keyboard and a microphone for inputting voice are conceivable. The image output unit 104 is a part for presenting the searched image candidates to the user, and a display or the like is assumed.

  An image / audio database 111 is included in the image apparatus. This database stores images captured from the imaging device and audio annotation data added to the images.

  FIG. 10 shows an example of an audio / image database. The elements of this database include image data picked up by the image pickup device, voice annotation data associated with the image data, and a voice recognition result storage file that stores the result of voice recognition of the voice annotation data by the information management device. It is. These three elements are not always aligned. For example, in an image without audio annotation data, the audio annotation data slot is empty. For voice annotation data that has not yet been voice-recognized, the slot of the voice recognition result storage file is empty. Correspondence between each data element may be held as correspondence information separately, and it is considered that the same file name and only the extension part is different as in the example of FIG. Good.

  The control unit 107 in the information management apparatus is a part that receives control commands from the outside of the apparatus and executes various commands. This command is at least a command for performing voice recognition of voice annotation data (voice recognition command) and a search command for searching for an image using a search query.

  When a voice recognition command is input, the control unit accesses the voice annotation data recognition unit 106. Then, the speech annotation data in the image / speech database 111 is speech-recognized using speech recognition data 105 including the feature values (acoustic model) of human voice and language constraints (language model) collected in advance. At this time, all voice annotation data in the image database may be voice-recognized, or may be performed only on voice annotation data that has not yet been voice-recognized. In the example of FIG. wav and 106. Since there is no voice recognition result storage file corresponding to wav, these two voice annotation data are recognized.

  For example, when the voice annotation data is a voice “Hakone no Yama”, a phoneme string “hakonenoyama” is ideally output as a voice recognition result. These speech recognition results are sent to the speech recognition result storage unit 110 and added to the image / speech database in the form of a speech recognition result storage file. FIG. 7 shows an example of a speech recognition result storage file. Here, five candidates are output as recognition results for one voice annotation data.

  When a search command is input from the control command issuing unit 102, the control unit 107 works on the search unit 108. Here, an example in which an image is retrieved by a text query will be described. When the search process starts, the search unit takes in the text from the search character string input unit 103 outside the apparatus. For example, when a character string “Hakone no Yama” is captured, the search unit automatically reads the input character string and converts it into a phoneme string (voice information) such as “hakonenoyama”. Further, the similarity between the utterance information and the phoneme string of the speech recognition result written in the speech recognition result storage file in the image / speech database is calculated, and N images having a large similarity are selected. Here, these images are called search result images.

  The related image extraction unit 112 extracts an image group related to the search result image. This extraction method will be described in detail later. The search result image and the related image are sent to the search result presentation unit 109 and output to the image output unit 104 outside the apparatus.

  2 and 3 are flowcharts of the information management apparatus. Here, based on this flow, a specific flow from image capture to search will be described.

  First, the user needs to capture an image using the portable image capturing device 401 of FIG. This portable image pickup apparatus has an image confirmation screen 403. Further, it is assumed that the imaging mode / captured image confirmation mode can be switched by switching the operation mode switching switch 405. In the captured image confirmation mode, it is assumed that the image captured so far can be confirmed on the image confirmation screen.

  The user can add a voice annotation to the specific image in the captured image confirmation mode. For example, an image to which a voice annotation is to be added may be displayed on the image confirmation screen, and the voice annotation data may be added to the image by pressing the voice annotation giving button 402 of the device. Specifically, when this button is pressed, sound for a certain period of time is captured from the microphone 404, and the sound data is associated with an image and stored in a memory in the imaging apparatus.

  However, it cannot be expected that the user attaches audio annotations to all images because the load on the user is large. Therefore, as shown in FIG. 5, a voice annotation is added to each scene break.

  FIG. 5 shows that an audio annotation “mountain of Hakone” is added 502 to the first image to be shot, and then two images without audio annotation 503 are captured. Then, after the next image is captured, a voice annotation “Lake Ashinoko” is added to the image.

  When the user returns from a trip and connects the portable image pickup device to the PC, a dialog window as shown in FIG. 6 is opened to prompt the user to upload the data in the portable image pickup device to the PC. The user selects an image to be uploaded and presses an upload instruction button 605. Then, the target image and audio annotation data added to the image are uploaded to the image / audio database on the PC.

  When this upload process is completed, the flow shown in FIG. 2 is started. When this flow is entered, the image / sound database in the apparatus is searched (S201), and it is determined whether there is speech annotation data that has not yet been speech-recognized (S202). If all voice annotation data has already been recognized, the flow is exited.

  If there is a voice annotation that has not yet been recognized, the voice annotation data is acquired (S203). Then, the voice data is recognized (S204). The voice recognition result is stored in the image / voice database as a voice recognition result storage file (S205).

  One day, you want to use an image uploaded by the user. Then, the user starts up an image search program as shown in FIG. 8, enters search text in the search character string input field 802, and presses a search start button 803.

  When the search start button is pressed, the flow shown in FIG. 3 is entered. In this flow, first, a search character string is fetched from the search character string input field 103 (S301). Next, this search character string is converted into a phoneme string using a language processing technique (S302). The converted phoneme string is stored in variable A.

  Next, the image / speech recognition result database in the apparatus is accessed, and one speech recognition result storage file is obtained therefrom. What is obtained at this time is set as a speech recognition result storage file α (S303). Next, the variable C_max is cleared with 0 (S304).

  Next, one speech recognition candidate is acquired from the speech recognition result storage file α acquired in S303. The speech recognition result candidate is information corresponding to each row of the speech recognition result storage file as shown in FIG. The recognition result phoneme string following “string =” of the voice recognition candidate acquired here is stored in the variable B (S305).

  Next, the similarity between the phoneme strings stored in the variables A and B is calculated (S306). At this time, the existing technology, dynamic programming, can be used. The calculated similarity is stored in the variable C. The stored C is compared with C_max (S307). If C is larger than C_max, C_max is updated with C (S308).

  These operations are performed on all the speech recognition candidates in one speech recognition result storage file α. When all the recognition candidates are completed (S309), C_max is set as the score of the speech recognition recognition file α (S310). .

  The above operation is performed on all the recognition result storage files in the image / speech recognition result database. When all the speech recognition result storage files are completed (S311), the files are sorted by the score calculated for each file, and the images corresponding to the top N are used as search result images. Further, an image related to the search result image is extracted from the image / sound database and presented to the user (S312).

  This related image is extracted by the following method. First, the image / sound database is arranged in the order of image data capturing date and time. Then, an image group from the search result image to an image having the next audio annotation data is set as a related image. For example, if the image / sound database shown in FIG. 102. For jpg images. jpg and 103. jpg is the related image. Similarly, 104. For jpg images, 105. jpg is the related image.

  For the sake of explanation, the example of FIG. 5 is returned again. Here, after the first photograph is taken, the user assigns a voice annotation “mountain of Hakone” to the image. In addition, two photographs were taken and no audio annotation was given to them. In this case, when a search is performed, the second and third pictures are presented simultaneously with the first picture (or in a form developed from the first picture). This operation is considered to be a relatively natural behavior from the viewpoint of the user.

  The search result image and the related image extracted based on the above idea are output by the presentation method as shown in FIG. In the search result presentation window A901, the searched images with voice annotations are presented in the order of search scores. Further, when the user places the mouse cursor 903 on the search result image, a related image of the image pops up and is presented.

  The search result presentation window B904 is an example in which a related image 905 is presented. The user can select a desired image from the related images popped up.

  The advantage of this embodiment is that it is possible to search even an image to which audio annotation data is not assigned. In addition, since an image without audio annotation data is not copied in the imaging device, the previous annotation data is copied (or linked information is added), so that the memory usage in the imaging device is increased. prevent.

(Example 2)
In the above-described embodiment, it is assumed that there is one portable image capturing apparatus that captures an image, but there may be a plurality of these. For example, even when one person performs imaging and voice annotation using a plurality of portable image pickup devices, it is possible to upload a common image / sound database. In that case, an image captured by a portable image capturing device different from the portable image capturing device to which the voice annotation is attached can be presented as a related image.

  In addition, it is possible to have images taken by other cameras not included in related images by having information on which portable image capture device was captured in the image or in the image / sound database. It is.

(Example 3)
In the above embodiment, it is not left in the image / sound database that an image is a related image of a specific image. However, it is also possible to explicitly hold information in the image / sound database which image is related to which image. For example, FIG. 11 shows an example of an image / sound database holding this information. Compared to that of FIG. 10, the element of related information increases, and the name of the image related to this part is held.

  This information can be created, for example, when an image is uploaded from the portable image pickup device to the PC. Further, this information may be registered when the voice recognition process is executed.

Example 4
In the above embodiment, when an image is regarded as a related image of an image, the difference in imaging time is not taken into consideration. However, it is unnatural to consider an image with a time difference as a related image. Thus, even in an image group captured from the time when an image A with an audio annotation is captured until the time when the next image with an audio annotation is captured, the imaging time is a fixed time from the imaging time of the image A. It is also possible to add a process that does not regard the image that has passed above as the related image of the image A.

(Example 5)
In the above embodiment, the processing when the image with audio annotation is deleted from the image / audio database is not touched. However, even when this image is deleted, the related information in the image / sound database can be automatically restored. For example, when related information is attached to this image, it can be considered that the related information is added to the related information of the immediately preceding data with the voice annotation to be deleted.

  For example, in FIG. Consider a case where an image of jpg (an image with audio annotation) is deleted. In this case, “105.jpg”, which is related information, is the previous data with voice annotation 101.jpg. What is necessary is just to add to the relevant information of jpg. In this example, 101. The related information of jpg is “102.jpg, 103.jpg, 104.jpg”.

  If the deleted image with voice annotation is the first one in the database, the related information may be deleted.

(Example 6)
In the above embodiment, an example in which audio annotation is added to image information has been described, but the present invention is not limited to this, and can be applied to various information such as document information and audio information. . Further, the present invention can also be applied when an annotation such as text is given instead of a voice annotation.

(Example 7)
An object of the present invention is to supply a storage medium recording a program code of software that realizes the functions of the above-described embodiments to a system or apparatus, and the computer (or CPU or MPU) of the system or apparatus stores the storage medium. Needless to say, this can also be achieved by reading and executing the program code stored in the.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention.

  As a storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) operating on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. It goes without saying that the CPU or the like provided in the board or the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

It is a functional block diagram of the information management apparatus of an Example. It is a flowchart of the speech recognition process part of the information management apparatus of an Example. It is a flowchart of the image search process part of the information management apparatus of an Example. It is an example of the portable imaging device of an Example. It is explanatory drawing which showed the relationship between a picked-up photograph and its related image. It is an example of the operation UI screen at the time of image upload of an Example. It is an example of the speech recognition result storage file of an Example. It is an example of the screen at the time of the image search of an Example. It is an example of presentation of an image search result of an example. It is an example of the image / sound database of the information management apparatus of an Example. It is an example of the image and audio | voice database of the information management apparatus of Examples 3-5.

Claims (14)

Search means for searching for the search target information based on the input search condition and annotation information associated with the search target information;
For each search target information searched by the search means, an extraction means for extracting related information from a storage means in which the search target information is stored based on time information associated with the search target information;
An information management apparatus comprising: a display control unit configured to control the search target information searched by the search unit and the related information extracted by the extraction unit to be related and presented to the user. The information is an image;
The extraction unit captures an image between the search target image and an image having imaging date and time information immediately after the search target image in an image associated with annotation information in the image stored in the storage unit. The information management apparatus according to claim 1, wherein an image having date information and not associated with the annotation information stored in the storage unit is extracted as a related image of the search target image. The information is an image;
The extraction unit extracts, as a related image, an image having a difference in imaging date and time between the search target image and the search target image within a predetermined time from the images stored in the storage unit. The information management apparatus according to claim 1.   The information management apparatus according to claim 1, wherein the extraction unit is performed after the operation of the search unit. A registration unit that registers the search target information and the annotation information in the storage unit;
The information management apparatus according to claim 1, wherein the extraction unit is performed after the operation of the registration unit. The annotation information includes voice annotation information,
The information management apparatus according to claim 1, further comprising voice recognition means for voice recognition of the voice annotation information and converting the voice annotation information into text information. A search process for searching for search target information based on the input search conditions and annotation information associated with the search target information;
For each search target information searched in the search step, an extraction step for extracting related information from a storage means in which the search target information is stored based on time information associated with the search target information;
An information management method, comprising: a search control step of controlling the search target information searched in the search step, the information extracted in the extraction step, and the related information to be related and presented to the user. The information is an image;
The extraction step includes imaging between the search target image and an image having imaging date and time information immediately after the search target image among images associated with annotation information in the image stored in the storage unit. The information management method according to claim 7, wherein an image having date information and not associated with the annotation information stored in the storage unit is extracted as a related image of the search target image.   In the extracting step, an image having a difference in imaging date and time between the search target image and the search target image within a predetermined time is extracted as a related image from the search target image and the image stored in the storage unit. The information management method according to claim 7.   The information management method according to claim 7, wherein the extraction step is performed after the operation of the search step. A registration step of registering the search target information and the annotation information in the storage unit;
The information management method according to claim 7, wherein the extraction step is performed after the operation of the registration step. The annotation information includes voice annotation information,
8. The information management method according to claim 7, further comprising a speech recognition step of recognizing the speech annotation information and converting the speech annotation information into text information.   A control program for causing a computer to execute the information processing method according to claim 7.   A computer-readable storage medium storing the control program according to claim 13.

Images