JP4587165B2 - Information processing apparatus and control method thereof - Google Patents

Description

  The present invention relates to a data search method and apparatus.

  Digital images taken by a portable imaging device such as a digital camera can be managed by a personal computer (PC), a server computer, or the like. For example, captured images can be organized in a folder on a PC or server, a specific image can be printed, or incorporated as a part of a New Year's card picture. In addition, when managed by the server, some images can be disclosed to other users.

  When performing such work, it is necessary to find a specific image intended by the user. When the number of images to be found is small, the images can be displayed as thumbnails, and can be found visually from the list. However, when the number of target images is hundreds, or when the target image group is divided into a plurality of folders and stored, it is difficult to find it visually.

  On the other hand, a voice annotation (annotation by voice) is attached to an image on an imaging device, and the information is used at the time of retrieval. For example, an image of a mountain is taken, and “Hakone no Yama” is uttered on the image. This audio data is stored in the imaging device as a pair with the previous image data, and then voice recognition is performed in the image imaging device or in the PC to which the image is uploaded, and the text information is “Hakoneyama”. Converted. If the annotation data is converted into text information, it can be processed later by a general text search technique, and the image can be searched for using words such as “yama” and “hakone”.

  In addition, Patent Document 1 discloses a technique for registering a search key input by a user as a prior art related to the present invention. This document discloses a technique for registering a search key input by a user in the system as an arithmetic expression for an existing keyword using a synonym or the like.

JP-A-2-27479

  In the case of searching by converting voice annotations by voice recognition, it is the current situation that misrecognition of voice recognition cannot be completely avoided. If the rate of misrecognition is large, even if the search key is correctly inserted, the matching correlation is poor and the search is not performed correctly. In such a case, no matter how the search word is entered, there is a problem that the desired image data is not searched as a higher candidate because the result of the speech recognition is poor.

  Therefore, it is necessary to introduce a technique capable of improving the data search accuracy even if annotation generated with erroneous recognition of voice recognition is attached to the data to be searched.

In order to solve the above problem, for example, a method for controlling an information processing apparatus according to one aspect of the present invention includes the following configuration. That is, it has a database that stores a plurality of pieces of content data in association with annotation data representing annotations of each piece of content data for searching the plurality of pieces of content data. An information processing apparatus control method configured to input voice data representing an annotation and perform voice recognition on the voice data, wherein the input unit responds to an operation by a user, An input step for inputting a search key corresponding to a search condition, and the analysis means divides the search key into word strings by performing morphological analysis on the search key input in the input step, and each word string Analysis step for obtaining a first phoneme string and a calculation means are registered in the database. The notation data is subjected to phoneme matching between the second phoneme string constituting the annotation data and the first phoneme string obtained in the analysis step, whereby the first phoneme string of the second phoneme string is obtained. A calculation step for calculating a degree of correlation with respect to the phoneme string; and a display control step for causing the display control unit to display the plurality of content data on the display unit in an order ranked by the degree of correlation calculated in the calculation step; A receiving unit that receives an operation in which a user selects one or more content data from the plurality of content data displayed on the display unit; and a registration unit that includes the first step obtained in the analyzing step. Is associated with one or more content data selected by the user in the accepting step. And having a registration step of registering in the database.

  According to the present invention, it is possible to improve data search accuracy even when annotation generated with erroneous recognition of voice recognition is attached to data to be searched.

  DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1A is a diagram showing a functional configuration of a data search apparatus according to an embodiment of the present invention. In the figure, reference numeral 100 denotes a database. Reference numeral 101 denotes a plurality of search target data whose contents are images and documents stored in the database 100. The search target data 101 has a structure as shown in FIG. 1B, for example, and includes content data 102 such as an image and a document, and voice annotation data (voice memo data) 103 annotated by voice corresponding to the content data 102. Attached is speech recognition result annotation data 104 as annotation data including a subword string (described in this embodiment with a phoneme string) such as a phoneme string, a syllable string, and a word string obtained by voice recognition of the voice annotation data 103.

  Reference numeral 105 denotes a search key input unit that inputs a search key in order to search for desired content data 102. Reference numeral 106 denotes a search key conversion unit that converts a search key into a subword string having the same format as the speech recognition result annotation data 104 in order to perform matching with a check key. 107 matches with the plurality of speech recognition result annotation data 104 in the database 100 and the search key, obtains a correlation score for each speech recognition result annotation data 104, and content data corresponding to the speech recognition result annotation data 104 The search unit ranks 102. A display unit 108 displays the content data 102 ranked by the search unit 107 in the order of ranking. Reference numeral 109 denotes a user selection unit that selects data desired by the user from the content data 102 displayed on the display unit 108. Reference numeral 110 denotes an annotation registration unit that additionally registers the subword string of the search key in the speech recognition result annotation data 104 corresponding to the data selected by the user in the user selection unit 109.

  The functional configuration of the data search apparatus in the present embodiment is generally as described above. Further, the processing in this data search apparatus is performed sequentially from the upper block of FIG. That is, FIG. 1 also shows a processing flow by the data search apparatus. In the following, the flow of processing of the data search apparatus in this embodiment will be described in detail with reference to FIG.

  As described above, the search target data 101 that is content such as an image or a document includes the corresponding voice annotation data 103 and the voice recognition result annotation data 104 that is a result of voice recognition of the voice annotation data (see FIG. 1B). reference.). Here, the voice recognition result annotation data 104 may be created by providing a voice recognition unit in this apparatus, or a voice recognition unit provided on another apparatus such as a camera that captures an image. You may make it create using. In addition, since the speech recognition result annotation data 104 is used for data retrieval in the present invention, the speech annotation data 103 may not exist after the speech recognition result annotation data 104 is created.

  FIG. 2 shows an example of the voice recognition result annotation data 104. 201 in the figure is a recognition result phoneme sequence obtained by speech recognition of the speech annotation data 103 and converted into a phoneme sequence, and the highest N recognition ranks (N is an integer of 1 or more) having a high recognition score based on likelihood or the like. Are lined up.

  First, the user inputs a search key into the search key input unit 105. The input search key is transferred to the search key conversion unit 106, and the search key is converted into a phoneme string having the same format as the recognition result phoneme string 201.

  FIG. 3 shows how the search key is converted into a phoneme string. The search key “Hakone no Yama” is morphologically analyzed and divided into word strings. Further, reading is given to the word string to obtain a phoneme string. A known natural language processing technique can be applied to the morphological analysis and reading method.

  Next, in the search unit 107, phoneme matching is performed between the phoneme string of the search key and the speech recognition result annotation data 104 in each search target data 101, and the correct phoneme accuracy indicating the degree of correlation with the search key is obtained. The matching can be performed by a known DP matching method or the like.

  FIG. 4 shows how the phoneme accuracy is obtained. The phoneme accuracy is obtained by the following equation when the number of correct answers, the number of insertion errors, the number of deletion errors, and the number of replacement errors are obtained by DP matching or the like.

    {(Number of correct answers)-(Number of insertion errors)-(Number of deletion errors)-(Number of replacement errors)} × 100 / (Number of correct answers)

  In FIG. 4, there are two insertion errors, “o” and “a”, and one erroneous replacement error with “h” as “f”, and the phoneme correct answer accuracy is 75%. The content data 102 is ranked as the score used for the search with the correct phoneme accuracy obtained in this way. Here, the speech recognition result annotation data in FIG. 2 has the top N recognition result phoneme strings, but matching is performed with each of them to obtain the correct phoneme accuracy, and the best correct phoneme accuracy and recognition result phoneme string are adopted. To do. However, the present invention is not limited to this, and the maximum value may be taken or the sum may be taken after applying the weighting coefficient to the correct phoneme accuracy according to the order.

  Next, the display unit 108 displays the data in the order of search order. FIG. 5 shows an example in which the display unit 108 displays data (here, an image) on a window. In the figure, when a search key is input in the left frame of the window and the search button is pressed, the content data 102 is displayed in the search frame in the right frame of the window.

  Here, the user can select one or more data from the displayed data. As described above, since there is a possibility that a recognition error has occurred during speech recognition, the desired content data may not appear at the upper level, but may barely appear at the lower level. Therefore, in this embodiment, even if the search is not performed at the top, if the user selects here as the desired image, the search is surely performed at the top when the search is performed again with the same search key by the following processing. It becomes possible to do.

  That is, the user selection unit 109 selects data in accordance with a user selection operation. In response to this, the annotation registration unit 110 additionally registers the phoneme string obtained by converting the search key in the speech recognition result annotation data 104 corresponding to the selected data.

  This is shown in FIG. In the figure, reference numeral 601 denotes a pointer, which is used by the user to select one of the displayed data. As a data selection method at this time, any method may be used as long as it can specify an image. For example, the image clicked by the user may be selected as it is, or the user may be asked whether to select the image clicked by the user, and may be selected when OK is obtained from the user thereafter. . Reference numeral 602 denotes a phoneme string obtained by converting the search key, and this phoneme string is additionally registered in the speech recognition result annotation data 104 corresponding to the selected data. As a result, when the same search key is input again to perform a search, the phoneme accuracy shown in FIG. 4 is 100%, and the search can be performed in the first place or the order corresponding thereto, and a part of the same search key is input. In this case, the search accuracy can be improved when the search is performed using the partial matching technique.

  FIG. 7 shows an example of the hardware configuration of the data search apparatus according to this embodiment.

  In the figure, reference numeral 701 denotes a display device that displays data, GUI, and the like. Reference numeral 702 denotes a keyboard / mouse for inputting a search key or the like or pressing a GUI button. Reference numeral 703 denotes an audio output device including a speaker that outputs sounds such as audio annotation data and warning sounds. A ROM 704 holds a control program for realizing the database 100 and the data search method. Of course, an external storage device such as a hard disk may be provided separately, and the database 100 and a control program for realizing the data search method may be held therein. Reference numeral 705 denotes a RAM that functions as a main storage device, and particularly temporarily stores the program, data, and the like when the data search method program is executed. Reference numeral 706 denotes a CPU that controls the apparatus, and particularly executes a control program for realizing the data search method.

  In the above-described embodiment, the score based on matching using phonemes as subwords is used. However, the present invention is not limited to this. For example, a score based on syllables instead of phonemes, or a score based on word-by-word matching may be used. In addition, the likelihood of recognition obtained by speech recognition may be added to this, or the score may be weighted using the similarity between phonemes (eg, “p” and “t” are high in similarity).

  In the above embodiment, as shown in FIG. 4, the correct phoneme accuracy based on the entire phoneme string matching is used as a search score. You may search by partial matching. In this embodiment, for example, when the annotation “Hakone no Yama” is added to the speech recognition result annotation data, the search can be performed using “Hakone” and “Mountain” as search keys by partial matching.

  Further, although the speech recognition result annotation data 104 in the above embodiment has been described as data in which only the phoneme string 201 is arranged as shown in FIG. 2, other modes can be applied. For example, an attribute is assigned to each phoneme string to distinguish whether it is a phoneme string created by speech recognition or a phoneme string of a search key added by the annotation registration unit 110.

  FIG. 8 shows the speech recognition result annotation data 104 in this modification. In the figure, reference numeral 801 denotes an attribute in which each phoneme string is generated. Here, the attribute value “phonemeASR” represents a phoneme sequence created by phoneme sequence recognition type speech recognition, and “user” is a phoneme sequence added by the annotation registration unit 110 when the user selects data. It represents that. By using this, the display method can be switched depending on the phoneme string used at the time of search, or the phoneme string additionally registered by the annotation registration unit 110 can be deleted. Further, this attribute is not limited to this, and it may be possible to determine phoneme string recognition / word string recognition based on attribute values in voice recognition.

  In the speech recognition result annotation data 104 of the above embodiment, the top N recognition results are held as subword strings such as phonemes as shown in FIG. 2, but the present invention is not limited to this. A lattice (subword graph) composed of subwords may be output, and the correct phoneme accuracy may be obtained in each path from the start to the end of the lattice.

  FIG. 9 shows an example of a subword graph. In FIG. 9, reference numeral 901 denotes a node of the subword graph. In the example of FIG. 9, a node is formed for each phoneme. Reference numeral 902 denotes a link connecting nodes, which represents a chain of phonemes. In general, a speech recognition interval likelihood between nodes to which the link is connected is assigned to the link, and when this speech recognition interval likelihood is used, the top N phoneme string candidates are obtained by a method called A * search. Can be extracted. The phoneme correct answer accuracy is obtained by matching the candidate with the search key.

  In this case, when the annotation registration unit 110 adds a phoneme string, a necessary node may be added to the subword graph of FIG. 9, and a graph of a phoneme string generated by speech recognition as shown in FIG. And the graph of the phoneme sequence registered by the annotation registration unit 110 may be held separately. If the phoneme sequence added by the annotation registration unit 110 already exists in the path of the subword graph of FIG. 9, the speech recognition interval likelihood of the link 902 is changed so that the path is selected by A * search. It may be implemented as follows.

  In the above-described embodiment, the annotation registration unit 110 has been described as additionally registering the phoneme string of the search key in the speech recognition result annotation data 104. However, the present invention is not limited to this. Of the N phoneme strings, the Nth phoneme string (that is, the phoneme string having the lowest recognition score in the speech recognition result annotation data 104) may be replaced with the phoneme string of the search key.

(Other embodiments)
As mentioned above, although embodiment of this invention was explained in full detail, this invention may be applied to the system comprised from several apparatuses, and may be applied to the apparatus which consists of one apparatus.

  In the present invention, a software program that realizes the functions of the above-described embodiments is directly or remotely supplied to a system or apparatus, and the computer of the system or apparatus reads and executes the supplied program code. Is also achieved. In that case, as long as it has the function of a program, the form does not need to be a program.

  Therefore, in order to realize the functional processing of the present invention with a computer, the program code itself installed in the computer and the storage medium storing the program also constitute the present invention. In other words, the claims of the present invention include the computer program itself for realizing the functional processing of the present invention and a storage medium storing the program.

  In this case, the program may be in any form as long as it has a program function, such as an object code, a program executed by an interpreter, or script data supplied to the OS.

  As a storage medium for supplying the program, for example, flexible disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R).

  As another program supply method, a client computer browser is used to connect to an Internet homepage, and the computer program of the present invention itself or a compressed file including an automatic installation function is downloaded from the homepage to a storage medium such as a hard disk. Can also be supplied. It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the claims of the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. It is also possible to execute the encrypted program by using the key information and install the program on a computer.

  In addition to the functions of the above-described embodiments being realized by the computer executing the read program, the OS running on the computer based on the instruction of the program is a part of the actual processing. Alternatively, the functions of the above-described embodiment can be realized by performing all of the processes.

  Furthermore, after the program read from the storage medium is written to a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion board or The CPU or the like provided in the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

FIG. 1A is a diagram showing a functional configuration and a processing flow of a data search device according to an embodiment of the present invention. FIG. 1B is a diagram showing a structure example of search target data in the embodiment of the present invention. FIG. 2 is a diagram illustrating an example of speech recognition result annotation data according to the embodiment of the present invention. FIG. 3 is a diagram illustrating processing by the search key conversion unit in the embodiment of the present invention. FIG. 4 is a diagram illustrating phoneme matching processing by the search unit according to the embodiment of the present invention. FIG. 5 is a diagram showing a display example of search results by the display unit in the embodiment of the present invention. FIG. 6 is a diagram illustrating processing performed by the annotation changing unit according to the embodiment of this invention. FIG. 7 is a block diagram illustrating a hardware configuration example of the data search apparatus according to the embodiment of the present invention. FIG. 8 is a diagram showing a modification of the speech recognition result annotation data in the embodiment of the present invention. FIG. 9 is a diagram illustrating an example of a subword graph in the embodiment of the present invention. FIG. 10 is a diagram illustrating a modification of the phoneme string addition processing by the annotation changing unit according to the embodiment of the present invention.

Claims (4)

There is a database that stores a plurality of content data and annotation data representing annotations of the respective content data in association with each other for searching the plurality of content data, and each annotation data includes an annotation of the corresponding content data. An information processing apparatus control method configured to be generated by inputting voice data to be represented and performing voice recognition on the voice data,
An input step in which an input means inputs a search key corresponding to a search condition in response to an operation by a user;
Analyzing means for dividing the search key into word strings by performing morphological analysis on the search key input in the input step, and adding a reading to each word string to obtain a first phoneme string Steps,
For each annotation data registered in the database, the calculation means performs phoneme matching between the second phoneme string constituting the annotation data and the first phoneme string obtained in the analysis step, A calculation step of calculating a degree of correlation of the second phoneme string with respect to the first phoneme string;
A display control step, wherein the display control means displays the plurality of content data on the display unit in an order ranked by the correlation degree calculated in the calculation step;
An accepting step for accepting an operation in which a user selects one or more content data from the plurality of content data displayed on the display unit;
A registration step in which a registration unit registers the first phoneme string obtained in the analysis step in the database in association with one or more content data selected by the user in the reception step;
A method for controlling an information processing apparatus, comprising: There is a database that stores a plurality of content data and annotation data representing annotations of the respective content data in association with each other for searching the plurality of content data, and each annotation data includes an annotation of the corresponding content data. An information processing apparatus configured to be inputted by inputting voice data to be represented and performing voice recognition on the voice data,
An input means for inputting a search key corresponding to the search condition in response to an operation by the user;
Analyzing means for dividing the search key into word strings by performing morphological analysis on the search key input by the input means, giving a reading to each word string, and obtaining a first phoneme string;
For each annotation data registered in the database, the second phoneme string constituting the annotation data and the first phoneme string obtained by the analyzing unit are subjected to phoneme matching, thereby performing the second phoneme matching. Calculation means for calculating a correlation degree of the phoneme string with respect to the first phoneme string;
Display control means for displaying the plurality of content data on a display unit in an order ranked by the correlation calculated by the calculation means;
Accepting means for accepting an operation for the user to select one or more content data from the plurality of content data displayed on the display unit;
Registration means for registering the first phoneme sequence obtained by the analysis means in the database in association with one or more content data selected by the user;
An information processing apparatus comprising: There is a database that stores a plurality of content data and annotation data representing annotations of the respective content data in association with each other for searching the plurality of content data, and each annotation data includes an annotation of the corresponding content data. A program executed by an information processing apparatus configured to input voice data to be represented and perform voice recognition on the voice data, the information processing apparatus comprising:
An input means for inputting a search key corresponding to a search condition in response to an operation by a user;
Analysis means for dividing the search key into word strings by performing morphological analysis on the search key input by the input means, giving a reading to each word string, and obtaining a first phoneme string;
For each annotation data registered in the database, the second phoneme string constituting the annotation data and the first phoneme string obtained by the analyzing unit are subjected to phoneme matching, thereby performing the second phoneme matching. Calculation means for calculating a correlation degree of the phoneme string with respect to the first phoneme string;
Display control means for displaying the plurality of content data on a display unit in an order ranked by the correlation calculated by the calculation means;
Accepting means for accepting an operation for a user to select one or more content data from the plurality of content data displayed on the display unit;
Registration means for registering the first phoneme string obtained by the analysis means in the database in association with one or more content data selected by the user;
Program to function as. A computer-readable storage medium storing the program according to claim 3 .

Images