JPH0883092A - Information inputting device and method therefor - Google Patents

Abstract

PURPOSE: To easily input the contents of information with high precision by providing a means to recognize visual sense information such as inputted characters, graphics and pictures and a means to recognize auditory sense information such as inputted voice and music. CONSTITUTION: An input switching section 3 switches and selects either one of a visual information input section 1 which inputs hand written characters and symbols and a visual sense information reading section 2 which reads already generated characters and symbols and the visual sense information is read. The contents of the information are recognized by a visual sense information recognition section 5 and the recognition results are rearranged in the order of liklihood by a recognition result sorting section 6. Similarly, the contents of the auditory sense information inputted by an auditory sense input section 4 are recognized and evaluated by an auditory sense information recognition section 7 and the recognition resluts are rearranged in the order of liklihood by a sorting section 8. When there are plural candidates to which both recognition results are matched, they are outputted to a recognition result output section 9 in the order of their overall degree of liklihood. Thus, correct information source is generated by combining relatively rough hand written character inputs, already existing hand written documents and voice recognition.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention mainly relates to a technique for recognizing visual information such as characters, symbols, figures, images, videos and combinations thereof, and auditory information such as voice, acoustic signals, or combinations thereof. In particular, the present invention intends to provide a better information input means by improving the recognition accuracy and the efficiency as compared with the case of using both of them individually, by combining the both recognition techniques.

[0002]

2. Description of the Related Art With the recent development of information equipment, computers and portable terminals have been downsized, and information equipment without a keyboard has appeared. In these, the information input means is mainly performed through handwritten character recognition or the like. Further, even in an information device equipped with a keyboard, when input from the keyboard is not desirable in some situations, for example, input by an unskilled person or input in a dark place. In addition to character input, there are also scenes where it is desired to recognize and retrieve information contained in figures, images, videos, and the like.

Taking handwritten character recognition as an example, the performance is not always satisfactory even in the case of carefully recognized character recognition. From practicality,
It is desirable that the correct recognition input can be made even when inputting quickly, such as scribble, but in this case, the recognition accuracy becomes lower than that when writing carefully. Generally, when inputting and creating data mainly consisting of characters including these symbols as an information source, the basics are the method of handwriting directly on paper or tablet as described above, word processor, personal computer, etc. Input is performed by the key operation of. In this case, except for the input by the key operation, the recognition is performed by using the technology of character recognition and character reading by a machine, but the performance is improved, but it is not sufficient.

On the other hand, the technical level of recognizing the utterance contents of voice by a machine has been surely improved. But,
It is difficult to realize a recognition rate of 100% especially in a situation where input environmental conditions are different, as in the case of recognition of handwritten characters. This is based on the premise that humans first perform the work of inputting information that is a source of information, such as characters and voice. In this case, handwritten characters and uttered voice are in a calm state. In addition to the existence of variable factors such as differences in individual habits and characteristics, if there are further changes in the working environment and psychology at that time, the above-mentioned habits and characteristics will cause even greater fluctuations. This is because it is difficult to avoid this.

Incidentally, there is an idea to improve the recognition performance by inputting kana characters as an auxiliary means for voice recognition (Japanese Patent Laid-Open No. 58-123596, a voice recognition system using auxiliary information). The idea here is that in addition to inputting the number of syllables of the vocalized voice by the number of times the push button is turned on and off, and giving the five Japanese vowels with the five keys, there are many consonant sounds. It is intended to provide an auxiliary function of inputting with kana characters instead of preparing with keys. However, these are used only as auxiliary means for voice recognition, and are difficult to apply in a situation where input by handwritten characters is preferentially used.

[0006]

SUMMARY OF THE INVENTION An object of the present invention is to solve the above-mentioned problems, and to input information contents to be input with high precision and high accuracy, such as visual information and auditory information. To provide a device.

[0007]

In any of the devices and methods of the present invention, visual information input means for inputting visual information such as characters, figures, images, and visual information recognition for recognizing the input visual information. Means, an auditory information input means for inputting auditory information such as voice and music, and an auditory information recognition means for recognizing the input auditory information.

In the apparatus of the present invention, when the visual information input means is further prepared for input, this is detected, and when the auditory information input means is prepared for input, this is detected, and when one of the two input preparations is detected. Only the input means corresponding to this can input information, and when both input preparations are detected, both input means can input information. Regarding one of the recognition result candidates of the visual information recognition means and the auditory information recognition means, the other recognition means has a means for narrowing down the recognition target. For a recognition candidate in which both the recognition results of the visual information recognizing means and the auditory information recognizing means are the same, the likelihood obtained by combining the likelihoods is calculated and used as the likelihood for the candidate.

In the method of the present invention, one of the visual information input means and the auditory information input means is used first to input and recognize the information, and then the other input means is used to input and recognize the corresponding information. However, the recognition target is narrowed down according to the previous recognition result. Further, the information input by the visual information input means and the input by the auditory information input means are performed at the same time, and the same candidates of the both recognition results are combined as the likelihood and the recognition is performed using this. Further, a method of inputting one of the above first and a method of inputting at the same time are selectively used.

[0010]

[Operation] By using the information input device or the information input method according to the present invention, if there is a relatively rough handwritten character input or an already handwritten document, this is used as the basis of the information input source. It is possible to use a voice recognition technology together to create a correct information source. on the other hand,
When there is no such document as a visual information source, the audio signal as auditory information is used as the basis of the information input source, and this is used together with the recognition technology by handwritten character input to create the correct information source. Can be done easily.

That is, on the assumption that the recognition rate of 100% cannot be achieved by the current individual techniques, it is conceivable to write a character and give the reading of the character by voice. As a typical method, a plurality of recognition candidates are listed by character recognition, and a correct character is selected by speech recognition using them as target words. For example, say “Sound” while handwriting “Sound” and say “Sound”. There are many types of such basic inventions. For example, in the case of identifying a man and a woman by a video (moving image), if the voice of the person is obtained, the overall discrimination performance can be improved in combination with the gender discrimination by the voice.

That is, the present invention can provide a highly accurate information input means by combining the techniques for recognizing such visual information and auditory information.

[0013]

DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows a functional configuration of an embodiment according to the present invention. The visual information input unit 1 is a handwritten character or symbol,
The visual information reading unit 2 is a unit for inputting information such as an image, a figure, and a video, and the visual information reading unit 2 is a unit for reading already created information such as characters and symbols, an image, a graphic, and a video. One of the visual information input unit 1 and the visual information reading unit 2 is switched and selected by 3 to read the visual information. Further, an auditory information input unit 4 is provided for taking in a voice uttered by the speaker and other acoustic information via a microphone or the like. The content of the visual information input by the input switching unit 3 is recognized by the visual information recognition unit 5,
The recognition result sorting unit 6 sorts the results recognized by the visual information recognizing unit 5 in the order of likelihoods (scores, similarities, or distance values). Similarly, the auditory information input unit 4
The content of the acoustic information input in (1) is recognized and evaluated by the auditory information recognition unit 7, and the results recognized by the auditory information recognition unit 7 are rearranged in the order of likelihood (same as above) by the recognition result sorting unit 8. The recognition result output unit 9 outputs the recognition results of the visual information and the auditory information described above. If there is an error in the output of the recognized result, the correction point specifying unit 11 can specify the correction point. The result after the input information is corrected correctly is recorded and accumulated in the input information accumulating unit 12 such as a semiconductor memory or a magnetic disk.
The information correctly input in 3 is output to the outside. The operation control unit 14 controls the procedure of the operation of the apparatus, such as the activation of the operation of each unit, the operation sequence, and the control of the input switching unit 3.

An example of the appearance of this device is shown in FIG. Case 21
Has a substantially thin rectangular parallelepiped shape, and the upper surface 21a is the rear surface 2
It is a tapered surface that rises slightly as it approaches 1b, and a tapered surface that rises further near the back surface 21b. This portion is a display surface 22 for displaying the recognition result. The central portion of the upper surface that rises extremely gradually is the visual information input surface 20, which forms a part of the visual information input unit 1, and is a touch panel, a CRT display for inputting with a light pen, or the like. As a part of the visual information input unit 1, a pen 23 for executing input by handwriting of characters, symbols, and graphics on the surface 20 is provided on the upper surface 21.
It is removably arranged parallel to the front edge of a. Between the pen 23 and the visual information input surface 20, a photoelectric conversion unit 24 that forms a part of an OCR (optical character reader) that constitutes the visual information reading unit 2 is detachably arranged. A microphone 25 that constitutes the auditory information input unit 4 along with the photoelectric conversion unit 24 is provided so that it can be taken out as needed. A plurality of control keys 26 are provided between the display surface 22 and the visual information input surface 20, and the control keys 26 move the cursor position when inputting information, switch the input switching unit 3, confirm input information, and execute printing. Select and control the output of information and information.

A speaker 27 and an earphone terminal 28 for outputting the recognition result are provided at one end of the side surface of the housing 21, and a data terminal 29 for outputting the result to the outside is provided. The visual information input surface 20 such as a touch panel or a CRT display device and the display surface 22 may be integrated to have both functions. Next, an example of the operation of the device of the present invention shown in FIGS. 1 and 2 will be described. First, when inputting information such as characters from the visual information input unit 1, the operation control unit 14
Thus, the input switching unit 3 is set so that the data on the visual information input unit 1 side is input to the visual information recognition unit 5. In the visual information input unit 1, for example, a combination of a touch panel as the visual information input surface 20 and a pen 23 is used to manually input the information content to be input as data. If there is an information source to be input, the input switching unit 3 is switched to the visual information reading unit 2 side, and the content of the information source is changed to data by using the optical scanning information reading unit 24 such as OCR. Enter as. As a result, the output of the visual information input unit 1 or the visual information reading unit 2 is input to the visual information recognition unit 5. The visual information recognition unit 5 recognizes the input information content by comparing and comparing with the content of characters and symbols stored in advance as a standard pattern. At this time, when the number of recognized candidates is one, the candidate is output to the recognition result output unit 9. However, when there are a plurality of recognized candidates instead of one, the recognition result sorting unit 6 rearranges the candidates so that the candidates have the highest likelihood, and the results are recognized. Output to. The recognition result output unit 9 can be visually confirmed by the display surface 22 or can be auditorily confirmed by the speaker 27, the earphone 28, or the like.

Next, a case where the auditory information input unit 4 is used for inputting an information source will be described. In this case, the speaker voice
Input to the auditory information input unit 4 using a device such as a microphone that converts an acoustic signal into an electric signal. Auditory information input unit 4
Then, it is converted to a plurality of parameters that express the characteristics of auditory information such as voice, and is input to the auditory information recognition unit 7 as a time-series data group. For example, it recognizes what the content of the input voice information is, by contrasting with phonemes, words, or sentences and symbols. When the number of recognized candidates is one, the candidate is output to the recognition result output unit 9 as in the previous case. However, when there are a plurality of recognized candidates instead of one, the recognition result sorting unit 8 rearranges the candidates so that the candidates have the highest likelihood, and the results are recognized. Output to.

As described above, the recognition result of the input contents can be output by both the input of visual information such as handwritten characters and symbols and the input of auditory information by voice. If the content to be input as the information source is correctly recognized by any of the methods, the result is recorded in the storage unit 12 of the correction result without correction, and the information is output via the information output unit 13 as necessary. Then, the printer can obtain a hard copy or provide information to the other party via the communication line. However, as described at the beginning, even if the input information source uses any of the input methods of the visual information input unit 1, the visual information reading unit 2, and the auditory information input unit 4 described above, there is no need for correction and recognition is performed. It is rare to be done. Therefore, an example of a procedure for recognition and a method of correction and confirmation when there is an error in the recognition result will be described below.

As described above, in the present invention,
When inputting desired information, there are the following three input methods for the information source. One of them can be selected, and the error correction method is set accordingly. To be done. The first is a method of inputting using one of the visual information recognition technology and the visual information reading technology, and when the recognition result has an error, the auditory information recognition technology is used to correct the content to be a correct answer. The second is a method of inputting using the auditory information recognition technology first, and if there is an error in the recognition result, correct it using either visual information recognition technology or visual information reading technology to correct the content. The third is that the recognition is performed by using both the visual information recognition technology or the visual information reading technology and the auditory information recognition technology at the same time, and if there is an error in the recognition result, the visual information recognition technology or the visual information recognition technology is used. This is a method of making corrections using either information reading technology or auditory information recognition technology.

As described above, the priority of the technology applied at the time of inputting information is selected by selecting one from the three types of handwritten character input means, voice input means, or simultaneous use of both, and inputting information. By enabling the error correction procedure and setting the error correction procedure accordingly, the following advantages occur. One of the advantages is that it can respond depending on the surrounding environment in which the information source is input.
This is because, for example, when the surroundings are noisy, input by visual information is given priority and correction is made by auditory information such as voice when the noise is quiet, and conversely in a relatively quiet environment. It is possible to give priority to voice input in a scene where the hand is shaking, such as in a moving car, or in a scene where there is darkness, and it is possible to make corrections by handwriting in a timely or bright scene where there is no shaking in the wrong place.

An example of the flow of recognition in this case is shown in FIG.
You In FIG. 3A, the visual information is input (S ₁), The visual information
Recognize the information (S₂), The result is from 1st to nth
And output (S₃). This information depends on auditory information
The same input information is used to narrow down the recognition target candidates during recognition.
Enter auditory information about the source (S_Four), Its auditory information
Recognize n candidates among the candidates that are recognized by visual information
Shi (S_Five), And outputs the recognition result (S₆). This
Cooperates with both sea urchins to reduce recognition processing time and recognition accuracy.
Is expected to improve. FIG. 9B shows the procedure for inputting information described above.
The opposite is true, that is, first enter the visual information, then
Recognize this and input the visual information about the recognition candidate
recognize. The effect obtained is similar to that of the case A. However
However, the user is free to decide which recognition procedure should be given priority.
It is done by selecting the control key 26, but the surrounding environment when entering information
Set for those who can obtain highly accurate recognition results
can do.

[0021] The second advantage is that the recognition is performed by using both the recognition or reading technology of visual information such as handwritten characters and the voice recognition technology at the same time. Since the format is such that input is done aloud and read aloud, it is expected that the possibility of reducing errors in the input of the content itself when creating a document will increase.

A specific method in this case will be shown below. Example 1: Along with inputting information to be input by handwriting, the same content is spoken and input. Due to the cooperation between the two, the input accuracy is higher than the accuracy of the handwritten character recognition alone and the accuracy of the voice recognition alone. The following input methods are conceivable as examples of a combination of both handwriting input and voice input.

Concrete example 1): "Sagayama" is input by handwriting, and "Sagayama" is input by voice. In this case, the person's name is handwritten in Kanji and its reading is input by voice. Specific example 2): Input "voice" by handwriting and "oto, koe" by voice. In this case, the reading for each kanji character is input as a kun reading.

Concrete example 3): Input "knowledge" by handwriting and "gomben" by voice. In this case, the radical name of "knowledge" is input by voice. Concrete example 4): Input "Saga" by handwriting and "Yamahen, Sa" by voice. In this case, the radical name of the input character and the reading of the part suggesting the character are input.

Concrete example 5): "Machine" is input by handwriting and "Kikai" is spoken and input. In this case, the idiom containing the desired character is uttered and input. (Specific example 6): Handwriting “machine” and voice input “kikai, kihen, hata”. In this case, the voice is input with a plurality of contents including the specific examples 2) to 5).

Concrete Example 7): "◇" is input by handwriting, and "Hishigata" is input by voice. In this case, the input of a symbol is taught by voice. As in these examples, when the visual information is a single or multiple handwritten characters or handwritten symbols, the auditory information should read or read the visual information or the radical name or symbol name or the character to be input. You can enter information with words or keywords that suggest.

In Examples 2 to 5 below, when an image or video is used as an information source for recognition, voice is also used to improve the recognition accuracy. Example 2: Image recognition input of Sagayama's face and voice input "Sagayama". By cooperation of both, "Sagayama" can be input without inputting from the keyboard. The accuracy is
It is higher than the accuracy of image recognition alone and the accuracy of voice recognition alone.

Example 3: An image input is a reddish purple object, and "Akamurasaki" is input as voice. By collaborating with each other, the color name "magenta" can be entered accurately. The accuracy is higher than the accuracy of video color recognition alone and the accuracy of voice recognition alone. Example 4: The image recognition input is the face of a man and the voice of the man is also input. By the cooperation of the two, it is recognized that the object is a male, that is, the accuracy is higher than the accuracy of the image recognition alone and the accuracy of the speech recognition alone, which is used for the input of gender of the male and female.

Example 5: The handwritten figure input is a square, and "seikei" is input by voice. Due to the cooperation of both, the accuracy is higher than the accuracy of the image recognition alone and the accuracy of the voice recognition alone. Example 6: Accurate input is possible by cooperation between the character recognition input for reading a book and the voice for reading the content.
The accuracy is higher than the accuracy of the character recognition alone and the accuracy of the voice recognition alone.

Next, a process of extracting and confirming that the input contents are desired information contents will be described. In this case, the candidates from the upper rank to the nth rank which are recognized when the visual information recognition technology or the visual information reading technology and the auditory information recognition technology are used together are sorted by the recognition result sorting units 6 and 8. After that, the result is output to the recognition result output unit 9. At this time, if there are a plurality of sets of candidates that the output results of both the visual information recognition technology or the visual information reading technology and the auditory information recognition technology are the same, the visual information recognition technology or the visual information recognition technology is used. From the results recognized by both the reading technique and the auditory information recognition technique, first, the sum of the respective likelihoods (score, similarity or distance value) is obtained, and then the sum of the likelihoods (same as above) is arranged in descending order. Sorting is performed by the sorting unit 6 or 8 and the result is displayed on the display as visual information or is output to the recognition result output unit 9 as auditory information so that the user can confirm it.

In this case, after assigning a certain weighting coefficient to each likelihood value calculated when recognition is performed using both the visual information recognition technology or the visual information reading technology and the auditory information recognition technology, In some cases, it may be necessary to find the sum of the two. This is because, for example, the value of the recognition likelihood when using visual information is 800 to 5 for the first to fifth positions.
Whereas the range of 0 is about 0, while the value of the recognition likelihood when using auditory information is the same as before, the number of digits differs from 10 to 0.2 for the 1st to 5th places. It is necessary to weight likelihood values, and it is important to experimentally set appropriate values in advance in order to obtain accurate recognition results. Since it is difficult to experimentally determine various cases, for example, normalization may be performed so that both recognition likelihoods are on the same order.

Next, when the recognition result is output to the recognition result output unit 9 and a part of the content is incorrect, the correction position specifying unit 11 moves the cursor to the position to be corrected. In this case, as a means for identifying a correction location for performing an error correction, a pen 23 or a finger, which is a functional component used for character input, is used, or the cursor is moved in the up / down / left / right directions to locate the correction location. The function key 26 of the operation control section that can be specified is used.

Further, when the correction portion for error correction is a partial or partial correction of a character, the portion to be corrected is designated by the above-mentioned method and a candidate other than the already recognized character is held. In addition to outputting a plurality of candidates as candidates sequentially from the dictionary, the function of specifying the correct character from the candidates is given.

If the information to be input is in particular characters and the voice recognition technique is used, the notation is changed and registered in advance so that either phonetic reading or kun reading can be accepted. I will. These contents are as described above as the embodiment of the combination of the handwriting input and the voice input. As another example, a series of procedures from input to confirmation is shown below.

First, the input mode is switched by, for example, the pen 2
3 is picked up from the housing 21, the visual information input priority mode is set, while the information reading unit 24, for example, OC.
By taking the R part as the priority mode of the visual information reading unit, the number of function keys 26 can be reduced. If you give priority to input by auditory information,
It may be selected by the function key 26 described above. For voice input, specify a function key to specify the start and end timings of utterance, and use this to simultaneously input the information of the utterance section to realize more accurate recognition. Sometimes you can.

In the information input device having such an input operation function, when the pen 23 is picked up from the housing 21 and the character "recognize" is input by handwriting on the visual information input surface 20 such as a touch panel, The character recognition operation is performed with the handwriting information input priority state. At this time, if voice input is desired to be performed at the same time, the above-described key selection 26 may be performed by pressing a key for performing both handwriting information input and voice information input. When the input of information is completed, the recognized result is displayed on the display surface 22 or the speaker 27 or the like in the order of the likelihood value described above. If the result is that the "knowledge" part is wrong, such as "recognize", is "job", the pen 23 will specify the error part, and the second candidate for that part will be output. . At this point the correct "knowledge"
Is output, the correct input information can be confirmed and taken in by depressing, for example, the "confirm" key of the function keys 26 or by inputting a command in place of this.

An overall processing flow chart is shown in FIG. That is, it is constantly monitored whether or not input preparation is detected (S ₁ ), that is, any key 26 indicating visual information manual input, visual information read input, or auditory input is operated, or the pen 23
When any of the above, the information reading section 24, and the microphone 25 are taken out, the input preparation is detected. From this, it is checked whether it is a manual input, that is, a handwriting input (S ₂ ), and if it is not a handwriting input, it is checked whether it is a reading input (S ₃ ). Auditory information from the input unit 4 is captured (S ₄ ). Recognition of the captured auditory information is performed by the auditory information recognition unit 7 (S ₅ ), and the recognition result is output from the output unit 13 to check whether there is an error (S ₆ ).

If there is an error, the visual information input means, the visual information input section 1, or the visual information reading section 2 inputs the visual information corresponding to the auditory information input by the auditory information input section 4 (S ₇ ). . The input visual information is recognized by the visual information recognizing unit 5, and the recognition target is a certain number of candidates from the predetermined upper rank among the candidates recognized in the auditory information recognition step S _5. It is carried out (S _8). The recognition result is visually displayed or audibly output on the output unit 13 in descending order of likelihood (S ₇ ). In the recognition result candidates, is performed modified as necessary (S _10), then, is carried out by confirming operation by the correct input is, for example, key 26 (S _11).

On the other hand, when the visual information input preparation state is detected in step S ₂ or S ₃ , it is checked whether or not the auditory information input preparation state is set (S ₁₂ ). If it is not in the auditory information input preparation state, it is checked whether or not the flag is set (S ₁₃ ). That is, when the read input preparation is detected in step S ₃ , the flag is set to 1 and the process proceeds to step S ₁₂ . Move. If the flag is 1 in step S ₁₃ , the input switching unit 3 is switched to the visual information reading unit 2 (S ₁₅ ). When the flag is 0, the input switching section 3 is switched to the visual information input section 1. Then the visual information is captured (S _16), the visual information can be recognized on the visual information recognition unit 5 (S _17). It is checked whether or not the recognition result has an error (S ₁₈ ), and if there is no error, the confirmation output is made. If there is an error, the information corresponding to the visual information inputted by the auditory information input unit 4 is inputted (S _19). ). The input auditory information is recognized by the auditory information recognizing unit 7 (S ₂₀ ), but this recognition is performed only on a predetermined number of candidates from the highest likelihood in the recognition result at step S ₁₇ . The recognition result is output by the output unit 13 in the order of highest likelihood (S ₉ ).

When it is detected in step S ₁₂ that the preparation for inputting auditory information is detected, in this case, the auditory information and the visual information are simultaneously input, and the input visual information and the auditory information are captured. is (S _21), these visual information and auditory information are recognized, respectively (S _22). The combined likelihood is calculated for the same one of these two recognition result candidates (S ₂₃ ), and from the combined likelihood and the likelihood of the non-matching candidate, the one that is higher by a predetermined number is output from the output unit 13. It is output (S ₉ ).

When the visual information is input and the auditory information is not input, the auditory information input unit 4 receives the ambient noise or noise from the auditory information input unit 4 or detects the information input preparation so as not to malfunction. From the input to the auditory information recognition unit 7 or the processing of the auditory information recognition unit 7 is prohibited. Although the audio has been described as an example of the auditory information in the above description, the siren of the ambulance may be input as the auditory information in correspondence with the input of the visual information of the ambulance. Corresponding to the input of the visual information of the animal, various sounds such as the sound of the animal may be input as the auditory information may be used as the auditory information.

[0042]

As described above, in the case of inputting information such as characters and symbols with high precision, compared with the case where many key inputs are required when using a conventional word processor or personal computer, By using the information input device and the information input method according to the present invention, if there is a relatively rough handwritten character input or a document already handwritten, this is used as the basis of the information input source and this is used in combination with the voice recognition technology. It is possible to create a correct information source, and it is possible to easily and quickly input this with the minimum number of key operations, so even users who are unfamiliar with keyboard operation do not feel uncomfortable. , Information input can be realized.

[Brief description of drawings]

FIG. 1 is a block diagram showing an embodiment of an apparatus according to the present invention.

FIG. 2 is a perspective view showing an example of an external configuration of a device according to the present invention.

FIG. 3 is a flowchart showing a recognition procedure when visual information and auditory information are used.

FIG. 4 is a flowchart showing an example of a processing procedure of a method according to the present invention.

Claims (11)

[Claims] 1. A visual information input means for inputting visual information such as characters, symbols, figures and images, an auditory information input means for inputting auditory information such as voice and music, and an input by the visual information input means. The visual information recognition means for recognizing visual information, the auditory information recognition means for recognizing the auditory information input from the auditory information input means, the recognition result of the visual information recognition means, and the recognition result of the auditory information recognition means An information input device comprising: a recognition result output means for outputting. 2. A means for detecting the input preparation of the visual information input means, a means for detecting the input preparation of the auditory information input means, and one of the both input preparation detecting means when the input preparation is detected. 2. Only the corresponding input means is capable of inputting information, and when both of the input preparation detecting means detect the input preparation, a means for enabling the information input of the both input means is included. Information input device. 3. The information input device according to claim 1, further comprising means for performing a recognition process by another recognition means for only one recognition result candidate of said visual information recognition means and said auditory information recognition means. . 4. A means for comparing a candidate recognized by the visual information recognition means with a candidate recognized by the auditory information recognition means, and both of the same candidates when there are a plurality of matched candidates by the comparison. The information input device according to claim 1, further comprising: a unit that obtains a total likelihood from the likelihood and uses it as a likelihood of the candidate. 5. The visual information input means, visual information manual input means for inputting after visual information is created, visual information reading means for reading the created visual information, these visual information manual input means, and visual information. 5. The information input device according to claim 1, further comprising input switching means for supplying one of the information reading means and the inputted visual information to the visual information recognition means. 6. A visual information input means for inputting visual information such as characters, symbols, figures and images, an auditory information input means for inputting auditory information such as voice and music, and an input by the visual information input means. An information input method comprising visual information recognition means for recognizing visual information, and auditory information recognition means for recognizing auditory information input from the auditory information input means, comprising: the auditory information input means; One of the input information is input, the input information is recognized by the corresponding recognition means, and then the other input means is used to input the information corresponding to the input information, and the recognition means corresponding to the input information is used. An information input method characterized in that recognition candidates are obtained by narrowing down recognition target candidates according to the recognition result. 7. A visual information input means for inputting visual information such as characters, symbols, figures and images, an auditory information input means for inputting auditory information such as voice and music, and an input by the visual information input means. In an information input method comprising visual information recognition means for recognizing visual information, and auditory information recognition means input by the auditory information input means, while inputting information using the visual information input means,
Information corresponding to the above information is simultaneously input using the auditory information input means, the input information is recognized by the corresponding recognition means, and the likelihoods of the same candidates in the recognition result are recognized by these recognition means. An information input method characterized by obtaining likelihood and obtaining a recognition candidate. 8. Information is input using one of the visual information input means and the auditory information input means, the input information is recognized by a corresponding recognition means, and then the other input means corresponds to the input information. Information to be input and the recognition means corresponding to the input information narrows down the recognition target candidates according to the recognition result, and the recognition means obtains recognition candidates by using both of the input means at the same time. 8. The information input method according to claim 7, wherein the method is selectively used. 9. The correction of the obtained recognition candidate is performed by inputting auditory information when the recognition candidate is based on the recognition result of visual information, and by inputting the visual information when based on the recognition result of auditory information. 9. The information input method according to claim 6, wherein the information input method is performed. 10. When the visual information to be input is a single character or a plurality of characters or symbols, the aural reading or the instructive reading of the visual information as the auditory information to be input,
9. The information input method according to claim 6, which is a radical name, a symbol, or a word or a keyword that suggests the visual information. 11. The total likelihood is obtained by adding a weighting coefficient to one of the likelihood of a visual information recognition result and the likelihood of an auditory information recognition result, and then adding both. The information input method according to item 7 or 8.