JP2016224608A - Character string input device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a character string input device that inputs a character string by detecting a vowel component of a character based upon a mouth shape and suppressing a burden on a user.SOLUTION: A character string input device comprises: imaging means 11 of imaging a mouth shape of a user P; vowel detecting means 12 of deriving, as a vowel component to be detected, a vowel component corresponding to the mouth shape found based upon a picked up image 23; character string selecting means 14 of selecting a character string starting with a character whose vowel component is the vowel component to be detected from a character database 13 in which a plurality of character strings are registered in advance; automatic scanning means 15 of specifying one of selected character strings as an inputtable item sequentially; display means 20 of displaying the inputtable item; input means 18 operated from the outside; and determining means 16 of determining a character string specified as the inputtable item as input information through an operation on the input means 18 from the outside.SELECTED DRAWING: Figure 1

Description

The present invention relates to a character string input device that detects characters and inputs a character string based on the shape of a user's mouth.

In a computer or a portable terminal, a keyboard, a mouse, and a button are mainstream as general interfaces for inputting characters.
On the other hand, with the development of science and technology, text is written using an interface based on handwritten character recognition technology that allows you to input text as if you were writing in everyday life, and speech, which is the most natural means of communication for humans. An input speech recognition interface has recently become available.

However, it is difficult for a person with a handicapped hand such as tremor (involuntary rhythmic movement caused by repeated muscle contraction and relaxation) to use a keyboard, a mouse, or a handwritten character recognition interface. In addition, in order to use the voice recognition interface, it is necessary to utter a voice, so that the content to be input is transmitted to the surroundings, lack of confidentiality, and the recognition accuracy decreases due to ambient noise and the like. Exists.

Accordingly, attention is paid to a technique for detecting characters corresponding to the mouth shape based on the mouth shape, and specific examples of apparatuses using the technology are described in Patent Documents 1 to 4.
Since the mouth shape naturally changes during utterance, it is possible to input the intended word naturally in the same way as speech recognition, and further, there is an advantage that the speech recognition problem can be solved because it is not necessary to utter speech.
Here, the mouth shape when the character is emitted has the property that it is the same as when the vowel component of the character is emitted (the vowel component of “K” is “A”). Therefore, “K” and “A” have the same mouth shape, and for example, a mouth shape that pronounces “Thank you” can be regarded as a mouth shape that pronounces “Aiou”.

The devices described in Patent Documents 1 to 4 utilize the property that the mouth shape that emits a character is the same as the mouth shape that emits the vowel component of the character, and based on the mouth shape, the vowel component of the character corresponding to the mouth shape is used. To detect. It has been reported that the accuracy of detecting the vowel component of a character based on the mouth shape is higher than the accuracy of detecting the character itself (for example, detecting “K” as “K”). Can stably detect characters from the mouth shape.

JP 2011-186994 A JP 2005-309952 A JP 2005-108079 A JP 2009-169464 A

However, the device of Patent Document 1 is assumed to be applied to a mobile phone, and it is necessary to operate a numeric keypad (10 keys) to input consonant characters, and a list of displayed candidate words When selecting a word from, the operation of the selection key and the confirmation key is required. Therefore, the apparatus of Patent Document 1 needs to operate a plurality of keys, which is heavy for a person with a handicapped hand.

The devices of Patent Documents 2 and 3 require key operations for each character when inputting consonant characters. For example, inputting “camera” requires at least three key operations. Therefore, the burden on the user is large in that the number of operations is large.
Moreover, after detecting the vowel component of the character corresponding to a mouth shape from the mouth shape, the apparatus of patent document 4 detects the movement of a user's head, and determines the character of the consonant to input. Therefore, in principle, it is necessary to move the head for each character, and the burden on the user is large.
The present invention is made in view of such circumstances, and an object of the present invention is to provide a character string input device that detects a vowel component of a character based on a mouth shape and inputs a character string while suppressing a burden on the user. To do.

A character string input device according to the present invention that meets the above-described object derives, as a detected vowel component, an imaging unit that captures a user's mouth shape, and a vowel component corresponding to the mouth shape obtained based on the captured image. A vowel detection means; a character string selection means for selecting the character string starting from a character whose vowel component is the detected vowel component; from the character string database in which a plurality of character strings are registered in advance; and the selected character string Sequentially, one (1) of the character string is designated as an inputable item, an auto-scanning means, a display means for displaying the inputable item, an input means operated from the outside, and the input enabled And a confirming means for confirming the character string designated as an input item as input information by externally operating the input means.

In the character string input device according to the present invention, the imaging unit intermittently captures the detected vowel component until the detected vowel component is derived, and the vowel detection unit is configured for each of the plurality of captured images. A feature amount of the mouth shape captured in the image is derived, and it is determined that the feature amounts corresponding to the plurality of images captured within a predetermined time are within a predetermined range, and It is preferable to derive the detected vowel component.

In the character string input device according to the present invention, the vowel detection means obtains a vowel component corresponding to the mouth shape based on the newly picked up image every time image pickup is intermittently performed, and the display means Each time a vowel component corresponding to the mouth shape is obtained, it is preferable to display the newly obtained vowel component corresponding to the mouth shape.

In the character string input device according to the present invention, the auto-scan means sequentially designates a mode switching item for causing the vowel detection means to start derivation of a new detected vowel component as the inputable item. The vowel detection means that derives the first detected vowel component is operated by operating the input means from outside in a state where the mode switching item is designated as the inputable item. The second detected vowel component is derived, and the character string selection means is the detected vowel component derived from the first and second vowel components of the first and second characters, respectively. It is preferable to select a certain character string.

In the character string input device according to the present invention, it is preferable that the display means also displays the total number of the character strings selected by the character string selection means.

In the character string input device according to the present invention, it is preferable that the character string selected from the character string database is determined for each user.

In the character string input device according to the present invention, a region corresponding to each vowel component is mapped, and further includes a plurality of mouth shape maps serving as a derivation reference of the detected vowel component by the vowel detection unit, The region corresponding to each vowel component is preferably adjusted for each user.

The character string input device according to the present invention derives (1) a vowel component corresponding to a mouth shape obtained based on a captured image as a detected vowel component, and (2) a plurality of character strings are registered in advance. A character string starting from a character whose vowel component is the detected vowel component is selected from the character string database. (3) For the selected character string, one of the character strings is sequentially designated as an inputable item. ) Since a character string designated as an item that can be input is determined as input information by operating the input means from the outside, one button (key) for determining the character string as input information on the input means The character string can be input by allowing the user to operate only one of the buttons. Therefore, the user does not need to operate a plurality of buttons to input the character string, and the user's burden for inputting the character string can be suppressed.

It is a block diagram of the character string input device which concerns on one embodiment of this invention. It is explanatory drawing of the content which the display means of the character string input device displays. (A)-(D) are explanatory drawings of a mouth shape map, respectively. It is explanatory drawing of the lip area | region extracted from the imaged image. It is a flowchart which shows the flow which produces a mouth shape map. It is a flowchart which shows a part of flow which determines input character string information. It is a flowchart which shows a part of flow which determines input character string information. It is a graph which shows the measurement result of input speed.

Next, embodiments of the present invention will be described with reference to the accompanying drawings for understanding of the present invention.
As shown in FIG. 1, a character string input device 10 according to an embodiment of the present invention has a mouth shape based on an image pickup unit 11 that picks up a mouth shape of a user P and an image 23 shown in FIG. A vowel detection unit 12 for obtaining a corresponding vowel component and deriving it as a detected vowel component; For the selected character string, auto-scan means 15 for sequentially specifying one (one) of the character string as an inputable item, and confirming that the character string specified for the inputable item is confirmed as input information And a means 16 for inputting a character string. Hereinafter, these will be described in detail.

As shown in FIG. 1, the character string input device 10 includes a computer (electronic computer) 17 and various hardware connected to the computer 17.
Various types of hardware connected to the computer 17 are an imaging unit 11 that is a camera, an input unit 18 that is an input device, a keyboard 19, a display unit 20 that is a display, and an output unit 21 that is a printer (printing device). .
In the present embodiment, the input unit 18 employs an input device with one operation button. However, an input device having a plurality of operation buttons may be used as long as the input device can be operated from the outside. . Further, one operation button on the keyboard may be handled as an input means.

The computer 17 includes a CPU, a hard disk, a memory, and a connection port, and the various hardware described above is connected to the connection port. The connection between the computer 17 and various hardware may be a wired connection or a wireless connection.
The vowel detection means 12, the character string selection means 14, the auto scan means 15, and the confirmation means 16 are software stored in the hard disk of the computer 17.

In the present embodiment, the image pickup unit 11, the input unit 18, the keyboard 19, and the display unit 20 are separate hardware from the computer 17, but may be integrated with the computer.
The character string input device 10 is designed on the assumption that the user P whose mouth is imaged by the imaging means 11 operates the input means 18 and the keyboard 19 while looking at the screen of the display means 20. . Therefore, the imaging unit 11 is arranged so as to capture the mouth of the user P who is looking at the screen of the display unit 20 (in the present embodiment, the entire face including the mouth).

As shown in FIGS. 1 and 2, the vowel detection unit 12 mainly uses a lip region of the user P (region surrounded by the contour of the lip) from an image 23 in which the imaging unit 11 images the face of the user P. 24 corresponds to the mouth shape of the user P (the shape of the contour of the lip) from the feature amount measuring unit 25 for obtaining the feature amount of the extracted lip region 24, and the feature amount of the lip region 24. It comprises a vowel discriminating unit 26 for obtaining a vowel component.

The hard disk of the computer 17 also stores a mouth shape map database 27 that can be accessed by the feature amount measurement unit 25 and the vowel discrimination unit 26, as shown in FIG. The mouth shape map database 27 stores a plurality of mouth shape maps 28 shown in FIGS. 3A to 3D in which the relationship between the lip region 24 and the vowel components corresponding to the lip region 24 is defined. A new mouth shape map 28 stored in the mouth shape map database 27 is created by the vowel detection means 12, and the created mouth shape map 28 is used as a reference for deriving the detected vowel component by the vowel detection means 12.

Each process of the vowel detection means 12 for the image 23 and the mouth shape map 28 taken by the image pickup means 11 is different from the processing common when creating the mouth shape map 28 and deriving the detected vowel component. Exists. Hereinafter, the common processing will be described.
As shown in FIG. 2, the image 23 captured by the imaging unit 11 includes parts other than the lip region 24 such as the eyes, eyebrows, and neck of the user P in addition to the lip region 24 of the user P. Yes.
The area extraction unit 22 acquires the image 23 captured by the imaging unit 11 and extracts the lip area 24 of the user P in the image 23. In the present embodiment, the region extraction unit 22 extracts the lip region 24 through two stages of processing.

First, the region extracting unit 22 extracts an entire region of the face of the user P from the image 23 using an algorithm devised by Viola and Jones (first stage processing), and for the extracted entire region of the face, Constrained local model (CLM) is applied to extract the lip region 24 of the user P together with the nose of the user P as shown in FIG. 4 (second stage process). CLM is one of the methods for detecting feature points by a statistical approach. In the present embodiment, in the first stage of processing, one feature point 24a is assigned to each of the left and right nostrils, and eight feature points 24a are assigned to the outline of the lip region 24. It is not limited to.

The feature amount measuring unit 25 derives the feature amount of the lip region 24 from the lip region 24 extracted by the region extracting unit 22. In the present embodiment, the feature amount measuring unit 25 derives the area of the lip region 24 as the first feature amount, and derives the aspect ratio of the lip region 24 as the second feature amount. The aspect ratio is a value obtained by dividing the vertical length of the lip region 24 by the horizontal width of the lip region 24.
In the present embodiment, two feature quantities are derived, but the present invention is not limited to this.

As shown in FIGS. 3A to 3D, the mouth map 28 is a two-dimensional map in which six areas are mapped. The five areas are “A”, “I”, “U”, “ E ”and“ O ”respectively correspond to the vowel components, and the remaining one region corresponds to the character“ n ”(for convenience, the character“ n ”will be described below as a vowel component). That is, the mouth map 28 is mapped with a region corresponding to each vowel component.
The vowel discrimination unit 26 determines the coordinates (position) of the lip region 24 in the mouth shape map 28 using the first and second feature values as the horizontal axis coordinate and the vertical axis coordinate on the mouth shape map 28, respectively. The coordinates of the lip region 24 are plotted on the map 28.

Next, processing when creating the mouth map 28 will be described.
As shown in FIG. 5, when the character string input device 10 logged in by the user P is in a standby state (step S1), the imaging unit 11 images the face of the user P (step S2). The area extraction unit 22 acquires the captured image 23 from the imaging unit 11 (step S3), and extracts the lip area 24 from the image 23 (step S4). The extracted mouth area 24 is displayed outside the display area of the image 23 on the screen of the display means 20, as shown in FIG. Then, as shown in FIG. 5, the feature amount measuring unit 25 measures the first and second feature amounts of the lip region 24 based on the extracted lip region 24 (step S5).

When the first and second feature values are measured, the display means 20 displays a mouth shape map 28 in which the coordinates of the lip region 24 are plotted according to the first and second feature values. At this time, a standard (default) mouth map 28 is displayed on the display means 20.
In this state, when the user P inputs a vowel component corresponding to his / her mouth shape by an operation (key input) from the keyboard 19 or the input means 18 (step S6), the coordinates of the lip region 24 are changed to the mouth shape map 28. The shape of each of the six areas corresponding to each vowel component on the mouth shape map 28 is adjusted so as to be arranged in the area corresponding to the input vowel component (step S7).

By repeating this cycle of step S1 to step S7 a plurality of times, a mouth shape map 28 corresponding to the user P can be created, and each mouth shape map 28 has a region corresponding to each vowel component. Adjustment is possible for each P. The mouth map 28 shown in FIGS. 3A to 3D is created for four users P, and the shapes of the six regions on the mouth map 28 are the same for each mouth map 28. It can be confirmed that they are different.
When it is detected by the operation from the keyboard 19 or the input means 18 that the adjustment of each area of the mouth map 28 has been completed (step S8), the mouth map 28 together with the identification information of the user P is stored in the mouth map map database. 27 (step S9).

Next, processing for obtaining a vowel component corresponding to the mouth shape of the user P and deriving the detected vowel component will be described.
As shown in FIG. 6, when the character string input device 10 logged in by the user P is in a standby state (step S1 ′), the imaging unit 11 images the face of the user P (step S2 ′). The area extracting unit 22 acquires the captured image 23 from the imaging unit 11 (step S3 ′), and extracts the lip area 24 from the image 23 (step S4 ′). Then, the feature amount measuring unit 25 derives the first and second feature amounts (that is, the feature amount of the mouth shape captured in the image 23) of the extracted lip region 24 (step S5 ′).

The vowel discrimination unit 26 selects a mouth shape map 28 corresponding to the user P in the mouth shape map database 27 based on the identification information of the user P, and the coordinates of the lip region 24 in the mouth shape map 28 are measured. By determining based on the first and second feature amounts, a vowel component corresponding to the mouth shape of the user P in the image 23 is obtained (step S6 ′).
Then, the display unit 20 displays the vowel component obtained by the vowel discrimination unit 26 on the vowel display unit 29 shown in FIG. 2 (step S7 ′). Further, the vowel discrimination unit 26 plots the coordinates of the lip region 24 determined in step S6 ′ on the lip map 28, and the display means 20 plots the coordinates of the lip region 24 as shown in FIG. A mouth map 28 is also displayed.

In the present embodiment, the imaging unit 11 captures the face of the user P intermittently (for example, 1 to 30 times per second), and the feature amount measurement unit 25 generates the image 23 by the imaging. Each time, the first and second feature quantities of the lip region 24 are derived from the image 23 and stored in the memory of the computer 17. The vowel discrimination unit 26 obtains a vowel component corresponding to the mouth shape of the user P every time the first and second feature quantities are newly derived. In other words, the vowel detection means 12 obtains a vowel component based on the newly picked up image 23 every time image pickup is performed intermittently.

And the vowel discrimination | determination part 26 is whether the 1st, 2nd feature-value corresponding to each of the several image 23 imaged within predetermined time (for example, 1-3 seconds) is settled in the predetermined range. Is determined (step S8 '). Specifically, for the image 23 captured within a predetermined time, (1) the coordinates of the lip region 24 on the mouth shape map 28 are all located within the same vowel component region, and (2) the mouth shape If the distance between the coordinates of the two most distant lip areas 24 on the map 28 is within a predetermined range, it is determined that the first and second feature values are within the predetermined range, and the others In this case, it is determined that the first and second feature amounts are not within the predetermined range.
Whether or not the coordinates of the lip region 24 on the mouth map 28 are all within the same vowel component region, whether or not the first and second feature values are within a predetermined range. It may be used as a criterion.

In step S8 ′, when it is determined that the first and second feature amounts corresponding to each of the plurality of images 23 captured within a predetermined time are within the predetermined range, the vowel determination unit 26 The vowel component obtained by itself is determined (derived) as the detected vowel component (step S9 ′).
On the other hand, when the vowel determination unit 26 determines that the first and second feature amounts corresponding to the plurality of images 23 captured within the predetermined time are not within the predetermined range, step S2 ′. To S8 ′ are performed again. Here, the detected vowel component is not fixed and the processing from step S2 ′ to step S8 ′ is repeated until the time during which intermittent imaging is performed by the imaging unit 11 reaches a predetermined time.

Every time the imaging means 11 intermittently captures the image 23 until the detected vowel component is determined, the region extraction unit 22 extracts the lip region 24 from the image 23 and whenever the lip region 24 is extracted. As shown in FIG. 2, the display unit 20 displays the lip region 24, and displays the newly obtained vowel component on the vowel display unit 29 every time the vowel discrimination unit 26 newly obtains the vowel component. .
The user P can confirm whether or not the extraction of the lip region 24 is appropriate by visually recognizing the lip region 24 displayed by the display means 20, and if the user P determines that the extraction of the lip region 24 is inappropriate, For example, the orientation of the face with respect to the imaging unit 11 can be adjusted. And the user P can judge whether the vowel component which he intended was calculated | required by visually recognizing the newly calculated | required vowel component displayed on the display means 20, and the unintended vowel component For example, it is possible to obtain the intended vowel component by changing its mouth shape.

Next, the character string input process based on the detected vowel component mainly performed by the character string database 13, the character string selection means 14, the auto scan means 15 and the confirmation means 16 will be described.
In the character string database 13, the user P can register in advance a plurality of character strings that the user P wants to input for each user P. In the present embodiment, the character string means a single word or a sentence composed of a plurality of words.

As shown in FIG. 1, the character string selection means 14 is designed to be accessible to the character string database 13, and based on the detected vowel component determined in step S9 ′ shown in FIG. Select a string.
After the detected vowel component is determined in step S9 ′, as shown in FIG. 6, the character string selection unit 14 inputs a character string whose vowel component starts from the character of the detected vowel component from the character string database 13. Is selected as a character string (step S10 ').

For example, if the detected vowel component is “A”, from words such as “Between”, “Tomorrow”, “Company”, “Position”, and multiple words such as “I will participate in tomorrow's meeting” Will be selected.
Here, in the character string database 13, each character string is stored with the identification information of the user P added thereto, and the character string selected as the input candidate character string from the character string database 13 is stored in the user P. Each is defined. Therefore, the user P can avoid selecting a character string that the user does not intend to input.
Then, as shown in FIG. 7, the display unit 20 displays the input candidate character strings selected by the character string selection unit 14 on the character string display unit 30 shown in FIG. 2 (step S11 ′).

The auto scan unit 15 sequentially selects one of the input candidate character strings selected by the character string selection unit 14 every predetermined time (1 to 3 seconds in the present embodiment). Specify one of the character strings as an inputable item. The order in which the input candidate character strings are designated as items that can be input is determined by the auto-scanning means 15 based on the past record as input character string information. That is, the auto-scanning means 15 has a learning function for determining the order of designating input candidate character strings as items that can be input based on past results.
The display means 20 highlights the input candidate character string designated as an inputable item on the character string display unit 30 (that is, displays the inputable item). Therefore, the user P can visually recognize the input candidate character string that is an inputable item.

As shown in FIG. 2, the character string display unit 30 is limited in the number of input candidate character strings that can be displayed at one time (10 in the present embodiment). When input candidate character strings exceeding the limit are selected, the display unit 20 can display all the input candidate character strings by repeatedly updating the character string display unit 30. When input candidate character strings exceeding the number that can be displayed at one time are selected in the character string display unit 30, the display means 20 updates the character string display update item 31 (in FIG. 2) for updating the character string display unit 30. , The item indicated as “next candidate”).

As the autoscan means 15 designates input candidate character strings in the character string display section 30 in order from the top to the bottom as items that can be entered, the display means 20 designates them as items that can be entered. The input candidate character string is highlighted. Then, after the character string of the input candidate displayed at the bottom is highlighted, the character string display update item 31 is designated as an item that can be input by the auto scan means 15 and is highlighted by the display means 20. .

As shown in FIG. 7, when the input candidate character string in the character string display unit 30 is highlighted, the button of the input unit 18 is pressed by the user P (the input unit 18 is operated). (Step S12 ′), the confirmation means 16 confirms the highlighted input candidate character string as input information (hereinafter also referred to as “input character string information”) and records it in the memory of the computer 17 (step S12 ′). 13 ′), when a process of confirming new input character string information is performed (step S14 ′), the process returns to step S2 ′. On the other hand, when the confirmation of the input character string information is finished in step S14 ′, the input character string information confirmed so far is arranged in the order of confirmation and output from the output means 21 (step S15 ′), and the process returns to step S1 ′. .

When the user P pushes the button of the input means 18 while the character string display update item 31 is highlighted (step S16 ′), the confirmation means 16 causes the display means 20 to display the character string display section. 30 is updated to display a new input candidate character string (step S17 '). After the character string display unit 30 is updated, highlighting of the newly displayed input candidate character string is started, and the process proceeds to step S12 '.

In addition, as shown in FIG. 2, the display means 20 includes a deletion item 32 (an item indicated as “deletion of word” in FIG. 2) for deleting the input character string information that has already been confirmed, In addition to the sound output item 33 (the item indicated as “speech output” in FIG. 2) for outputting the confirmed input character string information as a sound from the speaker of the computer 17, a new detected object is detected in the vowel detection means 12. A mode switching item 34 (an item indicated as “input of the next sound” in FIG. 2) to be set in a state in which derivation of a vowel component is started is also displayed.

When the character string display update item 31 is highlighted in step S16 ′ and the button of the input unit 18 is not pressed, the auto scan unit 15 designates the mode switching item 34 as an item that can be input (ie, The auto-scanning means 15 also designates the mode switching item 34 as an item that can be input sequentially), and the display means 20 highlights the mode switching item 34.
As shown in FIG. 7, when the mode switching item 34 is highlighted (that is, in a state where the mode switching item 34 is designated as an inputable item), the button of the input unit 18 is pressed by the user P. When pressed (step S18 ′), the process returns to step S2 ′, and the vowel detection means 12 passes from step S3 ′ to step S9 ′ as shown in FIG. A new detected vowel component is derived (when the first detected vowel component is already derived, the second detected vowel component is derived, and the Nth detected vowel component is already When it is derived, the (N + 1) th detected vowel component is derived).

When the second detected vowel component is derived by the vowel detection unit 12 that has already derived the first detected vowel component in step S9 ′, the character string selection unit 14 proceeds to step S10 ′. Then, the character string whose detected vowel component is derived from the first and second vowel components of the first and second characters, respectively, is selected from the character string database 13 as the input candidate character string. For example, if the first detected vowel component derived is “A” and the second detected vowel component is “Yes”, the first and second character vowel components are respectively Character strings such as “A”, “I”, “Ma”, “Company”, “Position”, etc. are selected.

When the N-th detected vowel component is derived in step S9 ′, the character string selection unit 14 determines the first, second,... Nth character in step S10 ′. A character string that is a detected vowel component derived from the first, second,..., Nth vowel components is selected from the character string database 13 as a character string as an input candidate. Therefore, the input candidate character strings to be selected are reduced (narrowed) by the increase in the derived detected vowel component.

On the other hand, if the button of the input unit 18 is not pressed while the mode switching item 34 is highlighted in step S18 ′, the autoscan unit 15 sequentially inputs the deletion item 32 and the sound output item 33. The display unit 20 highlights the deletion item 32 and the sound output item 33 sequentially at the timing when the item becomes an inputable item.
As shown in FIG. 7, when the button of the input means 18 is pressed while the deletion item 32 is highlighted (step S19 ′), the input candidate character string that has already been confirmed as the input character string information. Is deleted (step S20 ′), and the process returns to step S2 ′.

If the deletion item 32 is highlighted in step S19 'and the button of the input means 18 is not pressed, then the sound output item 33 is highlighted. If the button of the input means 18 is pressed while the sound output item 33 is highlighted (step S21 ′), the input character string information that has already been confirmed is output as a sound (step S22 ′), and step S14. Proceed to '. On the other hand, if the button of the input means 18 is not pressed while the sound output item 33 is highlighted in step S21 ′, the input candidate characters displayed at the top of the character string display section 30 are displayed. Highlighting from the column is resumed, and the process returns to step S12 ′.

The display unit 20 also displays the total number of input candidate character strings selected by the character string selection unit 14 by selecting input candidate character strings in step S10 '. Therefore, the user P visually recognizes the total number of character strings of the selected input candidates, and returns to step S2 ′ in step S18 ′ according to the number, thereby deriving a new detected vowel component, It is possible to determine whether to reduce the number of input candidate character strings to be selected.

Next, an experiment conducted for confirming the effect of the present invention will be described.
In the experiment, 400 words, which are the subjects of the national sign language certification test grade 5, were registered as character strings in the character string database. The vowel discrimination unit determines the detected vowel component on the condition that the first and second feature values corresponding to each of the plurality of images captured within 2 seconds are within a predetermined range. The auto-scanning means is set to sequentially switch items that can be input every second.

A text composed of two words shown in Table 1 created based on a character string registered in the character string database, a sentence composed of three words, and a sentence composed of four words are set as one set. The test subject (user) was input three sets of sentences, and the input speed per set was measured. The unit of the input speed is KPM (Kana Per Minute). For example, the sentence “buy camera” has five Kanas “ka”, “me”, “ra”, “ka”, “u”. Measured as containing (letters).

The average value which measured input speed with respect to 10 test subjects is shown in the graph of FIG. In the graph of FIG. 8, “1”, “2”, and “3” of “number of trials” on the horizontal axis indicate the measurement results of the first set, the second set, and the third set, respectively.
From the graph in Fig. 8, the average value of the input speed was 5.6 (KPM) in the first set, but it was 6.8 (KPM) in the third set. It was confirmed that the speed was improved. Note that the time for switching items that can be input by the auto-scanning means can be adjusted according to the subject, and it is conceivable that the input time can be shortened by shortening the time.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and all changes in conditions and the like that do not depart from the gist are within the scope of the present invention.
For example, it is not necessary for the imaging means to perform intermittent imaging until the detected vowel component is determined, and the detected vowel component may be determined by obtaining the vowel component based on one image. .
Further, instead of displaying the mode switching item on the display means, a button for causing the vowel detection means to start derivation of a new detected vowel component may be provided on the input means.
The display means does not necessarily display the number of character strings selected by the character string selection means.
Furthermore, a function for creating a mouth shape map in which a region corresponding to each vowel component is adjusted for each user and a function for determining a character string selected from the character string database for each user are not necessarily required.
Further, the display means may display the input candidate character string designated as an inputable item in a color different from other input candidate character strings, or may be designated as an inputable item. Only the input candidate character strings may be displayed.

10: Character string input device, 11: Imaging means, 12: Vowel detection means, 13: Character string database, 14: Character string selection means, 15: Auto scan means, 16: Determination means, 17: Computer, 18: Input means , 19: keyboard, 20: display means, 21: output means, 22: region extraction unit, 23: image, 24: lip region, 24a: feature point, 25: feature amount measurement unit, 26: vowel discrimination unit, 27: Mouth map database, 28: Mouth map, 29: Vowel display section, 30: Character string display section, 31: Character string display update item, 32: Delete item, 33: Sound output item, 34: Mode switching item, P: user

Claims (7)

Imaging means for imaging the mouth shape of the user;
Vowel detection means for deriving a vowel component corresponding to the mouth shape obtained based on the captured image as a detected vowel component;
A character string selection means for selecting the character string starting from a character whose vowel component is the detected vowel component, from a character string database in which a plurality of character strings are registered in advance;
For the selected character string, auto-scanning means for sequentially specifying one of the character strings as an inputable item,
Display means for displaying the inputable items;
Input means operated from the outside;
A character string input device comprising: a determining unit that determines the character string specified as the inputable item as input information by operating the input unit from the outside. 2. The character string input device according to claim 1, wherein the imaging unit intermittently captures the detected vowel component until the detected vowel component is derived, and the vowel detection unit performs each of the plurality of captured images. Deriving feature values of the mouth shape captured in the image, determining that the feature values corresponding to the plurality of images captured within a predetermined time are within a predetermined range, A character string input device for deriving a detected vowel component. 3. The character string input device according to claim 2, wherein the vowel detection means obtains a vowel component corresponding to the mouth shape based on the newly picked up image every time image pickup is intermittently performed, and the display means. Is a character string input device that displays a newly obtained vowel component corresponding to the mouth shape every time a vowel component corresponding to the mouth shape is obtained. The character string input device according to any one of claims 1 to 3, wherein the auto-scan unit includes a mode switching item for causing the vowel detection unit to start derivation of a new detected vowel component. The vowel detection means that sequentially designates the item that can be input and derives the first detected vowel component, in a state where the mode switching item is specified as the item that can be input, from the outside By operating the input means, the second detected vowel component is derived, and the character string selecting means has the first and second character vowel components as the first and second, respectively. A character string input device that selects the character string that is the derived detected vowel component. 5. The character string input device according to claim 1, wherein the display unit also displays a total number of the character strings selected by the character string selection unit. . The character string input device according to any one of claims 1 to 5, wherein the character string selected from the character string database is determined for each user. The character string input device according to any one of claims 1 to 6, wherein a plurality of mouth map is mapped to a region corresponding to each vowel component and serves as a reference for deriving the detected vowel component by the vowel detection unit. The character string input device further comprising: each mouth shape map, wherein the region corresponding to each vowel component is adjusted for each user.