JP2012118679A - Information processor, word discrimination device, screen display operation device, word registration device and method and program related to the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processor, a word discrimination device, a screen display operation device, a word registration device and their methods and programs, for processing information and discriminating a word only by a video image of a mouth of a user and enabling operating of screen display.SOLUTION: Word image extraction means 22 extracts a temporal change for a lip image extracted by lip image area extraction means 21 as a word image of one unit. Pattern comparison means 25 compares a word vocalization pattern obtained by extraction from the word image with a word dictionary 24 and specifies a word to be an object of the operation of a device.

Description

  The present invention relates to various information processing apparatuses such as a personal computer and a mobile phone, a word discriminating apparatus used therefor, a screen display operating apparatus for performing screen display operations, a word registering apparatus for performing word registration, and methods and programs thereof. About.

  In various information processing apparatuses such as personal computers and mobile phones, information related to various operations is displayed on a display, and input operations are often performed using a keyboard, an operation panel, or a pointing device such as a mouse.

  FIG. 25 shows the appearance of a mobile phone as an information processing apparatus according to the first related technology of the present invention. The cellular phone 200 is a foldable telephone in which a first casing 201 and a second casing 202 are foldably connected by a hinge mechanism 203. A display 205 is disposed in the first housing 201, and an operation unit 206 is disposed on a surface of the second housing 202 that faces the display 205 when folded.

  Taking this cellular phone 200 as an example, the user looks at the screen contents such as the menu screen displayed on the display 205 and performs key operations such as the enter key 206A, the direction key 206B, or the dial key 206C of the operation unit 206. Then, necessary processing such as transmission of e-mail and retrieval of information is executed. When a touch panel is attached to the display 205, various operations can be performed by pressing a desired display portion with a finger or a pen.

  By the way, it is assumed that such an operation is performed by the user's hand. Therefore, many information processing apparatuses are provided with an inconvenient interface for the handicapped person. Further, even for people who do not have any special obstacles in their hands, it is not preferable to use these information processing devices excessively because they place an excessive burden on the hands.

  In particular, in this type of information processing apparatus, when a user's desired item is selected from a menu screen, the menu is often classified into many layers. In such a case, it is necessary to repeat a manual operation such as pressing a key, which places a heavy burden on a handicapped person.

  Therefore, as a second related technique of the present invention, a personal computer input device has been proposed in which a mouse cursor is moved or clicked on a screen using a laser pointer and a switch operated by air (see, for example, Patent Document 1). In the first related technique, the movement of the mouse cursor is controlled by attaching a laser pointer to the user's head, irradiating a laser beam on the display, and detecting the position with a camera. The click is realized by setting a tube in the user's mouth and operating the switch by exhalation or inspiration.

  In this second related technique, it is necessary for the user to prepare a laser pointer, a camera for acquiring this image, and a special switch operated by air. Further, there is a problem that the user is deprived of the posture of the head, the viewpoint of the eyes, and the degree of freedom of the mouth while performing the input operation.

  Therefore, as a third related technique, a character input device using eye movement and mouth shape recognition has been proposed (see, for example, Patent Document 2). In the third related technique, image data of a user's video (face position, orientation, eye direction, etc.) and video (mouth shape) when a vowel is uttered are stored as a dictionary, and the user can The character of the line which is staring to input is selected.

Japanese Patent Laying-Open No. 2005-063101 (paragraph 0028, paragraph 0029, FIG. 1) JP 2002-269544 A (Claim 1, paragraphs 0007 to 0011, paragraph 0024, FIG. 1)

  In the third related technique, the movement of the mouth of the vowel of the character to be input is read by the camera, a button for selecting a line is displayed, and the character is discriminated together with the video staring at the button. Therefore, in order to implement the third related technique, the user follows the button of the line, finds the corresponding line, and utters only one character of the line while gazing at the line. There is a need. This is because if the word is spoken at a normal speed, it is impossible to trace the corresponding line one by one. For this reason, not only the input speed of characters is considerably reduced, but also the problem that eyes are tired arises because the viewpoint must be changed frequently.

  Therefore, in the third related technique, the direction of the line may be determined by key input, but this eliminates the operation of a handicapped person.

  Accordingly, an object of the present invention is to perform information processing and word discrimination only with a video of a user's mouth, enable an operation of screen display, an information processing device that performs word registration, a word discrimination device, a screen display operation device, The object is to provide a word registration device, a method thereof, and a program.

  In the present invention, (a) a display for visually displaying various types of information, (b) an imaging means for photographing at least an operator's mouth using the display, and (c) an operation obtained by the imaging means. Change discriminating means for discriminating temporal changes in the image of the person's mouth, and (d) a specific operation executing means for executing a predetermined specific operation associated with the discrimination result according to the discrimination result of the change discriminating means And the information processing apparatus.

  In the present invention, (a) lip image region extracting means for extracting a lip region image from a human face image to be recognized; and (b) a lip image region extracted by the lip image region extracting unit. A word image extraction means for extracting a series of temporal changes from the start to the end of the lip image change as a unit word image, and (c) closing upper and lower lips in the word image extracted by the word image extraction means. One unit of words utters the temporal changes in the vertical and horizontal distances passing through the center point of the opening of the lips when the seam at the time is arranged horizontally. Word utterance pattern recording means for recording as a word utterance pattern at the time of being recorded, and (d) a word utterance pattern previously recorded by the word utterance pattern recording means or equivalent means. A word dictionary associated with each word, and (e) a pattern for comparing the word utterance pattern to be recognized recorded by the word utterance pattern recording means with the word utterance pattern for each word in the word dictionary. And (f) a word discriminating unit for discriminating that the word corresponding to the word utterance pattern determined to be the best match as a result of the comparison by the pattern comparing unit is a word spoken by the person to be recognized as described above. Is provided with a word discrimination device.

  Further, in the present invention, (a) a lip image region extracting means for extracting a lip region image from a human face image to be recognized, and (b) a lip image region extracted by the lip image region extracting unit. A word image extracting means for extracting a series of temporal changes from the start to the end of the lip image as one unit of word image; and (c) timing at which the upper and lower lips in the word image extracted by the word image extracting means are closed. And (d) a vowel discrimination that discriminates each vowel constituting a word from the shape of the same lip opening that continues for a predetermined time or longer in the word image extracted by the word image extraction unit. Means, (e) the respective timings measured by the closing timing measuring means or the equivalent means, and the vowel discrimination means described above. Is a word dictionary prepared by associating each vowel combination determined by an equivalent means with a plurality of words in advance, and (f) the closing timing measurement in the word image extracted by the word image extracting means. A comparison means for comparing each timing measured by the means and a combination of each vowel constituting the word discriminated by the vowel discrimination means with the above combination in the word dictionary, and (g) a best match between the comparison means Then, the word discriminating device comprises word discriminating means for discriminating that the word corresponding to the above-described combination in the above-described word dictionary is the word uttered by the person to be recognized as described above.

  Furthermore, in the present invention, (a) a lip image region extracting means for extracting a lip region image from a human face image to be recognized, and (b) a lip image region extracted by the lip image region extracting unit. A word image extraction means for extracting a series of temporal changes from the start to the end of the lip image change as a unit word image, and (c) closing upper and lower lips in the word image extracted by the word image extraction means. One unit of words utters the temporal changes in the vertical and horizontal distances passing through the center point of the opening of the lips when the seam at the time is arranged horizontally. A word utterance pattern recording means for recording as a word utterance pattern at the time of being recorded, and (d) a word utterance pattern recorded in advance by the word utterance pattern recording means or an equivalent means. The word dictionary associated with each word and (e) the word utterance pattern to be recognized recorded by the word utterance pattern recording means are compared with the word utterance pattern for each word in the word dictionary. And (f) word discrimination means for discriminating that the word corresponding to the word utterance pattern determined to be the best match as a result of comparison by the pattern comparison means is the word spoken by the person to be recognized as described above. And (g) a display for displaying various types of information, and (h) a content operation unit for operating the display content displayed on the display with the operation content corresponding to the word determined by the word determination unit. A display operation device is provided.

  In the present invention, (a) lip image region extracting means for extracting a lip region image from a human face image to be recognized; and (b) a lip image region extracted by the lip image region extracting unit. A word image extraction means for extracting a series of temporal changes from the start to the end of the lip image change as a unit word image, and (c) upper and lower lips in the word image extracted by the word image extraction means are closed. A closed timing measuring means for measuring timing, and (d) a vowel for discriminating each vowel constituting a word from the shape of the same lip opening that lasts for a predetermined time or longer in the word image extracted by the word image extracting means. Discriminating means, (e) the respective timings measured by the closing timing measuring means described above or equivalent means, and the vowel discriminating means described above. Or a word dictionary prepared by associating each vowel combination determined by means equivalent to this with a plurality of words in advance, and (f) the closing timing described above in the word image extracted by the word image extracting means. A comparison means for comparing each timing measured by the measurement means and each vowel combination constituting the word discriminated by the vowel discrimination means with the above-mentioned combination in the word dictionary; A word discriminating means for discriminating that a word corresponding to the combination in the word dictionary determined to be a match is a word uttered by the person to be recognized as described above, and (h) a display for displaying various types of information (I) This display with the operation content corresponding to the word discriminated by the word discriminating means described above. And the content operating means for operating the display contents displayed on the upper screen display operation device comprises.

  Further, in the present invention, (b) lip image region extracting means for extracting a lip region image from a human face image to be recognized at the time of word registration, and (b) extraction by the lip image region extracting unit. Registration target word image extraction means for extracting as a word image when an image for registering a series of temporal changes from the start to the end of the lip image change in the lip image region, and (c) this registration target word image extraction In the word image extracted by the means, when the upper and lower lips are closed when the joints are arranged horizontally, the vertical and horizontal directions passing through the center point of the opening of the lips described above, respectively. A word utterance pattern recording means for recording a temporal change in distance as a word utterance pattern when a unit word is uttered; and (d) a word utterance pattern recording means or a hand equivalent thereto. And (e) a word utterance pattern to be registered in the word utterance pattern recorded by the word utterance pattern recording means in the word dictionary. Pattern comparison means for comparing with the word utterance pattern for each word, and (f) registering only unregistered words corresponding to the word utterance pattern determined to be not more than a predetermined value as a result of comparison by the pattern comparison means The word registration device includes a registration word permission / non-permission determining unit that enables the registration.

  Furthermore, in the present invention, (a) a lip image region extracting unit that extracts an image of a lip region from a face image of a person to be recognized at the time of word registration, and (b) the lip image region extracting unit Registration target word image extraction means for extracting as an image a word image when a series of chronological changes from the start to the end of the lip image region in the extracted lip image region is registered, and (c) this registration target word image A closing timing measuring means for measuring the timing at which the upper and lower lips are closed in the word image extracted by the extracting means; and (d) the same lip that continues for a predetermined time or longer in the word image extracted by the registration target word image extracting means. Vowel discriminating means for discriminating each vowel constituting the word from the shape of the opening, and (e) the closing timing measuring means described above or equivalent means Accordingly, a word dictionary prepared by previously associating each timing measured and a combination of each vowel determined by the above-described vowel determination unit or an equivalent unit with a plurality of words, and (f) the above-described registration target word Compare each timing measured by the closing timing measuring means in the word image extracted by the image extracting means and each vowel constituting the word determined by the vowel determining means with the combination in the word dictionary. And (g) a registered word propriety determining means for registering only unregistered words corresponding to the above-described combinations determined not to approximate more than a predetermined value as a result of comparison by the comparing means. A registration device is provided.

  In the present invention, (a) a lip image region extracting step for extracting a lip region image from a human face image to be recognized; and (b) a lip image region extracted in the lip image region extracting step. A word image extraction step for extracting a series of temporal changes from the start to the end of the lip image change as a unit word image, and (c) closing the upper and lower lips in the word image extracted by the word image extraction step. One unit of words utters the temporal changes in the vertical and horizontal distances passing through the center point of the opening of the lips when the seam at the time is arranged horizontally. A word utterance pattern recording step for recording as a word utterance pattern at the time of being recorded, and (d) a word utterance pattern to be recognized recorded in the word utterance pattern recording step. A pattern comparison step for comparing with the word utterance pattern for each word in the word dictionary registered in advance in association with each word in the word utterance pattern recording step; and (e) as a result of comparison by this pattern comparison step, The word discrimination method includes a word discrimination step of discriminating that a word corresponding to the discriminated word utterance pattern is a word uttered by the person to be recognized as described above.

  Further, in the present invention, (a) a lip image region extracting step for extracting a lip region image from a human face image to be recognized, and (b) a lip image region extracted by the lip image region extracting step. A word image extraction step for extracting a series of temporal changes from the start to the end of the lip image as one unit of word image; and (c) timing at which upper and lower lips are closed in the word image extracted by the word image extraction step. And (d) vowel discrimination that discriminates each vowel constituting a word from the shape of the same lip opening that lasts for a predetermined time or longer in the word image extracted by the word image extraction step. And (e) the closing timing measurement step in the word image extracted in the word image extraction step. Each timing determined in the above-mentioned step and each vowel combination constituting the word determined in the above-described vowel determination step are determined in each timing measured in the above-mentioned closed timing measurement step and each vowel determination step described above. A comparison step for comparing with a word dictionary registered in advance for each word in a combination of vowels, and (f) a word corresponding to the combination in the word dictionary determined to be the best match as a result of comparison in the comparison step. The word discrimination method includes a word discrimination step for discriminating a word uttered by a person to be recognized.

Furthermore, in the present invention, as a word discrimination program, (b) a lip image region extraction process for extracting a lip region image from a human face image to be recognized; A word image extraction process for extracting a series of temporal changes from the start to the end of the lip image change in the lip image region extracted by the extraction process as one unit of word image; and (c) a word extracted by the word image extraction process. A closing timing measurement process for measuring the timing at which the upper and lower lips in the image are closed, and (d) a word composed of the same lip opening shape that continues for a predetermined time or longer in the word image extracted by the word image extraction process. A vowel discrimination process for discriminating each vowel to be performed, and (e) the above-mentioned closed pattern in the word image extracted by the word image extraction process described above. The timings measured in the ming measurement process and the combinations of the vowels constituting the words determined in the vowel discrimination process are determined in the timings measured in the closed timing measurement process and the vowel discrimination process. A comparison process for comparing each vowel combination with a word dictionary registered in advance for each word, and (f) as a result of comparison by this comparison process, the word corresponding to the combination in the word dictionary determined to be the best match is And a word discrimination process for discriminating that the word is spoken by a person who is the object of recognition.

  As described above, according to the present invention, a camera (imaging device) often attached to an information processing apparatus such as a mobile phone or a small personal computer can be used without using a new device. It is possible to execute a predetermined operation based on a change in the image of the operator's mouth. In addition, even an information processing apparatus that does not include a camera (imaging device) can realize the present invention at low cost by a technique such as USB (Universal Serial Bus) connection.

  Furthermore, according to the present invention, not only the determination of vowels but also the measurement of the timing when the upper and lower lips in the word image are closed. As a result, it is possible to recognize words by combining vowels and lip sounds, and it is possible to increase the number of words registered as a dictionary to a level necessary for operation of the apparatus.

It is a claim corresponding | compatible figure of the information processing apparatus of this invention. It is a claim corresponding | compatible figure of the word discrimination device of this invention. It is another word discrimination device claim correspondence figure of the present invention. It is a claim corresponding | compatible figure of the screen display operation apparatus of this invention. It is a claim corresponding | compatible figure of the other screen display operation apparatus of this invention. It is a claim corresponding | compatible figure of the word registration apparatus of this invention. It is a claim corresponding | compatible figure of the other word registration apparatus of this invention. It is a claim correspondence diagram of the word discrimination method of the present invention. It is a claim corresponding | compatible figure of the other word discrimination | determination method of this invention. It is a claim corresponding | compatible figure of the word discrimination | determination program of this invention. It is a top view showing the structure of the mobile telephone as a screen display operation apparatus in embodiment of this invention. It is a block diagram showing the outline | summary of the circuit structure of the mobile telephone of this Embodiment. It is explanatory drawing which showed roughly the relationship between the Japanese vowel and the shape of the opening part of a lip. It is the flowchart showing the outline | summary of the process of the menu screen call of a mobile telephone and the word registration mode by voice in this Embodiment. It is explanatory drawing which showed the positional relationship of a mobile telephone and a user's face when the process of step S401 is performed in this Embodiment. It is explanatory drawing which showed an example of the content of utterance, an analysis result, and the registration content of a dictionary about the word utterance pattern registered into the word dictionary in this Embodiment. It is the top view which showed an example of the display content of a display in this Embodiment. It is the flowchart which showed the flow of the process in case a user operates the display content of a display by utterance in this Embodiment. FIG. 15 is a flowchart specifically showing the flow of word registration processing in step S404 of FIG. 14 in the present embodiment. It is explanatory drawing showing the mode of the change of an image when the user as a speaker utters a word one sound at a time in the 1st modification of the present invention. It is the top view which showed the example of a display of the display in a 1st modification. It is explanatory drawing showing the mode of the switching control of the active and non-active of the 1st and 2nd window in a 1st modification. It is explanatory drawing which showed selection of "tab 1"-"tab 4" as the 2nd example in this 1st modification. It is the perspective view which showed the screen display operation apparatus in the 2nd modification of this invention. It is a top view showing the external appearance of the mobile telephone as an information processing apparatus of the 1st related art of this invention.

  FIG. 1 is a diagram corresponding to claims of the information processing apparatus according to the present invention. The information processing apparatus 10 according to the present invention includes a display 11, an imaging unit 12, a change determination unit 13, and a specific operation execution unit 14. Here, the display 11 visually displays various information. The imaging unit 12 captures at least the mouth of the operator who uses the display 11. The change discriminating unit 13 discriminates a change with time of the image of the operator's mouth obtained by the imaging unit 12. The specific operation execution unit 14 executes a predetermined specific operation associated with the determination result according to the determination result of the change determination unit 13.

  FIG. 2 shows a claim correspondence diagram of the word discriminating apparatus of the present invention. The word discriminating apparatus 20 of the present invention includes a lip image region extracting unit 21, a word image extracting unit 22, a word utterance pattern recording unit 23, a word dictionary 24, a pattern comparing unit 25, and a word discriminating unit 26. Yes. Here, the lip image region extracting means 21 extracts a lip region image from the face image of the person to be recognized. The word image extraction unit 22 extracts a series of temporal changes from the start to the end of the lip image change in the lip image region extracted by the lip image region extraction unit 21 as one unit of word image. The word utterance pattern recording means 23 has the above-mentioned opening of the lips when the seam when the upper and lower lips are closed in the word image extracted by the word image extracting means 22 is arranged horizontally. The temporal changes in the vertical and horizontal distances passing through the center point are recorded as a word utterance pattern when one unit of word is uttered. The word dictionary 24 associates word utterance patterns previously recorded by the word utterance pattern recording means 23 or equivalent means with the words. The pattern comparison unit 25 compares the word utterance pattern to be recognized recorded by the word utterance pattern recording unit 23 with the word utterance pattern for each word in the word dictionary 24. The word discriminating unit 26 discriminates that the word corresponding to the word utterance pattern determined to be the best match by the pattern comparing unit 25 is the word uttered by the person to be recognized as described above.

  FIG. 3 shows a claim correspondence diagram of another word discriminating apparatus of the present invention. Another word discriminating apparatus 30 of the present invention includes a lip image region extracting unit 31, a word image extracting unit 32, a closing timing measuring unit 33, a vowel discriminating unit 34, a word dictionary 35, a comparing unit 36, a word A determination unit 37 is provided. Here, the lip image region extracting means 31 extracts a lip region image from the face image of the person to be recognized. The word image extraction unit 32 extracts a series of temporal changes from the start to the end of the lip image change in the lip image region extracted by the lip image region extraction unit 31 as a unit word image. The closing timing measuring unit 33 measures the timing at which the upper and lower lips in the word image extracted by the word image extracting unit 32 are closed. The vowel discrimination means 34 discriminates each vowel constituting a word from the shape of the same lip opening that continues for a predetermined time or longer in the word image extracted by the word image extraction means 32. The word dictionary 35 is prepared in advance for a plurality of words by combining each timing measured by the closing timing measuring means 33 or equivalent means with each vowel determined by the vowel discrimination means 34 or equivalent means. It is. The comparison means 36 uses the timing of the closed timing measurement means 33 in the word image extracted by the word image extraction means 32 and the combination of the vowels constituting the word determined by the vowel determination means 34 in the word dictionary 35. Compare with the combination. The word discriminating unit 37 discriminates that the word corresponding to the combination in the word dictionary 35 determined to be the best match as a result of the comparison by the comparing unit 36 is the word uttered by the person to be recognized.

  FIG. 4 is a diagram corresponding to the claims of the screen display operation device of the present invention. The screen display operation device 40 according to the present invention includes a lip image region extraction means 41, a word image extraction means 42, a word utterance pattern recording means 43, a word dictionary 44, a pattern comparison means 45, a word discrimination means 46, A display 47 and content operation means 48 are provided. Here, the lip image region extracting means 41 extracts a lip region image from the face image of the person to be recognized. The word image extracting unit 42 extracts a series of temporal changes from the start to the end of the lip image change in the lip image region extracted by the lip image region extracting unit 41 as one unit of word image. The word utterance pattern recording means 43 has the above-mentioned opening of the lips when the seam when the upper and lower lips are closed in the word image extracted by the word image extracting means 42 is arranged horizontally. The temporal changes in the vertical and horizontal distances passing through the center point are recorded as a word utterance pattern when one unit of word is uttered. The word dictionary 44 is a collection of a plurality of standard word utterance patterns previously recorded by the word utterance pattern recording means 43 or equivalent means. The pattern comparison means 45 compares the word utterance pattern to be recognized recorded by the word utterance pattern recording means 43 with the word utterance pattern for each word in the word dictionary 44. The word discriminating unit 46 discriminates that the word corresponding to the word utterance pattern determined to be the best match by the pattern comparing unit 45 is the word uttered by the person to be recognized as described above. The display 47 displays various information. The content operation unit 48 operates the display content displayed on the display 47 with the operation content corresponding to the word determined by the word determination unit 46.

  FIG. 5 is a diagram corresponding to a claim of another screen display operation device of the present invention. Another screen display operation device 50 according to the present invention includes a lip image region extraction means 51, a word image extraction means 52, a closing timing measurement means 53, a vowel discrimination means 54, a word dictionary 55, a comparison means 56, A word discriminating means 57, a display 58, and a content operating means 59 are provided. Here, the lip image region extracting means 51 extracts a lip region image from the face image of the person to be recognized. The word image extraction unit 52 extracts a series of temporal changes from the start to the end of the lip image change in the lip image region extracted by the lip image region extraction unit 51 as a unit word image. The closing timing measuring unit 53 measures the timing at which the upper and lower lips in the word image extracted by the word image extracting unit 52 are closed. The vowel discrimination means 54 discriminates each vowel constituting the word from the shape of the same lip opening that continues for a predetermined time or longer in the word image extracted by the word image extraction means 52. The word dictionary 55 is prepared for a plurality of words in advance by combining each timing measured by the closing timing measuring means 53 or equivalent means with each vowel determined by the vowel discrimination means 54 or equivalent means. It is. Comparing means 56 compares each timing combination measured by the closing timing measuring means 53 in the word image extracted by the word image extracting means 52 and each vowel constituting the word determined by the vowel determining means 54 with the word dictionary 55. To do. The word discriminating means 57 discriminates that the word in the word dictionary determined to be the best match by the comparing means 56 is the word spoken by the person to be recognized as described above. The display 58 displays various information. The content operation unit 59 operates the display content displayed on the display 58 with the operation content corresponding to the word determined by the word determination unit 57.

  FIG. 6 shows a claim correspondence diagram of the word registration device of the present invention. The word registration device 60 of the present invention includes a lip image region extraction unit 61, a registration target word image extraction unit 62, a word utterance pattern recording unit 63, a word dictionary 64, a pattern comparison unit 65, and a registered word availability determination unit. 66. Here, the lip image region extraction means 61 extracts a lip region image from the face image of the person to be recognized at the time of word registration. The registration target word image extraction unit 62 extracts an image for registering a series of temporal changes from the start to the end of the lip image change in the lip image region extracted by the lip image region extraction unit 61 as a word image in the case of speaking. . The word utterance pattern recording means 63 has the lip opening described above when the joints when the upper and lower lips are closed in the word image extracted by the registration target word image extraction means 62 are arranged horizontally. The temporal changes in the vertical and horizontal distances passing through the center point of the section are recorded as a word utterance pattern when one unit of word is uttered. The word dictionary 64 is a collection of standard words utterance patterns previously recorded by the word utterance pattern recording means 63 or equivalent means for a plurality of words. The pattern comparison means 65 compares the word utterance pattern to be registered in the word recorded by the word utterance pattern recording means 63 with the word utterance pattern for each word in the word dictionary 64. The registered word permission / non-permission determining means 66 can register only unregistered words that the pattern comparing means 65 has determined not to approximate more than a predetermined value.

  FIG. 7 shows a claim correspondence diagram of another word registration apparatus of the present invention. Another word registration device 70 of the present invention includes a lip image region extraction unit 71, a registration target word image extraction unit 72, a closing timing measurement unit 73, a vowel discrimination unit 74, a word dictionary 75, and a comparison unit 76. , A registered word availability determination means 77 is provided. Here, the lip image region extracting means 71 extracts a lip region image from a human face image to be recognized at the time of word registration. The registration target word image extraction unit 72 extracts an image for registering a series of temporal changes from the start to the end of the lip image change in the lip image region extracted by the lip image region extraction unit 71 as a word image in the case of speaking. . The closing timing measuring unit 73 measures the timing at which the upper and lower lips in the word image extracted by the registration target word image extracting unit 72 are closed. The vowel discriminating means 74 discriminates each vowel constituting the word from the shape of the same lip opening that continues for a predetermined time or longer in the word image extracted by the registration target word image extracting means 72. The word dictionary 75 is prepared for a plurality of words in advance by combining each timing measured by the closing timing measuring means 73 or equivalent means with each vowel determined by the vowel discrimination means 74 or equivalent means. It is. The comparison means 76 uses the word dictionary 75 to indicate combinations of the vowels constituting the words determined by the closing timing measurement means 73 and the vowel discrimination means 74 in the word image extracted by the registration target word image extraction means 72. Compare with The registered word permission / non-permission determining unit 77 can register only unregistered words determined by the comparing unit 76 as not to be approximated by a predetermined value or more.

  FIG. 8 shows a claim correspondence diagram of the word discrimination method of the present invention. The word discrimination method 80 of the present invention includes a lip image region extraction step 81, a word image extraction step 82, a word utterance pattern recording step 83, a pattern comparison step 84, and a word discrimination step 85. Here, in the lip image region extraction step 81, a lip region image is extracted from the face image of the person to be recognized. In the word image extraction step 82, a series of temporal changes from the start to the end of the lip image change in the lip image region extracted in the lip image region extraction step 81 is extracted as one unit of word image. In the word utterance pattern recording step 83, the opening of the lip when the upper and lower lips of the word image extracted by the word image extraction step 82 are horizontally arranged as a whole is arranged horizontally. The temporal changes in the vertical and horizontal distances passing through the center point are recorded as a word utterance pattern when one unit of word is uttered. In the pattern comparison step 84, the word utterance pattern to be recognized recorded in the word utterance pattern recording step 83 is registered for each word in the word dictionary registered in advance in association with each word in the word utterance pattern recording step 83. Compare with word utterance pattern. In the word determination step 85, it is determined that the word corresponding to the word utterance pattern determined to be the best match in the pattern comparison step 84 is the word uttered by the person to be recognized as described above.

  FIG. 9 shows a claim correspondence diagram of another word discrimination method of the present invention. Another word discrimination method 90 of the present invention includes a lip image region extraction step 91, a word image extraction step 92, a closing timing measurement step 93, a vowel discrimination step 94, a comparison step 95, and a word discrimination step 96. ing. Here, in the lip image region extracting step 91, a lip region image is extracted from the face image of the person to be recognized. In the word image extraction step 92, a series of temporal changes from the start to the end of the lip image change in the lip image region extracted in the lip image region extraction step 91 is extracted as one unit of word image. In the closing timing measurement step 93, the timing at which the upper and lower lips in the word image extracted by the word image extraction step 92 are closed is measured. In the vowel discrimination step 94, each vowel constituting the word is discriminated from the shape of the same lip opening that lasts for a predetermined time or longer in the word image extracted in the word image extraction step 92. In the comparison step 95, a combination of each timing measured in the closing timing measuring step 93 in the word image extracted in the word image extracting step 92 and each vowel constituting the word determined in the vowel determining step 94 is used as a closing timing measuring step. The combination of each timing measured in 93 and each vowel determined in the vowel determination step 94 is compared with the above-described combination in the word dictionary registered in advance for each word. In the word determination step 96, it is determined that the word in the word dictionary determined to be the best match in the comparison step 95 is the word spoken by the person to be recognized as described above.

  FIG. 10 shows a claim correspondence diagram of the word discrimination program of the present invention. The word discrimination program 100 of the present invention includes a lip image region extraction process 101, a word image extraction process 102, a closing timing measurement process 103, a vowel discrimination process 104, a comparison process 105, and a word discrimination process 106 on a computer. I am trying to execute it. Here, in the lip image region extraction processing 101, a lip region image is extracted from the face image of the person to be recognized. In the word image extraction process 102, a series of temporal changes from the start to the end of the lip image change in the lip image area extracted by the lip image area extraction process 101 is extracted as one unit of word image. In the closing timing measurement process 103, the timing at which the upper and lower lips are closed in the word image extracted by the word image extraction process 102 is measured. In the vowel discrimination processing 104, each vowel constituting a word is discriminated from the shape of the same lip opening that continues for a predetermined time or longer in the word image extracted by the word image extraction processing. In the comparison process 105, a combination of the timing measured in the closing timing measurement process 103 in the word image extracted by the word image extraction process 102 and the vowel constituting the word determined in the vowel discrimination process 104 is used as the closed timing measurement process. The combination of each timing measured in 103 and each vowel determined in the vowel determination processing 104 is compared with the above combination in the word dictionary registered in advance for each word. In the word discrimination processing 106, it is discriminated that the word in the word dictionary determined to be the best match in the comparison processing 105 is a word uttered by the person to be recognized as described above.

  <Embodiment of the Invention>

  Next, an embodiment of the present invention will be described.

  FIG. 11 shows a configuration of a mobile phone as a screen display operation device according to the embodiment of the present invention. The mobile phone 300 is a foldable telephone in which a first housing 301 and a second housing 302 are connected to each other by a hinge mechanism 303 so as to be foldable. A display 305 is disposed at the center of the first housing 301, and an imaging device 306 is disposed at the upper right thereof. An operation unit 307 is disposed on a surface of the second housing 302 that faces the display 305 when folded. Various keys such as an enter key 307A, a direction key 307B, and a dial key 307C are arranged on the operation unit 307.

  FIG. 12 shows an outline of the circuit configuration of the mobile phone according to the present embodiment. The cellular phone 300 includes a main control unit 323 including a CPU (Central Processing Unit) 321 and a memory 322 that stores a program executed by the CPU 321. The main control unit 323 is connected to each unit of the mobile phone 300 through a bus 324 such as a data bus, and performs these controls.

  Among these, the communication control unit 325 controls communication with a base station (not shown). The imaging device 306 captures still images and moving images. The display control device 326 controls display on the display 305. The operation unit 307 performs various operations of the mobile phone 300 by key input. In the mobile phone of this embodiment, an input operation using a lip image can be performed in association with the display content of the display 305.

  The word dictionary 327 is a collection of standard words utterance patterns for a plurality of words. Here, the word utterance pattern is a lip opening when the user speaks a word consisting of a plurality of sounds and the joints when the upper and lower lips are closed are arranged horizontally. Is a pattern of temporal changes in the distance in the vertical and horizontal directions passing through the center point. The word discriminating unit 328 discriminates the word utterance pattern for the word used when the user performs various operations by utterance by comparing with the standard word utterance pattern registered in the dictionary. The image memory 329 stores image data obtained from the imaging device 306. The lip image region extraction unit 330 extracts a lip image from the user's face image. For detection of lips, for example, a dynamic contour model (SNAKES) can be used. Also, lips can be detected by identifying a face and discriminating a region that is redr than the surrounding area.

  The word image extraction unit 331 extracts a series of temporal changes from the start to the end of the lip image change when the user utters a word from the image memory 329 as one unit of word image. The closing timing measuring unit 332 measures the closing timing of the upper and lower lips when the user utters a word. The vowel discriminating unit 333 discriminates each vowel constituting the word from the shape of the same lip opening that continues for a predetermined time or longer in the word image extracted by the word image extracting unit 331. The general function unit 334 is a part that summarizes the functions provided in the mobile phone 300 according to the present embodiment as general functions as a mobile phone. For example, when the mobile phone 300 of this embodiment has a television reception function and an electronic payment function, these functions exist in the general function unit 334.

  At least a part of the mobile phone 300 other than the main control unit 323 can be realized by software by the CPU 321 executing a program stored in the memory 322.

  FIG. 13 roughly shows the relationship between the Japanese vowels and the shape of the lip opening. A vowel is a voiced sound with vocal cord tremors, and differs from a consonant in that the same sound is emitted continuously for a certain period of time. In the case of Japanese, the vowels consist of “A”, “I”, “U”, “E”, and “O” sounds.

  Individual vowels can be identified in appearance by a combination of the size of the mouth and the shape of the mouth opening and the positions of the front and back of the tongue. However, it is difficult to specify the front and back positions of the tongue using the imaging device 306. Therefore, in the present embodiment, when the shape of the mouth opening continues for a predetermined time or longer when the user utters a word, from this shape, “A”, “I”, “U”, “E” It is estimated that one of the vowels "" and "o" has been uttered.

  Here, the sound of “a” is obtained when two fingers are vertically moved through the center point of the opening of the lip when it is assumed that the joint when the upper and lower lips are closed is arranged horizontally. It is emitted with the mouth open enough to enter. The sound of “I” is emitted with the mouth open enough for the tip of the little finger to enter. The sound of “D” is emitted with the mouth open to the middle of the sound of “A” and “I”. When you make these sounds, your mouth is not shrugged.

  Each sound of “U” and “O” is uttered with the mouth closed. Among these, the sound of “U” is slightly smaller in the opening of the lips than the sound of “O”. Left and right passing through the center point of the lip opening when assuming that the joints when the upper and lower lips are closed in the sounds of “a”, “d”, and “b” are arranged horizontally. The distance in the direction is longer than when the “U” and “O” sounds are uttered.

  Therefore, in this embodiment, each vowel that constitutes a word from the shape of the same lip opening that continues for a predetermined time or longer in the word image extracted by the word image extraction unit 331 by the vowel discrimination unit 333 shown in FIG. Is determined. Of course, there are individual differences in the size of the lips, and the relationship between the distance in the vertical and horizontal directions of the opening and each vowel is relative.

  Since the vowel sounds are continuously generated, for example, when the user utters the sound “I” and when the user utters the continuous sound “I”, the shape of the opening of the lips does not change, and the same opening The only difference is the time that the shape is maintained. Therefore, in the present embodiment, it is used for vowel discrimination that the shape of the lip opening continues for a predetermined time or more, and the same vowel is repeatedly uttered every time a specific time exceeding the predetermined time elapses. Determine. As a result, it is possible to determine whether the user has uttered the sound of “I” in a single shot or “I” in multiple shots, such as “I”.

  There are exceptions to this. “N” can be uttered while retaining the sounds “A”, “I”, “U”, “D”, “O” and the shape of the lips. In preparation for the next vowel to be uttered, the sound “N” can be uttered by changing the shape of the opening of the lips in advance. Therefore, even though the continuous sound “I” and the word “In” can be distinguished from the sound “I”, they may not be distinguished by the principle of the present invention. This is because there is a possibility that when the word “ink” is uttered, it is impossible to specify whether the vowel is used to identify “Iu” or “Iu”. Of course, it is also possible to restrict the word registration target from not including “n”.

  By the way, when trying to operate the mobile phone 300 based on the difference of words uttered by the user, if the difference between the individual words is determined only from the vowels, for example, the word “OK” and “Migi (right)”. Are both indistinguishable words consisting of vowels “I” and “I”. That is, the number and types of words registered for operation are greatly limited.

  Therefore, in the present embodiment, the closing timing measurement unit 332 is used to measure the closing timing of the upper and lower lips when the user utters a word. This uses the feature that the upper and lower lips are always closed once in Japanese, when the voices of both the line, line and line (both lip sounds) are uttered. For example, when the word “OK” is uttered, the state of “I” shown in the upper right of FIG. On the other hand, when the word “migi (right)” is uttered, the mouth is completely closed once, and then the state of “I” shown in the upper right of FIG. The state of “I” shown in the upper right occurs.

  When the word “migi (right)” is uttered, the number of times the mouth is completely closed cannot always be specified. This is because when the utterance of the word “migi (right)” is ended, the mouth may be open in a “b” state or may be closed.

  In any case, in this embodiment, each word used for the operation of the mobile phone 300 by the user is relatively inconvenient by using the characteristic that the word is a combination of each vowel and whether each sound is a bilateral sound. You can register without any problems.

  FIG. 14 shows an outline of processing in the cellular phone menu screen call and word registration mode by voice. This will be described with reference to FIGS. 11 and 12.

  First, as shown in FIG. 11, the user opens the first casing 301 and the second casing 302 from the folded state, and utters “menu” which means calling a menu screen. Then, the CPU 321 takes in the image of the user's face at this time from the imaging device 306 that has been activated, and determines that there has been an instruction for a “menu screen” (step S401).

  FIG. 15 shows the positional relationship between the mobile phone and the user's face when the process of step S401 is performed. This will be described with reference to FIG. In general, the face 352 of the user 351 is positioned in front of the imaging device 306 in a state where the first casing 301 and the second casing 302 of the mobile phone 300 are opened. Therefore, in this state, the lip image region extraction unit 330 can cut out the image region of the lips 353 of the user 351 from the image of the face 352 existing within the range of the angle of view (viewing angle) θ into which the subject is captured. The image area of the lips 353 cut out at this time is sequentially taken into the image memory 329, and when the user 351 utters a word, a series of images from the start point to the end point are extracted by the word image extraction unit 331.

  With respect to the extracted series of images, the closing timing measuring unit 332 measures the closing timing of the upper and lower lips when the user utters a word. In addition, the vowel discrimination unit 333 corrects the inclination of the upper and lower lips in the lip image cut out by the lip image region extraction unit 330, and then the shape shown in FIG. Discriminate each vowel.

  When discriminating the timing of closing the upper and lower lips and the vowels, with respect to temporal changes in the vertical and horizontal distances passing through the center point of the opening of the lips 353 in the image cut out by the lip image region extraction unit 330, 1 A word utterance pattern when a unit word is uttered is developed in a working memory area (not shown) of the image memory 329. When the timing of closing the upper and lower lips is recorded, and when the shape of the opening of the lips 353 (see FIG. 13) continues for a predetermined time (predetermined time) as a minimum time for uttering vowels, Based on the measured distances in the vertical and horizontal directions, it is determined whether these correspond to any of the five types of vowels. Also, if it is determined that the shape of the opening of the lip 353 corresponding to one type of vowel continues for a time exceeding a predetermined time, it is assumed that the same vowel is repeatedly uttered every time the specific time elapses. Will be determined.

  FIG. 16 shows an example of utterance contents, analysis results, and dictionary registration contents for the word utterance patterns registered in the word dictionary in the present embodiment. This will be described with reference to FIG.

  The word dictionary 327 is divided into a system dictionary that a user initially uses for basic operations and a user dictionary that the user registers later. In the system dictionary, the words “menu” and “register (registered)” shown in FIG. 16 are registered.

  Among them, the utterance of the phrase “menu” indicates the timing at which both lip sounds measured by the closing timing measurement unit 332 are represented by “◯”, and the consonant is represented by “Δ”, the vowels identified by the vowel discrimination unit 333. Can be used to represent “◯ ΔeΔiΔw”. Here, since “Me” is a lip sound, “O” is set as the starting point, followed by the consonant “Δ” of “Ma” and the vowel “E” of “Me”. “Di” is composed of consonant “Δ” in a row and vowel “I” in “d”. “Yu” is composed of the consonant “Δ” of the line and the vowel “u” of “yu”. Since “-” is a word obtained by extending “yu”, it is composed of a vowel “u” with a consonant “Δ” like “yu”. However, the time interval from when the sound “d” is uttered to when the sound “u” is uttered is the minimum interval at which the vowel “a” is recognized.

  Returning to FIG. 14, the description will be continued. When the user utters “menu”, the CPU 321 removes the consonant “Δ” from the utterance pattern, and processes it as “◯ ΔeΔiΔwoo”. When the CPU 321 searches the system dictionary area of the word dictionary 327 using the processing content, it matches the registered content of the “menu”. Therefore, it is determined that the user has instructed the menu screen (step S401). Therefore, the CPU 321 controls the display control device 326 so as to display the menu screen on the display 305 (step S402).

  When the menu screen is displayed on the display 305, the CPU 321 waits for the next utterance instruction in this display state. It is assumed that the user utters “Toroku (registration)” to perform word registration. Here, the utterance “TOROKU” is registered in the system dictionary area of the word dictionary 327 with the registered content “OOU” as shown in FIG.

  On the other hand, when the user utters “Toroku”, the CPU 321 performs the above-described processing based on the word utterance pattern developed in the image memory 329 and obtains “ΔOhΔOΔU” as an analysis result. Here, the sounds of “G”, “U”, “B”, and “K” are not both-lip sounds, and therefore “○” is not attached. The first “G” is composed of a consonant “Δ” and a vowel “O”. “U” is composed of the vowel “U” as it is. “Ro” is composed of consonant “Δ” and vowel “U”. The last “ku” is composed of a consonant “Δ” and a vowel “c”. As a result, when the consonant “Δ” is removed, “Ouo” is obtained, which completely matches the registered content of “Turoku (Registered)”.

  Therefore, if the user utters “Turoku (registration)” in the state where the menu screen is displayed on the display 305 in the process of step S402, the CPU 321 determines that there is an instruction by uttering “Toroku (registration)”. Then (step S403: Y), the word registration mode by voice is executed (step S404). Then, when the user utters “warming (end)”, the CPU 321 detects a match with the registered content “Oai” (see FIG. 16) on the word dictionary 327 side and utters “warming (end)”. After the determination (step S405: Y), the word registration mode is ended (END).

  On the other hand, when the menu screen is displayed on the display 305, the user utters a word other than “Toroku” (step S403: N), and it is determined as a mode other than the word registration mode. (Step S406: Y), the corresponding mode is executed. In other cases, for example, if the user coughs or utters a phrase that does not match the registered contents of the word dictionary 327 (step S406: N), the menu screen is displayed (step S402). Of course, depending on the system, the display state of the menu screen may be ended by the user saying “Wai (End)” in this state.

  By the way, in the word dictionary 327 shown in FIG. 16, utterance contents such as “Tabuchi (Tab 1)” and “Tabuni (Tab 2)” are registered as user dictionaries. Therefore, how the user 351 operates the tab displayed on the display 305 using such a word dictionary 327 will be described.

  FIG. 17 shows an example of display contents on the display. This will be described with reference to FIG.

  On the display 305, windows having a plurality of tabs are displayed in an overlapped state when a certain operation is performed. In FIG. 17, as a result of selecting “Tab 1”, the contents are displayed in a state where the first window is brought to the front. If the user uses “hand” when moving from this state to the second window of “tab 2”, the user selects the “2” key of the dial key 307C shown in FIG. The cursor is not moved to the position of the number “2” and clicked. In mobile phone 300 according to the present embodiment, a tab can be selected also by speaking.

  FIG. 18 shows the flow of processing when the user manipulates the display content by speaking. This process is performed as one of the “other instructions” in step S406 in FIG. This will be described with reference to FIGS. 12 and 15 to 17.

  It is assumed that the content of FIG. 17 is displayed on the display 305 and that the user 351 has selected the operation of the display content by speaking as one of the “other instructions” in step S406 of FIG. In this state, the image data captured by the imaging device 306 is stored endlessly in a predetermined ring memory area of the image memory 329, the lip image area extraction unit 330 sequentially extracts the lip image areas, and the vowel discrimination unit 333 Is checked (step S421).

As a result, when one of the vowels is detected from the analysis of the opening of the lips 353 (step S421: Y), the word image extraction unit 331 goes back the image stored in the ring memory area of the image memory 329 by time t _1. Then, the word extraction is started (step S422). Here, the time t ₁ is a length for confirming whether or not the sound including the vowel is a bilateral sound.

At the same time, the word image extraction unit 331 sequentially checks the time point after the start of extraction in the ring memory area, and the next vowel is detected (determined) within time t ₂ from the detection (determination) of the previous vowel. (Step S423, step S424). Here, the time t ₂ is a time obtained by adding a predetermined margin time to the maximum time that is normally assumed to be “between” when a plurality of sounds are uttered as words. If the word image extraction unit 331 determines the next vowel before the elapse of time t ₂ (step S423: Y), it repeatedly checks whether the next vowel is determined within the time t ₂ .

On the other hand, when the next vowel is not discriminated before the elapse of time t ₂ (step S423: N, step S424: Y), the word image extraction unit 331 ends the extraction of the word image at that time ( Step S425). The word discriminating unit 328 performs the word processing described in the part of FIG. 14 and acquires the content corresponding to the “registration content” in the word dictionary 327 (step S426).

  For example, it is assumed that an image of a word uttered by the user 351 as “Tabni (tab 2)” is cut out by the lip image region extraction unit 330. In this case, the processing result “A o ui” is obtained in the processing of step S426. The CPU 321 searches the registered contents of the word dictionary 327 (step S427), and if there is something that matches this (step S428: Y), the operation of the registered contents is executed (step S429), and the process is stepped. Return to S421 (return). In this example, as the operation in step S429, the state in which the first window in FIG. 17 is displayed is changed to the state in which the second window is selected instead by selecting “Tab 2”.

  If there is no registered content in the word dictionary 327 that matches the processing result “Aui” from the search result (step S428: N), an error display such as “operation content cannot be recognized” or the like is constant on the display 305. A period appears (step S430), and then the process returns to step S421 (return). Here, the user 351 can try the operation by utterance again. Of course, depending on the system, when there is no matching registered content in the search of step S427, the word having the highest degree of matching of the positions of individual vowels and lip sounds is displayed as a matching candidate to allow the user 351 to confirm, When the degree of matching exceeds a predetermined threshold value, it may be processed as matching and the corresponding operation may be executed.

  Next, a specific description will be given of a case where a user registers a word necessary for an operation. In the word dictionary 327 shown in FIG. 16, it is assumed that words from “Tabichi (Tab 1)” to “Tab San (Tab 3)” instructing expansion from the first window to the third window are registered. It is assumed that a word for instructing development of the fourth window has not been registered yet. In this case, the user 351 (FIG. 15) cannot expand the fourth window by voice instruction on the display 305 shown in FIG.

  FIG. 19 specifically shows the flow of the word registration process in step S404 of FIG. This will be described with reference to FIGS.

  When the word registration mode in step S404 in FIG. 14 is started, the CPU 321 displays a screen for selecting the operation content to be registered in the word on the display 305 (step S441). This display has a content that allows any operation related to display on the display 305 to be selected alternatively in consideration of a handicapped person. When requesting the next display content, the user 351 utters “tsugi (next)” (step S442: Y). In this case, the CPU 321 switches to the next selection screen (step S443) and returns to the display state of step S441. On the other hand, when the user 351 requests the display content previously displayed, the user utters “mae (previous)” (step S442: N, step S444: Y). In this case, the CPU 321 switches to the previous selection screen (step S445) and returns to the display state of step S441.

In this way, when a desired operation content to be registered at a certain point is displayed on the display 305 in this way, the user 351 utters “select” to select this operation content as a word registration target. . When recognizing this (step S442: N, step S444: N, step S446: Y), the CPU 321 displays a voice input instruction on the display 305 (step S447). When a time t ₃ sufficient for inputting a word elapses from the instruction (step S448: Y), the word image extraction unit 331 selects an image constituting the word from the image in the meantime extracted by the lip image region extraction unit 330. Extract.

  The word discriminating unit 328 performs the word processing described in the section of FIG. 14 on the extracted image, and acquires the content corresponding to “registration content” in the word dictionary 327 (step S449). The CPU 321 searches the registered content of the word dictionary 327 for the acquired content (step S450), and if there is no match (step S451: N), the content of the sound uttered by the user 351 is registered as a word (step S450). S452), the process ends (END).

  On the other hand, when the result obtained by performing the word processing in step S449 matches any of the registered contents of the word dictionary 327 (step S451: Y), an error is displayed on the display 305 to prevent duplicate registration. (Step S453). On the other hand, the user 351 tries again the process for registration or makes a selection to give up the word registration by voice utterance (steps S454 and S455). When it is determined that the user 351 has uttered “register (register)” for registration (step S454: Y), the CPU 321 returns the process to step S447 and waits for voice input again.

  For example, it is assumed that the user 351 utters “Tabsi (tab 4)” at the first time point in order to register a word that expands the fourth window by voice instruction. In this case, the result of the word processing in step S449 is “A o ui”, which matches the registered content of “Tabuni (tab 2)”. Therefore, “Tabushi (Tab 4)” matches the already registered contents of the word dictionary 327 (Step S451: Y) and cannot be registered (Step S453).

  In this case, it is assumed that the user 351 selects to try to register (step S454: Y), and reads “tabuyon (tab 4)” and speaks a word. Then, the result of the word processing in step S449 is “Ao Woo”, and the user 351 can perform word registration.

  At the time when the error display is performed, the user 351 can give up the word processing and say “Wai (End)”. In this case, when the CPU 321 determines “Warning” (step S454: N, step S455: Y), the word registration process ends (end). Even when the user 351 utters “warming (end)” while the operation content selection screen in step S441 is displayed (step S442: N, step S444: N, step S446: N, step S456: Y). The same (end).

  According to the embodiment described above, the words uttered by the user 351 are distinguished by the arrangement of vowels and lip sounds, and the word comparison process is performed. Therefore, a large number of words can be registered with words that are easy for the user 351 to remember, and many operations are possible depending on the shape of the opening of the lips when the display operation is sounded. In addition, the user 351 can perform the display operation only by changing the shape of the opening of the lips without necessarily generating a sound, so that it does not cause trouble to other people due to the generation of the sound.

  In the embodiment, the “n” sound is processed as the same vowel as the previously uttered sound, but it is processed as the same vowel as the next uttered sound, or the indefinite sound “−”. The contents may be classified and processed as not applicable. In the present embodiment, the process has been described on the assumption that the user 351 does not close his / her lips before and after the start of utterance of a word. However, in the case of a person who always closes his / her lips, The data processing similar to that of a person who does not close the lips can be performed by removing the “o” symbols one by one.

  <First Modification of Invention>

  In the embodiment described above, a word is uttered naturally, it is decomposed into vowels and lip sounds, registered in a dictionary, and associated with a display operation. However, the present invention is not limited to this. As a simple voice operation, each sound constituting a word may be divided by opening and closing the lips and uttered one sound at a time, and a combination of these vowels may be associated with a display operation. In this case, the types of words that can be registered are limited to some extent, but the contents displayed on the display can be sufficiently manipulated by making good use of the registered words.

  FIG. 20 shows a change in image when a user as a speaker utters a word one sound at a time. In the case of uttering a sound “Migi (right)” as a word used for the operation, in this first modification, “mi” and “gi” are pronounced one by one at intervals. FIG. 6A shows the state of the image before “mi” is uttered, and the lips 353 of the user 351 are closed.

  FIG. 5B is an image showing the shape of the lips in the utterance state, and the lips 353 of the user 351 are opened according to the type of sound uttered. When the utterance of one sound is finished, in this first modification, the user 351 closes the lips 353 and returns to the state shown in FIG.

  Since the mouth is opened and closed each time one sound is produced in this way, the image of FIG. 10C with the mouth open and the image of FIG. It will be repeated by the number. Of course, the user 351 can only open and close the lips 353 without utterance, and in this case as well, the screen display operation device of the first modification can perform various operations on the display.

  FIG. 21 is for explaining a first example of the operation of the display in the first modification. This will be described with reference to FIG.

  On the display 305, a first window 501 and a second window 502 are displayed. In such a case, the user 351 first performs an operation to activate one of the windows, and then performs data processing on the corresponding window. At this time, a simple operation of switching between the first window 501 and the second window 502 can be performed by the opening / closing determination of the mouth shown in FIGS.

 FIG. 22 shows a state of switching control between the active and inactive states of the first and second windows. This will be described with reference to FIGS.

  The first state shown in FIG. 22A is an initial state. In this initial state, the first window 501 is active and can be operated. At this time, the second window 502 is inactive and cannot be operated. In this state, when the user 351 opens and closes his / her mouth once as shown in FIGS. 20A and 20C, the first window 501 becomes inactive as shown in FIG. The second window 502 becomes active. When the user 351 opens and closes his / her mouth once from the state in which the second window 502 is active as shown in FIG. 22B, the first window 501 is again displayed as shown in FIG. 22A. Become active. At this time, the second window 502 is non-active.

  In this way, when there are two operation items on the display 305 and these must be selected alternatively, the selection operation is repeated each time the user 351 opens and closes the mouth in this first modification. Become. Therefore, the screen can be operated very easily.

  However, the types of words for instructing operations are extremely limited only by opening and closing the mouth. Therefore, it becomes difficult to control one of the plurality of windows shown in FIG. Therefore, in such a case, the operation by determining the shape of the opening of the lips 353 at the time of word utterance shown in FIGS. 20A and 20B is effective.

  FIG. 23 is a diagram for explaining selection of “tab 1” to “tab 4” described in FIG. 17 as a second example in the first modification. This will be described with reference to FIGS. 17 and 20.

  A state in which the first window is displayed on the front as shown in FIG. 17 is defined as a first state. At this time, the first window is active, and the user can perform more detailed operations on the first window. In this state, it is assumed that the user 351 utters “Mi” and “Gi” (right) while opening and closing his mouth each time he utters one sound. In this case, each vowel is discriminated by the recognition described in the previous embodiment, and a discrimination result “I, I” is obtained. In this first modification, the mouth is opened and closed each time one sound is uttered, so the notation by discrimination of both lip sounds is omitted.

  Each time the determination result “I, i” is obtained, the screen display operation device of the first modification advances the tab selection one by one in the right direction. Therefore, when the user 351 utters “Mi” and “Gi” (right) in the first state, the second state is set and “Tab 2” is selected and the second window is activated. When the user 351 utters “Mi” and “Gi” (right) in the second state, the third state is entered, “Tab 3” is selected, and the third window is activated. Further, when the user 351 utters “Mi” and “Gi” (right) in the third state, the fourth state is entered, “Tab 4” is selected, and the fourth window is activated. Furthermore, when the user 351 utters “mi” and “gi” (right) in the fourth state, the user returns to the first state and “tab 1” is selected and the first window becomes active. The same applies hereinafter.

  In addition, it is assumed that the user 351 utters “hi”, “da”, “li” (left) while opening and closing his / her mouth every time he utters one sound while the first window is active. . In this case, each vowel is discriminated by the recognition described in the previous embodiment, and a discrimination result of “A, A, A” is obtained.

  Each time the determination result “i, a, i” is obtained, the screen display operation device of the first modification advances the selection of the tabs one by one in the left direction. Therefore, when the user 351 utters “hi”, “da”, “li” (left) in the first state, the fourth state is entered, and “tab 4” is selected and the fourth window is activated. Become. In addition, when the user 351 utters “hi”, “da”, “li” (left) in the fourth state, the third state is entered and “tab 3” is selected and the third window becomes active. Become. Further, when the user 351 utters “hi”, “da”, “li” (left) in the third state, the second state is entered and “tab 2” is selected and the second window becomes active. . Furthermore, when the user 351 utters “hi”, “da”, “li” (left) in the second state, the user returns to the first state and “tab 1” is selected and the first window is activated. It becomes. The same applies hereinafter.

  According to the first modified example of the present invention described above, the mouth is opened and closed each time the user 351 utters one sound, so that each vowel constituting the word can be easily identified. Therefore, if you register the combination of opening and closing the mouth and vowels in advance for typical display operations such as screen switching, screen movement, backlight on / off, etc., you can use your hands. Various operations become possible. For this reason, there is an advantage that various operations are possible if the face can be turned to the screen display operation device even when the hand is occupied.

  Moreover, by registering operation patterns that are easy to understand intuitively for the user 351 such as “Mi”, “Gi” (right), “Hi”, “Da”, “Li” (left), Anyone can use it easily.

  <Second Modification of Invention>

  FIG. 24 shows a screen display operating device in a second modification of the present invention. The screen display operation device according to the second modification is a notebook personal computer 600. The personal computer 600 has a structure in which an imaging device 604 is attached together with a display 603 on the inner surface side of a lid portion 602 that can be freely opened and closed in the apparatus main body 601. Therefore, the user can not only operate the key operation unit 605 by hand, but also can perform various operations by opening and closing the mouth and recognizing the shape of the opening of the lip when speaking a word using the imaging device 604.

  Thus, the screen display operation device of the present invention is not necessarily limited to a small information processing device such as a mobile phone or a PHS (Personal Handy-phone System), and can be applied to any information processing device that can use an imaging device. is there.

  In the embodiments and modifications of the present invention, the sound discrimination based on the characteristics of vowels and lip sounds for Japanese is described. However, the present invention is also applied to other languages having different types and numbers of vowels. Of course you can do it.

  Some or all of the embodiments described above are described as in the following supplementary notes, but are not limited to the following descriptions.

(Appendix 1)
A display that visually displays various types of information;
Imaging means for photographing at least an operator's mouth using the display;
Change discriminating means for discriminating temporal changes in the image of the operator's mouth obtained by the imaging means;
An information processing apparatus comprising: a specific operation executing unit that executes a predetermined specific operation associated with the determination result according to the determination result of the change determination unit.

(Appendix 2)
The information processing apparatus according to claim 1, wherein the change determination unit determines a change in opening / closing of an operator's mouth.

(Appendix 3)
The information processing apparatus according to claim 1, wherein the change determining unit determines a specific shape formed for each sound when the operator's lip opening is uttered.

(Appendix 4)
The information processing apparatus according to appendix 3, wherein the unique shape formed for each sound is a shape when a vowel is uttered.

(Appendix 5)
Lip image region extracting means for extracting a lip region image from a human face image to be recognized;
A word image extraction means for extracting a series of temporal changes from the start to the end of the lip image change in the lip image area extracted by the lip image area extraction means as a unit word image;
Vertical and horizontal directions passing through the center point of the opening of the lips when the joints when the upper and lower lips are closed in the word image extracted by the word image extracting means are arranged horizontally. A word utterance pattern recording means for recording a temporal change in each distance as a word utterance pattern when a unit word is uttered;
A word dictionary in which each word utterance pattern previously recorded by the word utterance pattern recording means or equivalent means is associated with a word;
Pattern comparison means for comparing the word utterance pattern to be recognized recorded by the word utterance pattern recording means with the word utterance pattern for each word in the word dictionary; and (f) the result of comparison by the pattern comparison means, A word discriminating apparatus comprising: a word discriminating unit that discriminates a word corresponding to a word utterance pattern determined to be the best match as a word uttered by a person to be recognized.

(Appendix 6)
Lip image region extracting means for extracting a lip region image from a human face image to be recognized;
A word image extraction means for extracting a series of temporal changes from the start to the end of the lip image change in the lip image area extracted by the lip image area extraction means as a unit word image;
A closing timing measuring means for measuring the timing at which the upper and lower lips in the word image extracted by the word image extracting means are closed;
Vowel discrimination means for discriminating each vowel constituting a word from the shape of the same lip opening that continues for a predetermined time or longer in the word image extracted by the word image extraction means;
A word dictionary prepared by associating a combination of each timing measured by the closing timing measuring means or equivalent means and each vowel determined by the vowel discrimination means or equivalent means with a plurality of words in advance. When,
Comparison comparing each timing measured by the closing timing measuring unit in the word image extracted by the word image extracting unit and each vowel combination constituting the word determined by the vowel determining unit with the combination in the word dictionary Means,
Word discrimination means comprising: word discrimination means for discriminating that a word corresponding to the combination in the word dictionary determined to be the best match by the comparison means is a word uttered by the person to be recognized; apparatus.

(Appendix 7)
The vowel discrimination means discriminates that the same vowel is repeatedly uttered every time a specific time exceeding the predetermined time elapses when the shape of the same lip opening continues for the predetermined time or longer. The word discrimination device according to appendix 6.

(Appendix 8)
Lip image region extracting means for extracting a lip region image from a human face image to be recognized;
A word image extraction means for extracting a series of temporal changes from the start to the end of the lip image change in the lip image area extracted by the lip image area extraction means as a unit word image;
Vertical and horizontal directions passing through the center point of the opening of the lips when the joints when the upper and lower lips are closed in the word image extracted by the word image extracting means are arranged horizontally. A word utterance pattern recording means for recording a temporal change in each distance as a word utterance pattern when a unit word is uttered;
A word dictionary in which each word utterance pattern previously recorded by the word utterance pattern recording means or equivalent means is associated with a word;
Pattern comparing means for comparing the word utterance pattern to be recognized recorded by the word utterance pattern recording means with the word utterance pattern for each word in the word dictionary;
As a result of the comparison by the pattern comparison means, a word determination means for determining that a word corresponding to the word utterance pattern determined to be the best match is a word uttered by the person to be recognized,
A display that displays various information,
A screen display operation device comprising: content operation means for operating display contents displayed on the display with operation contents corresponding to the word determined by the word determination means.

(Appendix 9)
Lip image region extracting means for extracting a lip region image from a human face image to be recognized;
A word image extraction means for extracting a series of temporal changes from the start to the end of the lip image change in the lip image area extracted by the lip image area extraction means as a unit word image;
A closing timing measuring means for measuring the timing at which the upper and lower lips in the word image extracted by the word image extracting means are closed;
Vowel discrimination means for discriminating each vowel constituting a word from the shape of the same lip opening that continues for a predetermined time or longer in the word image extracted by the word image extraction means;
A word dictionary prepared by associating a combination of each timing measured by the closing timing measuring means or equivalent means and each vowel determined by the vowel discrimination means or equivalent means with a plurality of words in advance. When,
Comparison comparing each timing measured by the closing timing measuring unit in the word image extracted by the word image extracting unit and each vowel combination constituting the word determined by the vowel determining unit with the combination in the word dictionary Means,
A word discriminating unit for discriminating that a word corresponding to the combination in the word dictionary determined to be the best match by the comparing unit is a word uttered by the person to be recognized;
A display that displays various information,
A screen display operation device comprising: content operation means for operating display contents displayed on the display with operation contents corresponding to the word determined by the word determination means.

(Appendix 10)
The vowel discrimination means discriminates that the same vowel is repeatedly uttered every time a specific time exceeding the predetermined time elapses when the shape of the same lip opening continues for the predetermined time or longer. The screen display operation device according to appendix 9.

(Appendix 11)
Lip image region extracting means for extracting a lip region image from a human face image to be recognized at the time of word registration;
A registration target word image extraction unit that extracts a word image when a series of changes in the lip image region extracted from the lip image region extracted by the lip image region extraction unit from the start to the end of the lip image region is registered as a word image;
A vertical direction passing through the center point of the opening of the lips when the joints when the upper and lower lips are closed in the word image extracted by the registration target word image extracting means are horizontally arranged as a whole; Word utterance pattern recording means for recording temporal changes in the distances in the left and right directions as word utterance patterns when one unit of words is uttered;
A word dictionary in which each word utterance pattern previously recorded by the word utterance pattern recording means or equivalent means is associated with a word;
Pattern comparison means for comparing the word utterance pattern to be registered in the word utterance pattern recording means with the word utterance pattern for each word in the word dictionary;
A word comprising: a registered word availability judging means for registering only unregistered words corresponding to a word utterance pattern judged not to be approximated by a predetermined value or more as a result of comparison by the pattern comparing means; Registration device.

(Appendix 12)
Lip image region extracting means for extracting a lip region image from a human face image to be recognized at the time of word registration;
A registration target word image extraction unit that extracts a word image when a series of changes in the lip image region extracted from the lip image region extracted by the lip image region extraction unit from the start to the end of the lip image region is registered as a word image;
A closing timing measuring means for measuring a timing at which upper and lower lips are closed in the word image extracted by the registration target word image extracting means;
Vowel discrimination means for discriminating each vowel constituting a word from the shape of the same lip opening that lasts for a predetermined time or longer in the word image extracted by the registration target word image extraction means;
A word dictionary prepared by associating a combination of each timing measured by the closing timing measuring means or equivalent means and each vowel determined by the vowel discrimination means or equivalent means with a plurality of words in advance. When,
Compare each timing combination measured by the closing timing measurement unit in the word image extracted by the registration target word image extraction unit and each vowel constituting the word determined by the vowel determination unit with the combination in the word dictionary. Comparing means to
A word registration characterized by comprising: a registered word availability judging means capable of registering only unregistered words corresponding to a word utterance pattern judged not to approximate more than a predetermined value as a result of comparison by the comparing means apparatus.

(Appendix 13)
The vowel discrimination means discriminates that the same vowel is repeatedly uttered every time a specific time exceeding the predetermined time elapses when the shape of the same lip opening continues for the predetermined time or longer. The word registration device according to appendix 12.

(Appendix 14)
An imaging step of photographing at least an operator's mouth using a display that visually displays various types of information;
A change determination step for determining a change over time in an image of the operator's mouth obtained in this imaging step;
An information processing method comprising: a specific operation executing step for executing a predetermined specific operation associated with the determination result according to the determination result in the change determination step.

(Appendix 15)
A lip image region extracting step for extracting a lip region image from a human face image to be recognized;
A word image extraction step for extracting a series of temporal changes from the start to the end of the lip image change in the lip image region extracted in the lip image region extraction step as a unit word image;
The vertical and horizontal directions passing through the center point of the opening of the lips when the joints when the upper and lower lips are closed in the word image extracted by the word image extraction step are arranged horizontally. A word utterance pattern recording step of recording a temporal change of each distance of the word as a word utterance pattern when one unit of words is uttered;
The word utterance pattern to be recognized recorded in the word utterance pattern recording step is compared with the word utterance pattern for each word in the word dictionary registered in advance in association with each word in the word utterance pattern recording step. A pattern comparison step;
A word discrimination step for discriminating that the word corresponding to the word utterance pattern determined to be the best match as a result of the comparison in the pattern comparison step is a word uttered by the person to be recognized. Word discrimination method.

(Appendix 16)
A lip image region extracting step for extracting a lip region image from a human face image to be recognized;
A word image extraction step for extracting a series of temporal changes from the start to the end of the lip image change in the lip image region extracted by the lip image region extraction step as a unit word image;
A closing timing measurement step for measuring the timing at which the upper and lower lips in the word image extracted by the word image extraction step are closed;
A vowel discrimination step for discriminating each vowel constituting the word from the shape of the same lip opening that lasts for a predetermined time or longer in the word image extracted by the word image extraction step;
Each timing measured in the closing timing measurement step in the word image extracted in the word image extraction step and each vowel combination constituting the word determined in the vowel determination step is measured in the closing timing measurement step, respectively. A comparison step for comparing with the combination in the word dictionary registered in advance for each word with a combination of the timing of the vowel and the vowels determined in the vowel determination step;
A word determination step of determining that a word corresponding to the combination in the word dictionary determined to be the best match as a result of comparison in the comparison step is a word uttered by the person to be recognized. A word discrimination method.

(Appendix 17)
In the vowel determination step, when the shape of the same lip opening continues for the predetermined time or more, it is determined that the same vowel is repeatedly uttered each time a specific time exceeding the predetermined time elapses. The word discrimination method according to supplementary note 16.

(Appendix 18)
A lip image region extracting step for extracting a lip region image from a human face image to be recognized;
A word image extraction step for extracting a series of temporal changes from the start to the end of the lip image change in the lip image region extracted by the lip image region extraction step as a unit word image;
The vertical and horizontal directions passing through the center point of the opening of the lips when the joints when the upper and lower lips are closed in the word image extracted by the word image extraction step are arranged horizontally. A word utterance pattern recording step of recording a temporal change of each distance of the word as a word utterance pattern when one unit of words is uttered;
The word utterance pattern to be recognized recorded in the word utterance pattern recording step is compared with the word utterance pattern for each word in the word dictionary registered in advance in association with each word in the word utterance pattern recording step. A pattern comparison step;
As a result of comparison by this pattern comparison step, a word determination step for determining that a word corresponding to the word utterance pattern determined to be the best match is a word uttered by the person to be recognized,
A screen display operation method, comprising: a content operation step for operating display contents displayed on a display for displaying various types of information with operation contents corresponding to the word determined in the word determination step.

(Appendix 19)
A lip image region extracting step for extracting a lip region image from a human face image to be recognized;
A word image extraction step for extracting a series of temporal changes from the start to the end of the lip image change in the lip image region extracted by the lip image region extraction step as a unit word image;
A closing timing measurement step for measuring the timing at which the upper and lower lips in the word image extracted by the word image extraction step are closed;
A vowel discrimination step for discriminating each vowel constituting the word from the shape of the same lip opening that lasts for a predetermined time or longer in the word image extracted by the word image extraction step;
Each timing measured in the closing timing measurement step in the word image extracted in the word image extraction step and each vowel combination constituting the word determined in the vowel determination step is measured in the closing timing measurement step, respectively. A comparison step for comparing with a word dictionary registered in advance for each word with a combination of each vowel determined in the timing and the vowel determination step;
As a result of comparison by this comparison step, a word determination step of determining that a word in the word dictionary determined to be the best match is a word spoken by the person to be recognized,
A screen display operation method, comprising: a content operation step for operating display contents displayed on a display for displaying various types of information with operation contents corresponding to the word determined in the word determination step.

(Appendix 20)
In the vowel discrimination step, when the shape of the same lip opening continues for the predetermined time or more, it is determined that the same vowel is repeatedly uttered each time a specific time exceeding the predetermined time elapses. The screen display operation method according to appendix 19.

(Appendix 21)
A lip image region extracting step for extracting a lip region image from a human face image to be recognized at the time of word registration;
A registration target word image extraction step for extracting as a word image when an image for registering a series of temporal changes from the start to the end of the lip image change in the lip image region extracted by the lip image region extraction step;
The vertical direction passing through the center point of the opening of the lips when the joints when the upper and lower lips are closed in the word image extracted by the registration target word image extraction step are arranged horizontally. A word utterance pattern recording step for recording temporal changes in the distances in the left and right directions as a word utterance pattern when one unit of words is uttered;
The word utterance pattern to be registered in the word utterance pattern recording step is compared with the word utterance pattern for each word in the word dictionary registered in advance in association with each word in the word utterance pattern recording step. Pattern comparison step to
A word comprising: a registered word propriety determining step for registering only unregistered words corresponding to a word utterance pattern determined not to be approximated by a predetermined value or more as a result of comparison in the pattern comparing step; Registration method.

(Appendix 22)
A lip image region extracting step for extracting a lip region image from a human face image to be recognized at the time of word registration;
A registration target word image extraction step for extracting as a word image when an image for registering a series of temporal changes from the start to the end of the lip image change in the lip image region extracted by the lip image region extraction step;
A closing timing measuring step for measuring the timing at which the upper and lower lips in the word image extracted by the registration target word image extracting step are closed;
A vowel discrimination step for discriminating each vowel constituting a word from the shape of the same lip opening that continues for a predetermined time or longer in the word image extracted by the registration target word image extraction step;
In the closed timing measurement step, a combination of each timing measured in the closing timing measurement step in the word image extracted in the registration target word image extraction step and each vowel constituting the word determined in the vowel discrimination step is measured. A comparison step for comparing each word and a word dictionary registered in advance for each word with a combination of each vowel determined in the vowel determination step;
A word registration characterized by comprising: a registered word availability determination step that allows only unregistered words corresponding to the above-described combinations determined not to be approximated by a predetermined value or more as a result of comparison in the comparison step. Method.

(Appendix 23)
In the vowel discrimination step, when the shape of the same lip opening continues for the predetermined time or more, it is determined that the same vowel is repeatedly uttered each time a specific time exceeding the predetermined time elapses. The word registration method according to appendix 22.

(Appendix 24)
On the computer,
An imaging process for photographing at least an operator's mouth using a display that visually displays various types of information;
A change determination process for determining a temporal change in the image of the operator's mouth obtained by the imaging process;
An information processing program for executing a specific operation execution process for executing a predetermined specific operation associated with a determination result in accordance with a determination result by the change determination process.

(Appendix 25)
On the computer,
Lip image region extraction processing for extracting a lip region image from a human face image to be recognized;
A word image extraction process for extracting a series of temporal changes from the start to the end of the lip image change in the lip image area extracted by the lip image area extraction process as a unit word image;
Vertical and horizontal directions passing through the center point of the opening of the lips when the joints when the upper and lower lips are closed in the word image extracted by the word image extraction process are arranged horizontally. A word utterance pattern recording process for recording a temporal change in the distance of each as a word utterance pattern when one unit of words is uttered;
The word utterance pattern to be recognized recorded in the word utterance pattern recording process is compared with the word utterance pattern for each word in the word dictionary registered in advance in association with each word in the word utterance pattern recording process. Pattern comparison processing,
As a result of the comparison by the pattern comparison process, a word discrimination process for discriminating that the word corresponding to the word utterance pattern most closely matched is a word uttered by the person to be recognized is performed. Word discrimination program.

(Appendix 26)
On the computer,
Lip image region extraction processing for extracting a lip region image from a human face image to be recognized;
A word image extraction process for extracting a series of temporal changes from the start to the end of the lip image change in the lip image area extracted by the lip image area extraction process as a unit word image;
A closing timing measurement process for measuring the timing at which the upper and lower lips are closed in the word image extracted by the word image extraction process;
Vowel discrimination processing for discriminating each vowel constituting a word from the shape of the same lip opening that continues for a predetermined time or longer in the word image extracted by the word image extraction processing;
Each timing measured in the closing timing measurement process in the word image extracted by the word image extraction process and each vowel combination constituting the word determined in the vowel determination process is measured in the closed timing measurement process, respectively. A comparison process for comparing with a word dictionary registered in advance for each word with a combination of the timing of and the vowels determined in the vowel determination process;
A word discrimination process for discriminating that a word corresponding to the combination in the word dictionary determined to be the best match as a result of comparison by the comparison process is a word uttered by a person to be recognized; A word discrimination program.

(Appendix 27)
In the vowel discrimination process, when the shape of the same lip opening continues for the predetermined time or more, it is determined that the same vowel is repeatedly uttered each time a specific time exceeding the predetermined time elapses. The word discrimination program according to appendix 26.

(Appendix 28)
On the computer,
Lip image region extraction processing for extracting a lip region image from a human face image to be recognized;
A word image extraction process for extracting a series of temporal changes from the start to the end of the lip image change in the lip image area extracted by the lip image area extraction process as a unit word image;
Vertical and horizontal directions passing through the center point of the opening of the lips when the joints when the upper and lower lips are closed in the word image extracted by the word image extraction process are arranged horizontally. A word utterance pattern recording process for recording a temporal change in the distance of each as a word utterance pattern when one unit of words is uttered;
The word utterance pattern to be recognized recorded in the word utterance pattern recording process is compared with the word utterance pattern for each word in the word dictionary registered in advance in association with each word in the word utterance pattern recording process. Pattern comparison processing,
As a result of the comparison by the pattern comparison process, a word determination process for determining that the word corresponding to the word utterance pattern determined to be the best match is a word uttered by the person to be recognized,
What is claimed is: 1. A screen display operation program for executing content operation processing for operating display content displayed on a display for displaying various types of information with operation content corresponding to a word determined by the word determination processing.

(Appendix 29)
On the computer,
Lip image region extraction processing for extracting a lip region image from a human face image to be recognized;
A word image extraction process for extracting a series of temporal changes from the start to the end of the lip image change in the lip image area extracted by the lip image area extraction process as a unit word image;
A closing timing measurement process for measuring the timing at which the upper and lower lips are closed in the word image extracted by the word image extraction process;
Vowel discrimination processing for discriminating each vowel constituting a word from the shape of the same lip opening that continues for a predetermined time or longer in the word image extracted by the word image extraction processing;
Each timing measured in the closing timing measurement process in the word image extracted by the word image extraction process and each vowel combination constituting the word determined in the vowel determination process is measured in the closed timing measurement process, respectively. A comparison process for comparing with a word dictionary registered in advance for each word with a combination of the timing of and the vowels determined in the vowel determination process;
As a result of comparison by this comparison process, a word determination process for determining that a word in the word dictionary determined to be the best match is a word spoken by the person to be recognized;
What is claimed is: 1. A screen display operation program for executing content operation processing for operating display content displayed on a display for displaying various types of information with operation content corresponding to a word determined by the word determination processing.

(Appendix 30)
In the vowel discrimination process, when the shape of the same lip opening continues for the predetermined time or more, it is determined that the same vowel is repeatedly uttered each time a specific time exceeding the predetermined time elapses. The screen display operation program according to appendix 29.

(Appendix 31)
On the computer,
Lip image region extraction processing for extracting a lip region image from a human face image to be recognized at the time of word registration;
A registration target word image extraction process for extracting a word image when a series of temporal changes from the start to the end of a lip image change in the lip image area extracted by the lip image area extraction process is spoken;
The vertical direction passing through the center point of the opening of the lips when the joints when the upper and lower lips are closed in the word image extracted by the registration target word image extraction process are arranged horizontally. A word utterance pattern recording process for recording a temporal change in each distance in the left-right direction as a word utterance pattern when one unit of words is uttered;
The word utterance pattern to be registered in the word utterance pattern recording process is compared with the word utterance pattern for each word in the word dictionary registered in advance in association with each word in the word utterance pattern recording process. Pattern comparison processing,
A word that is subjected to a registered word availability determination process that allows only unregistered words corresponding to a word utterance pattern that is determined not to be approximated by a predetermined value or more as a result of comparison by the pattern comparison process. Registration program.

(Appendix 32)
On the computer,
Lip image region extraction processing for extracting a lip region image from a human face image to be recognized at the time of word registration;
A registration target word image extraction process for extracting a word image when a series of temporal changes from the start to the end of a lip image change in the lip image area extracted by the lip image area extraction process is spoken;
A closing timing measurement process for measuring the timing at which the upper and lower lips in the word image extracted by the registration target word image extraction process are closed;
Vowel discrimination processing for discriminating each vowel constituting a word from the shape of the same lip opening that continues for a predetermined time or longer in the word image extracted by the registration target word image extraction processing;
In the word image extracted by the registration target word image extraction process, the timing measured by the closing timing measurement process and the combination of the vowels constituting the word determined by the vowel determination process are measured by the closed timing measurement process. A comparison process for comparing with the above combination in the word dictionary registered in advance for each word with a combination of each timing and each vowel determined in the vowel determination process;
Word registration characterized in that, as a result of comparison by this comparison process, a registered word propriety determination process that allows only unregistered words corresponding to the above-described combinations determined not to approximate more than a predetermined value is executed. program.

(Appendix 33)
In the vowel discrimination process, when the shape of the same lip opening continues for the predetermined time or more, it is determined that the same vowel is repeatedly uttered each time a specific time exceeding the predetermined time elapses. The word registration program according to Supplementary Note 32.

DESCRIPTION OF SYMBOLS 10 Information processing apparatus 11, 58, 47, 305, 603 Display 12 Imaging means 13 Change discrimination means 14 Specific operation execution means 20, 30 Word discrimination apparatus 21, 31 Lip image area extraction means 22, 32 Word image extraction means 23 Word utterance Pattern recording means 24, 35, 44, 55, 64, 75, 327 Word dictionary 25, 45, 65 Pattern comparison means 26, 37, 46, 57 Word discrimination means 33, 53, 73 Closed timing measurement means 34, 54, 74 Vowel determination means 36, 56, 76 Comparison means 40, 50 Screen display operation device 41, 51, 61, 71 Lip image area extraction means 42, 52 Word image extraction means 43, 63 Word utterance pattern recording means 48, 59 Content operation means 60, 70 Word registration device 62, 72 Registration target word image extraction means 66, 77 Registration unit Visibility determining means 80, 90 Word determining method 81, 91 Lip image region extracting step 82, 92 Word image extracting step 83 Word utterance pattern recording step 84 Pattern comparing step 85, 96 Word determining step 93 Closing timing measuring step 94 Vowel determining step 95 Comparison step 100 Word discrimination program 101 Lip image region extraction processing 102 Word image extraction processing 103 Close timing measurement processing 104 Vowel discrimination processing 105 Comparison processing 106 Word discrimination processing 300 Mobile phone 306, 604 Imaging device 321 CPU
322 Memory 323 Main control unit 328 Word discrimination unit 329 Image memory 330 Lip image area extraction unit 331 Word image extraction unit 332 Close timing measurement unit 333 Vowel discrimination unit 351 User 352 Face 353 Lip 501 First window 502 Second window 600 Personal computer

Claims (10)

A display that visually displays various types of information;
Imaging means for photographing at least an operator's mouth using the display;
Change discriminating means for discriminating temporal changes in the image of the operator's mouth obtained by the imaging means;
An information processing apparatus comprising: a specific operation executing unit that executes a predetermined specific operation associated with the determination result according to the determination result of the change determination unit. Lip image region extracting means for extracting a lip region image from a human face image to be recognized;
A word image extraction means for extracting a series of temporal changes from the start to the end of the lip image change in the lip image area extracted by the lip image area extraction means as a unit word image;
Vertical and horizontal directions passing through the center point of the opening of the lips when the joints when the upper and lower lips are closed in the word image extracted by the word image extracting means are arranged horizontally. A word utterance pattern recording means for recording a temporal change in each distance as a word utterance pattern when a unit word is uttered;
A word dictionary in which each word utterance pattern previously recorded by the word utterance pattern recording means or equivalent means is associated with a word;
Pattern comparison means for comparing the word utterance pattern to be recognized recorded by the word utterance pattern recording means with the word utterance pattern for each word in the word dictionary; and (f) the result of comparison by the pattern comparison means, A word discriminating apparatus comprising: a word discriminating unit that discriminates a word corresponding to a word utterance pattern determined to be the best match as a word uttered by a person to be recognized. Lip image region extracting means for extracting a lip region image from a human face image to be recognized;
A word image extraction means for extracting a series of temporal changes from the start to the end of the lip image change in the lip image area extracted by the lip image area extraction means as a unit word image;
A closing timing measuring means for measuring the timing at which the upper and lower lips in the word image extracted by the word image extracting means are closed;
Vowel discrimination means for discriminating each vowel constituting a word from the shape of the same lip opening that continues for a predetermined time or longer in the word image extracted by the word image extraction means;
A word dictionary prepared by associating a combination of each timing measured by the closing timing measuring means or equivalent means and each vowel determined by the vowel discrimination means or equivalent means with a plurality of words in advance. When,
Comparison comparing each timing measured by the closing timing measuring unit in the word image extracted by the word image extracting unit and each vowel combination constituting the word determined by the vowel determining unit with the combination in the word dictionary Means,
A word discriminating unit for discriminating that the word corresponding to the combination in the word dictionary determined to be the best match as a result of comparison by the comparing unit is a word uttered by the person to be recognized; Word discrimination device. Lip image region extracting means for extracting a lip region image from a human face image to be recognized;
A word image extraction means for extracting a series of temporal changes from the start to the end of the lip image change in the lip image area extracted by the lip image area extraction means as a unit word image;
Vertical and horizontal directions passing through the center point of the opening of the lips when the joints when the upper and lower lips are closed in the word image extracted by the word image extracting means are arranged horizontally. A word utterance pattern recording means for recording a temporal change in each distance as a word utterance pattern when a unit word is uttered;
A word dictionary in which each word utterance pattern previously recorded by the word utterance pattern recording means or equivalent means is associated with a word;
Pattern comparing means for comparing the word utterance pattern to be recognized recorded by the word utterance pattern recording means with the word utterance pattern for each word in the word dictionary;
As a result of the comparison by the pattern comparison means, a word determination means for determining that a word corresponding to the word utterance pattern determined to be the best match is a word uttered by the person to be recognized,
A display that displays various information,
A screen display operation device comprising: content operation means for operating display contents displayed on the display with operation contents corresponding to the word determined by the word determination means. Lip image region extracting means for extracting a lip region image from a human face image to be recognized;
A word image extraction means for extracting a series of temporal changes from the start to the end of the lip image change in the lip image area extracted by the lip image area extraction means as a unit word image;
A closing timing measuring means for measuring the timing at which the upper and lower lips in the word image extracted by the word image extracting means are closed;
Vowel discrimination means for discriminating each vowel constituting a word from the shape of the same lip opening that continues for a predetermined time or longer in the word image extracted by the word image extraction means;
A word dictionary prepared by associating a combination of each timing measured by the closing timing measuring means or equivalent means and each vowel determined by the vowel discrimination means or equivalent means with a plurality of words in advance. When,
Comparison comparing each timing measured by the closing timing measuring unit in the word image extracted by the word image extracting unit and each vowel combination constituting the word determined by the vowel determining unit with the combination in the word dictionary Means,
A word discriminating unit for discriminating that a word corresponding to the combination in the word dictionary determined to be the best match by the comparing unit is a word uttered by the person to be recognized;
A display that displays various information,
A screen display operation device comprising: content operation means for operating display contents displayed on the display with operation contents corresponding to the word determined by the word determination means. Lip image region extracting means for extracting a lip region image from a human face image to be recognized at the time of word registration;
A registration target word image extraction unit that extracts a word image when a series of changes in the lip image region extracted from the lip image region extracted by the lip image region extraction unit from the start to the end of the lip image region is registered as a word image;
A vertical direction passing through the center point of the opening of the lips when the joints when the upper and lower lips are closed in the word image extracted by the registration target word image extracting means are horizontally arranged as a whole; Word utterance pattern recording means for recording temporal changes in the distances in the left and right directions as word utterance patterns when one unit of words is uttered;
A word dictionary in which each word utterance pattern previously recorded by the word utterance pattern recording means or equivalent means is associated with a word;
Pattern comparison means for comparing the word utterance pattern to be registered in the word utterance pattern recording means with the word utterance pattern for each word in the word dictionary;
A word comprising: a registered word availability judging means for registering only unregistered words corresponding to a word utterance pattern judged not to be approximated by a predetermined value or more as a result of comparison by the pattern comparing means; Registration device. Lip image region extracting means for extracting a lip region image from a human face image to be recognized at the time of word registration;
A registration target word image extraction unit that extracts a word image when a series of changes in the lip image region extracted from the lip image region extracted by the lip image region extraction unit from the start to the end of the lip image region is registered as a word image;
A closing timing measuring means for measuring a timing at which upper and lower lips are closed in the word image extracted by the registration target word image extracting means;
Vowel discrimination means for discriminating each vowel constituting a word from the shape of the same lip opening that lasts for a predetermined time or longer in the word image extracted by the registration target word image extraction means;
A word dictionary prepared by associating a combination of each timing measured by the closing timing measuring means or equivalent means and each vowel determined by the vowel discrimination means or equivalent means with a plurality of words in advance. When,
Compare each timing combination measured by the closing timing measurement unit in the word image extracted by the registration target word image extraction unit and each vowel constituting the word determined by the vowel determination unit with the combination in the word dictionary. Comparing means to
A word registration characterized by comprising: a registered word availability judging means capable of registering only unregistered words corresponding to a word utterance pattern judged not to approximate more than a predetermined value as a result of comparison by the comparing means apparatus. A lip image region extracting step for extracting a lip region image from a human face image to be recognized;
A word image extraction step for extracting a series of temporal changes from the start to the end of the lip image change in the lip image region extracted in the lip image region extraction step as a unit word image;
The vertical and horizontal directions passing through the center point of the opening of the lips when the joints when the upper and lower lips are closed in the word image extracted by the word image extraction step are arranged horizontally. A word utterance pattern recording step of recording a temporal change of each distance of the word as a word utterance pattern when one unit of words is uttered;
The word utterance pattern to be recognized recorded in the word utterance pattern recording step is compared with the word utterance pattern for each word in the word dictionary registered in advance in association with each word in the word utterance pattern recording step. A pattern comparison step;
A word discrimination step for discriminating that the word corresponding to the word utterance pattern determined to be the best match as a result of the comparison in the pattern comparison step is a word uttered by the person to be recognized. Word discrimination method. A lip image region extracting step for extracting a lip region image from a human face image to be recognized;
A word image extraction step for extracting a series of temporal changes from the start to the end of the lip image change in the lip image region extracted by the lip image region extraction step as a unit word image;
A closing timing measurement step for measuring the timing at which the upper and lower lips in the word image extracted by the word image extraction step are closed;
A vowel discrimination step for discriminating each vowel constituting the word from the shape of the same lip opening that lasts for a predetermined time or longer in the word image extracted by the word image extraction step;
Each timing measured in the closing timing measurement step in the word image extracted in the word image extraction step and each vowel combination constituting the word determined in the vowel determination step is measured in the closing timing measurement step, respectively. A comparison step for comparing with the combination in the word dictionary registered in advance for each word with a combination of the timing of the vowels and the vowels determined in the step,
A word determination step of determining that a word corresponding to the combination in the word dictionary determined to be the best match as a result of comparison in the comparison step is a word uttered by the person to be recognized. A word discrimination method. On the computer,
Lip image region extraction processing for extracting a lip region image from a human face image to be recognized;
A word image extraction process for extracting a series of temporal changes from the start to the end of the lip image change in the lip image area extracted by the lip image area extraction process as a unit word image;
A closing timing measurement process for measuring the timing at which the upper and lower lips are closed in the word image extracted by the word image extraction process;
Vowel discrimination processing for discriminating each vowel constituting a word from the shape of the same lip opening that continues for a predetermined time or longer in the word image extracted by the word image extraction processing;
Each timing measured in the closing timing measurement process in the word image extracted by the word image extraction process and each vowel combination constituting the word determined in the vowel determination process is measured in the closed timing measurement process, respectively. A comparison process for comparing with a word dictionary registered in advance for each word with a combination of the timing of and the vowels determined in the vowel determination process;
A word discrimination process for discriminating that a word corresponding to the combination in the word dictionary determined to be the best match as a result of comparison by the comparison process is a word uttered by a person to be recognized; A word discrimination program.

Images