JP2002259046A - System for entering character and symbol handwritten in air - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method which stably detects operation in various illumination conditions and background in a wearable computer environment by solving the problem of the difficulty in specifying the start and the end of user's character or symbol entry and the start and end points of segments constituting a character at recognizing a finger or a gesture, with which the character is written in air. SOLUTION: With respect to a system for entering of characters and symbols handwritten in the air, a terminal device for recognition, a method for representing and designing character and symbols handwritten in the air, and a virtual touch-panel button system, the operation of writing a character or a symbol in the air with a finger or a gesture is photographed by a moving image photographing camera 10, and voice and sounds during photographing are collected by the microphone 12, and the photographed video and collected voice and sounds are transmitted to a computer 14, and a character pattern plotted on the computer 14 by image analysis and voice and sound analysis is collated with patterns in a database and is recognized, and the recognition result is displayed on a display device 16, and the inputted character can be confirmed by voice and sounds from a speaker 18.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a character / symbol input system which is hand-drawn in the air, and more particularly to an operation of drawing a character / symbol in the air with a finger or a gesture by using a camera for moving image shooting, and the sound / sound during shooting is taken. Sound is picked up by a microphone, and the captured video and the picked up sound / sound are transmitted to a computer via a wireless / wired cable, and only image analysis or image analysis is performed on the computer. Character / symbol patterns drawn by sound / speech analysis are recognized against data-based patterns and recognized, the recognition result is displayed on a display, or a wireless / wired cable is transmitted from a computer. -Character / symbol input system hand-drawn in the air, which allows confirmation of characters / symbols input by sound or voice from a speaker connected with a cable, recognition terminal device, notation of characters / symbols drawn in the air Design method About the fine virtual touch panel button system.

[0002]

When a character or symbol is drawn in the air, since the written locus does not remain, the start and end of the user's input of the character or symbol, the starting point of the line segment constituting the character, It is difficult to identify the end point. Also, in a wearable computer (wearable computer) environment, a method for stably detecting operation under various lighting conditions and backgrounds is required.

As a solution to be considered, the user puts his hand outside the image frame when he finishes drawing characters / symbols. Under such shooting conditions, the background is not completely stationary and the shooting conditions are different each time, so that a design that operates stably in various backgrounds is required. Also, the lighting conditions are not constant.

A known handwritten character / symbol recognition system decodes a handwriting drawn on a pressure-sensitive panel or a handwriting drawn with a computer mouse. It was impossible to decipher the characters and symbols used.

Further, since the known gesture recognition does not presuppose the recognition of characters and symbols but presupposes that a video from a fixed camera is always analyzed, a wearable computer having a camera for photographing moving images is used. In the environment of a (worn computer) or mobile computer (portable computer) terminal, it has been difficult to recognize characters and symbols.

[0006] When writing characters in the air, wearable
In a computer or mobile computer terminal environment, under various lighting conditions and non-monochromatic background conditions,
Recognition processing must be performed by image analysis.

[0007] Under various lighting conditions, non-monochromatic background conditions, and conditions of shooting with an unfixed camera, it is very difficult to recognize characters and symbols drawn in the air by image analysis.

Also, when drawing characters and symbols in the air,
Since it is tilted in various directions, known gesture recognition recognizes the trajectory of the hand drawn by wearing special colored gloves and holding a special colored object in the hand, It was difficult to track the trajectory of the written characters in an environment without any of them.

As described above, since the trajectory of characters and symbols drawn in the air does not actually remain in the air, it is necessary to specify the start and end of character input and the start and end points of line segments constituting the character. was difficult. In addition, there are some characters such as 'K' and 'R' which are difficult to distinguish only from a locus drawn in the air.

[0010]

SUMMARY OF THE INVENTION According to the present invention, a character / symbol is drawn in the air in a state where nothing is attached to or held on a hand, and the image is processed on a computer. Then, the trajectory pattern of the fingertip or the gesture is specified from the amount of change in the analyzed image, and the data base pattern is determined.
Characters and symbols drawn in the air can be recognized by collating with characters, and sound and voice information can also be used for recognition. Camera, microphone for sound collection, computer for image and sound / speech analysis and display for displaying the recognition result, or in the air where characters / symbols composed of speakers shown by sound / speech can be confirmed Hand-painted characters
Symbol recognition terminal device, character / symbol notation / design method that can be recognized with high accuracy on the premise that no trajectory remains in the air, and hand region with high accuracy under various lighting / background conditions This is a virtual touch panel button system in an image frame that can acquire a character input start operation and area information of a hand being input at a desired time in order to obtain a desired time.

[0011]

According to the present invention, a method of roughly extracting a hand region by obtaining pixel information of a region of a user's hand immediately before character input, and a method of obtaining a luminance value between adjacent image frames. A method of analyzing the moving position of the finger based on the position of the center of gravity of the difference and continuous DP matching enables input of hand-drawn characters and symbols in the air without wearing / holding a special object on the hand, and a wearable computer, This is for inputting characters and symbols in a mobile computer terminal environment.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the character / symbol input system hand-drawn in the air according to claim 1 will be described.
1 to 4, in the character / symbol input system which is hand-drawn in the air according to the present invention, a video frame is acquired by the moving image photographing camera 10 (S101), and at the same time, sound and voice are acquired by the microphone 12 (FIG. 1). S102) The obtained video frame is transmitted to the computer 14 via a wireless / wired cable, and the video frame and the sound / audio signal are analyzed via the computer 14. Process (S1
03).

On the computer 14, only an image analysis or a combination of the image analysis and the sound / speech analysis is performed to obtain an input start signal for characters and symbols drawn in the air, and to specify a high-precision input operation. Therefore, region information of the hand in the video frame captured by the video capturing camera 10 is obtained (S104). The acquisition of the input start signal can be performed by a virtual touch panel type input system or a specific sound / voice according to the invention described in claim 4.

In FIG. 6, there is shown an embodiment of a virtual touch panel type input system according to the invention of claim 4,
This represents a camera image taken by the wearable computer or the mobile computer terminal 16 from the user side, and the user presses a gesture of pressing a different color (button portion) displayed in the center of the image frame. Then, it is determined by the image analysis on the computer 14 that a button has been pressed, and a specific process is performed. For example, when the user presses a button, it is assumed that the user's hand is at a specific position in the image frame, and by obtaining the luminance information and color information, various lighting and background conditions are obtained. And extract only the part of the user's hand.

In FIG. 6, in order to determine that the user has pressed the button, pixels having a large difference in luminance between adjacent image frames continuously obtained from the moving image camera 10 are used as image frames. Detects a state where the user concentrates only on the middle button area.

In FIG. 1, after obtaining the input start signal, the center of gravity G of the specific luminance variation between adjacent image frames is obtained.
Is determined (S105), the character / symbol drawn by the locus of the center of gravity G of the luminance change amount is determined, and an end signal of the input operation is obtained (S106). In order to determine a character / symbol drawn based on the locus of the center of gravity G of the luminance change amount, a character / symbol stored in a storage device on the computer 14 in advance is used.
It is collated with the trajectory pattern of the symbol, and the collation is performed by DP matching or HMM (hidden markovmode).
l) and the like. The end signal of the input operation is, for example, a state in which the user's hand is out of the image frame, or an operation of pressing the end determination button by the virtual touch panel. Is realized.

The drawn characters and symbols are displayed on the display 16 for confirmation by the user of the system (S10).
7) or sound from microphone 12 or speaker 18
Confirmation is made by voice (S108).

In FIG. 2, the recognition terminal device according to the second aspect of the present invention inputs an image frame from the moving image photographing camera 10 to the computer 14 and simultaneously sounds and sounds from the microphone 12. A signal is obtained and input to the computer 14, where the computer 14 performs image analysis, sound / voice analysis, and determines the trajectory of characters / symbols drawn in the air on gestures and the like. The result is displayed on the display 16 (S108) and output to the speaker or earphone 18. In FIG. 5, the recognition terminal device of FIG.
0 is replaced with an infrared camera 20, and an infrared light 22 is irradiated to shoot. 2 and 5, the microphone 1
2. It can be used only for image processing with the speaker 18 removed.

FIG. 3 shows an embodiment in which the recognition terminal device according to the second aspect of the present invention is used in a wearable computer environment. The present invention can also be applied to a system using a camera, a processor, and a display of a mobile computer terminal. The infrared camera 2 shown in FIG.
0, a system with the infrared light 22 can also be constructed.

FIG. 4 shows an embodiment in which the recognition terminal device according to the second aspect of the present invention is used in a fixed camera environment. This can be mounted on a robot, and the infrared camera 20 and the infrared light 2 shown in FIG.
2 can also be constructed.

Referring to FIG. 6, there is shown an embodiment of a touch panel type input system according to the fourth aspect of the present invention, which comprises a touch panel in an image frame 30 displayed on the display 16. The panel button 32 is displayed, the currently input character / symbol is displayed in the display area 34, and the type of the current input character / symbol is displayed in the display area 36.

FIG. 7 shows an embodiment of a method for writing and designing characters / symbols drawn in the air according to the third aspect of the present invention, in a wearable computer environment or a mobile computer terminal environment. Indicates alphanumeric characters assuming the use of. In this figure, the frame around each character shows the image frame, assuming that the characters start writing in the frame, make a single stroke, and finally reach out of the image frame. I have.

When hand-drawing characters and symbols in the air, the drawn trajectory becomes a single stroke,
Symbol design is required. In particular, characters such as 'K' and 'R' are extremely difficult to distinguish unless a trajectory remains, so that they can be clearly distinguished by a single-stroke notation. In addition, the characters and symbols can be easily distinguished from each other such as 'B' and 'P' by determining the direction in which the hand is to be moved out of the image frame after the writing.

[0024]

(1) The character / symbol input system hand-drawn in the air according to the present invention immediately recognizes the characters / symbols drawn in the air without holding anything in hand, and is provided with a video camera. In a wearable computer environment or a mobile computer terminal environment, a memo can be quickly taken. (2) The terminal device for recognizing characters and symbols drawn in the air according to the present invention can be mounted on a wearable computer or a mobile computer terminal with a camera for capturing moving images. (3) By mounting the terminal device for recognizing characters and symbols drawn in the air according to the present invention on a robot, characters and symbols can be transmitted to the robot, and communication between humans and the robot can be achieved. Means are expanded. (4) The character / symbol drawn in the air according to the present invention is mounted on a teaching material or a video game for children so that the character / symbol can be learned or used for games. (5) The terminal device for recognizing characters and symbols drawn in the air according to the present invention can perform input operation even at night by using an infrared light, so that a bedridden elderly person or a hospital patient does not get up. By drawing characters and symbols with one hand, input operation becomes possible. (6) According to the notation and design method of characters / symbols drawn in the air according to the present invention, when recognizing characters / symbols in the air, the feature of one-stroke writing is utilized, and it is effective and quick. Characters and symbols can be input. (7) It is desirable to apply the virtual touch panel button system according to the present invention to an input interface of a wearable computer or a mobile computer terminal with a video camera. At this time, the user can press the button to operate the terminal device and obtain the color information of a specific area such as a hand at an arbitrary moment, so that a highly accurate input system can be constructed.

[Brief description of the drawings]

FIG. 1 is a flowchart showing a processing flow of a character / symbol input system which is hand-drawn in the air according to the present invention.

FIG. 2 is a schematic diagram of a terminal for recognition of a character / symbol input system hand-drawn in the air according to the present invention.

FIG. 3 is a schematic diagram of an embodiment in which a terminal device for recognition of a character / symbol input system which is hand-drawn in the air according to the present invention is applied to a wearable computer.

FIG. 4 is a schematic diagram of an embodiment in which a terminal device for recognition of a character / symbol input system hand-drawn in the air according to the present invention is applied to a fixed camera.

FIG. 5 is a diagram illustrating the use of the terminal device for recognition of the character / symbol input system hand-drawn in the air according to the present invention in the dark;
4 is a schematic diagram of an embodiment when using an infrared camera and an infrared light.

FIG. 6 shows a virtual touch panel button according to the present invention.
FIG. 2 is a design diagram of a system.

FIG. 7 is an example of notation related to alphanumeric characters in the character / symbol input system hand-drawn in the air according to the present invention.

[Explanation of symbols]

10: Camera for video recording; 12: Microphone; 14: Computer for image processing, sound / voice processing, and arithmetic processing; 16: Display; 18: Speaker (or earphone); 20: Infrared camera; 22: Infrared Light; 30… Image frame like virtual touch panel
32: virtual touch panel button; 34: display area of characters / symbols being input on the virtual touch panel; 36: input mode display device of characters / symbols being input on the virtual touch panel .

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) H04N 7/18 H04N 7/18 KU F term (Reference) 5B064 AB04 BA05 DD16 EA24 FA03 FA16 5B068 AA05 BD09 BD17 BE08 CC06 CC17 CC19 5C054 AA01 CC02 CH01 EA01 EA03 EA05 FC00 FE00 HA00 HA05 5L096 BA16 BA20 CA04 DA02 FA60 FA70 GA45 HA02 HA08 JA16 JA20

Claims (4)

[Claims] 1. A character drawn in the air by a finger or a gesture.
In recognition of symbols, the user draws characters / symbols using a video camera, and the sound / speech input of the characters / symbols is collected by a microphone. Sound is transmitted to a computer via a wireless / wired cable, and images, sounds, and sounds are analyzed via the computer, and a locus pattern drawn by a hand or a fingertip in the image is displayed as an image only. Alternatively, it is obtained by a combination of image and sound / voice, and is recognized by comparing it with a pattern in a data base, and the recognized character / symbol is connected from the computer by the wireless / wired cable. Characters / symbols hand-drawn in the air, which can be displayed on a displayed display or the characters / symbols inputted from the computer by sound or voice from a speaker connected by the wireless / wired cable can be confirmed. Input system. 2. A moving image capturing camera for capturing an action of drawing characters and symbols in the air with a finger or a gesture, a microphone for collecting sounds and sounds, and a computer for analyzing images and sounds. It consists of a display and a speaker for confirming characters and symbols, and is used for recognizing characters and symbols drawn in the air with fingers and gestures. A terminal device for recognizing hand-drawn characters / symbols in the air, which makes it possible to confirm the characters / symbols that have been drawn. 3. Characters / symbols drawn in the air are input with a single stroke, so that characters / symbols with no handwriting can be easily distinguished, and characters / symbols drawn in the air can be input quickly. Notation and design method. 4. A moving picture camera for shooting a character or symbol in the air by a finger or a gesture.
A virtual touch panel button is assumed during the program, displayed on the display, and the user presses the virtual touch panel button by image analysis to recognize the operation of pressing the button and presses the button to the computer at any time. A virtual touch panel button system for input operations.