JP5152639B2 - Character input / function operation device, method, program, and portable terminal device - Google Patents

Description

  The present invention relates to a character input / function operation device for a portable terminal device. In particular, the present invention relates to a character input / function operation device, a method, a program, and a mobile terminal device that facilitate character input and function operation.

When inputting characters into the mobile terminal device that is the premise of the present invention, if the number of keys is small, it is necessary to perform key operations a plurality of times to input one character, and the display content on the screen is scrolled. -When performing operation to enlarge / reduce, there is a problem that if the number of keys is limited, it is necessary to press the key many times and the display contents cannot be operated intuitively.
In addition, when many keys such as a QWERTY keyboard are equipped, in many portable terminal devices, character input and function operations are performed using a keyboard with a limited number of keys, a QWERTY keyboard with a small key size, and a touch panel. However, the QWERTY keyboard has a problem that the interval between the keys is smaller than the size of the finger and is difficult to press, and the operability is impaired when the key size is small.

On the other hand, when the size of the portable terminal device is increased, there is a problem that portability and design freedom are impaired.
Further, when operating the display contents, it is difficult to operate intuitively and quickly compared with a mouse or the like for a personal computer by input using keys.
Furthermore, in the case of using the touch panel, when using the pen, not only is it time and effort required to take out the pen, but the pen may be lost. In addition, when a finger is directly touched on the touch panel, there is a problem that the screen is difficult to see due to dirt and the appearance is deteriorated.

As described above, there are the following conventional techniques related to the problems with the portable terminal device which is the premise of the present invention.
Conventionally, a plurality of characters have been assigned to buttons used for character input in order to make the input operation easier in a phone that requires many inputs such as a mobile phone, in order to improve the character input and input method to the phone. Depends on the time from when the button is pressed to when the button is released, it is provided with a motion detection device that determines which character is input and detects the motion of the user moving the phone. The screen is scrolled according to the content of the motion, and the user There is a finger motion detection means for detecting the motion of the finger in a non-contact manner, and an operation is instructed according to the content of the motion (for example, see Patent Document 1).

However, in Patent Document 1, since a plurality of characters are assigned to a button used for character input, as described above, it is necessary to press a key many times, and the display content cannot be operated intuitively. is there.
Conventionally, in order to provide an information terminal device that provides a space-saving and simple character input method and information selection method, the operator's head can be captured by a camera of an information terminal device that can be freely held and moved by hand, and digital image processing is performed. However, the movement of the information terminal is detected relative to the operator from the position of the head in the camera screen, and the virtual keyboard key or information group displayed on the terminal is temporarily selected with the digitized data. There is also a device that enables character input and information selection with one key by confirming by pressing one key provided on the terminal device or one virtual key on the display (for example, see Patent Document 2). .

However, in Patent Document 2, the movement of the information terminal is detected relative to the operator from the position of the head in the camera screen, and the key of the virtual keyboard displayed on the terminal is selected. Since the keys are selected by moving the hand holding the head or moving the head, there is a problem that the hand and head become tired for a long time operation.
In addition, in order to provide an information terminal device that provides a simple character input and information selection method based on the operator's minute part conventionally, a part of the operator is captured by a camera of an information terminal device that can be freely held by hand, The virtual keyboard keys or information displayed on the terminal using the digitized data by detecting the movement of the information terminal relative to the operator from the operator movement digitized by the digital signal processor according to the movement of the captured grayscale pattern There is also one that allows character input and information selection with one key by temporarily selecting a selection key and confirming by pressing one key provided on the information terminal device or one virtual key on the display. (For example, refer to Patent Document 3).

  However, in Patent Document 3, a part of the operator is captured by the camera, the movement of the information terminal is detected relative to the operator, and the virtual keyboard key or the information selection key displayed on the terminal is selected. , Move the hand holding the information terminal or move part of the operator (face, clothes) to select the key, there is a problem that the hand and head get tired for long time operation .

  Further, conventionally, the spatial handwritten character graphic input device of the present invention can be applied to a fingertip easily and easily in a situation where there is no planar entity such as a handwritten input panel. The spatial coordinates corresponding to the spatial position associated with the movement of the mounted magnetic sensor are sequentially detected, and the moving speed and acceleration of the magnetic sensor are measured based on the sequentially detected spatial coordinates, and the measured moving speed and In some cases, a trace image drawn by the magnetic sensor is extracted based on the acceleration, and the extracted trace image is stored (for example, see Patent Document 4).

However, in Patent Document 4 described above, spatial coordinates corresponding to the spatial position associated with the movement of the magnetic sensor attached to the fingertip are sequentially detected and spatial handwritten character graphic input is performed, but there is a problem that the magnetic sensor is an essential requirement. .
Conventionally, the movement of the operator's hand can be directly input as the trajectory of the character to be input, and the processing of the input character information such as confirmation, correction and conversion of the input character can be easily performed. In order to provide a character input device that can be used to capture the trajectory when an operator writes a character in the air with his / her fingers, the CCD camera captures the trajectory, and the captured trajectory is recognized as a character by the image processing device. Since the trajectory and recognized characters are displayed on the display device, and the character input by the operator can be performed while confirming the trajectory displayed on the display device, there are some that can be input easily and reliably (for example, (See Patent Document 5).

However, in the above-mentioned patent document 5, although an operator can input characters easily and reliably, the trajectory when the operator writes characters in the air with his / her fingers is captured by the CCD camera, and the captured characters are captured. There is a problem that it is an essential requirement to recognize the locus as a character by the image processing apparatus.
In addition, conventionally, the movement of the operator's hand can be directly input as the trajectory of the character to be input by a simple method without performing complicated operations, and input, such as confirmation, correction and conversion of input characters, can be performed. In order to provide a character input method that can easily process the character information and a character input device that implements the character input method, the operator moves the finger in the air following the character. While the position information is captured by the CCD camera, the image processing device detects whether the finger is in a moving state or a stopped state, and whether the locus drawn by the finger is valid each time the stopped state is detected. In some cases, the invalidity is alternately switched, and only the valid trajectory is taken into the RAM as a part of the trajectory following the shape of the character to be input (see, for example, Patent Document 6).

  However, in the above-mentioned Patent Document 6, it is an indispensable requirement to directly input the movement of the operator's hand as the trajectory of the character to be input as a character input method that can easily process the input character information. There is a problem.

International Publication No. W2003-024064 JP 2004-318793 A JP 2005-092839 A Japanese Patent Laid-Open No. 06-028096 JP-A-10-097361 JP-A-10-154036

  Therefore, in view of the above problems, an object of the present invention is to provide a character input / function operation device, a method, a program, and a mobile terminal device that facilitate character input and function operation.

In order to solve the above problems, the present invention provides a character input / function operating device having a camera and a display unit, a memory for storing a character list screen, and a character that is extracted from the memory and displayed on the display unit. Character input for performing character input based on a list display unit, a finger motion monitoring unit that monitors images of one finger or two fingers acquired by the camera, and an image of one finger monitored by the finger motion monitoring unit And a function operation unit that performs a function operation based on an image of two fingers monitored by the finger motion monitoring unit, and the character input unit is configured such that one finger monitored by the finger motion monitoring unit is In the case of an image that moves parallel to the screen of the display unit, the cursor on the character list screen is moved in the moving direction. In the case of an image in which one finger approaches the screen of the display unit, the cursor is displayed on the character list screen. The character at which In the case of an image that moves in parallel with the screen of the display unit when the two fingers monitored by the finger motion monitoring unit come into contact with each other, the function operation unit sets the input character in the moving direction. Scroll the displayed display content, reduce the display content displayed on the display unit in the case of an image approaching the screen of the display unit when two fingers touch, and Provided is a character input / function operation device characterized by enlarging display contents .

Further, the character list screen is a QWERTY keyboard screen.
Further, the character list screen is a screen of hiragana characters.
In addition, the character list screen Ru screen der of katakana characters.

Further, the display content displayed on the display unit includes an image and a document.
Further, when the character input / function operation device has a keyboard, when an arbitrary key on the keyboard is pressed, the camera is activated and the character input / function operation device starts operation.
Further, a sensor with low power consumption for detecting that a finger is approaching the character input / function operation device is provided, and the camera is activated when the sensor detects the approach of the finger, and the character input / function operation device. Starts operation.

Furthermore, the camera has a fisheye lens as a lens.
Further, the finger motion monitoring unit acquires a finger image with an input device including an ultrasonic sensor and an infrared sensor.
Further, when the character input / function operation device is set to the character input mode, the character list display unit has one finger placed in front of the camera, and the finger motion monitoring unit is the camera. When an image of one finger is acquired, a character list screen is taken out from the memory and displayed on the display unit.

In addition, the character input unit has a cursor on the character list screen in the case of an image in which one finger monitored by the finger motion monitoring unit moves at a certain speed or more, or in the case of an image that is stationary for a certain period of time. The position character is determined as the input character.
Further, the character input unit is configured to display an image of a cursor depending on a moving speed of an image of one finger when an image of one finger monitored by the finger motion monitoring unit moves in parallel with the screen of the display unit. Change the amount of movement and move the cursor on the character list screen in the direction of movement.

Further, the function operation unit depends on the moving speed of the two fingers that are in contact with each other when the two fingers monitored by the finger motion monitoring unit are in contact with each other and move parallel to the screen of the display unit. Then, the scroll amount of the display content displayed on the display unit is changed to scroll the display content displayed on the display unit in the moving direction.
Furthermore, the function operation unit, when the touched two fingers monitored by the finger motion monitoring unit move at a certain speed or more, displays the menu contents as the display content displayed on the display unit and remains stationary for a certain period of time. In the case of an image, the display content is changed to a thumbnail image.

Further, the character input unit displays a cursor over the entire character frame for the character on which the cursor is positioned on the character list screen displayed on the display unit, or includes an arrow, a finger image, and an icon. Display the image as a cursor.
Further, the function operation unit is displayed on the display unit in the moving direction in the case of an image in which one finger monitored by the finger motion monitoring unit moves in parallel after approaching the screen of the display unit. The display contents are scrolled, and in the case of an image that leaves after scrolling, the scrolling is stopped.

Further, the function operation unit may start the scrolling of the display content displayed on the display unit, and if the touched two fingers are separated from each other, or if the one finger is moved quickly, In the case of an image in which the scroll operation is continued and thereafter two fingers are in contact again, or in the case of an image in which one finger is close to the screen of the display unit, the scrolling is stopped.
Furthermore, after the function operation unit starts scrolling the display content displayed on the display unit, the moving speed of the two fingers touched from the image of the two fingers touching, and the images of the two fingers touched The scroll speed reduction rate is changed depending on the distance between the two fingers in contact with the screen and the screen of the display unit or the distance between the image of the one finger and the display unit and the screen of the one finger. .

Further, the function operation unit turns the display content page in the forward direction when the touched two fingers move upward, and turns the page back when the image moves downward.
Further, in the case of an image in which two touching fingers are moved diagonally upward, the function operation unit displays the display content displayed on the display unit horizontally, and the two touching fingers move diagonally downward. In the case of an image, the display content is displayed vertically.

Furthermore, the function operation unit performs screen operations on display contents including a character input document, an existing document, and an image depending on observation of three or more finger states.
Furthermore, the present invention provides a character input and function operation method for a portable phone having a camera and a display unit, a step of stored characters list screen that is displayed on the display unit is taken out, one finger by the camera or 2 one of the steps of image Ru is acquired finger, a step of image of one finger or two fingers acquired is Ru is monitored by the camera, an image in which one of the finger that is monitored is moved parallel to the screen of the display unit in the case of a moving cursor on the character list screen in the moving direction, it monitored one finger input character character position cursor to the character list screen in the case of the image to be closer to the screen of the display unit a deterministic is Ru step, the display content displayed on the display unit in the moving direction when in contact with two fingers which are monitored images moves parallel to the screen of the display unit is scrolled is monitored as Two There display content displayed on the display unit is reduced in the case of the image to be closer to the screen of the display unit in contact, in that it comprises a step in which the display contents Ru is magnified in the case of the image to be separated A character input / function operation method is provided.

Furthermore, the present invention is the camera and the cellular phone character input and function operation program having a display unit, the procedure is Eject is stored characters list screen that is displayed on the display unit, one finger by the camera or a step of images of two fingers Ru is acquired, the procedure in which the image of one finger or two fingers acquired by the camera Ru is monitored, one finger which is monitored is moved parallel to the screen of the display unit If the image is moved the cursor on the character list screen in the moving direction, character position cursor to the character list screen when one finger is monitored image to approach the screen of the display unit input the procedure is Ru determined as a character, the display content displayed on the display unit in the moving direction when in contact with two fingers which are monitored images moves parallel to the screen of the display unit is scrolled, monitoring Was One of the fingers in contact with the case of the image to be closer to the screen of the display unit display content displayed on the display unit is reduced, in the case of an image to be separated comprises a procedure in which the display contents Ru is enlarged A character input / function operation program is provided .

  Furthermore, the present invention provides a mobile phone having a camera and a display unit, a memory for storing a character list screen, a character list display unit for taking out a character list screen from the memory and displaying it on the display unit, and the camera. In the case of an image in which a finger motion monitoring unit that monitors an image of one finger or two fingers and an image in which one finger monitored by the finger motion monitoring unit moves in parallel with the screen of the display unit, characters are moved in the moving direction. Move the cursor on the list screen, and if the image of one finger monitored by the finger motion monitoring unit approaches the screen of the display unit, the character where the cursor is positioned on the character list screen is determined as the input character Display content displayed on the display unit in the moving direction in the case of an image in which two fingers monitored by the character input monitoring unit and the finger movement monitoring unit move in parallel with the screen of the display unit In the case of an image that scrolls and the two fingers monitored by the finger motion monitoring unit come into contact with each other and approaches the screen of the display unit, the display content displayed on the display unit is reduced, and the image is separated Includes a function operation unit for enlarging the display content.

  As described above, according to the present invention, the stored character list screen is taken out and displayed on the display unit, an image of one finger or two fingers is acquired by the camera, and one finger or two acquired by the camera is acquired. The image of one finger is monitored, and in the case of an image in which one monitored finger moves in parallel with the screen of the display unit, the cursor on the character list screen is moved in the moving direction, and one monitored finger is displayed. In the case of an image approaching the screen of the image, the character where the cursor is positioned on the character list screen is confirmed as the input character, and the image is moved in parallel with the screen of the display when the two monitored fingers touch Scrolls the display content displayed on the display unit in the moving direction, and reduces the display content displayed on the display unit in the case of an image in which two monitored fingers touch and approach the screen of the display unit Magnify the display content for images that are separated Because it was Unishi, the user can one character many times there is no need to press a key to enter, from the list of characters displayed on the screen, intuitively move the cursor to select a character.

In addition, since the character is selected by moving the cursor instead of writing and recognizing, not only can the input be performed with a few finger operations, but the possibility of erroneous recognition is low.
In addition, by setting the range of the virtual keyboard to be larger than the screen size, it is possible to reduce the frequency of pressing the adjacent key by mistake, rather than inputting from the software keyboard displayed on the touch panel.

In addition, the display content of images, documents, etc. can be operated more intuitively and quickly than when the key is pressed many times, so it is more efficient within the limited screen size of mobile terminal devices. You can browse the display contents.
Further, since the finger is not touched directly on the screen, there is no risk of the screen being soiled or damaged.
In addition, since there is no stress when a conventional keyboard is pressed and there is no frictional resistance when the touch panel is traced with a finger, smooth operation is possible without placing a burden on the finger.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a perspective view showing an appearance of a mobile terminal device according to the present invention. As shown in the figure, the mobile terminal device 101 includes a keyboard 116, a display unit 118, and a camera 120.
The keyboard 116 has a function of inputting information and operating the mobile terminal device 101.
The display unit 118 is composed of a monochrome or color liquid crystal panel or an organic EL (electrofluorescence).

The camera 120 is a sensor such as a CCD (Charge Coupled Device) and has a function of acquiring an image. As shown in FIGS. Based on the finger imaging result, one finger or two fingers are recognized and used to determine parallel movement, approach and separation of the finger with respect to the screen of the display unit 118.
FIG. 2 is a diagram showing a main part of the circuit configuration of the mobile terminal device 101 in FIG. As shown in the figure, the circuit 101A of the mobile terminal device 101 is provided with a key input unit 115, and the key input unit 115 is an input circuit that receives an input from the keyboard 116.

Further, the circuit 101A is provided with a display control unit 117, and the display control unit 117 is a control circuit of the display unit 118.
Further, the circuit 101 A is provided with a camera control unit 119, and the camera control unit 119 is a control circuit for the camera 120.
Further, the circuit 101A is equipped with a control unit 111. The control unit 111 includes a CPU (Central Processing Unit), and is connected to each unit in the circuit 101A via the bus 112 to control each unit, as will be described later. In addition, a character input / function operation control unit 111A is provided.

Further, the circuit 101A is provided with a ROM (Read Only Memory) 113, which stores various control programs and fixed data to be executed by the control unit 111, and as will be described later, It has a list screen storage unit 113A.
Further, the circuit 101A is provided with a RAM (Random Access Memory) 114. The RAM 114 is a working memory, and stores data temporarily necessary for the CPU to execute the program.

  FIG. 3 is a diagram showing an example of a list of characters stored in the character list screen storage unit 113A of the ROM 113 in FIG. As shown in the figure, the character list screen 118A stored in the character list screen storage unit 113A is a QWERTY keyboard screen, that is, a screen of alphabets, numbers, and the like. As will be described later, an input operation is performed on the display of characters and symbols constituting the character list screen 118A. For the input characters, a document is created by using the “conversion”, “confirmation”, “space”, “backspace”, and “delete” keys on the character list screen 118A.

  FIG. 4 is a diagram showing another list example of characters stored in the character list screen storage unit 113A of the ROM 113 in FIG. As shown in this figure, the character list screen 118B stored in the character list screen storage unit 113A is a screen of hiragana characters of 50 sounds, and includes a screen (not shown) of katakana characters. An input operation is performed on the display of characters and symbols constituting the character list screen 118B as will be described later. For the input characters, a document is created using the “conversion”, “confirm”, “space”, “backspace”, and “delete” keys on the character list screen 118B.

FIG. 5 is a diagram illustrating an arrangement in which a list of characters stored in the character list screen storage unit 113A in FIG. 3 or 4 is displayed on the display unit 118. As shown in the figure, the display unit 118 displays, for example, a character list screen 118A or 118B at the bottom of the screen, and the input characters are displayed at the top of the screen.
FIG. 6 is a block diagram showing a schematic configuration of the character input / function operation control unit 111A of the control unit 111 in FIG.

As shown in the figure, the character input / function operation control unit 111A includes a character list display unit 201, a finger motion monitoring unit 202, a character input unit 203, and a function operation unit 204.
The character list display unit 201 takes out the character list screen 118 </ b> A or 118 </ b> B shown in FIG. 3 or 4 from the ROM 113 and displays it on the display unit 118 when inputting characters to the mobile terminal device 101.

When the user brings one finger or two fingers close to the screen on which the character list screen 118 </ b> A or 118 </ b> B is displayed on the display unit 118 of the mobile terminal device 101, the finger motion monitoring unit 202 An image of two touched fingers is analyzed and recognized, and one finger or two touched fingers move in parallel with the screen, and an operation of approaching or moving away from the screen is monitored.
For example, a representative image of one finger or two fingers is stored in the finger motion monitoring unit 202 and is acquired by the camera 120 in comparison with the stored image of one finger or two fingers. Recognize images.
Representative images are also stored for the contact state and the non-contact state between the images of two fingers, and an image acquired by the camera 120 is recognized in comparison with the stored image.
Further, the finger movement monitoring unit 202 is configured to be parallel to the screen based on the movement of the image of one finger or two fingers acquired by the camera 120 to the left and right by continuously comparing the previous and current images. Monitor the direction of travel.
Further, the finger motion monitoring unit 202 continuously compares the previous and current images of one finger or two fingers acquired by the camera 120, and the approach is based on a state where the image becomes large and small. , Monitor the separation.
Based on the monitoring of the finger motion monitoring unit 202, the character input unit 203 displays a character when one finger moves in a direction parallel to the screen on which the character list screen 118A or 118B is displayed on the display unit 118 on the mobile terminal device 101. When the cursor on the list of characters displayed on the list screen 118A or 118B is scrolled in the direction of movement of one finger and one finger moves in the direction approaching the screen, the character at which the cursor is positioned is input character. Confirm as

  The function operation unit 204 is based on the monitoring of the finger motion monitoring unit 202 with respect to the display content displayed on the display unit 118 in the mobile terminal device 101, and moves in the direction of movement when two touched fingers move in parallel with the screen. If the two touched fingers move in the direction approaching the screen, the display content is reduced to display a wider range of content, and the two fingers move away from the screen. In such a case, the display contents are enlarged and detailed contents are enlarged.

FIG. 7 is a flowchart for explaining a series of operation examples of the character input / function operation control unit 111A of the control unit 111 in FIG. As shown in the figure, in step S301, the character list display unit 201 of the character input / function operation control unit 111A displays the character list screen 118A or 118B on the display unit 118.
In step S <b> 302, the finger motion monitoring unit 202 monitors the motion of the user's finger based on the image of the camera 120.

In step S303, the finger motion monitoring unit 202 confirms the presence of one finger of the user. If it cannot be confirmed, the process proceeds to step S309.
In step S304, when the presence of one finger can be confirmed, the character input unit 203 confirms the movement of the finger. If it cannot be confirmed, the process returns to step S302 and the above processing is repeated.

In step S <b> 305, the character input unit 203 determines whether the moving direction of one finger is parallel to the screen of the display unit 118 when the finger moves. If not parallel, the process proceeds to step S307.
In step S306, if the characters are parallel, the character input unit 203 moves the cursor in the moving direction and ends the process.

In step S <b> 307, the character input unit 203 determines whether the finger moving direction is approaching the screen of the display unit 118. If it is not approaching, the process is terminated.
In step S308, if the character input unit 203 is approaching, the character at the cursor position is determined as the input character, and the process ends.
By performing such processing, it becomes possible to input a character, and it is possible to make a space for the input character or delete the input character and perform character input again to convert the input character.

In step S309, the finger motion monitoring unit 202 confirms the presence of the user's two fingers. If it cannot be confirmed, the process is terminated.
In step S310, the finger motion monitoring unit 202 recognizes the positional relationship between the two fingers and determines whether the two fingers are in contact with each other. If not, the process returns to step S309 and the above process is repeated.

In step S <b> 311, when two fingers are in contact, the function operation unit 204 determines whether the movement direction of the two fingers is parallel to the screen of the display unit 118. If not, the process proceeds to step S313.
In step S312, the function operation unit 204 scrolls the display content of the display unit 118 in the moving direction of the two fingers, and ends the process.

In step S313, the function operation unit 204 determines whether the moving direction of the two fingers is approaching or moving away from the screen of the display unit 118. If so, the process proceeds to step S315.
In step S314, if the moving direction of the two fingers is close to the screen, the function operation unit 204 reduces the display content of the screen displayed on the display unit 118 and ends the processing.

In step S315, if the movement direction of the two fingers is separated from the screen, the function operation unit 204 enlarges the display content of the screen displayed on the display unit 118, and ends the process.
Therefore, according to the present invention, it is not necessary for the user to press a key many times to input one character, and the user can intuitively move the cursor and select a character from the list of characters displayed on the screen. Can do. In addition, since the character is selected by moving the cursor instead of writing and recognizing, not only can the input be performed with a few finger operations, but the possibility of erroneous recognition is low. In addition, by setting the range of the virtual keyboard to be larger than the screen size, it is possible to reduce the frequency of pressing the adjacent key by mistake, rather than inputting from the software keyboard displayed on the touch panel.

In addition, the display content of images, documents, etc. can be operated more intuitively and quickly than when the key is pressed many times, so it is more efficient within the limited screen size of mobile terminal devices. You can browse the display contents.
Further, since the finger is not touched directly on the screen, there is no risk of the screen being soiled or damaged. In addition, since there is no stress when a conventional keyboard is pressed and there is no frictional resistance when the touch panel is traced with a finger, smooth operation is possible without placing a burden on the finger.

  In order to suppress power consumption, the camera may be activated when the keyboard of the mobile terminal device is pressed, and the input operation may be started. Alternatively, the camera may be activated when it is detected that a finger is approaching by a sensor including an ultrasonic sensor or an infrared sensor as a sensor with low power consumption.

  Furthermore, in the present invention, an image of a finger is acquired by a camera. However, the motion of the finger may be recognized by an ultrasonic sensor, an infrared sensor, or other input devices. Further, a fish-eye lens may be provided as a camera lens so that a wider range of finger movements can be observed.

  When the portable terminal device is in the character input mode, if a finger is put out in front of the portable terminal device, a character list may be displayed on the screen.

  In the present invention, the character specified by the cursor on the character list is fixed when the finger is moved closer to the mobile terminal device, but when the finger is moved at a certain speed or more, Alternatively, it may be determined when the finger is stationary for a predetermined time or more. Further, the amount of movement of the cursor may be changed depending on the speed of moving the finger. Furthermore, the input method according to the present invention may be applied not only to character input but also to selection of menus and thumbnail images.

  To represent a selected character, it is conceivable to display a cursor over the entire frame of one character, highlight a character, or the like. In addition, an arrow, a finger image, or another icon image may be displayed as a cursor to indicate which position on the screen is designated.

  To manipulate the display content of images, documents, etc., move one finger close to the mobile terminal device and then move it in the direction parallel to the screen, the display content will scroll, and when it leaves, the scrolling will be stopped. May be.

  The scrolling operation may be continued when two fingers are released or one finger is quickly moved in the direction of scrolling after the scrolling of the display contents is started by the operation of the present invention. Thereafter, scrolling may be stopped when an operation of bringing two fingers into contact again or an operation of bringing the finger close to the mobile terminal device is performed. At that time, the scroll speed reduction rate may be changed depending on the speed of the operation and the distance between the finger and the mobile terminal device.

  Further, when the image of the two fingers touching each other moves upward, the page of the display content may be turned in the advancing direction, and when moved downward, the page may be turned in the returning direction.

  Further, when the screen is moved obliquely upward with two fingers in contact with each other, the screen is converted into a horizontal display, and when it is moved obliquely downward, it is converted into a vertical display.

  In addition, in character input and operation of display contents such as an image and a document, the screen operation may be performed by observing not only two fingers but also the states of three or more fingers.

  A three-dimensional space may be displayed on the screen, and the user may operate the cursor in the space by moving a finger. Further, when an operation of pinching with two fingers or an operation of grasping with a hand, the cursor may artificially grab the object in the space displayed on the screen. Further, when the finger of the hand is moved, the object in the space can be moved. Thereafter, when the two fingers are released or the hand is opened, the object in the screen may be released from the cursor.

  The mobile terminal device has been described above. However, the present invention is not limited to this, and includes mobile phones, PHS (simple mobile phones), PDAs (mobile terminal devices), images such as character input functions, viewers, editors, browsers, documents, etc. The present invention can also be used for an apparatus having a function of displaying.

It is a perspective view which shows the external appearance of the portable terminal device which concerns on this invention. It is a figure showing the principal part of the circuit structure of the portable terminal device 101 in FIG. It is a figure which shows the example of a list of the characters preserve | saved at the character list screen preservation | save part 113A of ROM113 in FIG. It is a figure which shows another example of a list of the character preserve | saved at the character list screen preservation | save part 113A of ROM113 in FIG. FIG. 5 is a diagram illustrating an arrangement in which a list of characters stored in a character list screen storage unit 113A in FIG. 3 or 4 is displayed on a display unit 118. It is a block diagram which shows schematic structure of 111 A of character input and function operation control parts of the control part 111 in FIG. 2 is a flowchart illustrating a series of operation examples of a character input / function operation control unit 111A of the control unit 111 in FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 ... Portable terminal device 101A ... Circuit 111 ... Control part 111A ... Character input and function operation control part 112 ... Bus 113 ... ROM
114 ... RAM
115 ... Key input unit 116 ... Keyboard 117 ... Display control unit 118 ... Display unit 118A, 118B ... Character list screen 118C ... Character input screen 119 ... Camera control unit 120 ... Camera 201 ... Character list display unit 202 ... Finger movement monitoring unit 203 ... Character input unit 204 ... Function operation unit

Claims (24)

In a character input / function operation device having a camera and a display unit,
Memory to save the character list screen,
A character list display unit that takes out a character list screen from the memory and displays it on the display unit;
A finger motion monitoring unit that monitors an image of one finger or two fingers acquired by the camera;
A character input unit for inputting characters based on an image of one finger monitored by the finger motion monitoring unit;
A function operation unit that performs a function operation based on an image of two fingers monitored by the finger motion monitoring unit ;
The character input unit moves the cursor on the character list screen in the moving direction when one finger monitored by the finger motion monitoring unit moves in parallel to the screen of the display unit. Is an image approaching the screen of the display unit, the character at which the cursor is positioned on the character list screen is determined as an input character, and the function operation unit is configured to display two fingers monitored by the finger motion monitoring unit. In the case of an image that touches and moves parallel to the screen of the display unit, the display content displayed on the display unit is scrolled in the moving direction, and two fingers touch and approach the screen of the display unit The character input / function operation device is characterized in that the display content displayed on the display unit is reduced in the case of, and the display content is enlarged in the case of a separated image . The character input / function operation device according to claim 1, wherein the character list screen is a QWERTY keyboard screen. The character input / function operation device according to claim 1, wherein the character list screen is a screen of hiragana characters. The character input / function operating device according to claim 1, wherein the character list screen is a katakana character screen. The character input / function operation device according to claim 1, wherein the display content displayed on the display unit includes an image and a document. 2. The character input / function operation device according to claim 1, wherein, when the character input / function operation device has a keyboard, the camera is activated when any key of the keyboard is pressed, and the character input / function operation device starts operation. The character input / function operation device described. A sensor with low power consumption for detecting that a finger is approaching the character input / function operation device is provided, and the camera is activated when the sensor detects the approach of the finger, and the character input / function operation device operates. The character input / function operation device according to claim 1 , wherein the character input / function operation device is started. The character input / function operation device according to claim 1, wherein the camera has a fisheye lens as a lens. The character input / function operation device according to claim 1, wherein the finger motion monitoring unit acquires a finger image with an input device including an ultrasonic sensor and an infrared sensor. In the character input / function operation device, when the character input mode is set, the character list display unit has one finger placed in front of the camera, and the finger motion monitoring unit has one camera. The character input / function operation device according to claim 1, wherein when a finger image is acquired, a character list screen is extracted from the memory and displayed on the display unit. In the character input unit, the cursor is positioned on the character list screen in the case of an image in which one finger monitored by the finger motion monitoring unit moves at a certain speed or more, or in the case of an image that remains stationary for a certain period of time. The character input / function operating device according to claim 1 , wherein the character is determined as an input character. In the case of an image in which one finger monitored by the finger motion monitoring unit moves in parallel with the screen of the display unit, the character input unit determines the amount of movement of the cursor depending on the moving speed of one finger image. The character input / function operation device according to claim 1 , wherein the cursor on the character list screen is moved in the moving direction by changing the character. In the case of an image in which the two fingers monitored by the finger motion monitoring unit come into contact with each other and move in parallel with the screen of the display unit, the function operation unit depends on the moving speed of the two touched fingers. wherein the scrolling of the display unit to the displayed display content in the moving direction by changing the scroll amount of the display contents displayed on the display unit, a character input and function operating device according to claim 1. In the case of an image in which two touched fingers monitored by the finger motion monitoring unit move at a certain speed or more, the function operation unit converts the display content displayed on the display unit to a menu image, and is an image that is stationary for a certain time or more. wherein characterized by the thumbnail image to display content, a character input and function operating device according to claim 1 in the case of. The character input unit displays a cursor on the entire character frame with respect to a character on which the cursor is positioned on the character list screen displayed on the display unit, or an image including an arrow, a finger image, and an icon. The character input / function operation device according to claim 1 , wherein the character input / function operation device is displayed as a cursor. In the case of an image in which one finger monitored by the finger motion monitoring unit moves in parallel after approaching the screen of the display unit, the function operation unit displays the display content displayed on the display unit in the moving direction. The character input / function operation device according to claim 1 , wherein scrolling is stopped in the case of an image that is scrolled and the image leaves after scrolling. The function operation unit performs a scroll operation in the case of an image in which two touched fingers are separated after starting scrolling of display contents displayed on the display unit, or in the case of an image in which one finger moves quickly There continues, then, if the image two finger touches again, or one finger in the case of an image close to the screen of the display unit is characterized by stopping the scrolling to claim 1 The character input / function operation device described. After the function operation unit starts scrolling the display content displayed on the display unit, the moving speed of the two fingers touched from the image of the two touched fingers moving, and the touch of the images of the two touched fingers The scroll speed reduction rate is changed depending on the distance between the two fingers and the screen of the display unit or the distance between one finger and the screen of the display unit from the image of one finger. The character input / function operating device according to claim 1 , wherein The function operation unit is characterized in that in the case of an image in which two touched fingers move in the upward direction, the display content page turns in the forward direction, and in the case of an image in the downward direction, the page turns in the return direction. The character input / function operation device according to claim 1 . In the case of an image in which two touched fingers are moved obliquely upward, the function operation unit displays the display content displayed on the display unit horizontally, and the image in which the two touched fingers move obliquely downward The character input / function operation device according to claim 1 , wherein in the case of the display, the display content is displayed vertically. 2. The character according to claim 1, wherein the function operation unit performs a screen operation of display contents including a character input document, an existing document, and an image depending on observation of three or more finger states. Input / function operation device. In a character input / function operation method for a mobile phone having a camera and a display unit,
A step that will be displayed on the display unit stored characters list screen is extracted,
A step of image of one finger or two fingers Ru acquired by the camera,
A step of image of one finger or two fingers acquired is Ru is monitored by the camera,
Cursor on the character list screen is moved in the moving direction when one finger is monitored images moves parallel to the screen of the display unit, one finger is monitored approaches the screen of the display unit a step of character position cursor to the character list screen is Ru is determined as an input character in the case of an image,
In the case of an image in which two monitored fingers touch and move in parallel with the screen of the display unit, the display content displayed on the display unit is scrolled in the moving direction, and the two monitored fingers touch. the display contents displayed on the display unit is reduced in the case of the image approaches the screen of the display unit, when the image to be separated is characterized by having a step of the display contents Ru is enlarged Te Character input / function operation method. In a character input / function operation program for a mobile phone having a camera and a display unit,
And procedures that saved character list screen is out takes that are displayed on the display unit,
A step of image of one finger or two fingers Ru acquired by the camera,
A step of image of one finger or two fingers acquired is Ru is monitored by the camera,
Cursor on the character list screen is moved in the moving direction when one finger is monitored images moves parallel to the screen of the display unit, one finger is monitored approaches the screen of the display unit and procedures that character to position the cursor to the character list on the screen is Ru is confirmed as the input characters in the case of an image,
In the case of an image in which two monitored fingers touch and move in parallel with the screen of the display unit, the display content displayed on the display unit is scrolled in the moving direction, and the two monitored fingers touch. the display contents displayed on the display unit is reduced in the case of the image approaches the screen of the display unit, when the image to be separated is characterized by having a procedure in which the display contents Ru is enlarged Te Character input / function operation program . In a mobile phone having a camera and a display unit,
Memory to save the character list screen,
A character list display unit that takes out a character list screen from the memory and displays it on the display unit;
A finger motion monitoring unit that monitors an image of one finger or two fingers acquired by the camera;
In the case of an image in which one finger monitored by the finger motion monitoring unit moves parallel to the screen of the display unit, the cursor on the character list screen is moved in the moving direction, and is monitored by the finger motion monitoring unit. In the case of an image in which one finger approaches the screen of the display unit, a character input unit that determines a character at which the cursor is positioned on the character list screen as an input character;
In the case of an image in which two fingers monitored by the finger motion monitoring unit come into contact and move parallel to the screen of the display unit, the display content displayed on the display unit is scrolled in the moving direction, and the finger motion is In the case of an image in which two fingers monitored by the monitoring unit come into contact with each other and approach the screen of the display unit, the display content displayed on the display unit is reduced, and in the case of an image that is separated, the display content is displayed. A cellular phone comprising: an enlarged function operation unit.

Images