JP2002049455A - Input interface device and portable information equipment equipped with the device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an input interface device for allowing an operator to input data by drawing virtual characters or graphics in a space by using his or her finger for portable information equipment equipped with an imaging means capable of imaging visible light beams without using any equipment to be mounted on the finger of an operator for measuring the movement of his or her finer or any special sensor capable of detecting infrared rays or a body temperature. SOLUTION: A locus extracting means 7 divides the picture of the finger of an operator inputted by an imaging means 6 capable of photographing visible light beams into plural areas, and extracts the locus of the movement of his or her finger by comparing the temporally consecutive images by the area units. Also the locus extracting means 7 extracts the locus of the finger of the operator by searching the outline from a reference point set inside the finger in the picture toward the outside. Also, the locus extracting means 7 extracts the locus of the movement of the finger of the operator by extracting an area including the reference point set inside his or her finger in the picture whose all pixel values are present within a specific range.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an input interface device for inputting an operator's instruction to electronic equipment, and more particularly to a portable information device such as a portable telephone or a portable terminal device equipped with a camera. The present invention relates to an input interface device that can input a trajectory.

[0002]

2. Description of the Related Art Information exchange by e-mail or the like using a mobile phone or a portable terminal device is rapidly spreading. In addition, e-mail can handle not only character data but also image data taken by digital still cameras, hand-drawn pictures and characters, etc., and portable information devices equipped with a small camera for image input Has also appeared.

In such a portable information device, a pen input using a touch panel is mainly used as an input interface for inputting hand-drawn pictures, characters, and the like. A pen input using this touch panel can set a work place for input on a liquid crystal screen, and thus can be said to be more suitable for a portable information device than a mouse or the like which requires a work place on a desk.

However, since the size of the input work area depends on the area of the touch panel, there is a problem that it is difficult to simultaneously secure operability and achieve downsizing of the device. That is, if the size of the touch panel is reduced in accordance with the downsizing of the device, the input operation must be performed on the small touch panel, so that input becomes difficult.

[0005] Therefore, in order to guarantee the convenience of the operator, the touch panel must have a certain area, which hinders the miniaturization of the portable information device.

As an input interface for solving such a problem, as disclosed in Japanese Patent Publication No. 4-15489, Japanese Patent Application Laid-Open No. 6-28096, and Japanese Patent Application Laid-Open No. 10-97361, virtual There are devices for inputting by drawing typical characters and figures. According to these devices, a virtual work space can be provided in the space, so that a work space of a required size can be provided regardless of the size of the device. Therefore,
The convenience of the operator can be guaranteed at the same time as the size of the portable information device is reduced.

[0007]

In the prior art as described above, special equipment is required to measure the movement of a finger in space. For example, in Japanese Patent Publication No. 4-15489, a light emitting device and a light receiving device must be mounted on a fingertip and a wrist, and in Japanese Patent Application Laid-Open No. 6-28096, a sensor must be mounted on a fingertip.

Further, in Japanese Patent Application Laid-Open No. 10-97361, an LED is attached to a fingertip, or a special sensor such as an infrared camera or a pyroelectric sensor capable of detecting infrared rays and body temperature is required.

However, there is a problem that the mounting of these special devices imposes a heavy burden on the operator and impairs a natural operation feeling. Due to the nature of portable information devices, where and how to store these devices that are not used when handwritten characters or graphics are not input is a major issue. The need for dedicated special equipment may hinder miniaturization of the equipment, which may lead to an increase in cost.

SUMMARY OF THE INVENTION Accordingly, the present invention has been made to solve the above-mentioned problem, and an apparatus for measuring the movement of a finger of an operator is not attached to a finger, and infrared rays and body temperature can be detected. It is an object of the present invention to provide an input interface device for a portable information device that can input a trajectory of finger movement without using a special sensor.

[0011]

According to a first aspect of the present invention, there is provided an input interface device, wherein an image pickup means for inputting a finger movement of an operator, and a locus of finger movement is extracted from an image input by the image pickup means. And a trajectory extracting means for performing the trajectory extraction.

According to a second aspect of the present invention, in the input interface device according to the first aspect,
The imaging unit is an imaging unit capable of capturing visible light.

According to a third aspect of the present invention, in the input interface device according to any one of the first to second aspects, the trajectory extracting means is provided with a device for extracting a finger position. A trajectory of finger movement can be extracted from an image of a finger that has not been picked up by image processing.

According to a fourth aspect of the present invention, in the input interface device of the third aspect,
The image processing used in the trajectory extraction means divides an image input from the imaging means into a plurality of regions, and performs comparison in units of regions divided between temporally continuous images.

According to a fifth aspect of the present invention, in the input interface device of the fourth aspect,
The image processing used in the trajectory extracting means compares a series of temporally continuous images with a reference image, and extracts a trajectory of finger movement between images in which the number of regions moved in the same direction is maximized. It is characterized by the following.

According to a sixth aspect of the present invention, in the input interface device according to the third aspect,
The image processing used in the trajectory extracting means is characterized in that a contour of a finger is extracted by searching for a contour from a reference point set inside a finger outward.

According to a seventh aspect of the present invention, in the input interface device according to the third aspect,
The image processing used in the trajectory extracting means extracts a region including a reference point set inside a finger and having all pixel values within a specified range.

According to an eighth aspect of the present invention, in the input interface device according to any one of the sixth to seventh aspects, the image input by the imaging means and the movement of the finger extracted by the trajectory extraction means. Display means for displaying the trajectory of the image, and by superimposing and displaying the image input by the imaging means and the graphic indicating the position of the reference point on the display means, at the start of the trajectory extraction, the operator can set the reference point with the finger. It is characterized in that it can be positioned inside.

According to a ninth aspect of the present invention, there is provided a portable information device including the input interface device according to any one of the first to eighth aspects.

[0020]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A preferred embodiment of the present invention will be described with reference to FIGS.

FIG. 1 is an image diagram of an embodiment of the input interface device of the present invention.

1 is a portable information device provided with the input interface of the present invention. Reference numeral 2 denotes an imaging device capable of capturing visible light, specifically, a CCD camera or
This applies to CMOS sensors. The imaging device 2 may function as an imaging unit of the input interface device of the present invention, and may function as a normal digital still camera when handwritten characters and graphics are not input. Reference numeral 3 denotes a display device, which specifically corresponds to a liquid crystal display.
Reference numeral 4 denotes an input button for operating the portable information device. Reference numeral 5 denotes an operator's finger.

FIG. 2 is a schematic block diagram of the input interface device of the present invention.

Reference numeral 6 denotes imaging means for imaging the fingers of the operator. Reference numeral 7 denotes a trajectory extraction unit that detects the movement of the operator's finger from the image data captured by the imaging unit 6 and extracts the trajectory. Reference numeral 8 denotes display means for displaying the trajectory of the finger movement of the operator extracted by the trajectory extraction means 7.

FIG. 3 is a diagram showing an operation procedure of the trajectory extracting means of the present invention. The operation will be described with reference to the image area shown in FIG.

First, in step 9, an image of the operator's finger is input, and in step 10, the image is divided into a plurality of regions, and the feature amount of each region is calculated. Here, the size of the area is 1
Any size from pixels to the same size as the image is possible, but since the size of the area is the unit of the size of the finger movement to be extracted, it can be set according to the size of the finger movement to be extracted desirable.

Here, the size of the image is 640 × 480 pixels, and the size of the region is 32 × 32 pixels. Further, as the feature amount of the region, an average value of pixel values included in the region, a combination of feature amounts of a plurality of small regions obtained by further dividing the region, and the like can be considered.

Next, in step 11, the operator's finger image is input again, and in step 12, the feature amount of the image is calculated as in step 10. Image A input in step 9
(See FIG. 4) and the image B (FIG.
Reference) is an image captured at a short interval of, for example, about 1/30 second while the operator moves his / her finger.

These images are obtained by imaging a finger at a short distance of about several centimeters from a lens, and there is a possibility that something other than a finger may be shown.
It can be assumed that it occupies a large area.

Next, in step 13, the feature amount of each area of the image A and the image B is compared to examine how the finger has moved in the imaging interval of the image A and the image B. For example,
As shown in FIG. 4, the feature amount of the area a of the image A is
If the feature quantity of the area b is the most similar to the feature quantity of each area, it is considered that the object shown in the area a of the image A has moved to the area b in the image B.

That is, in this example, it can be considered that the object shown in the area a of the image A has moved to the right in the image B by the size of the area. Further, the feature amount of the region a of the image A is compared with the feature amount of each region of the image B, and if the region b ′ is most similar, the object shown in the region a of the image A is moving. You can think that there is no.

Here, when comparing the feature amount of each area of the image A and the image B, if the magnitude of the finger movement can be predicted in advance, the area to be compared can be reduced. For example, assuming that the magnitude of the movement of the finger is smaller than the size of the block, the movement destination of the region of the image A is limited to eight regions that are located diagonally up, down, left and right in the image B.

Therefore, if each area of the image A is compared not with all the areas of the image B but only with eight areas adjacent to each area, the processing can be speeded up and an erroneous response can be achieved. The possibility of detection can be reduced.
Here, the movement destination of the area of the image A is limited to the adjacent area.

Since the finger is largely shown in the image, the direction in which the number of moved areas is the largest among the nine directions including the eight directions (up, down, left, right, and oblique) and the case where the finger does not move is defined as the finger movement direction. Can be considered. In this case, the magnitude of the extracted finger movement corresponds to one region.

However, in the images A and B, the finger does not always move by one area. For example, image A
In the case where the finger does not move corresponding to one region in the image and the image B, if a new image is required, a plurality of images are taken and compared with the image A, and the finger is moved corresponding to one region. It is necessary to extract the direction of movement when performing

Here, when the movement of the finger exceeds one area, the movement is not extracted. Therefore, when comparing the image A with the newly captured image, as the movement of the finger approaches the one area, The number of areas that are considered moved has increased,
Conversely, the number should decrease when it exceeds one area.

Therefore, when the number of regions regarded as moving reaches the maximum, it can be regarded that the finger has moved by one region. The above processing is realized in step 14 and subsequent steps.

First, in step S14, the image C is input. Next, in step 16, the feature amount of each area of the image A and the image C is compared, and the number M of areas moved in the direction in which the number of moved areas is the largest is calculated.

In step 17, step 1
The number N and the number of areas moved in the direction in which the number of areas moved between the image A and the image B calculated in step 3 are the largest are compared. If M is smaller than N, the movement between the image A and the image B Since the number of regions obtained is maximum, it is considered that the finger has moved by one region in the images A and B.

Therefore, the trajectory is displayed in step 18 based on the extracted moving direction. If M is equal to or greater than N in step 17, the values of the images B and N are updated in step 19. That is, the image B is replaced with the image C, and N = M. Then, the process returns to step S14.

In step 20, it is confirmed whether or not the trajectory extraction processing is to be continued.
Update the value of. That is, image A is replaced with image B, image B is replaced with image C, and N = M. Then, the process returns to step S14.

If continue is not selected in step 20,
The trajectory extraction processing ends. FIG. 5 is a diagram showing another operation procedure of the trajectory extraction means of the present invention.

First, in step 22, an image of the finger of the operator is input, and in step 23, the image of the finger is displayed on the display device 3, and as shown in FIG. The contour of the finger is extracted by searching the contour outward from “x” shown in FIG. 6). As a method of setting the reference point inside the finger, an image input by the imaging device 2 and a figure indicating the reference point (“X” shown in FIG. 6) are displayed on the display device 3 in a superimposed manner. A method of instructing the start of trajectory extraction by the input button 4 after the mark indicating the reference point is positioned inside the finger may be considered. At this time, if the mobile device is not provided with an automatic focusing unit (autofocus), the distance between the imaging device 2 and the finger must be adjusted to the focus matching distance, or the outline of the finger displayed on the display device 3 will be displayed. For blurring, it is conceivable to use a rectangle whose lower part is open as shown in FIG. 6 and to set the fingers so that the fingers coincide with the rectangle.

The contour can be extracted by comparing the brightness of adjacent pixels and finding a point where the brightness changes greatly. If the contour is searched from the inside of the finger, the contour found first can be regarded as the contour of the finger, so that the stable contour extraction independent of the background can be performed.

In step 24, the position of the finger is calculated from the outline of the finger extracted in step 23, and the locus of movement is displayed. In step 25, a point to be a reference point of the next image is set from the outline of the finger extracted in step 23.
This reference point must be a point located inside the finger in the next image, but if the moving distance of the finger in two consecutive images is small, it is expected that most of the finger area will overlap. Therefore, for example, the center of the fingertip of the index finger can be set as the reference point of the next image.

At step 26, the continuation of the trajectory extraction processing is confirmed. Step 26
If the continuation is not selected in, the trajectory extraction processing ends.
FIG. 7 is a diagram showing another operation procedure of the trajectory extraction means of the present invention.

First, in step 27, an image of the operator's finger is input, and in step 28, a finger region is extracted from the reference point set inside the finger outward by using region information using color information. The extraction of the finger region searches for a region that includes the reference point and in which all the pixel values are within the specified range. Specifically, first, an area formed by points whose pixel values are within a specified range among points adjacent to the reference point is regarded as a finger area.

Next, if the pixel values of points adjacent to the outside of the finger area are within the specified range, these points are also regarded as finger areas. This is repeated until the pixel values of all points adjacent to the outside of the finger region are outside the specified range. As the specified range of the pixel value, it is conceivable to set a range close to the reference value using the pixel value of the reference point or the average of the pixel values in the finger region as the reference value.

In step 29, the position of the finger is calculated from the finger region extracted in step 28, and the locus of movement is displayed. In step 30, a point to be a reference point of the next image is set from the finger region extracted in step. This reference point can be set in the same manner as in step 25.

In step 31, it is confirmed that the trajectory extraction process is to be continued, and if it is, the process returns to step 27. Step 31
If the continuation is not selected in, the trajectory extraction processing ends.

[0051]

As is apparent from the above description, according to the first to ninth aspects of the present invention, it is possible to detect a device mounted on the operator's finger to measure the movement of the finger, infrared rays, body temperature, and the like. A handwriting input interface can be provided in a portable information device including an imaging unit capable of imaging visible light without using a special sensor.

Further, since a work space virtually set in the space can be provided, the size of the portable information device can be reduced.

The trajectory of the finger movement extracted according to the present invention can be used as a handwritten character or graphic, or can be used as an input to character recognition means.

[Brief description of the drawings]

FIG. 1 is an image diagram of an input interface device of the present invention.

FIG. 2 is a schematic configuration diagram of an input interface device of the present invention.

FIG. 3 is a diagram showing an operation procedure of a trajectory extracting means of the present invention.

FIG. 4 is a diagram showing an example of comparison of a divided area of an image in the trajectory extraction means of the present invention.

FIG. 5 is a diagram showing another operation procedure of the trajectory extraction means of the present invention.

FIG. 6 is a diagram showing an input image and reference points.

FIG. 7 is a diagram showing another operation procedure of the trajectory extraction means of the present invention.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Portable equipment 2 ... Imaging device 3 ... Display device 4 ... Input button 5 ... Finger of an operator

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) H04M 11/00 302 H04B 7/26 109T F-term (Reference) 5B057 BA02 CA08 CA12 CA16 CB08 CB12 CB16 CC03 CH16 DB02 DB09 DC16 DC17 DC22 5K027 AA11 BB01 FF01 FF22 HH00 5K067 EE02 FF23 FF31 KK00 5K101 KK20 LL12 NN06 NN18 NN22 5L096 BA20 CA14 FA06 FA32 HA05

Claims (9)

[Claims] 1. An input interface, comprising: an imaging unit for inputting a finger movement of an operator; and a trajectory extraction unit for extracting a trajectory of a finger movement from an image input by the imaging unit. apparatus. 2. The input interface device according to claim 1, wherein said imaging means is an imaging means capable of imaging visible light. 3. The input interface device according to claim 1, wherein said trajectory extracting means is configured to extract an image of an image of a finger not wearing a device for extracting a position of the finger. An input interface device capable of extracting a trajectory of a finger movement by processing. 4. The input interface device according to claim 3, wherein the image processing used in said trajectory extracting means divides an image input from said image pickup means into a plurality of regions and divides the images between temporally continuous images. An input interface device, wherein the comparison is performed on a region-by-region basis. 5. The input interface device according to claim 4, wherein the image processing used in said trajectory extracting means is compared with a series of temporally continuous images, and the number of areas moved in the same direction is determined. An input interface device for extracting a locus of a finger movement between images having a maximum value. 6. The input interface device according to claim 3, wherein the image processing used by said trajectory extracting means searches for a contour line from a reference point set inside the finger to the outside, thereby forming a contour line of the finger. An input interface device for extracting. 7. The input interface device according to claim 3, wherein the image processing used in said trajectory extracting means extracts a region including a reference point set inside a finger and having all pixel values within a specified range. An input interface device, comprising: 8. The input interface device according to claim 6, further comprising a display unit that displays an image input by the imaging unit and a trajectory of a finger movement extracted by the trajectory extraction unit. By superimposing and displaying the image input by the imaging means and the graphic indicating the position of the reference point on the display means, it is possible for the operator to position the reference point inside the finger at the start of the trajectory extraction. An input interface device characterized by the above-mentioned. 9. A portable information device comprising the input interface device according to claim 1. Description: