JP2001092590A - Optical pointing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To use an optical image sensor also for functions except cursor display control as well. SOLUTION: This device has an optical image sensor 1 for reading the image of finger or the like touching a protecting cover 13, delay circuit 2 for outputting image data from the optical image sensor 1 after delaying them for a fixed time A, comparator 3 for detecting the difference between the image data from the optical image sensor 1 and the moving data from the delay circuit 2 and outputting the moving amount and moving direction of an image corresponding to the above difference to the main body of personal computer(PC) as pointing data PD, timer 4 for starting a pattern recognizing engine 5 and issuing an interruption to the main body of PC when there is no output from the computer 3 longer than fixed time B, and pattern recognizing engine 5 to be started from the timer 4 for extracting a feature by performing pattern recognition on the basis of the image data from the optical image sensor 1, and outputting this feature to the main body of PC as feature extraction data TD.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an optical pointing device used for controlling cursor display on a display in an information processing apparatus such as a personal computer, and more particularly to other functions in addition to a cursor display control function. The present invention relates to an optical pointing device that is also realized.

[0002]

2. Description of the Related Art Various pointing devices for controlling cursor display for moving a cursor on a display of an information processing apparatus are used due to their use environment and operability. For example, in a notebook personal computer, a glide point type pointing device in which an operator's finger is brought into contact with an operation surface and the finger slides in a direction in which a cursor is to be moved has become mainstream in recent years. As the glide point type pointing device, an electrostatic sensor shown in FIG. 5 for reading and pointing a change in capacitance due to a touch of an operator's finger, a pressure sensor for reading a change in pressure due to a touch of a finger, or the like An optical touch sensor or the like is used in which a large number of light-emitting elements and light-receiving elements are respectively arranged in an array so as to face each other, and a point shielded by finger contact is detected as coordinates. .

Another example is disclosed in Japanese Patent Application Laid-Open No. 4-1.
There is a pointing device of a display device disclosed in Japanese Patent No. 58434. This publication discloses that an optical scanner is used, and that movement of a fingerprint pattern read by the optical scanner is detected to control cursor display on a display.

[0004]

In the above-mentioned prior art, since the sensor portion with which the finger contacts is exposed,
There is a problem that the sensitivity of the sensor unit is reduced by oils and fats and dust, and the influence of destruction by electrostatic noise and the like is inevitable.

The sensor unit is used only for cursor display control, and is not used for other functions.

Further, in the conventional technique described in Japanese Patent Application Laid-Open No. 4-158434, in addition to the above-mentioned problem, there is a problem that the operating condition is limited to the finger because the detection of the movement of the fingerprint is assumed.

An object of the present invention is to provide an optical pointing device that solves the above problems.

[0008]

A first optical pointing device according to the present invention compares two image data with a time difference between images read by an optical image sensor to thereby reduce the amount of movement of the image. In addition to generating pointing data for detecting the moving direction and performing cursor display control, monitoring the presence or absence of a change in the image data output from the optical image sensor, and detecting that the image data has not changed for a predetermined time or more. When the fact is detected, pattern recognition and feature extraction are performed based on the image data to generate feature extraction data.

A second optical pointing device of the present invention detects a moving amount and a moving direction of the image by comparing two image data with a time difference between images read by an optical image sensor. Generating pointing data for performing cursor display control, monitoring the presence or absence of a change in image data output from the optical image sensor, and detecting that there has been no change in the image data for a predetermined time or more. Character recognition data is generated by performing character pattern recognition and character code conversion on the basis of the image data.

[0010] A third optical pointing device of the present invention comprises: an optical image sensor for reading an image and outputting the image data; an image data output means for outputting the image data as it is to a personal computer main body; By comparing two image data with a time difference of, the pointing data generation for detecting the moving amount and moving direction of the image, generating pointing data for performing cursor display control, and outputting to the personal computer main body, Output means for monitoring the presence or absence of a change in the image data, and detecting that the change in the image data has not been present for a predetermined time or more, performing pattern recognition and feature extraction based on the image data to obtain feature extraction data; And output it to the personal computer Features extracted data generation, and having an output means.

[0011] A fourth optical pointing device of the present invention comprises: an optical image sensor for reading an image and outputting the image data; an image data output means for outputting the image data as it is to a personal computer main body; By comparing two image data with a time difference of, the pointing data generation for detecting the moving amount and moving direction of the image, generating pointing data for performing cursor display control, and outputting to the personal computer main body, An output unit for monitoring whether or not the image data has changed, and when detecting that the image data has not changed for a predetermined time or more, performs character pattern recognition and character code conversion based on the image data to perform character conversion. Generates recognition data and personal computer Character recognition data generated to be output, and having an output means.

A fifth optical pointing device according to the present invention includes a light-transmitting protective cover for contacting a finger or the like as an operation surface, and displays an image of a finger, a bar code, a two-dimensional code, or the like contacting the operation surface. An optical image sensor for reading, a delay circuit for delaying image data of the image sent from the optical image sensor by a predetermined time A and outputting the image data, and image data of the image sent directly from the optical image sensor And a difference between the image data input through the delay circuit and the moving amount and the moving direction of the image according to the difference are converted into a mouse-compatible data format or the like, and the cursor display control is performed. A comparator that outputs to the main body of the personal computer as pointing data for performing Detecting the absence, if this output has not been continued for a predetermined period of time B or more, while activating the pattern recognition engine as image data read by the optical image sensor has not changed for a predetermined period of time, A timer for issuing an interrupt to the personal computer and, when activated by the timer, pattern recognition is performed on the basis of image data directly input from the optical image sensor to extract features. And a pattern recognition engine that outputs to the personal computer body as feature extraction data.

A sixth optical pointing device of the present invention includes a light-transmitting protective cover for contacting a finger or the like as an operation surface, and reads an image of a finger, a character, a two-dimensional code, or the like contacting the operation surface. An optical image sensor,
A delay circuit for delaying the image data of the image sent from the optical image sensor by a predetermined time A and outputting the image data; an image data of the image sent directly from the optical image sensor; The difference between the input image data and the input image data is detected, and the moving amount and the moving direction of the image according to the difference are converted into a mouse-compatible data format or the like, and this is used as pointing data for performing cursor display control. A comparator that outputs to the personal computer main body, and the presence or absence of the output of the comparator is detected. If this output does not continue for a predetermined period of time B or more, the image data read by the optical image sensor becomes constant. Activate the character recognition engine on the assumption that the A timer that issues an interrupt to the data body, and a character code conversion table for generating a character code from a character pattern, and when activated by the timer, an image directly input from the optical image sensor. A character recognition engine that recognizes a character pattern based on data, generates a character code from the character pattern by referring to the character code conversion table, and outputs the character code as character recognition data to a personal computer main body. And

A seventh optical pointing device of the present invention is the optical pointing device according to the third or fourth optical device of the present invention, wherein the image data of the image read by the optical image sensor is directly transmitted to a personal computer main body. It further comprises means for outputting.

According to an eighth optical pointing device of the present invention, in the fifth to seventh optical pointing devices of the present invention, the comparator extracts a contour of the image from the image data, and calculates a difference between the contours. By detecting, the moving amount and moving direction of the image are converted into a data format compatible with a mouse or the like, and this is used as pointing data for performing cursor display control.

According to a ninth optical pointing device of the present invention, in the fifth to eighth optical pointing devices of the present invention, the protective cover is made of glass.

According to a tenth optical pointing device of the present invention, in the fifth to eighth optical pointing devices of the present invention, the protective cover is a flat lens.

According to an eleventh optical pointing device of the present invention, in the fifth to eighth optical pointing devices of the present invention, the protective cover is made of glass, and the distance between the protective cover and the optical image sensor is increased. A flat lens.

A twelfth optical pointing device of the present invention is characterized in that any one of the first to eleventh optical pointing devices of the present invention is incorporated in a personal computer.

[0020]

Embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of an embodiment of the present invention, and FIG. 2 is a diagram showing the structure of the optical image sensor 1 of FIG.

Referring to FIG. 1, the optical pointing device of the present invention includes an optical image sensor 1, a delay circuit 2, a comparator 3, a timer 4, and a pattern recognition engine 5.

It is assumed that the optical pointing device of the present invention is built in a notebook personal computer.
As shown in FIG.
4 is mounted on a place suitable for the operation, for example, a palm rest portion or the like.

There are three types of outputs of the optical pointing device: pointing data PD, feature extraction data TD, and image data GD. The output of the personal computer operating system (OS) to which these data are output is provided. The data input unit, the data input unit of each application, and the like will be described below as a personal computer body.

Pointing data PD, one of the output data, means data sent to the main body of the personal computer in order to control cursor display on a display (not shown).

The feature extraction data TD means feature extraction data as a result of pattern recognition of image data obtained from the optical image sensor 1 and is sent to the personal computer. The personal computer main body can perform any action such as fingerprint authentication and application activation based on the feature extraction data.

The image data GD means image data sent directly from the optical image sensor 1 to the personal computer assuming that the personal computer itself has an application for directly processing the image itself. . The image data is converted into an image processing format such as TWAIN in the personal computer and processed.

Next, referring to FIG. 2, the optical image sensor 1 is provided with an image sensor 12 of, for example, an LSI on a substrate 11, and furthermore, a light-transmitting protection for contacting a finger or the like as an operation surface thereon. A cover 13 is provided. Note that the technology (CMOS image sensor, CCD image sensor) and resolution of the image sensor 12 are not limited, but it is desirable to use a device with lower power and higher accuracy. If a device with an image processor is used, the function can be covered up to the comparator 3, and the structure becomes simpler. Also, the image sensor 1
The focal length of 2 is adjusted to be on the upper surface of the protective cover 13. Accordingly, the image sensor 12 can perform accurate image processing on an object approaching or touching the protective cover 13 (usually an operator's finger, but may be an object other than the finger).

Next, the operation of the embodiment of the present invention will be described in detail with reference to FIGS. Note that FIG.
FIG. 4 is a diagram for explaining the operation of the pointing data PD, and FIG. 4 is a diagram for explaining the operation of the feature extraction data TD.

First, the operation of the pointing data PD will be described.

First, it is assumed that the operator moves his / her finger from the rear to the front on the protective cover 13 of the optical image sensor 1 as shown in FIG. FIG. 3-1 shows a time t
, And FIG. 3B shows the position of the finger at time t + Δ immediately after that. The square frame in the figure indicates the operation surface of the optical image sensor 1.

The optical image sensor 1 always reads an image of a finger or the like touching the operation surface as image data, and the image data read by the optical image sensor 1 is, as shown in FIG. , The comparator 3, the pattern recognition engine 5, and the personal computer. At this time, the delay circuit 2 delays the input image data by a preset time Δ (for example, several tens of milliseconds) and outputs the image data delayed by this time Δ to the comparator 3.

The comparator 3 receives image data sent directly from the optical image sensor 1 and image data sent via the delay circuit 2. As described above, the image data from the delay circuit 2 is the image data that is earlier by the time Δ than the image data directly from the optical image sensor 1.

The comparator 3 compares the two images and extracts a moving amount and a moving direction according to the difference. Specifically, the comparator 3 first extracts the outline of the input image data, then extracts the difference between the outlines of the two image data, and determines the amount of movement and the direction of movement in accordance with the difference with mouse-compatible data. The data is converted into a format or the like, and transmitted to the main body of the personal computer as pointing data PD. Since various technical means are known for such a pointing data generation function, and it is not the gist of the present invention to disclose the details of the technical means of this function, further detailed description is omitted. The comparator of the present invention is, of course, not limited to an example of the above-described means for generating the pointing data PD.

Thereafter, the main body of the personal computer
Based on the pointing data PD sent from the optical pointing device of the present invention, cursor display control of a display (not shown) is performed.

As described above, in the present invention, since the outline of the image data is extracted and the difference between the two outlines is extracted, the operation on the operation surface of the optical image sensor 1 is not performed. The object is not limited to the finger of the operator, and may be an object other than the finger.

Next, the operation of the feature extraction data TD will be described.

In the above-described operation of the pointing data PD, when the image data read by the optical image sensor 1 has not changed for a certain period of time, the operation of the feature extraction data TD is started. As described above, when there is a movement of the operator's finger, the comparator 3 compares the two images before and after the movement and outputs a movement amount and a movement direction according to the difference as the pointing data PD. However, if the image data has not changed for a certain period of time, the comparator 3 does not extract the movement of the operator's finger, and as a result, the pointing data PD, that is, the output of the comparator 3 disappears during that time.

A predetermined time (usually about 0.5 to 1 second) is set in the timer 4 in advance by a user program or a specified value. The timer 4 monitors the presence or absence of the output of the comparator 3, and activates the pattern recognition engine 5 when the output does not exist for a set time or more.
At this time, the pattern recognition engine 5 is activated only once irrespective of the length of time, no matter how long the time during which there is no output from the comparator 3. When there is an output from the comparator 3, the built-in count value is reset each time and the count is started again.

When activated by the timer 4, the pattern recognition engine 5 performs pattern recognition based on the image data input from the optical image sensor 1 at that time to extract features. As a result, feature extraction data TD of about several tens of bytes is generated. This pattern recognition,
There are various known techniques for feature extraction, and there is already a pattern recognition engine 5 that performs pattern recognition and feature extraction such as fingerprint extraction and barcode extraction with a one-chip microcomputer, and details of this technical means will be disclosed. However, since this is not the gist of the present invention, further detailed description will be omitted.

The feature extraction data TD thus generated
Is issued from the timer 4 and transmitted to the personal computer main body. The personal computer main body can perform any action such as fingerprint authentication and application activation based on the feature extraction data TD.

Since an interrupt is issued by the timer when there is no change in image data for a certain period of time or more, it is assumed that an image is not placed on the operation surface of the optical image sensor 1 or that it is assumed in advance. There is a possibility that the feature extraction data TD may be output even for an unexpected image other than an image such as a fingerprint or a barcode, but the mask processing for this needs to be performed on the personal computer main body side. In addition, as an input image to the optical image sensor 1 for such feature extraction data TD, as shown in FIG. 4, a fingerprint, a bar code, a two-dimensional code such as a QR code, or the like can be used. .

Next, the operation of the image data GD will be described.

The image data GD is the image data itself read by the optical image sensor 1, and is transmitted to the personal computer as it is. On the personal computer main body side, the image data GD
Is converted to an image processing format such as TWAIN for processing. As a result, the processing that does not depend on the comparator 3 or the pattern recognition engine 5 can be realized by the utility of the personal computer main body, and various applications are possible.

In the above-described embodiment of the present invention, the pattern recognition engine 5 performs pattern recognition and feature extraction of an image to generate feature extraction data TD. A character recognition engine may be provided instead of the recognition engine 5.

The character recognition engine includes a character code conversion table for generating a character code from a character pattern. Then, when activated by the timer 4, the character recognition engine recognizes a character pattern based on the image data of the character input from the optical image sensor 1 at that time, and further incorporates a built-in character recognition engine. A character code is generated with reference to the character code conversion table and transmitted to the personal computer as character recognition data. Although there are various known techniques for character pattern recognition, disclosure of the details of the technical means is not the gist of the present invention, and therefore, further detailed description will be omitted. As described above, by using the character recognition engine as an application of the personal computer, the optical pointing device of the present invention can be used as a character input tool.

In one embodiment of the present invention,
Although the optical pointing device has been described as being built in the notebook personal computer, the optical pointing device may be separately provided without being built in the personal computer.

In FIG. 2, the protective cover 13 may be glass or a flat lens. Alternatively, the protective cover 13 may be made of glass, and a flat lens may be provided between the protective cover 13 and the image sensor 12.

Further, by setting the focal length of the optical image sensor not at the position of the protective cover 13 but at an upper position away from the protective cover 13 using a flat lens or the like, the detection range of the optical image sensor is protected. Cover 13
Even if the user holds his hand or makes a gesture above the protective cover 13, the pointing operation, the authentication, the application activation, and the like for the action can be performed.

[0049]

As described above, the first effect of the present invention is to generate data for a plurality of functions such as pointing data, feature extraction data, and image data based on image data from one optical image sensor. Is realized in one optical pointing device,
This means that the optical image sensor can be shared, and the size of the device can be reduced particularly in a notebook personal computer or the like.

The second effect is that the optical image sensor is provided with a protective cover having a light-transmitting property, and the protective cover is used as an operation surface, so that a reduction in sensitivity due to dust, grease or the like and a destruction due to electrostatic noise or the like can be achieved. That can be prevented.

The third effect is that the outline of the image data from the optical image sensor is extracted and the difference between the two outlines is extracted. An object other than the finger is not limited to the finger of the other person.

A fourth effect is that the pattern recognition engine performs pattern recognition and feature extraction on specific images such as fingerprints and bar codes to generate feature extraction data. This is to reduce the load of the feature extraction processing.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of an embodiment of the present invention.

FIG. 2 is a diagram showing a structure of the optical image sensor 1 of FIG.

FIG. 3 is a diagram illustrating an operation of pointing data PD.

FIG. 4 is a diagram illustrating the operation of feature extraction data TD.

FIG. 5 is a diagram showing a structure of a conventional electrostatic sensor.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Optical image sensor 11 Substrate 12 Image sensor 13 Protective cover 14 Personal computer case 2 Delay circuit 3 Comparator 4 Timer 5 Pattern recognition engine GD Image data PD Pointing data TD Feature extraction data

Claims (12)

[Claims] 1. Pointing data for performing cursor display control by detecting a moving amount and a moving direction of an image by comparing two image data with a time difference between images read by an optical image sensor. While generating, monitoring the presence or absence of a change in the image data output from the optical image sensor, if it is detected that there is no change in the image data for a predetermined time or more,
An optical pointing device, wherein pattern extraction and feature extraction are performed based on the image data to generate feature extraction data. 2. Pointing data for detecting a movement amount and a movement direction of the image and performing cursor display control by comparing two image data with a time difference between images read by an optical image sensor. While generating, monitoring the presence or absence of a change in the image data output from the optical image sensor, if it is detected that there is no change in the image data for a predetermined time or more,
An optical pointing device, wherein character recognition data is generated by performing character pattern recognition and character code conversion based on the image data. 3. An optical image sensor for reading an image and outputting the image data, image data output means for outputting the image data as it is to a personal computer, and two image data with a time difference between the images. Pointing data generating and outputting means for detecting the moving amount and moving direction of the image, generating pointing data for performing cursor display control and outputting the pointing data to the personal computer main body by comparing, and a change in the image data. The presence or absence of the image data is monitored, and when it is detected that the image data has not changed for a predetermined time or more, pattern extraction and feature extraction are performed based on the image data to generate feature extraction data and output it to the personal computer main body. It has feature extraction data generation and output means. Optical pointing device. 4. An optical image sensor for reading an image and outputting the image data, an image data output means for outputting the image data as it is to a personal computer main body, and outputting two image data with a time difference between the images. Pointing data generating and outputting means for detecting the moving amount and moving direction of the image, generating pointing data for performing cursor display control and outputting the pointing data to the personal computer main body by comparing, and a change in the image data. The presence or absence of the image data is monitored, and when it is detected that there is no change in the image data for a predetermined time or more, character pattern recognition and character code conversion are performed based on the image data to generate character recognition data, and the personal computer main body is generated. It has character recognition data generation and output means to output An optical pointing device, comprising: 5. An optical image sensor comprising a light-transmitting protective cover for contacting a finger or the like as an operation surface, reading an image of a finger, a barcode, a two-dimensional code or the like in contact with the operation surface; A delay circuit for delaying the image data of the image sent from the image sensor by a predetermined time A and outputting the image data; an image data of the image sent directly from the optical image sensor; The difference between the image data and the moving amount and the moving direction of the image corresponding to the difference are converted into a data format compatible with a mouse or the like, and the converted data is used as pointing data for controlling cursor display. And a comparator for detecting the presence or absence of the output of the comparator, and when this output is set at a predetermined time. If it has not continued for more than B
A timer for issuing an interrupt to a personal computer body while activating a pattern recognition engine on the assumption that image data read by the optical image sensor has not changed for a predetermined time; and A pattern recognition engine that performs pattern recognition on the basis of image data directly input from an image sensor to extract features and outputs the extracted features as feature extraction data to a personal computer main body. pointing device. 6. An optical image sensor comprising a light-transmitting protective cover for contacting a finger or the like as an operation surface, reading an image of a finger, a character, a two-dimensional code or the like contacting the operation surface, and the optical image sensor. A delay circuit for delaying the image data of the image sent from the sensor by a fixed time A and outputting the image data; an image data of the image sent directly from the optical image sensor; and a signal input through the delay circuit. The difference between the image data and the image data is detected, and the moving amount and moving direction of the image according to the difference are converted into a mouse-compatible data format or the like, and the converted data is used as pointing data for performing cursor display control on the personal computer main body. A comparator to be output, and the presence or absence of the output of the comparator. If you did not continue,
A timer that activates the character recognition engine on the assumption that the image data read by the optical image sensor has not changed for a certain period of time and issues an interrupt to the personal computer main body, and a character for generating a character code from a character pattern. With a code conversion table, when started from the timer, recognizes a character pattern based on image data directly input from the optical image sensor, and refers to the character code conversion table from this character pattern. An optical pointing device comprising: a character recognition engine that generates a character code and outputs the character code as character recognition data to a personal computer main body. 7. The optical pointing device according to claim 5, further comprising: means for directly outputting image data of the image read by the optical image sensor to a personal computer main body. 8. The comparator extracts a contour of the image from the image data, detects a difference between the contours, and converts a moving amount and a moving direction of the image into a data format or the like compatible with a mouse. 8. The optical pointing device according to claim 5, wherein the data is used as pointing data for performing cursor display control. 9. The optical pointing device according to claim 5, wherein the protective cover is made of glass. 10. The optical pointing device according to claim 5, wherein the protective cover is a flat lens. 11. The optical pointing device according to claim 5, wherein the protective cover is made of glass, and has a planar lens between the protective cover and the optical image sensor. device. 12. The optical pointing device according to claim 1, wherein the optical pointing device is built in a personal computer.