JP2010113675A - Portable electronic appliance, instruction input device, instruction input method, instruction input processing program, and computer readable recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an instruction input device performing an operation instruction in the same way as conventional various single-handed operation instructions using a mouse. <P>SOLUTION: The instruction input device is provided with: an identification part 12 for identifying coordinates representing a position of an image on a planar member on the basis of the image detected by the planar member for detecting nearby images; a decision part 13 for deciding a position relation between first and second images on the planar member by comparing coordinates between the first and second images when the first image and the second image are consecutively detected in this order at different positions on the planar member; and a determination part 14 for determining content of the instruction in accordance with the position relation between the first and second images on the planar member. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a portable electronic device, an instruction input device, an instruction input method, an instruction input processing program, and a computer-readable recording medium, and in particular, a portable electronic device capable of inputting an instruction by a user by a simple operation, an instruction The present invention relates to an input device, an instruction input method, an instruction input processing program, and a computer-readable recording medium.

  Conventionally, various input devices are used as means for inputting instructions to an electronic device such as a personal computer. For example, a keyboard, a mouse, etc. are common as input devices for personal computers. In particular, since the mouse can output the actually moved direction as an input signal, the user can input an intuitive instruction while feeling the response of the operation.

  By the way, recently, the downsizing and portability of notebook personal computers have accelerated, and accordingly, input devices that replace the mouse have been newly developed. For example, Patent Document 1 describes a coordinate input device that can use an application based on an operation with a mouse having two buttons only by a finger input operation.

  In the coordinate input device described in Patent Document 1, data equivalent to a mouse having two buttons can be output to an application by an operation with a user's finger on a tablet connected to a terminal device such as a personal computer. .

  Further, Patent Document 2 describes a mouse pad that improves the operability of a personal computer and supports mouse operations.

  With the mouse pad described in Patent Document 2, it is possible to freely change the appearance of the mouse pad itself to have a different impression by enabling an arbitrary image to be displayed on the operation surface of the mouse.

  Further, Patent Document 3 discloses a display device that recognizes a user instruction based on at least one of a positional relationship, a temporal position change, and a shape of an object corresponding to an object that is in contact with or close to the display surface. Is described.

In the display device described in Patent Document 3, a user can realize an instruction operation of a mouse by performing an operation of bringing an object such as a finger on a display screen into contact or proximity.
JP-A-7-93094 (released on April 7, 1995) JP 2008-27120 A (published February 7, 2008) JP 2008-97172 A (published April 24, 2008)

  However, with the techniques described in the above Patent Documents 1 to 3, it is impossible to completely realize an instruction operation with one hand of a user using a conventional mouse, and further improvement in operability is indispensable. There was a problem.

  In view of the above problems, an object of the present invention is to provide a portable electronic device, an instruction input device, an instruction input method, and an operation instruction similar to various operation instructions with one hand of a user using a conventional mouse, An object of the present invention is to provide an instruction input processing program and a computer-readable recording medium.

  In order to achieve the above object, an instruction input device according to the present invention displays an image, and based on a planar member that detects a nearby image, and an image detected by the planar member, The first image and the second image are successively detected in this order at a different position on the planar member, with specifying means for identifying coordinates representing a position on the planar member. And determining means for determining the positional relationship of the first and second images on the planar member by comparing the coordinates of the first and second images specified by the specifying means; Determining means for determining instruction contents according to the positional relationship determined by the determining means.

  In the above instruction input device, the positional relationship between the first and second images on the planar member is determined by comparing the coordinates of the first and second images detected successively at different positions. Therefore, it is possible to determine the instruction content corresponding to the positional relationship between the first and second images on the planar member.

  For this reason, the user can input an instruction so as to operate a conventional mouse by sequentially touching the planar member with two fingers.

  Therefore, it is possible to realize operation instructions similar to various operation instructions with one hand of a user using a conventional mouse.

  A first image that is displayed at a position on the planar member of the first image based on the coordinates of the first image when the coordinates of the first image are specified; Arrangement means for arranging, on the planar member, a second image displayed at a position on the planar member of the second image having a predetermined positional relationship with the first image. Preferably it is.

  In this case, when the coordinates of the first image are specified, the first image is displayed at the position of the first image, and the second image is displayed at the position where the second image is to be detected. Can do.

  For this reason, the user can have an operation feeling similar to that of a conventional mouse by visually recognizing the first and second images.

  The arrangement means may move the first and second images on the planar member according to a change in coordinates of the first image when the first image moves on the planar member. preferable.

  In this case, even when the first image moves on the planar member, the first and second images can always be arranged at the positions of the first and second images.

  For this reason, the user can have the same operational feeling as a conventional mouse by visually recognizing the first and second images that move following the movement of the first and second images. .

  The second image includes a left detection image to be detected in a region on the left side of the straight line including the coordinates of the first image on the planar member, and a right side of the straight line on the planar member. A right detection image to be detected in a region, and the second image includes a left image displayed at a position on the planar member of the left detection image and the right detection image. And a right image displayed at a position on the planar member.

  In this case, even when the user touches three fingers on the planar member, the first image is positioned at the position of the first image, the left image is detected at the position where the left detection image is to be detected, and the right detection is performed. The right image can be displayed at the position where the image is to be detected.

  For this reason, the user can realize the same operation input as a conventional mouse having two buttons.

  A display control unit that generates display data for performing image display according to the instruction content determined by the determination unit; and an image display member that displays an image based on the display data generated by the display control unit It is preferable to provide.

  In this case, an image display based on the instruction content corresponding to the positional relationship between the first and second images can be displayed on the image display member.

  The image forming apparatus further includes display control means for generating display data for performing image display according to the instruction content determined by the determination means, and the planar member displays an image based on the display data generated by the display control means. It is preferable to display.

  In this case, an image display based on the instruction content corresponding to the positional relationship between the first and second images can be displayed on the planar member.

  An instruction input method according to the present invention is an instruction input method using a planar member that displays an image and detects a nearby image, and based on the image detected by the planar member, A specific step of specifying coordinates representing a position on the planar member and a first image and a second image are successively detected in this order at different positions on the planar member. A determination step of determining the positional relationship of the first and second images on the planar member by comparing the coordinates of the first and second images specified in the specifying step; And a determination step for determining instruction contents according to the positional relationship determined in the determination step.

  In the above instruction input method, the positional relationship of the first and second images on the planar member is determined by comparing the coordinates of the first and second images detected successively at different positions. Therefore, it is possible to determine the instruction content corresponding to the positional relationship between the first and second images on the planar member.

  For this reason, the user can input an instruction so as to operate a conventional mouse by sequentially touching the planar member with two fingers.

  Therefore, it is possible to realize operation instructions similar to various operation instructions with one hand of a user using a conventional mouse.

  A portable electronic device according to the present invention includes the instruction input device described above.

  In the portable electronic device, an electronic device including the instruction input device is realized.

  Further, a program for operating the instruction input device according to the present invention, which is characterized in that a computer is driven as each of the above-mentioned means and a computer-readable recording medium on which the program is recorded. Included in the category.

  The instruction input device of the present invention displays an image and detects the position of the image on the planar member based on the planar member that detects a nearby image and the image detected by the planar member. When the first image and the second image are sequentially detected in this order at different positions on the planar member, the specifying means for specifying the coordinates to represent is specified by the specifying means. A determination unit that determines the positional relationship of the first and second images on the planar member by comparing the coordinates of the first and second images, and the positional relationship determined by the determination unit. Determining means for determining the instruction content according to.

  Thereby, there is an effect that it is possible to realize operation instructions similar to various operation instructions with one hand of a user using a conventional mouse.

  One embodiment of the present invention is described below with reference to FIGS.

  First, an outline of the sensor built-in liquid crystal panel 301 provided in the data display / sensor device 100 according to the present embodiment will be described.

(Outline of LCD panel with built-in sensor)
The sensor built-in liquid crystal panel 301 provided in the data display / sensor device 100 is a liquid crystal panel capable of detecting an image of an object in addition to displaying data. Here, the image detection of the object is, for example, detection of a position pointed (touched) by the user with a finger or a pen, or reading (scanning) of an image of a printed material or the like. The device used for display is not limited to a liquid crystal panel, and may be an organic EL (Electro Luminescence) panel or the like.

  The structure of the sensor built-in liquid crystal panel 301 will be described with reference to FIG. FIG. 2 is a diagram schematically showing a cross section of the sensor built-in liquid crystal panel 301. The sensor built-in liquid crystal panel 301 described here is an example, and any structure can be used as long as the display surface and the reading surface are shared.

  As shown in the figure, the sensor built-in liquid crystal panel 301 includes an active matrix substrate 51A disposed on the back surface side and a counter substrate 51B disposed on the front surface side, and has a structure in which a liquid crystal layer 52 is sandwiched between these substrates. is doing. The active matrix substrate 51A is provided with a pixel electrode 56, a data signal line 57, an optical sensor circuit 32 (not shown), an alignment film 58, a polarizing plate 59, and the like. The counter substrate 51B is provided with color filters 53r (red), 53g (green), 53b (blue), a light shielding film 54, a counter electrode 55, an alignment film 58, a polarizing plate 59, and the like. In addition, a backlight 307 is provided on the back surface of the sensor built-in liquid crystal panel 301.

  The photodiode 6 included in the photosensor circuit 32 is provided in the vicinity of the pixel electrode 56 provided with the blue color filter 53b, but is not limited to this configuration. It may be provided in the vicinity of the pixel electrode 56 provided with the red color filter 53r, or may be provided in the vicinity of the pixel electrode 56 provided with the green color filter 53g.

  Next, with reference to FIGS. 3A and 3B, two types of methods for detecting a position where the user touches the sensor built-in liquid crystal panel 301 with a finger or a pen will be described.

  FIG. 3A is a schematic diagram illustrating a state in which a position touched by the user is detected by detecting a reflected image. When the light 63 is emitted from the backlight 307, the optical sensor circuit 32 including the photodiode 6 detects the light 63 reflected by the object 64 such as a finger. Thereby, the reflected image of the target object 64 can be detected. Thus, the sensor built-in liquid crystal panel 301 can detect the touched position by detecting the reflected image.

  FIG. 3B is a schematic diagram illustrating a state in which a position touched by the user is detected by detecting a shadow image. As shown in FIG. 3B, the optical sensor circuit 32 including the photodiode 6 detects external light 61 transmitted through the counter substrate 51B and the like. However, when there is an object 62 such as a pen, the incident of the external light 61 is hindered, so that the amount of light detected by the optical sensor circuit 32 is reduced. Thereby, a shadow image of the object 62 can be detected. Thus, the sensor built-in liquid crystal panel 301 can also detect a touched position by detecting a shadow image.

  As described above, the photodiode 6 may detect reflected light (shadow image) of the light emitted from the backlight 307 or may detect a shadow image caused by external light. Further, the two types of detection methods may be used in combination to detect both a shadow image and a reflected image at the same time.

(Data display / sensor configuration)
Next, with reference to FIG. 4, the configuration of the main part of the data display / sensor device 100 will be described. FIG. 4 is a block diagram showing a main configuration of the data display / sensor device 100. As shown in FIG. As shown, the data display / sensor device 100 includes one or more display / light sensor units 300, a circuit control unit 600, a data processing unit 700, a main control unit 800, a storage unit 901, a primary storage unit 902, and an operation unit. 903, an external communication unit 907, an audio output unit 908, and an audio input unit 909. Here, the data display / sensor device 100 will be described as including two display / light sensor units 300 (first display / light sensor unit 300A and second display / light sensor unit 300B). When the first display / light sensor unit 300A and the second display / light sensor unit 300B are not distinguished, they are referred to as the display / light sensor unit 300.

  The display / light sensor unit 300 is a so-called liquid crystal display device with a built-in light sensor. The display / light sensor unit 300 includes a sensor built-in liquid crystal panel 301, a backlight 307, and a peripheral circuit 309 for driving them.

  The sensor built-in liquid crystal panel 301 includes a plurality of pixel circuits 31 and photosensor circuits 32 arranged in a matrix. The detailed configuration of the sensor built-in liquid crystal panel 301 will be described later.

  The peripheral circuit 309 includes a liquid crystal panel drive circuit 304, an optical sensor drive circuit 305, a signal conversion circuit 306, and a backlight drive circuit 308.

  The liquid crystal panel drive circuit 304 outputs a control signal (G) and a data signal (S) in accordance with the timing control signal (TC1) and the data signal (D) from the display control unit 601 of the circuit control unit 600, and the pixel circuit 31. It is a circuit which drives. Details of the driving method of the pixel circuit 31 will be described later.

  The optical sensor driving circuit 305 is a circuit that drives the optical sensor circuit 32 by applying a voltage to the signal line (R) in accordance with a timing control signal (TC2) from the sensor control unit 602 of the circuit control unit 600. Details of the driving method of the optical sensor circuit 32 will be described later.

  The signal conversion circuit 306 is a circuit that converts the sensor output signal (SS) output from the optical sensor circuit 32 into a digital signal (DS) and transmits the converted signal to the sensor control unit 602.

  The backlight 307 includes a plurality of white LEDs (Light Emitting Diodes) and is disposed on the back surface of the sensor built-in liquid crystal panel 301. When a power supply voltage is applied from the backlight drive circuit 308, the backlight 307 is turned on and irradiates the sensor built-in liquid crystal panel 301 with light. Note that the backlight 307 is not limited to white LEDs, and may include LEDs of other colors. The backlight 307 may include, for example, a cold cathode fluorescent lamp (CCFL) instead of the LED.

  The backlight driving circuit 308 applies a power supply voltage to the backlight 307 when the control signal (BK) from the backlight control unit 603 of the circuit control unit 600 is at a high level, and conversely, the backlight control unit 603. When the control signal from is at a low level, no power supply voltage is applied to the backlight 307.

  Next, the circuit control unit 600 will be described. The circuit control unit 600 has a function as a device driver that controls the peripheral circuit 309 of the display / light sensor unit 300. The circuit control unit 600 includes a display control unit 601, a sensor control unit 602, a backlight control unit 603, and a display data storage unit 604.

  The display control unit 601 receives display data from the display data processing unit 701 of the data processing unit 700, and performs timing control on the liquid crystal panel driving circuit 304 of the display / light sensor unit 300 in accordance with an instruction from the display data processing unit 701. The signal (TC1) and the data signal (D) are transmitted, and the received display data is displayed on the sensor built-in liquid crystal panel 301.

  The display control unit 601 temporarily stores the display data received from the display data processing unit 701 in the display data storage unit 604. Then, a data signal (D) is generated based on the primary stored display data. The display data storage unit 604 is, for example, a video random access memory (VRAM).

  The sensor control unit 602 transmits a timing control signal (TC2) to the optical sensor driving circuit 305 of the display / optical sensor unit 300 in accordance with an instruction from the sensor data processing unit 703 of the data processing unit 700, and the sensor built-in liquid crystal panel 301. Run the scan with.

  In addition, the sensor control unit 602 receives a digital signal (DS) from the signal conversion circuit 306. Then, image data is generated based on the digital signal (DS) corresponding to the sensor output signal (SS) output from all the optical sensor circuits 32 included in the sensor built-in liquid crystal panel 301. That is, the image data read in the entire reading area of the sensor built-in liquid crystal panel 301 is generated. Then, the generated image data is transmitted to the sensor data processing unit 703.

  The backlight control unit 603 transmits a control signal (BK) to the backlight drive circuit 308 of the display / light sensor unit 300 in accordance with instructions from the display data processing unit 701 and the sensor data processing unit 703 to drive the backlight 307. Let

  When the data display / sensor device 100 includes a plurality of display / light sensor units 300, the display control unit 601 determines which display / light sensor unit 300 displays the display data from the data processing unit 700. When an instruction is received, the liquid crystal panel drive circuit 304 of the display / light sensor unit 300 is controlled according to the instruction. When the sensor control unit 602 receives an instruction from the data processing unit 700 as to which display / light sensor unit 300 is to scan the object, The sensor drive circuit 305 is controlled and a digital signal (DS) is received from the signal conversion circuit 306 of the display / light sensor unit 300 according to the instruction.

  Next, the data processing unit 700 will be described. The data processing unit 700 has a function as middleware that gives an instruction to the circuit control unit 600 based on a “command” received from the main control unit 800. Details of the command will be described later.

  The data processing unit 700 includes a display data processing unit 701 and a sensor data processing unit 703. When the data processing unit 700 receives a command from the main control unit 800, at least one of the display data processing unit 701 and the sensor data processing unit 703 depends on the value of each field (described later) included in the received command. One works.

  The display data processing unit 701 receives display data from the main control unit 800, and gives instructions to the display control unit 601 and the backlight control unit 603 according to the command received by the data processing unit 700, and displays the received display data. The image is displayed on the sensor built-in liquid crystal panel 301. The operation of the display data processing unit 701 according to the command will be described later.

  The sensor data processing unit 703 gives an instruction to the sensor control unit 602 and the backlight control unit 603 according to the command received by the data processing unit 700.

  The sensor data processing unit 703 receives image data from the sensor control unit 602 and stores the image data in the image data buffer 704 as it is. Then, in accordance with the command received by the data processing unit 700, the sensor data processing unit 703 performs “whole image data”, “partial image data (partial image coordinate data) based on the image data stored in the image data buffer 704. At least one of “including coordinate data” and “coordinate data” is transmitted to the main control unit 800. The whole image data, partial image data, and coordinate data will be described later. The operation of the sensor data processing unit 703 according to the command will be described later.

  Next, the main control unit 800 executes an application program. The main control unit 800 reads the program stored in the storage unit 901 into a primary storage unit 902 configured by, for example, a RAM (Random Access Memory) and executes the program.

  An application program executed by the main control unit 800 causes the data processing unit 700 to display display data on the sensor built-in liquid crystal panel 301 or to scan an object on the sensor built-in liquid crystal panel 301. Send commands and display data. When “data type” is designated as a command, at least one of whole image data, partial image data, and coordinate data is received from the data processing unit 700 as a response to the command.

  The circuit control unit 600, the data processing unit 700, and the main control unit 800 can each be configured with a CPU (Central Processing Unit), a memory, and the like. The data processing unit 700 may be configured by a circuit such as an ASIC (application specific integrate circuit).

  Next, the storage unit 901 stores programs and data executed by the main control unit 800 as shown in the figure. The program executed by the main control unit 800 may be separated into an application-specific program and a general-purpose program that can be shared by each application.

  Next, the operation unit 903 receives an input operation of the user of the data display / sensor device 100. The operation unit 903 includes input devices such as a switch, a remote controller, a mouse, and a keyboard, for example. Then, the operation unit 903 generates a control signal corresponding to the input operation of the user of the data display / sensor device 100, and transmits the generated control signal to the main control unit 800.

  As an example of the switch, a hinge switch 904 that is provided at the hinge portion of the housing and detects the open / closed state of the housing, a power switch 905 that switches power on and off, and a predetermined function are assigned in advance. A hardware switch such as a user switch 906 is assumed.

  In addition, the data display / sensor device 100 includes an external communication unit 907 for communicating with an external device by wireless / wired communication, an audio output unit 908 such as a speaker for outputting audio, and a microphone for inputting an audio signal. A voice input unit 909 such as the above may be provided as appropriate.

(Command details)
Next, details of commands transmitted from the main control unit 800 to the data processing unit 700 will be described with reference to FIGS. 5 and 6. FIG. 5 is a diagram schematically illustrating an example of a command frame structure. FIG. 6 is a diagram for explaining an example of values that can be specified in each field included in the command and an outline thereof.

  As shown in FIG. 5, the commands are “header”, “data acquisition timing”, “data type”, “scan method”, “scan image gradation”, “scan resolution”, “scan panel”, “display panel”. "And" Reserve "fields. In each field, for example, values shown in FIG. 6 can be designated.

  The “header” field is a field indicating the start of a frame. As long as it is possible to identify the “header” field, the value of the “header” field may be any value.

  Next, the “data acquisition timing” field is a field for designating a timing at which data should be transmitted to the main control unit 800. In the “data acquisition timing” field, for example, values of “00” (sense), “01” (event), and “10” (all) can be specified.

  Here, “sense” specifies that the latest data is to be transmitted immediately. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “sense”, the latest data specified in the “data type” field is immediately updated to the main control unit 800. Send to.

  The “event” designates transmission at a timing when a change occurs in the image data received from the sensor control unit 602. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “event”, the image that receives the data specified in the “data type” field from the sensor control unit 602. The data is transmitted to the main control unit 800 at a timing when a change larger than a predetermined threshold occurs.

  “All” designates data transmission at a predetermined cycle. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “all”, the data designated in the “data type” field is transferred to the main control unit 800 at a predetermined cycle. Send to. The predetermined period coincides with the period in which the optical sensor circuit 32 performs scanning.

  Next, the “data type” field is a field for designating the type of data acquired from the sensor data processing unit 703. In the “data type” field, for example, values “001” (coordinates), “010” (partial image), and “100” (entire image) can be specified. Furthermore, by adding these values, “coordinates” and “partial image” / “whole image” can be specified simultaneously. For example, when “coordinate” and “partial image” are specified at the same time, “011” can be specified.

  When the sensor data processing unit 703 receives a command whose value of the “data type” field is “whole image”, the sensor data processing unit 703 transmits the image data itself stored in the image data buffer 704 to the main control unit 800. The image data itself stored in the image data buffer 704 is referred to as “whole image data”.

  In addition, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 selects a portion where a change larger than a predetermined threshold has occurred from the image data received from the sensor control unit 602. A region to be included is extracted, and image data of the extracted region is transmitted to the main control unit 800. Here, the image data is referred to as “partial image data”. When a plurality of partial image data are extracted, the sensor data processing unit 703 transmits each extracted partial image data to the main control unit 800.

  Further, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 detects representative coordinates in the partial image data and indicates the position of the representative coordinates in the partial image data. The coordinate data is transmitted to the main control unit 800. The representative coordinates include, for example, the coordinates of the center of the partial image data, the coordinates of the center of gravity of the partial image data, and the like.

  Next, when receiving a command whose value of the “data type” field is “coordinate”, the sensor data processing unit 703 transmits coordinate data indicating the position of the representative coordinate in the entire image data to the main control unit 800. When a plurality of partial image data are extracted, the sensor data processing unit 703 detects representative coordinates in the entire image data of the extracted partial image data, and the coordinate data indicating the representative coordinates is detected. Each is transmitted to the main control unit 800 (multi-point detection).

  Specific examples of the whole image data, the partial image data, and the coordinate data will be described later with reference to schematic diagrams.

  Next, the “scan method” field is a field for designating whether or not the backlight 307 is turned on at the time of executing the scan. In the “scan method” field, for example, values “00” (reflection), “01” (transmission), and “10” (reflection / transmission) can be designated.

  “Reflection” specifies that scanning is performed with the backlight 307 turned on. Therefore, when the sensor data processing unit 703 receives a command whose “scan method” field value is “reflection”, the sensor data processing unit 703 performs sensor control so that the optical sensor driving circuit 305 and the backlight driving circuit 308 operate in synchronization. An instruction is given to the unit 602 and the backlight control unit 603.

  “Transmission” specifies that scanning is performed with the backlight 307 turned off. Therefore, when the sensor data processing unit 703 receives a command whose “scan method” field value is “transparent”, the sensor control unit 602 operates the optical sensor driving circuit 305 and does not operate the backlight driving circuit 308. Instructions to the backlight control unit 603. Note that “reflection / transmission” specifies that scanning is performed using both “reflection” and “transmission”.

  Next, the “scanned image gradation” field is a field for designating gradations of the partial image data and the entire image data. In the “scanned image gradation” field, for example, values of “00” (binary) and “01” (multivalue) can be designated.

  When the sensor data processing unit 703 receives a command whose “scanned image gradation” field value is “binary”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 as monochrome data. .

  In addition, when the sensor data processing unit 703 receives a command whose “scanned image gradation” field value is “multivalued”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 as multitone data. To do.

  Next, the “scan resolution” field is a field for designating the resolution of the partial image data and the entire image data. In the “resolution” field, for example, values of “0” (high) and “1” (low) can be specified.

  Here, “high” designates a high resolution. Therefore, when the sensor data processing unit 703 receives a command whose “scan resolution” field value is “high”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 with high resolution. For example, it is desirable to designate “high” for image data (image data such as a fingerprint) to be subjected to image processing such as image recognition.

  “Low” designates a low resolution. Therefore, when the sensor data processing unit 703 receives a command whose “scan resolution” field value is “low”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 at a low resolution. For example, it is desirable to designate “low” for image data (such as touched finger or hand image data) that only needs to be recognized.

  Next, the “scan panel” field is a field for designating which display / light sensor unit 300 is to scan the object. In the “scan panel” field, for example, values “001” (first display / light sensor unit 300A) and “010” (second display / light sensor unit 300B) can be designated. By adding these values, a plurality of display / light sensor units 300 can be specified at the same time. For example, when “first display / light sensor unit 300A” and “second display / light sensor unit 300B” are specified at the same time, “011” can be specified.

  When the sensor data processing unit 703 receives a command whose “scan panel” field value is “first display / photosensor unit 300A”, the sensor data processing unit 703 and the photosensor drive circuit 305 of the first display / photosensor unit 300A and An instruction is given to the sensor control unit 602 and the backlight control unit 603 so as to control the backlight drive circuit 308.

  Next, the “display panel” field is a field for designating which display / light sensor unit 300 displays the display data. In the “scanned image gradation” field, for example, values of “001” (first display / light sensor unit 300A) and “010” (second display / light sensor unit 300B) can be designated. By adding these values, a plurality of display / light sensor units 300 can be specified at the same time. For example, when “first display / light sensor unit 300A” and “second display / light sensor unit 300B” are specified at the same time, “011” can be specified.

  Here, for example, when the display data processing unit 701 receives a command whose value of the “display panel” field is “first display / light sensor unit 300A”, the display data processing unit 701 displays the display data on the first display / light sensor unit 300A. In order to achieve this, an instruction is given to the display control unit 601 and the backlight control unit 603 to control the liquid crystal panel drive circuit 304 and the backlight drive circuit 308 of the first display / light sensor unit 300A.

  Next, the “reserved” field is a field that is appropriately specified when it is necessary to further specify information other than information that can be specified in the above-described fields.

  Note that an application executed by the main control unit 800 does not need to use all the above-described fields when transmitting a command, and an invalid value (such as a NULL value) may be set for a field that is not used. .

  When the user wants to acquire coordinate data of a position touched with a finger or a pen, a command specifying “coordinate” in the “data type” field is transmitted to the data processing unit 700. Since a pen or the like moves, it is further desirable to specify “all” in the “data acquisition timing” field of the command to acquire coordinate data. Further, since it is only necessary to acquire coordinate data of the touched position, the scanning accuracy may not be high. Therefore, “low” may be specified as the value of the “resolution” field of the command.

  In addition, when “coordinate” is specified in the “data type” field of the command, for example, when the user touches the sensor built-in liquid crystal panel 301 with a plurality of fingers or pens at the same time, the coordinates of the touched position are used. Each of the data can be acquired (multi-point detection).

  When acquiring image data of an object such as a document, a command specifying “whole image” in the “data type” field is transmitted to the data processing unit 700. Since it is common to perform a scan in a stationary state, it is not necessary to periodically perform the scan. Therefore, in this case, it is desirable to designate “sense” or “event” in the “data acquisition timing” field. When scanning an object such as a document, it is desirable that the scanning accuracy is high so that the user can easily read the characters. Therefore, it is desirable to designate “high” in the “resolution” field.

(Whole image data / Partial image data / Coordinate data)
Next, the whole image data, the partial image data, and the coordinate data will be described with reference to FIG. The image data shown in FIG. 7A is image data obtained as a result of scanning the entire sensor built-in liquid crystal panel 301 when the object is not placed on the sensor built-in liquid crystal panel 301. The image data shown in FIG. 7B is image data obtained as a result of scanning the entire sensor-equipped liquid crystal panel 301 when the user touches the sensor-equipped liquid crystal panel 301 with a finger.

  When the user touches the sensor-equipped liquid crystal panel 301 with a finger, the amount of light received by the photosensor circuit 32 in the vicinity of the touch changes, so that the voltage output from the photosensor circuit 32 changes, and as a result In the image data generated by the sensor control unit 602, the brightness of the pixel value of the portion touched by the user changes.

  In the image data shown in FIG. 7B, the brightness of the pixel value of the portion corresponding to the user's finger is higher than the image data shown in FIG. In the image data shown in FIG. 7B, the smallest rectangular area (area PP) that includes all pixel values whose lightness changes more than a predetermined threshold is “partial image data”.

  The image data indicated by the area AP is “whole image data”.

  Also, the coordinate data in the whole image data (area AP) of the representative coordinates Z of the partial image data (area PP) is (Xa, Ya), and the coordinate data in the partial image data (area PP) is (Xp, Yp). It is.

(Configuration of sensor built-in liquid crystal panel)
Next, the configuration of the sensor built-in liquid crystal panel 301 and the configuration of the peripheral circuit 309 of the sensor built-in liquid crystal panel 301 will be described with reference to FIG. FIG. 8 is a block diagram showing a main part of the display / light sensor unit 300, particularly the configuration of the sensor built-in liquid crystal panel 301 and the configuration of the peripheral circuit 309.

  The sensor built-in liquid crystal panel 301 includes a pixel circuit 31 for setting a light transmittance (luminance) and an optical sensor circuit 32 that outputs a voltage corresponding to the intensity of light received by the sensor. Note that the pixel circuit 31 is a generic term for the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b corresponding to the red, green, and blue color filters, respectively.

  The pixel circuits 31 are arranged on the sensor built-in liquid crystal panel 301 in the column direction (vertical direction) and 3n in the row direction (horizontal direction). A set of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b is continuously arranged in the row direction (lateral direction). This set forms one pixel.

  In order to set the light transmittance of the pixel circuit 31, first, the high level voltage (TFT 33 is turned on) to the scanning signal line Gi connected to the gate terminal of the TFT (Thin Film Transistor) 33 included in the pixel circuit 31. Voltage). Thereafter, a predetermined voltage is applied to the data signal line SRj connected to the source terminal of the TFT 33 of the R pixel circuit 31r. Similarly, the light transmittance is also set for the G pixel circuit 31g and the B pixel circuit 31b. Then, by setting these light transmittances, an image is displayed on the sensor built-in liquid crystal panel 301.

  Next, the photosensor circuit 32 is arranged for each pixel. One pixel may be arranged in the vicinity of each of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b.

  In order for the optical sensor circuit 32 to output a voltage corresponding to the light intensity, first, the sensor readout line RWi connected to one electrode of the capacitor 35 and the anode terminal of the photodiode 36 are connected. A predetermined voltage is applied to the sensor reset line RSi. In this state, when light is incident on the photodiode 36, a current corresponding to the amount of incident light flows through the photodiode 36. Then, according to the current, the voltage at the connection point (hereinafter referred to as connection node V) between the other electrode of the capacitor 35 and the cathode terminal of the photodiode 36 decreases. When the power supply voltage VDD is applied to the voltage application line SDj connected to the drain terminal of the sensor preamplifier 37, the voltage at the connection node V is amplified and output from the source terminal of the sensor preamplifier 37 to the sensing data output line SPj. Based on the output voltage, the amount of light received by the optical sensor circuit 32 can be calculated.

  Next, the liquid crystal panel drive circuit 304, the optical sensor drive circuit 305, and the sensor output amplifier 44, which are peripheral circuits of the sensor built-in liquid crystal panel 301, will be described.

  The liquid crystal panel drive circuit 304 is a circuit for driving the pixel circuit 31, and includes a scanning signal line drive circuit 3041 and a data signal line drive circuit 3042.

  The scanning signal line driving circuit 3041 sequentially selects one scanning signal line from the scanning signal lines G1 to Gm for each line time based on the timing control signal TC1 received from the display control unit 601, and A high level voltage is applied to the selected scanning signal line, and a low level voltage is applied to the other scanning signal lines.

  Based on the display data D (DR, DG, and DB) received from the display control unit 601, the data signal line driver circuit 3042 applies a predetermined voltage corresponding to the display data for one row for each line time. The data signal lines SR1 to SRn, SG1 to SGn, and SB1 to SBn are applied (line sequential method). Note that the data signal line driver circuit 3042 may be driven by a dot sequential method.

  The optical sensor driving circuit 305 is a circuit for driving the optical sensor circuit 32. Based on the timing control signal TC2 received from the sensor control unit 602, the optical sensor driving circuit 305 selects a predetermined sensor readout signal line from the sensor readout signal lines RW1 to RWm for each line time. A read voltage is applied, and a voltage other than a predetermined read voltage is applied to the other sensor read signal lines. Similarly, based on the timing control signal TC2, a predetermined reset voltage is applied to the sensor reset signal line selected from the sensor reset signal lines RS1 to RSm for each line time, and the others. A voltage other than a predetermined reset voltage is applied to the sensor reset signal line.

  The sensing data output signal lines SP1 to SPn are grouped into p groups (p is an integer of 1 to n), and the sensing data output signal lines belonging to each group are connected via a switch 47 that is sequentially turned on in time division. And connected to the sensor output amplifier 44. The sensor output amplifier 44 amplifies the voltage from the group of sensing data output signal lines connected by the switch 47 and outputs the amplified voltage to the signal conversion circuit 306 as sensor output signals SS (SS1 to SSp).

(Electronic equipment with data display / sensor device)
Next, an electronic apparatus provided with the data display / sensor device 100 according to the present embodiment will be described. The data display / sensor device 100 is attached to a miniaturized and portable electronic device such as a notebook personal computer. Of course, the attachment of the data display / sensor device 100 is not limited to a notebook personal computer, and may be an electronic device that can be moved by being carried by a person. For example, it is a portable information terminal such as a game machine, a mobile phone, or a PDA. In any electronic device, the same effect can be obtained.

  Hereinafter, the present embodiment will be described as an embodiment of a data display / sensor device 100 attached to a notebook personal computer. FIG. 9 is an external view of a notebook personal computer 101 provided with the data display / sensor device 100.

  As shown in FIG. 9, the notebook personal computer 101 includes an input pad (planar member) 300 </ b> C configured by the first display / light sensor unit 300 </ b> A of the data display / sensor device 100, and the data display / sensor device 100. Display panel (image display member) 300D configured by the second display / light sensor unit 300B.

  Here, the input pad 300C and the display panel 300D are realized by the first display / light sensor unit 300A and the second display / light sensor unit 300B which are different from each other. It may be realized by one display / light sensor unit 300A. In other words, one display / light sensor unit 300A may serve as the input pad 300C and the display panel 300D.

  When the user operates the sensor-embedded liquid crystal panel 301 of the input pad 300C with a finger, the input pad 300C moves the pointer displayed on the sensor-embedded liquid crystal panel 301 of the display panel 300D or the sensor-embedded liquid crystal. Selection operation of each menu screen on the panel 301 is enabled. The pointer is for pointing an object to an instruction or operation on the sensor built-in liquid crystal panel 301 of the display panel 300D.

  For example, if the user's finger moves while touching the sensor built-in liquid crystal panel 301 of the input pad 300C, the pointer on the sensor built-in liquid crystal panel 301 of the display panel 300D can be moved according to the locus.

  FIG. 10 is an enlarged view of the periphery of the input pad 300C shown in FIG. As shown in FIG. 10, the input pad 300C is arranged in the vicinity of the keyboard included in the operation unit 903 of the data display / sensor device 100. For example, when the user pauses the keyboard input operation, The input pad 300C can be operated with a finger.

  The finger position image that the user can visually recognize that the input pad 300C is an area where the user should place his / her finger so that the user can feel the same operation feeling as a conventional mouse when the user operates the input pad 300C with a finger. 30 is displayed on its own sensor built-in liquid crystal panel 301. The finger position image 30 includes a central image (first image) 30a representing an area to be touched by the first finger (first image) of the user, and a left side of the first finger in the user's hand. A left image (second image) 30b representing an area to be touched by a second finger (second image) that is positioned, and a third finger (second image) that is positioned to the right of the first finger in the user's hand 2 image) includes a right image (second image) 30c representing an area to be touched. That is, the finger position image 30 constitutes a virtual mouse (virtual mouse) for the user.

  FIG. 11 is a diagram illustrating a state when the user's finger is actually touching the finger position image 30. As shown in FIG. 11, for example, when the input pad 300C is operated by the user's right hand, the middle finger of the user touches the center image 30a as the first finger 41, and the left image 30b The user's index finger touches as the second finger 42, and the user's ring finger touches the right image 30 c as the third finger 43.

  Alternatively, if the user's index finger touches the center image 30a as the first finger 41, the user's thumb touches the left image 30b as the second finger 42, and the right image 30c shows the user's thumb. The middle finger touches as the third finger 43.

  Of course, if the input pad 300C is operated by the left hand of the user, the first finger 41, the second finger 42, and the third finger 43 to the center image 30a, the left image 30b, and the right image 30c will be described. Needless to say, touch is the opposite of the above case.

  The operation of the input pad 300C by the user's finger is executed as follows, for example.

  First, when the first finger of the user touches an arbitrary position on the sensor-embedded liquid crystal panel 301 of the input pad 300C, the finger position image 30 is displayed so that the central image 30a is arranged at the touch position. .

  When the user moves the input pad 300C on the sensor built-in liquid crystal panel 301 while touching the first finger, the position of the pointer on the sensor built-in liquid crystal panel 301 on the display panel 300D is moved in accordance with the movement. Can be made.

  In this case, the movement distance, direction, speed, etc. of the first finger on the sensor built-in liquid crystal panel 301 of the input pad 300C are detected, and the pointer displayed on the sensor built-in liquid crystal panel 301 of the display panel 300D is detected accordingly. The relative movement may be controlled. When one display / light sensor unit 300A serves as both the input pad 300C and the display panel 300D, the coordinates of the first finger on the sensor built-in liquid crystal panel 301 of the input pad 300C and the sensor of the display panel 300D are used. The coordinates of the pointer on the built-in liquid crystal panel 301 are associated by absolute coordinates.

  When the first finger stops at the end of the sensor built-in liquid crystal panel 301 of the input pad 300C and can no longer be moved, the first finger is returned to the vicinity of the center of the sensor built-in liquid crystal panel 301 of the input pad 300C to continue the operation. Thus, the movement of the pointer on the sensor built-in liquid crystal panel 301 of the display panel 300D can be continued.

  Here, the finger position image 30 may be displayed before the first finger touches the input pad 300C. In this case, when the first finger of the user does not touch the center image 30a of the finger position image 30 that is already displayed and touches a different position, the center image 30a is arranged at the touch position. Then, the finger position image 30 may be moved.

  Alternatively, the finger position image 30 may not be displayed before the first finger touches the sensor built-in liquid crystal panel 301 of the input pad 300C and before the second and third fingers touch. In this case, the finger position image 30 may be displayed after the second and third fingers described later are touched.

  Next, when the second finger touches the left image 30b with the first finger touching the input pad 300C, the icon and file at the position of the pointer on the sensor built-in liquid crystal panel 301 of the display panel 300D. Etc. are selected.

  Then, when the user moves the input pad 300C on the sensor built-in liquid crystal panel 301 while touching the first and second fingers, the pointer on the sensor built-in liquid crystal panel 301 on the display panel 300D is moved accordingly. The icon or the like selected by can be moved.

  Further, when the second finger leaves the input pad 300C while being touched by the first finger, the selected state of the icon or the like is canceled at the position of the movement destination, and the icon or the like can be held at the position. .

  On the other hand, when the first finger leaves the input pad 300C while the second finger is touched, the selected state of the icon or the like is maintained at the position of the movement destination. When the first finger touches again, the icon selected by the pointer is moved by moving the input pad 300C on the sensor-embedded liquid crystal panel 301 while touching the first and second fingers. Can continue.

  Similarly, when the third finger touches the right image 30c with the first finger touching the input pad 300C, the icon, file, etc. at the position of the pointer on the sensor built-in liquid crystal panel 301 of the display panel 300D. Is selected.

  Then, when the user moves on the sensor-embedded liquid crystal panel 301 of the input pad 300C while touching the first and third fingers, the pointer on the sensor-embedded liquid crystal panel 301 of the display panel 300D in accordance with the movement. The icon or the like selected by can be moved.

  Further, when the third finger leaves the input pad 300C while the first finger is touched, the selected state of the icon or the like is canceled at the position of the movement destination, and the icon or the like can be held at the position. .

  On the other hand, when the first finger leaves the input pad 300C while the third finger is touched, the selected state of the icon or the like is maintained at the position of the movement destination. When the first finger touches again, the icon selected by the pointer is moved by moving the input pad 300C on the sensor built-in liquid crystal panel 301 while touching the first and third fingers. Can continue.

Note that the operation of the input pad 300C with the user's finger in the present embodiment is not limited to the above-described content. For example, the operation of the input pad 300C by the user's finger includes the following.
(A) “Point” operation in which an instruction or an operation target is indicated by a pointer when the first finger touches the sensor-embedded liquid crystal panel 301 of the input pad 300C. (B) On the sensor-embedded liquid crystal panel 301 of the input pad 300C. “Left click” operation in which the second finger touches only once (C) “Right click” operation in which the third finger touches the sensor built-in liquid crystal panel 301 of the input pad 300C only once (D) Input pad 300C “Double-click” operation in which the second finger touches the sensor built-in liquid crystal panel 301 twice continuously (E) The first and second fingers remain touching the sensor built-in liquid crystal panel 301 of the input pad 300C. (F) Move with the first and third fingers touching on the sensor built-in liquid crystal panel 301 of the input pad 300C. “Right drag” operation (G) After the “left drag” operation or the “right drag” operation, the second or third finger is released from the sensor built-in liquid crystal panel 301 of the input pad 300C, thereby performing the “left drag” operation or “ "Drop" operation that completes movement of the target during "Right drag" operation (Configuration of main control unit and storage unit)
Next, the configuration of the main control unit 800 and the storage unit 901 of the data display / sensor device 100 according to the present embodiment will be described. Here, in order to make the description easy to understand, the description of the operations of the data processing unit 700 and the circuit control unit 600 located between the main control unit 800 and the display / light sensor unit 300 is omitted.

  However, to be precise, when displaying data and scanning an object, each unit of the main control unit 800 transmits a command to the data processing unit 700, and the data processing unit 700 sets the circuit control unit 600 based on the command. Then, the circuit control unit 600 transmits a signal to the display / light sensor unit 300.

  Further, the main control unit 800 acquires the entire image data, the partial image data, and the coordinate data from the data processing unit 700 as a response to the command transmitted to the data processing unit 700.

  FIG. 1 is a block diagram illustrating the configuration of the main control unit 800 and the storage unit 901 of the data display / sensor device 100. As shown in FIG. 1, the main control unit 800 of the data display / sensor device 100 includes a data acquisition unit 11, a specification unit (specification unit) 12, a determination unit (determination unit) 13, and a determination unit (determination unit). 14, an arrangement unit (arrangement unit) 15, and a display control unit (display control unit) 16.

  Further, as shown in FIG. 1, the storage unit 901 of the data display / sensor device (instruction input device) 100 includes a coordinate information storage unit 21, a determination condition storage unit 22, a determination result storage unit 23, and an instruction content storage. Part 24.

  The data acquisition unit 11 acquires coordinate data transmitted from the sensor data processing unit 703 of the data processing unit 700 to the main control unit 800 in response to a command from the main control unit 800.

  Here, in order to acquire coordinate data from the sensor data processing unit 703, the data acquisition unit 11 designates “coordinate” in the “data type” field of the command transmitted from the main control unit 800 to the data processing unit 700.

  Further, in addition to specifying the “data type” field, the data acquisition unit 11 specifies “all” in the “data acquisition timing” field, and acquires coordinate data at a predetermined cycle.

  Further, when a plurality of partial image data is extracted by the sensor data processing unit 703, the data acquisition unit 11 acquires coordinate data of each partial image data.

  The coordinate information storage unit 21 stores the coordinate data acquired by the data acquisition unit 11. Then, when the data acquisition unit 11 acquires the coordinate data of a plurality of partial image data at the same time, the coordinate information storage unit 21 stores the partial image data and the coordinates so that the coordinate data of each partial image data can be identified. Associate and store data.

  When the coordinate data acquired by the data acquisition unit 11 is input, the specifying unit 12 reads the coordinate data stored in the coordinate information storage unit 21 and compares it with the coordinate data from the data acquisition unit 11. Then, the specifying unit 12 matches the partial image data having the coordinate data acquired by the data acquiring unit 11 among the partial image data having the coordinate data stored in the coordinate information storage unit 21 based on the comparison result. Identify whether there is something to do.

  Here, the coordinate information storage unit 21 has partial image data having coordinate data stored in the coordinate information storage unit 21 that matches the partial image data having coordinate data acquired by the data acquisition unit 11. Stores the coordinate data acquired by the data acquisition unit 11 in association with the specified partial image based on the identification result.

  In this case, the object on the sensor built-in liquid crystal panel 301 corresponding to the partial image data is moving or waiting while touching the sensor built-in liquid crystal panel 301. And about the movement or stand-by, and the movement amount and movement direction in the case of moving, the specifying unit 12 uses the coordinate data from the data acquisition unit 11 and the coordinate data from the coordinate information storage unit 21, Can be identified.

  On the other hand, when there is no partial image data having the coordinate data stored in the coordinate information storage unit 21 that matches the partial image data having the coordinate data acquired by the data acquisition unit 11, the coordinate information storage unit 21 does not exist. Based on the identification result, the coordinate data acquired by the data acquisition unit 11 is stored as coordinate data of new partial image data.

  In this case, the object on the sensor built-in liquid crystal panel 301 corresponding to the partial image data is newly touched on the sensor built-in liquid crystal panel 301.

  Further, the specifying unit 12 has no match with the partial image data having the coordinate data acquired by the data acquiring unit 11 among the partial image data having the coordinate data stored in the coordinate information storage unit 21. And deleted from the coordinate information storage unit 21.

  In this case, the object on the sensor built-in liquid crystal panel 301 corresponding to the partial image data is detached from the sensor built-in liquid crystal panel 301.

  The determination unit 13 determines the operation content of the input pad 300 </ b> C by the user's finger based on the identification result by the identification unit 12. Specifically, the determination unit 13 determines whether or not the first to third fingers have touched the sensor built-in liquid crystal panel 301 of the input pad 300C, and whether or not the first to third liquid crystal panels 301 of the input pad 300C are touched. It is determined whether or not the finger has moved while touching, and whether or not the first to third fingers have been detached from the sensor built-in liquid crystal panel 301 of the input pad 300C. Further, when the determination unit 13 determines that the first to third fingers have moved while touching on the sensor built-in liquid crystal panel 301 of the input pad 300C, the specifying unit 12 also determines the movement direction and the movement amount. Get from.

  Further, when determining the operation of the input pad 300C by the user's finger, the determination unit 13 refers to determination conditions stored in advance in a determination condition storage unit 22 described later.

The determination condition storage unit 22 stores in advance determination conditions to be referred to when the determination unit 13 determines the operation of the input pad 300C by the user's finger. Examples of the determination condition include the following.
(A) Conditions for determining that the finger touching the sensor built-in liquid crystal panel 301 of the input pad 300C is the first finger. (B) The finger touching the sensor built-in liquid crystal panel 301 of the input pad 300C is the second. (C) Condition for Determining that the Finger is Touched on the Sensor Built-in Liquid Crystal Panel 301 of the Input Pad 300C Specifically, the condition for determining that the finger is the third finger (A) The condition is, for example, that the finger is newly touched from a state in which none of the user's fingers is touching the sensor built-in liquid crystal panel 301 of the input pad 300C.

  The above condition (B) is that, for example, in the state where the first finger is touching the sensor-embedded liquid crystal panel 301 of the input pad 300C, a new area is touched on the left side of the first finger. Being a finger.

  In the case of the condition (B), an area to be touched by the second finger based on the touch position of the first finger is determined in advance. Then, the left image 30b of the finger position image 30 is displayed in the area to be touched by the second finger so that the user can visually recognize the area to be touched by the second finger.

  It should be noted that the shape of the region to be touched by the second finger and the shape of the left image 30b do not have to completely match. In short, it suffices if the user can touch the area to be touched by the second finger when the user touches the second image toward the left image 30b.

  In addition, for example, in the case where the first finger is the middle finger and the second finger is the index finger as described above, the index finger moves normally after fixing the position of the middle finger. It may be set based on the range that is considered possible.

  The above condition (C) is that, for example, when the first finger is touching the sensor built-in liquid crystal panel 301 of the input pad 300C, a new touch is made in the region located on the right side of the first finger. Being a finger.

  In the case of the condition (C), an area to be touched by the third finger based on the touch position of the first finger is determined in advance. Then, the right image 30c of the finger position image 30 is displayed in the area to be touched by the third finger so that the user can visually recognize the area to be touched by the third finger.

  Note that the shape of the region to be touched by the third finger and the shape of the right image 30c do not have to completely match. In short, it suffices if the third finger can touch the area to be touched when the user touches the third finger toward the right image 30c.

  In addition, for example, when the first finger is the middle finger and the second finger is the ring finger as described above, the ring finger normally moves after fixing the position of the middle finger. It may be set based on the range that is considered possible.

  The determination unit 13 determines the operation content of the input pad 300 </ b> C by the user's finger based on the identification result by the identification unit 12 while referring to the determination condition stored in the determination condition storage unit 22.

  The determination result storage unit 23 stores the determination result from the determination unit 13. And the determination result memory | storage part 23 outputs the determination result which self has memorize | stored to the determination part 14 and the arrangement | positioning part 15 which are mentioned later.

  Based on the determination result from the determination unit 13, the determination unit 14 determines the instruction content to be executed on the sensor built-in liquid crystal panel 301 of the display panel 300D. Then, the determination unit 14 outputs the determined instruction content to the display control unit 16 described later. Further, the determination unit 14 determines the instruction content to be executed on the sensor built-in liquid crystal panel 301 of the display panel 300 </ b> D while referring to the past determination results stored in the determination result storage unit 23.

  Furthermore, when determining the instruction content to be executed on the sensor built-in liquid crystal panel 301 of the display panel 300D, the determination unit 14 determines whether the instruction content stored in advance in the instruction content storage unit 24 described later is determined from the determination unit 13. , I.e., the instruction content corresponding to the operation content of the input pad 300 </ b> C by the user's finger is extracted, and the instruction content is output to the display control unit 16.

  The instruction content storage unit 24 stores in advance instruction contents corresponding to the operation contents of the input pad 300C by the user's finger. This instruction content is an instruction content executed on the sensor built-in liquid crystal panel 301 of the display panel 300D based on the operation of the input pad 300C by the user's finger. As described above, the operation of the input pad 300C by the user's finger includes a “point” operation, a “left click” operation, a “right click” operation, and the like. The instruction content storage unit 24 performs a pair of these operations. Instruction contents corresponding to one are stored in advance.

  The placement unit 15 places the finger position image 30 on the sensor built-in liquid crystal panel 301 of the input pad 300 </ b> C based on the determination result stored in the determination result storage unit 23.

  Specifically, the determination result storage unit 23 determines whether or not the determination result by the determination unit 13, that is, whether or not the first finger touches the sensor-embedded liquid crystal panel 301 of the input pad 300 </ b> C, and the sensor-embedded liquid crystal of the input pad 300 </ b> C. It stores whether or not the first finger has moved while touching on the panel 301 and whether or not the first finger has detached from the sensor built-in liquid crystal panel 301 of the input pad 300C. Based on the determination result, the placement unit 15 displays the finger position image 30 on the sensor-embedded liquid crystal panel 301 of the input pad 300C in accordance with the touch of the first finger, and keeps the touch of the first finger. The finger position image 30 is moved on the sensor built-in liquid crystal panel 301 of the input pad 300C in accordance with the movement of the input pad 300C, and the finger position image 30 is displayed on the sensor built-in liquid crystal panel 301 in the input pad 300C in accordance with the release of the first finger. To stop.

  In addition, the placement unit 15 instructs the display control unit 16 on the behavior of the finger position image 30 on the sensor-embedded liquid crystal panel 301 of the input pad 300C.

  The display control unit 16 generates display data corresponding to the outputs of the determination unit 14 and the arrangement unit 15. Specifically, when the display control unit 16 receives the instruction content on the sensor-embedded liquid crystal panel 301 of the display panel 300D from the determination unit 14, the display control unit 16 outputs the display data generated by the display control unit 16 to the display panel 300D. “010” is designated in the “display panel” field of the command transmitted from the main control unit 800 to the data processing unit 700.

  On the other hand, when the display control unit 16 receives an instruction about the finger position image 30 on the sensor built-in liquid crystal panel 301 of the input pad 300C from the placement unit 15, the display control unit 16 outputs display data generated by itself to the input pad 300C. Therefore, “001” is designated in the “display panel” field of the command transmitted from the main control unit 800 to the data processing unit 700.

  Next, a typical example of processing performed by the notebook personal computer 101 in response to an operation of the input pad 300C by a user's finger will be described with reference to FIGS.

  FIG. 12 is a flowchart showing a procedure of processing performed in response to the operation of the input pad 300C by the user's first finger.

  As illustrated in FIG. 12, the determination unit 13 specifies the touch position (first position) of the first finger on the sensor-embedded liquid crystal panel 301 of the input pad 300 </ b> C based on the specification result by the specifying unit 12 ( Step S101). In step S <b> 101, the determination result storage unit 23 stores the identification result of the first position by the determination unit 13.

  The placement unit 15 places the finger position image 30 that is a virtual mouse on the sensor built-in liquid crystal panel 301 of the input pad 300C based on the first position specifying result stored in the determination result storage unit 23 (step S30). S102).

  When the first position is moved (YES in step S103), the placement unit 15 moves the finger position image 30 that is a virtual mouse on the sensor built-in liquid crystal panel 301 of the input pad 300C according to the movement (step S103 YES). Step S104), it returns to step S103 again.

  On the other hand, when the first position does not move (step S103 NO), the placement unit 15 waits on the sensor built-in liquid crystal panel 301 of the input pad 300C that is a virtual mouse (step S105).

  FIG. 13 and FIG. 14 are flowcharts showing a procedure of processing performed in accordance with the operation of the input pad 300C by the user's first, second and third fingers. Here, it is assumed that the first finger has already touched the sensor-embedded liquid crystal panel 301 of the input pad 300C.

  As illustrated in FIG. 13, the determination unit 13 determines the touch position (second position) of the second or third finger on the sensor built-in liquid crystal panel 301 of the input pad 300 </ b> C based on the specification result by the specification unit 12. Specify (step S201). In step S <b> 201, the determination result storage unit 23 stores the second position specifying result by the determination unit 13.

  If the second position is the left side of the first position (the left side of step S202), the determination unit 14 stores the instruction content corresponding to the case where the second position is the left side of the first position. Extracted from the unit 24 and determined (step S203).

  On the other hand, if the second position is the right side of the first position (the right side of step S202), the determination unit 14 stores the instruction content corresponding to the case where the second position is the right side of the first position. Extracted from the unit 24 and determined (step S204).

  The determination unit 14 outputs the instruction content determined in step S203 or step S204 to the display control unit 16. And the display control part 16 produces | generates display data according to the instruction | indication content output from the determination part 14 (step S205).

  If the first and second positions do not move (step S206 NO), the process ends.

  On the other hand, when the first and second positions are moved (YES in step S206), as shown in FIG. 14, the placement unit 15 displays the finger position image 30 that is a virtual mouse on the input pad 300C according to the movement. The sensor is moved on the sensor built-in liquid crystal panel 301 (step S301).

  Then, as long as both the first and second positions can be specified (NO in step S302), the process returns to step S301 again.

  On the other hand, when the first position or the second position cannot be specified (YES in step S302), if the first position cannot be specified (step S303, the first position), the placement unit 15 detects the sensor of the input pad 300C. On the built-in liquid crystal panel 301, the finger position image 30, which is a virtual mouse, is kept on standby (step S304). Then, again, the determination unit 13 specifies the first position of the input pad 300C on the sensor built-in liquid crystal panel 301 based on the specification result by the specifying unit 12 (step S305), and returns to step S206 again. .

  On the other hand, if the second position cannot be specified (second position in step S303), the determination unit 14 stops display data generation in the display control unit 16 in order to stop execution of the instruction content determined by itself. Is instructed (step S306). And it will return to said step S201 again.

(Program, computer-readable recording medium)
Further, each block of the data display / sensor device 100, particularly the main control unit 800, may be configured by hardware logic, or may be realized by software using a computer as follows.

  That is, the main control unit 800 includes a CPU (central processing unit) that executes instructions of a control program for realizing each function, a ROM (read only memory) that stores the program, and a RAM (random access memory) that expands the program. And a storage device (recording medium) such as a memory for storing the program and various data. An object of the present invention is to provide a recording medium on which a program code (execution format program, intermediate code program, source program) of a control program of the main control unit 800, which is software that realizes the functions described above, is recorded so as to be readable by a computer. This can also be achieved by supplying to the main control unit 800 and reading and executing the program code recorded on the recording medium by the computer (or CPU or MPU).

  Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy (registered trademark) disks / hard disks, and disks including optical disks such as CD-ROM / MO / MD / DVD / CD-R. Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.

  The main control unit 800 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Also, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  The present invention can be applied as an instruction input device for various electronic devices. In particular, when it is applied as an input device for a portable electronic device, the same input operation as a conventional mouse can be performed with one hand, so that it can be applied to a mobile phone, a PDA (Personal Digital Assistance), a portable game machine, etc. It can be suitably applied.

It is a block diagram which shows the structure of the main control part and memory | storage part with which the data display / sensor apparatus which concerns on one Embodiment of this invention is provided. It is a figure which shows typically the cross section of the liquid crystal panel with a built-in sensor with which the data display / sensor apparatus which concerns on one Embodiment of this invention is provided. FIG. 3A is a schematic diagram showing a state in which a position touched by a user is detected by detecting a reflected image on a sensor-embedded liquid crystal panel included in the data display / sensor device according to the embodiment of the present invention. It is. FIG. 3B is a schematic diagram illustrating a state in which a position touched by a user is detected by detecting a shadow image with a sensor built-in liquid crystal panel included in the data display / sensor device according to the embodiment of the present invention. is there. It is a block diagram which shows the principal part structure of the data display / sensor apparatus which concerns on one Embodiment of this invention. It is a figure which shows typically an example of the frame structure of the command used with the data display / sensor apparatus which concerns on one Embodiment of this invention. It is a figure explaining an example of the value which can be specified to each field contained in the command shown in FIG. 5, and its outline. FIG. 7A is a result of scanning the entire sensor-equipped liquid crystal panel when the object is not placed on the sensor-equipped liquid crystal panel in the data display / sensor device according to the embodiment of the present invention. Image data. FIG. 7B shows image data obtained as a result of scanning when the user touches the sensor-equipped liquid crystal panel with a finger in the data display / sensor device according to the embodiment of the present invention. . It is a block diagram which shows the structure of the liquid crystal panel with a sensor with which the data display / sensor apparatus which concerns on one Embodiment of this invention is provided, and the structure of its peripheral circuit. 1 is an external view of a notebook personal computer including a data display / sensor device according to an embodiment of the present invention. FIG. 10 is an enlarged view of a peripheral portion of the input pad shown in FIG. 9. It is a figure which shows a mode when a user's finger | toe is actually touching the finger position image. It is a flowchart which shows the procedure of the process performed according to operation of the input pad by a user's 1st finger | toe. It is a flowchart which shows the procedure of the process performed according to operation of the input pad by a user's 1st, 2nd and 3rd finger | toe (the 1). It is a flowchart which shows the procedure of the process performed according to operation of the input pad by a user's 1st, 2nd and 3rd finger | toe (the 2).

Explanation of symbols

11 Data acquisition unit 12 Identification unit (identification means)
13 Determination part (determination means)
14. Determination part (determination means)
15 Arrangement part (arrangement means)
16 Display control unit (display control means)
21 Coordinate information storage unit 22 Judgment condition storage unit 23 Judgment result storage unit 24 Instruction content storage unit 30 Finger position image 30a Center image (first image)
30b Left image (second image)
30c Right image (first image)
41 First finger (first image)
42 Second finger (second image)
43 Third finger (second image)
100 Data display / sensor device (instruction input device)
101 Notebook personal computer (portable electronic device)
300 Display / light sensor unit 300C Input pad (planar member)
300D display panel (image display member)
301 Liquid crystal panel with built-in sensor 700 Data processing unit 800 Main control unit 901 Storage unit

Claims (10)

A planar member for displaying an image and detecting a nearby image;
A specifying means for specifying coordinates representing a position of the image on the planar member based on the image detected by the planar member;
When the first image and the second image are successively detected in this order at different positions on the planar member, the first and second images specified by the specifying means are detected. Determining means for determining a positional relationship of the first and second images on the planar member by comparing coordinates;
An instruction input device comprising: a determination unit that determines instruction content according to the positional relationship determined by the determination unit.   A first image that is displayed at a position on the planar member of the first image based on the coordinates of the first image when the coordinates of the first image are specified; Arrangement means for arranging, on the planar member, a second image displayed at a position on the planar member of the second image having a predetermined positional relationship with the first image. The instruction input device according to claim 1, wherein:   The arrangement means moves the first and second images on the planar member according to a change in coordinates of the first image when the first image moves on the planar member. The instruction input device according to claim 2, wherein: The second image includes a left detection image to be detected in a region on the left side of the straight line including the coordinates of the first image on the planar member, and a right side of the straight line on the planar member. Right side detection image to be detected in the area,
The second image includes a left side image displayed at a position on the planar member of the left side detection image, and a right side image displayed at a position on the planar member of the right side detection image. The instruction input device according to claim 2, comprising: an instruction input device. Display control means for generating display data for performing image display according to the instruction content determined by the determination means;
The instruction input device according to claim 1, further comprising an image display member that displays an image based on display data generated by the display control unit. Display control means for generating display data for performing image display according to the instruction content determined by the determination means;
The instruction input device according to claim 1, wherein the planar member displays an image based on display data generated by the display control unit. An instruction input method using a planar member that displays an image and detects a nearby image,
A specifying step for specifying coordinates representing a position of the image on the planar member based on the image detected by the planar member;
The first and second images specified in the specifying step when the first image and the second image are successively detected in this order at different positions on the planar member. A determination step of determining a positional relationship of the first and second images on the planar member by comparing the coordinates of
An instruction input method comprising: a determination step for determining instruction contents according to the positional relationship determined in the determination step.   A portable electronic device comprising the instruction input device according to claim 1.   A program for operating the instruction input device according to any one of claims 1 to 6, wherein the instruction input processing program causes a computer to function as each of the above means.   A computer-readable recording medium in which the instruction input processing program according to claim 9 is recorded.

Images