JP2010113481A - Input device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a new input device for generating an input signal by detecting the movement of an object. <P>SOLUTION: The input device 100 includes a display/photosensor part 300 for detecting a neighboring image; a plurality of input signal generation parts 12 for detecting the movement of an object on the basis of the image of the object detected by the display/photosensor part 300, and for generating the input signal associated with the movement of the object; and an input signal generation control part 11 for selecting any one input signal generation part 12 from the plurality of input signal generation parts 12 on the basis of the image of the object detected by the display/photosensor part 300, and for making the selected input signal generation part 12 generate the input signal. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an input device used when a user inputs information to an information device, and more particularly to an input device that detects an object motion and generates an input signal.

  There are various types of input devices used when a user performs input to an information device such as a computer, such as a mouse, a pen tablet, a light pen, a joystick, a touch panel, a trackball, a pointing stick, and a keyboard. It is used properly.

  For example, a mouse is suitable for input via a GUI (graphical user interface), and a user can obtain a comfortable operation feeling by combining mouse movement, click, wheel operation, and the like. Further, the pen tablet can be operated with the same feeling as when drawing with a pen on paper, and is suitable for operation of drawing software or the like. There are suitable applications for other input devices.

Further, Patent Document 1 proposes an input device that is easy to fatigue and has good operability, which recognizes the shape of a manual operation as image information and generates an input signal.
JP 59-1332079 (released July 30, 1984)

  However, as indicated by the existence of various input devices depending on the application, humans further seek input devices that are easier to use, and it is beneficial to propose input devices having new characteristics. In other words, the input device described in Patent Document 1 is not an input device that satisfies all human needs.

  The present invention has been made in view of the above problems, and an object thereof is to provide a new input device.

  In order to solve the above problems, an input device according to the present invention detects a movement of an object based on a planar member that detects a nearby image and an image of the object detected by the planar member. One of the plurality of input signal generation means for generating an input signal associated with the movement of the object and the plurality of input signal generation means based on the image of the object detected by the planar member. An input signal generation control unit that selects one input signal generation unit and causes the selected input signal generation unit to generate the input signal is provided.

  According to the above configuration, the input device includes a plurality of input signal generation means, and selects the input signal generation means for generating the input signal based on the image of the object detected by the planar member. Therefore, the user can easily generate the input signal by selecting the more appropriate input signal generation means via the object. Furthermore, since the input signal generation means generates the input signal based on the image of the object detected by the planar member, the user can easily input without physically operating a special instrument. In addition, the selection of the input signal generation means and the input to the input device can be performed seamlessly. The correspondence between the movement of the object and the input signal is preferably different for each input signal generating means.

  In the input device according to the present invention, at least one of the plurality of input signal generating means detects that the object has pointed to a specific position on the planar member and indicates the specific position. A tablet signal generating means for generating an input signal is preferable.

  According to the above configuration, the input device can detect that the target object has indicated a specific position on the planar member and generate the input signal indicating the specific position. Even if the user does not hold the pen in the hand, it is possible to input with the same operational feeling as the operation of the pen tablet.

  In the input device according to the present invention, at least one of the plurality of input signal generation means detects that the object has moved on the planar member and generates an input signal indicating the amount of the movement. In addition, it is preferable that the mouse signal generation means generates an input signal indicating that a click operation has been performed by detecting that a part of the object has approached or contacted after being separated from the planar member.

  According to the above configuration, the input device detects that the object has moved on the planar member and generates the input signal indicating the amount of the movement, and a part of the object is It is possible to generate the above input signal indicating that there was a click operation by detecting contact after leaving the planar member, so that even if the user does not actually hold the mouse, it is equivalent to the mouse operation It is possible to input with a feeling of operation.

  In the input device according to the present invention, the mouse signal generation means detects that a part of the object has moved in a state of approaching or contacting the planar member, and indicates that there has been a wheel operation. May be generated.

  According to the above configuration, the input device detects that a part of the object has moved in contact with the planar member and generates the input signal indicating that there has been a wheel operation. Therefore, even if the user does not actually hold the mouse with a wheel, the user can input with an operation feeling equivalent to the mouse operation with a wheel.

  In the input device according to the present invention, the input signal generation control means compares the image of the object detected by the planar member with a reference image associated with each of the plurality of input signal generation means. Accordingly, it is preferable to select any one input signal generation unit from the plurality of input signal generation units.

  According to the above configuration, the input device selects an input signal generation unit to be used by comparing the image of the object with the reference image. Here, since the reference image is associated with any one of the input signal generation means, when the input signal generation means intended by the user is indicated by the form of the object, the input device Based on the image of the object, the input signal generation means intended by the user can be successfully selected.

  In the input device according to the present invention, the input signal generation control means is one selected from the group consisting of an image of the object detected by the planar member and a certain reference image that is moved, rotated, enlarged and reduced. It is preferable to determine whether or not to select an input signal generation unit associated with the certain reference image based on the image subjected to the above image processing or the overlapping rate with the certain reference image.

  According to the above configuration, the input device is configured such that the object presented by the user is standardized by performing processing such as movement, rotation, enlargement, and reduction on the reference image and then comparing it with the image of the object. Even if it deviates from this form, the input signal generation means intended by the user can be successfully selected.

  In the input device according to the present invention, the planar member displays an image, and an auxiliary image associated with the input signal generation unit selected by the input signal generation control unit is displayed on the planar member. It is preferable to further include auxiliary image display control means.

  According to the said structure, the auxiliary | assistant image linked | related with the said input signal production | generation means can be displayed on the said planar member. As a result, the input device can present to the user which input signal generation means is currently used to detect the movement of the object and generate a signal. An image that assists in performing input suitable for the input signal generation means, for example, the selected input signal generation means detects that the object has pointed to a specific position, and When an input signal indicating a specific position is generated, an image such as a grid (grid) that makes it easy for the user to specify the target position can be displayed on the planar member.

  In the input device according to the present invention, the auxiliary image may be an auxiliary image selected from the group consisting of a pen image, a mouse image, and a grid image.

  According to the above configuration, since the pen image, the mouse image, and the grid image are used as the auxiliary image, particularly when the input signal generation unit that performs input by an input method simulating tablet input or mouse input is selected. A suitable auxiliary image can be displayed.

  The control method of the input device according to the present invention includes a planar member that detects an image in the vicinity and a motion of the target object based on the image of the target object detected by the planar member, and the target object A plurality of input signal generating means for generating the input signal associated with the movement of the input device, the plurality of input signal generating means, and based on the image of the object detected by the planar member, It includes an input signal generation control step of selecting any one of the input signal generation means from the input signal generation means and causing the selected input signal generation means to generate the input signal.

  According to the said structure, there can exist an effect equivalent to the input device which concerns on this invention.

  According to said structure, there exists an effect similar to the image display / image detection apparatus which concerns on this invention.

  Further, a program for operating the image display / image detection apparatus according to the present invention, which is characterized in that the computer is driven as each of the above means, is also included in the scope of the present invention.

  Since the input device according to the present invention includes a plurality of input signal generation units, the user can select and input an appropriate input signal generation unit, and the input device can generate the input signal. Since the selection of the means is performed based on the detected image of the target object, the user can easily select an appropriate input signal generation means, and thereby can perform input to the input device successfully.

  The input device 100 according to an embodiment of the present invention includes a sensor built-in liquid crystal panel (planar member) 301 that can detect a nearby image, thereby reading and reading an image of the object 64. This is an apparatus for generating an input signal based on the image of the target object 64.

  In the present specification, the object refers to an object used for the user to input. Specifically, a hand is assumed. The input signal is a signal indicating information input by the user using the object. In addition, reading of an image of an object is also referred to as a scan below.

  In one embodiment, the input device 100 includes a plurality of input modes of a tablet mode and a mouse mode, and can be used by switching them.

  FIG. 15 is a schematic diagram showing an outline of the tablet mode. As shown in FIG. 15, in this mode, the user performs an operation with his / her hand (object 64) tied in a shape holding a pen. At this time, the input device 100 reads the shape of the user's hand using the sensor built-in liquid crystal panel 301 and detects the position of the pen nib virtually held by the hand as an input signal. Note that the input device 100 preferably displays a pen image 171 at the detected position, as shown in FIG. In the tablet mode, it is preferable to display a grid image 172 that assists the user in inputting.

  FIG. 17 is a schematic diagram showing an outline of the mouse mode. As shown in FIG. 17, in this mode, the user operates the hand (object 64) with a mouse. At this time, the input device 100 reads the shape of the user's hand using the sensor built-in liquid crystal panel 301, detects the movement of the mouse virtually held by the hand, and the like as an input signal. Note that the input device 100 preferably displays a mouse image 174 at the detected mouse position, as shown in FIG.

  The user can naturally switch between the tablet mode and the mouse mode in the input device 100. That is, the user can intuitively switch the input mode depending on whether the shape of the hand (object 64) is a shape with a pen or a shape with a mouse.

  Hereinafter, the structure of each member which implement | achieves the function mentioned above is demonstrated in detail. First, an outline of the sensor built-in liquid crystal panel 301 included in the input device 100 will be described.

(Outline of LCD panel with built-in sensor)
The sensor built-in liquid crystal panel 301 provided in the input device 100 is a liquid crystal panel capable of detecting an image of an object in addition to displaying data. Here, the image detection of the object is, for example, detection of a position pointed (touched) by the user with a finger or a pen, or reading (scanning) of an image of a printed material. The device used for display is not limited to a liquid crystal panel, and may be an organic EL (Electro Luminescence) panel or the like.

  The structure of the sensor built-in liquid crystal panel 301 will be described with reference to FIG. FIG. 2 is a diagram schematically showing a cross section of the sensor built-in liquid crystal panel 301. The sensor built-in liquid crystal panel 301 described here is an example, and any structure can be used as long as the display surface and the reading surface are shared.

  As shown in the figure, the sensor built-in liquid crystal panel 301 includes an active matrix substrate 51A disposed on the back surface side and a counter substrate 51B disposed on the front surface side, and has a structure in which a liquid crystal layer 52 is sandwiched between these substrates. is doing. The active matrix substrate 51A is provided with a pixel electrode 56, a data signal line 57, an optical sensor circuit 32 (not shown), an alignment film 58, a polarizing plate 59, and the like. The counter substrate 51B is provided with color filters 53r (red), 53g (green), 53b (blue), a light shielding film 54, a counter electrode 55, an alignment film 58, a polarizing plate 59, and the like. In addition, a backlight 307 is provided on the back surface of the sensor built-in liquid crystal panel 301.

  The photodiode 6 included in the photosensor circuit 32 is provided in the vicinity of the pixel electrode 56 provided with the blue color filter 53b, but is not limited to this configuration. It may be provided in the vicinity of the pixel electrode 56 provided with the red color filter 53r, or may be provided in the vicinity of the pixel electrode 56 provided with the green color filter 53g.

  Next, with reference to FIGS. 3A and 3B, two types of methods for detecting the position where the user touches the sensor built-in liquid crystal panel 301 with a finger or a pen will be described.

  FIG. 3A is a schematic diagram illustrating a state in which a position touched by the user is detected by detecting a reflected image. When the light 63 is emitted from the backlight 307, the optical sensor circuit 32 including the photodiode 6 detects the light 63 reflected by the object 64. Thereby, the reflected image of the target object 64 can be detected. Thus, the sensor built-in liquid crystal panel 301 can detect the touched position by detecting the reflected image.

  FIG. 3B is a schematic diagram illustrating a state in which a position touched by the user is detected by detecting a shadow image. As shown in FIG. 3B, the optical sensor circuit 32 including the photodiode 6 detects external light 61 transmitted through the counter substrate 51B and the like. However, when the object 64 is present, the incident of the external light 61 is hindered, so that the amount of light detected by the optical sensor circuit 32 is reduced. Thereby, the image of the target object 64 can be detected. Thus, the sensor built-in liquid crystal panel 301 can also detect a touched position by detecting a shadow image.

  As described above, the photodiode 6 may detect reflected light (shadow image) of the light emitted from the backlight 307 or may detect a shadow image caused by external light. Further, the two types of detection methods may be used in combination to detect both a shadow image and a reflected image at the same time.

(Main components of the input device)
Next, with reference to FIG. 4, the configuration of the main part of the input device 100 will be described. FIG. 4 is a block diagram illustrating a main configuration of the input device 100. As illustrated, the input device 100 includes one or more display / light sensor units 300, a circuit control unit 600, a data processing unit 700, a main control unit 800, a storage unit 901, a primary storage unit 902, an operation unit 903, an external device A communication unit 907, an audio output unit 908, and an audio input unit 909 are provided.

  The display / light sensor unit 300 is a so-called liquid crystal display device with a built-in light sensor. The display / light sensor unit 300 includes a sensor built-in liquid crystal panel 301, a backlight 307, and a peripheral circuit 309 for driving them.

  The sensor built-in liquid crystal panel 301 includes a plurality of pixel circuits 31 and photosensor circuits 32 arranged in a matrix. The detailed configuration of the sensor built-in liquid crystal panel 301 will be described later.

  The peripheral circuit 309 includes a liquid crystal panel drive circuit 304, an optical sensor drive circuit 305, a signal conversion circuit 306, and a backlight drive circuit 308.

  The liquid crystal panel driving circuit 304 outputs a control signal (G) and a data signal (S) in accordance with the timing control signal (TC1) and the data signal (D) from the display control unit 601 of the circuit control unit 600, and the pixel circuit 31. It is a circuit which drives. Details of the driving method of the pixel circuit 31 will be described later.

  The optical sensor driving circuit 305 is a circuit that drives the optical sensor circuit 32 by applying a voltage to the signal line (R) in accordance with a timing control signal (TC2) from the sensor control unit 602 of the circuit control unit 600. Details of the driving method of the optical sensor circuit 32 will be described later.

  The signal conversion circuit 306 is a circuit that converts the sensor output signal (SS) output from the optical sensor circuit 32 into a digital signal (DS) and transmits the converted signal to the sensor control unit 602.

  The backlight 307 includes a plurality of white LEDs (Light Emitting Diodes) and is disposed on the back surface of the sensor built-in liquid crystal panel 301. When a power supply voltage is applied from the backlight drive circuit 308, the backlight 307 is turned on and irradiates the sensor built-in liquid crystal panel 301 with light. Note that the backlight 307 is not limited to white LEDs, and may include LEDs of other colors. The backlight 307 may include, for example, a cold cathode fluorescent lamp (CCFL) instead of the LED.

  The backlight driving circuit 308 applies a power supply voltage to the backlight 307 when the control signal (BK) from the backlight control unit 603 of the circuit control unit 600 is at a high level, and conversely, the backlight control unit 603. When the control signal from is at a low level, no power supply voltage is applied to the backlight 307.

  Next, the circuit control unit 600 will be described. The circuit control unit 600 has a function as a device driver that controls the peripheral circuit 309 of the display / light sensor unit 300. The circuit control unit 600 includes a display control unit 601, a sensor control unit 602, a backlight control unit 603, and a display data storage unit 604.

  The display control unit 601 receives display data from the display data processing unit 701 of the data processing unit 700, and performs timing control on the liquid crystal panel driving circuit 304 of the display / light sensor unit 300 in accordance with an instruction from the display data processing unit 701. A signal (TC1) and a data signal (D) are transmitted, and the received display data is displayed on the sensor built-in liquid crystal panel 301.

  The display control unit 601 temporarily stores the display data received from the display data processing unit 701 in the display data storage unit 604. Then, a data signal (D) is generated based on the primary stored display data. The display data storage unit 604 is, for example, a video random access memory (VRAM).

  The sensor control unit 602 transmits a timing control signal (TC2) to the optical sensor driving circuit 305 of the display / optical sensor unit 300 in accordance with an instruction from the sensor data processing unit 703 of the data processing unit 700, and the sensor built-in liquid crystal panel 301. Run the scan with.

  In addition, the sensor control unit 602 receives a digital signal (DS) from the signal conversion circuit 306. Then, image data is generated based on the digital signal (DS) corresponding to the sensor output signal (SS) output from all the optical sensor circuits 32 included in the sensor built-in liquid crystal panel 301. That is, the image data read in the entire reading area of the sensor built-in liquid crystal panel 301 is generated. Then, the generated image data is transmitted to the sensor data processing unit 703.

  The backlight control unit 603 transmits a control signal (BK) to the backlight drive circuit 308 of the display / light sensor unit 300 in accordance with instructions from the display data processing unit 701 and the sensor data processing unit 703 to drive the backlight 307. Let

  When the input device 100 includes a plurality of display / light sensor units 300, the display control unit 601 receives an instruction from the data processing unit 700 regarding which display / light sensor unit 300 displays the display data. In response, the liquid crystal panel drive circuit 304 of the display / light sensor unit 300 is controlled according to the instruction. When the sensor control unit 602 receives an instruction from the data processing unit 700 as to which display / light sensor unit 300 is to scan the object, The sensor drive circuit 305 is controlled and a digital signal (DS) is received from the signal conversion circuit 306 of the display / light sensor unit 300 according to the instruction.

  Next, the data processing unit 700 will be described. The data processing unit 700 has a function as middleware that gives an instruction to the circuit control unit 600 based on a “command” received from the main control unit 800. Details of the command will be described later.

  The data processing unit 700 includes a display data processing unit 701 and a sensor data processing unit 703. When the data processing unit 700 receives a command from the main control unit 800, at least one of the display data processing unit 701 and the sensor data processing unit 703 depends on the value of each field (described later) included in the received command. One works.

  The display data processing unit 701 receives display data from the main control unit 800, and gives instructions to the display control unit 601 and the backlight control unit 603 according to the command received by the data processing unit 700, and displays the received display data. The image is displayed on the sensor built-in liquid crystal panel 301. The operation of the display data processing unit 701 according to the command will be described later.

  The sensor data processing unit 703 gives an instruction to the sensor control unit 602 and the backlight control unit 603 according to the command received by the data processing unit 700.

  The sensor data processing unit 703 receives image data from the sensor control unit 602 and stores the image data in the image data buffer 704 as it is. Then, in accordance with the command received by the data processing unit 700, the sensor data processing unit 703 performs “whole image data”, “partial image data (partial image coordinate data) based on the image data stored in the image data buffer 704. At least one of “including coordinate data” and “coordinate data” is transmitted to the main control unit 800. The whole image data, partial image data, and coordinate data will be described later. The operation of the sensor data processing unit 703 according to the command will be described later.

  Next, the main control unit 800 executes an application program. The main control unit 800 reads the program stored in the storage unit 901 into a primary storage unit 902 configured by, for example, a RAM (Random Access Memory) and executes the program.

  An application program executed by the main control unit 800 causes the data processing unit 700 to display display data on the sensor built-in liquid crystal panel 301 or to scan an object on the sensor built-in liquid crystal panel 301. Send commands and display data. When “data type” is designated as a command, at least one of whole image data, partial image data, and coordinate data is received from the data processing unit 700 as a response to the command.

  The circuit control unit 600, the data processing unit 700, and the main control unit 800 can be configured by a CPU (Central Processing Unit), a memory, and the like, respectively. The data processing unit 700 may be configured by a circuit such as an ASIC (application specific integrate circuit).

  Next, the storage unit 901 stores programs and data executed by the main control unit 800 as shown in the figure. The program executed by the main control unit 800 may be separated into an application-specific program and a general-purpose program that can be shared by each application.

  Next, the operation unit 903 receives an input operation of the user of the input device 100. The operation unit 903 includes input devices such as a switch, a remote controller, a mouse, and a keyboard, for example. Then, the operation unit 903 generates a control signal corresponding to the input operation of the user of the input device 100 and transmits the generated control signal to the main control unit 800.

  As an example of the switch, a hardware switch such as a power switch 905 that switches power on and off and a user switch 906 to which a predetermined function is assigned in advance is assumed.

  In addition, the input device 100 includes an external communication unit 907 for communicating with an external device by wireless / wired communication, an audio output unit 908 such as a speaker for outputting audio, and an audio such as a microphone for inputting an audio signal. An input unit 909 or the like may be provided as appropriate.

(Command details)
Next, details of commands transmitted from the main control unit 800 to the data processing unit 700 will be described with reference to FIGS. 5 and 6. FIG. 5 is a diagram schematically illustrating an example of a command frame structure. FIG. 6 is a diagram for explaining an example of values that can be specified for each field included in the command and an outline thereof.

  As shown in FIG. 5, the commands are “header”, “data acquisition timing”, “data type”, “scan method”, “scan image gradation”, “scan resolution”, “scan panel”, “display panel”. "And" Reserve "fields. In each field, for example, values shown in FIG. 6 can be designated.

  The “header” field is a field indicating the start of a frame. As long as it is possible to identify the “header” field, the value of the “header” field may be any value.

  Next, the “data acquisition timing” field is a field for designating a timing at which data should be transmitted to the main control unit 800. In the “data acquisition timing” field, for example, values “00” (sense), “01” (event), and “10” (all) can be specified.

  Here, “sense” designates that the latest data is transmitted immediately. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “sense”, the latest data specified in the “data type” field is immediately updated to the main control unit 800. Send to.

  The “event” designates transmission at a timing when a change occurs in the image data received from the sensor control unit 602. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “event”, the image that receives the data specified in the “data type” field from the sensor control unit 602. The data is transmitted to the main control unit 800 at a timing when a change larger than a predetermined threshold occurs.

  “All” designates data transmission at a predetermined cycle. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “all”, the data designated in the “data type” field is transferred to the main control unit 800 at a predetermined cycle. Send to. The predetermined period coincides with the period in which the optical sensor circuit 32 performs scanning.

  Next, the “data type” field is a field for designating the type of data acquired from the sensor data processing unit 703. In the “data type” field, for example, values of “001” (coordinates), “010” (partial image), and “100” (entire image) can be specified. Furthermore, by adding these values, “coordinates” and “partial image” / “whole image” can be specified simultaneously. For example, when “coordinate” and “partial image” are specified at the same time, “011” can be specified.

  When the sensor data processing unit 703 receives a command whose value of the “data type” field is “whole image”, the sensor data processing unit 703 transmits the image data itself stored in the image data buffer 704 to the main control unit 800. The image data itself stored in the image data buffer 704 is referred to as “whole image data”.

  In addition, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 selects a portion where a change larger than a predetermined threshold has occurred from the image data received from the sensor control unit 602. A region to be included is extracted, and image data of the extracted region is transmitted to the main control unit 800. Here, the image data is referred to as “partial image data”. When a plurality of partial image data are extracted, the sensor data processing unit 703 transmits each extracted partial image data to the main control unit 800.

  Further, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 detects representative coordinates in the partial image data and indicates the position of the representative coordinates in the partial image data. The coordinate data is transmitted to the main control unit 800. The representative coordinates include, for example, the coordinates of the center of the partial image data, the coordinates of the center of gravity of the partial image data, and the like.

  Next, when receiving a command whose value of the “data type” field is “coordinate”, the sensor data processing unit 703 transmits coordinate data indicating the position of the representative coordinate in the entire image data to the main control unit 800. When a plurality of partial image data are extracted, the sensor data processing unit 703 detects representative coordinates in the entire image data of the extracted partial image data, and the coordinate data indicating the representative coordinates is detected. Each is transmitted to the main control unit 800 (multi-point detection).

  Specific examples of the whole image data, the partial image data, and the coordinate data will be described later with reference to schematic diagrams.

  Next, the “scan method” field is a field for designating whether or not the backlight 307 is turned on at the time of executing the scan. In the “scan method” field, for example, values of “00” (reflection), “01” (transmission), and “10” (reflection / transmission) can be designated.

  “Reflection” designates that scanning is performed with the backlight 307 turned on. Therefore, when the sensor data processing unit 703 receives a command whose “scan method” field value is “reflection”, the sensor data processing unit 703 performs sensor control so that the optical sensor driving circuit 305 and the backlight driving circuit 308 operate in synchronization. An instruction is given to the unit 602 and the backlight control unit 603.

  “Transmission” specifies that scanning is performed with the backlight 307 turned off. Therefore, when the sensor data processing unit 703 receives a command whose “scan method” field value is “transparent”, the sensor control unit 602 operates the optical sensor driving circuit 305 and does not operate the backlight driving circuit 308. Instructions to the backlight control unit 603. Note that “reflection / transmission” specifies that scanning is performed using both “reflection” and “transmission”.

  Next, the “scanned image gradation” field is a field for designating gradations of the partial image data and the entire image data. In the “scanned image gradation” field, for example, values of “00” (binary) and “01” (multivalue) can be designated.

  When the sensor data processing unit 703 receives a command whose “scan image gradation” field value is “binary”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 as monochrome data. .

  When the sensor data processing unit 703 receives a command whose “scanned image gradation” field value is “multivalued”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 as multitone data. To do.

  Next, the “scan resolution” field is a field for designating the resolution of the partial image data and the entire image data. In the “resolution” field, for example, values of “0” (high) and “1” (low) can be designated.

  Here, “high” designates a high resolution. Therefore, when the sensor data processing unit 703 receives a command whose “scan resolution” field value is “high”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 with high resolution. For example, it is desirable to designate “high” for image data (image data such as a fingerprint) to be subjected to image processing such as image recognition.

  “Low” designates a low resolution. Therefore, when the sensor data processing unit 703 receives a command whose “scan resolution” field value is “low”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 at a low resolution. For example, it is desirable to designate “low” for image data (such as touched finger or hand image data) that only needs to be recognized.

  Next, the “scan panel” field is a field for designating which display / light sensor unit 300 is to scan the object. In the “scan panel” field, for example, values of “001” (first display / light sensor unit 300) and “010” (second display / light sensor unit 300) can be designated. By adding these values, a plurality of display / light sensor units 300 can be specified at the same time. For example, when the first and second display / light sensor units 300 are specified at the same time, “011” can be specified.

  Here, the sensor data processing unit 703 controls the light sensor driving circuit 305 and the backlight driving circuit 308 of the display / light sensor unit 300 specified in the “scan panel” field of the received command. An instruction is given to 602 and the backlight control unit 603.

  Next, the “display panel” field is a field for designating which display / light sensor unit 300 displays the display data. In the “display panel” field, for example, values “001” (first display / light sensor unit 300) and “010” (second display / light sensor unit 300) can be designated. By adding these values, a plurality of display / light sensor units 300 can be specified at the same time. For example, when the first and second display / light sensor units 300 are specified at the same time, “011” can be specified.

  For example, when the display data processing unit 701 receives a command whose value of the “display panel” field is the first display / light sensor unit 300, the display data processing unit 701 displays the display data on the first display / light sensor unit 300. Therefore, an instruction is given to the display control unit 601 and the backlight control unit 603 so as to control the liquid crystal panel driving circuit 304 and the backlight driving circuit 308 of the display / light sensor unit 300.

  Next, the “reserved” field is a field that is appropriately specified when it is necessary to further specify information other than information that can be specified in the above-described fields.

  Note that an application executed by the main control unit 800 does not need to use all the above-described fields when transmitting a command, and an invalid value (such as a NULL value) may be set for a field that is not used. .

  When the user wants to acquire coordinate data of a position touched with a finger or pen, a command specifying “coordinate” in the “data type” field is transmitted to the data processing unit 700. Therefore, it is desirable to specify “all” in the “data acquisition timing” field of the command to acquire coordinate data. Further, since it is only necessary to acquire coordinate data of the touched position, the scanning accuracy may not be high. Therefore, “low” may be specified as the value of the “resolution” field of the command.

  Further, when “coordinate” is specified in the “data type” field of the command, for example, when the user touches the sensor built-in liquid crystal panel 301 with a plurality of fingers or pens at the same time, the coordinate data of the touched position is used. Can be acquired (multi-point detection).

  When acquiring image data of an object such as a document, a command specifying “whole image” in the “data type” field is transmitted to the data processing unit 700. Since it is common to perform a scan in a stationary state, it is not necessary to periodically perform the scan. Therefore, in this case, it is desirable to designate “sense” or “event” in the “data acquisition timing” field. When scanning an object such as a document, it is desirable that the scanning accuracy is high so that the user can easily read the characters. Therefore, it is desirable to designate “high” in the “resolution” field.

(Whole image data / Partial image data / Coordinate data)
Next, the whole image data, the partial image data, and the coordinate data will be described with reference to FIG. The image data shown in FIG. 7A is image data obtained as a result of scanning the entire sensor built-in liquid crystal panel 301 when the object is not placed on the sensor built-in liquid crystal panel 301. The image data shown in FIG. 7B is image data obtained as a result of scanning the entire sensor-equipped liquid crystal panel 301 when the user touches the sensor-equipped liquid crystal panel 301 with a finger.

  When the user touches the sensor built-in liquid crystal panel 301 with a finger, the amount of light received by the photosensor circuit 32 in the vicinity of the touch changes, so that the voltage output from the photosensor circuit 32 changes, and as a result, In the image data generated by the sensor control unit 602, the brightness of the pixel value of the portion touched by the user changes.

  In the image data shown in FIG. 7B, the brightness of the pixel value of the portion corresponding to the user's finger is higher than that in the image data shown in FIG. In the image data shown in FIG. 7B, the smallest rectangular area (area PP) that includes all pixel values whose lightness changes more than a predetermined threshold is “partial image data”.

  The image data indicated by the area AP is “whole image data”.

  Also, the coordinate data in the whole image data (area AP) of the representative coordinates Z of the partial image data (area PP) is (Xa, Ya), and the coordinate data in the partial image data (area PP) is (Xp, Yp). It is.

(Configuration of sensor built-in liquid crystal panel)
Next, the configuration of the sensor built-in liquid crystal panel 301 and the configuration of the peripheral circuit 309 of the sensor built-in liquid crystal panel 301 will be described with reference to FIG. FIG. 8 is a block diagram showing the main part of the display / light sensor unit 300, particularly the configuration of the sensor built-in liquid crystal panel 301 and the configuration of the peripheral circuit 309.

  The sensor built-in liquid crystal panel 301 includes a pixel circuit 31 for setting light transmittance (brightness) and an optical sensor circuit 32 that outputs a voltage corresponding to the intensity of light received by the sensor. The pixel circuit 31 is used as a general term for the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b corresponding to the red, green, and blue color filters, respectively.

  The pixel circuits 31 are arranged on the sensor built-in liquid crystal panel 301 in the column direction (vertical direction) and 3n in the row direction (horizontal direction). A set of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b is continuously arranged in the row direction (lateral direction). This set forms one pixel.

  In order to set the light transmittance of the pixel circuit 31, first, the high level voltage (TFT 33 is turned on) to the scanning signal line Gi connected to the gate terminal of the TFT (Thin Film Transistor) 33 included in the pixel circuit 31. Voltage). Thereafter, a predetermined voltage is applied to the data signal line SRj connected to the source terminal of the TFT 33 of the R pixel circuit 31r. Similarly, the light transmittance is also set for the G pixel circuit 31g and the B pixel circuit 31b. Then, by setting these light transmittances, an image is displayed on the sensor built-in liquid crystal panel 301.

  Next, the photosensor circuit 32 is arranged for each pixel. One pixel may be arranged in the vicinity of each of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b.

  In order for the optical sensor circuit 32 to output a voltage according to the light intensity, first, the sensor readout line RWi connected to one electrode of the capacitor 35 and the anode terminal of the photodiode 6 are connected. A predetermined voltage is applied to the sensor reset line RSi. In this state, when light is incident on the photodiode 6, a current corresponding to the amount of incident light flows through the photodiode 6. And according to the said electric current, the voltage of the connection point (henceforth connection node V) of the other electrode of the capacitor | condenser 35 and the cathode terminal of the photodiode 6 falls. When the power supply voltage VDD is applied to the voltage application line SDj connected to the drain terminal of the sensor preamplifier 37, the voltage at the connection node V is amplified and output from the source terminal of the sensor preamplifier 37 to the sensing data output line SPj. Based on the output voltage, the amount of light received by the optical sensor circuit 32 can be calculated.

  Next, the liquid crystal panel drive circuit 304, the optical sensor drive circuit 305, and the sensor output amplifier 44, which are peripheral circuits of the sensor built-in liquid crystal panel 301, will be described.

  The liquid crystal panel drive circuit 304 is a circuit for driving the pixel circuit 31, and includes a scanning signal line drive circuit 3041 and a data signal line drive circuit 3042.

  The scanning signal line driving circuit 3041 sequentially selects one scanning signal line from the scanning signal lines G1 to Gm for each line time based on the timing control signal TC1 received from the display control unit 601, and A high level voltage is applied to the selected scanning signal line, and a low level voltage is applied to the other scanning signal lines.

  Based on the display data D (DR, DG, and DB) received from the display controller 601, the data signal line driver circuit 3042 generates a predetermined voltage corresponding to the display data for one row for each line time. The data signal lines SR1 to SRn, SG1 to SGn, and SB1 to SBn are applied (line sequential method). Note that the data signal line driver circuit 3042 may be driven by a dot sequential method.

  The optical sensor driving circuit 305 is a circuit for driving the optical sensor circuit 32. Based on the timing control signal TC2 received from the sensor control unit 602, the optical sensor driving circuit 305 selects a predetermined sensor readout signal line from the sensor readout signal lines RW1 to RWm for each line time. A read voltage is applied, and a voltage other than a predetermined read voltage is applied to the other sensor read signal lines. Similarly, based on the timing control signal TC2, a predetermined reset voltage is applied to the sensor reset signal line selected from the sensor reset signal lines RS1 to RSm for each line time, and the others. A voltage other than a predetermined reset voltage is applied to the sensor reset signal line.

  The sensing data output signal lines SP1 to SPn are grouped into p groups (p is an integer of 1 to n), and the sensing data output signal lines belonging to each group are connected via a switch 47 that is sequentially turned on in time division. And connected to the sensor output amplifier 44. The sensor output amplifier 44 amplifies the voltage from the group of sensing data output signal lines connected by the switch 47 and outputs the amplified voltage to the signal conversion circuit 306 as sensor output signals SS (SS1 to SSp).

(More detailed configuration of input device)
Next, a more detailed configuration of the input device 100 will be described with reference to FIG. Here, in order to make the description easy to understand, the description of the operations of the data processing unit 700 and the circuit control unit 600 located between the main control unit 800 and the display / light sensor unit 300 is omitted. However, to be precise, when displaying data and scanning an object, each unit of the main control unit 800 transmits a command to the data processing unit 700, and the data processing unit 700 controls the circuit control unit 600 based on the command. Then, the circuit control unit 600 transmits a signal to the display / light sensor unit 300. Further, the main control unit 800 acquires the entire image data, the partial image data, and the coordinate data from the data processing unit 700 as a response to the command transmitted to the data processing unit 700.

  FIG. 1 is a block diagram showing a more detailed configuration of the input device 100. As shown in FIG. 1, the main controller 800 includes an input signal generation controller (input signal generation controller) 11, a tablet signal generator (input signal generator, tablet signal generator) 13, and a mouse signal generator (input). A signal generation unit, a mouse signal generation unit) 14, an auxiliary image display control unit (auxiliary image display control unit) 15, a scan data holding unit 16, a reference image processing data holding unit 17, and a mouse state holding unit 18. The storage unit 901 includes a reference image storage unit 21 and an auxiliary image storage unit 22.

  Both the tablet signal generation unit 13 and the mouse signal generation unit 14 are an input signal generation unit (input signal generation unit) 12 that generates an input signal. The tablet signal generation unit 13 detects the movement of the object 64 as an operation on the tablet, and generates an input signal associated with the operation. The mouse signal generator 14 detects the movement of the object 64 as a mouse operation, and generates an input signal associated with the operation.

  The input signal generation control unit 11 selects any one of the input signal generation units 12 and causes the selected input signal generation unit 12 to generate an input signal. As described above, in one embodiment, the input device 100 has two input modes of the tablet mode and the mouse mode, and the input signal generation control unit 11 determines which input mode to execute. When the tablet mode is executed, the tablet signal generation unit 13 causes the mouse signal generation unit 14 to generate an input signal when the mouse mode is executed.

  The auxiliary image display control unit 15 displays the auxiliary image 170 corresponding to each input mode on the sensor built-in liquid crystal panel 301.

  The scan data holding unit 16 temporarily stores scan data (image) 160 of the object 64 read by the sensor built-in liquid crystal panel 301.

  The reference image processing data holding unit 17 temporarily stores data indicating the content of the image processing added to the reference image 161 in steps to be described later.

  The reference image storage unit 21 stores in advance the reference image 161 used by the input signal generation control unit 11 and the input signal generation unit 12 to analyze the scan data 160 and data associated with the reference image 161. It is.

  The auxiliary image storage unit 22 stores the auxiliary image 170 that the auxiliary image display control unit 15 displays on the sensor built-in liquid crystal panel 301 in advance.

  Next, an example of the operation of the input device 100 will be described in detail with reference to FIGS.

(Mode switching process)
FIG. 9 is a flowchart illustrating the mode switching process of the input device 100. For example, the input device 100 performs mode switching processing in an aspect described later when input is interrupted for a certain period of time immediately after activation. Hereinafter, each step of the mode switching process will be described with reference to the drawings. The mode switching process is started from step S1.

  In step S1, the input device 100 scans the object 64. In one embodiment, the input signal generation control unit 11 acquires the entire image data from the sensor data processing unit 703 and stores it in the scan data holding unit 16 as scan data 160.

  Specifically, first, the input signal generation control unit 11 sends the above-mentioned command with “data type” as “whole image data” and “data acquisition timing” as “event” to the data processing unit 700. The input signal generation control unit 11 causes the scan data holding unit 16 to store the entire image data obtained from the sensor data processing unit 703 as a response to the command. When the input signal generation control unit 11 determines the data acquisition timing and the change in the acquired data, the “data acquisition timing” of the command may be “all” or “sense”.

  In step S2 and subsequent steps, the input device 100 selects the input mode to be executed by sequentially comparing the scan data 160 and the reference image 161 corresponding to each input mode.

  In one embodiment, in step S2, the input signal generation control unit 11 compares the scan data 160 acquired from the scan data holding unit 16 with the reference image 161 corresponding to the mouse mode called from the reference image storage unit 21. . If the scan data 160 is similar to the reference image 161 corresponding to the mouse mode, the mouse mode is executed. If the scan data 160 is not similar, the process proceeds to step S3. The mouse mode execution process will be described later.

  More specifically, first, the input signal generation control unit 11 acquires the scan data 160 from the scan data holding unit 16. At this time, as shown in FIG. 10 (a), when the good hand (object 64) holding the mouse is placed on the sensor-equipped liquid crystal panel 301, the scan data 160 is shown in FIG. 10 (b). It will be as shown in In the figure, the shaded area is an approach / contact area where the object 64 and the sensor built-in liquid crystal panel 301 are approaching or in contact with each other, and the above-described scanning method is “reflection”. Is a high luminance region that is equal to or higher than a predetermined threshold value, and is a low luminance region that is equal to or lower than a predetermined threshold value when the scanning method is “transmission”.

  Here, as the reference image 161 corresponding to the mouse mode (associated with the mouse signal generation unit 14), an image similar to that shown in FIG. 11B is prepared, and a reference image storage unit is prepared. 21 is stored in advance. In addition, since how to hold the mouse and the shape of the hand may differ depending on the person, it is preferable to prepare a plurality of reference images corresponding to each and store them in the reference image storage unit 21 in advance.

  The input signal generation control unit 11 sequentially calls the reference image 161 (associated with the mouse signal generation unit 14) corresponding to such a mouse mode from the reference image storage unit 21, compares it with the scan data 160, It is determined whether or not they are similar.

  In one embodiment, the similarity between the scan data 160 and the reference image 161 indicates, for example, the following case. That is, first, the scan data 160 and the reference image 161 are superimposed, and the contact / contact area overlap rate is calculated. Next, as shown in FIG. 11B, the reference image 161 is rotated, and the rotated reference image 161 and the scan data 160 are overlapped to calculate the contact / contact area overlap rate. The reference image 161 that has been rotated in advance may be stored in the reference image storage unit 21 in advance. In addition to the rotation of the reference image 161, the reference image 161 may be overlapped with the scan data 160 and the overlap ratio of the contact / contact area may be calculated even when the reference image 161 is moved up, down, left and right, and enlarged / reduced. . When the duplication rate obtained by the above processing exceeds a predetermined value, for example, 80%, it is determined that the scan data 160 and the reference image 161 are similar. The overlap rate can be obtained by a calculation formula, for example, (overlapping contact / area of contact region / area of contact / contact region in reference image 161). Further, the processing such as rotation, movement, and enlargement / reduction of the reference image 161 does not need to be performed in detail, and may be appropriately set within a range that can be assumed when the user generally uses a mouse.

  Here, as shown in FIG. 10A, when the object 64 is a good hand holding the mouse and the scan data 160 is an image as shown in FIG. 11 (a), the input signal generation control unit 11 determines that the scan data 160 and the reference image 161 are similar, and instructs the mouse signal generation unit 14 to generate an input signal. As a result, the mouse mode is executed. If it is determined that the scan data 160 and the reference image 161 are not similar, the process proceeds to step S3.

  In one embodiment, in step S <b> 3, the input signal generation control unit 11 compares the scan data 160 acquired from the scan data holding unit 16 with the reference image 161 corresponding to the tablet mode called from the reference image storage unit 21. . If the scan data 160 is similar to the reference image 161 corresponding to the tablet mode, the tablet mode is executed. If the scan data 160 is not similar, no input mode is selected, and the process returns to step S1.

  More specifically, the input signal generation control unit 11 acquires the scan data 160 from the scan data holding unit 16 in substantially the same manner as step S2. At this time, as shown in FIG. 12A, when a good hand (object 64) holding the pen is placed on the sensor-equipped liquid crystal panel 301, the scan data 160 is shown in FIG. It will be as shown in In the drawing, the description of the grayed out area is the same as in step S2.

  Here, as a reference image 161 corresponding to the tablet mode (associated with the tablet signal generation unit 13), an image similar to that shown in FIG. 13B is prepared, and a reference image storage unit is prepared. 21 is stored in advance. Since how to hold the pen and the shape of the hand may differ depending on the person, it is preferable to prepare a plurality of reference images corresponding to the pens and store them in the reference image storage unit 21 in advance.

  The input signal generation control unit 11 sequentially calls the reference image 161 (associated with the tablet signal generation unit 13) corresponding to such a tablet mode from the reference image storage unit 21, compares it with the scan data 160, It is determined whether or not they are similar. The description of the similarity determination is based on step S2. For reference, a rotated reference image 161 corresponding to the tablet mode is shown in FIG. Processing such as rotation, movement, and enlargement / reduction of the reference image 161 in this step does not need to be performed in detail, and may be set as appropriate within a range that a user can generally assume when using a pen.

  Here, as shown in FIG. 12A, when the object 64 is a good hand holding a pen and the scan data 160 is an image as shown in FIG. 13 (a), the input signal generation control unit 11 determines that the scan data 160 and the reference image 161 are similar, and instructs the tablet signal generation unit 13 to generate an input signal. The tablet mode is executed. If it is determined that the scan data 160 and the reference image 161 are not similar, the process returns to step S1.

(Tablet mode)
FIG. 14 is a flowchart illustrating tablet mode processing of the input device 100. Hereinafter, each step of the tablet mode process will be described with reference to the drawings. The tablet mode process is started from step S11.

  In step S <b> 11, the input device 100 displays a grid image 172 as shown in FIG. 15 on the sensor built-in liquid crystal panel 301. In one embodiment, when the tablet mode is executed (when an instruction to generate an input signal is given from the input signal generation control unit 11 to the tablet signal generation unit 13), the tablet signal generation unit 13 displays an auxiliary image. The control unit 15 is instructed to display the grid image 172 on the sensor built-in liquid crystal panel 301. In response to this, the auxiliary image display control unit 15 calls the data of the grid image 172 from the auxiliary image storage unit 22 and displays it on the sensor built-in liquid crystal panel 301.

  Specifically, when receiving an instruction from the input signal generation control unit 11 to generate an input signal, the tablet signal generation unit 13 first places the grid image 172 on the sensor built-in liquid crystal panel 301 on the auxiliary image display control unit 15. Command to display. Upon receiving the command, the auxiliary image display control unit 15 calls the data of the grid image 172 from the auxiliary image storage unit 22 and sends the data to the display data processing unit 701, and uses the “display panel” as the display / light sensor unit 300. By sending the command to the data processing unit 700, a grid image is displayed on the sensor built-in liquid crystal panel 301. Thereafter, the process proceeds to step S12.

  As the grid image 172, for example, a grid (grid) that covers the entire sensor-equipped liquid crystal panel 301 as shown in FIG. 15 can be used. Also good. In addition, the size of the grid of the grid may be set as appropriate depending on the application. If there is an image already displayed on the sensor built-in liquid crystal panel 301, the auxiliary image display control unit 15 may send an image obtained by combining the image and the grid image 172 to the data processing unit 700. .

  In step S12, the input device 100 scans the object 64. In one embodiment, the tablet signal generation unit 13 acquires the entire image data from the sensor data processing unit 703 and stores it in the scan data holding unit 16 as scan data 160. Thereafter, the process proceeds to step S13. If the mode switching process has been performed immediately before and the previous scan data 160 has already been stored in the scan data holding unit 16, this step can be omitted.

  More specifically, first, the tablet signal generation unit 13 sends the above-mentioned command with “data type” as “whole image data” and “data acquisition timing” as “event” to the data processing unit 700. The tablet signal generation unit 13 stores the entire image data obtained from the sensor data processing unit 703 in response to the command as scan data 160 in the scan data holding unit 16. When the tablet signal generation unit 13 determines the data acquisition timing and the change in the acquired data, the “data acquisition timing” of the command may be “all” or “sense”.

  In step S13, the input apparatus 100 determines whether or not to execute the mode switching process by comparing the scan data 160 with the reference image 161 corresponding to the tablet mode. In one embodiment, in step S <b> 13, the tablet signal generation unit 13 compares the scan data 160 acquired from the scan data holding unit 16 with the reference image 161 corresponding to the tablet mode called from the reference image storage unit 21. If the scan data 160 is similar to the reference image 161 corresponding to the tablet mode, the process proceeds to step S14. If the scan data 160 is not similar, the tablet mode is terminated and the above-described mode switching process is executed. To do.

  The description of the comparison between the scan data 160 and the reference image 161 is based on step S3. In addition, for the reference image 161 in which the overlap ratio between the scan data 160 and the contact / contact area exceeds the predetermined value, the reference data Data (horizontal movement distance X, vertical movement distance Y, rotation angle θ, magnification M) used for movement, rotation, enlargement / reduction, and the like applied to the image and information P indicating the reference image 161 are used as a reference. It is stored in the image processing data holding unit 17.

  In step S <b> 14, the input device 100 detects a pen position that the user instructs using the object 64. In one embodiment, the tablet signal generation unit 13 calls data related to the reference image 161 processed to be similar to the scan data 160 in step S13 from the reference image processing data holding unit 17 and the reference image storage unit 21, and Based on the data, the pen position is calculated.

  More specifically, the tablet signal generation unit 13 sends a horizontal movement distance X, a vertical movement distance Y, and a rotation angle θ stored in the reference image processing data holding unit 17 from the reference image processing data holding unit 17. The enlargement ratio M and information P are acquired. Next, the tablet signal generation unit 13 acquires the designated coordinates 162 (PX, PY) associated with the reference image 161 indicated by the information P from the reference image storage unit 21. As shown in FIG. 13, the designated coordinates 162 indicate a position (position pointed by the tip of the pen) indicated by the object 64 (a hand holding the pen).

  The tablet signal generation unit 13 calculates the pen position based on these data. For example, the horizontal coordinate of the pen position is (PXcos θ−PY sin θ) × M + X, and the vertical coordinate is (PX sin θ + PY cos θ) × M + Y. Thereafter, the process proceeds to step S15.

  In steps S15 to S18, the input device 100 determines whether or not the detection of the pen position is successful in step S14 (step S15), and if successful, displays the pen image 171 at the pen position (step S16). The pen position is output as an input signal (step S17). On the other hand, if the detection of the pen position fails, the pen image 171 is erased (step S18). Then, it returns to step S12. In one embodiment, the series of operations is performed by the tablet signal generation unit 13. Further, whether or not the detection of the pen position is successful is determined, for example, based on whether or not the pen position exists within a predetermined area.

  Specifically, in step S16, the tablet signal generation unit 13 instructs the auxiliary image display control unit 15 to display the pen image 171 at the pen position. The auxiliary image display control unit 15 calls the pen image 171 from the auxiliary image storage unit 22, combines the pen image 171 with the pen position on the screen (grid image 172 or the like) displayed below the pen image 171, and the display data processing unit 701. To the data processing unit 700, the pen image 171 is displayed on the sensor built-in liquid crystal panel 301 (see FIG. 15).

  In step S <b> 17, the tablet signal generation unit 13 instructs the auxiliary image display control unit 15 not to display the pen image 171. The auxiliary image display control unit 15 sends only the screen (grid image 172 or the like) displayed below the pen image 171 to the display data processing unit 701 and uses the “display panel” as the display / light sensor unit 300. Is sent to the data processing unit 700 to erase the pen image 171 from the sensor built-in liquid crystal panel 301.

  As described above, when the user makes the hand (object 64) hold the pen, the input device 100 shifts to the tablet mode. In the tablet mode, the input device 100 holds the tablet pen although it is not actually held. Input can be performed with the same feeling of operation. The input pen position is a position pointed by the tip of the pen when actually held by the user's hand. Thus, according to the present invention, the user can comfortably obtain the same operational feeling as a pen tablet without having a pen.

(Mouse mode)
FIG. 16 is a flowchart for explaining processing in the mouse mode of the input device 100. Hereinafter, each step of the processing in the mouse mode will be described with reference to the drawings. The mouse mode process starts from step S21.

  In step S <b> 21, the input device 100 displays a mouse image 174 as shown in FIG. 17 on the sensor built-in liquid crystal panel 301. In one embodiment, when the mouse mode is executed (when the input signal generation control unit 11 is instructed to generate an input signal to the mouse signal generation unit 14), the mouse signal generation unit 14 displays the auxiliary image. The control unit 15 is instructed to display the mouse image 174 at the mouse position on the sensor built-in liquid crystal panel 301. In response to this, the auxiliary image display control unit 15 calls the data of the mouse image 174 from the auxiliary image storage unit 22 and displays it at the mouse position on the sensor built-in liquid crystal panel 301. Note that this step is not executed when the mouse position is not yet determined.

  More specifically, when the mouse signal generation unit 14 receives a command to generate an input signal from the input signal generation control unit 11, first, the mouse image 174 is displayed on the auxiliary image display control unit 15 on the sensor built-in liquid crystal panel 301. Command to display at mouse position. Upon receiving the command, the auxiliary image display control unit 15 calls the data of the mouse image 174 from the auxiliary image storage unit 22 and combines the mouse image 174 with the mouse position on the screen to be displayed below the mouse image 174. Is sent to the display data processing unit 701, and the above-mentioned command with “display panel” as the display / light sensor unit 300 is sent to the data processing unit 700, thereby displaying a grid image on the sensor built-in liquid crystal panel 301. Thereafter, the process proceeds to step S22.

  In step S22, the input device 100 scans the object 64. In one embodiment, the mouse signal generation unit 14 acquires the entire image data from the sensor data processing unit 703 and stores it in the scan data holding unit 16 as scan data 160. Thereafter, the process proceeds to step S23. If the mode switching process has been performed immediately before and the previous scan data 160 has already been stored in the scan data holding unit 16, this step can be omitted.

  Specifically, first, the mouse signal generation unit 14 sends the above-mentioned command with “data type” as “whole image data” and “data acquisition timing” as “event” to the data processing unit 700. The mouse signal generation unit 14 causes the scan data holding unit 16 to store the entire image data obtained from the sensor data processing unit 703 in response to the command. When the mouse signal generation unit 14 determines the data acquisition timing and the change in the acquired data, the “data acquisition timing” of the command may be “all” or “sense”.

  In steps S23 to S28, the input device 100 detects the movement of the object 64 based on the scan data 160 acquired in step S22 and generates an input signal. In one embodiment, the mouse signal generation unit 14 first compares the scan data 160 acquired from the scan data holding unit 16 with the reference image 161 corresponding to the mouse mode called from the reference image storage unit 21.

  The description of the comparison between the scan data 160 and the reference image 161 is based on step S2. In addition, the reference image 161 in which the overlap ratio between the scan data 160 and the contact / contact area exceeds the predetermined value is related to the reference data 161. Data (horizontal movement distance X, vertical movement distance Y, rotation angle θ, magnification M) used for movement, rotation, enlargement / reduction, and the like applied to the image and information P indicating the reference image 161 are used as a reference. It is stored in the image processing data holding unit 17.

  The mouse signal generation unit 14 calculates the mouse position when the scan data 160 is similar to any of the reference images 161 corresponding to the mouse mode. That is, the mouse signal generation unit 14 sends from the reference image processing data holding unit 17 the horizontal movement distance X, the vertical movement distance Y, the rotation angle θ, and the magnification stored in the reference image processing data holding unit 17. Get rate M and information P. Next, the mouse signal generation unit 14 acquires the central coordinates 163 (PX, PY) associated with the reference image 161 indicated by the information P from the reference image storage unit 21. The center coordinates 163 indicate the center position of the object 64 (with the mouse) as shown in FIG.

  The mouse signal generation unit 14 calculates the mouse position based on these data. For example, the horizontal coordinate of the mouse position is (PXcos θ−PY sin θ) × M + X, and the vertical coordinate is (PX sin θ + PY cos θ) × M + Y.

  In step S23, the calculated mouse position is compared with the mouse position stored in the mouse state holding unit 18 in advance. If there is a change, an input signal indicating the amount of change (movement direction and amount of movement) is generated and output, the mouse state holding unit 18 stores the calculated mouse position, and the mouse display position is further changed. Change (step S29) and return to step S21. If there is no change in the mouse position, the process proceeds to step S24.

  Specifically, the mouse signal generation unit 14 first instructs the auxiliary image display control unit 15 to display the mouse image 174 at the calculated mouse position on the sensor built-in liquid crystal panel 301. To do. Upon receiving the command, the auxiliary image display control unit 15 calls the data of the mouse image 174 from the auxiliary image storage unit 22 and combines the mouse image 174 with the mouse position on the screen to be displayed below the mouse image 174. Is sent to the display data processing unit 701, and the above-mentioned command with “display panel” as the display / light sensor unit 300 is sent to the data processing unit 700, thereby displaying a grid image on the sensor built-in liquid crystal panel 301.

  In step S24, it is detected whether or not the object 64 has performed a left click operation. In one embodiment, a left click operation is detected when scan data 160 appears in a sequence as shown in FIG. That is, it is detected that only the forefinger is separated from the state where all five fingers are in contact, and that all the five fingers are in contact. For this detection, for example, a reference image 161 in which all five fingers are in contact and a reference image 161 in which only the index finger is separated are prepared, and both correspond to the mouse mode (mouse signal generating means). 13 is stored in advance in the reference image storage unit 21 as a reference image 161 (related to 13) and compared with the scan data 160 as described above. When the scan data 160 is similar to the reference image 161 in a state where only the index finger is separated, for example, “only the index finger is separated” is stored in the mouse state holding unit 18 as the mouse state. When the scan data 160 is similar to the reference image 161 in which all five fingers are in contact, the mouse state stored in the mouse state holding unit 18 is “only the index finger is released”. If so, an input signal indicating that there has been a left click is generated and output, and the mouse state holding unit 18 stores, for example, “all five fingers are in contact” as the mouse state. Is highlighted (step S30), and the process returns to step S21. If no left click is detected, the process proceeds to step S25.

  In step S25, it is detected whether or not the object 64 has performed a right click operation. The detection method is substantially the same as the detection of the left click in step S24. When the scan data 160 appears in the sequence shown in FIG. 19, the right click operation is detected. That is, it is detected that the state where all five fingers are in contact, the state where only the middle finger is separated, and the state where all five fingers are in contact is detected. For this detection, for example, a reference image 161 in which all five fingers are in contact and a reference image 161 in which only the middle finger is separated are prepared, and both correspond to the mouse mode (mouse signal generating means). 13 is stored in advance in the reference image storage unit 21 as a reference image 161 (related to 13) and compared with the scan data 160 as described above. When the scan data 160 is similar to the reference image 161 in a state where only the middle finger is separated, for example, “only the middle finger is separated” is stored in the mouse state holding unit 18 as the mouse state. When the scan data 160 is similar to the reference image 161 in a state where all five fingers are in contact, the mouse state stored in the mouse state holding unit 18 is “only the middle finger is released”. If so, an input signal indicating that there has been a right click is generated and output, and the mouse state holding unit 18 stores, for example, “all five fingers are in contact” as the mouse state, and further the mouse Is highlighted (step S31), and the process returns to step S21. If a right click is not detected, the process proceeds to step S26.

  In step S26, it is detected whether or not the object 64 has operated the wheel. The detection method is almost the same as the detection of the left click in step S24, and the wheel operation is detected when the scan data 160 as shown in FIG. 20 appears. That is, it is detected that only the index finger has shifted upward or downward from the state where all five fingers are in normal contact. For this detection, for example, a reference image 161 in which all five fingers are in normal contact and a reference image 161 in which only the index finger is displaced are prepared, and both correspond to the mouse mode (mouse signal). A reference image 161 (associated with the generation means 13) is stored in advance in the reference image storage unit 21 and compared with the scan data 160 as described above. When the scan data 160 is similar to the reference image 161 in which only the index finger is displaced, it is detected that the wheel has made one rotation, and an input indicating the amount of wheel operation (direction, number of displacement (rotation amount)). A signal is generated and output, and the mouse wheel is highlighted (step S32), and the process returns to step S21. If no wheel operation is detected, the process proceeds to step S27.

  In step S <b> 27, it is determined whether or not the scan data 160 includes an image of the object 64. If the scan data 160 does not include any image (when the scan / contact area is not included), the mouse is erased (step S33), and the process returns to step S21. If any image is included, the process proceeds to step S28.

  In step S28, it is determined whether or not the scan data 160 and the reference image 161 corresponding to the mouse mode are similar. If the scan data 160 and the reference image 161 corresponding to the mouse mode are not similar, the process proceeds to mode switching processing. If they are similar, the mouse is normally displayed (step S34).

  To describe the mouse left button, right button or wheel highlight display or normal display in detail, first, the mouse signal generation unit 14 first sends the mouse image 174 to the auxiliary image display control unit 15 on the sensor built-in liquid crystal panel 301. Instructs the mouse position to highlight the left button, right button, or wheel, or to display in the normal state. Upon receiving the command, the auxiliary image display control unit 15 calls the data of the mouse image 174 in which the left button, the right button, or the wheel is emphasized or in a normal state from the auxiliary image storage unit 22 and displays it under the mouse image 174. By sending an image obtained by synthesizing the mouse image 174 to the mouse position on the screen to be displayed to the display data processing unit 701, and sending the above command with the “display panel” as the display / light sensor unit 300 to the data processing unit 700. Then, a grid image is displayed on the sensor built-in liquid crystal panel 301.

  When the mouse display is erased, the mouse signal generation unit 14 instructs the auxiliary image display control unit 15 not to display the mouse image 174. The auxiliary image display control unit 15 sends only the screen displayed under the mouse image 174 to the display data processing unit 701, and sends the above-described command to the data processing unit 700 using the “display panel” as the display / light sensor unit 300. By sending it, the mouse image 174 is erased from the sensor built-in liquid crystal panel 301.

  Thus, the input device 100 shifts to the mouse mode when the user makes the hand (object 64) hold the mouse, and in the mouse mode, the input device 100 has the mouse although it does not actually have. Input can be performed with the same operation feeling. Thus, according to the present invention, the user can comfortably obtain the same operational feeling as the mouse without holding the mouse.

(Modification)
In the above description, the sensor built-in liquid crystal panel 301 is capable of detecting a nearby image and displaying an image. However, the sensor built-in liquid crystal panel 301 may simply be configured to detect and display a nearby image. Even with such a configuration, only the auxiliary image 170 is not displayed, and the above-described input in a plurality of modes and switching of the modes are realized.

  Further, the auxiliary image 170 is not limited to the pen image 171, the grid image 172, and the mouse image 174 as described above, but may be other images. For example, icons such as an icon 173 shown in FIG. 15 and an icon 175 shown in FIG.

  Further, as other modes of the tablet mode and the mouse mode described above, a touch panel mode, a screen keyboard mode, and the like may be provided. In that case, the touch panel signal generation unit and the screen keyboard signal generation unit generate input signals based on the movement of the object 64, respectively.

(Program and recording medium)
In the above, the configuration including the data processing unit 700 and the circuit control unit 600 has been described as the configuration of the input device 100. However, the configuration is not limited to this configuration. For example, the main control unit 800 may directly control the circuit control unit 600, and the main control unit 800 may generate whole image data / partial image data / coordinate data. The main control unit 800 may directly control the display / light sensor unit 300.

  Finally, the main control unit 800, the data processing unit 700, and the circuit control unit 600 of the input device 100 may be configured by hardware logic, or may be realized by software using a CPU as follows.

  That is, the input device 100 includes a CPU (central processing unit) that executes instructions of a control program that realizes each function, a ROM (read only memory) that stores the program, a RAM (random access memory) that expands the program, A storage device (recording medium) such as a memory for storing the program and various data is provided. An object of the present invention is to provide a recording medium in which a program code (execution format program, intermediate code program, source program) of a control program of the input device 100, which is software that realizes the above-described functions, is recorded in a computer-readable manner. This can also be achieved by supplying the input device 100 and reading and executing the program code recorded on the recording medium by the computer (or CPU or MPU).

  Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy (registered trademark) disks / hard disks, and disks including optical disks such as CD-ROM / MO / MD / DVD / CD-R. Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.

  The input device 100 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Also, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), IEEE802.11 radio, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

  Since the present invention can be used for input devices of various information devices, it can be used in the field of manufacturing information devices.

It is a block diagram which shows the detailed structure of the input device which concerns on one Embodiment of this invention. It is a schematic diagram which shows the cross section of the liquid crystal panel with a built-in sensor with which the input device which concerns on one Embodiment of this invention is provided. FIG. 3A is a schematic diagram illustrating a state in which a position touched by the user is detected by detecting a reflected image with a sensor-equipped liquid crystal panel included in the input device according to the embodiment of the present invention. FIG. 3B is a schematic diagram illustrating a state in which a position touched by the user is detected by detecting a shadow image with the sensor-embedded liquid crystal panel included in the input device according to the embodiment of the present invention. It is a block diagram which shows the principal part structure of the input device which concerns on one Embodiment of this invention. It is a schematic diagram which shows an example of the frame structure of the command used with the input device which concerns on one Embodiment of this invention. It is explanatory drawing which shows an example of the value which can be designated to each field contained in the command shown in FIG. 5, and its outline. FIG. 7A shows image data obtained as a result of scanning the entire liquid crystal panel with a sensor when an object is not placed on the liquid crystal panel with a sensor in the input device according to the embodiment of the present invention. It is. FIG. 7B shows image data obtained as a result of scanning when the user touches the sensor built-in liquid crystal panel with a finger in the input device according to the embodiment of the present invention. It is a block diagram which shows the structure of the liquid crystal panel with a sensor with which the input device which concerns on one Embodiment of this invention is provided, and the structure of its peripheral circuit. It is a flowchart explaining the mode switching process of the input device which concerns on one Embodiment of this invention. FIG. 10A is a diagram illustrating an example of an object placed on a planar member, and FIG. 10B is an image of the object detected by the planar member in FIG. FIG. FIG. 11A is a diagram showing an example of a reference image associated with mouse signal input means according to an embodiment of the present invention, and FIG. 11B is a reference image shown in FIG. It is a figure which shows the result of having rotated. Fig.12 (a) is a figure which shows an example of the target object set | placed on the planar member, FIG.12 (b) is an image of the said target object which the said planar member in Fig.10 (a) detected. FIG. FIG. 13A is a view showing an example of a reference image associated with the tablet signal input means according to the embodiment of the present invention, and FIG. 13B is a reference image shown in FIG. It is a figure which shows the result of having rotated. It is a flowchart explaining operation | movement of the tablet mode of the input device which concerns on one Embodiment of this invention. It is a schematic diagram which shows the outline of the tablet mode of the input device which concerns on one Embodiment of this invention. It is a flowchart explaining operation | movement of the mouse | mouth mode of the input device which concerns on one Embodiment of this invention. It is a schematic diagram which shows the outline of the mouse | mouth mode of the input device which concerns on one Embodiment of this invention. It is a schematic diagram which shows left click operation in the mouse | mouth mode of the input device which concerns on one Embodiment of this invention. It is a schematic diagram which shows right click operation in the mouse | mouth mode of the input device which concerns on one Embodiment of this invention. It is a schematic diagram which shows wheel operation in the mouse | mouth mode of the input device which concerns on one Embodiment of this invention.

Explanation of symbols

6 Photodiode 11 Input signal generation control unit (input signal generation control means)
12 Input signal generator (input signal generator)
13 Tablet signal generator (tablet signal generator)
14 Mouse signal generator (mouse signal generator)
15 Auxiliary image display control unit (auxiliary image display control means)
16 Scan data holding unit 17 Reference image processing data holding unit 18 Mouse state holding unit 21 Reference image storage unit 22 Auxiliary image storage unit 64 Object 100 Input device 160 Scan data (object image)
161 Reference image 162 Pointed coordinate 163 Center coordinate 170 Auxiliary image 171 Pen image 172 Grid image 173 Icon 174 Mouse image 175 Icon 300 Display / light sensor unit 301 Sensor built-in liquid crystal panel (planar member)
600 circuit control unit 601 display control unit 602 sensor control unit 603 backlight control unit 700 data processing unit 701 display data processing unit 703 sensor data processing unit 800 main control unit 901 storage unit

Claims (10)

A planar member for detecting a nearby image;
A plurality of input signal generating means for detecting the movement of the object based on the image of the object detected by the planar member and generating an input signal associated with the movement of the object;
Based on the image of the object detected by the planar member, any one input signal generation unit is selected from the plurality of input signal generation units, and the selected input signal generation unit generates the input signal. An input device comprising an input signal generation control means.   At least one of the plurality of input signal generation means detects that the object indicates a specific position on the planar member, and generates a tablet signal that generates the input signal indicating the specific position The input device according to claim 1, wherein the input device is a means.   At least one of the plurality of input signal generation means detects that the object has moved on the planar member, generates an input signal indicating the amount of the movement, and 3. The mouse signal generating means for generating an input signal indicating that a click operation has been performed by detecting that the part has approached or contacted after being separated from the planar member. Input device.   The mouse signal generation means detects that a part of the object has moved in a state of approaching or contacting the planar member, and generates an input signal indicating that there has been a wheel operation. The input device according to claim 3.   The input signal generation control unit compares the image of the object detected by the planar member with a reference image associated with each of the plurality of input signal generation units, thereby generating the plurality of input signal generation units. 5. The input device according to claim 1, wherein any one of the input signal generation means is selected from the means.   The input signal generation control means includes an image of the object detected by the planar member and an image obtained by performing one or more image processing selected from the group consisting of movement, rotation, enlargement, and reduction on the certain reference image. 6. The input device according to claim 5, wherein whether or not to select an input signal generation unit associated with the certain reference image is determined based on an overlap rate with the certain reference image. The planar member displays an image,
The auxiliary image display control means for displaying on the planar member an auxiliary image associated with the input signal generation means selected by the input signal generation control means. The input device according to claim 1.   The input device according to claim 7, wherein the auxiliary image is an auxiliary image selected from the group consisting of a pen image, a mouse image, and a grid image. A planar member that detects an image in the vicinity, and a plurality of members that detect the movement of the object based on the image of the object detected by the planar member and generate an input signal associated with the movement of the object An input device control method comprising:
Based on the image of the object detected by the planar member, the input signal generating unit is selected from the plurality of input signal generating units, and the input signal is input to the selected input signal generating unit. A control method for an input device, comprising an input signal generation control step to be generated.   A program for operating the input device according to any one of claims 1 to 8, wherein the program causes a computer to function as each of the means.

Images