JP2010244409A - Input device, and method of controlling the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an input device with high convenience and ease of operation for a user. <P>SOLUTION: The data display/sensor device 100 includes: a first display/optical sensor unit 300A and a second display/optical sensor unit 300B disposed on both surfaces thereof; a first input detection unit 810a which specifies a motion of input operation detected by the sensor unit 300A and generates a first input signal corresponding to the specified motion of input operation, which is associated with an operation in a first direction of an application which executes a predetermined operation; and a second input detection unit 810b which specifies a motion of input operation detected by the sensor unit 300B and generates a second input signal corresponding to the motion of input operation specified by the sensor unit 300B, which is associated with an operation in a second direction different from the first direction of the operation. Accordingly, the data display/sensor device 100 can provide an input device with superior convenience and ease of operation to the user. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an input device that inputs from two input surfaces and a method for controlling the input device.

  2. Description of the Related Art In recent years, touch panel type input devices are widely used in mobile devices (portable terminals) represented by mobile phones, PDAs and the like. As an example, there is known an input device that performs input according to the position of the touched finger when the input surface changes from a state where the finger is not in contact to a state where the finger is in contact ( For example, Patent Document 1).

  In addition, input devices that allow more convenient input depending on applications as well as accepting input simply by a touch panel type have been proposed. For example, in a portable terminal device, a user touches an icon on the display screen and then moves the finger in the direction in which the icon is to be moved, so that the icon is moved to a desired position on the display screen. Can be moved. Further, the folder can be opened by double-clicking the folder on the display screen with a finger (for example, Patent Document 2).

JP 2008-141519 A (published June 19, 2008) JP 11-39093 A (published on February 12, 1999)

  However, each of the conventional input devices is a method of inputting from one input surface (hereinafter referred to as a single-sided input method), and a method of inputting from two input surfaces arranged on the front and back surfaces (hereinafter referred to as a single-sided input method). In the first place, there was no input device of a double-sided input type.

  Therefore, the user cannot experience the various three-dimensional operations realized by the double-sided input type input device and the excellent convenience and operability of the conventional single-sided input type input device.

  Further, the conventional input device has the following problems. That is, in the one-side input type input device, the possibility that an unintended operation is performed without the user's knowledge cannot be excluded. Specifically, the user's finger may unintentionally touch the icon on the touch panel and drag the icon to somewhere else on the display screen. If the dragged destination is a trash folder, the icon is deleted without the user's knowledge. As described above, the conventional input device also has an inherent problem caused by inputting from one input surface.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an input device that is convenient and easy to operate for a user, and a method for controlling the input device.

  In order to solve the above problems, the input device according to the present invention is a plate-like input device in which the planar member for detecting the input operation is disposed on both the front and back surfaces, and the planar member on the front surface is detected. First input detection that identifies a movement of the input operation and generates a first input signal corresponding to the movement of the identified input operation that is associated with the movement in the first direction of the application that executes the predetermined movement A movement of the input operation detected by the means and the sheet member on the back surface, and the movement of the application in the second direction different from the first direction is associated with the movement of the application in the sheet member on the back surface And second input detection means for generating a second input signal corresponding to the identified movement of the input operation.

  The input device control method of the present invention is a plate-like input device control method in which the planar member for detecting an input operation is disposed on both the front and back surfaces, and the surface planar member is detected. First input detection that identifies a movement of the input operation and generates a first input signal corresponding to the movement of the identified input operation that is associated with the movement in the first direction of the application that executes the predetermined movement Identifying the movement of the input operation detected by the step and the sheet member on the back surface, and corresponding to the movement of the application in the second direction different from the first direction, in the sheet member on the back surface And a second input detection step of generating a second input signal corresponding to the identified input operation movement.

  Accordingly, in the input device and the input device control method according to the present invention, the first input signal associated with the operation in the first direction of the application that executes the predetermined operation is detected by the first input detection unit. Second input signals associated with movements in a second direction different from the first direction are respectively generated by the second input detection means. Then, according to the first and second input signals, the application performs operations in the first and second directions separately or in combination.

  Accordingly, the surface members on both the front and back surfaces detect various input operation movements, thereby realizing various operations that could not be realized with the conventional input device. That is, various operations can be realized by fusing the operation of the application in the first direction by the input operation from the front surface and the operation of the application in the second direction by the input operation from the back surface. Moreover, if various operations are assigned as the operations in the first and second directions, more various operations can be realized. In addition, since the input device is plate-shaped, the user can easily experience the various operations described above by sandwiching the front and back surfaces with fingers.

  Thus, the input device and the control method for the input device according to the present invention can provide the user with excellent operability and convenience.

  Note that there may be one or a plurality of operation execution devices, the operation execution devices may be integrated with the input device, or may be present at a position physically separated from the input device. .

  Further, in the input device according to the present invention, an X axis and a Y axis are set for the surface planar member, and the first input detecting means moves the input operation detected by the surface planar member. Is specified as a movement amount and movement direction on the XY coordinates, a first input signal corresponding to the movement amount and movement direction on the specified XY coordinates is generated, and the second input detection means The movement of the input operation detected by the member is specified as the movement amount and movement direction on the Z axis that defines three dimensions together with the X axis and the Y axis, and corresponds to the specified movement amount and movement direction on the Z axis. The second input signal may be generated.

  As described above, the input device according to the present invention realizes various operations that cannot be realized by the conventional input device by detecting the movement of the input operation by the planar members on both the front and back surfaces. In addition, the first input signal is a signal corresponding to the movement of the input operation detected by the surface planar member specified by the first input detection unit as the movement amount and movement direction on the XY coordinates. The second input signal is a signal corresponding to the movement of the input operation detected by the planar member on the back surface, which is specified as the movement amount and movement direction on the Z-axis by the second input detection means.

  Accordingly, the first and second input signals that realize various three-dimensional operations that cannot be realized by the conventional input device are generated by detecting the movements of various input operations by the planar members on both the front and back surfaces. Based on the first and second input signals, the application executes operations in the first and second directions separately or in combination.

  As described above, by detecting the movements of various input operations by the planar members on both the front and back surfaces, various three-dimensional operations that cannot be realized by a conventional input device are realized. Therefore, an input device having excellent convenience and operability is provided to the user. Further, the user can imagine a three-dimensional motion realized by his / her own operation, and can detect the movement of a predetermined input operation that causes the three-dimensional motion to be performed on the planar members on both the front and back surfaces. You can experience excellent operation.

  Furthermore, in the input device according to the present invention, in the configuration described above, the first input detection unit executes a three-dimensional operation defined by the movement amount and movement direction on the specified XY coordinates and the XYZ axes. A movement amount on the specified XY coordinates based on first command correspondence information associated with a command for causing the motion execution device in which the application operates to execute a motion in the first direction defined by the XY axes. And a command associated with the moving direction is determined, the first input signal indicating the command is transmitted to the operation executing device, and the second input detecting means is configured to determine the movement amount and the movement on the identified Z axis. Based on the second command correspondence information in which the direction and the command for causing the motion execution device to execute the motion in the second direction defined by the Z-axis direction are correlated, Determining an instruction kinematic direction associated, may the second input signal indicative of the instruction as a configuration for transmitting to the operation execution unit.

  According to the configuration, in the input device according to the present invention, the first direction is a direction defined by the XY axis in the application that executes a three-dimensional operation defined by the XYZ axes. The first input is based on the first command correspondence information in which the movement amount and movement direction on the XY coordinates specified by the first input detection means are associated with the command for executing the operation in the first direction. The detection means determines a command in which the movement amount and the movement direction on the specified XY coordinates are associated with each other.

  Therefore, by reading the first correspondence command information from the recording means provided inside or outside, the first input detection means can easily issue a command in which the movement amount and the movement direction on the specified XY coordinates are associated with each other. Can be determined. Then, the first input detection means transmits the first input signal indicating the command to the operation execution device.

  The second direction is a direction defined by the Z-axis direction in the application. Then, based on the second command correspondence information in which the movement amount and movement direction on the Z-axis specified by the second input detection means and the command for executing the operation in the second direction are associated with each other, the second input The detection means determines a command associated with the identified movement amount and movement direction on the Z axis.

  Therefore, by reading the second correspondence command information from the recording means, the second input detection means can easily determine a command associated with the identified movement amount and movement direction on the Z axis. . Then, the second input detection means transmits the second input signal indicating the command to the operation execution device.

  As described above, the first and second input detecting means can easily issue a command in which the movement amount and the movement direction on the XY coordinates and the Z axis are associated with each other based on the first and second command correspondence information. Can be determined. Furthermore, if the first and second instruction correspondence information is associated with each application and recorded in the recording unit, various three-dimensional operations can be experienced quickly and reliably by the user for various applications. be able to.

  Furthermore, the input device according to the present invention further comprises image display control means for displaying an image on at least one of the surface members on both the front and back surfaces in the above-described configuration, and corresponds to the first command. The information and the second command correspondence information may be associated with the image.

  According to the above configuration, in the input device according to the present invention, the image display control means displays an image on at least one of the planar members. The image is associated with the first and second instruction correspondence information.

  That is, when the image display control means displays an image, the operation target is clearly indicated to the user. Then, since the first and second command correspondence information is associated with the image, the user visually recognizes the image and shows the movement of the input operation that brings about a desired operation on the image. The planar member may be detected.

  Therefore, since a desired operation can be executed after visually recognizing the image, an input device that is easy for the user to operate intuitively and is highly convenient is realized.

  In order to solve the above problems, the input device according to the present invention is a plate-like input device in which the planar member for detecting the input operation is disposed on both the front and back surfaces, and the planar member on the front surface is detected. First input detecting means for identifying a movement of the input operation and generating a first input signal corresponding to the identified movement of the input operation, which is associated with the first movement of the application that executes the predetermined movement; In the first operation of the application, the movement of the input operation detected by the sheet member on the back surface is specified, or the movement of the input operation from detection to non-detection on the sheet member on the back surface is specified. Second input detection means for generating a second input signal corresponding to the movement of the specified input operation in the planar member on the back surface associated with the accompanying second action. .

  Thereby, in the input device according to the present invention, the first input signal associated with the first operation of the application that executes the predetermined operation is detected by the first input detection means by the second input associated with the first operation. Second input signals associated with the operations are respectively generated by the second input detection means. In the second input signal, the movement of the input operation detected by the sheet member on the back surface or the movement of the input operation from detection to non-detection on the sheet member on the back surface is specified by the second input detection means. Is generated.

  Thus, for example, it is assumed that the sheet member on the back side is assigned to execute operations such as “confirm” and “conversion” by detecting the movement of the input operation. Alternatively, when the movement of the input operation detected by the sheet member on the back surface changes to a non-detection state, the application operation that has been executed so far is assigned to be “cancelled (invalidated)”. Suppose. At this time, when the user wants to execute operations such as “confirm”, “conversion”, and “cancel (invalid)”, the back surface member detects the movement of the input operation, or the back surface member The movement of the input operation from detection to non-detection may be given to. Thereby, each said operation | movement of the application which a user desires can be performed.

  That is, if the movement of the input operation with respect to the planar member on the back surface is assigned as described above, the user can easily execute a desired application operation. In addition, since the input device has a plate shape, the user can perform the above-described operation so that both sides of the input device are held between fingers.

  Thus, the input device according to the present invention can provide the user with excellent operability and convenience.

  Note that there may be one or a plurality of operation execution devices, the operation execution devices may be integrated with the input device, or may be present at a position physically separated from the input device. .

  As described above, the input device according to the present invention specifies the movement of the input operation detected by the surface planar member, and is associated with the operation in the first direction of the application that executes the predetermined operation. The first input detection means for generating a first input signal corresponding to the identified input operation movement, the input operation movement detected by the sheet member on the back surface, and the first direction of the application Second input detection means for generating a second input signal corresponding to the movement of the specified input operation in the planar member on the back surface, which is associated with a different movement in the second direction.

  As described above, the input device according to the present invention specifies the movement of the input operation detected by the surface planar member, and is associated with the first operation of the application that executes a predetermined operation. The first input detection means for generating the first input signal corresponding to the input operation movement and the movement of the input operation detected by the back surface member are identified or not detected from the detection of the back surface member. The movement of the input operation for detection is identified, and the movement of the identified input operation in the planar member on the back surface associated with the second movement accompanying the first movement of the application is identified. Second input detection means for generating a second input signal.

  In the input device control method of the present invention, the movement of the input operation detected by the surface planar member is identified and associated with the operation in the first direction of the application that executes the predetermined operation. A first input detection step for generating a first input signal corresponding to the identified input operation movement is different from the first direction of the application by identifying the input operation movement detected by the planar member on the back surface. And a second input detecting step of generating a second input signal corresponding to the movement of the specified input operation in the planar member on the back surface associated with the movement in the second direction.

  Therefore, it is possible to provide the user with an input device that is input from two input surfaces and has excellent operability and convenience, and an input device control method.

  This embodiment will be described below with reference to FIGS.

  Generally speaking, the data display / sensor device 100 (input device) according to the present embodiment specifies the movement of the input operation detected by the first display / light sensor unit 300A (surface planar member). First input detection unit 810a (first input detection means) that generates a first input signal corresponding to the movement of the specified input operation associated with the movement of the application that executes the predetermined movement in the first direction ) And the movement of the input operation detected by the second display / light sensor unit 300B (the sheet member on the back surface) are identified and associated with the movement in the second direction different from the first direction of the application. And a second input detection unit 810b (second input detection means) that generates a second input signal corresponding to the movement of the specified input operation in the second display / light sensor unit 300B.

  First, an outline of the sensor built-in liquid crystal panel 301 included in the data display / sensor device 100 will be described. However, the data display / sensor device 100 does not necessarily need to use the sensor built-in liquid crystal panel 301, and may be configured to use another touch panel (for example, a resistive film type).

(Outline of LCD panel with built-in sensor)
The sensor built-in liquid crystal panel 301 included in the data display / sensor device 100 is a liquid crystal panel capable of detecting an image of an object in addition to displaying data. Here, the image detection of the object is, for example, detection of a position pointed (touched) by the user with a finger or a pen, or reading (scanning) of an image of a printed material. The device used for display is not limited to a liquid crystal panel, and may be an organic EL (Electro Luminescence) panel or the like.

  The structure of the sensor built-in liquid crystal panel 301 will be described with reference to FIG. FIG. 2 is a diagram schematically showing a cross section of the sensor built-in liquid crystal panel 301. The sensor built-in liquid crystal panel 301 described here is an example, and any structure can be used as long as the display surface and the reading surface are shared.

  As shown in the figure, the sensor built-in liquid crystal panel 301 includes an active matrix substrate 51A disposed on the back surface side and a counter substrate 51B disposed on the front surface side, and has a structure in which a liquid crystal layer 52 is sandwiched between these substrates. is doing. The active matrix substrate 51A is provided with a pixel electrode 56, a data signal line 57, an optical sensor circuit 32 (not shown), an alignment film 58, a polarizing plate 59, and the like. The counter substrate 51B is provided with color filters 53r (red), 53g (green), 53b (blue), a light shielding film 54, a counter electrode 55, an alignment film 58, a polarizing plate 59, and the like. In addition, a backlight 307 is provided on the back surface of the sensor built-in liquid crystal panel 301.

  The photodiode 6 included in the photosensor circuit 32 is provided in the vicinity of the pixel electrode 56 provided with the blue color filter 53b, but is not limited to this configuration. It may be provided in the vicinity of the pixel electrode 56 provided with the red color filter 53r, or may be provided in the vicinity of the pixel electrode 56 provided with the green color filter 53g.

  Next, with reference to FIGS. 3A and 3B, two types of methods for detecting the position where the user touches the sensor built-in liquid crystal panel 301 with a finger or a pen will be described.

  FIG. 3A is a schematic diagram illustrating a state in which a position touched by the user is detected by detecting a reflected image. When the light 63 is emitted from the backlight 307, the optical sensor circuit 32 including the photodiode 6 detects the light 63 reflected by the object 64 such as a finger. Thereby, the reflected image of the target object 64 can be detected. Thus, the sensor built-in liquid crystal panel 301 can detect the touched position by detecting the reflected image.

  FIG. 3B is a schematic diagram illustrating a state in which a position touched by the user is detected by detecting a shadow image. As shown in FIG. 3B, the optical sensor circuit 32 including the photodiode 6 detects external light 61 transmitted through the counter substrate 51B and the like. However, when there is an object 62 such as a pen, the incident of the external light 61 is hindered, so that the amount of light detected by the optical sensor circuit 32 is reduced. Thereby, a shadow image of the object 62 can be detected. Thus, the sensor built-in liquid crystal panel 301 can also detect a touched position by detecting a shadow image.

  As described above, the photodiode 6 may detect reflected light (shadow image) of the light emitted from the backlight 307 or may detect a shadow image caused by external light. Further, the two types of detection methods may be used in combination to detect both a shadow image and a reflected image at the same time.

(Data display / sensor configuration)
Next, the configuration of the main part of the data display / sensor device 100 will be described with reference to FIG. FIG. 4 is a block diagram showing a main configuration of the data display / sensor device 100. As shown in FIG. As illustrated, the data display / sensor device 100 includes one or more display / light sensor units 300, a circuit control unit 600, a data processing unit 700, a main control unit 800, a storage unit 901, a primary storage unit 902, and an operation unit. 903, an external communication unit 907, an audio output unit 908, and an audio input unit 909. Here, the data display / sensor device 100 will be described as including two display / light sensor units 300 (first display / light sensor unit 300A and second display / light sensor unit 300B). When the first display / light sensor unit 300A and the second display / light sensor unit 300B are not distinguished, they are referred to as the display / light sensor unit 300.

  The display / light sensor unit 300 is a so-called liquid crystal display device with a built-in light sensor. The display / light sensor unit 300 includes a sensor built-in liquid crystal panel 301, a backlight 307, and a peripheral circuit 309 for driving them.

  The sensor built-in liquid crystal panel 301 includes a plurality of pixel circuits 31 and photosensor circuits 32 arranged in a matrix. The detailed configuration of the sensor built-in liquid crystal panel 301 will be described later.

  The peripheral circuit 309 includes a liquid crystal panel drive circuit 304, an optical sensor drive circuit 305, a signal conversion circuit 306, and a backlight drive circuit 308.

  The liquid crystal panel driving circuit 304 outputs a control signal (G) and a data signal (S) in accordance with the timing control signal (TC1) and the data signal (D) from the display control unit 601 of the circuit control unit 600, and the pixel circuit 31. It is a circuit which drives. Details of the driving method of the pixel circuit 31 will be described later.

  The optical sensor driving circuit 305 is a circuit that drives the optical sensor circuit 32 by applying a voltage to the signal line (R) in accordance with a timing control signal (TC2) from the sensor control unit 602 of the circuit control unit 600. Details of the driving method of the optical sensor circuit 32 will be described later.

  The signal conversion circuit 306 is a circuit that converts the sensor output signal (SS) output from the optical sensor circuit 32 into a digital signal (DS) and transmits the converted signal to the sensor control unit 602.

  The backlight 307 includes a plurality of white LEDs (Light Emitting Diodes) and is disposed on the back surface of the sensor built-in liquid crystal panel 301. When a power supply voltage is applied from the backlight drive circuit 308, the backlight 307 is turned on and irradiates the sensor built-in liquid crystal panel 301 with light. Note that the backlight 307 is not limited to white LEDs, and may include LEDs of other colors. The backlight 307 may include, for example, a cold cathode fluorescent lamp (CCFL) instead of the LED.

  The backlight driving circuit 308 applies a power supply voltage to the backlight 307 when the control signal (BK) from the backlight control unit 603 of the circuit control unit 600 is at a high level, and conversely, the backlight control unit 603. When the control signal from is at a low level, no power supply voltage is applied to the backlight 307.

  Next, the circuit control unit 600 will be described. The circuit control unit 600 has a function as a device driver that controls the peripheral circuit 309 of the display / light sensor unit 300. The circuit control unit 600 includes a display control unit 601, a sensor control unit 602, a backlight control unit 603, and a display data storage unit 604.

  The display control unit 601 receives display data from the display data processing unit 701 of the data processing unit 700, and performs timing control on the liquid crystal panel driving circuit 304 of the display / light sensor unit 300 in accordance with an instruction from the display data processing unit 701. A signal (TC1) and a data signal (D) are transmitted, and the received display data is displayed on the sensor built-in liquid crystal panel 301.

  The display control unit 601 temporarily stores the display data received from the display data processing unit 701 in the display data storage unit 604. Then, a data signal (D) is generated based on the primary stored display data. The display data storage unit 604 is, for example, a video random access memory (VRAM).

  The sensor control unit 602 transmits a timing control signal (TC2) to the optical sensor driving circuit 305 of the display / optical sensor unit 300 in accordance with an instruction from the sensor data processing unit 703 of the data processing unit 700, and the sensor built-in liquid crystal panel 301. Run the scan with.

  In addition, the sensor control unit 602 receives a digital signal (DS) from the signal conversion circuit 306. Then, image data is generated based on the digital signal (DS) corresponding to the sensor output signal (SS) output from all the optical sensor circuits 32 included in the sensor built-in liquid crystal panel 301. That is, the image data read in the entire reading area of the sensor built-in liquid crystal panel 301 is generated. Then, the generated image data is transmitted to the sensor data processing unit 703.

  The backlight control unit 603 transmits a control signal (BK) to the backlight drive circuit 308 of the display / light sensor unit 300 in accordance with instructions from the display data processing unit 701 and the sensor data processing unit 703 to drive the backlight 307. Let

  When the data display / sensor device 100 includes a plurality of display / light sensor units 300, the display control unit 601 determines which display / light sensor unit 300 displays the display data from the data processing unit 700. When an instruction is received, the liquid crystal panel drive circuit 304 of the display / light sensor unit 300 is controlled according to the instruction. When the sensor control unit 602 receives an instruction from the data processing unit 700 as to which display / light sensor unit 300 is to scan the object, The sensor drive circuit 305 is controlled and a digital signal (DS) is received from the signal conversion circuit 306 of the display / light sensor unit 300 according to the instruction.

  Next, the data processing unit 700 will be described. The data processing unit 700 has a function as middleware that gives an instruction to the circuit control unit 600 based on a “command” received from the main control unit 800. Details of the command will be described later.

  The data processing unit 700 includes a display data processing unit 701 and a sensor data processing unit 703. When the data processing unit 700 receives a command from the main control unit 800, at least one of the display data processing unit 701 and the sensor data processing unit 703 depends on the value of each field (described later) included in the received command. One works.

  The display data processing unit 701 receives display data from the main control unit 800, and gives instructions to the display control unit 601 and the backlight control unit 603 according to the command received by the data processing unit 700, and displays the received display data. The image is displayed on the sensor built-in liquid crystal panel 301. The operation of the display data processing unit 701 according to the command will be described later.

  The sensor data processing unit 703 gives an instruction to the sensor control unit 602 and the backlight control unit 603 according to the command received by the data processing unit 700.

  The sensor data processing unit 703 receives image data from the sensor control unit 602 and stores the image data in the image data buffer 704 as it is. Then, in accordance with the command received by the data processing unit 700, the sensor data processing unit 703 performs “whole image data”, “partial image data (partial image coordinate data) based on the image data stored in the image data buffer 704. At least one of “including coordinate data” and “coordinate data” is transmitted to the main control unit 800. The whole image data, partial image data, and coordinate data will be described later. The operation of the sensor data processing unit 703 according to the command will be described later.

  Next, the main control unit 800 executes an application program. The main control unit 800 reads the program stored in the storage unit 901 into a primary storage unit 902 configured by, for example, a RAM (Random Access Memory) and executes the program.

  An application program executed by the main control unit 800 causes the data processing unit 700 to display display data on the sensor built-in liquid crystal panel 301 or to scan an object on the sensor built-in liquid crystal panel 301. Send commands and display data. When “data type” is designated as a command, at least one of whole image data, partial image data, and coordinate data is received from the data processing unit 700 as a response to the command.

  The circuit control unit 600, the data processing unit 700, and the main control unit 800 can be configured by a CPU (Central Processing Unit), a memory, and the like, respectively. The data processing unit 700 may be configured by a circuit such as an ASIC (application specific integrate circuit).

  Next, the storage unit 901 stores programs and data executed by the main control unit 800 as shown in the figure. The program executed by the main control unit 800 may be separated into an application-specific program and a general-purpose program that can be shared by each application.

  Next, the operation unit 903 receives an input operation of the user of the data display / sensor device 100. The operation unit 903 includes input devices such as a switch, a remote controller, a mouse, and a keyboard, for example. Then, the operation unit 903 generates a control signal corresponding to the user's input operation of the data display / sensor device 100, and transmits the generated control signal to the main control unit 800.

  As an example of the switch, a hardware switch such as a power switch 905 that switches power on and off and a user switch 906 to which a predetermined function is assigned in advance is assumed.

  In addition, the data display / sensor device 100 includes an external communication unit 907 for communicating with an external device by wireless / wired communication, an audio output unit 908 such as a speaker for outputting audio, and a microphone for inputting an audio signal. A voice input unit 909 such as the above may be provided as appropriate.

(Command details)
Next, details of commands transmitted from the main control unit 800 to the data processing unit 700 will be described with reference to FIGS. 5 and 6. FIG. 5 is a diagram schematically illustrating an example of a command frame structure. FIG. 6 is a diagram for explaining an example of values that can be specified for each field included in the command and an outline thereof.

  As shown in FIG. 5, the commands are “header”, “data acquisition timing”, “data type”, “scan method”, “scan image gradation”, “scan resolution”, “scan panel”, “display panel”. "And" Reserve "fields. In each field, for example, values shown in FIG. 6 can be designated.

  The “header” field is a field indicating the start of a frame. As long as it is possible to identify the “header” field, the value of the “header” field may be any value.

  Next, the “data acquisition timing” field is a field for designating a timing at which data should be transmitted to the main control unit 800. In the “data acquisition timing” field, for example, values “00” (sense), “01” (event), and “10” (all) can be specified.

  Here, “sense” designates that the latest data is transmitted immediately. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “sense”, the latest data specified in the “data type” field is immediately updated to the main control unit 800. Send to.

  The “event” designates transmission at a timing when a change occurs in the image data received from the sensor control unit 602. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “event”, the image that receives the data specified in the “data type” field from the sensor control unit 602. The data is transmitted to the main control unit 800 at a timing when a change larger than a predetermined threshold occurs.

  “All” designates data transmission at a predetermined cycle. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “all”, the data designated in the “data type” field is transferred to the main control unit 800 at a predetermined cycle. Send to. The predetermined period coincides with the period in which the optical sensor circuit 32 performs scanning.

  Next, the “data type” field is a field for designating the type of data acquired from the sensor data processing unit 703. In the “data type” field, for example, values of “001” (coordinates), “010” (partial image), and “100” (entire image) can be specified. Furthermore, by adding these values, “coordinates” and “partial image” / “whole image” can be specified simultaneously. For example, when “coordinate” and “partial image” are specified at the same time, “011” can be specified.

  When the sensor data processing unit 703 receives a command whose value of the “data type” field is “whole image”, the sensor data processing unit 703 transmits the image data itself stored in the image data buffer 704 to the main control unit 800. The image data itself stored in the image data buffer 704 is referred to as “whole image data”.

  In addition, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 selects a portion where a change larger than a predetermined threshold has occurred from the image data received from the sensor control unit 602. A region to be included is extracted, and image data of the extracted region is transmitted to the main control unit 800. Here, the image data is referred to as “partial image data”. When a plurality of partial image data are extracted, the sensor data processing unit 703 transmits the extracted partial image data to the main control unit 800.

  Further, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 detects representative coordinates in the partial image data and indicates the position of the representative coordinates in the partial image data. The coordinate data is transmitted to the main control unit 800. The representative coordinates include, for example, the coordinates of the center of the partial image data, the coordinates of the center of gravity of the partial image data, and the like.

  Next, when receiving a command whose value of the “data type” field is “coordinate”, the sensor data processing unit 703 transmits coordinate data indicating the position of the representative coordinate in the entire image data to the main control unit 800. When a plurality of partial image data are extracted, the sensor data processing unit 703 detects representative coordinates in the entire image data of the extracted partial image data, and the coordinate data indicating the representative coordinates is detected. Each is transmitted to the main control unit 800 (multi-point detection).

  Specific examples of the whole image data, the partial image data, and the coordinate data will be described later with reference to schematic diagrams.

  Next, the “scan method” field is a field for designating whether or not the backlight 307 is turned on at the time of executing the scan. In the “scan method” field, for example, values of “00” (reflection), “01” (transmission), and “10” (reflection / transmission) can be designated.

  “Reflection” designates that scanning is performed with the backlight 307 turned on. Therefore, when the sensor data processing unit 703 receives a command whose “scan method” field value is “reflection”, the sensor data processing unit 703 performs sensor control so that the optical sensor driving circuit 305 and the backlight driving circuit 308 operate in synchronization. An instruction is given to the unit 602 and the backlight control unit 603.

  “Transmission” specifies that scanning is performed with the backlight 307 turned off. Therefore, when the sensor data processing unit 703 receives a command whose “scan method” field value is “transparent”, the sensor control unit 602 operates the optical sensor driving circuit 305 and does not operate the backlight driving circuit 308. Instructions to the backlight control unit 603. Note that “reflection / transmission” specifies that scanning is performed using both “reflection” and “transmission”.

  Next, the “scanned image gradation” field is a field for designating gradations of the partial image data and the entire image data. In the “scanned image gradation” field, for example, values of “00” (binary) and “01” (multivalue) can be designated.

  When the sensor data processing unit 703 receives a command whose “scan image gradation” field value is “binary”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 as monochrome data. .

  When the sensor data processing unit 703 receives a command whose “scanned image gradation” field value is “multivalued”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 as multitone data. To do.

  Next, the “scan resolution” field is a field for designating the resolution of the partial image data and the entire image data. In the “scan resolution” field, for example, values of “0” (high) and “1” (low) can be designated.

  Here, “high” designates a high resolution. Therefore, when the sensor data processing unit 703 receives a command whose “scan resolution” field value is “high”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 with high resolution. For example, it is desirable to designate “high” for image data (image data such as a fingerprint) to be subjected to image processing such as image recognition.

  “Low” designates a low resolution. Therefore, when the sensor data processing unit 703 receives a command whose “scan resolution” field value is “low”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 at a low resolution. For example, it is desirable to designate “low” for image data (such as touched finger or hand image data) that only needs to be recognized.

  Next, the “scan panel” field is a field for designating which display / light sensor unit 300 is to scan the object. In the “scan panel” field, for example, values “001” (first display / light sensor unit 300A) and “010” (second display / light sensor unit 300B) can be designated. By adding these values, a plurality of display / light sensor units 300 can be specified at the same time. For example, when “first display / light sensor unit 300A” and “second display / light sensor unit 300B” are specified at the same time, “011” can be specified.

  When the sensor data processing unit 703 receives a command whose “scan panel” field value is “first display / photosensor unit 300A”, the sensor data processing unit 703 and the photosensor drive circuit 305 of the first display / photosensor unit 300A and An instruction is given to the sensor control unit 602 and the backlight control unit 603 so as to control the backlight drive circuit 308.

  Next, the “display panel” field is a field for designating which display / light sensor unit 300 displays the display data. In the “display panel” field, for example, values of “001” (first display / light sensor unit 300A) and “010” (second display / light sensor unit 300B) can be designated. By adding these values, a plurality of display / light sensor units 300 can be specified at the same time. For example, when “first display / light sensor unit 300A” and “second display / light sensor unit 300B” are specified at the same time, “011” can be specified.

  Here, for example, when the display data processing unit 701 receives a command whose value of the “display panel” field is “first display / light sensor unit 300A”, the display data processing unit 701 displays the display data on the first display / light sensor unit 300A. Therefore, an instruction is given to the display control unit 601 and the backlight control unit 603 to control the liquid crystal panel driving circuit 304 and the backlight driving circuit 308 of the first display / light sensor unit 300A.

  Next, the “reserved” field is a field that is appropriately specified when it is necessary to further specify information other than information that can be specified in the above-described fields.

  Note that an application executed by the main control unit 800 does not need to use all the above-described fields when transmitting a command, and an invalid value (such as a NULL value) may be set for a field that is not used. .

  When the user wants to acquire coordinate data of a position touched with a finger or pen, a command specifying “coordinate” in the “data type” field is transmitted to the data processing unit 700. Therefore, it is desirable to specify “all” in the “data acquisition timing” field of the command to acquire coordinate data. Further, since it is only necessary to acquire coordinate data of the touched position, the scanning accuracy may not be high. Therefore, “low” may be specified as the value of the “scan resolution” field of the command.

  Further, when “coordinate” is specified in the “data type” field of the command, for example, when the user touches the sensor built-in liquid crystal panel 301 with a plurality of fingers or pens at the same time, the coordinate data of the touched position is used. Can be acquired (multi-point detection).

  When acquiring image data of an object such as a document, a command specifying “whole image” in the “data type” field is transmitted to the data processing unit 700. Since it is common to perform a scan in a stationary state, it is not necessary to periodically perform the scan. Therefore, in this case, it is desirable to designate “sense” or “event” in the “data acquisition timing” field. When scanning an object such as a document, it is desirable that the scanning accuracy is high so that the user can easily read the characters. Therefore, it is desirable to designate “high” in the “scan resolution” field.

(Whole image data / Partial image data / Coordinate data)
Next, the whole image data, the partial image data, and the coordinate data will be described with reference to FIG. The image data shown in FIG. 7A is image data obtained as a result of scanning the entire sensor built-in liquid crystal panel 301 when the object is not placed on the sensor built-in liquid crystal panel 301. The image data shown in FIG. 7B is image data obtained as a result of scanning the entire sensor-equipped liquid crystal panel 301 when the user touches the sensor-equipped liquid crystal panel 301 with a finger.

  When the user touches the sensor built-in liquid crystal panel 301 with a finger, the amount of light received by the photosensor circuit 32 in the vicinity of the touch changes, so that the voltage output from the photosensor circuit 32 changes, and as a result, In the image data generated by the sensor control unit 602, the brightness of the pixel value of the portion touched by the user changes.

  In the image data shown in FIG. 7B, the brightness of the pixel value of the portion corresponding to the user's finger is higher than that in the image data shown in FIG. In the image data shown in FIG. 7B, the smallest rectangular area (area PP) that includes all pixel values whose lightness changes more than a predetermined threshold is “partial image data”.

  The image data indicated by the area AP is “whole image data”.

  Also, the coordinate data in the whole image data (area AP) of the representative coordinates Z of the partial image data (area PP) is (Xa, Ya), and the coordinate data in the partial image data (area PP) is (Xp, Yp). It is.

(Configuration of sensor built-in liquid crystal panel)
Next, the configuration of the sensor built-in liquid crystal panel 301 and the configuration of the peripheral circuit 309 of the sensor built-in liquid crystal panel 301 will be described with reference to FIG. FIG. 8 is a block diagram showing the main part of the display / light sensor unit 300, particularly the configuration of the sensor built-in liquid crystal panel 301 and the configuration of the peripheral circuit 309.

  The sensor built-in liquid crystal panel 301 includes a pixel circuit 31 for setting light transmittance (brightness) and an optical sensor circuit 32 that outputs a voltage corresponding to the intensity of light received by the sensor. Note that the pixel circuit 31 is a generic term for the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b corresponding to the red, green, and blue color filters, respectively.

  The pixel circuits 31 are arranged on the sensor built-in liquid crystal panel 301 in the column direction (vertical direction) m and in the row direction (horizontal direction) n. A set of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b is continuously arranged in the row direction (lateral direction). This set forms one pixel.

  In order to set the light transmittance of the pixel circuit 31, first, the high level voltage (TFT 33 is turned on) to the scanning signal line Gi connected to the gate terminal of the TFT (Thin Film Transistor) 33 included in the pixel circuit 31. Voltage). Thereafter, a predetermined voltage is applied to the data signal line SRj connected to the source terminal of the TFT 33 of the R pixel circuit 31r. Similarly, the light transmittance is also set for the G pixel circuit 31g and the B pixel circuit 31b. Then, by setting these light transmittances, an image is displayed on the sensor built-in liquid crystal panel 301.

  Next, the photosensor circuit 32 is arranged for each pixel. One pixel may be arranged in the vicinity of each of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b.

  In order for the optical sensor circuit 32 to output a voltage according to the light intensity, first, the sensor readout line RWi connected to one electrode of the capacitor 35 and the anode terminal of the photodiode 6 are connected. A predetermined voltage is applied to the sensor reset line RSi. In this state, when light is incident on the photodiode 6, a current corresponding to the amount of incident light flows through the photodiode 6. And according to the said electric current, the voltage of the connection point (henceforth connection node V) of the other electrode of the capacitor | condenser 35 and the cathode terminal of the photodiode 6 falls. When the power supply voltage VDD is applied to the voltage application line SDj connected to the drain terminal of the sensor preamplifier 37, the voltage at the connection node V is amplified and output from the source terminal of the sensor preamplifier 37 to the sensing data output line SPj. Based on the output voltage, the amount of light received by the optical sensor circuit 32 can be calculated.

  Next, the liquid crystal panel drive circuit 304, the optical sensor drive circuit 305, and the sensor output amplifier 44, which are peripheral circuits of the sensor built-in liquid crystal panel 301, will be described.

  The liquid crystal panel drive circuit 304 is a circuit for driving the pixel circuit 31, and includes a scanning signal line drive circuit 3041 and a data signal line drive circuit 3042.

  The scanning signal line driving circuit 3041 sequentially selects one scanning signal line from the scanning signal lines G1 to Gm for each line time based on the timing control signal TC1 received from the display control unit 601, and A high level voltage is applied to the selected scanning signal line, and a low level voltage is applied to the other scanning signal lines.

  Based on the display data D (DR, DG, and DB) received from the display controller 601, the data signal line driver circuit 3042 generates a predetermined voltage corresponding to the display data for one row for each line time. The data signal lines SR1 to SRn, SG1 to SGn, and SB1 to SBn are applied (line sequential method). Note that the data signal line driver circuit 3042 may be driven by a dot sequential method.

  The optical sensor driving circuit 305 is a circuit for driving the optical sensor circuit 32. Based on the timing control signal TC2 received from the sensor control unit 602, the optical sensor driving circuit 305 selects a predetermined sensor readout signal line from the sensor readout signal lines RW1 to RWm for each line time. A read voltage is applied, and a voltage other than a predetermined read voltage is applied to the other sensor read signal lines. Similarly, based on the timing control signal TC2, a predetermined reset voltage is applied to the sensor reset signal line selected from the sensor reset signal lines RS1 to RSm for each line time, and the others. A voltage other than a predetermined reset voltage is applied to the sensor reset signal line.

The sensing data output signal lines SP1 to SPn are grouped into p groups (p is an integer of 1 to n), and the sensing data output signal lines belonging to each group are connected via a switch 47 that is sequentially turned on in time division. And connected to the sensor output amplifier 44. The sensor output amplifier 44 amplifies the voltage from the group of sensing data output signal lines connected by the switch 47 and outputs the amplified voltage to the signal conversion circuit 306 as sensor output signals SS (SS1 to SSp).
[Embodiment 1]
The main feature of the present invention is that in the data display / sensor device 100, the movement of the input operation detected by the first display / light sensor unit 300A is specified, and the operation in the first direction of the application for executing a predetermined operation is performed. The first input detection unit 810a that generates the first input signal corresponding to the identified input operation movement, and the input operation movement detected by the second display / light sensor unit 300B are identified. Generating a second input signal corresponding to the movement of the specified input operation in the second display / light sensor unit 300B, which is associated with an operation of the application in a second direction different from the first direction. And an input detection unit 810b.

  As a result, in the data display / sensor device 100, the first input signal associated with the operation of the application in the first direction is moved by the first input detection unit 810a in a second direction different from the first direction of the application. Second input signals associated with the operations are respectively generated by the second input detection unit 810b. Based on the first and second input signals, the application executes operations in the first and second directions separately or in combination.

  Accordingly, the surface members on both the front and back surfaces detect various input operation movements, thereby realizing various operations that could not be realized with the conventional input device. In other words, a variety of operations are realized by combining the operation in the first direction of the application by the input operation from the front surface and the operation in the second direction of the application by the input operation from the back surface. In addition, if various operations are assigned as the operations in the first and second directions, more various operations can be realized. In addition, since the input device is plate-shaped, the user can easily experience the various operations described above by sandwiching the front and back surfaces with fingers.

Thus, the data display / sensor device 100 can provide the user with excellent operability and convenience.
(Description of data display / operation of sensor device 100)
The data display / sensor device 100 according to the first embodiment will be described with reference to FIGS. 1 and 9 to 11.

  First, a specific example of the operation performed by the data display / sensor device 100 will be described with reference to FIG. Next, the configuration of the data display / sensor device 100 will be described with reference to FIGS. Then, the flow of processing for realizing the operation will be described with reference to FIG.

The data display / sensor device 100 may display an image on either the first display / light sensor unit 300A or the second display / light sensor unit 300B. Description will be made assuming that an image is displayed on the optical sensor unit 300A. Note that the first display / light sensor unit 300A is located on the front side of the drawing in FIG.
(Operations performed on the display image)
The outline of FIG. 9 will be described below.

  FIG. 9A is a diagram showing a state in which a photograph of a certain region taken from above (hereinafter, a photograph taken from above is referred to as a bird's-eye view) is displayed on the first display / light sensor unit 300A. is there.

  FIG. 9B is a diagram showing a state in which a bird's-eye view of the surrounding area centering on the region shown in FIG. 9A is displayed. The bird's-eye view is more than that in FIG. 9A. Furthermore, it is a picture when the area is photographed from the sky. Note that the change from FIG. 9A to FIG. 9B is that the user touches the second display / light sensor unit 300B with the index finger, and the finger is constant in a predetermined direction (hereinafter referred to as A direction). Obtained by moving the amount.

  FIG. 9C is a diagram showing a state in which a bird's-eye view of the surrounding area centered on the region shown in FIG. 9B is displayed. The bird's-eye view is more than that in FIG. 9B. Furthermore, it is a picture when the area is photographed from the sky. The change from FIG. 9B to FIG. 9C is obtained when the user touches the second display / light sensor unit 300B with the index finger and the finger moves a certain amount in the A direction. .

  FIG. 9D is a diagram illustrating a state in which a bird's-eye view in which a region different from the region illustrated in FIG. 9C has moved to the center position is displayed. The change from FIG. 9C to FIG. 9D is that the user touches the first display / light sensor unit 300A with the thumb and moves the finger in a predetermined direction (hereinafter referred to as B direction) by a certain amount. The second display / light sensor unit 300B is touched with an index finger, and the finger is moved by a certain amount in the A direction. In addition, when the thumb touching the first display / light sensor unit 300A moves in the B direction, an area different from the area at the center of the display screen in FIGS. 9A to 9C is displayed. Move to the center of

  FIG. 9 (e) is a diagram showing a state in which a bird's-eye view of the surrounding area centered on the central area of FIG. 9 (d) is displayed. The bird's-eye view is from the sky above FIG. 9 (d). This is a picture taken when the area was photographed. The change from FIG. 9D to FIG. 9E is obtained when the user touches the second display / light sensor unit 300B with the index finger and the finger moves a certain amount in the A direction. .

  FIG. 9 (f) is a diagram showing a state in which a bird's-eye view of the surrounding area centered on the region shown in FIG. 9 (e) is displayed. The bird's-eye view is from a lower sky than FIG. 9 (e). This is a picture taken when the area is photographed. Note that the change from FIG. 9E to FIG. 9F is that the user touches the second display / light sensor unit 300B with the index finger and the finger is referred to as a predetermined direction (hereinafter referred to as A ′ direction). Note that the A ′ direction is obtained by moving a certain amount in the direction opposite to the A direction).

  FIG. 9G is a diagram showing a state in which a bird's-eye view of the surrounding area centered on the region shown in FIG. 9F is displayed. The bird's-eye view is from a lower altitude than FIG. This is a picture taken when the area is photographed. Note that the change from FIG. 9F to FIG. 9E is obtained when the user touches the second display / light sensor unit 300B with the index finger and the finger moves a certain amount in the A ′ direction. It is done.

Here, if the X axis and the Y axis are set in the first display / light sensor unit 300A, the change from FIG. 9C to FIG. 9D is the direction defined by the XY coordinates ( It is a change in the first direction). Also, assuming that the second display / light sensor unit 300B is set with a Z axis that defines three dimensions together with the X axis and the Y axis, the change from FIG. 9A to FIG. This is a change in the axial direction (second direction). That is, the movement in the first and second directions, which are different from each other, is realized by the movement of the input operation detected by the first display / light sensor unit 300A and the second display / light sensor unit 300B.
(Specific description of operation performed on display image)
Next, typical operations among the operations described with reference to FIGS. 9A to 9F will be described more specifically.

  First, the operation of changing the displayed photograph from FIG. 9A to FIG. 9B will be described. As shown in FIG. 9A, a bird's-eye view of a certain area is displayed on the first display / light sensor unit 300A. In this state, second display / light sensor unit 300B detects the image of the index finger and also detects that the image of the finger has moved a certain amount in the A direction (for the sake of convenience, it is upward in the drawing). By this operation, the first display / light sensor unit 300A further displays a photograph when the area is photographed from the sky (FIG. 9B).

  That is, the second display / light sensor unit 300B detects that the image of the index finger has moved in the A direction, whereby a photograph displayed on the first display / light sensor unit 300A is shown in FIG. 9A. Is also displayed as a photograph when the area is photographed from above (this is hereinafter referred to as map reduction). The rate at which the map is reduced is determined by the amount of movement of the image moved in the A direction. The larger the amount of movement of the image, the higher the reduction rate, and the smaller the amount of movement of the image, the smaller the reduction rate.

  Contrary to the above operation, when the display photograph changes from FIG. 9E to FIG. 9F, the second display / light sensor unit 300B detects the image of the index finger and the image of the finger is changed. This is realized by detecting a certain amount of movement in the A ′ direction (downward in the drawing).

  That is, the second display / light sensor unit 300B detects that the image of the index finger has moved in the A ′ direction, whereby a photograph displayed on the first display / light sensor unit 300A is shown in FIG. It is displayed as a picture when the area is photographed from a lower altitude (hereinafter, this is referred to as enlargement of the map). The rate at which the map is enlarged is determined by the amount of movement of the image moved in the A ′ direction. The larger the amount of movement of the image, the higher the enlargement rate. The smaller the amount of movement of the image, the smaller the enlargement rate.

  Next, of the operations in which the displayed photograph changes from FIG. 9C to FIG. 9D, the operation in which the area displayed at the center position of the first display / light sensor unit 300A moves will be described.

  As shown in FIG. 9C, a bird's-eye view of a certain area is displayed on the first display / light sensor unit 300A. In this state, the first display / light sensor unit 300A detects the thumb image and also detects that the finger image has moved a certain amount in the B direction (for the sake of convenience, the left direction in the drawing).

  By this operation, a bird's-eye view photo in which an area different from the area shown in FIG. 9C is moved to the center position is displayed on the first display / light sensor unit 300A (FIG. 9D). That is, the first display / light sensor unit 300A detects that the image of the thumb has moved in the B direction, whereby the central area of the photograph displayed on the first display / light sensor unit 300A is shown in FIG. In c), the area located on the east side of the screen moves to the center position (hereinafter referred to as map movement).

  The amount of movement of the map is determined by the amount of movement of the image moved in the B direction. The amount of movement increases as the amount of movement of the image increases, and the amount of movement decreases as the amount of movement of the image decreases. Further, the direction B may be any direction from east to west, north and south, and is located in the direction opposite to the direction in which the thumb is moved due to the movement of the thumb and is not displayed on the first display / light sensor unit 300A. Is displayed on the first display / light sensor unit 300A.

  In the above description, the above operations are realized by the thumb and the index finger, but any finger may be used. In other words, any finger may be used to cause the display / light sensor unit 300 to detect the movement of the finger image, which is predetermined with respect to the map displayed on the first display / light sensor unit 300A. It is a precondition to execute the operation. The detection of the finger image by the display / light sensor unit 300 does not require the finger to actually touch (touch) the first display / light sensor unit 300A and / or the second display / light sensor unit 300B. / The optical sensor unit 300 may detect the image of the finger. However, in the following description, for convenience of explanation, the display / light sensor unit 300 is moved with the thumb from the upper surface (first display / light sensor unit 300A) side and the index finger from the lower surface (second display / light sensor unit 300B) side. It will be described as touching.

  Furthermore, although the image displayed on the first display / light sensor unit 300A is described as a map, it is not necessarily a map, and an application that executes a three-dimensional operation defined by the XYZ axes functions. Other images displayed by doing so may be used. However, in the present embodiment, the description will be made assuming that the image displayed on first display / light sensor unit 300A is a map.

  Although the speed of movement of the thumb and index finger is not particularly mentioned, the movement associated with the movement of the image of the finger is executed according to the speed of movement of the finger.

  The specific examples of the various operations executed in response to the movement of the finger image have been described above with reference to FIGS. 9 (a) to 9 (g). In the above description, the movement direction of the thumb and forefinger, the movement amount, and the action to be executed are associated with each other. However, the action to be executed is not limited to the above-described actions (movement, enlargement / reduction). . That is, if the movement of the finger image and the action to be executed are associated with each other and registered in a finger-command correspondence information table described later, various actions can be realized.

  Here, the moving direction of the finger image detected by the second display / light sensor unit 300B can be specified by the following method, for example. That is, a reference axis (X / Y axis) is set for the second display / light sensor unit 300B, and the image detected by the second display / light sensor unit 300B is displayed if it moves in the positive direction of the Y axis. It may be determined that the image is enlarged and the display image is reduced if the image moves in the negative direction of the Y axis. In this manner, the moving direction of the finger image detected by the second display / light sensor unit 300B is specified.

  Further, the movement amount of the finger image detected by the second display / light sensor unit 300B is determined by specifying the finger image by the upper reference coordinates (X · Y), and the movement of the finger image is determined based on the change in the coordinates. The amount can be specified. Then, the display image may be enlarged or reduced according to the amount of movement.

  Further, as described above, the operation to be executed is not limited to the above-described operations (movement, enlargement / reduction). The various operations to be executed can be made different depending on the operating state (scene) of the application, the mode, and the like.

(Explanation of main parts of data display / sensor device)
Next, a detailed configuration of the data display / sensor device 100 will be described with reference to FIG. Here, in order to make the description easy to understand, the description of the operations of the data processing unit 700 and the circuit control unit 600 located between the main control unit 800 and the display / light sensor unit 300 is omitted. However, to be precise, when performing image display and image detection, the input interface control unit 805 of the main control unit 800 transmits a command to the data processing unit 700, and the data processing unit 700 performs circuit control based on the command. The circuit control unit 600 transmits a signal to the display / light sensor unit 300. Further, the main control unit 800 acquires the entire image data, the partial image data, and the coordinate data from the data processing unit 700 as a response to the command transmitted to the data processing unit 700.

  FIG. 1 is a block diagram showing a main configuration of the data display / sensor device 100. As shown in the figure, the main control unit 800 includes an input interface control unit 805 and an application execution unit 850 (operation execution device). The input interface control unit 805 includes an input detection unit 810 and an image display control unit 860 (image display control means).

  The input detection unit 810 includes a first input detection unit 810a and a second input detection unit 810b. The first input detection unit 810 a acquires coordinate data, whole image data, and partial image data from the data processing unit 700 when the first display / light sensor unit 300 </ b> A detects a finger image. The first display / light sensor unit 300A has an X axis and a Y axis (see FIG. 7 and the like), and the first input detection unit 810a detects the finger detected by the first display / light sensor unit 300A. The movement of the image is specified by XY coordinates, and a first input signal corresponding to the XY coordinates is generated. Then, the first input detection unit 810a outputs the first input signal to the application execution unit 850.

  The first input detection unit 810a may output the information indicating the XY coordinates as it is to the application execution unit 850 as the first input signal. Alternatively, the first input detection unit 810a generates a first input signal indicating a command associated with the specified XY coordinates based on first finger-command correspondence information (first command correspondence information) to be described later. You may output to 850.

  The second input detection unit 810b acquires coordinate data, whole image data, and partial image data from the data processing unit 700 when the second display / light sensor unit 300B detects a finger image. Then, the second input detection unit 810b specifies the movement of the finger image detected by the second display / light sensor unit 300B as the movement amount and movement direction on the Z axis that defines three dimensions together with the X axis and the Y axis. Then, a second input signal corresponding to the identified movement amount and movement direction on the Z axis is generated. Then, the second input detection unit 810b outputs the second input signal to the application execution unit 850.

  Note that the second input detection unit 810b may output the movement amount and movement direction of the finger image detected by the second display / light sensor unit 300B to the application execution unit 850 as they are as the second input signal. Alternatively, the second input detection unit 810b is a first command that indicates a command in which the identified movement amount and movement direction on the Z axis are associated with each other based on second finger-command correspondence information (second command correspondence information) described later. Two input signals may be output to the application execution unit 850.

  Here, the input detection unit 810 (when there is no need to distinguish between the first input detection unit 810a and the second input detection unit 810b, in the following description, the input detection unit 810 is collectively referred to as coordinate data, It is not necessary to acquire all of the entire image data and partial image data, and a configuration in which each of the above data is appropriately selected and acquired may be employed.

  Further, in FIG. 1, coordinate data, whole image data, and partial image data transmitted from the data processing unit 700 to the input detection unit 810 are indicated by solid lines and broken lines. The first input detection unit 810a receives coordinate data, whole image data, and partial image data from the first display / light sensor unit 300A, and the second input detection unit 810b receives the second display / light sensor unit 300B, respectively. The configuration to be acquired is shown. The former is indicated by a solid line and the latter is indicated by a broken line.

  Next, the first finger-command correspondence information and the second finger-command correspondence information that the input detection unit 810 reads from the storage unit 901 will be described with reference to FIG. In FIG. 1, the storage unit 901 is provided outside the data display / sensor device 100. However, the storage unit 901 may be configured to be provided inside the data display / sensor device 100.

  FIG. 10 is a diagram illustrating an example of finger-command correspondence information when the application is a map. FIG. 10A illustrates first finger-command correspondence information, and FIG. 10B illustrates second finger-command correspondence information. It is a table which shows correspondence information.

  First, FIG. 10A will be described. FIG. 10A is a table showing first finger-command correspondence information, and is composed of two items (XY coordinates, command). That is, the X axis and the Y axis are set in the first display / light sensor unit 300A, and the first input detection unit 810a detects the movement of the finger image detected by the first display / light sensor unit 300A in the XY coordinates. Specified. Then, the first input detection unit 810a calculates the motion of the finger image as a movement vector from the specified XY coordinates. Specifically, as described with reference to FIG. 7, the coordinate data in the entire image data (region AP) of the representative coordinate Z of the partial image data (region PP) is represented by (Xa, Ya), The coordinate data in the image data (region PP) is represented as (Xp, Yp). Accordingly, the first input detection unit 810a can calculate the movement vector of the thumb image by acquiring the coordinate data before and after the movement of the thumb image and calculating the difference of the coordinate data.

  Then, the first input detection unit 810a reads the “command” item corresponding to the XY coordinates in the first finger-command correspondence information. Then, since the corresponding command is “move the map according to the movement vector”, the first input detection unit 810a determines the command associated with the XY coordinates, and receives the first input signal indicating the command. The data is transmitted to the application execution unit 850.

  As an example, the case of a change from FIG. 9C to FIG. 9D in which an area different from the area at the center position of the display screen moves to the center of the display screen will be described. At this time, the user touches the first display / light sensor unit 300A with the thumb and moves the finger in a predetermined direction (hereinafter referred to as B direction) by a certain amount. The first input detection unit 810a obtains XY coordinates related to the movement of the image from the movement of the image of the finger, and calculates the movement vector. Then, the first input detection unit 810a generates a first input signal indicating an instruction to move the map according to the movement vector, and transmits the first input signal to the application execution unit 850.

  In this way, the first input detection unit 810a determines a command associated with the identified XY coordinates based on the first finger-command correspondence information, and sends the first input signal indicating the command to the application execution unit. To 850.

  Next, FIG. 10B will be described. FIG. 10B is a table showing the second finger-command correspondence information, and includes two items (a movement amount and a movement direction on the Z axis, and a command). That is, since the movement of the finger image detected by the second display / light sensor unit 300B is specified as the movement amount and movement direction on the Z axis that defines three dimensions together with the X axis and the Y axis, the second input detection is performed. The unit 810b calculates the movement amount on the Z axis and the movement direction (+/−) on the Z axis from the identified movement amount and movement direction on the Z axis.

  Specifically, as described with reference to FIG. 7, the coordinate data in the entire image data (region AP) of the representative coordinate Z of the partial image data (region PP) is represented by (Xa, Ya), The coordinate data in the image data (region PP) is represented as (Xp, Yp). Therefore, the second input detection unit 810b can acquire the coordinate data before and after the movement of the index finger image, and calculate the movement amount of the index finger image based on the coordinate data.

  As for the moving direction, the second display / light sensor unit 300B has a reference axis (X / Y axis), and the finger image detected by the second display / light sensor unit 300B is positive on the Y axis. “+” If moving in the direction, “−” if moving in the negative direction of the Y-axis. Thereby, the moving direction of the finger image detected by the second display / light sensor unit 300B can be specified by “+” and “−”.

  Then, the second input detection unit 810b moves the movement amount and movement direction acquired by the above method on the Z axis that defines three dimensions together with the X axis and the Y axis set in the first display / light sensor unit 300A. Identified as quantity and direction of movement.

  Subsequently, the second input detection unit 810b reads the “command” item corresponding to the movement direction on the Z axis in the second finger-command correspondence information. Then, since the corresponding command is “magnify the map if“ + ”/ shrink the map if“ − ””, the second input detection unit 810b corresponds to the movement direction on the Z axis. Determine the attached instruction. At the same time, the second input detection unit 810b reads the “command” item corresponding to the movement amount on the Z axis in the second finger-command correspondence information. Then, since the corresponding command is “enlarge / reduce at a rate proportional to the movement amount”, the second input detection unit 810b determines the command associated with the movement amount on the Z axis. Then, the second input detection unit 810b transmits a second input signal indicating the two instructions to the application execution unit 850.

  Here, “in proportion to the amount of movement” means, for example, that the amount of movement is divided into 10 stages according to the size of the screen. Then, if the map is enlarged or reduced by 10%, the map can be enlarged or reduced at a rate proportional to the calculated movement amount. Note that the 10 steps described here are merely examples, and may be divided into more steps. This is because the larger the number of steps, the more accurately the map is scaled at a rate proportional to the amount of movement. Alternatively, the map may be enlarged or reduced by a proportion proportional to the movement amount without dividing the movement amount into stages.

  As described above, when the first finger-command correspondence information and the second finger-command correspondence information stored in the storage unit 901 are read, the input detection unit 810 moves the identified movement amount and movement direction on the Z axis. Can be determined, and a first input signal and a second input signal indicating the instruction can be transmitted to the application execution unit 850.

  In the above description, the application is a map. However, the application is not limited to a map. Further, the first finger-command correspondence information and the second finger-command correspondence information may be prepared in advance and registered in the storage unit 901 for each operating state (scene) such as an application to be executed and a mode.

  The application execution unit 850 receives the first input signal and the second input signal from the input detection unit 810 and executes an operation indicated by the command. Then, the application execution unit 850 transmits operation result information that is information indicating a result of the operation to the image display control unit 860.

  For example, the finger image detected by the first display / light sensor unit 300A moves from point A to point B, and the command associated with the movement of the finger image is “the finger image is point A to point B. In the case of “move the map according to the movement vector when moving to”, the application execution unit 850 performs a process of moving the map according to the movement vector. Then, the application execution unit 850 transmits operation result information indicating the result of the operation to the image display control unit 860.

  Next, the application execution unit 850 receives the first input signal and the second input signal from the input detection unit 810, executes the operation indicated by each input signal, and further, operation result information indicating the result of the operation Is transmitted to the image display control unit 860.

  In FIG. 1, the application execution unit 850 is included in the main control unit 800. However, the application execution unit 850 may be provided outside the main control unit 800, and a predetermined operation may be executed via the external communication unit 907. Further, the application execution unit 850 may have one or a plurality of applications.

  Upon receiving the operation result information from the application execution unit 850, the image display control unit 860 transmits display data to the data processing unit 700. The display data prompts the data processing unit 700 to display the operation result executed by the application execution unit 850 on the first display / light sensor unit 300A.

  Therefore, when the application execution unit 850 performs an operation of “enlarging the map displayed on the first display / light sensor unit 300A by 20%”, the image display control unit 860 informs the data processing unit 700 of that. The display data shown is transmitted. As a result, the operation result is displayed on the first display / light sensor unit 300A. That is, a state where the map is enlarged by 20% is displayed on the first display / light sensor unit 300A.

  Finally, the input detection unit 810 performs data processing to transmit the entire image data, partial image data, and coordinate data of the finger image for each of the first display / light sensor unit 300A and the second display / light sensor unit 300B. Request to the part 700. This is because the application program executed by the main control unit 800 displays the entire image data, partial image data, and coordinate data of the finger image for each of the first display / light sensor unit 300A and the second display / light sensor unit 300B. By controlling the data processing unit 700 to transmit.

(Flowchart explanation)
Next, the processing flow of the data display / sensor device 100 and the details of the command transmitted from the input interface control unit 805 to the data processing unit 700 will be described with reference to FIG.

  FIG. 11 is a flowchart showing a process flow of the data display / sensor device 100.

  First, when the data display / sensor device 100 is powered off, the data display / sensor device 100 is turned on when the user presses the power switch 905 (S11).

  In step S <b> 13, when the user presses the power switch 905, an image is displayed on the display / light sensor unit 300. Note that the image is displayed on the first display / light sensor unit 300A. In this state, the first display / light sensor unit 300A and the second display / light sensor unit 300B stand by until a finger image is detected.

  Here, when displaying an image on the first display / light sensor unit 300A, a command specifying the upper screen (“001”) as the value of the “display panel” field is sent from the input interface control unit 805 to the data processing unit 700. Is output. When an image is displayed on the second display / light sensor unit 300B together with the first display / light sensor unit 300A, a command specifying both screens (“011”) as the value of the “display panel” field. Is output from the input interface control unit 805 to the data processing unit 700. Here, in order to display an image only on the first display / light sensor unit 300A, a command specifying the upper screen (“001”) as the value of the “display panel” field is input from the input interface control unit 805 to the data processing unit 700. Is output.

  Further, at this stage, it is assumed that the display / light sensor unit 300 is in a state in which a nearby image can be detected. In order to perform the detection, “event” is specified in the “data acquisition timing” field (“01”), and both “coordinate” and “whole image” are specified in the “data type” field (“101” is specified). The designated command is output from the input interface control unit 805 to the data processing unit 700. Note that a command specifying only “coordinates” or “entire image” may be output from the input interface control unit 805 to the data processing unit 700.

  Further, in order to perform the detection in the first display / light sensor unit 300A and the second display / light sensor unit 300B, a command in which both the display / light sensor units 300 are specified (“011”) in the “scan panel” field. Is output from the input interface control unit 805 to the data processing unit 700.

  Accordingly, when the sensor built-in liquid crystal panel 301 detects a nearby image, the input detection unit 810 can acquire coordinate data, whole image data, and partial image data from the data processing unit 700.

  That is, when the user's finger touches the first display / light sensor unit 300A and the second display / light sensor unit 300B, the light sensor circuit 32 detects at least one of a shadow image and a reflection image of the touched finger. As a result, the image data acquired by the data processing unit 700 from the sensor control unit 602 changes.

  In this way, the input detection unit 810 obtains the coordinate data, the entire image data, and the partial image data that are output from the data processing unit 700 when a change occurs in the image data, thereby performing the first display. / The optical sensor unit 300A and the second display / optical sensor unit 300B can recognize that the finger image has been detected.

  In step S15, the first display / light sensor unit 300A detects the thumb image, and the coordinate data, the entire image data, and the partial image data are output from the data processing unit 700 to the first input detection unit 810a. In FIG. 11, for convenience, the first display / light sensor unit 300 </ b> A is referred to as “A”, and the second display / light sensor unit 300 </ b> B is referred to as “B”.

  Subsequently, in step S17, the first input detection unit 810a generates a first input signal. Note that the X axis and the Y axis are set in the first display / light sensor unit 300A, and the first input detection unit 810a indicates the movement of the finger image detected by the first display / light sensor unit 300A in the XY coordinates. The first input signal is generated so as to correspond to the XY coordinates. Then, the first input detection unit 810a outputs the generated first input signal to the application execution unit 850.

  Similarly, in step S19, the second display / light sensor unit 300B detects the index finger image, and the coordinate data, the entire image data, and the partial image data are output from the data processing unit 700 to the second input detection unit 810b.

  Subsequently, in step S21, the second input detection unit 810b generates a second input signal. The second input detection unit 810b uses the movement of the finger image detected by the second display / light sensor unit 300B as the movement amount and movement direction on the Z axis that defines three dimensions together with the X axis and the Y axis. The second input signal is generated so as to correspond to the identified movement amount and movement direction on the Z-axis. Then, the second input detection unit 810b outputs the generated second input signal to the application execution unit 850.

  Subsequently, in step S25, the application execution unit 850 receives the first input signal and / or the second input signal from the input detection unit 810, and executes an operation indicated by each input signal. Then, the application execution unit 850 outputs operation result information that is information indicating the result of the operation to the image display control unit 860.

  In step S <b> 27, the image display control unit 860 receives operation result information from the application execution unit 850 and transmits display data to the data processing unit 700. The display data prompts the data processing unit 700 to display the operation result executed by the application execution unit 850 on the first display / light sensor unit 300A. As a result, the operation result executed by the application execution unit 850 is displayed on the first display / light sensor unit 300A.

  At this time, when displaying an image on the first display / light sensor unit 300A, a command specifying the upper screen (“001”) as the value of the “display panel” field is sent from the input interface control unit 805 to the data processing unit 700. Is output. When an image is displayed on the second display / light sensor unit 300B together with the first display / light sensor unit 300A, a command specifying both screens ("011") as the value of the "display panel" field Is output from the input interface control unit 805 to the data processing unit 700. Here, in order to display an image only on the first display / light sensor unit 300A, a command specifying the upper screen (“001”) as the value of the “display panel” field is input from the input interface control unit 805 to the data processing unit 700. Is output.

  Here, the processing flow of each step (step S13 to step S27) will be described by applying the operation from FIG. 9C to FIG. 9D.

  In FIG. 9C, a bird's-eye view of the surrounding area centered on the area shown in FIG. 9B is displayed on the first display / light sensor unit 300A. This is because, as a result of the command specifying the upper screen (“001”) as the value of the “display panel” field being output from the input interface control unit 805 to the data processing unit 700, the bird's-eye view is displayed on the first display / light sensor unit 300A. A photograph is displayed (step S13).

  At this stage, “Event” is specified in the “Data acquisition timing” field (“01”), and both “Coordinate” and “Entire image” are specified in the “Data type” field (“101” is set). The designated command is output from the input interface control unit 805 to the data processing unit 700, and the display / light sensor unit 300 is in a state in which a nearby image can be detected. Note that a command specifying only “coordinates” or “entire image” may be output from the input interface control unit 805 to the data processing unit 700.

  In addition, in order to perform the detection in the first display / light sensor unit 300A and the second display / light sensor unit 300B, a command in which both display / light sensor units 300 are specified (“011”) in the “scan panel” field. Is output from the input interface control unit 805 to the data processing unit 700.

  In such a state, the user can use the thumb from the upper surface (the side on which the first display / light sensor unit 300A is disposed) and the index finger from the lower surface (the side on which the second display / light sensor unit 300B is disposed). The data display / sensor device 100 is held. The thumb detected by the first display / light sensor unit 300A moves in the B direction, and the index finger detected by the second display / light sensor unit 300B moves in the A direction by a certain amount. Thereby, both the first display / light sensor unit 300A and the second display / light sensor unit 300B detect the image of the finger, and the first input detection unit 810a performs before and after the movement of the images of both fingers and during the movement. The coordinate data, the whole image data, and the partial image data are acquired from the data processing unit 700 (steps S15 and S19).

  Subsequently, the first input detection unit 810a and the second input detection unit 810b that have acquired the coordinate data, the whole image data, and the partial image data from the data processing unit 700 generate the first input signal and the second input signal, respectively. The first input signal is associated with a certain amount of movement of the thumb image detected by the first display / light sensor unit 300A in the B direction. That is, the first input detection unit 810a specifies the movement of the finger image detected by the first display / light sensor unit 300A (movement moved by a certain amount in the B direction) using the XY coordinates, and corresponds to the XY coordinates. A first input signal is generated (step S17).

  Similarly, the second input signal corresponds to the fact that the index finger image detected by the second display / light sensor unit 300B has moved a certain amount in the A direction. That is, the second input detection unit 810b defines the movement of the finger image detected by the second display / light sensor unit 300B (movement moved by a certain amount in the direction A) in three dimensions along with the X axis and the Y axis. The movement amount and the movement direction on the Z axis are specified, and a second input signal is generated so as to correspond to the specified movement amount and movement direction on the Z axis (step S21).

  The first input signal and the second input signal generated in this way are output from the input detection unit 810 to the application execution unit 850, and the application execution unit 850 executes an operation indicated by each input signal. Then, the application execution unit 850 outputs operation result information, which is information indicating the operation result, to the image display control unit 860 (step S25).

  Then, the image display control unit 860 receives operation result information from the application execution unit 850 and transmits display data to the data processing unit 700. The display data prompts the data processing unit 700 to display the operation result executed by the application execution unit 850 on the first display / light sensor unit 300A. As a result, the operation result executed by the application execution unit 850 is displayed on the first display / light sensor unit 300A (step S27). Then, the execution result is displayed as a bird's-eye view photograph shown in FIG.

  At this time, since the map is displayed on the first display / light sensor unit 300A, the command specifying the upper screen (“001”) as the value of the “display panel” field is input from the input interface control unit 805 to the data processing unit 700. Is output. When the map is to be displayed on the second display / light sensor unit 300B together with the first display / light sensor unit 300A, both screens ("011") are designated as the value of the "display panel" field. A command is output from the input interface control unit 805 to the data processing unit 700. However, here, in order to display an image on the first display / light sensor unit 300A, a command specifying the upper screen ("001") as the value of the "display panel" field is sent from the input interface control unit 805 to the data processing unit 700. Is output.

  In this way, the operation described with reference to FIGS. 9C and 9D is executed, and the execution result is displayed on the first display / light sensor unit 300A.

  The first display / light sensor unit 300A and the second display / light sensor unit 300B have been described as using the thumb and index finger as detection targets. However, the types of fingers to be detected are limited to these fingers. The middle finger, ring finger, and little finger may be used. This is because a predetermined operation is performed on the image displayed on the first display / light sensor unit 300A based on the detected movement of the finger image instead of the type of the finger.

  Furthermore, in the above description, a configuration has been described in which a map is displayed on the first display / light sensor unit 300A, and the map / movement is detected when the display / light sensor unit 300 detects the movement of a finger image. However, the object to be moved by the movement of the finger is not necessarily a map, and may be a pointer on the map.

  The effects obtained by the processing of the data display / sensor device 100 will be described as follows.

  With the above-described configuration, in the data display / sensor device 100, the first input signal associated with the operation in the first direction of the application executed by the application execution unit 850 is detected by the first input detection unit 810a. A second input signal associated with an operation in a second direction different from the first direction is generated by the second input detection unit 810b. Then, the first and second input signals are transmitted to the application execution unit 850, and the application execution unit 850 executes the operation of the application according to the first and second input signals.

  Therefore, when the first display / light sensor unit 300A and the second display / light sensor unit 300B detect the movements of various input operations, various operations that cannot be realized by the conventional input device are realized. In other words, a variety of operations are realized by combining the operation in the first direction of the application by the input operation from the front surface and the operation in the second direction of the application by the input operation from the back surface. In addition, if various operations are assigned as the operations in the first and second directions, more various operations can be realized. In addition, since the data display / sensor device 100 has a plate shape, the user can easily experience the above-described various operations by holding both sides of the data display / sensor device 100 with fingers.

  Furthermore, the first input signal is a signal corresponding to the movement of the input operation detected by the first display / light sensor unit 300A, which is specified as the movement amount and movement direction on the XY coordinates by the first input detection unit 810a. The second input signal is a signal corresponding to the movement of the input operation detected by the second display / light sensor unit 300B specified by the second input detection unit 810b as the movement amount and movement direction on the Z axis.

  Therefore, the first display / light sensor unit 300A and the second display / light sensor unit 300B detect various movements of the input operation, thereby realizing a variety of three-dimensional operations that could not be realized by the conventional input device. First and second input signals are generated. Then, the first and second input signals are transmitted to the application execution unit 850, and the application execution unit 850 operates the application based on the first and second input signals.

  Therefore, the first display / light sensor unit 300A and the second display / light sensor unit 300B detect various input operation movements, thereby realizing a variety of three-dimensional operations that could not be realized by conventional input devices. Therefore, the data display / sensor device 100 having excellent convenience and operability is provided to the user. In addition, the user imagines a three-dimensional motion realized by his / her own operation, and then performs a predetermined input operation that brings the three-dimensional motion to the first display / light sensor unit 300A and the second display / light sensor unit 300B. It is possible to detect movement and to experience a sensory superior operation.

  Further, in the data display / sensor device 100, the first direction is a direction defined by the XY axes in an application that executes a three-dimensional operation defined by the XYZ axes. Then, based on the first finger-command correspondence information in which the movement amount and movement direction on the XY coordinates specified by the first input detection unit 810a and the command for executing the operation in the first direction are associated with each other, the first input The detection unit 810a determines a command associated with the movement amount and movement direction on the specified XY coordinates.

  Accordingly, by referring to the first correspondence command information recorded in the storage unit 901 provided inside or outside, the first input detection unit 810a associates the movement amount and movement direction on the specified XY coordinates. Can be easily determined.

  The second direction is a direction defined by the Z-axis direction in the application. Then, based on the second finger-command correspondence information in which the movement amount and movement direction on the Z axis specified by the second input detection unit 810b and the command for executing the operation in the second direction are associated with each other, the second input The detection unit 810b determines a command in which the identified movement amount and movement direction on the Z axis are associated with each other.

  Accordingly, by referring to the second correspondence command information recorded in the storage unit 901, the second input detection unit 810b easily determines a command associated with the identified movement amount and movement direction on the Z axis. be able to.

  As described above, the first input detection unit 810a and the second input detection unit 810b are associated with the movement amount and the movement direction on the XY coordinates and the Z axis based on the first and second finger-command correspondence information. Can be easily determined. If the first and second finger-command correspondence information is associated with each application and recorded in the storage unit 901, various three-dimensional operations can be experienced to the user quickly and surely for various applications. Can be made.

  Further, in the data display / sensor device 100, in the above configuration, when an application for displaying a map is operating in the application execution unit 850, the first input detection unit 810a moves the map or a pointer on the map. The second input detection unit 810b may transmit a second input signal indicating a command for enlarging / reducing the map to the application execution unit 850, respectively. .

  With the above configuration, the map or the pointer on the map moves according to the finger movement detected by the first display / light sensor unit 300A, and the finger movement detected by the second display / light sensor unit 300B. The map is enlarged or reduced accordingly.

  Accordingly, the movement of the finger on the front surface (first display / light sensor unit 300A) side moves the map up, down, left and right, and the movement of the finger on the back surface (second display / light sensor unit 300B) side enlarges / reduces the map. Therefore, an input device that is easy for the user to operate intuitively and is highly convenient is realized.

  Further, the data display / sensor device 100 includes an image display control unit 860 that displays an image on at least one of the display / light sensor units 300 among the display / light sensor units 300, and supports the first finger-command. The information and the second finger-command correspondence information may be associated with the image.

  According to the above configuration, in the data display / sensor device 100, the image display control unit 860 displays an image on at least one of the display / light sensor units 300. The image is associated with the first finger-command correspondence information and the second finger-command correspondence information.

  That is, when the image display control unit 860 displays an image, the operation target is clearly indicated to the user. Since the first finger-command correspondence information and the second finger-command correspondence information are associated with the image, the user visually recognizes the image and executes a desired operation on the image. Therefore, the display / light sensor unit 300 may detect the movement of the finger for this purpose.

  Therefore, the user can execute a desired operation after visually recognizing the image, and an input device that is easy to operate intuitively and is convenient for the user is realized.

  Further, in the data display / sensor device 100, in the above configuration, the image display control unit 860 may display an image based on the result of the operation executed by the application execution unit 850 on the display / light sensor unit 300. .

  According to the above configuration, in the data display / sensor device 100, the image display control unit 860 causes the display / light sensor unit 300 to display an image based on the result of the operation executed by the application execution unit 850.

  Therefore, the image display control unit 860 can display the result of the operation executed in association with the movement of the user's finger on the display / light sensor unit 300 and allow the user to confirm the result.

  In addition, when it is desired to give some operation to the image after the operation, the display / light sensor unit 300 may detect the desired movement of the finger image. That is, the user can continuously perform the next operation by checking the operation result on the screen.

  Further, the application execution unit 850 may be one or plural, may be configured integrally with the input device, and may be present at a position physically separated from the input device. Good.

Thus, the data display / sensor device 100 can provide the user with more excellent operability and convenience.
[Embodiment 2]
Next, a data display / sensor device 110 according to another embodiment will be described with reference to FIGS. In addition, the description similar to the content demonstrated with reference to FIG. 1, FIG.

  The main feature of the data display / sensor device 110 is that the display / light sensor unit 300 for detecting the input operation is provided on both the front and back surfaces, and the movement of the input operation detected by the first display / light sensor unit 300A. And a first input generation unit 880a (first input generation) that generates a first input signal corresponding to the movement of the specified input operation associated with the first movement of the application that executes the predetermined movement. And the movement of the input operation detected by the second display / light sensor unit 300B, or the movement of the input operation from detection to non-detection with respect to the second display / light sensor unit 300B, A second that generates a second input signal corresponding to the movement of the specified input operation in the second display / light sensor unit 300B, which is associated with a second action associated with the first action of the application; In that it comprises force generating unit 880b (second input generating means), a.

  Thereby, in the data display / sensor device 110, the first input signal associated with the first operation of the application is associated with the second operation associated with the first operation by the first input generation unit 880a. The second input signal is generated by the second input generation unit 880b. Then, the second input signal is the second input of the movement of the input operation detected by the second display / light sensor unit 300B or the movement of the input operation from detection to non-detection with respect to the second display / light sensor unit 300B. It is generated by being specified by the detection means.

More specifically, when the movement of the input operation detected by the second display / light sensor unit 300B changes from the detection state to the non-detection state, the operation of the application executed so far is “cancel (invalid)”. ”Is assigned to be. At this time, when the user wants to “cancel (invalidate)” the operation of the application that has been executed, the input operation from detection to non-detection is made to the second display / light sensor unit 300B. Give it. As a result, the operation of the application that has been executed is canceled.
(Description of Data Display / Operation of Sensor Device 110)
A data display / sensor device 110 according to Embodiment 2 will be described with reference to FIGS.

  First, a specific example of the operation performed by the data display / sensor device 110 will be described with reference to FIG. Next, the configuration of the data display / sensor device 110 will be described with reference to FIGS. Then, the flow of processing for realizing the operation will be described with reference to FIG.

The data display / sensor device 110 may display an image on either the first display / light sensor unit 300A or the second display / light sensor unit 300B. Description will be made assuming that an image is displayed on the optical sensor unit 300A. The first display / light sensor unit 300A is located on the front side of the drawing in FIG.
(Operations performed on the display image)
The outline of FIG. 12 will be described below. In FIG. 12A, a square image 20 is displayed on the first display / light sensor unit 300A, and the image 20 is touched with the thumb from the top and the index finger from the bottom. Then, as shown in FIG. 12B, the user is moving the image 20 to the outside of the screen by moving the thumb and index finger in the same direction (rightward in the horizontal direction of the drawing). That is, the user moves the image 20 by the movement of the finger according to the distance that the finger moves. Next, in FIG.12 (c), the user has separated the index finger from the 2nd display / light sensor part 300B. Then, the image 20 is automatically returned to the initial position shown in FIG. As described above, the second display / light sensor unit 300B shifts from the detection state of the finger image to the non-detection state, thereby canceling the operation executed so far (movement of the image 20).

  In FIGS. 12A and 12B, it has been described that both the thumb and the index finger are moved to the right in the horizontal direction of the drawing. However, the direction in which the thumb and index finger are moved is not limited to this, and may be any direction. Therefore, the user can move the image to an arbitrary place.

  In FIGS. 12A and 12B, the thumb and the index finger are both moved to the right. However, the image moves even when only the thumb moves.

  Furthermore, as described above, the image 20 displayed on the first display / light sensor unit 300A is touched with the thumb from the top surface but is not actually touched with the index finger from the bottom surface. However, in the following description according to the present embodiment, for convenience of description, such a state is expressed as “touching with the thumb from the top surface and the index finger from the bottom surface, respectively”.

  In addition, although it has been described that the thumb and index finger touch the image 20 from both the front and back sides, any finger may be used. That is, the image 20 may be sandwiched from both the front and back surfaces using any finger, and this is a premise for performing a predetermined operation on the image 20. Therefore, even if the image 20 is touched only from either the upper surface or the lower surface, a predetermined operation cannot be performed on the image 20 by itself. Further, although it is described that the image (or screen) is touched with a finger, it is not necessary to actually touch the image, and the display / light sensor unit 300 may detect the image of the finger. However, in the following description, for convenience of explanation, it is assumed that the image 20 is touched with the thumb from the upper surface side and the index finger from the lower surface side.

  In addition, although the image 20 has been described as being square, the image 20 may have any shape. Although the speed of movement of the thumb and index finger is not particularly mentioned, the movement associated with the movement of the image of the finger is executed according to the speed of movement of the finger.

(Explanation of main parts of data display / sensor device)
Next, a detailed configuration of the data display / sensor device 110 will be described with reference to FIG. Here, in order to make the description easy to understand, the description of the operations of the data processing unit 700 and the circuit control unit 600 located between the main control unit 800 and the display / light sensor unit 300 is omitted. However, to be precise, when performing image display and image detection, the input interface control unit 805 of the main control unit 800 transmits a command to the data processing unit 700, and the data processing unit 700 performs circuit control based on the command. The circuit control unit 600 transmits a signal to the display / light sensor unit 300. Further, the main control unit 800 acquires the entire image data, the partial image data, and the coordinate data from the data processing unit 700 as a response to the command transmitted to the data processing unit 700.

  FIG. 13 is a block diagram illustrating a main configuration of the data display / sensor device 110. As shown in the figure, the main control unit 800 includes an input interface control unit 805 and an application execution unit 850. The input interface control unit 805 includes an image display control unit 860, a detection determination unit 870, a first input generation unit 880a, a second input generation unit 880b, and a superimposition determination unit 890.

  The detection determination unit 870 determines whether the first display / light sensor unit 300A and the second display / light sensor unit 300B have both detected a finger image. Specifically, when the first display / light sensor unit 300A and the second display / light sensor unit 300B detect a finger image, coordinate data, whole image data, and partial image data are detected from the data processing unit 700. 870 is transmitted. Therefore, when the detection determination unit 870 receives coordinate data, whole image data, and partial image data from both the first display / light sensor unit 300A and the second display / light sensor unit 300B, the first display / light sensor It can be determined that both the unit 300A and the second display / light sensor unit 300B have detected a finger image. Note that the detection determination unit 870 receives at least one of the coordinate data, the entire image data, and the partial image data, so that the first display / light sensor unit 300A and the second display / light sensor unit 300B together. It can also be determined that a finger image has been detected.

  Here, in FIG. 13, the coordinate data, the entire image data, and the partial image data transmitted from the data processing unit 700 to the detection determination unit 870 (and the first input generation unit 880a and the superimposition determination unit 890) are indicated by solid lines and broken lines. Are listed. This is because the detection determination unit 870 acquires coordinate data, whole image data, and partial image data from both the first display / light sensor unit 300A and the second display / light sensor unit 300B. The coordinate data, the entire image data, and the partial image data transmitted from the optical sensor unit 300A to the detection determination unit 870 (and the first input generation unit 880a and the superimposition determination unit 890) are solid lines, and the second display / optical sensor unit 300B. The coordinate data, the whole image data, and the partial image data transmitted to the detection determination unit 870 (and the first input generation unit 880a and the superimposition determination unit 890) are indicated by broken lines, respectively.

  Next, the detection determination unit 870 outputs a determination result indicating that “the first display / light sensor unit 300A and the second display / light sensor unit 300B have both detected a finger image” to the first input generation unit 880a. To do. Accordingly, when only the first display / light sensor unit 300A or only the second display / light sensor unit 300B detects an image of a finger, or the first display / light sensor unit 300A and the second display / light sensor unit 300B When neither finger image is detected, the detection determination unit 870 indicates that “the first display / light sensor unit 300A and the second display / light sensor unit 300B have not detected the finger image”. Is output to the first input generation unit 880a. Alternatively, the detection determination unit 870 outputs nothing to the first input generation unit 880a.

  The superimposition determination unit 890 includes a display image region that is a region of an image displayed on the first display / light sensor unit 300A, and a finger detected by the first display / light sensor unit 300A and the second display / light sensor unit 300B. It is determined whether or not an image region that is an image region overlaps. In the display / light sensor unit 300, an image is displayed on the first display / light sensor unit 300A and no image is displayed on the second display / light sensor unit 300B.

  More specifically, the superimposition determination unit 890 acquires coordinate data, whole image data, and partial image data from the data processing unit 700, and specifies an image region from these coordinate data, whole image data, and partial image data. In addition, the superimposition determination unit 890 reads image information related to the image displayed on the first display / light sensor unit 300 </ b> A from the storage unit 901. In the image information, since the display image area related to the image is specified by coordinates, the superimposition determination unit 890 can specify the display image area by reading the image information from the storage unit 901.

  As an example, image information will be described with reference to FIGS. 14 and 15. FIG. 14 is a diagram for explaining the image information about the circular image 26. FIG. 15 is a diagram for explaining image information about the square image 20.

  As shown in FIG. 14, the circular image 26 is a circle having a diameter φT1 with the coordinates (A, B) as the center point. Here, the display image area related to the circular image 26 is specified by the coordinates (A, B) and the diameter φT1, but may be specified by the coordinates of each circumference of the circular image 26. I do not care. This is because the display image area related to the circular image 26 is specified in any manner.

  As shown in FIG. 15, the square image 20 is a square image whose one side is represented by a length C with the position (A, B1) as the center of the image. Here, the image 20 is specified by coordinates (A, B1) and a length C, but may be specified by coordinates indicating four sides of the image 20. This is because the display image area related to the image 20 is specified in any manner.

  In this way, the information of coordinates (A, B) and φT1 in FIG. 14 and the information of coordinates (A, B1) and side length C in FIG. 15 are handled as image information. The superimposition determination unit 890 can specify the display image region by reading out the image information from the storage unit 901.

  By the above method, the superimposition determination unit 890 can specify the image region and the display image region, and therefore determines whether or not there is a superimposition region that overlaps between the image region and the display image region. Can do. Then, the superimposition determination unit 890 outputs the determination result to the first input generation unit 880a. In addition, when it determines with the said superimposition area | region not existing, the superimposition determination part 890 does not output the determination result to the 1st input production | generation part 880a, and it is set as the determination result that a superimposition area | region does not exist by it. You can also.

  In the above description, the superimposition determination unit 890 has been described as acquiring coordinate data, whole image data, and partial image data from the data processing unit 700. However, the superimposition determination unit 890 may acquire coordinate data, whole image data, and partial image data from the data processing unit 700 via the detection determination unit 870.

  In determining whether or not there is a superimposition region, the superimposition determination unit 890 includes a display image region that is a region of an image displayed on the first display / light sensor unit 300A and the first display / light sensor unit 300A. Or you may determine whether the image area | region which is the area | region of the image of the finger | toe which any one detected among the 2nd display / light sensor part 300B overlaps. Thus, “there is a superimposed region”, compared to the case where it is determined whether or not the image region of both the first display / light sensor unit 300A and the second display / light sensor unit 300B overlap the display image region. Can be relaxed.

  Here, the reason why it is preferable to relax the condition for superimposition determination will be described using a specific example. In FIG. 12A, the user touches the area on the second display / light sensor unit 300B corresponding to the image 20 displayed on the first display / light sensor unit 300A with the index finger. That is, if the image 20 is displayed on the second display / light sensor unit 300B, the index finger touches the area where the image 20 will be displayed.

  At this time, since the second display / light sensor unit 300B does not display the image 20 on the screen, the user can use the second display / light sensor corresponding to the image 20 displayed on the first display / light sensor unit 300A. The area on the part 300B is sensibly recognized, and then the area is touched. That is, touching the area with the index finger is performed sensibly and not always accurately.

  Accordingly, an image region, which is a region of a finger image detected by at least one of the first display / light sensor unit 300A or the second display / light sensor unit 300B, is superimposed on the display image region. If it is determined that “exists”, the data display / sensor device 110 can provide the user with better operability and convenience.

  The first input generation unit 880a determines that the detection determination unit 870 has detected a finger image by both the first display / light sensor unit 300A and the second display / light sensor unit 300B, and the superimposition determination unit 890 When it is determined that the overlapping region exists, the movement of the finger image detected by the first display / light sensor unit 300A is detected, and a first input signal associated with the movement of the finger image is generated. The first input signal is output to the application execution unit 850. Note that the first input signal may be output to the application execution unit 850 as indicating the instruction associated with the identified XY coordinates based on the first finger-command correspondence information described in the first embodiment. Good.

  Further, the first input generation unit 880a may generate the first input signal in association with an operating state (scene) such as an application and a mode, or an image attribute. Thereby, the application execution unit 850 can execute a predetermined operation on the image displayed on the first display / light sensor unit 300A. Note that the first input generation unit 880a can acquire image information related to the image or the like directly from the storage unit 901 or via the superimposition determination unit 890. Accordingly, the first input generation unit 880a can generate a first input signal associated with the image displayed on the first display / light sensor unit 300A.

  Note that the detailed description of the first input signal has already been described in the first embodiment, and will not be repeated. In the above description, the data display / sensor device 110 includes the superimposition determination unit 890 that determines whether or not there is a superimposition region that overlaps between the image region and the display image region, and the first input generation unit 880a. Has been described as generating the first input signal when the overlapping region exists. However, the data display / sensor device 110 does not necessarily need to include the superimposition determination unit 890. In this case, it is not necessary for the user to touch the image displayed on the first display / light sensor unit 300A from both the front and back surfaces, and the first display / light sensor unit 300A and the second display / light sensor unit 300B both indicate the finger. It is only necessary to detect the image.

  When the second display / light sensor unit 300B determines that the second display / light sensor unit 300B no longer detects the finger image while the first display / light sensor unit 300A detects the finger image, the second input generation unit 880b Is received from the first input generation unit 880a or the detection determination unit 870. Then, by receiving the signal, the second input generation unit 880b generates a second input signal for reproducing the state before generating the first input signal. Then, the second input generation unit 880b transmits the second input signal to the application execution unit 850.

  The application execution unit 850 receives the first input signal from the first input generation unit 880a and the second input signal from the second input generation unit 880b. Then, the operation indicated by the first input signal and the second input signal is executed. Subsequently, the application execution unit 850 transmits operation result information, which is information indicating the result of the operation, to the image display control unit 860.

  Specifically, with reference to FIGS. 12A to 12C, the application execution unit 850 moves the image 20 by receiving the first input signal from the first input generation unit 880a. Execute. When the detection determination unit 870 determines that the second display / light sensor unit 300B no longer detects the finger image while the first display / light sensor unit 300A detects the finger image, The input generation unit 880b transmits a second input signal for reproducing a state before the first input signal is generated to the application execution unit 850. By receiving the second input signal, the application execution unit 850 moves the image 20 shown in FIG. 12B to the position shown in FIG. 12C, that is, the position before the moving operation is executed. Perform the revert action.

  As described above, the first input generation unit 880a outputs the first input signal and the second input generation unit 880b outputs the second input signal to the application execution unit 850, respectively. Then, the application execution unit 850 executes an operation based on the command indicated by the first input signal and / or the second input signal, and further, operation result information that is information indicating the result of the operation is sent to the image display control unit 860. Send. As a result, the image 20 is displayed at the position shown in FIG. 12C on the first display / light sensor unit 300A.

  In FIG. 13, the application execution unit 850 is included in the main control unit 800. However, the application execution unit 850 may be provided outside the main control unit 800, and a predetermined operation may be executed via the external communication unit 907. Further, the application execution unit 850 may have one or a plurality of applications.

  Upon receiving the operation result information from the application execution unit 850, the image display control unit 860 transmits display data to the data processing unit 700. The display data prompts the data processing unit 700 to display the operation result executed by the application execution unit 850 on the first display / light sensor unit 300A.

  Therefore, for example, when the operation of “moving the image from point A to point B” is executed by the application execution unit 850, the image display control unit 860 transmits display data indicating that to the data processing unit 700. As a result, the operation result is displayed on the first display / light sensor unit 300A. When the operation of “moving the image from point A to point B” is executed by the application execution unit 850, the image display control unit 860 transmits display data indicating that to the data processing unit 700, As a result, the operation result is displayed on the first display / light sensor unit 300A.

  Note that the detection determination unit 870, the first input generation unit 880a, and the superimposition determination unit 890 are, for each of the first display / light sensor unit 300A and the second display / light sensor unit 300B, the entire image data of the finger image, The data processing unit 700 is requested to transmit image data and coordinate data. This is because the application program executed by the main control unit 800 displays the entire image data, partial image data, and coordinate data of the finger image for each of the first display / light sensor unit 300A and the second display / light sensor unit 300B. This is done by controlling the data processing unit 700 to transmit.

  Further, in FIG. 13, the storage unit 901 is provided outside the main control unit 800, but may be configured inside the main control unit 800.

(Flowchart explanation)
Next, the processing flow of the data display / sensor device 110 and the details of the command transmitted from the input interface control unit 805 to the data processing unit 700 will be described with reference to FIG.

  FIG. 16 is a flowchart showing a process flow of the data display / sensor device 110.

  First, when the data display / sensor device 110 is turned off, the user presses the power switch 905 to turn on the data display / sensor device 110 (step S31).

  In step S <b> 33, when the user presses the power switch 905, an image is displayed on the display / light sensor unit 300. Here, it is assumed that an image is displayed on the first display / light sensor unit 300A. In this state, the data display / sensor device 110 stands by until the first display / light sensor unit 300A and the second display / light sensor unit 300B detect the image of the finger.

  Here, when displaying an image on the first display / light sensor unit 300A, a command specifying the upper screen (“001”) as the value of the “display panel” field is sent from the input interface control unit 805 to the data processing unit 700. Is output. When displaying an image on the second display / light sensor unit 300B, a command specifying the lower screen (“010”) as the value of the “display panel” field is sent from the input interface control unit 805 to the data processing unit 700. Is output. Here, in order to display an image on the first display / light sensor unit 300A, a command specifying the upper screen (“001”) as the value of the “display panel” field is sent from the input interface control unit 805 to the data processing unit 700. Is output.

  Further, at this stage, it is assumed that the display / light sensor unit 300 is in a state in which a nearby image can be detected. In order to perform the detection, “event” is specified in the “data acquisition timing” field (“01”), and both “coordinate” and “whole image” are specified in the “data type” field (“101” is specified). The designated command is output from the input interface control unit 805 to the data processing unit 700. Note that a command specifying only “coordinates” or “entire image” may be output from the input interface control unit 805 to the data processing unit 700.

  Further, in order to perform the detection in the first display / light sensor unit 300A and the second display / light sensor unit 300B, a command in which both the display / light sensor units 300 are specified (“011”) in the “scan panel” field. Is output from the input interface control unit 805 to the data processing unit 700.

  As a result, when the sensor built-in liquid crystal panel 301 detects a nearby image, the detection determination unit 870, the first input generation unit 880a, and the superimposition determination unit 890 receive the coordinate data, the entire image data, and the partial data from the data processing unit 700. Image data can be acquired.

  That is, when the user's finger touches the first display / light sensor unit 300A and the second display / light sensor unit 300B, the light sensor circuit 32 detects at least one of a shadow image and a reflection image of the touched finger. As a result, the image data acquired by the data processing unit 700 from the sensor control unit 602 changes.

  In this manner, the detection determination unit 870 obtains the coordinate data, the entire image data, and the partial image data that are output from the data processing unit 700 when the image data is changed, thereby performing the first display. It is possible to recognize that both the optical sensor unit 300A and the second display / optical sensor unit 300B have detected the finger image.

  In step S35, the detection determination unit 870 determines whether the first display / light sensor unit 300A and the second display / light sensor unit 300B have both detected a finger image. That is, the coordinate data, the entire image data, and the partial image data for each display / light sensor unit 300 are input from the data processing unit 700 to the detection determination unit 870, whereby the first display / light sensor unit 300A and the second display / light sensor unit 300A. It is determined that both of the optical sensor units 300B detect the finger image. In FIG. 16, for the sake of convenience, the first display / light sensor unit 300A is referred to as “A”, and the second display / light sensor unit 300B is referred to as “B”.

  When both the first display / light sensor unit 300A and the second display / light sensor unit 300B detect a finger image (Yes in step S35), the process proceeds to step S37. In step S37, the superimposition determining unit 890 includes a display image region that is a region of an image displayed on the first display / light sensor unit 300A, and the first display / light sensor unit 300A and the second display / light sensor unit 300B. It is determined whether or not the detected image area that is the image area of the finger is superimposed. That is, by acquiring the coordinate data, the entire image data, and the partial image data for each display / light sensor unit 300 from the data processing unit 700, the superimposition determination unit 890 can obtain the coordinate data, the entire image data, and the partial image data, A display image area which is an area of an image displayed on the first display / light sensor unit 300A is compared, and it is determined whether or not an overlapping area (superimposition area) exists. Note that the superimposition determination unit 890 reads image information related to the image displayed on the first display / light sensor unit 300 </ b> A from the storage unit 901.

  If it is determined that the overlapping area exists (Yes in step S37), the process proceeds to step S39. In step S39, the first input generation unit 880a identifies the movement of the finger image detected by the first display / light sensor unit 300A and generates a first input signal associated with the movement of the finger image. The input signal is output to the application execution unit 850.

  Subsequently, in step S41, the application execution unit 850 receives the first input signal from the first input generation unit 880a, and executes the operation indicated by the command. Then, the application execution unit 850 transmits operation result information that is information indicating a result of the operation to the image display control unit 860.

  Subsequently, in step S43, the detection determination unit 870 determines that the second display / light sensor unit 300B no longer detects the finger image while the first display / light sensor unit 300A detects the finger image. In this case, the second input generation unit 880b generates a second input signal for reproducing the state before the first input signal is generated. Then, the second input generation unit 880b outputs the second input signal to the application execution unit 850.

  In step S45, even if any operation is performed as a result of the generation of the first input signal, the second display / light sensor unit 300B stops detecting the image of the finger so far. The executed action is canceled. Then, the application execution unit 850 transmits operation result information that is information indicating a result of the operation to the image display control unit 860.

  Here, the image display control unit 860 receives operation result information from the application execution unit 850 and transmits display data to the data processing unit 700. The display data prompts the data processing unit 700 to display the operation result executed by the application execution unit 850 on the first display / light sensor unit 300A. As a result, the first display / light sensor unit 300A displays the operation result executed by the application execution unit 850 based on the commands received from the first input generation unit 880a and the second input generation unit 880b.

  When an image is displayed on the first display / light sensor unit 300A, a command specifying the upper screen (“001”) as the value of the “display panel” field is sent from the input interface control unit 805 to the data processing unit 700. Is output. When displaying an image on the second display / light sensor unit 300B, a command specifying the lower screen (“010”) as the value of the “display panel” field is sent from the input interface control unit 805 to the data processing unit 700. Is output. Here, in order to display an image on the first display / light sensor unit 300A, a command specifying the upper screen (“001”) as the value of the “display panel” field is sent from the input interface control unit 805 to the data processing unit 700. Is output.

  On the other hand, when it progresses to "No" by step S35 and step S37, it returns to step S35 again. Although the above description includes step S37, the step is not essential and may be omitted as appropriate.

  Here, the process flow of each step (step S33 to step S45) in the case of FIG. 12 will be described.

  In FIG. 12A, the image 20 is displayed on the first display / light sensor unit 300A. This is because the command specifying the upper screen (“001”) as the value of the “display panel” field is output from the input interface control unit 805 to the data processing unit 700, and as a result, the image 20 is displayed on the first display / light sensor unit 300A. Is displayed (step S33).

  At this stage, “Event” is specified in the “Data acquisition timing” field (“01”), and both “Coordinate” and “Entire image” are specified in the “Data type” field (“101” is set). The designated command is output from the input interface control unit 805 to the data processing unit 700, and the display / light sensor unit 300 is in a state in which a nearby image can be detected. Note that a command specifying only “coordinates” or “entire image” may be output from the input interface control unit 805 to the data processing unit 700.

  In addition, in order to perform the detection in the first display / light sensor unit 300A and the second display / light sensor unit 300B, a command in which both display / light sensor units 300 are specified (“011”) in the “scan panel” field. Is output from the input interface control unit 805 to the data processing unit 700.

  In such a state, the user can use the thumb from the upper surface (the side on which the first display / light sensor unit 300A is disposed) and the index finger from the lower surface (the side on which the second display / light sensor unit 300B is disposed). , Touch the image 20 respectively. Since both the first display / light sensor unit 300A and the second display / light sensor unit 300B can detect the image of the finger, the detection determination unit 870 includes the first display / light sensor unit 300A and the first display / light sensor unit 300A. The second display / light sensor unit 300B determines that both are detecting a finger image, and outputs the determination result to the first input generation unit 880a (step S35).

  Also, as shown in FIG. 12 (a), a display image area indicating the area of the image 20 displayed on the first display / light sensor unit 300A, the first display / light sensor unit 300A, and the second display / light sensor. The image area which is the image area of the thumb and index finger detected by the unit 300B is superimposed. Therefore, the superimposition determination unit 890 determines that there is a superimposition region where the display image region and the image region overlap, and outputs the determination result to the first input generation unit 880a (step S37).

  Note that the superimposition determination unit 890 acquires coordinate data, whole image data, and partial image data of a finger image from the data processing unit 700, and specifies the image region from these data. In addition, the superimposition determination unit 890 identifies the display image area of the image 20 by reading out the image information related to the image displayed on the first display / light sensor unit 300A from the storage unit 901. In this way, since the superimposition determination unit 890 can specify the image area and the display image area, it can determine whether or not there is a superimposition area that overlaps between the image area and the display image area. it can.

  In the first input generation unit 880a, the detection determination unit 870 determines that both the first display / light sensor unit 300A and the second display / light sensor unit 300B detect the image of the finger, and the superimposition determination unit 890 superimposes. In response to the determination that the region exists, the movement of the finger image detected by the first display / light sensor unit 300A is specified.

  Here, in FIG. 12B, since the thumb (and the index finger) is moving in the right direction in the drawing, the first input generation unit 880a outputs the first input signal associated with the movement of the image of the finger. Generate. In addition, the first input generation unit 880a acquires image information related to the image 20 directly from the storage unit 901 or via the superimposition determination unit 890, and the first input signal is generated in association with the image 20. ing. Then, the first input generation unit 880a outputs the input signal to the application execution unit 850 (step S39).

  The generation of the first input signal in association with the image 20 means that when the application is a map, the map also moves in the direction in which the thumb moves. Therefore, in an application other than a map, different operation results may be output depending on the operation of moving the thumb. Thus, the first input signal may be generated in association with an image displayed by the application.

  The application execution unit 850 receives the first input signal from the first input generation unit 880a and executes the operation indicated by the command. That is, the application execution unit 850 executes an operation for moving the image 20 in the right direction of the drawing. In addition, the application execution unit 850 transmits operation result information that is information indicating a result of the operation to the image display control unit 860.

  Upon receiving the operation result information from the application execution unit 850, the image display control unit 860 transmits display data to the data processing unit 700. The display data is data instructing to move and display the image 20 from the position of FIG. 12A to the position of FIG. Then, based on the display data transmitted from the image display control unit 860, the data processing unit 700 moves the image 20 displayed on the first display / light sensor unit 300A to the position shown in FIG. Display.

  It should be noted that the image 20 after moving is displayed as a result of a command specifying the upper screen (“001”) as the value of the “display panel” field being output from the input interface control unit 805 to the data processing unit 700. / The image 20 after movement is displayed on the optical sensor unit 300A.

  Subsequently, in FIG. 12C, the detection / determination unit 870 stops detecting the finger image in the second display / light sensor unit 300 </ b> B in a state where the first display / light sensor unit 300 </ b> A detects the finger image. The second input generation unit 880b generates a second input signal for reproducing the state before the first input signal is generated. Then, the second input generation unit 880b outputs the second input signal to the application execution unit 850.

  In step S45, the second display / light sensor unit 300B no longer detects the finger image, so that the operation of moving the image 20 to the position shown in FIG. 12B is cancelled. Then, the application execution unit 850 transmits operation result information of the operation of returning the image 20 to the position illustrated in FIG. 12C to the image display control unit 860.

  Since the subsequent operation of the image display control unit 860 is as described above, description thereof is omitted here.

  In this way, the operation described with reference to FIGS. 12A to 12C is executed, and the execution result is displayed on the first display / light sensor unit 300A.

  Further, here, the display image is displayed on the first display / light sensor unit 300A, but the image may be displayed on the second display / light sensor unit 300B.

  Furthermore, although it has been described that the thumb and index finger are used as detection targets by the first display / light sensor unit 300A and the second display / light sensor unit 300B, the types of fingers to be detected are limited to these fingers. The middle finger, ring finger, and little finger may be used. This is because a predetermined operation is performed on the image displayed on the first display / light sensor unit 300A based on the detected movement of the finger image instead of the type of the finger.

  The effects obtained by the processing of the data display / sensor device 110 will be described as follows.

  The data display / sensor device 110 has the above-described configuration, so that when the first display / light sensor unit 300A and the second display / light sensor unit 300B both detect a finger image, the first input generation unit 880a The first input signal associated with the movement of the finger image is generated. When the second display / light sensor unit 300B stops detecting the finger image while the first display / light sensor unit 300A detects the finger image, the second input generation unit 880b A second input signal for reproducing the state before the one input signal is generated is generated.

  Therefore, even if some operation is performed as a result of the generation of the first input signal, the second display / light sensor unit 300B is in a state where it does not detect the image of the finger, and thus has been performed so far. Canceled operation is canceled. In other words, conventionally, operations such as entering a certain function (mode) to execute a cancel operation and entering the edit mode to cancel the executed operation are necessary. However, in the data display / sensor device 110, When the second display / light sensor unit 300B does not detect the image of the finger, the operation executed so far is easily canceled.

  The operation that can be executed by the data display / sensor device 110 is not limited to “cancel (invalid)”. For example, when the second display / light sensor unit 300B detects the movement of the input operation, Other operations such as “conversion” may be executed. Hereinafter, the modification is demonstrated.

  In the above description, the first input generation unit 880a determines that the detection determination unit 870 has detected the image of the finger by both the first display / light sensor unit 300A and the second display / light sensor unit 300B, and superimposes. When the determination unit 890 determines that there is an overlapping area, the first display / light sensor unit 300A detects the movement of the finger image detected by the first display / light sensor unit 300A, and the first input signal associated with the movement of the finger image And the first input signal is output to the application execution unit 850.

  However, the first input generation unit 880a detects the movement of the finger image detected by the first display / light sensor unit 300A, regardless of the detection result by the detection determination unit 870 and the determination result by the superimposition determination unit 890. The first input signal associated with the movement of the finger image may be generated, and the first input signal may be output to the application execution unit 850.

  In this case, the first input generation unit 880a generates a first input signal associated with the movement of the finger and outputs the first input signal to the application execution unit 850. Then, the application execution unit 850 executes a predetermined operation associated with the first input signal.

  In this state, the second display / light sensor unit 300B detects the movement of the finger image. Thus, the second input generation unit 880b generates a second input signal that instructs to “confirm” the predetermined operation executed by the application execution unit 850. Alternatively, for example, when the predetermined operation executed by the application execution unit 850 is “character input”, the second input generation unit 880b instructs to “convert” the input character into another kanji. A second input signal is generated.

  In this way, if the second display / light sensor unit 300B is assigned to execute operations such as “confirm” and “conversion” by detecting the movement of the input operation, the user executes the above-described operations. If desired, the second display / light sensor unit 300B may detect the movement of the input operation. Thereby, each operation of the application desired by the user is executed. Note that the operation examples are not limited to “determined” and “conversion”, and various operations are possible.

  Further, the movement of the input operation that should be detected by the second display / light sensor unit 300B is not particularly limited. For example, even if the movement of the input operation changes from non-detection to detection, or when the movement of the finger image goes in a specific direction, an operation such as “confirm” or “conversion” may be executed. Good.

  As described above, by allocating the movement of the input operation to the second display / light sensor unit 300B in advance, the user can easily execute the operation of the desired application. In addition, since the data display / sensor device 110 has a plate shape, the user can perform the above-described operation so that both the front and back surfaces are sandwiched between fingers.

  Note that the user can select whether the data display / sensor device 110 is a device that performs a “cancel (invalid)” operation or a device that performs other operations such as “confirm” and “conversion”. You can do it. For example, by providing an execution operation selection button or the like and allowing the user to select one, the single data display / sensor device 110 can be used for various purposes.

  Thus, the data display / sensor device 110 can provide the user with excellent operability and convenience.

  The invention is not limited to the above-described embodiments, and various modifications can be made within the scope shown in the claims, and the embodiments can be obtained by appropriately combining technical means disclosed in different embodiments. The form is also included in the technical scope of the present invention.

An information terminal device such as a mobile phone provided with the input device according to the present invention is also included in the technical scope of the present invention.
(Other)
The input device according to the present invention may have the following configuration.

  Specifically, the input device according to the present invention is a plate-like input device in which the planar members for detecting an input operation are respectively disposed on both the front and back surfaces, and the surface planar member has an X axis, A Y-axis is set, a first input detection means for specifying a movement of an input operation detected by the planar member on the front surface by XY coordinates and generating a first input signal corresponding to the XY coordinates; And a second input detection unit that generates a second input signal corresponding to the movement of the input operation detected by the planar member.

  Thus, in the input device and the input device control method according to the present invention, when the surface planar member detects the input operation, the first input signal corresponding to the movement of the input operation specified by the XY coordinates. Are detected by the first input detecting means, the second input signal corresponding to the movement of the input operation is respectively generated by the second input detecting means.

  Accordingly, by detecting the movements of various input operations by the surface members on both the front and back surfaces, various operations that could not be realized by the conventional input device, that is, operations for realizing movements in the XY axes, and the second input A first input signal and a second input signal are generated that enable various operations fused with other operations realized based on the signals. Then, the operation realized based on the first input signal and the second input signal provides the user with an input device having excellent convenience and operability. In addition, since the operation is performed so that the screen is sandwiched between the front and back surfaces of the plate-like input device with a finger, the user can use the device very easily.

  In the input device according to the present invention, in the configuration described above, the second input detection unit defines a three-dimensional motion of the input operation detected by the planar member on the back surface together with the X axis and the Y axis. The second input signal corresponding to the movement amount and the movement direction on the specified Z axis may be generated by specifying the movement amount and the movement direction on the Z axis.

  Thereby, in the input device according to the present invention, when the planar member on the back surface detects the input operation, the second input corresponding to the movement of the input operation specified as the movement amount and the movement direction on the Z axis. A signal is generated by the second input detection means.

  Therefore, when the planar members on both the front and back surfaces detect various input operations, the first input signal and the second input signal that realize various three-dimensional operations that cannot be realized by the conventional input device are generated. The operation realized based on the first input signal and the second input signal provides the user with an input device having excellent convenience and operability.

  In addition, the surface planar member corresponds to a first input signal that is specified in two dimensions, XY coordinates, and the amount of movement on the Z axis that defines the three-dimensional together with the X and Y axes. And the second input signal specified by the moving direction. Therefore, when an application that executes a three-dimensional operation defined by the XYZ axes is operated by the above input signals, the user imagines a three-dimensional operation realized by the user's own operation, and then performs both front and back surfaces. It is possible to detect a predetermined input operation that causes the planar member to cause the three-dimensional motion, and to experience a sensuously excellent operation.

  Furthermore, in the input device according to the present invention, in the configuration described above, the first input detection unit is an operation execution device in which an application for executing a three-dimensional operation defined by the specified XY coordinates and XYZ axes operates. Based on the first finger-command correspondence information associated with the command for executing the two-dimensional motion defined by the XY axes, and determining the command associated with the specified XY coordinates, The first input signal is transmitted to the operation execution device, and the second input detection unit causes the operation execution device to execute an operation in the Z-axis direction and the movement amount and movement direction on the specified Z-axis. Based on the second finger-command correspondence information associated with the command, a command associated with the identified movement amount and movement direction on the Z-axis is determined, and the second input signal indicating the command is determined. Said operation It may be transmitted device.

  According to the above configuration, in the input device according to the present invention, the first input detection means outputs the first input signal indicating a command for causing the operation execution device to execute a two-dimensional operation, and the second input detection means operates as an operation. A first input signal indicating a command for causing the execution device to execute an operation in the Z-axis direction is transmitted to the operation execution device. In the operation execution apparatus, an application that executes a three-dimensional operation defined by the XYZ axes operates.

  Accordingly, the first and second input signals that realize various three-dimensional operations that could not be realized by the conventional input device are generated by detecting movements of various input operations by the planar members on both the front and back surfaces. Therefore, the application can make the user experience a sensory operation.

  Note that there may be one or a plurality of operation execution devices, the operation execution devices may be integrated with the input device, or may be present at a position physically separated from the input device. .

  Furthermore, in the input device according to the present invention, in the configuration described above, when an application for displaying a map is operating in the operation execution device, the first input detection unit moves the map or a pointer on the map. The second input detection means transmits the second input signal indicating the command for enlarging / reducing the map to the operation executing device, respectively, the first input signal indicating the command for causing the map to be enlarged / reduced Also good.

  By providing the above configuration, the map or the pointer on the map moves according to the movement of the input operation detected by the surface planar member, and according to the movement of the input operation detected by the back surface planar member, The map is enlarged or reduced.

  Therefore, the input operation on the front side moves the map up, down, left and right, and the movement of the input operation on the back side enlarges / reduces the map. Is realized.

  Furthermore, in the input device according to the present invention, in the configuration described above, the image display control unit may display an image based on a result of an operation executed by the operation execution device on the planar member.

  According to the above configuration, in the input device according to the present invention, the image display control means causes the planar member to display an image based on the result of the operation executed by the operation execution device.

  Therefore, the image display control means can display the result of the operation executed in association with the movement of the input operation from the user on the screen and allow the user to confirm the result.

  In addition, when it is desired to give some operation to the image after the operation, it is only necessary to detect the movement of a predetermined input operation on the surface members on both the front and back surfaces. That is, the user can continuously perform the next operation by checking the operation result on the screen.

  The input device may be realized by a computer. In this case, an input device control program for causing the input device to be realized by a computer by causing the computer to operate as the respective means, and the program recorded therein. Computer-readable recording media are also within the scope of the present invention. An information terminal device provided with the input device also falls within the scope of the present invention.

(Program, computer-readable recording medium)
Finally, each block of the data display / sensor devices 100 and 110 according to the present embodiment, in particular, the input detection unit 810 of the input interface control unit 805, the application execution unit 850, the image display control unit 860, the detection determination unit 870, the first The input generation unit 880a, the second input generation unit 880b, and the superimposition determination unit 890 may be configured by hardware logic, or may be realized by software using a CPU as follows.

  That is, the data display / sensor devices 100 and 110 according to the present embodiment include a CPU (Central Processing Unit) that executes instructions of a control program that realizes each function, a ROM (Read Only Memory) that stores the program, and the program. And a storage device (recording medium) such as a memory for storing the program and various data. The object of the present invention is to record the program code (execution format program, intermediate code program, source program) of the control program of the data display / sensor devices 100, 110, which is software that realizes the above-described functions, in a computer-readable manner. This can also be achieved by supplying the recorded medium to the data display / sensor device 100 and reading and executing the program code recorded on the recording medium by the computer (or CPU or MPU).

  Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy (registered trademark) disks / hard disks, and disks including optical disks such as CD-ROM / MO / MD / DVD / CD-R. Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.

  The data display / sensor devices 100 and 110 according to the present embodiment may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Also, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope shown in the claims. That is, embodiments obtained by combining technical means appropriately modified within the scope of the claims are also included in the technical scope of the present invention.

  The present invention can be applied to mobile devices (portable terminals) such as mobile phones and PDAs that input from two input surfaces. In particular, it is suitable for a portable terminal equipped with an image detection device.

It is a block diagram showing the principal part structure of the data display / sensor apparatus which concerns on one Embodiment of this invention. It is a figure which shows typically the cross section of the liquid crystal panel with a built-in sensor with which the data display / sensor apparatus which concerns on one Embodiment of this invention is provided. (A) is a schematic diagram which shows a mode that the position which the user touched is detected by detecting a reflected image with the sensor built-in liquid crystal panel with which the data display / sensor apparatus which concerns on one Embodiment of this invention is provided. (B) is a schematic diagram which shows a mode that the position which the user touched is detected by detecting a shadow image with the sensor built-in liquid crystal panel with which the data display / sensor apparatus which concerns on one Embodiment of this invention is provided. It is a block diagram which shows the principal part structure of the data display / sensor apparatus which concerns on one Embodiment of this invention. It is a figure which shows typically an example of the frame structure of the command used with the data display / sensor apparatus which concerns on one Embodiment of this invention. It is a figure explaining an example of the value which can be specified to each field contained in the command shown in FIG. 5, and its outline. (A) is an image obtained as a result of scanning the entire liquid crystal panel with built-in sensor when the object is not placed on the liquid crystal panel with built-in sensor in the data display / sensor device according to one embodiment of the present invention. It is data. (B) is image data obtained as a result of scanning when the user touches the sensor-equipped liquid crystal panel with a finger in the data display / sensor device according to the embodiment of the present invention. It is a block diagram which shows the structure of the liquid crystal panel with a sensor with which the data display / sensor apparatus which concerns on one Embodiment of this invention is provided, and the structure of its peripheral circuit. (A)-(g) is a figure which shows a mode when the bird's-eye view photo which image | photographed the certain area from the sky is displayed on the display / light sensor part. It is a figure which shows an example of the finger-command corresponding information in case an application is a map, (a) is a table which shows 1st finger-command corresponding information, (b) is the 2nd finger-command corresponding information. It is a flowchart showing the flow of a process of the data display / sensor apparatus which concerns on one Embodiment of this invention. In (a), an image is displayed on the first display / light sensor unit, and the image is touched with the thumb from the top and the index finger from the bottom, and in (b), the user holds the thumb and index finger. The image is moved to the outside of the screen by moving in the same direction (rightward in the horizontal direction of the drawing). In (c), the user moves the index finger away from the second display / light sensor unit, and It is a figure which shows a mode that the image has automatically returned to the first position shown by 12 (a). It is a block diagram showing the principal part structure of the other data display / sensor apparatus which concerns on embodiment of this invention. It is a figure for demonstrating the image information about a circular image. It is a figure for demonstrating the image information about a square image. It is a flowchart showing the flow of a process of the data display / sensor apparatus which concerns on other embodiment of this invention.

100, 110 Data display / sensor device (input device)
800 Main control unit 805 Input interface control unit 810 Input detection unit 810a First input detection unit (first input detection means)
810b Second input detection unit (second input detection means)
850 application execution unit (operation execution device)
860 Image display control unit (image display control means)
870 detection determination unit (detection determination means)
880a 1st input production | generation part (1st input production | generation means)
880b Second input generation unit (second input generation means)
890 Superimposition determination unit

Claims (6)

The planar members for detecting the input operation are plate-like input devices respectively disposed on the front and back surfaces,
A first input corresponding to the movement of the specified input operation is identified and associated with the movement in the first direction of the application that executes the predetermined movement by identifying the movement of the input operation detected by the surface planar member. First input detecting means for generating a signal;
The movement of the input operation detected by the sheet member on the back surface is specified, and the specified input in the sheet member on the back surface is associated with the movement of the application in the second direction different from the first direction. An input device comprising: second input detection means for generating a second input signal corresponding to the movement of the operation. An X axis and a Y axis are set on the surface planar member, and the first input detection means detects the movement of the input operation detected by the surface planar member on the XY coordinates and the movement direction. And generating a first input signal corresponding to the movement amount and movement direction on the specified XY coordinates,
The second input detection means specifies the movement of the input operation detected by the planar member on the back surface as a movement amount and a movement direction on the Z axis that defines three dimensions together with the X axis and the Y axis. The input device according to claim 1, wherein the second input signal corresponding to the movement amount and movement direction on the Z axis is generated. The first input detection means is defined by an XY axis in an operation execution device that operates the application that executes a three-dimensional operation defined by the specified movement amount and movement direction on the XY coordinates and the XYZ axes. Based on the first instruction correspondence information associated with the instruction for executing the operation in the first direction, the instruction associated with the movement amount and the movement direction on the specified XY coordinates is determined, and the instruction is Transmitting the first input signal indicating to the operation executing device;
In the second input detection means, the movement amount and movement direction on the specified Z axis are associated with a command for causing the operation execution device to execute an operation in the second direction defined by the Z axis direction. Determining a command associated with the identified movement amount and movement direction on the Z-axis based on the second command correspondence information, and transmitting the second input signal indicating the command to the operation execution device; The input device according to claim 2, wherein Image display control means for displaying an image on at least one of the surface members on both the front and back surfaces,
The input device according to claim 3, wherein the first instruction correspondence information and the second instruction correspondence information are associated with the image. The planar members for detecting the input operation are plate-like input devices respectively disposed on the front and back surfaces,
A first input signal corresponding to the identified input operation movement is associated with the first movement of the application that performs the predetermined movement by identifying the movement of the input operation detected by the surface planar member. First input detecting means for generating;
Accompanying the first operation of the application by identifying the movement of the input operation detected by the sheet member on the back surface or identifying the movement of the input operation from detection to non-detection for the sheet member on the back surface And second input detection means for generating a second input signal corresponding to the movement of the specified input operation in the sheet member on the back surface associated with the second movement. apparatus. A planar member for detecting an input operation is a control method for a plate-like input device arranged on both the front and back surfaces,
A first input corresponding to the movement of the specified input operation is identified and associated with the movement in the first direction of the application that executes the predetermined movement by identifying the movement of the input operation detected by the surface planar member. A first input detection step for generating a signal;
The movement of the input operation detected by the sheet member on the back surface is specified, and the specified input in the sheet member on the back surface is associated with the movement of the application in the second direction different from the first direction. And a second input detection step of generating a second input signal corresponding to the movement of the operation.