JP2010160689A - Input device and method of controlling the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an input device with high convenience and operability for a user. <P>SOLUTION: A data display/sensing device 100 includes: a liquid crystal panel 301 with a built-in sensor which can detect a nearby image, while displaying a part of a map; a scroll control section 840 which changes a position of the map responding to move of the detected image; a localization section 830 which specifies a position of the image which the liquid crystal panel 301 with a built-in sensor has detected as the position on the map; an area determination section which selects whether the scroll control section 840 changes the position of the map according to the area of the detected image or the localization section 830 pinpoints the position on the map. Thereby the data display/sensing device 100 distinguishes the operation of the scroll control section 840 to change the position of the map and the operation of the localization section 830 which pinpoints the position of the image which the liquid crystal panel 301 with a built-in sensor has detected as the position on the map, and performs the operations. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an input device capable of simultaneously displaying an image and reading an object, and a method for controlling the input device.

  2. Description of the Related Art In recent years, touch panel type input devices are widely used in mobile devices (portable terminals) represented by mobile phones, PDAs and the like. As an example, an input device that performs input according to the position of a touched finger when the input surface (operation surface) is changed from a state where the finger is not in contact to a state where the finger is in contact is known. It has been.

  Patent Document 1 discloses a vehicle navigation device that determines that a single pen operation has ended by releasing a pen from a screen and scrolls a map displayed on the screen. Is disclosed.

In Patent Document 2, the travel route from the current position on the map to the destination can be set by tracing the touch panel with a finger, and the map displayed on the screen according to the movement vector of the finger on the touch panel A route guidance device for an automobile to be scrolled is disclosed.
JP-A-9-101171 (published on April 15, 1997) Japanese Patent Laid-Open No. 7-91974 (published April 7, 1995)

  However, in the conventional configuration, the operation of moving an image and the operation of inputting to the image cannot be performed independently.

  For example, in the vehicle navigation device described in Patent Document 1, the map displayed on the screen does not scroll unless a predetermined pen input operation is finished and the pen is released from the screen. Accordingly, the vehicular navigation device cannot meet the user's request to perform an input operation while scrolling the screen.

  Further, in the automobile route guidance device described in Patent Document 2, the map does not scroll unless the finger on the touch panel moves. Therefore, the route guidance device of the automobile cannot respond to a user's request to perform an input operation after scrolling only the map and sufficiently grasping the surrounding situation.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to realize an input device that is convenient and easy to operate for a user, and a method for controlling the input device.

  In order to solve the above problems, the input device of the present invention is an input device that acquires a user input by detecting a nearby image, displays a sub-image cut out from one image, A planar member capable of detecting an image, and position changing means for changing the position of the sub-image on the image displayed on the planar member in accordance with the movement of the image detected by the planar member. Position specifying means for specifying the position of the image detected by the planar member as a position on the image, and further calculating the area of the image detected by the planar member, and the area of the image Is determined to be greater than a predetermined threshold, and the position changing unit changes the position of the sub-image or the position specifying unit specifies the position on the image according to the determination result. Or a selection means to select It is characterized in that.

  The input device control method according to the present invention includes a planar member that displays a sub-image cut out from one image and can detect a user's input by detecting a nearby image. A position changing step for changing a position of a sub-image on the image to be displayed on the planar member according to a movement of the image detected by the planar member, and an image detected by the planar member. A position specifying step of specifying the position of the image as a position on the image, and further calculating an area of the image detected by the planar member and determining whether the area of the image is larger than a predetermined threshold value. And a selection step of selecting whether the position changing step changes the position of the sub-image or whether the position specifying step specifies a position on the image according to the determination result. Specially It is set to.

  Thereby, in the input device and the control method of the input device according to the present invention, the planar member detects a nearby image, calculates the area of the detected image, and the area of the image is larger than a predetermined threshold value. Two operations, that is, an operation in which the position changing unit changes the position of the sub-image, and a position specifying unit determines the position of the image detected by the planar member according to the determination result. The operation specified as the position on the image can be performed separately. Therefore, even when the planar member detects two images having different areas, the two operations can be performed independently. That is, the input device and the method for controlling the input device according to the present invention do not need to execute the other operation after waiting for one of the two operations to end, and therefore, without being interrupted. And various operations can be provided to the user. Since this operation cannot be realized by a conventional input device, the user is provided with excellent operability and convenience.

  In addition, when it is desired to give some operation to the image after the operation, the user can cause the planar member to detect the desired image movement while considering the area of the image detected by the planar member. That's fine. Thereby, the next operation is continuously executed.

  Thus, the input device and the control method for the input device according to the present invention can provide the user with excellent operability and convenience.

  Furthermore, in the input device according to the present invention, in the configuration described above, the selection unit responds to the movement of the image detected by the planar member when determining that the area of the image is larger than the predetermined threshold. In this case, the position on the image may be specified when the position of the sub-image is changed and it is determined that the area of the image is equal to or smaller than the predetermined threshold.

  According to the above configuration, in the input device according to the present invention, the selection unit can determine whether or not the area of the image detected by the planar member is larger than the predetermined threshold value.

  Therefore, the input device according to the present invention compares the size of the image area with a predetermined threshold value, and easily and quickly detects two images after the planar member detects the image according to the image area. The operation, that is, the operation in which the position changing unit changes the position of the sub-image and the operation in which the position specifying unit specifies the position of the image detected by the planar member as the position on the image are performed separately. it can.

  In addition, since the input device according to the present invention can change the determination result by the selection means by changing the predetermined threshold value, a highly convenient input device can be provided to the user.

  Furthermore, in the input device according to the present invention, in the configuration described above, the input device includes an image display control unit that superimposes and displays a position image indicating the position on the image specified by the position specifying unit on the image. Also good.

  According to the above configuration, in the input device according to the present invention, the image display control means can superimpose and display a position image indicating the position on the image specified by the position specifying means on the image.

  Therefore, the image display control means specifies the operation in which the position changing means has changed the position of the sub-image, and the position specifying means specifies the position of the image detected by the planar member as the position on the image, and the specified image Since the operation of displaying the position image indicating the upper position on the image is displayed in a superimposed manner, the operation result can be easily and quickly confirmed by the user.

  Therefore, the user can continuously perform the next operation by visually recognizing the operation result displayed on the planar member.

  Furthermore, in the input device according to the present invention, in the above configuration, the image is a map, and a distance on the map corresponding to a length of a locus drawn by the position image based on a scale of the map A configuration may be provided that includes a distance calculating means for calculating.

  According to the above configuration, in the input device according to the present invention, the distance calculating unit can calculate the distance on the map corresponding to the length of the locus drawn by the position image.

  Therefore, by using the distance on the map calculated by the distance calculation means, various processes utilizing the data can be performed. For example, when the input device according to the present invention is used in an automobile navigation system, it is possible to perform a process of comparing the distances for each route or a process for calculating a moving distance for each category such as year, month, and day It becomes.

  Thus, in the input device according to the present invention, the distance calculation means can calculate the distance on the map corresponding to the length of the locus of the image detected by the planar member drawn by the position image. Therefore, excellent operability and convenience can be provided to the user.

  Furthermore, in the input device according to the present invention, in the above configuration, the image display control unit may display the distance on the map calculated by the distance calculation unit on the planar member.

  As a result, the input device according to the present invention allows the user to visually recognize the distance on the map accurately and promptly, and as a result, can inform the user of information such as the length of the distance for each route or the shortest route. .

  Therefore, the input device according to the present invention can accurately and quickly inform the user of the distance on the map, which can only be measured sensuously with the conventional input device, so the distance on the map can be accurately determined. It can respond to the user's desire to know.

  As described above, the input device according to the present invention is an input device that acquires a user input by detecting a nearby image, displays a sub-image cut out from one image, and displays a nearby image. A planar member that can be detected, and a position changing unit that changes the position of the sub-image on the image to be displayed on the planar member according to the movement of the image detected by the planar member; Position specifying means for specifying the position of the image detected by the planar member as the position on the image, and further calculating the area of the image detected by the planar member so that the area of the image is predetermined. Whether the position changing unit changes the position of the sub-image, or the position specifying unit specifies the position on the image, according to the determination result, With a selection means for selecting That.

  In addition, as described above, the control method of the input device according to the present invention displays a sub-image cut out from one image and a planar member that can acquire user input by detecting a nearby image. A position changing step of changing a position of a sub-image on the image to be displayed on the planar member according to a movement of an image detected by the planar member; A position specifying step of specifying the position of the image detected by the planar member as a position on the image, and further calculating the area of the image detected by the planar member so that the area of the image is a predetermined value. It is determined whether or not it is larger than a threshold value, and whether the position changing step changes the position of the sub-image or whether the position specifying step specifies a position on the image according to the determination result. Selection step to select A configuration that includes a flop.

  Thereby, the input device excellent in operability and convenience and the control method of the input device can be provided to the user.

It is a block diagram showing the principal part structure of the data display / sensor apparatus which concerns on one Embodiment of this invention. It is a figure which shows typically the cross section of the liquid crystal panel with a built-in sensor with which the data display / sensor apparatus which concerns on one Embodiment of this invention is provided. (A) is a schematic diagram which shows a mode that the position which the user touched is detected by detecting a reflected image with the sensor built-in liquid crystal panel with which the data display / sensor apparatus which concerns on one Embodiment of this invention is provided. (B) is a schematic diagram which shows a mode that the position which the user touched is detected by detecting a shadow image with the sensor built-in liquid crystal panel with which the data display / sensor apparatus which concerns on one Embodiment of this invention is provided. It is a block diagram which shows the principal part structure of the data display / sensor apparatus which concerns on one Embodiment of this invention. It is a figure which shows typically an example of the frame structure of the command used with the data display / sensor apparatus which concerns on one Embodiment of this invention. It is a figure explaining an example of the value which can be specified to each field contained in the command shown in FIG. 5, and its outline. (A) is an image obtained as a result of scanning the entire liquid crystal panel with built-in sensor when the object is not placed on the liquid crystal panel with built-in sensor in the data display / sensor device according to one embodiment of the present invention. It is data. (B) is image data obtained as a result of scanning when the user touches the sensor-equipped liquid crystal panel with a finger in the data display / sensor device according to the embodiment of the present invention. It is a block diagram which shows the structure of the liquid crystal panel with a sensor with which the data display / sensor apparatus which concerns on one Embodiment of this invention is provided, and the structure of its peripheral circuit. It is a figure which shows a mode that a finger | toe is moved to the right side since the trace of the path | route by a stylus pen reached the left end of the screen. It is a figure which shows a mode that a map scrolls by operation | movement of the finger | toe by FIG. 9, and traces a path | route again using a stylus pen. It is a flowchart showing the flow of a process of the data display / sensor apparatus which concerns on one Embodiment of this invention. It is a figure which shows the Example which concerns on other embodiment of this invention. It is a figure which shows another Example in other embodiment of this invention.

[Embodiment 1]
This embodiment will be described below with reference to FIGS.

  Generally speaking, the data display / sensor device 100 (input device) according to the present embodiment displays a sub-image cut out from one image and can detect a nearby image and can detect a nearby image. A scroll control unit 840 (position change) for changing the position of the sub image on the image displayed on the sensor-incorporated liquid crystal panel 301 according to the movement of the image detected by the panel 301 (planar member) and the sensor-incorporated liquid crystal panel 301. And a position specifying unit 830 (position specifying means) for specifying the position of the image detected by the sensor built-in liquid crystal panel 301 as a position on the image. The area is calculated to determine whether the area of the image is larger than a predetermined threshold, and the scroll control unit 840 changes the position of the sub-image according to the determination result. Either, or comprises or position specifying part 830 specifies the position on the image, an area judging unit 820 for selecting (selecting means), a.

  First, an outline of the sensor built-in liquid crystal panel 301 included in the data display / sensor device 100 will be described. Note that the data display / sensor device 100 may include one or more sensor-equipped liquid crystal panels 301.

(Outline of LCD panel with built-in sensor)
The sensor built-in liquid crystal panel 301 included in the data display / sensor device 100 is a liquid crystal panel capable of detecting an image of an object in addition to displaying data. Here, the image detection of the object is, for example, detection of a position pointed (touched) by the user with a finger or a pen, or reading (scanning) of an image of a printed material. The device used for display is not limited to a liquid crystal panel, and may be an organic EL (Electro Luminescence) panel or the like.

  The structure of the sensor built-in liquid crystal panel 301 will be described with reference to FIG. FIG. 2 is a diagram schematically showing a cross section of the sensor built-in liquid crystal panel 301. The sensor built-in liquid crystal panel 301 described here is an example, and any structure can be used as long as the display surface and the reading surface are shared.

  As shown in the figure, the sensor built-in liquid crystal panel 301 includes an active matrix substrate 51A disposed on the back surface side and a counter substrate 51B disposed on the front surface side, and has a structure in which a liquid crystal layer 52 is sandwiched between these substrates. is doing. The active matrix substrate 51A is provided with a pixel electrode 56, a data signal line 57, an optical sensor circuit 32 (not shown), an alignment film 58, a polarizing plate 59, and the like. The counter substrate 51B is provided with color filters 53r (red), 53g (green), 53b (blue), a light shielding film 54, a counter electrode 55, an alignment film 58, a polarizing plate 59, and the like. In addition, a backlight 307 is provided on the back surface of the sensor built-in liquid crystal panel 301.

  The photodiode 6 included in the photosensor circuit 32 is provided in the vicinity of the pixel electrode 56 provided with the blue color filter 53b, but is not limited to this configuration. It may be provided in the vicinity of the pixel electrode 56 provided with the red color filter 53r, or may be provided in the vicinity of the pixel electrode 56 provided with the green color filter 53g.

  Next, with reference to FIGS. 3A and 3B, two types of methods for detecting the position where the user touches the sensor built-in liquid crystal panel 301 with a finger or a pen will be described.

  FIG. 3A is a schematic diagram illustrating a state in which a position touched by the user is detected by detecting a reflected image. When the light 63 is emitted from the backlight 307, the optical sensor circuit 32 including the photodiode 6 detects the light 63 reflected by the object 64 such as a finger. Thereby, the reflected image of the target object 64 can be detected. Thus, the sensor built-in liquid crystal panel 301 can detect the touched position by detecting the reflected image.

  FIG. 3B is a schematic diagram illustrating a state in which a position touched by the user is detected by detecting a shadow image. As shown in FIG. 3B, the optical sensor circuit 32 including the photodiode 6 detects external light 61 transmitted through the counter substrate 51B and the like. However, when there is an object 62 such as a pen, the incident of the external light 61 is hindered, so that the amount of light detected by the optical sensor circuit 32 is reduced. Thereby, a shadow image of the object 62 can be detected. Thus, the sensor built-in liquid crystal panel 301 can also detect a touched position by detecting a shadow image.

  As described above, the photodiode 6 may detect reflected light (shadow image) of the light emitted from the backlight 307 or may detect a shadow image caused by external light. Further, the two types of detection methods may be used in combination to detect both a shadow image and a reflected image at the same time.

(Data display / sensor configuration)
Next, the configuration of the main part of the data display / sensor device 100 will be described with reference to FIG. FIG. 4 is a block diagram showing a main configuration of the data display / sensor device 100. As shown in FIG. As illustrated, the data display / sensor device 100 includes one or more display / light sensor units 300, a circuit control unit 600, a data processing unit 700, a main control unit 800, a storage unit 901, a primary storage unit 902, and an operation unit. 903, an external communication unit 907, an audio output unit 908, and an audio input unit 909. Here, the data display / sensor device 100 will be described as including two display / light sensor units 300 (first display / light sensor unit 300A and second display / light sensor unit 300B). When the first display / light sensor unit 300A and the second display / light sensor unit 300B are not distinguished, they are referred to as the display / light sensor unit 300.

  The display / light sensor unit 300 is a so-called light sensor built-in liquid crystal display device and has a df. The display / light sensor unit 300 includes a sensor built-in liquid crystal panel 301, a backlight 307, and a peripheral circuit 309 for driving them.

  The sensor built-in liquid crystal panel 301 includes a plurality of pixel circuits 31 and photosensor circuits 32 arranged in a matrix. The detailed configuration of the sensor built-in liquid crystal panel 301 will be described later.

  The peripheral circuit 309 includes a liquid crystal panel drive circuit 304, an optical sensor drive circuit 305, a signal conversion circuit 306, and a backlight drive circuit 308.

  The liquid crystal panel driving circuit 304 outputs a control signal (G) and a data signal (S) in accordance with the timing control signal (TC1) and the data signal (D) from the display control unit 601 of the circuit control unit 600, and the pixel circuit 31. It is a circuit which drives. Details of the driving method of the pixel circuit 31 will be described later.

  The optical sensor driving circuit 305 is a circuit that drives the optical sensor circuit 32 by applying a voltage to the signal line (R) in accordance with a timing control signal (TC2) from the sensor control unit 602 of the circuit control unit 600. Details of the driving method of the optical sensor circuit 32 will be described later.

  The signal conversion circuit 306 is a circuit that converts the sensor output signal (SS) output from the optical sensor circuit 32 into a digital signal (DS) and transmits the converted signal to the sensor control unit 602.

  The backlight 307 includes a plurality of white LEDs (Light Emitting Diodes) and is disposed on the back surface of the sensor built-in liquid crystal panel 301. When a power supply voltage is applied from the backlight drive circuit 308, the backlight 307 is turned on and irradiates the sensor built-in liquid crystal panel 301 with light. Note that the backlight 307 is not limited to white LEDs, and may include LEDs of other colors. The backlight 307 may include, for example, a cold cathode fluorescent lamp (CCFL) instead of the LED.

  The backlight driving circuit 308 applies a power supply voltage to the backlight 307 when the control signal (BK) from the backlight control unit 603 of the circuit control unit 600 is at a high level, and conversely, the backlight control unit 603. When the control signal from is at a low level, no power supply voltage is applied to the backlight 307.

  Next, the circuit control unit 600 will be described. The circuit control unit 600 has a function as a device driver that controls the peripheral circuit 309 of the display / light sensor unit 300. The circuit control unit 600 includes a display control unit 601, a sensor control unit 602, a backlight control unit 603, and a display data storage unit 604.

  The display control unit 601 receives display data from the display data processing unit 701 of the data processing unit 700, and performs timing control on the liquid crystal panel driving circuit 304 of the display / light sensor unit 300 in accordance with an instruction from the display data processing unit 701. A signal (TC1) and a data signal (D) are transmitted, and the received display data is displayed on the sensor built-in liquid crystal panel 301.

  The display control unit 601 temporarily stores the display data received from the display data processing unit 701 in the display data storage unit 604. Then, a data signal (D) is generated based on the primary stored display data. The display data storage unit 604 is, for example, a video random access memory (VRAM).

  The sensor control unit 602 transmits a timing control signal (TC2) to the optical sensor driving circuit 305 of the display / optical sensor unit 300 in accordance with an instruction from the sensor data processing unit 703 of the data processing unit 700, and the sensor built-in liquid crystal panel 301. Run the scan with.

  In addition, the sensor control unit 602 receives a digital signal (DS) from the signal conversion circuit 306. Then, image data is generated based on the digital signal (DS) corresponding to the sensor output signal (SS) output from all the optical sensor circuits 32 included in the sensor built-in liquid crystal panel 301. That is, the image data read in the entire reading area of the sensor built-in liquid crystal panel 301 is generated. Then, the generated image data is transmitted to the sensor data processing unit 703.

  The backlight control unit 603 transmits a control signal (BK) to the backlight drive circuit 308 of the display / light sensor unit 300 in accordance with instructions from the display data processing unit 701 and the sensor data processing unit 703 to drive the backlight 307. Let

  When the data display / sensor device 100 includes a plurality of display / light sensor units 300, the display control unit 601 determines which display / light sensor unit 300 displays the display data from the data processing unit 700. When an instruction is received, the liquid crystal panel drive circuit 304 of the display / light sensor unit 300 is controlled according to the instruction. When the sensor control unit 602 receives an instruction from the data processing unit 700 as to which display / light sensor unit 300 is to scan the object, The sensor drive circuit 305 is controlled and a digital signal (DS) is received from the signal conversion circuit 306 of the display / light sensor unit 300 according to the instruction.

  Next, the data processing unit 700 will be described. The data processing unit 700 has a function as middleware that gives an instruction to the circuit control unit 600 based on a “command” received from the main control unit 800. Details of the command will be described later.

  The data processing unit 700 includes a display data processing unit 701 and a sensor data processing unit 703. When the data processing unit 700 receives a command from the main control unit 800, at least one of the display data processing unit 701 and the sensor data processing unit 703 depends on the value of each field (described later) included in the received command. One works.

  The display data processing unit 701 receives display data from the main control unit 800, and gives instructions to the display control unit 601 and the backlight control unit 603 according to the command received by the data processing unit 700, and displays the received display data. The image is displayed on the sensor built-in liquid crystal panel 301. The operation of the display data processing unit 701 according to the command will be described later.

  The sensor data processing unit 703 gives an instruction to the sensor control unit 602 and the backlight control unit 603 according to the command received by the data processing unit 700.

  The sensor data processing unit 703 receives image data from the sensor control unit 602 and stores the image data in the image data buffer 704 as it is. Then, in accordance with the command received by the data processing unit 700, the sensor data processing unit 703 performs “whole image data”, “partial image data (partial image coordinate data) based on the image data stored in the image data buffer 704. At least one of “including coordinate data” and “coordinate data” is transmitted to the main control unit 800. The whole image data, partial image data, and coordinate data will be described later. The operation of the sensor data processing unit 703 according to the command will be described later.

  Next, the main control unit 800 executes an application program. The main control unit 800 reads the program stored in the storage unit 901 into a primary storage unit 902 configured by, for example, a RAM (Random Access Memory) and executes the program.

  An application program executed by the main control unit 800 causes the data processing unit 700 to display display data on the sensor built-in liquid crystal panel 301 or to scan an object on the sensor built-in liquid crystal panel 301. Send commands and display data. When “data type” is designated as a command, at least one of whole image data, partial image data, and coordinate data is received from the data processing unit 700 as a response to the command.

  The circuit control unit 600, the data processing unit 700, and the main control unit 800 can be configured by a CPU (Central Processing Unit), a memory, and the like, respectively. The data processing unit 700 may be configured by a circuit such as an ASIC (application specific integrate circuit).

  Next, the storage unit 901 stores programs and data executed by the main control unit 800 as shown in the figure. The program executed by the main control unit 800 may be separated into an application-specific program and a general-purpose program that can be shared by each application.

  Next, the operation unit 903 receives an input operation of the user of the data display / sensor device 100. The operation unit 903 includes input devices such as a switch, a remote controller, a mouse, and a keyboard, for example. Then, the operation unit 903 generates a control signal corresponding to the user's input operation of the data display / sensor device 100, and transmits the generated control signal to the main control unit 800.

  As an example of the switch, a hardware switch such as a power switch 905 that switches power on and off and a user switch 906 to which a predetermined function is assigned in advance is assumed.

  In addition, the data display / sensor device 100 includes an external communication unit 907 for communicating with an external device by wireless / wired communication, an audio output unit 908 such as a speaker for outputting audio, and a microphone for inputting an audio signal. A voice input unit 909 such as the above may be provided as appropriate.

(Command details)
Next, details of commands transmitted from the main control unit 800 to the data processing unit 700 will be described with reference to FIGS. 5 and 6. FIG. 5 is a diagram schematically illustrating an example of a command frame structure. FIG. 6 is a diagram for explaining an example of values that can be specified for each field included in the command and an outline thereof.

  As shown in FIG. 5, the commands are “header”, “data acquisition timing”, “data type”, “scan method”, “scan image gradation”, “scan resolution”, “scan panel”, “display panel”. "And" Reserve "fields. In each field, for example, values shown in FIG. 6 can be designated.

  The “header” field is a field indicating the start of a frame. As long as it is possible to identify the “header” field, the value of the “header” field may be any value.

  Next, the “data acquisition timing” field is a field for designating a timing at which data should be transmitted to the main control unit 800. In the “data acquisition timing” field, for example, values “00” (sense), “01” (event), and “10” (all) can be specified.

  Here, “sense” designates that the latest data is transmitted immediately. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “sense”, the latest data specified in the “data type” field is immediately updated to the main control unit 800. Send to.

  The “event” designates transmission at a timing when a change occurs in the image data received from the sensor control unit 602. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “event”, the image that receives the data specified in the “data type” field from the sensor control unit 602. The data is transmitted to the main control unit 800 at a timing when a change larger than a predetermined threshold occurs.

  “All” designates data transmission at a predetermined cycle. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “all”, the data designated in the “data type” field is transferred to the main control unit 800 at a predetermined cycle. Send to. The predetermined period coincides with the period in which the optical sensor circuit 32 performs scanning.

  Next, the “data type” field is a field for designating the type of data acquired from the sensor data processing unit 703. In the “data type” field, for example, values of “001” (coordinates), “010” (partial image), and “100” (entire image) can be specified. Furthermore, by adding these values, “coordinates” and “partial image” / “whole image” can be specified simultaneously. For example, when “coordinate” and “partial image” are specified at the same time, “011” can be specified.

  When the sensor data processing unit 703 receives a command whose value of the “data type” field is “whole image”, the sensor data processing unit 703 transmits the image data itself stored in the image data buffer 704 to the main control unit 800. The image data itself stored in the image data buffer 704 is referred to as “whole image data”.

  In addition, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 selects a portion where a change larger than a predetermined threshold has occurred from the image data received from the sensor control unit 602. A region to be included is extracted, and image data of the extracted region is transmitted to the main control unit 800. Here, the image data is referred to as “partial image data”. When a plurality of partial image data are extracted, the sensor data processing unit 703 transmits the extracted partial image data to the main control unit 800.

  Further, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 detects representative coordinates in the partial image data and indicates the position of the representative coordinates in the partial image data. The coordinate data is transmitted to the main control unit 800. The representative coordinates include, for example, the coordinates of the center of the partial image data, the coordinates of the center of gravity of the partial image data, and the like.

  Next, when receiving a command whose value of the “data type” field is “coordinate”, the sensor data processing unit 703 transmits coordinate data indicating the position of the representative coordinate in the entire image data to the main control unit 800. When a plurality of partial image data are extracted, the sensor data processing unit 703 detects representative coordinates in the entire image data of the extracted partial image data, and the coordinate data indicating the representative coordinates is detected. Each is transmitted to the main control unit 800 (multi-point detection).

  Specific examples of the whole image data, the partial image data, and the coordinate data will be described later with reference to schematic diagrams.

  Next, the “scan method” field is a field for designating whether or not the backlight 307 is turned on at the time of executing the scan. In the “scan method” field, for example, values of “00” (reflection), “01” (transmission), and “10” (reflection / transmission) can be designated.

  “Reflection” designates that scanning is performed with the backlight 307 turned on. Therefore, when the sensor data processing unit 703 receives a command whose “scan method” field value is “reflection”, the sensor data processing unit 703 performs sensor control so that the optical sensor driving circuit 305 and the backlight driving circuit 308 operate in synchronization. An instruction is given to the unit 602 and the backlight control unit 603.

  “Transmission” specifies that scanning is performed with the backlight 307 turned off. Therefore, when the sensor data processing unit 703 receives a command whose “scan method” field value is “transparent”, the sensor control unit 602 operates the optical sensor driving circuit 305 and does not operate the backlight driving circuit 308. Instructions to the backlight control unit 603. Note that “reflection / transmission” specifies that scanning is performed using both “reflection” and “transmission”.

  Next, the “scanned image gradation” field is a field for designating gradations of the partial image data and the entire image data. In the “scanned image gradation” field, for example, values of “00” (binary) and “01” (multivalue) can be designated.

  When the sensor data processing unit 703 receives a command whose “scan image gradation” field value is “binary”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 as monochrome data. .

  When the sensor data processing unit 703 receives a command whose “scanned image gradation” field value is “multivalued”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 as multitone data. To do.

  Next, the “scan resolution” field is a field for designating the resolution of the partial image data and the entire image data. In the “resolution” field, for example, values of “0” (high) and “1” (low) can be designated.

  Here, “high” designates a high resolution. Therefore, when the sensor data processing unit 703 receives a command whose “scan resolution” field value is “high”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 with high resolution. For example, it is desirable to designate “high” for image data (image data such as a fingerprint) to be subjected to image processing such as image recognition.

  “Low” designates a low resolution. Therefore, when the sensor data processing unit 703 receives a command whose “scan resolution” field value is “low”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 at a low resolution. For example, it is desirable to designate “low” for image data (such as touched finger or hand image data) that only needs to be recognized.

  Next, the “scan panel” field is a field for designating which display / light sensor unit 300 is to scan the object. In the “scan panel” field, for example, values “001” (first display / light sensor unit 300A) and “010” (second display / light sensor unit 300B) can be designated. By adding these values, a plurality of display / light sensor units 300 can be specified at the same time. For example, when “first display / light sensor unit 300A” and “second display / light sensor unit 300B” are specified at the same time, “011” can be specified.

  When the sensor data processing unit 703 receives a command whose “scan panel” field value is “first display / photosensor unit 300A”, the sensor data processing unit 703 and the photosensor drive circuit 305 of the first display / photosensor unit 300A and An instruction is given to the sensor control unit 602 and the backlight control unit 603 so as to control the backlight drive circuit 308.

  Next, the “display panel” field is a field for designating which display / light sensor unit 300 displays the display data. In the “display panel” field, for example, values of “001” (first display / light sensor unit 300A) and “010” (second display / light sensor unit 300B) can be designated. By adding these values, a plurality of display / light sensor units 300 can be specified at the same time. For example, when “first display / light sensor unit 300A” and “second display / light sensor unit 300B” are specified at the same time, “011” can be specified.

  Here, for example, when the display data processing unit 701 receives a command whose value of the “display panel” field is “first display / light sensor unit 300A”, the display data processing unit 701 displays the display data on the first display / light sensor unit 300A. Therefore, an instruction is given to the display control unit 601 and the backlight control unit 603 to control the liquid crystal panel driving circuit 304 and the backlight driving circuit 308 of the first display / light sensor unit 300A.

  Next, the “reserved” field is a field that is appropriately specified when it is necessary to further specify information other than information that can be specified in the above-described fields.

  Note that an application executed by the main control unit 800 does not need to use all the above-described fields when transmitting a command, and an invalid value (such as a NULL value) may be set for a field that is not used. .

  When the user wants to acquire coordinate data of a position touched with a finger or pen, a command specifying “coordinate” in the “data type” field is transmitted to the data processing unit 700. Therefore, it is desirable to specify “all” in the “data acquisition timing” field of the command to acquire coordinate data. Further, since it is only necessary to acquire coordinate data of the touched position, the scanning accuracy may not be high. Therefore, “low” may be specified as the value of the “resolution” field of the command.

  Further, when “coordinate” is specified in the “data type” field of the command, for example, when the user touches the sensor built-in liquid crystal panel 301 with a plurality of fingers or pens at the same time, the coordinate data of the touched position is used. Can be acquired (multi-point detection).

  When acquiring image data of an object such as a document, a command specifying “whole image” in the “data type” field is transmitted to the data processing unit 700. Since it is common to perform a scan in a stationary state, it is not necessary to periodically perform the scan. Therefore, in this case, it is desirable to designate “sense” or “event” in the “data acquisition timing” field. When scanning an object such as a document, it is desirable that the scanning accuracy is high so that the user can easily read the characters. Therefore, it is desirable to designate “high” in the “resolution” field.

(Whole image data / Partial image data / Coordinate data)
Next, the whole image data, the partial image data, and the coordinate data will be described with reference to FIG. The image data shown in FIG. 7A is image data obtained as a result of scanning the entire sensor built-in liquid crystal panel 301 when the object is not placed on the sensor built-in liquid crystal panel 301. The image data shown in FIG. 7B is image data obtained as a result of scanning the entire sensor-equipped liquid crystal panel 301 when the user touches the sensor-equipped liquid crystal panel 301 with a finger.

  When the user touches the sensor built-in liquid crystal panel 301 with a finger, the amount of light received by the photosensor circuit 32 in the vicinity of the touch changes, so that the voltage output from the photosensor circuit 32 changes, and as a result, In the image data generated by the sensor control unit 602, the brightness of the pixel value of the portion touched by the user changes.

  In the image data shown in FIG. 7B, the brightness of the pixel value of the portion corresponding to the user's finger is higher than that in the image data shown in FIG. In the image data shown in FIG. 7B, the smallest rectangular area (area PP) that includes all pixel values whose lightness changes more than a predetermined threshold is “partial image data”.

  The image data indicated by the area AP is “whole image data”.

  Also, the coordinate data in the whole image data (area AP) of the representative coordinates Z of the partial image data (area PP) is (Xa, Ya), and the coordinate data in the partial image data (area PP) is (Xp, Yp). It is.

(Configuration of sensor built-in liquid crystal panel)
Next, the configuration of the sensor built-in liquid crystal panel 301 and the configuration of the peripheral circuit 309 of the sensor built-in liquid crystal panel 301 will be described with reference to FIG. FIG. 8 is a block diagram showing the main part of the display / light sensor unit 300, particularly the configuration of the sensor built-in liquid crystal panel 301 and the configuration of the peripheral circuit 309.

  The sensor built-in liquid crystal panel 301 includes a pixel circuit 31 for setting light transmittance (brightness) and an optical sensor circuit 32 that outputs a voltage corresponding to the intensity of light received by the sensor. Note that the pixel circuit 31 is a generic term for the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b corresponding to the red, green, and blue color filters, respectively.

  The pixel circuits 31 are arranged on the sensor built-in liquid crystal panel 301 in the column direction (vertical direction) and 3n in the row direction (horizontal direction). A set of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b is continuously arranged in the row direction (lateral direction). This set forms one pixel.

  In order to set the light transmittance of the pixel circuit 31, first, the high level voltage (TFT 33 is turned on) to the scanning signal line Gi connected to the gate terminal of the TFT (Thin Film Transistor) 33 included in the pixel circuit 31. Voltage). Thereafter, a predetermined voltage is applied to the data signal line SRj connected to the source terminal of the TFT 33 of the R pixel circuit 31r. Similarly, the light transmittance is also set for the G pixel circuit 31g and the B pixel circuit 31b. Then, by setting these light transmittances, an image is displayed on the sensor built-in liquid crystal panel 301.

  Next, the photosensor circuit 32 is arranged for each pixel. One pixel may be arranged in the vicinity of each of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b.

  In order for the optical sensor circuit 32 to output a voltage corresponding to the light intensity, first, the sensor readout line RWi connected to one electrode of the capacitor 35 and the anode terminal of the photodiode 36 are connected. A predetermined voltage is applied to the sensor reset line RSi. In this state, when light is incident on the photodiode 36, a current corresponding to the amount of incident light flows through the photodiode 36. Then, according to the current, the voltage at the connection point (hereinafter referred to as connection node V) between the other electrode of the capacitor 35 and the cathode terminal of the photodiode 36 decreases. When the power supply voltage VDD is applied to the voltage application line SDj connected to the drain terminal of the sensor preamplifier 37, the voltage at the connection node V is amplified and output from the source terminal of the sensor preamplifier 37 to the sensing data output line SPj. Based on the output voltage, the amount of light received by the optical sensor circuit 32 can be calculated.

  Next, the liquid crystal panel drive circuit 304, the optical sensor drive circuit 305, and the sensor output amplifier 44, which are peripheral circuits of the sensor built-in liquid crystal panel 301, will be described.

  The liquid crystal panel drive circuit 304 is a circuit for driving the pixel circuit 31, and includes a scanning signal line drive circuit 3041 and a data signal line drive circuit 3042.

  The scanning signal line driving circuit 3041 sequentially selects one scanning signal line from the scanning signal lines G1 to Gm for each line time based on the timing control signal TC1 received from the display control unit 601, and A high level voltage is applied to the selected scanning signal line, and a low level voltage is applied to the other scanning signal lines.

  Based on the display data D (DR, DG, and DB) received from the display controller 601, the data signal line driver circuit 3042 generates a predetermined voltage corresponding to the display data for one row for each line time. The data signal lines SR1 to SRn, SG1 to SGn, and SB1 to SBn are applied (line sequential method). Note that the data signal line driver circuit 3042 may be driven by a dot sequential method.

  The optical sensor driving circuit 305 is a circuit for driving the optical sensor circuit 32. Based on the timing control signal TC2 received from the sensor control unit 602, the optical sensor driving circuit 305 selects a predetermined sensor readout signal line from the sensor readout signal lines RW1 to RWm for each line time. A read voltage is applied, and a voltage other than a predetermined read voltage is applied to the other sensor read signal lines. Similarly, based on the timing control signal TC2, a predetermined reset voltage is applied to the sensor reset signal line selected from the sensor reset signal lines RS1 to RSm for each line time, and the others. A voltage other than a predetermined reset voltage is applied to the sensor reset signal line.

The sensing data output signal lines SP1 to SPn are grouped into p groups (p is an integer of 1 to n), and the sensing data output signal lines belonging to each group are connected via a switch 47 that is sequentially turned on in time division. And connected to the sensor output amplifier 44. The sensor output amplifier 44 amplifies the voltage from the group of sensing data output signal lines connected by the switch 47 and outputs the amplified voltage to the signal conversion circuit 306 as sensor output signals SS (SS1 to SSp).
[Embodiment 1]
The main feature of the present invention is that the data display / sensor device 100 displays a sub-image cut out from one image and can detect a nearby image, and a sensor-embedded liquid crystal panel 301 and a sensor-embedded liquid crystal panel The scroll control unit 840 that changes the position of the sub-image on the image displayed on the sensor built-in liquid crystal panel 301 according to the movement of the image detected by the 301, and the position of the image detected by the sensor built-in liquid crystal panel 301 A position specifying unit 830 that specifies the upper position, further calculates the area of the image detected by the sensor built-in liquid crystal panel 301, and determines whether the area of the image is larger than a predetermined threshold. Depending on the determination result, the scroll control unit 840 selects whether to change the position of the sub-image, or the position specifying unit 830 specifies the position on the image. An area determination unit 820, in that it comprises a.

  As a result, the data display / sensor device 100 detects a nearby image by the sensor built-in liquid crystal panel 301, calculates the area of the detected image, and determines whether the area of the image is larger than a predetermined threshold. According to the determination result, two operations, that is, an operation in which the scroll control unit 840 changes the position of the sub image, and a position specifying unit 830 displays the position of the image detected by the sensor built-in liquid crystal panel 301. The operation specified as the upper position can be performed separately. The sensor built-in liquid crystal panel 301 can detect a nearby image. Therefore, even when the sensor built-in liquid crystal panel 301 detects two images having different areas, the two operations can be performed independently.

Therefore, in the data display / sensor device 100, it is not necessary to wait for one of the two operations to be completed before executing the other operation. Therefore, the data display / sensor device 100 can perform continuous and various operations without interruption. Can be provided to the user. Since this operation cannot be realized by a conventional input device, the user is provided with excellent operability and convenience.
(Description of data display / sensor device operation)
The data display / sensor device 100 according to the first embodiment will be described with reference to FIGS. 1 and 9 to 11.

  First, a specific example of the operation performed by the data display / sensor device 100 will be described with reference to FIGS. Next, the configuration of the data display / sensor device 100 will be described with reference to FIG. Then, the flow of processing for realizing the operation will be described with reference to FIG.

Further, a map is displayed on the sensor built-in liquid crystal panel 301 of the data display / sensor device 100, and the user inputs a route to the map using a stylus pen.
(Operations performed on the display image)
The outline of FIGS. 9 and 10 will be described below. FIG. 9 is a diagram illustrating how the finger is moved to the right side because the trace of the route by the stylus pen has reached the left end of the screen. FIG. 10 is a diagram showing a state in which the position of the map displayed on the sensor built-in liquid crystal panel 301 is changed by the operation of the finger shown in FIG. 9 and the route is traced again using the stylus pen.

  As shown in FIG. 9, the sensor built-in liquid crystal panel 301 of the data display / sensor device 100 can display a map and simultaneously detect images of a finger and a stylus pen. The data display / sensor device 100 distinguishes whether the image detected by the sensor built-in liquid crystal panel 301 is a finger image or a stylus pen image as follows.

  Generally, the area of the finger image is larger than the area of the stylus pen image. Therefore, the area of the detected image is compared with a predetermined threshold, and a determination is made that if the detected image is larger than the predetermined threshold, a finger image, and if the detected image is less than the predetermined threshold, a stylus pen image is determined. To do. By determining in this way, the data display / sensor device 100 can distinguish whether the image detected by the sensor built-in liquid crystal panel 301 is a finger image or a stylus pen image.

  When the image detected by the sensor built-in liquid crystal panel 301 is a finger image, the data display / sensor device 100 displays a map displayed on the sensor built-in liquid crystal panel 301 according to the movement of the finger image. The position is changed (in the following description, changing the position of the map is called scrolling).

  That is, when the finger moves to the right side of the drawing (from the west to the east on the map), the map displayed on the sensor built-in liquid crystal panel 301 scrolls to the east by the same amount of movement, and as a result, the sensor is built in. The liquid crystal panel 301 displays a map of the western region that was not displayed on the screen. Similarly, although not shown, when the finger moves in the upper left direction (from southeast to northwest), the map displayed on the sensor built-in liquid crystal panel 301 with the same movement amount and movement direction is northwest. As a result, a map of the southeastern region that was not displayed on the screen is displayed on the sensor built-in liquid crystal panel 301.

  At this time, the map is also scrolled at the same speed as the moving speed of the finger. Therefore, if the movement amount, movement direction, and movement speed are paraphrased as movement vectors, the data display / sensor device 100 scrolls the map in accordance with the movement vector of the finger image detected by the sensor built-in liquid crystal panel 301.

  On the other hand, when the image detected by the sensor built-in liquid crystal panel 301 is a stylus pen image, the data display / sensor device 100 identifies the position of the stylus pen image as the position on the map, and the identified position on the map Is displayed on the sensor built-in liquid crystal panel 301. That is, when a stylus pen image is detected, a position image indicating the position of the stylus pen image on the map is displayed on the map as if the map is traced with the stylus pen. Therefore, by tracing the road on the map and moving the stylus pen, the user can trace the road.

  In FIG. 9, the user traces the road on the map with the stylus pen from the right side (east) of the drawing to the left side (west). Then, the trace reaches the left end of the screen (near the area indicated as A city on the map), and the trace is not possible any more. Therefore, the user tries to scroll the map in the east direction by causing the sensor built-in liquid crystal panel 301 to detect the image of the finger and then moving the finger from the west direction to the east direction.

  Next, FIG. 10 will be described. FIG. 10 shows a state where the map displayed on the sensor built-in liquid crystal panel 301 is scrolled by the operation of the finger according to FIG. 9 and the route is traced again using the stylus pen. As a result of the scrolling shown in FIG. 9, City A is displayed near the center of the screen (FIG. 10). Thereby, the traceable range is further expanded in the west direction, and the user is tracing further in the west direction than the city A. Similarly, although not shown, for example, when tracing is performed up to the upper end of the screen (north direction), the map can be scrolled in the south direction by moving the finger from the north to the south direction. it can. This extends the traceable range further northward.

  Here, “distance 8 km” is displayed in the upper left of the drawing in FIG. 9, and “distance 12 km” is displayed in the upper left of the drawing in FIG. 10. This indicates the distance on the map traced by the user with the stylus pen. That is, the trace corresponding to the distance of 8 km on the map is performed at the time shown in FIG. 9, and the trace corresponding to the distance of 12 km on the map is performed at the time shown in FIG.

  Thus, in the data display / sensor device 100, the distance on the map corresponding to the traced length is displayed on the sensor built-in liquid crystal panel 301. Therefore, the user can easily and quickly confirm the distance. it can.

(Explanation of main components of data display / sensor device)
Next, a more detailed configuration of the data display / sensor device 100 will be described with reference to FIG. Here, in order to make the description easy to understand, the description of the operations of the data processing unit 700 and the circuit control unit 600 located between the main control unit 800 and the display / light sensor unit 300 is omitted. However, to be precise, when performing image display and image detection, the image display control unit 860 of the main control unit 800 transmits a command to the data processing unit 700, and the data processing unit 700 performs circuit control based on the command. The circuit control unit 600 transmits a signal to the display / light sensor unit 300. Further, the main control unit 800 acquires the entire image data, the partial image data, and the coordinate data from the data processing unit 700 as a response to the command transmitted to the data processing unit 700.

  FIG. 1 is a block diagram showing a main configuration of the data display / sensor device 100. As illustrated, the main control unit 800 includes a data acquisition unit 810, an area determination unit 820, a position specifying unit 830, a scroll control unit 840, a distance calculation unit 850, and an image display control unit 860. . The distance calculation unit 850 can be omitted.

  The data acquisition unit 810 acquires the entire image data and coordinate data output from the data processing unit 700. Then, the data acquisition unit 810 outputs the entire image data and coordinate data acquired from the data processing unit 700 to the area determination unit 820.

  The area determination unit 820 acquires the entire image data and coordinate data from the data acquisition unit 810, and calculates the area of an image included in the entire image data (hereinafter referred to as an image area). Specifically, the area determination unit 820 determines whether the calculated image area is larger than a predetermined threshold. The predetermined threshold value is stored in the storage unit 901, and the area determination unit 820 reads the predetermined threshold value from the storage unit 901.

  Next, the area determination unit 820 outputs the coordinate data acquired from the data acquisition unit 810 to the position specifying unit 830 or the scroll control unit 840. At this time, the area determination unit 820 changes the output destination according to the image area included in the entire image data acquired from the data acquisition unit 810. The image area calculation method may be a general method performed by conventional image processing software.

  More specifically, the area determination unit 820 outputs the coordinate data acquired from the data acquisition unit 810 to the scroll control unit 840 when the calculated image area is larger than a predetermined threshold. On the other hand, the area determination unit 820 outputs the coordinate data acquired from the data acquisition unit 810 to the position specifying unit 830 when the calculated image area is equal to or less than a predetermined threshold.

  The position specifying unit 830 obtains coordinate data from the area determining unit 820 when the image area calculated by the area determining unit 820 is smaller than a predetermined threshold, and based on the coordinate data, the position of the detected image is Specify as the top position.

  More specifically, as described with reference to FIG. 7, the coordinate data in the entire image data (area AP) of the representative coordinates Z of the partial image data (area PP) is represented by (Xa, Ya). The coordinate data in the partial image data (region PP) is expressed as (Xp, Yp). Accordingly, the position specifying unit 830 can specify the position of the detected image on the sensor-equipped liquid crystal panel 301 based on the coordinate data of the image detected by the sensor-equipped liquid crystal panel 301.

  The position specifying unit 830 further reads out map information regarding the map displayed on the sensor built-in liquid crystal panel 301 from the storage unit 901. The map information includes coordinate information specifying each point on the map by coordinates, and the position specifying unit 830 reads the coordinate information from the storage unit 901 and displays it on the sensor built-in liquid crystal panel 301. Any point on the map can be specified by coordinates. Therefore, the position specifying unit 830 can specify the position of the image detected by the sensor built-in liquid crystal panel 301 as the position on the map based on the map information. Therefore, when the road on the map is traced with the stylus pen, the position specifying unit 830 specifies the traced position as the position on the map, and the traced position is incorporated in the sensor by a method described later. Displayed on the liquid crystal panel 301.

  Note that the position specifying unit 830 continuously acquires coordinate data from the data acquisition unit 810 when the sensor built-in liquid crystal panel 301 continuously detects moving images. Based on the map information, the position of the moving image detected by the sensor built-in liquid crystal panel 301 is specified as the position on the map.

  The position specifying unit 830 further specifies the position of the image detected by the sensor built-in liquid crystal panel 301 as the position on the map, and then displays the information, that is, the trace information indicating the position on the map traced by the user with the stylus pen. The data is output to the storage unit 901. The position specifying unit 830 may be configured to store the trace information by itself. The position specifying unit 830 also outputs the trace information to the image display control unit 860, details of which will be described later.

  The scroll control unit 840 acquires coordinate data from the area determination unit 820 when the image area calculated by the area determination unit 820 is larger than a predetermined threshold. The scroll control unit 840 continuously acquires coordinate data from the data acquisition unit 810 when the image detected by the sensor built-in liquid crystal panel 301 moves. Note that the scroll control unit 840 can calculate the moving amount and moving direction of the image by calculating the difference of the coordinate data before and after the moving of the image. Further, the scroll control unit 840 can calculate the moving speed of the image at the same time by calculating the difference in time when the coordinate data is acquired.

  The scroll control unit 840 further reads out map information relating to the map displayed on the sensor built-in liquid crystal panel 301 from the storage unit 901. Then, the scroll control unit 840 displays the map displayed on the sensor built-in liquid crystal panel 301 with the same movement amount, movement direction, and movement speed as the movement amount, movement direction, and movement speed of the image calculated based on the coordinate data. An instruction for scrolling is output to the image display control unit 860. That is, the scroll control unit 840 outputs an instruction for scrolling the map to the image display control unit 860 according to the movement of the image detected by the sensor built-in liquid crystal panel 301.

  The distance calculation unit 850 reads the trace information from the storage unit 901, and calculates the movement amount of the image based on the trace information. Note that a method for calculating the amount of movement of the image can be easily calculated using coordinate data related to the position of the detected image, and may be a general method performed by conventional image processing software.

  At the same time, the distance calculation unit 850 reads map information related to the map displayed on the sensor built-in liquid crystal panel 301 from the storage unit 901. Since the map information includes map scale information indicating the scale of the map, the distance calculation unit 850 can acquire the scale of the map displayed on the sensor built-in liquid crystal panel 301. Then, the distance calculation unit 850 calculates a distance on the map corresponding to the amount of movement of the image based on the calculated amount of movement of the image and the scale of the map. In addition, the distance calculation unit 850 outputs an instruction for displaying a distance on the map corresponding to the moving amount of the image detected by the sensor built-in liquid crystal panel 301 to the image display control unit 860. The movement amount of the image detected by the sensor built-in liquid crystal panel 301 is detected by the sensor built-in liquid crystal panel 301 drawn by a position image displayed on the map and indicating the position of the stylus pen image. This corresponds to the length of the locus of the image.

  The image display control unit 860 displays a display screen corresponding to each output on the sensor built-in liquid crystal panel 301 based on instructions (outputs) from the position specifying unit 830, the scroll control unit 840, and the distance calculation unit 850. Generate display data. Then, the generated display data is output to the data processing unit 700, and a display screen corresponding to each output is displayed on the sensor built-in liquid crystal panel 301.

  More specifically, it is as follows. As described above, the position specifying unit 830 specifies the position of the image detected by the sensor built-in liquid crystal panel 301 as the position on the map. Then, the position specifying unit 830 outputs an instruction for displaying the position image indicating the position of the image of the stylus pen on the map on the sensor built-in liquid crystal panel 301 to the image display control unit 860. Therefore, the image display control unit 860 generates display data for displaying the position image on the sensor built-in liquid crystal panel 301.

  Further, the image display control unit 860 generates display data for scrolling the map displayed on the sensor built-in liquid crystal panel 301 in accordance with the output from the scroll control unit 840. Furthermore, the image display control unit 860 generates display data for displaying the distance on the map on the sensor built-in liquid crystal panel 301 in accordance with the output from the distance calculation unit 850. As a result, a display screen corresponding to each output is superimposed and displayed on the sensor built-in liquid crystal panel 301.

  Here, in FIG. 1, the storage unit 901 is provided outside the main control unit 800. However, the storage unit 901 may be configured to be provided inside the main control unit 800. Further, the predetermined threshold acquired by the area determination unit 820 from the storage unit 901 may be stored in the storage unit 901 in advance, or may be appropriately set by the user from an external input device.

(Flowchart explanation)
Next, the processing flow of the data display / sensor device 100 and the details of commands transmitted from the image display control unit 860 to the data processing unit 700 will be described with reference to FIGS. 1 and 9 to 11.

  Note that the image detected by the sensor built-in liquid crystal panel 301 is assumed to be an image of a finger and a stylus pen. The finger may be an adult finger or a child's finger, and may be a thumb or a little finger. In general, the area of the finger image is much larger than the area of the stylus pen image. Therefore, if the predetermined threshold is set to a value smaller than the finger image and larger than the stylus pen image, the type of finger does not pose a problem.

  In the data display / sensor device 100, the first display / light sensor unit 300A will be described as detecting a nearby image and displaying a map.

  FIG. 11 is a flowchart showing a process flow of the data display / sensor device 100.

  First, when the data display / sensor device 100 is powered off, the user presses the power switch 905 to turn on the data display / sensor device 100 (step S11).

  In step S <b> 13, when the user presses the power switch 905, a map is displayed on the sensor built-in liquid crystal panel 301. In this state, the data display / sensor device 100 stands by until the sensor built-in liquid crystal panel 301 detects a finger image.

  Here, when a map is displayed on the sensor built-in liquid crystal panel 301, a command specifying the upper screen (first display / light sensor unit 300A) (“001”) as the value of the “display panel” field is an image display control unit. The data is output from 860 to the data processing unit 700.

  At this stage, it is assumed that the sensor built-in liquid crystal panel 301 is in a state where a nearby image can be detected. To detect nearby images, specify “Event” in the “Data Acquisition Timing” field, and specify both “Coordinates” and “Whole Image” in the “Data Type” field (specify “101”). The command is output from the image display control unit 860 to the data processing unit 700. Note that a command designating only “coordinates” or “entire image” may be output from the image display control unit 860 to the data processing unit 700.

  Further, in order to detect a nearby image in the sensor built-in liquid crystal panel 301, a command specifying the first display / light sensor unit 300A (“001”) in the “scan panel” field is received from the image display control unit 860 as data. The data is output to the processing unit 700.

  Thus, when the sensor built-in liquid crystal panel 301 detects a finger and / or stylus pen image, the data acquisition unit 810 can acquire the entire image data and coordinate data from the data processing unit 700.

  That is, when the user's finger or stylus pen touches the sensor built-in liquid crystal panel 301, the optical sensor circuit 32 detects at least one of a touched finger or a shadow image and a reflected image of the stylus pen. As a result, a change occurs in the image data acquired by the data processing unit 700 from the sensor control unit 602, and the data acquisition unit 810 displays the entire image data and coordinates output from the data processing unit 700 when the image data changes. Get the data.

  In step S <b> 15, the sensor built-in liquid crystal panel 301 detects a finger and / or stylus pen image, and the data acquisition unit 810 acquires the entire image data and coordinate data from the data processing unit 700. Then, the data acquisition unit 810 outputs the entire image data and coordinate data acquired from the data processing unit 700 to the area determination unit 820.

  Then, the area determination unit 820 acquires the entire image data and coordinate data from the data acquisition unit 810, and calculates the area of the image (image area) included in the entire image data. Furthermore, the area determination unit 820 determines whether the calculated image area is larger than a predetermined threshold. Here, the predetermined threshold value is stored in the storage unit 901, and the area determination unit 820 reads the threshold value from the storage unit 901.

  Here, the predetermined threshold is set to a value smaller than the image area of the finger and larger than the image area of the stylus pen, and the area determination unit 820 includes the image included in the entire image data acquired from the data acquisition unit 810. The output destination of the coordinate data acquired from the data acquisition unit 810 is changed to the position specifying unit 830 or the scroll control unit 840 according to the area.

  If the area determination unit 820 determines that the calculated image area is smaller than the predetermined threshold (Yes in step S15), the process proceeds to step S17. Step S17 corresponds to the case where the image area calculated by the area determination unit 820 is smaller than a predetermined threshold, that is, the case where the sensor built-in liquid crystal panel 301 detects a stylus pen image.

  At this time, the position specifying unit 830 acquires coordinate data from the area determining unit 820, and specifies the detected position of the stylus pen image as a position on the map based on the coordinate data.

  Next, in step S <b> 19, the position specifying unit 830 outputs an instruction for displaying a position image indicating the position of the stylus pen image on the map on the sensor built-in liquid crystal panel 301 to the image display control unit 860. This instruction is for displaying the trace by the stylus pen on the sensor built-in liquid crystal panel 301. Note that the position specifying unit 830 also outputs trace information indicating the position on the map traced by the user to the storage unit 901, and the storage unit 901 stores the trace information.

  Next, in step S21, the distance calculation unit 850 reads the trace information from the storage unit 901, and calculates the movement amount of the image based on the trace information.

  At the same time, the distance calculation unit 850 reads map information related to the map displayed on the sensor built-in liquid crystal panel 301 from the storage unit 901. Since the map information includes map scale information indicating the scale of the map, the distance calculation unit 850 can acquire the scale of the map (for example, 1/25000) displayed on the sensor built-in liquid crystal panel 301. it can.

  Then, the distance calculation unit 850 calculates a distance on the map corresponding to the amount of movement of the image based on the calculated amount of movement of the image and the scale of the map. In addition, the distance calculation unit 850 outputs an instruction for displaying the distance on the map on the sensor built-in liquid crystal panel 301 to the image display control unit 860. This instruction is for causing the sensor built-in liquid crystal panel 301 to display a distance on the map such as “distance 8 km”.

  In step S23, the image display control unit 860 generates display data for causing the sensor built-in liquid crystal panel 301 to display a position image indicating the position of the stylus pen image on the map. Further, the image display control unit 860 generates display data for displaying the distance on the map on the sensor built-in liquid crystal panel 301 according to the output from the distance calculation unit 850. Based on the display data, operation results corresponding to outputs from the position specifying unit 830 and the distance calculating unit 850 are displayed on the sensor built-in liquid crystal panel 301.

  On the other hand, when the area determination unit 820 determines that the calculated image area is larger than the predetermined threshold (No in step S15), the process proceeds to step S25. Step S25 corresponds to the case where the image area calculated by the area determination unit 820 is larger than a predetermined threshold, that is, the case where the sensor built-in liquid crystal panel 301 detects a finger image.

  At this time, the scroll control unit 840 acquires the coordinate data acquired from the data processing unit 700 by the data acquisition unit 810 from the area determination unit 820. Note that the scroll control unit 840 continuously acquires coordinate data from the data acquisition unit 810 when the finger image detected by the sensor-equipped liquid crystal panel 301 moves. Further, the scroll control unit 840 can calculate the movement amount and the moving direction of the image by calculating the difference between the coordinate data before and after the movement of the image. Furthermore, the scroll control unit 840 can calculate the moving speed of the image at the same time by calculating the difference in time when the coordinate data is acquired.

  At the same time, the scroll control unit 840 reads out map information related to the map displayed on the sensor built-in liquid crystal panel 301 from the storage unit 901. Then, the scroll control unit 840 scrolls the map displayed on the sensor built-in liquid crystal panel 301 with the same movement amount, movement direction, and movement speed as the movement amount, movement direction, and movement speed of the image calculated based on the coordinate data. An instruction for causing the image to be displayed is output to the image display control unit 860. That is, the scroll control unit 840 outputs an instruction for scrolling the map to the image display control unit 860 according to the movement of the image detected by the sensor built-in liquid crystal panel 301. In the case of FIG. 10 as an example, this instruction corresponds to an instruction to scroll the map to the right side with the same movement amount, movement direction, and movement speed as a result of moving the finger to the right side of the drawing.

  In step S <b> 23, the image display control unit 860 generates display data for scrolling the map displayed on the sensor built-in liquid crystal panel 301 according to the output from the scroll control unit 840. Based on the display data, an operation result corresponding to the output from the image display control unit 860 is displayed on the sensor built-in liquid crystal panel 301.

  When the sensor built-in liquid crystal panel 301 detects the image of the finger and the stylus pen at the same time, the image display control unit 860 responds to the outputs from the position specifying unit 830, the scroll control unit 840, and the distance calculation unit 850, Display data for displaying a display screen (input screen) corresponding to each output on the sensor built-in liquid crystal panel 301 is generated. The sensor built-in liquid crystal panel 301 displays an operation result corresponding to each output.

  At this time, in order to display an image (operation result) on the sensor built-in liquid crystal panel 301, the command specifying the first display / light sensor unit 300A (“001”) as the value of the “display panel” field is an image display control unit. The data is output from 860 to the data processing unit 700.

  The processing flow of the data display / sensor device 100 has been described in steps S11 to S25. In the above description, it has been described that the distance calculation (step S21) is performed following the trace to the map (step S19), but step S19 and step S21 may be performed simultaneously.

  In this manner, the operation described with reference to FIGS. 9 and 10 is executed, and the operation result is displayed on the sensor built-in liquid crystal panel 301.

  Next, the effect obtained by the processing of the data display / sensor device 100 will be described.

  Since the data display / sensor device 100 has the above-described configuration, the sensor built-in liquid crystal panel 301 detects a nearby image, calculates the area of the detected image, and the area of the image is larger than a predetermined threshold value. In accordance with the determination result, two operations, that is, an operation in which the scroll control unit 840 changes (scrolls) the map position, and a position specifying unit 830 are detected by the sensor built-in liquid crystal panel 301. The operation of specifying the position of the image as the position on the map can be distinguished and executed. Therefore, even when the sensor built-in liquid crystal panel 301 simultaneously detects two images having different areas, the two operations can be performed independently.

  Thus, the data display / sensor device 100 does not need to wait for one of the two operations to be completed before executing the other operation, and therefore, the continuous and various operations without interruption. Can be provided to the user. Since this operation cannot be realized by a conventional input device, the user is provided with excellent operability and convenience.

  Further, in the data display / sensor device 100, the image display control unit 860 can superimpose and display a position image indicating the position on the map specified by the position specifying unit 830 on the map displayed on the sensor built-in liquid crystal panel 301. .

  Therefore, the image display control unit 860 specifies the operation in which the scroll control unit 840 scrolls the map, and the position specifying unit 830 specifies the position of the image detected by the sensor built-in liquid crystal panel 301 as the position on the map. Since the position image indicating the position on the map is displayed in a superimposed manner with the operation of displaying the position image on the sensor built-in liquid crystal panel 301, the operation result can be easily and quickly made visible to the user.

  Furthermore, in the data display / sensor device 100, the area determination unit 820 determines whether the area of the image detected by the sensor built-in liquid crystal panel 301 is larger than a predetermined threshold.

  Therefore, after the data display / sensor device 100 detects the image according to the area of the image easily and quickly, the data display / sensor device 100 compares the size of the image with the predetermined threshold value. , That is, the scroll control unit 840 scrolls the map, and the position specifying unit 830 executes the operation separately from the operation of specifying the position of the image detected by the sensor built-in liquid crystal panel 301 as the position on the map. can do.

  Moreover, since the data display / sensor device 100 can change the determination result by the area determination unit 820 by changing the predetermined threshold, it is possible to provide the user with high convenience.

  In the data display / sensor device 100, the distance calculation unit 850 can calculate a distance on the map corresponding to the length of the locus of the image drawn by the position image and detected by the sensor built-in liquid crystal panel 301.

  Therefore, by using the distance on the map calculated by the distance calculation unit 850, various processes utilizing the data can be performed.

  As an example, consider the case where the data display / sensor device 100 is used in an automobile navigation system. The sensor built-in liquid crystal panel 301 detects the image of the finger and the stylus pen, the map is scrolled by the movement of the finger (the one with the larger area of the image), and the movement of the stylus pen (the one with the smaller area of the image) It is assumed that input is performed by tracing the road.

  If it is possible to select several routes to go to a destination, the user may want to know which route is the shortest distance. In such a case, the user traces (inputs) each route with a stylus pen. Accordingly, the distance calculation unit 850 calculates a distance on the map corresponding to the length of the locus of the image detected by the sensor-equipped liquid crystal panel 301 drawn by the position image related to the image of the stylus pen based on the scale of the map. To do. Then, by using the calculation result, it is possible to compare the distances on the map for each route, and as a result, the car navigation system can inform the user of the shortest route.

  Moreover, the following cases are assumed as other uses. The user may have a request to know the moving distance for each category such as year, month, and day. In such a case, the distance on the map calculated by the distance calculation unit 850 is aggregated for each term such as year, month, day, etc., and the total distance on the map is notified to the user as the total movement distance. be able to. As a result, the user's desire to know the movement distance for each category such as year, month, and day can be fulfilled.

  As described above, in the data display / sensor device 100, the distance calculation unit 850 can calculate the distance on the map corresponding to the length of the locus of the image detected by the sensor built-in liquid crystal panel 301 drawn by the position image. Therefore, excellent operability and convenience can be provided to the user.

  Further, in the data display / sensor device 100, the image display control unit 860 can display the distance on the map calculated by the distance calculation unit 850 on the sensor built-in liquid crystal panel 301.

  As a result, the data display / sensor device 100 can cause the user to visually recognize the distance on the map accurately and quickly, and as a result, can inform the user of information such as the length of the distance for each route or the shortest route. .

  Accordingly, the data display / sensor device 100 can accurately and promptly notify the user of the distance on the map, which could only be measured sensibly with the conventional input device, so that the distance on the map can be accurately determined. It can respond to the user's desire to know.

Thus, the data display / sensor device 100 can provide the user with more excellent operability and convenience.
[Embodiment 2]
Next, a data display / sensor device 110 according to another embodiment of the present invention will be described with reference to FIGS. The application differs between the first embodiment and the present embodiment, and this embodiment uses a drawing software as a use case. Details of the operation described with reference to FIGS. 1 and 9 to 11, details of processing for realizing the operation, and effects obtained by the data display / sensor device 110 are omitted.

  The difference between the data display / sensor device 100 and the data display / sensor device 110 is that the stylus pen is used for route input (trace) in the data display / sensor device 100, whereas the data display / sensor device 110 plots. In that it is used. The data display / sensor device 110 does not need to measure the distance on the map, and therefore may not include the distance calculation unit 850.

  12 and 13 are diagrams showing an embodiment of the data display / sensor device 110. FIG. As shown in FIG. 12, a “star” diagram being drawn by the user is displayed on the sensor built-in liquid crystal panel 301 of the data display / sensor device 110. The upper half of the star is drawn, but the lower half of the star is in the stage of drawing. As shown in the figure, there is no drawing space on the lower side of the screen (lower side of the drawing), and the user cannot draw the lower half of the star in this state. Therefore, in order to draw the lower half of the star, it is necessary to scroll the screen displayed on the sensor built-in liquid crystal panel 301 upward (upward in the drawing).

  Therefore, in FIG. 12, the sensor built-in liquid crystal panel 301 detects the finger and moves the finger upward in the drawing. In accordance with the movement of the finger, the display screen scrolls upward in the drawing with the same movement amount, movement direction, and movement speed as the finger. FIG. 13 shows the operation result. In FIG. 13, the display screen scrolls to the upper side of the drawing in accordance with the movement of the finger (more precisely, the movement of the finger image), and as a result, a new drawing space is provided on the lower side of the screen (lower side of the drawing). Is generated. Thereby, the user can draw the lower half of the star.

  In the second embodiment, it has been described that the drawing software can be used as a use case. That is, the input device according to the present invention can be applied not only to route input (trace) to a map but also to drawing software. As described above, the present invention is not limited to the above-described embodiment, and various modifications can be made within the scope of the claims. That is, embodiments obtained by combining technical means appropriately modified within the scope of the claims are also included in the technical scope of the present invention.

(Program, computer-readable recording medium)
Finally, each block of the data display / sensor devices 100 and 110 according to the present embodiment, in particular, the area determination unit 820, the position specifying unit 830, the scroll control unit 840, the distance calculation unit 850, and the image display control unit 860 includes hardware. You may comprise by a logic and may implement | achieve by software using CPU as follows.

  That is, the data display / sensor devices 100 and 110 according to the present embodiment include a CPU (Central Processing Unit) that executes instructions of a control program that realizes each function, a ROM (Read Only Memory) that stores the program, and the program And a storage device (recording medium) such as a memory for storing the program and various data. The object of the present invention is to record the program code (execution format program, intermediate code program, source program) of the control program of the data display / sensor devices 100, 110, which is software that realizes the above-described functions, in a computer-readable manner. This can also be achieved by supplying the recorded medium to the data display / sensor devices 100, 110, and the computer (or CPU or MPU) reading and executing the program code recorded on the recording medium.

  Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy (registered trademark) disks / hard disks, and disks including optical disks such as CD-ROM / MO / MD / DVD / CD-R. Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.

  The data display / sensor devices 100 and 110 according to the present embodiment may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Also, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope shown in the claims. That is, embodiments obtained by combining technical means appropriately modified within the scope of the claims are also included in the technical scope of the present invention.

  The present invention can be applied to an input device capable of simultaneously displaying an image and reading an object. In particular, it is suitable for a path input device provided with an image detection device.

100 Data display / sensor device (input device)
810 Data acquisition unit 820 Area determination unit (selection means)
830 Position specifying unit (position specifying means)
840 Scroll control unit (position changing means)
850 Distance calculation unit (distance calculation means)
860 Image display control unit (image display control means)
901 Memory unit

Claims (6)

An input device that acquires a user's input by detecting a nearby image,
A planar member capable of displaying a sub-image cut out from one image and detecting a nearby image;
Position changing means for changing the position of the sub-image on the image to be displayed on the planar member according to the movement of the image detected by the planar member;
Position specifying means for specifying the position of the image detected by the planar member as a position on the image, and
The area of the image detected by the planar member is calculated, it is determined whether the area of the image is larger than a predetermined threshold, and the position changing unit determines the position of the sub-image according to the determination result. An input device comprising: selection means for selecting whether to change the position or whether the position specifying means specifies a position on the image.   When the selection unit determines that the area of the image is larger than the predetermined threshold, the selection unit changes the position of the sub-image according to the movement of the image detected by the planar member, and the area of the image 2. The input device according to claim 1, wherein a position on the image is specified when it is determined that is less than or equal to the predetermined threshold value.   The input apparatus according to claim 1, further comprising: an image display control unit configured to superimpose and display a position image indicating the position on the image specified by the position specifying unit on the image. The image is a map,
The input device according to claim 3, further comprising a distance calculating unit that calculates a distance on the map corresponding to a length of a locus drawn by the position image based on a scale of the map.   The input device according to claim 4, wherein the image display control unit displays the distance on the map calculated by the distance calculation unit on the planar member. A method of controlling an input device including a planar member that displays a sub-image cut out from one image and can detect a user's input by detecting a nearby image,
A position changing step for changing the position of the sub-image on the image to be displayed on the planar member according to the movement of the image detected by the planar member;
A position specifying step of specifying the position of the image detected by the planar member as a position on the image, and
The area of the image detected by the planar member is calculated to determine whether the area of the image is larger than a predetermined threshold, and the position changing step determines the position of the sub-image according to the determination result. Or a selection step of selecting whether the position specifying step specifies a position on the image.

Images