JP2010092272A - Image display/image detection device, auxiliary image display method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To smooth a dialogue between information equipment and a user. <P>SOLUTION: An image display/image detection device is provided with: a display/optical sensor 300 for displaying an image and detecting a nearby image; a specified coordinate detection part 12 for detecting a specified coordinate indicating a point ahead where an object on the display/optical sensor part 300 approaches on the basis of an image of the object detected by the display/optical sensor part 300; and an auxiliary image display control unit 13 which identifies an area on the display/optical sensor part 300 covered by the object on the basis of the image of the object detected by the display/optical sensor part 300 and displays an auxiliary image associated with the specified coordinate outside the identified area. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image display / image detection apparatus capable of displaying an image and detecting an image of an object, and an auxiliary image display method in the apparatus. The present invention also relates to a program for operating such an image display / image detection apparatus, and a recording medium on which such a program is recorded.

With the development of the information society, information equipment is used in various situations. Therefore, a technique for facilitating a dialogue between an information device and its user is very important. For example, Patent Documents 1 to 4 disclose techniques for facilitating a dialogue between an information device and a user using a touch panel.
Japanese Patent Laid-Open No. 8-21205 (published on August 20, 1996) Japanese Patent Laid-Open No. 10-269022 (released on October 9, 1998) JP 2003-5912 A (published on January 10, 2003) JP 2004-302958 A (released on October 28, 2004)

  However, in the technologies described in Patent Documents 1 to 4, the dialogue between the information device and the user is not sufficiently smooth, and a technology for further facilitating the dialogue is required.

  SUMMARY An advantage of some aspects of the invention is that it provides a technique for facilitating a dialogue between an information device and a user.

  In order to solve the above problems, an image display / image detection apparatus according to the present invention displays an image, and based on a planar member that detects a nearby image, and an image of an object detected by the planar member. A designated coordinate detecting means for detecting designated coordinates indicating a point to which the object approaches on the planar member, and an area on the planar member that is covered by the object. And an auxiliary image display control means for specifying the auxiliary image associated with the designated coordinates outside the specified area, while specifying based on the image of the object detected by the shape member.

  According to the above configuration, when the object indicates a point on the planar member, the image display / image detection device according to the present invention is an area covered by the object on the planar member. In addition, an auxiliary image associated with the coordinates of the point can be displayed. As a result, the image display / image detection apparatus successfully presents the user with information on the destination indicated by the object that is obscured by the object and has become difficult for the user to see. be able to. Therefore, the interaction between the image display / image detection apparatus and the user can be facilitated.

  In the image display / image detection apparatus according to the present invention, the planar member further includes a commanded image display unit that displays a plurality of commanded images, and the auxiliary image display control unit includes the plurality of commanded images. It is preferable that an image associated with the pointed image including the point indicated by the pointed coordinates in the image is displayed on the planar member as the auxiliary image.

  According to the above configuration, when the object indicates an arbitrary point in the specified image, the image display / image detection apparatus according to the present invention displays an image associated with the specified image as the auxiliary image. Can be displayed as As a result, the image display / image detection apparatus can successfully present to the user which of the indicated images the object indicates. Therefore, the interaction between the image display / image detection apparatus and the user can be facilitated.

  In the image display / image detection apparatus according to the present invention, each of the plurality of designated images may indicate a character, a number, or a symbol, and a screen keyboard may be formed on the planar member.

  According to the above configuration, in each of the screen keyboards, each button (instructed image) is generally small, and the arbitrary button is often obscured by the object when the arbitrary button is pressed by the object. It is possible to successfully present to the user which button the user is pressing. Therefore, the interaction between the image display / image detection apparatus and the user can be facilitated.

  In the image display / image detection apparatus according to the present invention, the planar member includes a photodiode group that detects the amount of light received at each position on the planar member, and the amount of received light detected by the photodiode group. An image in the vicinity may be detected based on the change.

  According to said structure, the said planar member can detect the image of the said vicinity successfully.

  In the image display / image detection apparatus according to the present invention, the auxiliary image display control means may display the auxiliary image before the object contacts the planar member.

  According to said structure, the information about the destination which the said target object points can be shown to a user immediately. Therefore, the interaction between the image display / image detection apparatus and the user can be facilitated.

  An auxiliary image display method according to the present invention is an auxiliary image display method in which an image display / image detection apparatus including a planar member that displays an image and detects a nearby image displays an auxiliary image on the planar member. An instruction coordinate detecting step for detecting an instruction coordinate indicating a point on the planar member to which the object approaches, based on an image of the object detected by the planar member; and the object The area on the planar member covered by the area is identified based on the image of the object detected by the planar member, and the auxiliary image associated with the designated coordinates is displayed outside the identified area. And an image display step.

  According to said structure, there exists an effect similar to the image display / image detection apparatus which concerns on this invention.

  Further, a program for operating the image display / image detection apparatus according to the present invention, which is characterized in that the computer is driven as each of the above means, is also included in the scope of the present invention.

  According to the present invention, the image display / image detection apparatus displays an image, and when the object indicates a point on the planar member that detects a nearby image, the object on the planar member is displayed. Thus, an auxiliary image associated with the coordinates of the point can be displayed outside the area covered by. As a result, the image display / image detection apparatus successfully presents the user with information on the destination indicated by the object that is obscured by the object and has become difficult for the user to see. be able to. Therefore, the interaction between the image display / image detection apparatus and the user can be facilitated.

  FIG. 1 is a diagram schematically showing an appearance of a data display / sensor device (image display / image detection device) 100 according to an embodiment of the present invention. As shown in FIG. 1, the data display / sensor device 100 is provided with a sensor built-in liquid crystal panel (planar member) 301 on one surface of a housing, and is operated by, for example, thumbs (objects) 64 of both hands. Device.

  FIG. 2 is a diagram schematically illustrating an example of the operation of the data display / sensor device 100. In one example, as shown in FIG. 2A, a screen keyboard 351 including a plurality of buttons (instructed images) 350 is displayed on the sensor built-in liquid crystal panel 301.

  Then, as shown in FIG. 2B, when the finger 64 indicates a specific button 350a (in the case of FIG. 2B, a button indicating “S”), the finger 64 is placed on the sensor built-in liquid crystal panel 301. An auxiliary image 352 associated with the button 350 indicating which button 350 is indicated is displayed. By referring to the auxiliary image 352, the user can easily know which button 350 the position indicated by the finger 64 is on, and can smoothly perform input via the screen keyboard 351.

  The auxiliary image 352 is not particularly limited as long as it is associated with the button 350a indicated by the finger 64. For example, it can be an image similar to the button 350a, such as an enlarged image of the button 350a. In addition, the auxiliary image 352 may be provided with a drawer portion 353 having the display position of the button 350a as a tip, so that the user can easily recognize that the auxiliary image 352 is associated with the button 350a. it can.

  In the present embodiment, the auxiliary image 352 is displayed on the sensor built-in liquid crystal panel 301 in the vicinity of the position indicated by the finger 64 and outside the area covered by the finger 64. By displaying the auxiliary image 352 outside the area covered by the finger 64, the user can reliably refer to the auxiliary image 352. As will be described in detail later, this is realized by the data display / sensor device 100 detecting an image of the finger 64 and specifying an area covered by the finger 64. Also, the user's line of sight is usually at the point indicated by the finger 64. Therefore, by displaying the auxiliary image 352 in the vicinity of the position indicated by the finger 64, the user can easily recognize the auxiliary image 352.

  Note that the timing for displaying the auxiliary image 352 may be, for example, when the user does not select the button 350a. For example, when the button 350a is selected by the operation of releasing the finger 64 once after touching the sensor built-in liquid crystal panel 301 (up operation), the up operation is performed after the finger 64 touches the sensor built-in liquid crystal panel 301. The auxiliary image 352 may be displayed until

  Further, for example, when the button 350a is selected by an operation (touch operation) in which the finger 64 is once brought into contact with the sensor built-in liquid crystal panel 301, the auxiliary image 352 is displayed before the finger 64 comes into contact with the sensor built-in liquid crystal panel 301. You may let them. In this configuration, since the auxiliary image 352 indicating the information of the button 350a that can be selected is displayed before the finger 64 touches the sensor built-in liquid crystal panel 301, the user can quickly obtain the information, and the above input is performed. Furthermore, it can be performed smoothly.

  In addition to the finger 64, the user may instruct on the sensor built-in liquid crystal panel 301 using a pen or the like.

  Hereinafter, the structure of each member which implement | achieves the function mentioned above is demonstrated in detail. First, an outline of the sensor built-in liquid crystal panel 301 included in the data display / sensor device 100 will be described.

(Outline of LCD panel with built-in sensor)
The sensor built-in liquid crystal panel 301 provided in the data display / sensor device 100 is a liquid crystal panel capable of detecting an image of an object in addition to displaying data. Here, the image detection of the object is, for example, detection of a position pointed (touched) by the user with a finger or a pen, or reading (scanning) of an image of a printed material. The device used for display is not limited to a liquid crystal panel, and may be an organic EL (Electro Luminescence) panel or the like.

  The structure of the sensor built-in liquid crystal panel 301 will be described with reference to FIG. FIG. 3 is a diagram schematically showing a cross section of the sensor built-in liquid crystal panel 301. The sensor built-in liquid crystal panel 301 described here is an example, and any structure can be used as long as the display surface and the reading surface are shared.

  As shown in the figure, the sensor built-in liquid crystal panel 301 includes an active matrix substrate 51A disposed on the back surface side and a counter substrate 51B disposed on the front surface side, and has a structure in which a liquid crystal layer 52 is sandwiched between these substrates. is doing. The active matrix substrate 51A is provided with a pixel electrode 56, a data signal line 57, an optical sensor circuit 32 (not shown), an alignment film 58, a polarizing plate 59, and the like. The counter substrate 51B is provided with color filters 53r (red), 53g (green), 53b (blue), a light shielding film 54, a counter electrode 55, an alignment film 58, a polarizing plate 59, and the like. In addition, a backlight 307 is provided on the back surface of the sensor built-in liquid crystal panel 301.

  The photodiode 6 included in the photosensor circuit 32 is provided in the vicinity of the pixel electrode 56 provided with the blue color filter 53b, but is not limited to this configuration. It may be provided in the vicinity of the pixel electrode 56 provided with the red color filter 53r, or may be provided in the vicinity of the pixel electrode 56 provided with the green color filter 53g.

  Next, with reference to FIGS. 4A and 4B, two types of methods for detecting the position where the user touches the sensor built-in liquid crystal panel 301 with a finger or a pen will be described.

  FIG. 4A is a schematic diagram showing how the position touched by the user is detected by detecting the reflected image. When the light 63 is emitted from the backlight 307, the optical sensor circuit 32 including the photodiode 6 detects the light 63 reflected by the object 64 such as a finger. Thereby, the reflected image of the target object 64 can be detected. Thus, the sensor built-in liquid crystal panel 301 can detect the touched position by detecting the reflected image.

  FIG. 4B is a schematic diagram showing a state in which a position touched by the user is detected by detecting a shadow image. As shown in FIG. 4B, the optical sensor circuit 32 including the photodiode 6 detects the external light 61 transmitted through the counter substrate 51B and the like. However, when there is an object 62 such as a pen, the incident of the external light 61 is hindered, so that the amount of light detected by the optical sensor circuit 32 is reduced. Thereby, a shadow image of the object 62 can be detected. Thus, the sensor built-in liquid crystal panel 301 can also detect a touched position by detecting a shadow image.

  As described above, the photodiode 6 may detect reflected light (shadow image) of the light emitted from the backlight 307 or may detect a shadow image caused by external light. Further, the two types of detection methods may be used in combination to detect both a shadow image and a reflected image at the same time.

(Data display / sensor configuration)
Next, the configuration of the main part of the data display / sensor device 100 will be described with reference to FIG. FIG. 5 is a block diagram showing a main configuration of the data display / sensor device 100. As illustrated, the data display / sensor device 100 includes one or more display / light sensor units 300, a circuit control unit 600, a data processing unit 700, a main control unit 800, a storage unit 901, a primary storage unit 902, and an operation unit. 903, an external communication unit 907, an audio output unit 908, and an audio input unit 909. Here, although the data display / sensor device 100 is described as including a single display / light sensor unit 300, a plurality of data display / sensor devices 100 may be included.

  The display / light sensor unit 300 is a so-called liquid crystal display device with a built-in light sensor. The display / light sensor unit 300 includes a sensor built-in liquid crystal panel 301, a backlight 307, and a peripheral circuit 309 for driving them.

  The sensor built-in liquid crystal panel 301 includes a plurality of pixel circuits 31 and photosensor circuits 32 arranged in a matrix. The detailed configuration of the sensor built-in liquid crystal panel 301 will be described later.

  The peripheral circuit 309 includes a liquid crystal panel drive circuit 304, an optical sensor drive circuit 305, a signal conversion circuit 306, and a backlight drive circuit 308.

  The liquid crystal panel driving circuit 304 outputs a control signal (G) and a data signal (S) in accordance with the timing control signal (TC1) and the data signal (D) from the display control unit 601 of the circuit control unit 600, and the pixel circuit 31. It is a circuit which drives. Details of the driving method of the pixel circuit 31 will be described later.

  The optical sensor driving circuit 305 is a circuit that drives the optical sensor circuit 32 by applying a voltage to the signal line (R) in accordance with a timing control signal (TC2) from the sensor control unit 602 of the circuit control unit 600. Details of the driving method of the optical sensor circuit 32 will be described later.

  The signal conversion circuit 306 is a circuit that converts the sensor output signal (SS) output from the optical sensor circuit 32 into a digital signal (DS) and transmits the converted signal to the sensor control unit 602.

  The backlight 307 includes a plurality of white LEDs (Light Emitting Diodes) and is disposed on the back surface of the sensor built-in liquid crystal panel 301. When a power supply voltage is applied from the backlight drive circuit 308, the backlight 307 is turned on and irradiates the sensor built-in liquid crystal panel 301 with light. Note that the backlight 307 is not limited to white LEDs, and may include LEDs of other colors. The backlight 307 may include, for example, a cold cathode fluorescent lamp (CCFL) instead of the LED.

  The backlight driving circuit 308 applies a power supply voltage to the backlight 307 when the control signal (BK) from the backlight control unit 603 of the circuit control unit 600 is at a high level, and conversely, the backlight control unit 603. When the control signal from is at a low level, no power supply voltage is applied to the backlight 307.

  Next, the circuit control unit 600 will be described. The circuit control unit 600 has a function as a device driver that controls the peripheral circuit 309 of the display / light sensor unit 300. The circuit control unit 600 includes a display control unit 601, a sensor control unit 602, a backlight control unit 603, and a display data storage unit 604.

  The display control unit 601 receives display data from the display data processing unit 701 of the data processing unit 700, and performs timing control on the liquid crystal panel driving circuit 304 of the display / light sensor unit 300 in accordance with an instruction from the display data processing unit 701. A signal (TC1) and a data signal (D) are transmitted, and the received display data is displayed on the sensor built-in liquid crystal panel 301.

  The display control unit 601 temporarily stores the display data received from the display data processing unit 701 in the display data storage unit 604. Then, a data signal (D) is generated based on the primary stored display data. The display data storage unit 604 is, for example, a video random access memory (VRAM).

  The sensor control unit 602 transmits a timing control signal (TC2) to the optical sensor driving circuit 305 of the display / optical sensor unit 300 in accordance with an instruction from the sensor data processing unit 703 of the data processing unit 700, and the sensor built-in liquid crystal panel 301. Run the scan with.

  In addition, the sensor control unit 602 receives a digital signal (DS) from the signal conversion circuit 306. Then, image data is generated based on the digital signal (DS) corresponding to the sensor output signal (SS) output from all the optical sensor circuits 32 included in the sensor built-in liquid crystal panel 301. That is, the image data read in the entire reading area of the sensor built-in liquid crystal panel 301 is generated. Then, the generated image data is transmitted to the sensor data processing unit 703.

  The backlight control unit 603 transmits a control signal (BK) to the backlight drive circuit 308 of the display / light sensor unit 300 in accordance with instructions from the display data processing unit 701 and the sensor data processing unit 703 to drive the backlight 307. Let

  When the data display / sensor device 100 includes a plurality of display / light sensor units 300, the display control unit 601 determines which display / light sensor unit 300 displays the display data from the data processing unit 700. When an instruction is received, the liquid crystal panel drive circuit 304 of the display / light sensor unit 300 is controlled according to the instruction. When the sensor control unit 602 receives an instruction from the data processing unit 700 as to which display / light sensor unit 300 is to scan the object, The sensor drive circuit 305 is controlled and a digital signal (DS) is received from the signal conversion circuit 306 of the display / light sensor unit 300 according to the instruction.

  Next, the data processing unit 700 will be described. The data processing unit 700 has a function as middleware that gives an instruction to the circuit control unit 600 based on a “command” received from the main control unit 800. Details of the command will be described later.

  The data processing unit 700 includes a display data processing unit 701 and a sensor data processing unit 703. When the data processing unit 700 receives a command from the main control unit 800, at least one of the display data processing unit 701 and the sensor data processing unit 703 depends on the value of each field (described later) included in the received command. One works.

  The display data processing unit 701 receives display data from the main control unit 800, and gives instructions to the display control unit 601 and the backlight control unit 603 according to the command received by the data processing unit 700, and displays the received display data. The image is displayed on the sensor built-in liquid crystal panel 301. The operation of the display data processing unit 701 according to the command will be described later.

  The sensor data processing unit 703 gives an instruction to the sensor control unit 602 and the backlight control unit 603 according to the command received by the data processing unit 700.

  The sensor data processing unit 703 receives image data from the sensor control unit 602 and stores the image data in the image data buffer 704 as it is. Then, in accordance with the command received by the data processing unit 700, the sensor data processing unit 703 performs “whole image data”, “partial image data (partial image coordinate data) based on the image data stored in the image data buffer 704. At least one of “including coordinate data” and “coordinate data” is transmitted to the main control unit 800. The whole image data, partial image data, and coordinate data will be described later. The operation of the sensor data processing unit 703 according to the command will be described later.

  Next, the main control unit 800 executes an application program. The main control unit 800 reads the program stored in the storage unit 901 into a primary storage unit 902 configured by, for example, a RAM (Random Access Memory) and executes the program.

  An application program executed by the main control unit 800 causes the data processing unit 700 to display display data on the sensor built-in liquid crystal panel 301 or to scan an object on the sensor built-in liquid crystal panel 301. Send commands and display data. When “data type” is designated as a command, at least one of whole image data, partial image data, and coordinate data is received from the data processing unit 700 as a response to the command.

  The circuit control unit 600, the data processing unit 700, and the main control unit 800 can be configured by a CPU (Central Processing Unit), a memory, and the like, respectively. The data processing unit 700 may be configured by a circuit such as an ASIC (application specific integrate circuit).

  Next, the storage unit 901 stores programs and data executed by the main control unit 800 as shown in the figure. The program executed by the main control unit 800 may be separated into an application-specific program and a general-purpose program that can be shared by each application.

  Next, the operation unit 903 receives an input operation of the user of the data display / sensor device 100. The operation unit 903 includes input devices such as a switch, a remote controller, a mouse, and a keyboard, for example. Then, the operation unit 903 generates a control signal corresponding to the user's input operation of the data display / sensor device 100, and transmits the generated control signal to the main control unit 800.

  As an example of the switch, a hardware switch such as a power switch 905 that switches power on and off and a user switch 906 to which a predetermined function is assigned in advance is assumed.

  In addition, the data display / sensor device 100 includes an external communication unit 907 for communicating with an external device by wireless / wired communication, an audio output unit 908 such as a speaker for outputting audio, and a microphone for inputting an audio signal. A voice input unit 909 such as the above may be provided as appropriate.

(Command details)
Next, details of commands transmitted from the main control unit 800 to the data processing unit 700 will be described with reference to FIGS. 6 and 7. FIG. 6 is a diagram schematically illustrating an example of a command frame structure. FIG. 7 is a diagram for explaining an example of values that can be specified in each field included in the command and an outline thereof.

  As shown in FIG. 6, the commands are “header”, “data acquisition timing”, “data type”, “scan method”, “scan image gradation”, “scan resolution”, “scan panel”, “display panel”. "And" Reserve "fields. In each field, for example, values shown in FIG. 7 can be designated.

  The “header” field is a field indicating the start of a frame. As long as it is possible to identify the “header” field, the value of the “header” field may be any value.

  Next, the “data acquisition timing” field is a field for designating a timing at which data should be transmitted to the main control unit 800. In the “data acquisition timing” field, for example, values “00” (sense), “01” (event), and “10” (all) can be specified.

  Here, “sense” specifies that the latest data is to be transmitted immediately. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “sense”, the latest data specified in the “data type” field is immediately updated to the main control unit 800. Send to.

  The “event” designates transmission at a timing when a change occurs in the image data received from the sensor control unit 602. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “event”, the image that receives the data specified in the “data type” field from the sensor control unit 602. The data is transmitted to the main control unit 800 at a timing when a change larger than a predetermined threshold occurs.

  “All” designates data transmission at a predetermined cycle. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “all”, the data designated in the “data type” field is transferred to the main control unit 800 at a predetermined cycle. Send to. The predetermined period coincides with the period in which the optical sensor circuit 32 performs scanning.

  Next, the “data type” field is a field for designating the type of data acquired from the sensor data processing unit 703. In the “data type” field, for example, values of “001” (coordinates), “010” (partial image), and “100” (entire image) can be specified. Furthermore, by adding these values, “coordinates” and “partial image” / “whole image” can be specified simultaneously. For example, when “coordinate” and “partial image” are specified at the same time, “011” can be specified.

  When the sensor data processing unit 703 receives a command whose value of the “data type” field is “whole image”, the sensor data processing unit 703 transmits the image data itself stored in the image data buffer 704 to the main control unit 800. The image data itself stored in the image data buffer 704 is referred to as “whole image data”.

  In addition, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 selects a portion where a change larger than a predetermined threshold has occurred from the image data received from the sensor control unit 602. A region to be included is extracted, and image data of the extracted region is transmitted to the main control unit 800. Here, the image data is referred to as “partial image data”. When a plurality of partial image data are extracted, the sensor data processing unit 703 transmits each extracted partial image data to the main control unit 800.

  Further, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 detects representative coordinates in the partial image data as the coordinates of the point to which the object approaches. The representative coordinates in the partial image data are detected, and coordinate data indicating the position of the representative coordinates in the partial image data is transmitted to the main control unit 800. The representative coordinates include, for example, the coordinates of the center of the partial image data, the coordinates of the center of gravity of the partial image data, and the like.

  Next, when receiving a command whose value of the “data type” field is “coordinate”, the sensor data processing unit 703 transmits coordinate data indicating the position of the representative coordinate in the entire image data to the main control unit 800. When a plurality of partial image data are extracted, the sensor data processing unit 703 detects representative coordinates in the entire image data of the extracted partial image data, and the coordinate data indicating the representative coordinates is detected. Each is transmitted to the main control unit 800 (multi-point detection).

  Specific examples of the whole image data, the partial image data, and the coordinate data will be described later with reference to schematic diagrams.

  Next, the “scan method” field is a field for designating whether or not the backlight 307 is turned on at the time of executing the scan. In the “scan method” field, for example, values of “00” (reflection), “01” (transmission), and “10” (reflection / transmission) can be designated.

  “Reflection” specifies that scanning is performed with the backlight 307 turned on. Therefore, when the sensor data processing unit 703 receives a command whose “scan method” field value is “reflection”, the sensor data processing unit 703 performs sensor control so that the optical sensor driving circuit 305 and the backlight driving circuit 308 operate in synchronization. An instruction is given to the unit 602 and the backlight control unit 603.

  “Transmission” specifies that scanning is performed with the backlight 307 turned off. Therefore, when the sensor data processing unit 703 receives a command whose “scan method” field value is “transparent”, the sensor control unit 602 operates the optical sensor driving circuit 305 and does not operate the backlight driving circuit 308. Instructions to the backlight control unit 603. Note that “reflection / transmission” specifies that scanning is performed using both “reflection” and “transmission”.

  Next, the “scanned image gradation” field is a field for designating gradations of the partial image data and the entire image data. In the “scanned image gradation” field, for example, values of “00” (binary) and “01” (multivalue) can be designated.

  When the sensor data processing unit 703 receives a command whose “scan image gradation” field value is “binary”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 as monochrome data. .

  When the sensor data processing unit 703 receives a command whose “scanned image gradation” field value is “multivalued”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 as multitone data. To do.

  Next, the “scan resolution” field is a field for designating the resolution of the partial image data and the entire image data. In the “resolution” field, for example, values of “0” (high) and “1” (low) can be designated.

  Here, “high” designates a high resolution. Therefore, when the sensor data processing unit 703 receives a command whose “scan resolution” field value is “high”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 with high resolution. For example, it is desirable to designate “high” for image data (image data such as a fingerprint) to be subjected to image processing such as image recognition.

  “Low” designates a low resolution. Therefore, when the sensor data processing unit 703 receives a command whose “scan resolution” field value is “low”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 at a low resolution. For example, it is desirable to designate “low” for image data (such as touched finger or hand image data) that only needs to be recognized.

  Next, in the “scan panel” field, when the data display / sensor device 100 includes a plurality of display / light sensor units 300, which display / light sensor unit 300 performs scanning of an object. This field is specified. In the “scan panel” field, for example, values “001” (first display / light sensor unit 300) and “010” (second display / light sensor unit 300) can be designated. By adding these values, a plurality of display / light sensor units 300 can be specified at the same time. For example, when both the first and second display / light sensor units 300 are specified at the same time, “011” can be specified.

  Here, the sensor data processing unit 703 controls the photo sensor driving circuit 305 and the backlight driving circuit 308 of the designated display / light sensor unit 300 according to the value of the “scan panel” field of the received command. An instruction is given to the sensor control unit 602 and the backlight control unit 603.

  Next, the “display panel” field specifies which display / light sensor unit 300 displays the display data when the data display / sensor device 100 includes a plurality of display / light sensor units 300. It is a field to be. In the “scanned image gradation” field, for example, values of “001” (first display / light sensor unit 300) and “010” (second display / light sensor unit 300) can be designated. By adding these values, a plurality of display / light sensor units 300 can be specified at the same time. For example, when both the first and second display / light sensor units 300 are specified at the same time, “011” can be specified.

  Here, for example, when the display data processing unit 701 receives a command whose value of the “display panel” field is the display / light sensor unit 300, the display / light sensor unit 300 displays the display / light sensor unit 300 to display the display data. An instruction is given to the display control unit 601 and the backlight control unit 603 so as to control the liquid crystal panel driving circuit 304 and the backlight driving circuit 308 of the optical sensor unit 300.

  Next, the “reserved” field is a field that is appropriately specified when it is necessary to further specify information other than information that can be specified in the above-described fields.

  Note that an application executed by the main control unit 800 does not need to use all the above-described fields when transmitting a command, and an invalid value (such as a NULL value) may be set for a field that is not used. .

  When the user wants to acquire coordinate data of a position touched with a finger or pen, a command specifying “coordinate” in the “data type” field is transmitted to the data processing unit 700. Therefore, it is desirable to specify “all” in the “data acquisition timing” field of the command to acquire coordinate data. Further, since it is only necessary to acquire coordinate data of the touched position, the scanning accuracy may not be high. Therefore, “low” may be specified as the value of the “resolution” field of the command.

  Further, when “coordinate” is specified in the “data type” field of the command, for example, when the user touches the sensor built-in liquid crystal panel 301 with a plurality of fingers or pens at the same time, the coordinate data of the touched position is used. Can be acquired (multi-point detection).

  When acquiring image data of an object such as a document, a command specifying “whole image” in the “data type” field is transmitted to the data processing unit 700. Since it is common to perform a scan in a stationary state, it is not necessary to periodically perform the scan. Therefore, in this case, it is desirable to designate “sense” or “event” in the “data acquisition timing” field. When scanning an object such as a document, it is desirable that the scanning accuracy is high so that the user can easily read the characters. Therefore, it is desirable to designate “high” in the “resolution” field.

(Whole image data / Partial image data / Coordinate data)
Next, the whole image data, the partial image data, and the coordinate data will be described with reference to FIG. The image data shown in FIG. 8A is image data obtained as a result of scanning the entire sensor-equipped liquid crystal panel 301 when the object is not placed on the sensor-equipped liquid crystal panel 301. 8B is image data obtained as a result of scanning the entire sensor-equipped liquid crystal panel 301 when the user touches the sensor-equipped liquid crystal panel 301 with a finger.

  When the user touches the sensor built-in liquid crystal panel 301 with a finger, the amount of light received by the photosensor circuit 32 in the vicinity of the touch changes, so that the voltage output from the photosensor circuit 32 changes, and as a result, In the image data generated by the sensor control unit 602, the brightness of the pixel value of the portion touched by the user changes.

  In the image data shown in FIG. 8B, the brightness of the pixel value of the portion corresponding to the user's finger is higher than that in the image data shown in FIG. In the image data shown in FIG. 8B, the smallest rectangular area (area PP) that includes all pixel values whose lightness changes more than a predetermined threshold value is “partial image data”.

  The image data indicated by the area AP is “whole image data”.

  Also, the coordinate data in the whole image data (area AP) of the representative coordinates Z of the partial image data (area PP) is (Xa, Ya), and the coordinate data in the partial image data (area PP) is (Xp, Yp). It is.

(Configuration of sensor built-in liquid crystal panel)
Next, the configuration of the sensor built-in liquid crystal panel 301 and the configuration of the peripheral circuit 309 of the sensor built-in liquid crystal panel 301 will be described with reference to FIG. FIG. 9 is a block diagram showing the main part of the display / light sensor unit 300, particularly the configuration of the sensor built-in liquid crystal panel 301 and the configuration of the peripheral circuit 309.

  The sensor built-in liquid crystal panel 301 includes a pixel circuit 31 for setting light transmittance (brightness) and an optical sensor circuit 32 that outputs a voltage corresponding to the intensity of light received by the sensor. Note that the pixel circuit 31 is a generic term for the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b corresponding to the red, green, and blue color filters, respectively.

  The pixel circuits 31 are arranged on the sensor built-in liquid crystal panel 301 in the column direction (vertical direction) and 3n in the row direction (horizontal direction). A set of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b is continuously arranged in the row direction (lateral direction). This set forms one pixel.

  In order to set the light transmittance of the pixel circuit 31, first, the high level voltage (TFT 33 is turned on) to the scanning signal line Gi connected to the gate terminal of the TFT (Thin Film Transistor) 33 included in the pixel circuit 31. Voltage). Thereafter, a predetermined voltage is applied to the data signal line SRj connected to the source terminal of the TFT 33 of the R pixel circuit 31r. Similarly, the light transmittance is also set for the G pixel circuit 31g and the B pixel circuit 31b. Then, by setting these light transmittances, an image is displayed on the sensor built-in liquid crystal panel 301.

  Next, the photosensor circuit 32 is arranged for each pixel. One pixel may be arranged in the vicinity of each of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b.

  In order for the optical sensor circuit 32 to output a voltage according to the light intensity, first, the sensor readout line RWi connected to one electrode of the capacitor 35 and the anode terminal of the photodiode 6 are connected. A predetermined voltage is applied to the sensor reset line RSi. In this state, when light is incident on the photodiode 6, a current corresponding to the amount of incident light flows through the photodiode 6. And according to the said electric current, the voltage of the connection point (henceforth connection node V) of the other electrode of the capacitor | condenser 35 and the cathode terminal of the photodiode 6 falls. When the power supply voltage VDD is applied to the voltage application line SDj connected to the drain terminal of the sensor preamplifier 37, the voltage at the connection node V is amplified and output from the source terminal of the sensor preamplifier 37 to the sensing data output line SPj. Based on the output voltage, the amount of light received by the optical sensor circuit 32 can be calculated.

  Next, the liquid crystal panel drive circuit 304, the optical sensor drive circuit 305, and the sensor output amplifier 44, which are peripheral circuits of the sensor built-in liquid crystal panel 301, will be described.

  The liquid crystal panel drive circuit 304 is a circuit for driving the pixel circuit 31, and includes a scanning signal line drive circuit 3041 and a data signal line drive circuit 3042.

  The scanning signal line driving circuit 3041 sequentially selects one scanning signal line from the scanning signal lines G1 to Gm for each line time based on the timing control signal TC1 received from the display control unit 601, and A high level voltage is applied to the selected scanning signal line, and a low level voltage is applied to the other scanning signal lines.

  Based on the display data D (DR, DG, and DB) received from the display controller 601, the data signal line driver circuit 3042 generates a predetermined voltage corresponding to the display data for one row for each line time. The data signal lines SR1 to SRn, SG1 to SGn, and SB1 to SBn are applied (line sequential method). Note that the data signal line driver circuit 3042 may be driven by a dot sequential method.

  The optical sensor driving circuit 305 is a circuit for driving the optical sensor circuit 32. Based on the timing control signal TC2 received from the sensor control unit 602, the optical sensor driving circuit 305 selects a predetermined sensor readout signal line from the sensor readout signal lines RW1 to RWm for each line time. A read voltage is applied, and a voltage other than a predetermined read voltage is applied to the other sensor read signal lines. Similarly, based on the timing control signal TC2, a predetermined reset voltage is applied to the sensor reset signal line selected from the sensor reset signal lines RS1 to RSm for each line time, and the others. A voltage other than a predetermined reset voltage is applied to the sensor reset signal line.

  The sensing data output signal lines SP1 to SPn are grouped into p groups (p is an integer of 1 to n), and the sensing data output signal lines belonging to each group are connected via a switch 47 that is sequentially turned on in time division. And connected to the sensor output amplifier 44. The sensor output amplifier 44 amplifies the voltage from the group of sensing data output signal lines connected by the switch 47 and outputs the amplified voltage to the signal conversion circuit 306 as sensor output signals SS (SS1 to SSp).

(Data display / more detailed configuration of sensor device)
Next, a more detailed configuration of the data display / sensor device 100 will be described with reference to FIG. Here, in order to make the description easy to understand, the description of the operations of the data processing unit 700 and the circuit control unit 600 located between the main control unit 800 and the display / light sensor unit 300 is omitted. However, to be precise, when displaying data and scanning an object, each unit of the main control unit 800 transmits a command to the data processing unit 700, and the data processing unit 700 sets the circuit control unit 600 based on the command. Then, the circuit control unit 600 transmits a signal to the display / light sensor unit 300. Further, the main control unit 800 acquires the entire image data, the partial image data, and the coordinate data from the data processing unit 700 as a response to the command transmitted to the data processing unit 700.

  FIG. 10 is a block diagram showing the configuration of the data display / sensor device 100 according to the present embodiment, centering on the main control unit 800 and the storage unit 901. As shown in FIG. 10, the main control unit 800 includes an instruction screen display control unit 11, an instruction coordinate detection unit (instruction coordinate detection unit) 12, and an auxiliary image display control unit (auxiliary image display control unit) 13. Yes. The auxiliary image display control unit 13 includes an auxiliary image display determination unit 131, an auxiliary image determination unit 132, and an auxiliary image display position determination unit 133. The storage unit 901 includes an instruction screen data storage unit 21, an instruction coordinate storage unit 22, an auxiliary image data storage unit 23, and an image data storage unit 24.

  Next, an example of the operation of the data display / sensor device 100 will be described with reference to FIGS.

  FIG. 11 is a flowchart for explaining the operation of the data display / sensor device 100 according to the present embodiment.

  In step S <b> 1, the data display / sensor device 100 displays an instruction screen on the sensor built-in liquid crystal panel 301. In one embodiment, the instructed screen display control unit 11 causes the sensor built-in liquid crystal panel 301 to display a screen keyboard 351 including a plurality of buttons 350.

  More specifically, first, the instructed screen display control unit 11 retrieves data from the instructed screen data storage unit 21 and acquires image data to be displayed. The data to be called from the instructed screen data storage unit 21 includes, for example, the coordinates of each button 350 (for example, (Xbi, Ybi) for the horizontal position and the vertical position of the button i) and the dimensions (for example, the button i (Wbi, Hbi)) and image data for the width in the horizontal direction and the width in the vertical direction, and by combining the image data of all buttons based on the respective coordinates and dimensions, Data can be generated. Further, the data of the entire image to be displayed (the entire image of the screen keyboard 351) may be directly called from the instructed screen data storage unit 21.

  Next, the instructed screen display control unit 11 sends the data of the entire screen to be displayed obtained as described above to the display data processing unit 701 and uses the “display panel” as the display / light sensor unit 300. Is sent to the data processing unit 700. As described above, the data processing unit 700 controls the display control unit 601 and the backlight control unit 603 to display the screen keyboard 351 on the sensor built-in liquid crystal panel 301.

  Note that step S <b> 1 may be started as necessary, for example, when the application is started.

  Next, in step S <b> 2, the data display / sensor device 100 detects the coordinates (indicated coordinates 65) of the previous point indicated by the finger 64 on the sensor built-in liquid crystal panel 301. In one embodiment, the designated coordinate detection unit 12 acquires the designated coordinate 65 from the sensor data processing unit 703 and stores it in the designated coordinate storage unit 22.

  Specifically, the designated coordinate detection unit 12 sends the above-mentioned command with “data type” as “coordinate” and “data acquisition timing” as “event” to the data processing unit 700. The designated coordinate detection unit 12 stores the coordinate obtained from the sensor data processing unit 703 as a response to the command in the designated coordinate storage unit 22 as the designated coordinate 65. When the designated coordinate detection unit 12 determines the data acquisition timing and the change in the acquired data, the “data acquisition timing” of the command may be “all” or “sense”. The designated coordinate detection unit 12 stores the designated coordinate 65 in the designated coordinate storage unit 22, and then causes the auxiliary image display control unit 13 to start step S3.

  Note that step S2 is started together with the start of the application, for example, similarly to step S1.

  Next, in step S <b> 3, the data display / sensor device 100 determines whether to display an auxiliary image on the sensor built-in liquid crystal panel 301. In one embodiment, the auxiliary image display determination unit 131 determines to display an auxiliary image when the indicated coordinate 65 indicates any of the buttons 350.

  Specifically, the auxiliary image display determination unit 131 reads the coordinates of each button 350 from the indicated screen data storage unit 21 (for example, (Xbi, Ybi) for the horizontal position and the vertical position of the button i), and Call a dimension (for example, (Wbi, Hbi) for the horizontal width and vertical width of the button i), and the point indicated by the indicated coordinate 65 (Xa, Ya) stored in the indicated coordinate storage unit 22 is It is determined which button 350 is inside and all buttons 350 are outside. When the designated coordinate 65 (Xa, Ya) is in any one of the buttons 350 (that is, for example, when Xbk ≦ Xa ≦ Xbk + Wbi and Ybk ≦ Ya ≦ Ybk + Hbk are satisfied for the button k), the auxiliary image is displayed. Decide to display. If the auxiliary image display determination unit 131 determines to display the auxiliary image, the auxiliary image display determination unit 131 causes the auxiliary image determination unit 132 to execute step S4.

  In one embodiment, the data display / sensor device 100 has two states of “selected state” and “non-selected state”, and the point indicated by the indicated coordinate 65 in one of the buttons 350 is as follows. Even if it exists, the auxiliary image 352 may not be displayed when the state of the data display / sensor device 100 is “selected”, and the auxiliary image 352 may be displayed only when “not selected”.

  “Selected state” refers to a state in which information to be input to the data display / sensor device 100 by the user via the screen keyboard 351 has been determined. “Unselected state” refers to a state in which the information has not been determined. Point to.

  In the selection of “unselected state” and “selected state”, for example, a state in which the finger 64 is not in contact with the sensor built-in liquid crystal panel 301 is referred to as “non-selected state”, and a state in which the finger 64 is in contact is referred to as “selected state”. be able to. Since the data display / sensor device 100 according to the present embodiment can detect the shadow or reflected light of the finger 64, the coordinates of the sensor built-in liquid crystal panel 301 to be pointed by the finger 64 even in the non-contact state. Can be detected.

  In addition, it is possible to set a “non-selection state” until it is released after being contacted once, and a “selected state” after it is released.

  Next, in step S4, the data display / sensor device 100 determines an auxiliary image 352 to be displayed on the sensor built-in liquid crystal panel 301. In one embodiment, the auxiliary image determination unit 132 calls data from the storage unit 901 to determine the auxiliary image 352.

  More specifically, the auxiliary image determination unit 132 determines, as the auxiliary image 352, an image associated with a button 350 (referred to as a button 350a) including the previous point indicated by the indicated coordinate 65 detected in step S3. For example, the auxiliary image determination unit 132 may call an image associated with the button 350a from the auxiliary image data storage unit 23 using information indicating the button 350a. For example, an image obtained by enlarging the button 350a can be used as the associated image. Further, the auxiliary image determination unit 132 may generate an image associated with the button 350a by calling the image data of the button 350a from the designated screen data storage unit 21 and processing it. As the processing, commonly used image decoration processing such as enlargement processing and processing for adding a lead line can be used. After determining the auxiliary image 352, the auxiliary image determination unit 132 causes the auxiliary image display position determination unit 133 to execute step S5.

  Next, in step S5, the data display / sensor device 100 determines the display position (Xs, Ys) of the auxiliary image 352 to be displayed on the sensor built-in liquid crystal panel 301. In one embodiment, the auxiliary image display position determination unit 133 displays the display position (Xs, Ys) based on the instruction coordinates 65 stored in the instruction coordinate storage unit 22 and the image of the finger 64 acquired from the sensor data processing unit 703. ).

  More specifically, first, the auxiliary image display position determination unit 133 calls the designated coordinates 65 from the designated coordinate storage unit 22. The auxiliary image display position determination unit 133 also sends the above-described command with “data type” as “whole image” and “data acquisition timing” as “sense” to the data processing unit 700, and the sensor data processing unit 703 sends the above-mentioned command. “Whole image data” (hereinafter referred to as image data) is acquired. Then, the obtained image data is stored in the image data storage unit 24. In step S2, the “whole image” may be acquired in advance at the same time as the coordinates.

  Next, the display position of the auxiliary image 352 is determined so that the auxiliary image 352 is displayed in the vicinity of the designated coordinates 65 and outside the area covered by the finger 64. An example of the method will be described below.

  The auxiliary image display position determination unit 133 specifies an area covered by the finger 64 on the sensor built-in liquid crystal panel 301 based on the image data stored in the image data storage unit 24. As described above, the image data is image data obtained as a result of scanning the entire sensor built-in liquid crystal panel 301. When the finger 64 covers the sensor built-in liquid crystal panel 301, the light in the covered area is displayed. Since the amount of light received by the sensor circuit 32 changes, a change occurs in the voltage output from the photosensor circuit 32. As a result, in the image data generated by the sensor control unit 602, the region of the area covered by the finger 64 is covered. A change occurs in the brightness of the pixel value. That is, when the “scan method” in the command for the scan is “00” (reflection), the brightness of the area is high, and the “scan method” is “01” (transmission). In some cases, the brightness of the region is low. Therefore, when the “scan method” is “00” (reflection), the brightness is higher than a predetermined threshold value. When the “scan method” is “01” (transmission), the brightness is An area lower than the predetermined threshold is specified as an area covered by the finger 64. As the identified result, for example, information on whether each pixel of the image data belongs to the region may be stored in the image data storage unit 24 in association with the image data.

  Subsequently, as shown in FIG. 12, candidates for the display position of the auxiliary image 352 are set (positions indicated by 352a to 352d in the figure). For example, the candidate position A, candidate position B, candidate position C, and candidate position D of the auxiliary image 352 so as to surround the button 350 indicated by the finger 64 (including the point indicated by the indicated coordinate 65). To decide. FIG. 12 shows auxiliary images 352a to 352d when displayed at each of the candidate positions A to D. When the position of the button 350 is (Xb, Yb) and the dimensions are (Wb, Hb), in one embodiment, the candidate position A is (Xb-Wb-3k, Yb-k), and the candidate position B is (Xb- k, Yb-Hb-3k), candidate position C can be (Xb + Wb + 2k, Yb-k), and candidate position D (Xb-k, Yb + Hb + 2k). Note that k is a constant, and the dimensions of the auxiliary image are (Wb + 2k, Hb + 2k).

  Next, it is determined whether the auxiliary image displayed at each candidate position overlaps the area covered by the identified finger 64. This determination is performed, for example, by calculating the percentage of pixels belonging to the area covered by the specified finger 64 among the pixels included in each auxiliary image candidate, and overlapping when the percentage is equal to or greater than a predetermined threshold. You may judge that it is. Then, among the candidate positions A to D, the candidate position determined that the auxiliary image does not overlap the image of the finger 64 is determined as the display position. For example, in the case of FIG. 12, candidate position B or candidate position C is determined as the display position. At this time, when the display position is selected from a plurality of candidate positions determined not to overlap the image of the finger 64, the display position may be selected based on the direction of the finger 64. That is, a candidate position that exists in a direction close to the direction of the finger 64 is selected as the display position from the designated coordinates 65. The direction of the finger 64 may be calculated as, for example, the direction opposite to the direction in which there are many pixels belonging to the area covered by the finger 64 from the designated coordinates 65. The auxiliary image display control unit 13 causes the instructed screen display control unit 11 to execute step S6 after the auxiliary image display position determination unit 133 determines the display position.

  The selection of the candidate positions is not limited to the positions described above, and is not limited to four. For example, it may be two places on the left and right or top and bottom, may be six places every 60 °, or may be more places.

  Next, in step S6, the data display / sensor device 100 causes the sensor built-in liquid crystal panel 301 to display the auxiliary image 352 determined in step S4 at the display position determined in step S5.

  In one embodiment, the instructed screen display control unit 11 displays the instructed screen (screen keyboard 351) that has been displayed by a known image composition technique based on the image data of the auxiliary image 352 and the display position. The auxiliary image 352 is synthesized to generate data for the entire image to be displayed and sent to the display data processing unit 701, and the above-mentioned command using the “display panel” as the display / light sensor unit 300 is sent to the data processing unit 700. send. As described above, the data processing unit 700 controls the display control unit 601 and the backlight control unit 603 to display the auxiliary image 352 together with the screen keyboard 351 on the sensor-equipped liquid crystal panel 301.

  Thereafter, the process returns to step S2 and waits for a new instruction from the user.

  The instructed screen displayed in step S1 does not need to be the screen keyboard 351, and can be various images. For example, buttons 350 corresponding to specific functions (for example, an OK button, a cancel button, etc.) may be displayed. In this case, what is necessary is just to perform similarly to the above about step S1-S6 except displaying the above buttons 350 in step S1. Further, it is not necessary to display the button 350 on the instructed screen. In that case, in step S4, as the auxiliary image 350, an image associated with the designated coordinate 65, for example, the vicinity of the point indicated by the designated coordinate 65 is enlarged. As with the data display / sensor device 100 according to the above-described embodiment, by making an image or the like, the information about the destination indicated by the finger 64 that is obscured by the finger 64 or the like and is difficult for the user to see. Can be successfully presented to the user. Therefore, the interaction between the image display / image detection apparatus and the user can be facilitated.

(Program and recording medium)
Finally, the circuit control unit 600, the data processing unit 700, and the main control unit 800 included in the data display / sensor device 100 may be configured by hardware logic. Alternatively, it may be realized by software using a CPU (Central Processing Unit) as follows.

  That is, the circuit control unit 600, the data processing unit 700, and the main control unit 800 are a CPU such as an MPU that executes instructions of a program that realizes each function, a ROM (Read Only Memory) that stores the program, and executes the program. A RAM (Random Access Memory) that expands into a possible format, and a storage device (recording medium) such as a memory that stores the program and various data are provided.

  The object of the present invention is not limited to the case where the program memory of the circuit control unit 600, the data processing unit 700, and the main control unit 800 is fixedly supported, but the program code of the above program (execution format program, intermediate code) Also, a recording medium in which a program or a source program is recorded is supplied to the data display / sensor device 100, and the data display / sensor device 100 reads and executes the program code recorded in the recording medium. Achievable.

  The recording medium is not limited to a specific structure or type. That is, the recording medium includes, for example, a tape system such as a magnetic tape and a cassette tape, a magnetic disk such as a floppy (registered trademark) disk / hard disk, and an optical disk such as a CD-ROM / MO / MD / DVD / CD-R. System, a card system such as an IC card (including a memory card) / optical card, or a semiconductor memory system such as a mask ROM / EPROM / EEPROM / flash ROM.

  The object of the present invention can be achieved even if the circuit control unit 600, the data processing unit 700, and the main control unit 800 (or the data display / sensor device 100) are configured to be connectable to a communication network. In this case, the program code is supplied to the circuit control unit 600, the data processing unit 700, and the main control unit 800 via the communication network. The communication network only needs to be able to supply program codes to the circuit control unit 600, the data processing unit 700, and the main control unit 800, and is not limited to a specific type or form. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication network, etc. may be used.

  The transmission medium constituting the communication network may be any medium that can transmit the program code, and is not limited to a specific configuration or type. For example, even with wired lines such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL (Asymmetric Digital Subscriber Line) line, infrared rays such as IrDA and remote control, Bluetooth (registered trademark), 802.11 wireless, HDR, mobile phone It can also be used by radio such as a telephone network, a satellite line, and a terrestrial digital network. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  The present invention has been specifically described above based on the embodiments. However, the present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims, and different implementations are possible. Embodiments obtained by appropriately combining the technical means disclosed in each form are also included in the technical scope of the present invention.

  The present invention can be used in various information devices.

1 is a diagram schematically illustrating the appearance of a data display / sensor device according to an embodiment of the present invention. It is a figure which shows typically an example of operation | movement of the data display / sensor apparatus which concerns on one Embodiment of this invention. It is a figure which shows typically the cross section of the liquid crystal panel with a built-in sensor with which the data display / sensor apparatus which concerns on one Embodiment of this invention is provided. FIG. 4A is a schematic diagram illustrating a state in which a position touched by the user is detected by detecting a reflected image on the sensor-equipped liquid crystal panel included in the data display / sensor device according to the embodiment of the present invention. is there. FIG. 4B is a schematic diagram illustrating a state in which a position touched by the user is detected by detecting a shadow image on the sensor-embedded liquid crystal panel included in the data display / sensor device according to the embodiment of the present invention. . It is a block diagram which shows the principal part structure of the data display / sensor apparatus which concerns on one Embodiment of this invention. It is a figure which shows typically an example of the frame structure of the command used with the data display / sensor apparatus which concerns on one Embodiment of this invention. It is a figure explaining an example of the value which can be specified to each field contained in the command shown in FIG. 6, and its outline. FIG. 8A is obtained as a result of scanning the entire liquid crystal panel with built-in sensor when the object is not placed on the liquid crystal panel with built-in sensor in the data display / sensor device according to one embodiment of the present invention. Image data. FIG. 8B shows image data obtained as a result of scanning when the user touches the sensor built-in liquid crystal panel with a finger in the data display / sensor device according to the embodiment of the present invention. It is a block diagram which shows the structure of the liquid crystal panel with a sensor with which the data display / sensor apparatus which concerns on one Embodiment of this invention is provided, and the structure of its peripheral circuit. It is a block diagram which shows the structure of the data display / sensor apparatus which concerns on one Embodiment of this invention centering on a main-control part and a memory | storage part. It is a flowchart explaining an example of operation | movement of the data display / sensor apparatus which concerns on one Embodiment of this invention. It is a figure which illustrates typically a part of example of operation | movement of the data display / sensor apparatus which concerns on one Embodiment of this invention.

Explanation of symbols

6 Photodiode 64 Finger (object)
65 Indicated coordinates 100 Data display / sensor device (image display / image detection device)
300 Display / light sensor unit 301 Liquid crystal panel with built-in sensor (planar member)
350 button (instructed image)
351 Screen keyboard 352 Auxiliary image 600 Circuit controller 601 Display controller 602 Sensor controller 603 Backlight controller 700 Data processor 701 Display data processor 703 Sensor data processor 800 Main controller 901 Storage unit

Claims (7)

A planar member for displaying an image and detecting a nearby image;
On the basis of the image of the object detected by the planar member, indicated coordinate detection means for detecting indicated coordinates on the planar member indicating the point to which the object approaches;
An area on the planar member that is covered by the object is identified based on the image of the object detected by the planar member, and an auxiliary image associated with the designated coordinates is outside the identified area. Auxiliary image display control means for displaying;
An image display / image detection apparatus comprising: The sheet member further includes an instruction image display means for displaying a plurality of instruction images,
The auxiliary image display control means displays an image associated with the pointed image including the point indicated by the pointed coordinate among the plurality of pointed images as the auxiliary image on the planar member. The image display / image detection apparatus according to claim 1.   3. The image display / image detection device according to claim 2, wherein each of the plurality of designated images indicates a character, a number, or a symbol, and a screen keyboard is formed on the planar member. .   The planar member includes a photodiode group that detects the amount of light received at each position on the planar member, and detects an image in the vicinity based on a change in the amount of received light detected by the photodiode group. The image display / image detection apparatus according to claim 1, wherein   The image display / image according to any one of claims 1 to 4, wherein the auxiliary image display control means displays the auxiliary image before the object contacts the planar member. Detection device. An image display / image detection apparatus provided with a planar member for displaying an image and detecting a nearby image is an auxiliary image display method for displaying an auxiliary image on the planar member,
Based on the image of the object detected by the planar member, an indicated coordinate detection step of detecting indicated coordinates on the planar member that indicate a point to which the object approaches,
An area on the planar member that is covered by the object is identified based on the image of the object detected by the planar member, and an auxiliary image associated with the designated coordinates is outside the identified area. A method for displaying an auxiliary image, comprising the step of displaying an auxiliary image.   A program for operating the image display / image detection apparatus according to any one of claims 1 to 5, wherein the program causes a computer to function as each of the means.

Images