JP2010109467A - Image display/image detection apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To read out a document or the like set on a display screen at high accuracy in a state that an image is still displayed on the whole display screen. <P>SOLUTION: An image display/image detection apparatus includes: a display/optical sensor unit 300 for displaying an image and reading out a nearby image; a display data generation unit 15 for displaying an image on the display/optical sensor unit 300; an object detection unit 131 for detecting a setting area on which an object is set in the display/optical sensor unit 300; and a monochromatic display control unit 133 which, when object reading is instructed during the display of the image on the display/optical sensor unit 300, instructs the display data generation part 15 to display the setting area on which the object is set by a predetermined monochromatic color. In the state that the setting area is displayed by the monochromatic color, the display/optical sensor unit 300 reads out an image of the object on the setting area. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image display / image detection apparatus capable of simultaneously displaying an image and reading an object.

2. Description of the Related Art Conventionally, an apparatus that can display an image and read a document or the like on the same screen is known. Patent Document 1 discloses a display device in which a plurality of light emitting elements for displaying an image and a plurality of light receiving elements for reading a document or the like placed on the display screen are arranged in a display screen. .
Japanese Patent Laid-Open No. 11-282411 (Publication date: October 15, 1999)

  In an apparatus that displays an image and reads an original on the same screen, when the original is read with high accuracy, it is necessary to prevent or eliminate the reflection of light emission for displaying the image. In the display device disclosed in Patent Document 1, when reading a document or the like, the light receiving element receives reflected light from the document or the like in a state where all the light emitting elements emit light, and the received reflected light is converted into an electric signal. Thus, after detecting the area of the original, only the light emitting element in the vicinity regarded as the original area is caused to emit light again, thereby reading the original or the like. That is, in this display device, the light emitting elements other than those in the vicinity of the document area temporarily stop emitting light during the reading period. For this reason, since the light receiving element does not receive the reflected light emitted to display the image, the reading accuracy of the document or the like can be improved.

  However, in the above display device, it is impossible to read a document or the like placed on the display screen while an image is displayed on the display screen. There is a problem that it is necessary to erase the image.

  In addition, when reading a document placed on the display screen while the image is displayed on the display screen, the image displayed in the region where the document is placed is read from the read image. It is conceivable to remove the reflection of light emission for displaying an image by using a method of subtracting (subtracting) by general image processing.

  However, there are cases in which the reflection is not completely removed even when the image processing is performed, and there is a problem that a load on a CPU (Central Processing Unit) that performs the image processing becomes high.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to read a document or the like placed on a display screen with high accuracy while an image is displayed on the display screen. An object of the present invention is to provide an image display / image detection apparatus capable of

  In order to solve the above-described problems, an image display / image detection device according to the present invention is an image display / image detection device including a planar member that displays an image and reads a nearby image. Image display control means for displaying an image on the sheet, object detection means for detecting a placement area where the object is placed on the planar member, and the image display control means on the planar member for the image When the reading of the object is instructed, the display color control that instructs the image display control means to display the placement area on which the object is placed in a predetermined single color And the planar member reads an image of the object in the placement area in a state where the placement area is displayed in the single color.

  According to said structure, an image can be displayed on the said planar member. And the mounting area | region where the target object was mounted with respect to the said planar member is detectable. When the image is displayed on the planar member and the reading of the object is instructed, the placement area on which the object is placed can be displayed in a predetermined single color. And the image of the said target object can be read in the above-mentioned placement area in the state where the above-mentioned placement area is displayed by the above-mentioned single color.

  Therefore, when the image is displayed on the planar member, after the object is placed on the planar member, the area where the object is placed is displayed in a predetermined single color. The image of the object can be read.

  Therefore, the user can read the image of the object while viewing the image displayed on the planar member.

  Furthermore, since the image of the object is read in a state where the area where the object is placed is displayed in a predetermined single color, the image displayed on the area where the object is placed is prevented from being reflected. As a result, the accuracy of reading the image of the object can be improved.

  Furthermore, since only the area where the object is placed is displayed in a predetermined single color, there is an effect that other visible display areas are not affected.

  Furthermore, the image display / image detection apparatus according to the present invention may be configured to instruct the image display control means to display the vicinity of the outer edge of the placement area in the single color in the above configuration.

  According to said structure, the vicinity of the outer edge of the said mounting area | region can be further displayed with the said single color.

  Therefore, when the image is displayed on the planar member, after the object is placed on the planar member, the area where the object is placed and the vicinity are displayed in a predetermined single color. In addition, the image of the object can be read.

  Therefore, the image displayed near the area where the object is placed can be prevented from being reflected, and as a result, the accuracy of reading the image of the object can be further improved. Play.

  Furthermore, in the image display / image detection apparatus according to the present invention, in the above configuration, the object detection unit further detects that the object is not placed on the planar member. When the object detection means detects that the object is not placed on the planar member, the display color control means stops the display in the single color and displays the image. It may be configured to instruct the display control means.

  According to said structure, it can further detect that the target object is not mounted in the said planar member. And when it detects that the target object is not mounted in the said planar member, the display in the said single color can be stopped and the said image can be displayed.

  Therefore, when the object is removed from the planar member, it is possible to return to a state before the object is placed (a state where an image is displayed on the planar member).

  Therefore, when the user removes the object from the planar member, the user can see the image displayed on the planar member in the same state as before placing the object.

  Furthermore, in the image display / image detection apparatus according to the present invention, in the above configuration, when the reading of the image of the object is completed by the planar member, the display color control unit is The image display control means may be instructed to stop the display and display the image.

  According to said structure, when reading of the image of the said target object is completed with the said planar member, the display in the said single color can be stopped and the said image can be displayed.

  Therefore, when the reading of the image of the object is completed, it is possible to return to the state before the object is placed (the state where the image is displayed on the planar member).

  Therefore, in particular, the user can see the image displayed in the area where the object is placed.

  Furthermore, the image display / image detection apparatus according to the present invention is a detection image storage that stores the image of the object image read by the planar member in the storage unit in a state where the image is mirror-inverted in the above configuration. It is good also as a structure provided with a part.

  According to said structure, the image of the said target object read with the said planar member can be further stored in a memory | storage part in the state which reversed the mirror image.

  Therefore, the image of the object can be stored in the storage unit in a normal state.

  Therefore, it is possible to separately read an image of the object image from the storage unit and execute a desired process using the read image.

  In addition, since the image of the object image is stored in the storage unit in a mirror image inverted state, it can be prevented from being read out in the state of the image of the object as it is (in a mirror image inverted state).

  Further, in the image display / image detection apparatus according to the present invention, in the above configuration, when the image display control unit displays an image on the planar member, the object displayed by the planar member is read. An image display control unit that instructs the image display control unit to display an image of the image on the planar member in a mirror image inverted state may be provided.

  According to the above configuration, the image display control means further displays an image of the image of the object read by the planar member on the planar member on the planar member in a mirror image inverted state. Can be made.

  Therefore, the image of the object can be displayed without interrupting or stopping the display of the image displayed on the planar member.

  Therefore, the user can confirm the image of the object while viewing the image displayed on the planar member.

  In addition, since the image of the object is displayed in a mirror-inverted state, it can be prevented from being displayed as it is (in the state of being mirror-inverted) as it is.

  Furthermore, in the image display / image detection apparatus according to the present invention, in the above configuration, a user operation for moving the display position of the image of the object displayed on the planar member by the planar member. The image display control means instructs the image display control means to move and display the displayed image of the object image to a position corresponding to the user operation. It is good.

  According to said structure, when the user operation which moves the display position of the image of the said object displayed on the said planar member is detected further, the image of the said image of the displayed object is detected. , It can be moved to a position corresponding to the user operation and displayed.

  Therefore, the display position of the image of the object can be changed according to the user operation.

  Therefore, the user has an effect that the region that the user wants to watch in the image displayed on the planar member and the image of the object image displayed on the planar member can be prevented from overlapping each other.

  The image display / image detection device may be realized by a computer. In this case, the image display / image detection device is realized by the computer by operating the computer as each of the above-described means. A control program for the image detection apparatus and a computer-readable recording medium on which the control program is recorded also fall within the scope of the present invention.

  As described above, the image display / image detection apparatus according to the present invention is an image display / image detection apparatus including a planar member that displays an image and reads a nearby image, and displays an image on the planar member. Image display control means for displaying, object detection means for detecting a placement area where the object is placed on the planar member, and the image display control means for displaying the image on the planar member. Display color control means for instructing the image display control means to display a placement area on which the object is placed in a predetermined single color when reading of the object is instructed. The planar member reads an image of the object in the placement area in a state where the placement area is displayed in the single color.

  Therefore, when the image is displayed on the planar member, after the object is placed on the planar member, the area where the object is placed is displayed in a predetermined single color. The image of the object can be read.

  Therefore, the user can read the image of the object while viewing the image displayed on the planar member.

  Furthermore, since the image of the object is read in a state where the area where the object is placed is displayed in a predetermined single color, the image displayed on the area where the object is placed is prevented from being reflected. As a result, the accuracy of reading the image of the object can be improved.

  Furthermore, since only the area where the object is placed is displayed in a predetermined single color, there is an effect that other visible display areas are not affected.

  The data display / sensor device (image display / image detection device) 100 according to the present invention can be broadly described as a sensor built-in liquid crystal panel (planar member) capable of displaying an image and detecting a nearby image. 301 (which will be described later) is an apparatus capable of reading an image of an object while displaying an image or the like on the sensor built-in liquid crystal panel 301.

  In the present specification, the object refers to an object to be read. Specifically, a document in which characters, images, and the like are written is assumed. For example, business cards, various materials, books, cards, flyers, and the like correspond to the objects.

  In addition, reading of an image of an object is also referred to as a scan below.

(Key points of the invention)
The main feature of the data display / sensor device 100 is that when the object placed on the sensor built-in liquid crystal panel 301 displaying an image such as a moving image or a still image is scanned, the object is placed on the sensor built-in liquid crystal panel 301. The object is scanned while the area is displayed in a predetermined single color. At this time, the area other than the placement area is in a state where the image is displayed.

  With the above feature, the user can scan the object while viewing the image. Further, since the object is scanned in a state where the placement area and its vicinity are displayed in a single color, the object is scanned without being affected by the image displayed on the sensor-equipped liquid crystal panel 301. Can do. That is, since the image can be prevented from being reflected during scanning, the scanning accuracy can be improved.

  Note that when the object is scanned while the image is displayed on the placement area, the light of the image may be reflected by the object, and the image may be reflected in the scan data. Lower.

  Further, as described above, only the placement area may be displayed in a single color, but it is desirable to display the placement area and its vicinity in a single color. This is because the reflection of the image displayed in the vicinity of the placement area can be prevented, and the scanning accuracy can be further improved.

  Note that the above-mentioned vicinity is a region outside the placement region and is a region within a predetermined range from the outer edge of the placement region. Further, the area of the predetermined range is an area having a size capable of preventing the reflection of an image or the like when scanning in the placement area.

  Further, the data display / sensor device 100 is further characterized in that when the object is removed from the sensor built-in liquid crystal panel 301 or when the object scan is completed, the placement area and its vicinity are displayed in a single color. Is displayed, and the image displayed before displaying a single color is displayed.

  Therefore, in particular, after removing the object, the user can continue to see the image in a state before placing the object (a state where an image such as a moving image or a still image is displayed).

  In addition, as described above, when the object is scanned, the placement area and its vicinity are displayed in a single color, and when the object is removed or when the object scan is finished, the placement area is displayed. Since the display in a single color in the area and its vicinity is stopped, it is not necessary to always provide a single color display area for scanning the object, and the display area of the image is limited. Absent.

  Further, the scan data of the object may be displayed superimposed on the image at a position on the sensor built-in liquid crystal panel 301 that does not overlap the placement area. Thereby, the user can confirm the scan data while viewing the image.

  Furthermore, the display position of the displayed scan data may be moved by a user operation. Accordingly, the user can prevent the scan data from overlapping the area in the image that the user wants to watch. It is assumed that the user operation is performed by dragging the sensor built-in liquid crystal panel 301 with a finger or a pen from a state in which the user touches the scan data with a finger or a pen.

(Outline of LCD panel with built-in sensor)
The sensor built-in liquid crystal panel 301 provided in the data display / sensor device 100 is a liquid crystal panel capable of detecting an image of an object in addition to displaying data. Here, the image detection of the object is, for example, detection of a position pointed (touched) by the user with a finger or a pen, or reading (scanning) of an image of a printed material. The device used for display is not limited to a liquid crystal panel, and may be an organic EL (Electro Luminescence) panel or the like.

  The structure of the sensor built-in liquid crystal panel 301 will be described with reference to FIG. FIG. 2 is a diagram schematically showing a cross section of the sensor built-in liquid crystal panel 301. The sensor built-in liquid crystal panel 301 described here is an example, and any structure can be used as long as the display surface and the reading surface are shared.

  As shown in the figure, the sensor built-in liquid crystal panel 301 includes an active matrix substrate 51A disposed on the back surface side and a counter substrate 51B disposed on the front surface side, and has a structure in which a liquid crystal layer 52 is sandwiched between these substrates. is doing. The active matrix substrate 51A is provided with a pixel electrode 56, a data signal line 57, an optical sensor circuit 32 (not shown), an alignment film 58, a polarizing plate 59, and the like. The counter substrate 51B is provided with color filters 53r (red), 53g (green), 53b (blue), a light shielding film 54, a counter electrode 55, an alignment film 58, a polarizing plate 59, and the like. In addition, a backlight 307 is provided on the back surface of the sensor built-in liquid crystal panel 301.

  The photodiode 6 included in the photosensor circuit 32 is provided in the vicinity of the pixel electrode 56 provided with the blue color filter 53b, but is not limited to this configuration. It may be provided in the vicinity of the pixel electrode 56 provided with the red color filter 53r, or may be provided in the vicinity of the pixel electrode 56 provided with the green color filter 53g.

  Next, with reference to FIGS. 3A and 3B, two types of methods for detecting the position where the user touches the sensor built-in liquid crystal panel 301 with a finger or a pen will be described.

  FIG. 3A is a schematic diagram illustrating a state in which a position touched by the user is detected by detecting a reflected image. When the light 63 is emitted from the backlight 307, the optical sensor circuit 32 including the photodiode 6 detects the light 63 reflected by the object 64 such as a finger. Thereby, the reflected image of the target object 64 can be detected. Thus, the sensor built-in liquid crystal panel 301 can detect the touched position by detecting the reflected image.

  FIG. 3B is a schematic diagram illustrating a state in which a position touched by the user is detected by detecting a shadow image. As shown in FIG. 3B, the optical sensor circuit 32 including the photodiode 6 detects external light 61 transmitted through the counter substrate 51B and the like. However, when there is an object 62 such as a pen, the incident of the external light 61 is hindered, so that the amount of light detected by the optical sensor circuit 32 is reduced. Thereby, a shadow image of the object 62 can be detected. Thus, the sensor built-in liquid crystal panel 301 can also detect a touched position by detecting a shadow image.

  As described above, the photodiode 6 may detect reflected light (shadow image) of the light emitted from the backlight 307 or may detect a shadow image caused by external light. Further, the two types of detection methods may be used in combination to detect both a shadow image and a reflected image at the same time.

(Data display / sensor configuration)
Next, with reference to FIG. 4, the configuration of the main part of the data display / sensor device 100 will be described. FIG. 4 is a block diagram showing a main configuration of the data display / sensor device 100. As shown in FIG. As illustrated, the data display / sensor device 100 includes one or more display / light sensor units 300, a circuit control unit 600, a data processing unit 700, a main control unit 800, a storage unit 901, a primary storage unit 902, and an operation unit. 903, an external communication unit 907, an audio output unit 908, and an audio input unit 909.

  The display / light sensor unit 300 is a so-called liquid crystal display device with a built-in light sensor. The display / light sensor unit 300 includes a sensor built-in liquid crystal panel 301, a backlight 307, and a peripheral circuit 309 for driving them.

  The sensor built-in liquid crystal panel 301 includes a plurality of pixel circuits 31 and photosensor circuits 32 arranged in a matrix. The detailed configuration of the sensor built-in liquid crystal panel 301 will be described later.

  The peripheral circuit 309 includes a liquid crystal panel drive circuit 304, an optical sensor drive circuit 305, a signal conversion circuit 306, and a backlight drive circuit 308.

  The liquid crystal panel driving circuit 304 outputs a control signal (G) and a data signal (S) in accordance with the timing control signal (TC1) and the data signal (D) from the display control unit 601 of the circuit control unit 600, and the pixel circuit 31. It is a circuit which drives. Details of the driving method of the pixel circuit 31 will be described later.

  The optical sensor driving circuit 305 is a circuit that drives the optical sensor circuit 32 by applying a voltage to the signal line (R) in accordance with a timing control signal (TC2) from the sensor control unit 602 of the circuit control unit 600. Details of the driving method of the optical sensor circuit 32 will be described later.

  The signal conversion circuit 306 is a circuit that converts the sensor output signal (SS) output from the optical sensor circuit 32 into a digital signal (DS) and transmits the converted signal to the sensor control unit 602.

  The backlight 307 includes a plurality of white LEDs (Light Emitting Diodes) and is disposed on the back surface of the sensor built-in liquid crystal panel 301. When a power supply voltage is applied from the backlight drive circuit 308, the backlight 307 is turned on and irradiates the sensor built-in liquid crystal panel 301 with light. Note that the backlight 307 is not limited to white LEDs, and may include LEDs of other colors. The backlight 307 may include, for example, a cold cathode fluorescent lamp (CCFL) instead of the LED.

  The backlight driving circuit 308 applies a power supply voltage to the backlight 307 when the control signal (BK) from the backlight control unit 603 of the circuit control unit 600 is at a high level, and conversely, the backlight control unit 603. When the control signal from is at a low level, no power supply voltage is applied to the backlight 307.

  Next, the circuit control unit 600 will be described. The circuit control unit 600 has a function as a device driver that controls the peripheral circuit 309 of the display / light sensor unit 300. The circuit control unit 600 includes a display control unit 601, a sensor control unit 602, a backlight control unit 603, and a display data storage unit 604.

  The display control unit 601 receives display data from the display data processing unit 701 of the data processing unit 700, and performs timing control on the liquid crystal panel driving circuit 304 of the display / light sensor unit 300 in accordance with an instruction from the display data processing unit 701. A signal (TC1) and a data signal (D) are transmitted, and the received display data is displayed on the sensor built-in liquid crystal panel 301.

  The display control unit 601 temporarily stores the display data received from the display data processing unit 701 in the display data storage unit 604. Then, a data signal (D) is generated based on the primary stored display data. The display data storage unit 604 is, for example, a video random access memory (VRAM).

  The sensor control unit 602 transmits a timing control signal (TC2) to the optical sensor driving circuit 305 of the display / optical sensor unit 300 in accordance with an instruction from the sensor data processing unit 703 of the data processing unit 700, and the sensor built-in liquid crystal panel 301. Run the scan with.

  In addition, the sensor control unit 602 receives a digital signal (DS) from the signal conversion circuit 306. Then, image data is generated based on the digital signal (DS) corresponding to the sensor output signal (SS) output from all the optical sensor circuits 32 included in the sensor built-in liquid crystal panel 301. That is, the image data read in the entire reading area of the sensor built-in liquid crystal panel 301 is generated. Then, the generated image data is transmitted to the sensor data processing unit 703.

  The backlight control unit 603 transmits a control signal (BK) to the backlight drive circuit 308 of the display / light sensor unit 300 in accordance with instructions from the display data processing unit 701 and the sensor data processing unit 703 to drive the backlight 307. Let

  When the data display / sensor device 100 includes a plurality of display / light sensor units 300, the display control unit 601 determines which display / light sensor unit 300 displays the display data from the data processing unit 700. When an instruction is received, the liquid crystal panel drive circuit 304 of the display / light sensor unit 300 is controlled according to the instruction. When the sensor control unit 602 receives an instruction from the data processing unit 700 as to which display / light sensor unit 300 is to scan the object, The sensor drive circuit 305 is controlled and a digital signal (DS) is received from the signal conversion circuit 306 of the display / light sensor unit 300 according to the instruction.

  Next, the data processing unit 700 will be described. The data processing unit 700 has a function as middleware that gives an instruction to the circuit control unit 600 based on a “command” received from the main control unit 800. Details of the command will be described later.

  The data processing unit 700 includes a display data processing unit 701 and a sensor data processing unit 703. When the data processing unit 700 receives a command from the main control unit 800, at least one of the display data processing unit 701 and the sensor data processing unit 703 depends on the value of each field (described later) included in the received command. One works.

  The display data processing unit 701 receives display data from the main control unit 800, and gives instructions to the display control unit 601 and the backlight control unit 603 according to the command received by the data processing unit 700, and displays the received display data. The image is displayed on the sensor built-in liquid crystal panel 301. The operation of the display data processing unit 701 according to the command will be described later.

  The sensor data processing unit 703 gives an instruction to the sensor control unit 602 and the backlight control unit 603 according to the command received by the data processing unit 700.

  The sensor data processing unit 703 receives image data from the sensor control unit 602 and stores the image data in the image data buffer 704 as it is. Then, in accordance with the command received by the data processing unit 700, the sensor data processing unit 703 performs “whole image data”, “partial image data (partial image coordinate data) based on the image data stored in the image data buffer 704. At least one of “including coordinate data” and “coordinate data” is transmitted to the main control unit 800. The whole image data, partial image data, and coordinate data will be described later. The operation of the sensor data processing unit 703 according to the command will be described later.

  Next, the main control unit 800 executes an application program. The main control unit 800 reads the program stored in the storage unit 901 into a primary storage unit 902 configured by, for example, a RAM (Random Access Memory) and executes the program.

  An application program executed by the main control unit 800 causes the data processing unit 700 to display display data on the sensor built-in liquid crystal panel 301 or to scan an object on the sensor built-in liquid crystal panel 301. Send commands and display data. When “data type” is designated as a command, at least one of whole image data, partial image data, and coordinate data is received from the data processing unit 700 as a response to the command.

  The circuit control unit 600, the data processing unit 700, and the main control unit 800 can be configured by a CPU (Central Processing Unit), a memory, and the like, respectively. The data processing unit 700 may be configured by a circuit such as an ASIC (application specific integrate circuit).

  Next, the storage unit 901 stores programs and data executed by the main control unit 800 as shown in the figure. The program executed by the main control unit 800 may be separated into an application-specific program and a general-purpose program that can be shared by each application.

  Next, the operation unit 903 receives an input operation of the user of the data display / sensor device 100. The operation unit 903 includes input devices such as a switch, a remote controller, a mouse, and a keyboard, for example. Then, the operation unit 903 generates a control signal corresponding to the user's input operation of the data display / sensor device 100, and transmits the generated control signal to the main control unit 800.

  As an example of the switch, a hardware switch such as a power switch 905 that switches power on and off and a user switch 906 to which a predetermined function is assigned in advance is assumed.

  In addition, the data display / sensor device 100 includes an external communication unit 907 for communicating with an external device by wireless / wired communication, an audio output unit 908 such as a speaker for outputting audio, and a microphone for inputting an audio signal. A voice input unit 909 such as the above may be provided as appropriate.

(Command details)
Next, details of commands transmitted from the main control unit 800 to the data processing unit 700 will be described with reference to FIGS. 5 and 6. FIG. 5 is a diagram schematically illustrating an example of a command frame structure. FIG. 6 is a diagram for explaining an example of values that can be specified for each field included in the command and an outline thereof.

  As shown in FIG. 5, the commands are “header”, “data acquisition timing”, “data type”, “scan method”, “scan image gradation”, “scan resolution”, “scan panel”, “display panel”. "And" Reserve "fields. In each field, for example, values shown in FIG. 6 can be designated.

  The “header” field is a field indicating the start of a frame. As long as it is possible to identify the “header” field, the value of the “header” field may be any value.

  Next, the “data acquisition timing” field is a field for designating a timing at which data should be transmitted to the main control unit 800. In the “data acquisition timing” field, for example, values “00” (sense), “01” (event), and “10” (all) can be specified.

  Here, “sense” specifies that the latest data is to be transmitted immediately. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “sense”, the latest data specified in the “data type” field is immediately updated to the main control unit 800. Send to.

  The “event” designates transmission at a timing when a change occurs in the image data received from the sensor control unit 602. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “event”, the image that receives the data specified in the “data type” field from the sensor control unit 602. The data is transmitted to the main control unit 800 at a timing when a change larger than a predetermined threshold occurs.

  “All” designates data transmission at a predetermined cycle. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “all”, the data designated in the “data type” field is transferred to the main control unit 800 at a predetermined cycle. Send to. The predetermined period coincides with the period in which the optical sensor circuit 32 performs scanning.

  Next, the “data type” field is a field for designating the type of data acquired from the sensor data processing unit 703. In the “data type” field, for example, values of “001” (coordinates), “010” (partial image), and “100” (entire image) can be specified. Furthermore, by adding these values, “coordinates” and “partial image” / “whole image” can be specified simultaneously. For example, when “coordinate” and “partial image” are specified at the same time, “011” can be specified.

  When the sensor data processing unit 703 receives a command whose value of the “data type” field is “whole image”, the sensor data processing unit 703 transmits the image data itself stored in the image data buffer 704 to the main control unit 800. The image data itself stored in the image data buffer 704 is referred to as “whole image data”.

  In addition, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 selects a portion where a change larger than a predetermined threshold has occurred from the image data received from the sensor control unit 602. A region to be included is extracted, and image data of the extracted region is transmitted to the main control unit 800. Here, the image data is referred to as “partial image data”. When a plurality of partial image data are extracted, the sensor data processing unit 703 transmits each extracted partial image data to the main control unit 800.

  Further, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 detects representative coordinates in the partial image data and indicates the position of the representative coordinates in the partial image data. The coordinate data is transmitted to the main control unit 800. The representative coordinates include, for example, the coordinates of the center of the partial image data, the coordinates of the center of gravity of the partial image data, and the like.

  Next, when receiving a command whose value of the “data type” field is “coordinate”, the sensor data processing unit 703 transmits coordinate data indicating the position of the representative coordinate in the entire image data to the main control unit 800. When a plurality of partial image data are extracted, the sensor data processing unit 703 detects representative coordinates in the entire image data of the extracted partial image data, and the coordinate data indicating the representative coordinates is detected. Each is transmitted to the main control unit 800 (multi-point detection).

  Specific examples of the whole image data, the partial image data, and the coordinate data will be described later with reference to schematic diagrams.

  Next, the “scan method” field is a field for designating whether or not the backlight 307 is turned on at the time of executing the scan. In the “scan method” field, for example, values of “00” (reflection), “01” (transmission), and “10” (reflection / transmission) can be designated.

  “Reflection” specifies that scanning is performed with the backlight 307 turned on. Therefore, when the sensor data processing unit 703 receives a command whose “scan method” field value is “reflection”, the sensor data processing unit 703 performs sensor control so that the optical sensor driving circuit 305 and the backlight driving circuit 308 operate in synchronization. An instruction is given to the unit 602 and the backlight control unit 603.

  “Transmission” specifies that scanning is performed with the backlight 307 turned off. Therefore, when the sensor data processing unit 703 receives a command whose “scan method” field value is “transparent”, the sensor control unit 602 operates the optical sensor driving circuit 305 and does not operate the backlight driving circuit 308. Instructions to the backlight control unit 603. Note that “reflection / transmission” specifies that scanning is performed using both “reflection” and “transmission”.

  Next, the “scanned image gradation” field is a field for designating gradations of the partial image data and the entire image data. In the “scanned image gradation” field, for example, values of “00” (binary) and “01” (multivalue) can be designated.

  When the sensor data processing unit 703 receives a command whose “scan image gradation” field value is “binary”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 as monochrome data. .

  When the sensor data processing unit 703 receives a command whose “scanned image gradation” field value is “multivalued”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 as multitone data. To do.

  Next, the “scan resolution” field is a field for designating the resolution of the partial image data and the entire image data. In the “resolution” field, for example, values of “0” (high) and “1” (low) can be designated.

  Here, “high” designates a high resolution. Therefore, when the sensor data processing unit 703 receives a command whose “scan resolution” field value is “high”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 with high resolution. For example, it is desirable to designate “high” for image data (image data such as a fingerprint) to be subjected to image processing such as image recognition.

  “Low” designates a low resolution. Therefore, when the sensor data processing unit 703 receives a command whose “scan resolution” field value is “low”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 at a low resolution. For example, it is desirable to designate “low” for image data (such as touched finger or hand image data) that only needs to be recognized.

  Next, the “scan panel” field is a field for designating which display / light sensor unit 300 is to scan the object. In the “scan panel” field, for example, values of “001” (first display / light sensor unit 300) and “010” (second display / light sensor unit 300) can be designated. By adding these values, a plurality of display / light sensor units 300 can be specified at the same time. For example, when the first and second display / light sensor units 300 are specified at the same time, “011” can be specified.

  Here, the sensor data processing unit 703 controls the light sensor driving circuit 305 and the backlight driving circuit 308 of the display / light sensor unit 300 specified in the “scan panel” field of the received command. An instruction is given to 602 and the backlight control unit 603.

  Next, the “display panel” field is a field for designating which display / light sensor unit 300 displays the display data. In the “scanned image gradation” field, for example, values of “001” (first display / light sensor unit 300) and “010” (second display / light sensor unit 300) can be designated. By adding these values, a plurality of display / light sensor units 300 can be specified at the same time. For example, when the first and second display / light sensor units 300 are specified at the same time, “011” can be specified.

  For example, when the display data processing unit 701 receives a command whose value of the “display panel” field is the first display / light sensor unit 300, the display data processing unit 701 displays the display data on the first display / light sensor unit 300. Therefore, an instruction is given to the display control unit 601 and the backlight control unit 603 so as to control the liquid crystal panel driving circuit 304 and the backlight driving circuit 308 of the display / light sensor unit 300.

  Next, the “reserved” field is a field that is appropriately specified when it is necessary to further specify information other than information that can be specified in the above-described fields.

  Note that an application executed by the main control unit 800 does not need to use all the above-described fields when transmitting a command, and an invalid value (such as a NULL value) may be set for a field that is not used. .

  When the user wants to acquire coordinate data of a position touched with a finger or pen, a command specifying “coordinate” in the “data type” field is transmitted to the data processing unit 700. Therefore, it is desirable to specify “all” in the “data acquisition timing” field of the command to acquire coordinate data. Further, since it is only necessary to acquire coordinate data of the touched position, the scanning accuracy may not be high. Therefore, “low” may be specified as the value of the “resolution” field of the command.

  Further, when “coordinate” is specified in the “data type” field of the command, for example, when the user touches the sensor built-in liquid crystal panel 301 with a plurality of fingers or pens at the same time, the coordinate data of the touched position is used. Can be acquired (multi-point detection).

  When acquiring image data of an object such as a document, a command specifying “whole image” in the “data type” field is transmitted to the data processing unit 700. Since it is common to perform a scan in a stationary state, it is not necessary to periodically perform the scan. Therefore, in this case, it is desirable to designate “sense” or “event” in the “data acquisition timing” field. When scanning an object such as a document, it is desirable that the scanning accuracy is high so that the user can easily read the characters. Therefore, it is desirable to designate “high” in the “resolution” field.

(Whole image data / Partial image data / Coordinate data)
Next, the whole image data, the partial image data, and the coordinate data will be described with reference to FIG. The image data shown in FIG. 7A is image data obtained as a result of scanning the entire sensor built-in liquid crystal panel 301 when the object is not placed on the sensor built-in liquid crystal panel 301. The image data shown in FIG. 7B is image data obtained as a result of scanning the entire sensor-equipped liquid crystal panel 301 when the user touches the sensor-equipped liquid crystal panel 301 with a finger.

  When the user touches the sensor built-in liquid crystal panel 301 with a finger, the amount of light received by the photosensor circuit 32 in the vicinity of the touch changes, so that the voltage output from the photosensor circuit 32 changes, and as a result, In the image data generated by the sensor control unit 602, the brightness of the pixel value of the portion touched by the user changes.

  In the image data shown in FIG. 7B, the brightness of the pixel value of the portion corresponding to the user's finger is higher than that in the image data shown in FIG. In the image data shown in FIG. 7B, the smallest rectangular area (area PP) that includes all pixel values whose lightness changes more than a predetermined threshold is “partial image data”.

  The image data indicated by the area AP is “whole image data”.

  Also, the coordinate data in the whole image data (area AP) of the representative coordinates Z of the partial image data (area PP) is (Xa, Ya), and the coordinate data in the partial image data (area PP) is (Xp, Yp). It is.

(Configuration of sensor built-in liquid crystal panel)
Next, the configuration of the sensor built-in liquid crystal panel 301 and the configuration of the peripheral circuit 309 of the sensor built-in liquid crystal panel 301 will be described with reference to FIG. FIG. 8 is a block diagram showing the main part of the display / light sensor unit 300, particularly the configuration of the sensor built-in liquid crystal panel 301 and the configuration of the peripheral circuit 309.

  The sensor built-in liquid crystal panel 301 includes a pixel circuit 31 for setting light transmittance (brightness) and an optical sensor circuit 32 that outputs a voltage corresponding to the intensity of light received by the sensor. Note that the pixel circuit 31 is a generic term for the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b corresponding to the red, green, and blue color filters, respectively.

  The pixel circuits 31 are arranged on the sensor built-in liquid crystal panel 301 in the column direction (vertical direction) and 3n in the row direction (horizontal direction). A set of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b is continuously arranged in the row direction (lateral direction). This set forms one pixel.

  In order to set the light transmittance of the pixel circuit 31, first, the high level voltage (TFT 33 is turned on) to the scanning signal line Gi connected to the gate terminal of the TFT (Thin Film Transistor) 33 included in the pixel circuit 31. Voltage). Thereafter, a predetermined voltage is applied to the data signal line SRj connected to the source terminal of the TFT 33 of the R pixel circuit 31r. Similarly, the light transmittance is also set for the G pixel circuit 31g and the B pixel circuit 31b. Then, by setting these light transmittances, an image is displayed on the sensor built-in liquid crystal panel 301.

  Next, the photosensor circuit 32 is arranged for each pixel. One pixel may be arranged in the vicinity of each of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b.

  In order for the optical sensor circuit 32 to output a voltage corresponding to the light intensity, first, the sensor readout line RWi connected to one electrode of the capacitor 35 and the anode terminal of the photodiode 36 are connected. A predetermined voltage is applied to the sensor reset line RSi. In this state, when light is incident on the photodiode 36, a current corresponding to the amount of incident light flows through the photodiode 36. Then, according to the current, the voltage at the connection point (hereinafter referred to as connection node V) between the other electrode of the capacitor 35 and the cathode terminal of the photodiode 36 decreases. When the power supply voltage VDD is applied to the voltage application line SDj connected to the drain terminal of the sensor preamplifier 37, the voltage at the connection node V is amplified and output from the source terminal of the sensor preamplifier 37 to the sensing data output line SPj. Based on the output voltage, the amount of light received by the optical sensor circuit 32 can be calculated.

  Next, the liquid crystal panel drive circuit 304, the optical sensor drive circuit 305, and the sensor output amplifier 44, which are peripheral circuits of the sensor built-in liquid crystal panel 301, will be described.

  The liquid crystal panel drive circuit 304 is a circuit for driving the pixel circuit 31, and includes a scanning signal line drive circuit 3041 and a data signal line drive circuit 3042.

  The scanning signal line driving circuit 3041 sequentially selects one scanning signal line from the scanning signal lines G1 to Gm for each line time based on the timing control signal TC1 received from the display control unit 601, and A high level voltage is applied to the selected scanning signal line, and a low level voltage is applied to the other scanning signal lines.

  Based on the display data D (DR, DG, and DB) received from the display controller 601, the data signal line driver circuit 3042 generates a predetermined voltage corresponding to the display data for one row for each line time. The data signal lines SR1 to SRn, SG1 to SGn, and SB1 to SBn are applied (line sequential method). Note that the data signal line driver circuit 3042 may be driven by a dot sequential method.

  The optical sensor driving circuit 305 is a circuit for driving the optical sensor circuit 32. Based on the timing control signal TC2 received from the sensor control unit 602, the optical sensor driving circuit 305 selects a predetermined sensor readout signal line from the sensor readout signal lines RW1 to RWm for each line time. A read voltage is applied, and a voltage other than a predetermined read voltage is applied to the other sensor read signal lines. Similarly, based on the timing control signal TC2, a predetermined reset voltage is applied to the sensor reset signal line selected from the sensor reset signal lines RS1 to RSm for each line time, and the others. A voltage other than a predetermined reset voltage is applied to the sensor reset signal line.

  The sensing data output signal lines SP1 to SPn are grouped into p groups (p is an integer of 1 to n), and the sensing data output signal lines belonging to each group are connected via a switch 47 that is sequentially turned on in time division. And connected to the sensor output amplifier 44. The sensor output amplifier 44 amplifies the voltage from the group of sensing data output signal lines connected by the switch 47 and outputs the amplified voltage to the signal conversion circuit 306 as sensor output signals SS (SS1 to SSp).

[Embodiment 1]
An embodiment of the present invention will be described below with reference to FIGS. 1 and 9 to 15. Here, a case where the data display / sensor device 100 has a function of performing a video conference with a video conference device on a communication network will be described as an example. Although the data display / sensor device 100 according to the present embodiment is configured to include one display / light sensor unit 300, a plurality of data display / sensor devices 100 may be provided.

  A schematic configuration of a video conference system including the data display / sensor device 100 and a video conference device 1030 on a communication network will be described with reference to FIG. FIG. 9 is a schematic diagram showing a schematic configuration of the video conference system. The data display / sensor device 100 and the video conference device 1030 are communicably connected to the communication network 1020 and can perform a video conference with each other using a generally known technique. And Then, an image of the user of the data display / sensor device 100 captured by the imaging device 1010 connected to the data display / sensor device 100 is displayed on the display unit 1031 of the video conference device 1030, and conversely, the television It is assumed that the video of the user of the video conference device 1030 captured by the imaging device 1032 connected to the conference device 1030 is displayed on the sensor built-in liquid crystal panel 301 of the data display / sensor device 100. The video conference apparatus 1030 includes a data storage unit 1033 that stores various data.

  In the video conference system having the above-described configuration, when a video conference is being performed between the data display / sensor device 100 and the video conference device 1030, a document possessed by the user of the data display / sensor device 100 is obtained. A case will be described as an example in which scanning is performed by the sensor built-in liquid crystal panel 301 of the data display / sensor device 100 and the resulting image data is transmitted to the user of the video conference device 1030. At this time, since the video conference is being performed, it is desirable not to interrupt the display of the image or the like of the other party (the user of the video conference device 1030) displayed on the sensor built-in liquid crystal panel 301.

(Data display / scanning object using sensor device)
Next, a typical example of processing in which the data display / sensor device 100 scans an object while displaying an image will be described with reference to FIGS. Here, an outline of the processing will be described, and details of each processing will be described later.

  First, a specific example of an object to be scanned will be described with reference to FIG. As shown in the figure, here, a material P on which characters, diagrams, etc. are written is assumed as an object to be scanned. FIG. 10A is a diagram showing the surface of the material P. FIG. On the surface of the material P, letters and diagrams are written. Note that when the material P is scanned, the surface of the material P is scanned. FIG. 10B is a diagram showing the back surface of the material P.

  Next, with reference to FIG. 11 and FIG. 12, a state in which the data P is scanned while the video is displayed using the data display / sensor device 100 will be described.

  First, FIG. 11A is a diagram showing a state in which an image of the other party of the video conference (user of the video conference device 1030) is displayed on the sensor built-in liquid crystal panel 301. FIG.

  Next, FIG. 11B is a diagram illustrating a state in which the user places the material P on the sensor built-in liquid crystal panel 301 for the purpose of scanning the material P. In addition, in order to scan the surface of the material P, the surface of the material P is placed with the surface facing the liquid crystal panel 301 with a built-in sensor. Therefore, the user of the data display / sensor device 100 is in a state where the back surface of the material P is visible as illustrated.

  Next, FIG. 11C shows that the data display / sensor device 100 detects that the material P is placed and displays the placement region of the material P and its vicinity in a single color such as white. FIG. As shown in the drawing, in the placement area of material P and in the vicinity thereof (referred to as area SA), the video of the other party of the video conference is not displayed, but is displayed in a single color.

  It is assumed that how much area is allocated as the neighborhood is determined in advance. In addition, when a nearby area is not assigned, only the placement area may be displayed in a single color.

  Then, the surface of the material P is scanned with the area SA displayed in a single color. Thus, by displaying the placement area and its vicinity in a single color, it is possible to scan the material P without being influenced by the video of the other party of the video conference, and to improve the scanning accuracy. Can be increased.

  Since the video of the other party of the video conference is still displayed except for the area where the object is placed and the vicinity thereof, the user scans the material P while viewing the video of the other party of the video conference. be able to.

  Next, FIG. 11D shows the scan data SCD, which is image data obtained as a result of scanning the material P in the state shown in FIG. 11C, superimposed on the video of the other party of the video conference. It is a figure which shows a mode. At this time, the scan data SCD is displayed at a position that does not overlap the area SA.

  Since the surface of the material P is scanned, the scan data SCD is image data of the surface of the material P.

  When an object is scanned, generally, image data having a mirror image relationship with the scanned surface of the object is obtained, and thus the obtained image data is obtained by a generally known method. The mirrored image data is displayed and stored.

  Next, FIG. 11E is a diagram illustrating a state where the user removes the material P from the sensor built-in liquid crystal panel 301 after the scanning of the material P is completed. As shown in the figure, in the area SA, the display in the single color is stopped and the video of the other party of the video conference is displayed.

  Next, FIG. 11 (f) is a diagram showing a state in which the display position of the scan data SCD is changed by the user moving the scan data SCD with the hand F by dragging the hand F from the state of FIG. 11 (e). It is. Thus, the scan data SCD can be moved to an arbitrary position on the sensor built-in liquid crystal panel 301.

  Next, FIG. 12 is a diagram illustrating a state in which the scan data SCD is transferred to a device (video conference device 1030) on the other side of the video conference using a predetermined user operation as a trigger. Here, it is assumed that a user operation for transferring the scan data SCD to the video conference device 1030 is performed in the state shown in FIG. At this time, the scan data SCD is stored in the data storage unit 1033 of the video conference device 1030 via the communication network 1020 and then displayed on the display unit 1031 of the video conference device 1030.

(Mounting area and its vicinity)
Next, an example of the placement area and the vicinity thereof will be described with reference to FIG. FIG. 13 is a schematic diagram illustrating an example of the placement area and the vicinity thereof. Here, it is assumed that the shape of the material P, which is the object, is a rectangle, the horizontal length is X, and the vertical length is Y.

  First, an area RA in the sensor built-in liquid crystal panel 301 is an area on which the material P is placed. Since the region RA has the same shape as the material P, the region RA is a rectangle whose horizontal length is X and whose vertical length is Y.

  The vicinity of the placement area is an area outside the placement area, and is an area within a predetermined range from the outer edge of the placement area. If the length of the predetermined range is W, the lateral length is X + 2W. A rectangular area SA having a vertically long Y + 2W is a placement area and its vicinity.

  Note that the value of W may be determined in advance or may be set by the user. Further, the shape of the object is not limited to a rectangle. For this reason, the shapes of the region RA and the region SA are not limited to rectangles, and are shapes corresponding to the shapes of the countermeasures.

(Data display / more detailed configuration of sensor device)
Next, a more detailed configuration of the data display / sensor device 100 will be described with reference to FIG. Here, in order to make the description easy to understand, the description of the operations of the data processing unit 700 and the circuit control unit 600 located between the main control unit 800 and the display / light sensor unit 300 is omitted. However, to be precise, when displaying data and scanning an object, each unit of the main control unit 800 transmits a command to the data processing unit 700, and the data processing unit 700 sets the circuit control unit 600 based on the command. Then, the circuit control unit 600 transmits a signal to the display / light sensor unit 300. Further, the main control unit 800 acquires the entire image data, the partial image data, and the coordinate data from the data processing unit 700 as a response to the command transmitted to the data processing unit 700.

  FIG. 1 is a block diagram showing a more detailed configuration of the data display / sensor device 100. As illustrated, the storage unit 901 includes a display color information storage unit 21 and a scan data storage unit 22.

  The display color information storage unit 21 stores in advance color information capable of specifying a single color to be displayed in the placement area and its neighboring areas. Here, in particular, white, red, green, blue, and the like are assumed as single colors. The color information stored in the display color information storage unit 21 is not particularly limited as long as it is information that can specify a single color. For example, a set of R value, G value, B value (for example, “255, 255, 255”) may be stored. The color information may be determined in advance or may be set by the user.

  The scan data storage unit 22 stores scan data of an object in a readable state. In order to be able to read the scan data, it is desirable that the scan data is stored in the scan data storage unit 22 in association with the identification information. The identification information may be any information that can uniquely specify scan data, and includes information including a storage date and time, for example.

  Next, the main control unit 800 will be described. As illustrated, the main control unit 800 includes an object scan processing unit, a display data generation unit 15, a data transfer processing unit 17, and a video conference communication processing unit 19.

  First, the object scan processing unit 13 will be described. The object scan processing unit 13 scans the object placed on the sensor built-in liquid crystal panel 301 and displays the scan data on the sensor built-in liquid crystal panel 301. For this purpose, the object scan processing unit includes an object detection unit 131, a placement area information holding unit 132, a single color display control unit 133, an object scan execution unit 134, a scan data holding unit 135, and a scan data display control unit 136. And a scan data storage unit 137.

  The object detection unit 131 first detects that the object has been placed on the sensor built-in liquid crystal panel 301 and that the object has been removed from the sensor built in liquid crystal panel 301.

  The detection is performed by scanning the entire sensor-equipped liquid crystal panel 301 at a predetermined cycle, and continuous in time among image data obtained as a result (that is, the entire image data obtained from the data processing unit 700 at a predetermined cycle). This is done by examining the difference between the two image data obtained in this way.

  Here, when the object is not placed on the sensor built-in liquid crystal panel 301, the reflected image of the object is not detected. For example, as shown in FIG. The brightness of each pixel value of the data is almost constant. Even when an image or the like is displayed on the sensor built-in liquid crystal panel 301, it may be considered that a reflected image of the image or the like is not detected when an object is not placed. On the other hand, when the object is placed on the sensor built-in liquid crystal panel 301, a reflected image of the object is detected. For example, as shown in FIG. 7B, the part where the object is placed. The brightness of the pixel value of is changed.

  Accordingly, when two pieces of image data obtained in succession in time among the whole image data obtained in a predetermined cycle are compared, a region in which the brightness of the pixel value has changed by a predetermined threshold or more in the previous image data in time And the object detection unit 131 places the object on the sensor-equipped liquid crystal panel 301 when there is an area in which the brightness of the pixel value has changed by a predetermined threshold value or more in the image data after time. You can see that

  In addition, there is a region in which the brightness of the pixel value changes by a predetermined threshold or more in the temporally previous image data, and the brightness of the pixel value changes by a predetermined threshold or more in the temporally subsequent image data. When there is no area, the object detection unit 131 knows that the object has been removed from the sensor-equipped liquid crystal panel 301.

  Note that the object detection unit 131 acquires the partial image data transmitted from the data processing unit 700 at a timing when the image data generated by the sensor control unit 602 changes, so that the object is a liquid crystal panel 301 with a built-in sensor. You may detect having mounted on the top.

  Next, the object detection unit 131 recognizes that the object has been placed on the sensor built-in liquid crystal panel 301, recognizes the placement area, and represents the recognized recognition area (hereinafter referred to as placement). Area information) is acquired, and the acquired placement area information is held in the placement area information holding unit 132 in a readable manner.

  The placement area information includes, for example, image data of the placement area itself, data indicating the shape of the placement area (for example, coordinates serving as feature points on the outer edge of the placement area), representative coordinates of the placement area (for example, , Center coordinates and barycentric coordinates), or a combination thereof.

  The image data of the placement area can be acquired by extracting an area where the brightness of the pixel value is changed by a predetermined threshold or more from the entire image data. Further, the data indicating the shape of the placement area and the representative coordinates of the placement area can be obtained by image analysis of the placement area image data by a general method. When the placement area is a rectangle, the data indicating the shape of the placement area is, for example, coordinate data of four vertices of the rectangle.

  Note that the timing for acquiring the placement area information is as follows: (1) The object detection unit 131 immediately detects that the object has been placed on the sensor built-in liquid crystal panel 301 as a trigger. Information may be acquired, or (2) placement area information may be obtained after a predetermined time (T1) has elapsed while the object is placed on the sensor built-in liquid crystal panel 301. . When a predetermined time is required from the start of placing the object to the end of placement, after a predetermined time T1 (for example, several seconds) has elapsed since it was detected that the object was placed on the sensor built-in liquid crystal panel 301 It is desirable to acquire the placement area information.

  When the position and size of the region in which the brightness of the pixel value has changed by a predetermined threshold or more in the entire image data acquired at a predetermined cycle has not changed for a predetermined time T1, the object is a sensor. It is determined that the predetermined time T1 has elapsed while being placed on the built-in liquid crystal panel 301.

  Further, after the object is placed on the sensor built-in liquid crystal panel 301, the placement area information may be acquired using a user operation as a trigger.

  Note that the detection by the object detection unit 131 that the object is placed on the sensor built-in liquid crystal panel 301 is performed by a predetermined user operation (for example, a predetermined button pressed through the operation unit 903). It may be started as a trigger.

  Next, the single color display control unit 133 will be described. The single color display control unit 133 controls to display the placement area and its vicinity in the sensor built-in liquid crystal panel 301 in a single color according to a predetermined trigger. The predetermined trigger is not particularly limited as long as it indicates that reading of an object has been instructed. For example, (1) the object detection unit 131 may hold the placement region information in the placement region information holding unit 132 as the predetermined trigger, or (2) the user via the operation unit 903 The predetermined trigger may be an instruction to read an object.

  For this purpose, the single color display control unit 133 first reads color information preset in the display color information storage unit 21. Next, the placement area information is read from the placement area information holding unit 132, the placement area is calculated based on the read placement area information, and the vicinity of the placement area is calculated.

  Thereafter, the single color display control unit 133 notifies the display data generation unit 15 of an instruction to display the calculated placement area and its vicinity in a single color based on the color information read from the display color information storage unit 21. .

  Note that the above-mentioned vicinity is a region outside the placement region and is a region within a predetermined range from the outer edge of the placement region. The predetermined range may be determined in advance, or may be settable by the user.

  In addition, the single color display control unit 133 (1) the object detection unit 131 detects that the object has been removed from the sensor built-in liquid crystal panel 301, or (2) the object scan execution unit 134. However, the display data generation unit 15 is notified of an instruction to stop the display of the placement area and the vicinity thereof in a single color using the scan data holding unit 135 as a trigger.

  Next, the object scan execution unit 134 will be described. The object scan execution unit 134 uses the single-color display control unit 133 as a trigger to notify the display data generation unit 15 of an instruction to display the placement area and the vicinity thereof in a single color. Run a scan. Then, when the entire image data is acquired from the data processing unit 700 as a response, the image data of the region portion of the object in the acquired entire image data is extracted. The object scan execution unit 134 reads the placement area information from the placement area information holding unit 132, and determines the area of the object based on the read placement area information.

  Then, the object scan execution unit 134 causes the scan data holding unit 135 to hold the extracted image data as scan data in a readable manner.

  Here, when the characteristics of the photosensor are high in sensitivity to red, the single color displayed in the mounting area and its vicinity is set to red, and the obtained single red image is applied to green and blue.・ By combining, it is possible to obtain highly accurate monochrome images. The same applies when the characteristics of the optical sensor are high in sensitivity to green or blue.

  Note that the image data obtained by mirror-inversion of the extracted image data may be held in the scan data holding unit 135 as scan data, or at this time, it may be held in the scan data holding unit 135 without being mirror-inverted.

  Next, the scan data display control unit 136 will be described. The scan data display control unit 136 controls the display of the scan data, the redisplay of the scan data according to the user's movement operation, and the non-display of the scan data according to the user's non-display operation.

  The scan data display control unit 136 reads the latest scan data held in the scan data holding unit 135 and notifies the display data generation unit 15 of an instruction to display the read scan data on the sensor built-in liquid crystal panel 301. . At this time, the scan data display control unit 136 reads the placement area information from the placement area information holding unit 132 and can display the scan data so as not to overlap the placement area based on the read placement area information. The display data generation unit 15 is notified of an instruction to calculate a correct position and display the scan data at the position. In order to make it easier for the user to check the scan data, the distance between the scan data display area and the placement area is preferably as short as possible.

  When the object detection unit 131 detects that the object has been removed from the sensor built-in liquid crystal panel 301, the scan data display control unit 136 may display the scan data again in the placement area. In this case, the scan data display control unit 136 reads the placement area information from the placement area information holding unit 132 and displays an instruction to display the scan data on the placement area indicated by the read placement area information. Notify the generation unit 15. At the same time, the display data generation unit 15 is instructed to return the display of the area where the scan data was displayed before the scan data is displayed again to the placement area to the state before the scan data is displayed in the area. To do.

  When the scan data held in the scan data holding unit 135 is not in a mirror image inverted state, the scan data display control unit 136 reads the scan data held in the scan data holding unit 135, and The display data generation unit 15 is notified of an instruction to reverse the mirror image of the read scan data by general image processing, and to display the mirror-inverted scan data on the sensor built-in liquid crystal panel 301.

  Next, when the scan data display control unit 136 detects that a user operation for moving the scan data displayed on the sensor built-in liquid crystal panel 301 is performed, the scan data display control unit 136 instructs the scan data to be displayed again according to the user operation. The display data generation unit 15 is notified. It is assumed that the user operation is performed by tracing the sensor built-in liquid crystal panel 301 with a finger or a pen from a state in which the user touches the scan data with a finger or a pen.

  Here, when the user touches the sensor built-in liquid crystal panel 301 with a finger, a pen, or the like, the optical sensor circuit 32 detects a reflected image of the finger, the pen, or the like. Coordinate data indicating the touched position can be acquired. Therefore, when the position indicated by the acquired coordinate data is within the area where the scan data is displayed on the sensor built-in liquid crystal panel 301, it is understood that the user has touched the scan data.

  If coordinate data such as a user's finger or pen is acquired from the data processing unit 700, a user operation for moving scan data (a liquid crystal panel with a built-in sensor from a state in which the scan data is touched with a finger or pen) Based on the coordinate data acquired during the period during which the operation 301 is performed), the position to which the scan data is moved can be obtained.

  Therefore, the scan data display control unit 136 notifies the display data generation unit 15 of an instruction to redisplay the scan data from the position where the scan data was displayed immediately before to the position of the movement destination.

  Next, the scan data storage unit 137 will be described. The scan data storage unit 137 stores the scan data held in the scan data holding unit 135 in the scan data storage unit 22 triggered by a user operation for storing the scan data. The scan data storage unit 137 desirably stores the scan data and the identification information of the scan data in the scan data storage unit 22 in a state of being associated with each other. The identification information uniquely identifies the scan data, and may be any information that can identify the scan data when reading the scan data from the scan data storage unit 22, and includes, for example, the storage date and time. Things.

  If the scan data held in the scan data holding unit 135 is not in a mirror image inverted state, the scan data storage unit 137 reads the scan data held in the scan data holding unit 135 and then The read scan data is mirror-inverted by general image processing, and the scan data that has been mirror-inverted is stored in the scan data storage unit 22.

  Next, the display data generation unit 15 will be described. The display data generation unit 15 displays the image received from the multipoint connection transmission unit 11 on the sensor built-in liquid crystal panel 301, and in accordance with an instruction from the object scan processing unit 13, the placement area and its vicinity are displayed in a single color. Display data is generated so that the display data is displayed on the display or the scan data is displayed on the sensor built-in liquid crystal panel 301, and the generated display data is transmitted to the data processing unit 700.

  First, when there is no instruction from the object scan processing unit 13 and only video data is received from the multipoint connection transmission unit 11, the display data generation unit 15 displays the received video data on the sensor built-in liquid crystal panel 301. Display data to be displayed is generated, and the generated display data is transmitted to the data processing unit 700.

  Next, when an instruction to display the placement area and its vicinity in a single color is received from the single color display control unit 133 of the object scan processing unit 13, the display data generation unit 15 places the placement area on the basis of the instruction. Display data for displaying the region and its vicinity in a single color is generated, and the generated display data is transmitted to the data processing unit 700. When video data is received from the multipoint connection transmission unit 11, only the placement area and its vicinity are displayed in a single color, and display data is generated to display the video data in the other areas. Then, the generated display data is transmitted to the data processing unit 700.

  The display data generation unit 15 displays display data for displaying the placement region and its vicinity in a single color until receiving an instruction from the single color display control unit 133 to stop displaying the placement region and its vicinity in a single color. Continue to generate.

  On the other hand, when receiving an instruction from the single color display control unit 133 of the object scan processing unit 13 to stop the display in the single color of the placement area and its vicinity, the display data generation unit 15 Display data that does not display (return to the original state) in a single color in the mounting area and its vicinity is generated, and the generated display data is transmitted to the data processing unit 700. Therefore, when video data is received from the multipoint connection transmission unit 11, display data for displaying the video data is generated in the placement area and the vicinity thereof, and the generated display data is used as the data processing unit. To 700.

  Next, upon receiving an instruction to display scan data from the scan data display control unit 136 of the object scan processing unit 13, the display data generation unit 15 generates display data for displaying the scan data based on the instruction. The generated display data is transmitted to the data processing unit 700. When video data is received from the multipoint connection transmission unit 11, display data in which scan data is superimposed on the video data is generated, and the generated display data is transmitted to the data processing unit 700.

  The display data generation unit 15 continues to generate display data on which the scan data is superimposed until an instruction to stop displaying the scan data is received by a user operation or the like.

  Next, the data transfer processing unit 17 will be described. The data transfer processing unit 17 reads the scan data to be transferred stored in the scan data holding unit 135 or the scan data storage unit 22 triggered by a user operation for transferring the scan data to the external device, The read scan data is transmitted to the external device via the external communication unit 907. The external device is, for example, a device of a video conference partner.

  When transferring the scan data being displayed to the sensor built-in liquid crystal panel 301, the scan data is read from the scan data holding unit 135, while when the scan data stored in the scan data storage unit 22 is transferred, The scan data is read from the scan data storage unit 22 by, for example, causing the user to select scan data to be transferred.

  Next, the video conference communication processing unit 19 will be described. The video conference communication processing unit 19 performs various processes for realizing a video conference between the own device and one or a plurality of video conference devices on the communication network. Various processes for realizing the video conference are generally known processes, and are not an essential part of the present invention.

  The video conference communication processing unit 19 is used to realize a video conference with a video conference device on a communication network or between multiple points on a communication network by a user operating an operation unit 903 such as a remote controller. It communicates with a multipoint connection control device that performs various controls, and transmits and receives audio and video.

  The video conference communication processing unit 19 receives the audio signal input by the audio input unit 909 such as a microphone and the video signal captured by the imaging device 1010 such as a camera, and outputs the audio and video signals. Then, via the external communication unit 907, the data is transmitted to a video conference device on the communication network or a multi-point connection control device that performs various controls for realizing one video conference between multiple points on the communication network.

  In addition, the audio and video signals of the other party are received from the video conference device on the communication network or the multipoint connection control device via the external communication unit 907, and the audio data is displayed on the audio output unit 908. Video data is transmitted to the data generator 15.

  The video conference communication processing unit 19 may include a multi-point connection control unit 191 that performs various controls for realizing one video conference between multi-points on the communication network, and is provided on the communication network. The multi-point connection control device having a function corresponding to the multi-point connection control unit 191 may be connected to be communicable via the external communication unit 907.

  The various user operations are assumed to be performed by operating the operation unit 903 (switch, remote controller, mouse, keyboard, etc.), but the present invention is not limited to this, and the sensor built-in liquid crystal panel 301 is used. The menu image or button image displayed above may be touched by the user. In this case, it is understood that the user has touched the menu image, the button image, etc. by receiving the coordinate data located in the area displaying the menu image, the button image, etc. from the data processing unit 700. .

(Operation example of the main control unit)
Next, referring to FIG. 11 again, an operation example of the main control unit 800 will be described while touching commands transmitted from the main control unit 800 to the data processing unit 700.

  Since the data display / sensor device 100 according to the present embodiment includes one display / light sensor unit 300, when displaying data on the display / light sensor unit 300, a “display panel” is used. It is assumed that a command specifying “first display / light sensor unit (001)” as a field value is transmitted to the data processing unit 700, and when scanning is performed by the display / light sensor unit 300, “scan panel” It is assumed that a command designating “first display / light sensor unit (001)” as the value of the “” field is transmitted to the data processing unit 700.

  First, as shown in FIG. 11A, in order to display an image on the sensor built-in liquid crystal panel 301, the display data generating unit 15 displays the image data received from the video conference communication processing unit 19 on the sensor built-in liquid crystal panel 301. Display data for display is generated, and the generated display data is transmitted to the data processing unit 700. In addition, the display data generation unit 15 transmits a command designating “first display / light sensor unit (001)” as the value of the “display panel” field to the data processing unit 700.

  Next, as shown in FIG. 11B, in order to detect that the material P, which is the object, is placed, the object detection unit 131 designates “all” in the “data acquisition timing” field in advance. Then, a command specifying “whole image” in the “data type” field and “first display / light sensor unit (001)” in the “scan panel” field is transmitted to the data processing unit 700. It is desirable to specify “reflection” in the “scan method” field. Further, the “scanned image gradation” field may be “high” or “low”.

  As described above, the entire image data is obtained from the data processing unit 700 in a predetermined cycle. Then, the object detection unit 131 compares two pieces of image data obtained continuously in time among the entire image data obtained in a predetermined cycle, and the brightness of the pixel value is compared with the previous image data in terms of time. When there is no region that has changed by a predetermined threshold value or more and there is a region in which the brightness of a pixel value has changed by a predetermined threshold value or more in temporally subsequent image data, it is detected that the material P has been placed.

  As another method for detecting that the material P is placed on the sensor built-in liquid crystal panel 301, the object detection unit 131 designates “event” in the “data acquisition timing” field and sets “data type”. A command in which “partial image” is designated in the field and “first display / light sensor unit (001)” is designated in the “scan panel” field may be transmitted to the data processing unit 700.

  In this case, when the material P is placed on the sensor built-in liquid crystal panel 301, the optical sensor circuit 32 detects a reflected image of the material P, and as a result, the data processing unit 700 receives from the sensor control unit 602. A change occurs in the image data. Therefore, as a response to the above-described command, the data processing unit 700 transmits partial image data (including coordinate data in the partial image) of the portion where the change has occurred (that is, the region of the target) to the target detection unit 131. .

  Therefore, when the object detection unit 131 receives partial image data indicating the region of the object from the data processing unit 700, the object detection unit 131 detects that the object is placed on the sensor built-in liquid crystal panel 301.

  Then, when detecting that the material P is placed in the placement area, the object detection unit 131 extracts, from the entire image data, an area in which the brightness of the pixel value changes by a predetermined threshold value or more. Recognize the placement area.

  Next, the data display / sensor device 100 displays the placement region of the material P and the vicinity thereof in a single color (FIG. 11C). For this purpose, the single color display control unit 133 transmits an instruction to display the above-described placement area and its vicinity in a single color based on the color information read from the display color information storage unit 21 to the display data generation unit 15 to display the display area. In response to the instruction, the data generation unit 15 generates display data for displaying the placement area and the vicinity thereof in the single color, and transmits the generated display data to the data processing unit 700. In addition, the display data generation unit 15 transmits a command designating “first display / light sensor unit (001)” as the value of the “display panel” field to the data processing unit 700.

  Thereafter, the data display / sensor device 100 performs a scan. Therefore, the object scan execution unit 134 designates “sense” in the “data acquisition timing” field, designates “whole image” in the “data type” field, and “first display / image” in the “scan panel” field. A command designating the optical sensor unit (001) "is transmitted to the data processing unit 700. It is desirable to specify “reflection” in the “scan method” field. In addition, it is desirable to designate “high” in the “scanned image gradation” field.

  As described above, the latest entire image data is obtained from the data processing unit 700 when the command is transmitted. Then, the object detection unit 131 reads the image data of the region of the material P based on the placement area information read from the placement region information holding unit 132 out of the obtained entire image data, and reads the read image data. The image data is held in the scan data holding unit 135 as scan data SCD.

  Next, the data display / sensor device 100 displays the scan data SCD superimposed on the video (FIG. 11D). For this purpose, the scan data display control unit 136 transmits an instruction to display the scan data SCD read from the scan data holding unit 135 to the display data generation unit 15, and the display data generation unit 15 performs the above scan according to the instruction. Display data in which the data SCD is superimposed on the video is generated, and the generated display data is transmitted to the data processing unit 700. In addition, the display data generation unit 15 transmits a command designating “first display / light sensor unit (001)” as the value of the “display panel” field to the data processing unit 700.

  Next, after the scanning of the material P is completed, when the user removes the material P from the sensor built-in liquid crystal panel 301 (FIG. 11 (e)), the object detection unit 131 displays the material P as the sensor built-in liquid crystal panel 301. Detects removal from above. Here, the object detection unit 131 obtains the entire image data from the data processing unit 700 in a predetermined cycle as a response to the command already transmitted to the data processing unit 700. Therefore, the object detection unit 131 compares two pieces of image data obtained continuously in time among the entire image data obtained in a predetermined cycle, and the brightness of the pixel value is compared with the previous image data in terms of time. When there is an area that has changed by a predetermined threshold or more and there is no area in which the brightness of a pixel value has changed by a predetermined threshold or more in image data that is later in time, the material P has been removed from the sensor built-in liquid crystal panel 301 Is detected.

  Thereafter, the data display / sensor device 100 stops displaying the single color in the area where the material P is placed and in the vicinity thereof. For this purpose, the single color display control unit 133 transmits an instruction to stop the display in the above-described placement area and a single color in the vicinity thereof to the display data generation unit 15, and the display data generation unit 15 responds to the instruction. Then, display data for displaying only the video data received from the video conference communication processing unit 19 on the sensor built-in liquid crystal panel 301 is generated, and the generated display data is transmitted to the data processing unit 700.

  Next, when the user performs a moving operation by dragging the scan data SCD with the hand F, the data display / sensor device 100 changes the display position of the scan data SCD (FIG. 11 (f)). For this purpose, the scan data display control unit 136 acquires coordinate data of the user's finger, pen, and the like from the data processing unit 700 while the user touches the scan data and traces on the sensor built-in liquid crystal panel 301. . Therefore, after displaying the scan data, the scan data display control unit 136 designates “event” in the “data acquisition timing” field, designates “coordinate” in the “data type” field, and “scan panel”. A command designating “first display / light sensor unit (001)” in the field is transmitted to the data processing unit 700. It is desirable to specify “reflection” in the “scan method” field. Further, the “scanned image gradation” field may be “high” or “low”.

  Then, the scan data display control unit 136 obtains the position of the movement destination of the scan data SCD based on the coordinate data acquired from the data processing unit 700 and then instructs to redisplay the scan data SCD at the position of the movement destination. Is transmitted to the display data generation unit 15.

  Then, the display data generation unit 15 generates display data for superimposing and displaying the scan data SCD at the movement destination position, and transmits the generated display data to the data processing unit 700. In addition, the display data generation unit 15 transmits a command designating “first display / light sensor unit (001)” as the value of the “display panel” field to the data processing unit 700.

(Processing flow 1 for scanning an object)
Next, an example of a flow of processing for scanning an object while displaying an image in the data display / sensor device 100 according to the present embodiment will be described with reference to FIG. FIG. 14 is a flowchart illustrating an example of a flow of processing for scanning an object while displaying an image in the data display / sensor device 100 according to the present embodiment.

  Here, it is assumed that the video conference communication processing unit 19 is in a state where a video conference is being performed between the data display / sensor device 100 and one or more video conference devices on the communication network. . The video conference communication processing unit 19 receives the audio and video signals of the other party of the video conference via the external communication unit 907.

  First, the video conference communication processing unit 19 transmits video data to the display data generation unit 15, and the display data generation unit 15 generates display data for displaying the video data on the sensor built-in liquid crystal panel 301. By transmitting the display data to the data processing unit 700, an image is displayed on the sensor built-in liquid crystal panel 301 (step S21). Step S21 corresponds to the state described with reference to FIG.

  Next, when the object detection unit 131 detects that the object is placed on the sensor built-in liquid crystal panel 301 (step S22), the object detection unit 131 has a predetermined time T1 (for example, several seconds) while the object is placed. When it is determined that the time has elapsed (YES in step S23), the placement area is recognized, and the placement area information is held in the placement area information holding unit 132 (step S24).

  Next, based on the placement area information held in the placement area information holding unit 132 and the color information preset in the display color information storage unit 21, the single color display control unit 133 sets the placement area and The display data generation unit 15 is notified of an instruction to display the vicinity in a single color, and the display data generation unit 15 generates display data for displaying the placement area and the vicinity in a single color, and the generated display data Is transmitted to the data processing unit 700, so that the placement area on the sensor built-in liquid crystal panel 301 and its vicinity are displayed in a single color (step S25). Step S25 corresponds to the state described with reference to FIG.

  Next, the object scan execution unit 134 causes the sensor built-in liquid crystal panel 301 to execute a scan (step S26), and extracts the area of the object as scan data from the entire image data obtained as a response (step S27). . Then, the object scan execution unit 134 holds the extracted scan data in the scan data holding unit 135 (step S28).

  Next, the scan data display control unit 136 reads the placement area information from the placement area information holding unit 132, and scans an arbitrary area that does not overlap the placement area based on the read placement area information. By notifying the display data generation unit 15 to display the data, the display data generation unit 15 generates display data for displaying the scan data in a superimposed manner, and transmits the generated display data to the data processing unit 700. Scan data is displayed on the sensor built-in liquid crystal panel 301 (step S29). Note that step S29 corresponds to the state described with reference to FIG.

  Next, when the object detection unit 131 detects that the object has been removed from the sensor built-in liquid crystal panel 301 (YES in step S30), the single-color display control unit 133 detects the placement area and its vicinity. An instruction to stop displaying in a single color is notified to the display data generation unit 15, and the scan data display control unit 136 notifies the display data generation unit 15 of an instruction to display the scan data in the placement area. Then, the display data generation unit 15 generates display data for displaying the scan data on the placement area based on the instruction, and transmits the generated display data to the data processing unit 700, thereby incorporating the sensor. The display of the placement area on the liquid crystal panel 301 and the vicinity thereof in a single color is stopped, and the display of the scan data displayed in step S29 is changed to the display that was displayed before the scan data was displayed. Return (step S31). Step S31 corresponds to the state described with reference to FIG. However, in step S31, the scan data SCD is displayed in the area SA in FIG. 11E, and the area in which the scan data SCD is displayed in FIG. 11E is displayed before the scan data SCD is displayed. Return to display.

  If there is a user operation on the scan data (YES in step S32), if the user operation is an operation for moving scan data (“movement operation” in step S33), the scan data display control unit 136 is operated. Notifies the display data generation unit 15 of an instruction to redisplay the scan data at the movement destination position, and the display data generation unit 15 generates display data for displaying the scan data superimposed on the movement destination position. By transmitting the displayed data to the data processing unit 700, the scan data is displayed on the sensor built-in liquid crystal panel 301 (step S34). Note that step S34 corresponds to the state described in FIG.

  If the user operation is a scan data transfer operation (“transfer operation” in step S33), the data transfer processing unit 17 transfers the data stored in the scan data holding unit 135 or the scan data storage unit 22. The target scan data is read, and the read scan data is transmitted to the external device via the external communication unit 907 (step S35).

  If the user operation is a scan data saving operation (“save operation” in step S33), the scan data storage unit 137 stores the scan data held in the scan data holding unit 135 in the scan data storage. The data is stored in the unit 22 (step S36).

(Processing flow 2 for scanning an object)
In the above description, when the object detection unit 131 detects that the object has been removed from the sensor built-in liquid crystal panel 301 (step S30), display of the placement area and its vicinity in a single color is stopped (step S31). ) Was taken as an example. However, since the monochrome display of the placement area and its vicinity is performed in order to increase the scanning accuracy of the object, the display of the placement area and its vicinity in a single color is performed when the scan of the object is completed. May be canceled. The processing flow in this case will be described below.

  With reference to FIG. 15, another example of the flow of processing for scanning an object while displaying an image in the data display / sensor device 100 will be described. FIG. 15 is a flowchart showing another example of the flow of processing for scanning an object while displaying an image in the data display / sensor device 100.

  The flow from step S21 to step S28 is the same as the flow of processing described with reference to FIG.

  In step S28, the single-color display control unit 133 uses the single-color display control unit 133 in a single color in the placement area and the vicinity thereof, triggered by the object scan execution unit 134 holding the scan data in the scan data holding unit 135. The display data generation unit 15 is notified of an instruction to stop the display, and the display data generation unit 15 generates display data that does not display in the single color of the placement area and its vicinity based on the instruction, By transmitting the generated display data to the data processing unit 700, display of the placement area on the sensor built-in liquid crystal panel 301 and the vicinity thereof in a single color is stopped (step S41).

  Next, the scan data display control unit 136 reads the placement area information from the placement area information holding unit 132, and scans an arbitrary area that does not overlap the placement area based on the read placement area information. By notifying the display data generation unit 15 to display the data, the display data generation unit 15 generates display data for displaying the scan data in a superimposed manner, and transmits the generated display data to the data processing unit 700. Scan data is displayed on the sensor built-in liquid crystal panel 301 (step S42).

  The processing flow after step S42 is the same as the flow from step S32 to step S36 shown in FIG.

[Embodiment 2]
In the above-described embodiment, when it is found that a predetermined time T1 (for example, several seconds) has elapsed with the object placed (YES in step S23), the object is scanned only once (step S24). ˜S28) has been described. However, the timing of scanning the object is not limited to the above. In the present embodiment, a mode in which scanning of an object is repeatedly performed until the object is removed from the sensor built-in liquid crystal panel 301 will be described. This embodiment will be described below with reference to FIG. 1 and FIG.

  For convenience of explanation, members having the same functions as those shown in the first embodiment are denoted by the same reference numerals, and description thereof is omitted. Further, the description of the same processing as that described in Embodiment 1 is omitted.

(Configuration of main control unit 800)
Of the configuration of the main control unit 800 of the data display / sensor device 100 according to the present embodiment, a configuration different from the configuration described in the first embodiment will be described below. The storage unit 901 of the data display / sensor device 100 according to the present embodiment has the same configuration as that described in the first embodiment.

  The object detection unit 131 immediately after detecting that the object is placed on the sensor built-in liquid crystal panel 301 and when it does not detect that the object is removed from the sensor built-in liquid crystal panel 301 The placement area is recognized, the placement area information of the recognized recognition area is acquired, and the obtained placement area information is held in the placement area information holding unit 132 so as to be readable.

  Next, the object scan execution unit 134 extracts scan data from the entire image data acquired from the data processing unit 700 as a response to causing the sensor built-in liquid crystal panel 301 to execute the scan, and holds the extracted scan data as scan data The unit 135 holds the data in a readable manner. At this time, the held scan data is associated with the time when it is held, and is held in the scan data holding unit 135.

  The scan data holding unit 135 holds a plurality of predetermined numbers of scan data by a first-in first-out method. Note that scan data that has been stored for a predetermined time may be deleted.

  Next, when the scan data display control unit 136 reads the scan data to be displayed on the sensor built-in liquid crystal panel 301 from the scan data holding unit 135, the object detection unit 131 removes the object from the sensor built-in liquid crystal panel 301. The latest scan data held in the scan data holding unit 135 is read from the scan data holding unit 135 at a time point a predetermined time (referred to as T2) from the time when this is detected. Then, the display data generation unit 15 is notified of an instruction to display the read scan data on the sensor built-in liquid crystal panel 301.

  As described above, the reason for reading the latest scan data held in the scan data holding unit 135 at a time point before the predetermined time T2 from the time when the object is removed is that the object is removed. The latest scan data held in the scan data holding unit 135 may be scan data obtained in the middle of the removal of the object by the user, and when the object is removed If the latest scan data held in the scan data holding unit 135 at a time point before the predetermined time T2 from the state before the user performs an operation of removing the target object (the target object was completely placed) This is because it may be considered that the scan data is obtained in the state).

(Process flow to scan the object)
With reference to FIG. 16, another example of the flow of processing for scanning an object while displaying an image in the data display / sensor device 100 according to the present embodiment will be described. FIG. 16 is a flowchart showing another example of the flow of processing for scanning an object while displaying an image in the data display / sensor device 100 according to the present embodiment.

  Since the flow up to step S21 and step S22 is the same as the flow of processing described with reference to FIG. 14 of the first embodiment, description thereof is omitted.

  In step S <b> 22, the object detection unit 131 recognizes the placement area using the detection that the object detection unit 131 detects that the object is placed on the sensor built-in liquid crystal panel 301. The area information is held in the placement area information holding unit 132 (step S51).

  Next, similarly to step S25, the single color display control unit 133 displays the placement area on the sensor built-in liquid crystal panel 301 and its vicinity in a single color (step S52). Then, the object scan execution unit 134 causes the sensor built-in liquid crystal panel 301 to execute the scan (step S53), and extracts the area of the object as scan data from the entire image data obtained as a response (step S54). Then, the object scan execution unit 134 holds the extracted scan data in the scan data holding unit 135 (step S55).

  Next, when the object detection unit 131 does not detect that the object has been removed from the sensor built-in liquid crystal panel 301 (NO in step S56), the processes from step S51 to step S55 are performed again.

  On the other hand, when it is detected that the object has been removed from liquid crystal panel 301 with a built-in sensor (YES in step S56), single color display control unit 133 has a single color in the placement area and its vicinity, as in step S31. The display data generation unit 15 is notified of an instruction to stop the display at the display data, and the display data generation unit 15 generates display data that does not display in the single color of the placement area and its vicinity based on the instruction. Then, by transmitting the generated display data to the data processing unit 700, display of the placement area on the sensor built-in liquid crystal panel 301 and its vicinity in a single color is stopped (step S57).

  Thereafter, the scan data display unit 136 scans the scan data holding unit 135 at a time before the predetermined time T2 from the time when the object detection unit 131 detects that the object has been removed from the sensor built-in liquid crystal panel 301. The latest scan data held in the sensor is read from the scan data holding unit 135, and an instruction to display the read scan data on the sensor built-in liquid crystal panel 301 is notified to the display data generating unit 15, thereby Scan data is displayed on the panel 301 (step S58).

  Note that the flow of processing following step S58 is the same as the flow from step S32 to step S36 shown in FIG.

[Additional notes]
In each of the above-described embodiments, the mode in which the object is scanned only once in a state where the placement area and the vicinity thereof are displayed in a single color has been described. Color image scan data may be obtained by performing each of the scans continuously once. The three types of scans are (1) a scan with the placement area and its vicinity displayed in red, (2) a scan with the placement area and its vicinity displayed in green, and (3) a placement. This is a scan with the area and its vicinity displayed in blue.

  Here, when the object is scanned in a state where the placement area and its vicinity are displayed in red, red characters and the like written on the object are not read, and characters that are far from red are more clearly displayed. The read image data can be obtained.

  Similarly, when the object is scanned with the placement area and the vicinity thereof displayed in green, the green characters and the like written on the object are not read, and the characters farther away from the green and the like are clearer. The read image data can be obtained.

  Similarly, when the object is scanned with the placement area and the vicinity thereof displayed in blue, the blue characters written on the object are not read, and the characters farther away from blue are clearly The read image data can be obtained.

  Accordingly, color image scan data can be obtained by performing general image processing based on the difference between the three image data obtained by executing the above-described three types of scans.

[Additional Notes]
In the above description, the configuration including the data processing unit 700 and the circuit control unit 600 has been described as the configuration of the data display / sensor device 100. However, the configuration is not limited to this configuration. For example, the main control unit 800 may directly control the circuit control unit 600, and the main control unit 800 may generate whole image data / partial image data / coordinate data. The main control unit 800 may directly control the display / light sensor unit 300.

  Finally, the main control unit 800, the data processing unit 700, and the circuit control unit 600 of the data display / sensor device 100 may be configured by hardware logic, or realized by software using a CPU as follows. Also good.

  That is, the data display / sensor device 100 includes a CPU (central processing unit) that executes instructions of a control program for realizing each function, a ROM (read only memory) that stores the program, and a RAM (random access) that expands the program. memory), a storage device (recording medium) such as a memory for storing the program and various data. An object of the present invention is a record in which a program code (execution format program, intermediate code program, source program) of a control program for the data display / sensor device 100, which is software for realizing the functions described above, is recorded so as to be readable by a computer. This can also be achieved by supplying a medium to the data display / sensor device 100 and reading and executing the program code recorded on the recording medium by the computer (or CPU or MPU).

  Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy (registered trademark) disks / hard disks, and disks including optical disks such as CD-ROM / MO / MD / DVD / CD-R. Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.

  Further, the data display / sensor device 100 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Also, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), IEEE802.11 radio, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

  The present invention can be applied to an apparatus that performs display processing of an image or the like and scanning processing of an object. In particular, the present invention can be suitably applied to a video conference apparatus that can be assumed to perform scan processing of an object while displaying an image.

It is a block diagram which shows the detailed structure of the data display / sensor apparatus which concerns on one Embodiment of this invention. It is a schematic diagram which shows the cross section of the liquid crystal panel with a built-in sensor with which the data display / sensor apparatus which concerns on one Embodiment of this invention is provided. FIG. 3A is a schematic diagram showing a state in which a position touched by a user is detected by detecting a reflected image on a sensor built-in liquid crystal panel included in the data display / sensor device according to the embodiment of the present invention. is there. FIG. 3B is a schematic diagram illustrating a state in which a position touched by the user is detected by detecting a shadow image on the sensor built-in liquid crystal panel included in the data display / sensor device according to the embodiment of the present invention. . It is a block diagram which shows the principal part structure of the data display / sensor apparatus which concerns on one Embodiment of this invention. It is a schematic diagram which shows an example of the frame structure of the command used with the data display / sensor apparatus which concerns on one Embodiment of this invention. It is explanatory drawing which shows an example of the value which can be designated to each field contained in the command shown in FIG. 5, and its outline. FIG. 7A is a result of scanning the entire sensor-equipped liquid crystal panel when the object is not placed on the sensor-equipped liquid crystal panel in the data display / sensor device according to the embodiment of the present invention. Image data. FIG. 7B shows image data obtained as a result of scanning when the user is touching the sensor-equipped liquid crystal panel with a finger in the data display / sensor device according to the embodiment of the present invention. It is a block diagram which shows the structure of the liquid crystal panel with a sensor with which the data display / sensor apparatus which concerns on one Embodiment of this invention is provided, and the structure of its peripheral circuit. It is a schematic diagram which shows schematic structure of the video conference system containing the data display / sensor apparatus which concerns on one Embodiment of this invention. FIG. 10A is an explanatory diagram showing the surface of a material that is an object. FIG. 10B is an explanatory diagram showing the back side of the material that is the object. FIG. 11A is a diagram illustrating a state in which the data display / sensor device according to the embodiment of the present invention displays an image of the other party of the video conference. FIG. 11B is a diagram showing a state in which a material as an object is placed on the data display / sensor device according to the embodiment of the present invention. FIG. 11C is a diagram illustrating a state in which the placement area and the vicinity thereof are displayed in a single color. FIG. 11D is a diagram illustrating a state in which scan data, which is image data obtained as a result of scanning a material that is an object, is superimposed and displayed on the video of the other party of the video conference. FIG. 11E is a diagram illustrating a state where the user has removed the material that is the object. FIG. 11F is a diagram illustrating a state in which the user moves the scan data and changes the display position of the scan data. It is a figure which shows a mode that the scan data was transferred to the other party apparatus of a video conference from a data display / sensor apparatus in the video conference system containing the data display / sensor apparatus which concerns on one Embodiment of this invention. It is a schematic diagram which shows an example of a mounting area | region and its vicinity. 6 is a flowchart illustrating an example of a flow of processing for scanning an object while displaying an image in the data display / sensor device according to the embodiment of the present invention. 7 is a flowchart showing another example of a process flow for scanning an object while displaying an image in the data display / sensor device according to the embodiment of the present invention. 7 is a flowchart showing another example of a process flow for scanning an object while displaying an image in the data display / sensor device according to the embodiment of the present invention.

Explanation of symbols

22 Scan data storage (storage)
13 Object Scan Processing Unit 15 Display Data Generation Unit (Image Display Control Unit)
100 Data display / sensor device (image display / image detection device)
131 Object detection unit (object detection means)
132 Placement area information holding unit 133 Single color display control unit (display color control means)
134 Object scan execution unit 135 Scan data holding unit 136 Scan data display control unit (image display control means)
137 Scan data storage (detected image storage)
300 Display / light sensor unit 301 Liquid crystal panel with built-in sensor (planar member)
700 Data processing unit 800 Main control unit RA Placement area P Material (object)
SCD scan data (image of object image)

Claims (7)

An image display / image detection apparatus including a planar member that displays an image and reads a nearby image,
Image display control means for displaying an image on the planar member;
An object detection means for detecting a placement area where the object is placed on the planar member;
When the image display control means is displaying the image on the planar member, when the reading of the object is instructed, the placement area on which the object is placed is displayed in a predetermined single color. Display color control means for instructing the image display control means,
An image display / image detection apparatus, wherein the planar member reads an image of the object in the placement area in a state where the placement area is displayed in the single color.   2. The image display / image detection according to claim 1, wherein the display color control means further instructs the image display control means to display the vicinity of the outer edge of the placement area in the single color. apparatus. The object detection means further detects that the object is not placed on the planar member,
When the object detection means detects that the object is not placed on the planar member, the display color control means stops the display in the single color and displays the image. 3. The image display / image detection apparatus according to claim 1, wherein the image display control means is instructed.   When reading of the image of the object is completed on the planar member, the display color control unit instructs the image display control unit to stop displaying the single color and display the image. The image display / image detection apparatus according to claim 1, wherein:   5. The detection image storage unit according to claim 1, further comprising a detection image storage unit that stores an image of the object image read by the planar member in a storage unit in a mirror image inverted state. The image display / image detection apparatus described.   When the image display control means is displaying an image on the planar member, the image of the object read by the planar member is displayed on the planar member in a mirror image inverted state. 6. The image display / image detection apparatus according to claim 1, further comprising image display control means for instructing the image display control means.   In the planar member, when detecting a user operation to move the display position of the image of the target object displayed on the planar member, the image display control unit is configured to display the target displayed 7. The image display / image detection apparatus according to claim 6, wherein the image display control means is instructed to move and display an image of an object to a position corresponding to the user operation.

Images