JP2010092340A - Image display/image detection apparatus, image display method, image display program, and recording medium with program recorded thereon - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image display/image detection apparatus which activates an application in an image display area desired by a user without urging complicated operation to the user. <P>SOLUTION: The image display/image detection apparatus (data display/sensor apparatus) 100 includes: an indicating direction detection unit 12 for detecting an indicating direction based on a positional coordinate of a point indicating an icon displayed on any display/optical sensor unit 300 and a positional coordinate of a point indicated next to the point; a display position decision unit 14 for setting, as the display/optical sensor unit 300 for displaying a window image, any of the display/optical sensor units 300 disposed in a direction indicated by the indicating direction; and a display processing control unit 15 for displaying the window image in the decided display/optical sensor unit 300. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image display / image detection apparatus that displays an image and detects an image of an object located nearby. More specifically, the present invention relates to application processing in the image display / image detection apparatus.

  Conventionally, when an application is activated on a computer having a plurality of image display devices, the application is activated on a predetermined image display device. For this reason, when an application window is to be displayed on another image display device, it is necessary to move the application window onto the desired image display device by user operation such as dragging using a mouse after the application is started. There is.

  Specific examples of conventional application activation and application window movement corresponding to the activated application will be described with reference to FIGS. First, as shown in FIG. 18A, the user touches (or selects with an pointer) an icon for starting an application. Then, as shown in FIG. 18B, the application is activated, and an application window is displayed on a predetermined screen (here, the display unit 310B in the lower housing 41).

  At this time, even if the user wants to display the application window on the upper screen (display unit 310A in the upper housing 40), as shown in FIG. After the application window is displayed on the display unit 310A, the application window must be dragged to the display unit 310B.

  As a technique for automatically changing the display position of the application window, there is a technique described in Patent Document 1, for example. However, the technique described in Patent Document 1 only adjusts the display position and the display area so that the displayed application windows do not overlap each other, and determining the display position of the application window at the time of starting the application is not possible. Can not.

As a technique for displaying an application window at a position desired by the user when the application is activated, for example, a technique described in Patent Document 2 can be cited. Patent Document 2 discloses a technique for acquiring the display position and display size of an application window from the drag start position and drag end position of the mouse, and starting the application accordingly.
JP 11-194867 A (publication date: July 21, 1999) JP 2001-296945 A (publication date: October 26, 2001)

  However, in the technique described in Patent Document 2, the mouse must be dragged as the key on the keyboard is pressed, and the number of user operations required to start the application increases. For this reason, there is still a problem that the user is forced to perform complicated operations when determining the display position of the application window.

  In addition, when the image display device has a large screen, the amount of movement for moving the mouse for dragging increases, which makes it very difficult to start the application at the position and size desired by the user. Also have.

  The present invention has been made in view of the above-described problems, and a main object of the present invention is an image display device that can display an application window at a position desired by the user without forcing the user to perform complicated operations. An image display / image detection apparatus) is provided.

In the image display / image detection apparatus according to the present invention, in order to solve the above problems,
An image display / image detection device including one or more planar members for displaying an image and detecting an image of an object located in the vicinity, provided on the one or more planar members When a point on an icon displayed in any of a plurality of image display areas is designated by an object located in the vicinity of the planar member, the position coordinates of this point and the point of the planar member next to this point Based on the position coordinates of a point indicated by an object located in the vicinity, an indication direction detection means for detecting an indication direction, and another image in the indication direction with respect to the image display area where the icon is displayed Display area setting means for setting any one of the display areas as an image display area for displaying a window image displayed by an application associated with the icon, and an image table set in the display area setting means In the region, it is characterized by comprising a display processing control means for displaying a window displayed by the application associated with the icon.

  In the image display / image detection apparatus according to the present invention, an indication direction detection means for detecting an indication direction designated by a user and a window displayed by an application from an image display area arranged in the direction indicated by the indication direction are displayed. Display area setting means for setting an image display area to be displayed, and display processing control means for displaying a window in the set image display area.

  Thus, in the image display / image detection apparatus according to the present invention, the user can display a window in a desired image display area only by indicating the direction of the image display area in which the window displayed by the application is to be displayed. Can do. Therefore, the image display / image detection apparatus according to the present invention can display the window image displayed by the activated application in the image display area desired by the user without forcing the user to perform complicated operations. There is an effect.

In the image display method according to the present invention, in order to solve the above problems,
An image display method in an image display / image detection apparatus including one or more planar members for displaying an image and detecting an image of an object located in the vicinity, wherein the image display method includes the above-described one or more planar members. When the point on the icon displayed in any of the plurality of image display areas provided in is pointed by an object located in the vicinity of the planar member, the position coordinates of this point, and the point Based on the position coordinates of a point designated by an object located in the vicinity of the planar member, an indication direction detection step for detecting an indication direction, and an indication direction relative to the image display area where the icon is displayed A display area setting step for setting any one of the other image display areas as an image display area for displaying a window displayed by the application associated with the icon; and the display area setting step. The image display area set in flop, is characterized by comprising a display processing control step of displaying a window that is displayed by the application associated with the icon, the.

  According to said structure, there exists an effect similar to the image display / image detection apparatus mentioned above.

In the image display / image detection apparatus according to the present invention, in order to solve the above problems,
An image display / image detection device including one or more planar members for displaying an image and detecting an image of an object located in the vicinity, provided on the one or more planar members When a point on the first image displayed in any of the plurality of image display areas is designated by an object located in the vicinity of the planar member, the point is located in the vicinity of the planar member indicating this point A calculation unit that calculates the number of images of the object, and a process determination that determines a process to be executed in the image display / image detection device according to the type of the first image and the number of images calculated by the calculation unit. Based on the means, the position coordinates of the point on the first image indicated by the object, and the position coordinates of the point indicated by the object located in the vicinity of the planar member next to this point, Pointing direction detecting means for detecting the pointing direction; A second image obtained by executing the process determined by the process determining means in any one of the other image display areas in the indicated direction with respect to the image display area in which the image is displayed is displayed. It is characterized by comprising display area setting means for setting as an image display area and display processing control means for displaying the second image in the image display area set by the display area setting means.

  According to the above configuration, the process executed in the image display / image detection device according to the number of images of the object covering at least a part of the first image displayed in any of the image display areas. decide.

  As a result, the image display / image detection apparatus can execute a desired process only by changing the number of images of the object, for example, by changing the number of fingers of the user or the number of touch pens.

  Therefore, the image display / image detection apparatus according to the present invention can display the second image in the image display area desired by the user without forcing the user to perform complicated operations, and can also display the image display / image. There is an effect that the process executed in the detection device can be set by an easy operation.

  In the image display / image detection apparatus according to the present invention, the processing determination unit may further include the case where the first image is an icon and the number of images calculated by the calculation unit is a first value. It is preferable that the process to be executed in the image display / image detection apparatus is determined to execute the application associated with the icon.

  In the image display / image detection apparatus according to the present invention, the processing determination means is a window in which the first image is displayed by the application, and the number of images calculated by the calculation means is the first. When the value is 3, it is preferable that the processing to be executed in the image display / image detection apparatus is determined to end the display of the window.

  In the image display / image detection apparatus according to the present invention, the process determination unit further executes a process to be executed in the image display / image detection apparatus when the number of images calculated by the calculation unit is the second value. Is preferably determined to change the image display area for displaying the first image.

  According to said structure, there exists an effect which can perform each process with the number of the image of the object which a user can understand intuitively with one hand.

  In the image display / image detection apparatus according to the present invention, the pointing direction detection means detects the pointing direction with reference to a locus of position coordinates of an image obtained by sequentially detecting an object located in the vicinity of the planar member. It is preferable.

  According to the above configuration, the pointing direction detection means detects the pointing direction according to the locus of the position coordinates of the temporally continuous image by sequentially detecting the image of the object located in the vicinity of the planar member. To do. That is, the indication direction is detected according to the scroll direction of the object.

  Thereby, the user can set the image display area where the image is displayed more intuitively. Therefore, the image display / image detection apparatus according to the present invention provides instructions by an operation that can be easily understood and practiced even by a user who is not familiar with the operation of the image display / image detection apparatus (for example, a young person or an elderly person) There is an effect that the direction can be set.

  The image display / image detection apparatus according to the present invention preferably further includes an image display apparatus corresponding to each of the image display areas.

  Also, a program for operating the image display / image detection apparatus according to the present invention, which is characterized in that the computer is driven as each of the above-mentioned means and a computer-readable recording medium on which the program is recorded. It is included in the category of the present invention.

  As described above, the image display / image detection apparatus according to the present invention is displayed by the application from the indication direction detection means for detecting the indication direction indicated by the user and the image display area displayed in the direction indicated by the indication direction. Display area setting means for setting an image display area for displaying the window to be displayed, and display processing control means for displaying the window in the determined image display area.

  Thus, the image display / image detection apparatus according to the present invention can display the window image of the activated application in the image display area desired by the user without forcing the user to perform complicated operations. Play.

  An embodiment of a data display / sensor device (image display / image detection device) according to the present invention will be described below with reference to FIGS.

(Data display / appearance of sensor device)
First, the external appearance of the data display / sensor device according to the present embodiment will be described below with reference to FIG. The data display / sensor device 100 includes two housings, an upper housing 40 and a lower housing 41, as shown in FIG. The upper housing 40 includes a first display / light sensor unit 300A, and the lower housing 41 includes a second display / light sensor unit 300B.

  In the present embodiment, the data display / sensor device 100 is mainly described as an example of a data display / sensor device including the upper housing 40 and the lower housing 41. The configuration of the sensor device 100 is not limited to this. For example, as shown in FIG. 9B, the configuration may include two housings, a right housing 42 and a left housing 43, or the first housing as shown in FIG. 9C. 44, a second housing 45, and a third housing 46 may be used. Moreover, although not shown here, the structure which consists of four or more housing | casing may be sufficient. In addition, the arrow in Fig.9 (a)-(c) has shown the movable direction of each housing | casing.

(Outline of LCD panel with built-in sensor)
Here, prior to detailed description of the configuration and processing operation of the data display / sensor device 100, an outline of the sensor built-in liquid crystal panel 301 provided in the data display / sensor device 100 according to the present invention will be described.

  The sensor built-in liquid crystal panel 301 provided in the data display / sensor device 100 is a liquid crystal panel capable of detecting an image of an object in addition to displaying data. Here, the image detection of the object is, for example, detection of a position pointed (touched) by the user with a finger or a pen, or reading (scanning) of an image of a printed material or the like. The device used for display is not limited to a liquid crystal panel, and may be an organic EL (Electro Luminescence) panel or the like.

  The structure of the sensor built-in liquid crystal panel 301 will be described with reference to FIG. FIG. 2 is a diagram schematically showing a cross section of the sensor built-in liquid crystal panel 301. The sensor built-in liquid crystal panel 301 described here is an example, and any structure can be used as long as the display surface and the reading surface are shared.

  As shown in the figure, the sensor built-in liquid crystal panel 301 includes an active matrix substrate 51A disposed on the back surface side and a counter substrate 51B disposed on the front surface side, and has a structure in which a liquid crystal layer 52 is sandwiched between these substrates. is doing. The active matrix substrate 51A is provided with a pixel electrode 56, a data signal line 57, an optical sensor circuit 32 (not shown), an alignment film 58, a polarizing plate 59, and the like. The counter substrate 51B is provided with color filters 53r (red), 53g (green), 53b (blue), a light shielding film 54, a counter electrode 55, an alignment film 58, a polarizing plate 59, and the like. In addition, a backlight 307 is provided on the back surface of the sensor built-in liquid crystal panel 301.

  The photodiode 6 included in the photosensor circuit 32 is provided in the vicinity of the pixel electrode 56 provided with the blue color filter 53b, but is not limited to this configuration. It may be provided in the vicinity of the pixel electrode 56 provided with the red color filter 53r, or may be provided in the vicinity of the pixel electrode 56 provided with the green color filter 53g.

  Next, with reference to FIGS. 3A and 3B, two types of methods for detecting the position where the user touches the sensor built-in liquid crystal panel 301 with a finger or a pen will be described.

  FIG. 3A is a schematic diagram illustrating a state in which a position touched by the user is detected by detecting a reflected image. When the light 63 is emitted from the backlight 307, the optical sensor circuit 32 including the photodiode 6 detects the light 63 reflected by the object 64 such as a finger. Thereby, the reflected image of the target object 64 can be detected. Thus, the sensor built-in liquid crystal panel 301 can detect the touched position by detecting the reflected image.

  FIG. 3B is a schematic diagram illustrating a state in which a position touched by the user is detected by detecting a shadow image. As shown in FIG. 3B, the optical sensor circuit 32 including the photodiode 6 detects external light 61 transmitted through the counter substrate 51B and the like. However, when there is an object 62 such as a pen, the incident of the external light 61 is hindered, so that the amount of light detected by the optical sensor circuit 32 is reduced. Thereby, a shadow image of the object 62 can be detected. Thus, the sensor built-in liquid crystal panel 301 can also detect a touched position by detecting a shadow image.

  As described above, the photodiode 6 may detect reflected light (shadow image) of the light emitted from the backlight 307 or may detect a shadow image caused by external light. Further, the two types of detection methods may be used in combination to detect both a shadow image and a reflected image at the same time.

(Data display / sensor configuration)
Next, with reference to FIG. 4, the configuration of the main part of the data display / sensor device 100 will be described. FIG. 4 is a block diagram showing a main configuration of the data display / sensor device 100. As shown in FIG. As illustrated, the data display / sensor device 100 includes one or more display / light sensor units 300, a circuit control unit 600, a data processing unit 700, a main control unit 800, a storage unit 901, a primary storage unit 902, and an operation unit. 903, an external communication unit 907, an audio output unit 908, and an audio input unit 909. Here, the data display / sensor device 100 will be described as including two display / light sensor units 300 (first display / light sensor unit 300A and second display / light sensor unit 300B). When the first display / light sensor unit 300A and the second display / light sensor unit 300B are not distinguished, they are referred to as the display / light sensor unit 300.

  The display / light sensor unit 300 is a so-called liquid crystal display device with a built-in light sensor. The display / light sensor unit 300 includes a sensor built-in liquid crystal panel 301, a backlight 307, and a peripheral circuit 309 for driving them.

  The sensor built-in liquid crystal panel 301 includes a plurality of pixel circuits 31 and photosensor circuits 32 arranged in a matrix. The detailed configuration of the sensor built-in liquid crystal panel 301 will be described later.

  The peripheral circuit 309 includes a liquid crystal panel drive circuit 304, an optical sensor drive circuit 305, a signal conversion circuit 306, and a backlight drive circuit 308.

  The liquid crystal panel driving circuit 304 outputs a control signal (G) and a data signal (S) in accordance with the timing control signal (TC1) and the data signal (D) from the display control unit 601 of the circuit control unit 600, and the pixel circuit 31. It is a circuit which drives. Details of the driving method of the pixel circuit 31 will be described later.

  The optical sensor driving circuit 305 is a circuit that drives the optical sensor circuit 32 by applying a voltage to the signal line (R) in accordance with a timing control signal (TC2) from the sensor control unit 602 of the circuit control unit 600. Details of the driving method of the optical sensor circuit 32 will be described later.

  The signal conversion circuit 306 is a circuit that converts the sensor output signal (SS) output from the optical sensor circuit 32 into a digital signal (DS) and transmits the converted signal to the sensor control unit 602.

  The backlight 307 includes a plurality of white LEDs (Light Emitting Diodes) and is disposed on the back surface of the sensor built-in liquid crystal panel 301. When a power supply voltage is applied from the backlight drive circuit 308, the backlight 307 is turned on and irradiates the sensor built-in liquid crystal panel 301 with light. Note that the backlight 307 is not limited to white LEDs, and may include LEDs of other colors. The backlight 307 may include, for example, a cold cathode fluorescent lamp (CCFL) instead of the LED.

  The backlight driving circuit 308 applies a power supply voltage to the backlight 307 when the control signal (BK) from the backlight control unit 603 of the circuit control unit 600 is at a high level, and conversely, the backlight control unit 603. When the control signal from is at a low level, no power supply voltage is applied to the backlight 307.

  Next, the circuit control unit 600 will be described. The circuit control unit 600 has a function as a device driver that controls the peripheral circuit 309 of the display / light sensor unit 300. The circuit control unit 600 includes a display control unit 601, a sensor control unit 602, a backlight control unit 603, and a display data storage unit 604.

  The display control unit 601 receives display data from the display data processing unit 701 of the data processing unit 700, and performs timing control on the liquid crystal panel driving circuit 304 of the display / light sensor unit 300 in accordance with an instruction from the display data processing unit 701. A signal (TC1) and a data signal (D) are transmitted, and the received display data is displayed on the sensor built-in liquid crystal panel 301.

  The display control unit 601 temporarily stores the display data received from the display data processing unit 701 in the display data storage unit 604. Then, a data signal (D) is generated based on the primary stored display data. The display data storage unit 604 is, for example, a video random access memory (VRAM).

  The sensor control unit 602 transmits a timing control signal (TC2) to the optical sensor driving circuit 305 of the display / optical sensor unit 300 in accordance with an instruction from the sensor data processing unit 703 of the data processing unit 700, and the sensor built-in liquid crystal panel 301. Run the scan with.

  In addition, the sensor control unit 602 receives a digital signal (DS) from the signal conversion circuit 306. Then, image data is generated based on the digital signal (DS) corresponding to the sensor output signal (SS) output from all the optical sensor circuits 32 included in the sensor built-in liquid crystal panel 301. That is, the image data read in the entire reading area of the sensor built-in liquid crystal panel 301 is generated. Then, the generated image data is transmitted to the sensor data processing unit 703.

  The backlight control unit 603 transmits a control signal (BK) to the backlight drive circuit 308 of the display / light sensor unit 300 in accordance with instructions from the display data processing unit 701 and the sensor data processing unit 703 to drive the backlight 307. Let

  When the data display / sensor device 100 includes a plurality of display / light sensor units 300, the display control unit 601 determines which display / light sensor unit 300 displays the display data from the data processing unit 700. When an instruction is received, the liquid crystal panel drive circuit 304 of the display / light sensor unit 300 is controlled according to the instruction. When the sensor control unit 602 receives an instruction from the data processing unit 700 as to which display / light sensor unit 300 is to scan the object, The sensor drive circuit 305 is controlled and a digital signal (DS) is received from the signal conversion circuit 306 of the display / light sensor unit 300 according to the instruction.

  Next, the data processing unit 700 will be described. The data processing unit 700 has a function as middleware that gives an instruction to the circuit control unit 600 based on a “command” received from the main control unit 800. Details of the command will be described later.

  The data processing unit 700 includes a display data processing unit 701 and a sensor data processing unit 703. When the data processing unit 700 receives a command from the main control unit 800, at least one of the display data processing unit 701 and the sensor data processing unit 703 depends on the value of each field (described later) included in the received command. One works.

  The display data processing unit 701 receives display data from the main control unit 800, and gives instructions to the display control unit 601 and the backlight control unit 603 according to the command received by the data processing unit 700, and displays the received display data. The image is displayed on the sensor built-in liquid crystal panel 301. The operation of the display data processing unit 701 according to the command will be described later.

  The sensor data processing unit 703 gives an instruction to the sensor control unit 602 and the backlight control unit 603 according to the command received by the data processing unit 700.

  The sensor data processing unit 703 receives image data from the sensor control unit 602 and stores the image data in the image data buffer 704 as it is. Then, in accordance with the command received by the data processing unit 700, the sensor data processing unit 703 performs “whole image data”, “partial image data (partial image coordinate data) based on the image data stored in the image data buffer 704. At least one of “including coordinate data” and “coordinate data” is transmitted to the main control unit 800. The whole image data, partial image data, and coordinate data will be described later. The operation of the sensor data processing unit 703 according to the command will be described later.

  Next, the main control unit 800 executes an application program. The main control unit 800 reads the program stored in the storage unit 901 into a primary storage unit 902 configured by, for example, a RAM (Random Access Memory) and executes the program.

  An application program executed by the main control unit 800 causes the data processing unit 700 to display display data on the sensor built-in liquid crystal panel 301 or to scan an object on the sensor built-in liquid crystal panel 301. Send commands and display data. When “data type” is designated as a command, at least one of whole image data, partial image data, and coordinate data is received from the data processing unit 700 as a response to the command.

  The circuit control unit 600, the data processing unit 700, and the main control unit 800 can be configured by a CPU (Central Processing Unit), a memory, and the like, respectively. The data processing unit 700 may be configured by a circuit such as an ASIC (application specific integrate circuit).

  Next, the storage unit 901 stores programs and data executed by the main control unit 800 as shown in the figure. The program executed by the main control unit 800 may be separated into an application-specific program and a general-purpose program that can be shared by each application.

  Next, the operation unit 903 receives an input operation of the user of the data display / sensor device 100. The operation unit 903 includes input devices such as a switch, a remote controller, a mouse, and a keyboard, for example. Then, the operation unit 903 generates a control signal corresponding to the user's input operation of the data display / sensor device 100, and transmits the generated control signal to the main control unit 800.

  As an example of the switch, a hinge switch 904 that is provided at the hinge portion of the housing and detects the open / closed state of the housing, a power switch 905 that switches power on and off, and a predetermined function are assigned in advance. A hardware switch such as a user switch 906 is assumed.

  In addition, the data display / sensor device 100 includes an external communication unit 907 for communicating with an external device by wireless / wired communication, an audio output unit 908 such as a speaker for outputting audio, and a microphone for inputting an audio signal. A voice input unit 909 such as the above may be provided as appropriate.

(Command details)
Next, details of commands transmitted from the main control unit 800 to the data processing unit 700 will be described with reference to FIGS. 5 and 6. FIG. 5 is a diagram schematically illustrating an example of a command frame structure. FIG. 6 is a diagram for explaining an example of values that can be specified for each field included in the command and an outline thereof.

  As shown in FIG. 5, the commands are “header”, “data acquisition timing”, “data type”, “scan method”, “scan image gradation”, “scan resolution”, “scan panel”, “display panel”. "And" Reserve "fields. In each field, for example, values shown in FIG. 5 can be designated.

  The “header” field is a field indicating the start of a frame. As long as it is possible to identify the “header” field, the value of the “header” field may be any value.

  Next, the “data acquisition timing” field is a field for designating a timing at which data should be transmitted to the main control unit 800. In the “data acquisition timing” field, for example, values “00” (sense), “01” (event), and “10” (all) can be specified.

  Here, “sense” designates that the latest data is transmitted immediately. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “sense”, the latest data specified in the “data type” field is immediately updated to the main control unit 800. Send to.

  The “event” designates transmission at a timing when a change occurs in the image data received from the sensor control unit 602. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “event”, the image that receives the data specified in the “data type” field from the sensor control unit 602. The data is transmitted to the main control unit 800 at a timing when a change larger than a predetermined threshold occurs.

  “All” designates data transmission at a predetermined cycle. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “all”, the data designated in the “data type” field is transferred to the main control unit 800 at a predetermined cycle. Send to. The predetermined period coincides with the period in which the optical sensor circuit 32 performs scanning.

  Next, the “data type” field is a field for designating the type of data acquired from the sensor data processing unit 703. In the “data type” field, for example, values of “001” (coordinates), “010” (partial image), and “100” (entire image) can be specified. Furthermore, by adding these values, “coordinates” and “partial image” / “whole image” can be specified simultaneously. For example, when “coordinate” and “partial image” are specified at the same time, “011” can be specified.

  When the sensor data processing unit 703 receives a command whose value of the “data type” field is “whole image”, the sensor data processing unit 703 transmits the image data itself stored in the image data buffer 704 to the main control unit 800. The image data itself stored in the image data buffer 704 is referred to as “whole image data”.

  In addition, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 selects a portion where a change larger than a predetermined threshold has occurred from the image data received from the sensor control unit 602. A region to be included is extracted, and image data of the extracted region is transmitted to the main control unit 800. Here, the image data is referred to as “partial image data”. When a plurality of partial image data are extracted, the sensor data processing unit 703 transmits each extracted partial image data to the main control unit 800.

  Further, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 detects representative coordinates in the partial image data and indicates the position of the representative coordinates in the partial image data. The coordinate data is transmitted to the main control unit 800. The representative coordinates include, for example, the coordinates of the center of the partial image data, the coordinates of the center of gravity of the partial image data, and the like.

  Next, when receiving a command whose value of the “data type” field is “coordinate”, the sensor data processing unit 703 transmits coordinate data indicating the position of the representative coordinate in the entire image data to the main control unit 800. When a plurality of partial image data are extracted, the sensor data processing unit 703 detects representative coordinates in the entire image data of the extracted partial image data, and the coordinate data indicating the representative coordinates is detected. Each is transmitted to the main control unit 800 (multi-point detection).

  Specific examples of the whole image data, the partial image data, and the coordinate data will be described later with reference to schematic diagrams.

  Next, the “scan method” field is a field for designating whether or not the backlight 307 is turned on at the time of executing the scan. In the “scan method” field, for example, values of “00” (reflection), “01” (transmission), and “10” (reflection / transmission) can be designated.

  “Reflection” designates that scanning is performed with the backlight 307 turned on. Therefore, when the sensor data processing unit 703 receives a command whose “scan method” field value is “reflection”, the sensor data processing unit 703 performs sensor control so that the optical sensor driving circuit 305 and the backlight driving circuit 308 operate in synchronization. An instruction is given to the unit 602 and the backlight control unit 603.

  “Transmission” specifies that scanning is performed with the backlight 307 turned off. Therefore, when the sensor data processing unit 703 receives a command whose “scan method” field value is “transparent”, the sensor control unit 602 operates the optical sensor driving circuit 305 and does not operate the backlight driving circuit 308. Instructions to the backlight control unit 603. Note that “reflection / transmission” specifies that scanning is performed using both “reflection” and “transmission”.

  Next, the “scanned image gradation” field is a field for designating gradations of the partial image data and the entire image data. In the “scanned image gradation” field, for example, values of “00” (binary) and “01” (multivalue) can be designated.

  When the sensor data processing unit 703 receives a command whose “scan image gradation” field value is “binary”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 as monochrome data. .

  When the sensor data processing unit 703 receives a command whose “scanned image gradation” field value is “multivalued”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 as multitone data. To do.

  Next, the “scan resolution” field is a field for designating the resolution of the partial image data and the entire image data. In the “scan resolution” field, for example, values of “0” (high) and “1” (low) can be specified.

  Here, “high” designates a high resolution. Therefore, when the sensor data processing unit 703 receives a command whose “scan resolution” field value is “high”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 with high resolution. For example, it is desirable to designate “high” for image data (image data such as a fingerprint) to be subjected to image processing such as image recognition.

  “Low” designates a low resolution. Therefore, when the sensor data processing unit 703 receives a command whose “scan resolution” field value is “low”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 at a low resolution. For example, it is desirable to designate “low” for image data (such as touched finger or hand image data) that only needs to be recognized.

  Next, the “scan panel” field is a field for designating which display / light sensor unit 300 is to scan the object. In the “scan panel” field, for example, values “001” (first display / light sensor unit 300A) and “010” (second display / light sensor unit 300B) can be designated. By adding these values, a plurality of display / light sensor units 300 can be specified at the same time. For example, when “first display / light sensor unit 300A” and “second display / light sensor unit 300B” are specified at the same time, “011” can be specified.

  When the sensor data processing unit 703 receives a command whose “scan panel” field value is “first display / photosensor unit 300A”, the sensor data processing unit 703 and the photosensor drive circuit 305 of the first display / photosensor unit 300A and An instruction is given to the sensor control unit 602 and the backlight control unit 603 so as to control the backlight drive circuit 308.

  Next, the “display panel” field is a field for designating which display / light sensor unit 300 displays the display data. In the “display panel” field, for example, values of “001” (first display / light sensor unit 300A) and “010” (second display / light sensor unit 300B) can be designated. By adding these values, a plurality of display / light sensor units 300 can be specified at the same time. For example, when “first display / light sensor unit 300A” and “second display / light sensor unit 300B” are specified at the same time, “011” can be specified.

  Here, for example, when the display data processing unit 701 receives a command whose value of the “display panel” field is “first display / light sensor unit 300A”, the display data processing unit 701 displays the display data on the first display / light sensor unit 300A. Therefore, an instruction is given to the display control unit 601 and the backlight control unit 603 to control the liquid crystal panel driving circuit 304 and the backlight driving circuit 308 of the first display / light sensor unit 300A.

  Next, the “reserved” field is a field that is appropriately specified when it is necessary to further specify information other than information that can be specified in the above-described fields.

  Note that an application executed by the main control unit 800 does not need to use all the above-described fields when transmitting a command, and an invalid value (such as a NULL value) may be set for a field that is not used. .

  When the user wants to acquire coordinate data of a position touched with a finger or pen, a command specifying “coordinate” in the “data type” field is transmitted to the data processing unit 700. Therefore, it is desirable to specify “all” in the “data acquisition timing” field of the command to acquire coordinate data. Further, since it is only necessary to acquire coordinate data of the touched position, the scanning accuracy may not be high. Therefore, “low” may be specified as the value of the “scan resolution” field of the above command.

  Further, when “coordinate” is specified in the “data type” field of the command, for example, when the user touches the sensor built-in liquid crystal panel 301 with a plurality of fingers or pens at the same time, the coordinate data of the touched position is used. Can be acquired (multi-point detection).

  When acquiring image data of an object such as a document, a command specifying “whole image” in the “data type” field is transmitted to the data processing unit 700. Since it is common to perform a scan in a stationary state, it is not necessary to periodically perform the scan. Therefore, in this case, it is desirable to designate “sense” or “event” in the “data acquisition timing” field. When scanning an object such as a document, it is desirable that the scanning accuracy is high so that the user can easily read the characters. Therefore, it is desirable to designate “high” in the “scan resolution” field.

(Whole image data / Partial image data / Coordinate data)
Next, the whole image data, the partial image data, and the coordinate data will be described with reference to FIG. The image data shown in FIG. 7A is image data obtained as a result of scanning the entire sensor built-in liquid crystal panel 301 when the object is not placed on the sensor built-in liquid crystal panel 301. The image data shown in FIG. 7B is image data obtained as a result of scanning the entire sensor-equipped liquid crystal panel 301 when the user touches the sensor-equipped liquid crystal panel 301 with a finger.

  When the user touches the sensor built-in liquid crystal panel 301 with a finger, the amount of light received by the photosensor circuit 32 in the vicinity of the touch changes, so that the voltage output from the photosensor circuit 32 changes, and as a result, In the image data generated by the sensor control unit 602, the brightness of the pixel value of the portion touched by the user changes.

  In the image data shown in FIG. 7B, the brightness of the pixel value of the portion corresponding to the user's finger is higher than that in the image data shown in FIG. In the image data shown in FIG. 7B, the smallest rectangular area (area PP) that includes all pixel values whose lightness changes more than a predetermined threshold is “partial image data”.

  The image data indicated by the area AP is “whole image data”.

  Also, the coordinate data in the whole image data (area AP) of the representative coordinates Z of the partial image data (area PP) is (Xa, Ya), and the coordinate data in the partial image data (area PP) is (Xp, Yp). It is.

(Configuration of sensor built-in liquid crystal panel)
Next, the configuration of the sensor built-in liquid crystal panel 301 and the configuration of the peripheral circuit 309 of the sensor built-in liquid crystal panel 301 will be described with reference to FIG. FIG. 8 is a block diagram showing the main part of the display / light sensor unit 300, particularly the configuration of the sensor built-in liquid crystal panel 301 and the configuration of the peripheral circuit 309.

  The sensor built-in liquid crystal panel 301 includes a pixel circuit 31 for setting light transmittance (brightness) and an optical sensor circuit 32 that outputs a voltage corresponding to the intensity of light received by the sensor. The pixel circuit 31 is used as a general term for the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b corresponding to the red, green, and blue color filters, respectively.

  The pixel circuits 31 are arranged on the sensor built-in liquid crystal panel 301 in the column direction (vertical direction) and 3n in the row direction (horizontal direction). A set of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b is continuously arranged in the row direction (lateral direction). This set forms one pixel.

  In order to set the light transmittance of the pixel circuit 31, first, the high level voltage (TFT 33 is turned on) to the scanning signal line Gi connected to the gate terminal of the TFT (Thin Film Transistor) 33 included in the pixel circuit 31. Voltage). Thereafter, a predetermined voltage is applied to the data signal line SRj connected to the source terminal of the TFT 33 of the R pixel circuit 31r. Similarly, the light transmittance is also set for the G pixel circuit 31g and the B pixel circuit 31b. Then, by setting these light transmittances, an image is displayed on the sensor built-in liquid crystal panel 301.

  Next, the photosensor circuit 32 is arranged for each pixel. One pixel may be arranged in the vicinity of each of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b.

  In order for the optical sensor circuit 32 to output a voltage corresponding to the light intensity, first, the sensor readout line RWi connected to one electrode of the capacitor 35 and the anode terminal of the photodiode 36 are connected. A predetermined voltage is applied to the sensor reset line RSi. In this state, when light is incident on the photodiode 36, a current corresponding to the amount of incident light flows through the photodiode 36. Then, according to the current, the voltage at the connection point (hereinafter referred to as connection node V) between the other electrode of the capacitor 35 and the cathode terminal of the photodiode 36 decreases. When the power supply voltage VDD is applied to the voltage application line SDj connected to the drain terminal of the sensor preamplifier 37, the voltage at the connection node V is amplified and output from the source terminal of the sensor preamplifier 37 to the sensing data output line SPj. Based on the output voltage, the amount of light received by the optical sensor circuit 32 can be calculated.

  Next, the liquid crystal panel drive circuit 304, the optical sensor drive circuit 305, and the sensor output amplifier 44, which are peripheral circuits of the sensor built-in liquid crystal panel 301, will be described.

  The liquid crystal panel drive circuit 304 is a circuit for driving the pixel circuit 31, and includes a scanning signal line drive circuit 3041 and a data signal line drive circuit 3042.

  The scanning signal line driving circuit 3041 sequentially selects one scanning signal line from the scanning signal lines G1 to Gm for each line time based on the timing control signal TC1 received from the display control unit 601, and A high level voltage is applied to the selected scanning signal line, and a low level voltage is applied to the other scanning signal lines.

  Based on the display data D (DR, DG, and DB) received from the display controller 601, the data signal line driver circuit 3042 generates a predetermined voltage corresponding to the display data for one row for each line time. The data signal lines SR1 to SRn, SG1 to SGn, and SB1 to SBn are applied (line sequential method). Note that the data signal line driver circuit 3042 may be driven by a dot sequential method.

  The optical sensor driving circuit 305 is a circuit for driving the optical sensor circuit 32. Based on the timing control signal TC2 received from the sensor control unit 602, the optical sensor driving circuit 305 selects a predetermined sensor readout signal line from the sensor readout signal lines RW1 to RWm for each line time. A read voltage is applied, and a voltage other than a predetermined read voltage is applied to the other sensor read signal lines. Similarly, based on the timing control signal TC2, a predetermined reset voltage is applied to the sensor reset signal line selected from the sensor reset signal lines RS1 to RSm for each line time, and the others. A voltage other than a predetermined reset voltage is applied to the sensor reset signal line.

  The sensing data output signal lines SP1 to SPn are grouped into p groups (p is an integer of 1 to n), and the sensing data output signal lines belonging to each group are connected via a switch 47 that is sequentially turned on in time division. And connected to the sensor output amplifier 44. The sensor output amplifier 44 amplifies the voltage from the group of sensing data output signal lines connected by the switch 47 and outputs the amplified voltage to the signal conversion circuit 306 as sensor output signals SS (SS1 to SSp).

(Data display / configuration of main part of sensor device 100)
Next, the configuration of the main part of the main control unit 800 in the data display / sensor device 100 will be described below with reference to FIG. FIG. 1 is a block diagram showing a main configuration of the main control unit 800 in the data display / sensor device 100.

  As shown in FIG. 1, the main control unit 800 in the data display / sensor device 100 includes an image number calculation unit 11, an instruction direction detection unit 12, an execution processing determination unit 13, a display position determination unit 14, and a display processing control unit 15. I have. These members will be described below.

(Image number calculation unit 11)
The image number calculation unit 11 calculates the number of images located on the image displayed on the display / light sensor unit 300 among the images of the object detected by the display / light sensor unit 300.

(Instruction direction detector 12)
The indication direction detection unit 12 is based on the position coordinates of the object that indicates a point on the image displayed on the display / light sensor unit 300 and the position coordinates of the point indicated next to this position coordinate. The indicated direction is detected.

  Here, in the present embodiment, a case where the indication direction is detected based on the direction of a vector obtained from temporally continuous position coordinates in the image of the target object will be described as an example. That is, in the present embodiment, a case where the pointing direction is detected based on a stroke obtained by sliding an object (for example, a user's finger or a touch pen) will be described as an example. Of course, the method of detecting the indication direction is not limited to this, and any method may be used as long as it is detected in accordance with a change in the position coordinates of the target object.

(Execution processing determination unit 13)
The execution process determination unit 13 determines a process to be executed in the data display / sensor device 100 according to the type of image at least partially covered by the object and the number of images calculated by the image number calculation unit 11. To do.

  In the present embodiment, when the image is an icon associated with the execution file of the application and the number of images of the object covering the icon is one, the application is started by executing the execution file. More specifically, the process executed in the data display / sensor device 100 is started by the application, and a window image of the started application is displayed.

  When the number of images of the object covering the image is two, the image is moved as it is with respect to another display / light sensor unit 300.

  If the image is a window image of the application and the number of images of the target covering the window image is three, the display of the window image is terminated. More specifically, an icon associated with the execution file of the application is displayed instead of the window image of the application.

  Note that program data including an execution file of an application executed in the data display / sensor device 100 and an icon associated with the execution file are preferably stored in the storage unit 901. That is, the main control unit 800 reads out icons from the storage unit 901 and displays them.

(Display position determination unit 14)
The display position determination unit 14 displays the display / light sensor unit 300 that displays an image obtained by executing the process determined by the execution process determination unit 13 based on the instruction direction detected by the instruction direction detection unit 12. decide.

  At this time, the display position determination unit 14 determines the display / light sensor unit 300 based on the recognition of the arrangement of the display / light sensor unit 300 in the data display / sensor device 100.

  For example, when the data display / sensor device 100 has the configuration shown in FIG. 9A, when the instruction direction designated by the user in the second display / light sensor unit 300B is the upward direction, the second display / light An image is displayed on the first display / light sensor unit 300A in the upper housing 40 provided on the upper part of the sensor unit 300B.

  In addition, when the data display / sensor apparatus 100 is the structure shown in FIG.9 (c), ie, the definition of the instruction | indication direction in the case where it consists of three housings is shown in FIG. FIG. 10 is a diagram showing the definition of the pointing direction when the data display / sensor device 100 is composed of three housings.

  As shown in FIG. 10, when the direction indicated by the user in the first display / light sensor unit 300 </ b> A in the first housing 44 is the direction A <b> 1, the second display / light sensor unit in the second housing 45. When an image is displayed on 300B and the pointing direction is the direction A2, the image is displayed on the third display / light sensor unit 300C in the third housing 46.

  Similarly, when the direction indicated by the user in the second display / light sensor unit 300B is the direction B1, an image is displayed on the first display / light sensor unit 300A, and the direction indicated is the direction B2. Displays an image on the third display / light sensor unit 300C. When the pointing direction indicated by the user in the third display / light sensor unit 300C is the direction C1, an image is displayed on the second display / light sensor unit 300B, and when the pointing direction is the direction C2, 1 display / image is displayed on the optical sensor unit 300A.

  Thus, each indication direction when the data display / sensor device 100 is configured as shown in FIG. 9C is determined based on a fan-shaped region centered on the representative coordinates in the partial image of the object. Good. The size of each sector, that is, the central angle of the sector, is based on recognition of the arrangement of the display / light sensor unit 300 in the data display / sensor device 100.

  The size of each sector center may be a preset invariant value or may be set as appropriate by the user. Further, the object may cover the image, and the fan-shaped area shown in FIG. 10 may be actually displayed on the display / light sensor unit 300.

  In the case where there are four or more display / light sensor units 300, the number of fan-shaped regions around the representative coordinates in the partial image of the object may be increased.

(Display processing control unit 15)
The display process control unit 15 instructs the display / light sensor unit 300 determined by the display position determination unit 14 to execute the process determined by the execution process determination unit 13.

  For example, the process determined by the execution process determination unit 13 is application activation, and the display / light sensor unit 300 determined by the display position determination unit 14 is the first display / light sensor unit in the upper housing 40. In the case of 300A, the display processing control unit 15 reads an application to be activated from the storage unit 901, and outputs an instruction to display a window image of the read application on the first display / light sensor unit 300A.

(Details of operation processing when displaying window images)
Next, details of operation processing when displaying a window image of an application on the desired display / light sensor unit 300 in the data display / sensor device 100 will be described with reference to FIG. FIG. 11 is a flowchart showing an operation when displaying a window of an application on the desired display / light sensor unit 300.

  First, the image number calculation unit 11 determines how many images are located on the image displayed on the display / light sensor unit 300 among the images of the object input via the data processing unit 700. Is determined (step S1).

  When the image number calculation unit 11 determines that the number of images is one, the execution process determination unit 13 determines whether the image covered by the object is an icon associated with the execution file that executes the application. Is determined (step S2). If the image covered by the object is not an icon (No in step S2), the data display / sensor device 100 ends the process. On the other hand, when the image covered with the object is an icon (Yes in step S2), the execution process determining unit 13 determines the process to be executed in the data display / sensor device 100 to be an application activation (step S3). ).

  The pointing direction detection unit 12 detects the pointing direction from temporally continuous position coordinates of the image of the object covering the icon, which is input via the data processing unit 700 (step S4). Then, the display position determination unit 14 determines the display / light sensor unit 300 that displays the window image displayed by starting the application, based on the instruction direction detected in step S4 (step S5).

  Finally, the display processing control unit 15 causes the display / light sensor unit 300 determined by the display position determination unit 14 to display the window image of the application to be activated by executing the execution file associated with the icon. An instruction is output to the processing unit 700 (step S6).

(Specific Example of Displaying Window Image on Desired Display / Optical Sensor Unit 300)
Next, a specific example of displaying a window image on the display / light sensor unit 300 desired by the user will be described below with reference to FIGS. FIGS. 12A and 12B are diagrams showing a specific example of displaying a window image of the executed application on the display / light sensor unit 300 desired by the user, and FIG. It is a figure which shows operation | movement, (b) is a figure which shows the state which displayed the window image based on the operation | movement shown to (a).

  In this section, by touching (tapping) an icon associated with an execution file for executing the “spreadsheet” application displayed on the second display / light sensor unit 300B in the lower casing 41, An example will be described in which a window image of the “spreadsheet” application is displayed on the display / light sensor unit 300A in the housing 40.

  As shown in FIG. 12A, the user touches (tap) the icon displayed on the second display / light sensor unit 300 </ b> B in the lower housing 41 with one finger and raises the finger touching the icon. Slide in the direction. At this time, the image count calculation unit 11 in the data display / sensor device 100 detects that there is one image of the user's finger covering the icon for starting the “spreadsheet” application. The execution process determination unit 13 determines the execution of the process for starting the “spreadsheet” application by executing the execution file associated with the icon based on the detection result in the image number calculation unit 11.

  At the same time, the indication direction detection unit 12 detects that the user's finger has been slid upward. Based on this, the display position determination unit 14 displays the window image displayed by executing the execution file associated with the icon for starting the “spreadsheet” application in the first display / It is determined to display on the optical sensor unit 300A.

  As a result, as shown in FIG. 12B, the window image of the activated “spreadsheet” application can be displayed on the first display / light sensor unit 300 </ b> A in the upper housing 40. At this time, as shown in FIG. 12B, the icon continues to be displayed in the second display / light sensor unit 300B in the lower housing 41 by displaying the color lightly (for example, gray-out display). Alternatively, the display may be turned off.

(Details of operation processing when moving and displaying images)
Next, in the data display / sensor device 100, details of operation processing when moving and displaying an image on the desired display / light sensor unit 300 will be described below with reference to FIG. FIG. 13 is a flowchart showing an operation when moving and displaying an image on the desired display / light sensor unit 300.

  The image number calculation unit 11 determines that the number of images positioned on the image displayed on the display / light sensor unit 300 among the images of the object input via the data processing unit 700 is two. In this case, the execution process determining unit 13 determines the process to be executed in the data display / sensor device 100 as the movement process of the image covered by the object (that is, the image selected by the user) (step S11). .

  The pointing direction detection unit 12 detects the pointing direction from temporally continuous position coordinates in the image of the object covering the image, which is input via the data processing unit 700 (step S12). Then, the display position determination unit 14 determines the display / light sensor unit 300 that moves the image covered with the object based on the indication direction detected in step S12 (step S13).

  Finally, the display processing control unit 15 outputs an instruction to the data processing unit 700 so that the display / light sensor unit 300 determined by the display position determination unit 14 displays an image (step S14).

(Specific example of moving and displaying an image on the desired display / light sensor unit 300)
Next, a specific example in which the window image is moved and displayed on the display / light sensor unit 300 desired by the user will be described below with reference to FIGS. 14 (a) and 14 (b). FIGS. 14A and 14B are diagrams illustrating a specific example in which a window image is moved and displayed on the display / light sensor unit 300 desired by the user, and FIG. 14A is a diagram illustrating an operation of moving the window image. (B) is a figure which shows the state which moved and displayed the window image based on the operation | movement shown to (a).

  14A and 14B, the window image of the “spreadsheet” application displayed on the first display / light sensor unit 300 </ b> A in the upper housing 40 is displayed in the second display in the lower housing 41. A case will be described as an example where the display is moved to the optical sensor unit 300B.

  As shown in FIG. 14A, the user touches (tap) the window image displayed on the first display / light sensor unit 300 </ b> A in the upper housing 40 with two fingers and touches the window image. Slide down. At this time, the image number calculation unit 11 in the data display / sensor device 100 detects that there are two images of the user's finger covering a part of the window image of the “spreadsheet” application. The execution process determination unit 13 determines the execution of the process for moving the window image based on the detection result in the image number calculation unit 11.

  At the same time, the indication direction detection unit 12 detects that the user's finger has been slid downward. Based on this, the display position determination unit 14 determines to move the window image of the “spreadsheet” application to the second display / light sensor unit 300 </ b> B in the lower housing 41.

  As a result, as shown in FIG. 14B, the window image of the “spreadsheet” application displayed on the first display / light sensor unit 300 </ b> A in the upper housing 40 is left as it is in the lower housing 41. The second display / light sensor unit 300B can be moved and displayed.

  In FIGS. 14A and 14B, an example of moving the window image of the “spreadsheet” application is described as an example. However, the images that can be moved are not limited to window images. Absent. For example, an icon displayed on the display / light sensor unit 300 can be moved to another display / light sensor unit 300 to be displayed.

  A specific example of moving and displaying icons on the display / light sensor unit 300 desired by the user will be described below with reference to FIGS. 15 (a) and 15 (b). FIGS. 15A and 15B are diagrams showing a specific example in which an icon is moved and displayed on the display / light sensor unit 300 desired by the user, and FIG. 15A is a diagram showing an operation of moving the icon. (B) is a figure which shows the state which moved and displayed the icon based on the operation | movement shown to (a).

  In FIGS. 15A and 15B, the icon associated with the execution file of the “spreadsheet” application displayed on the second display / light sensor unit 300B in the lower casing 41 is displayed in the upper casing. A case where the display is moved to the first display / light sensor unit 300A in 40 and displayed will be described as an example.

  As shown in FIG. 15A, while the user touches (tap) the icon displayed on the second display / light sensor unit 300B in the lower housing 41 with two fingers, Slide in the direction. At this time, the image number calculation unit 11 in the data display / sensor device 100 detects that there are two images of the user's finger covering a part of the icon. The execution process determination unit 13 determines the execution of the process of moving the icon based on the detection result in the image number calculation unit 11.

  At the same time, the indication direction detection unit 12 detects that the user's finger has been slid upward. Based on this, the display position determination unit 14 determines to move the icon to the first display / light sensor unit 300 </ b> A in the upper housing 40.

  As a result, as shown in FIG. 15B, the icon associated with the execution file of the “spreadsheet” application displayed on the second display / light sensor unit 300 </ b> B in the lower housing 41 remains unchanged. Thus, the first display / light sensor unit 300A in the upper housing 40 can be moved and displayed.

(Details of operation processing when window image display is terminated)
Next, details of the operation processing when the display of the window image displayed on the display / light sensor unit 300 in the data display / sensor device 100 is terminated will be described below with reference to FIG. FIG. 16 is a flowchart showing an operation when the display of the window image displayed on the display / light sensor unit 300 is ended.

  The image number calculation unit 11 determines that the number of images located on the image displayed on the display / light sensor unit 300 among the images of the object input via the data processing unit 700 is three. When it does, the execution process determination part 13 determines whether the image covered with a target object is a window image (step S21). If the image covered by the object is not a window image (No in step S21), the data display / sensor device 100 ends the process. On the other hand, when the execution process determining unit 13 determines that the image covered by the object is a window image (Yes in step S21), the execution process determining unit 13 executes in the data display / sensor device 100. The processing to be performed is determined to end the display of the window image (step S22).

  The pointing direction detection unit 12 detects the pointing direction from temporally continuous position coordinates in the image of the object covering the window image, which is input via the data processing unit 700 (step S23). Then, the display position determination unit 14 displays an icon associated with the execution file to be displayed instead of the window image covered with the object based on the indication direction detected in step S23. 300 is determined (step S24).

  Finally, the display processing control unit 15 ends the display of the displayed window image and instructs the data processing unit 700 to display an icon on the display / light sensor unit 300 determined by the display position determination unit 14. To output an instruction (step S25).

(Specific Example of Moving and Displaying Images on Desired Display / Optical Sensor Unit 300)
Next, a specific example of terminating the display of the window image displayed on the display / light sensor unit 300 will be described below with reference to FIGS. 17 (a) and 17 (b). FIGS. 17A and 17B are diagrams showing a specific example of ending the display of the window image displayed on the display / light sensor unit 300, and FIG. 17A shows the operation of ending the display of the window image. (B) is a figure which shows the state which ended the display of the window image based on the operation | movement shown to (a).

  17A and 17B, the display of the window image of the “spreadsheet” application displayed on the first display / light sensor unit 300A in the upper housing 40 is terminated, and the “spreadsheet” is displayed. An example in which an icon associated with an execution file for executing the application is displayed on the second display / light sensor unit 300B in the lower housing 41 will be described as an example.

  As shown in FIG. 17A, the user touches (tap) the window image displayed on the first display / light sensor unit 300A of the upper housing 40 with three fingers and touches the window image. Slide down. At this time, the image count calculation unit 11 in the data display / sensor device 100 detects that there are three images of the user's finger covering the window image of the “spreadsheet” application. The execution process determination unit 13 determines the execution of the process for ending the display of the window image based on the detection result in the image number calculation unit 11.

  At the same time, the indication direction detection unit 12 detects that the user's finger has been slid downward. Based on this, the display position determination unit 14 determines to display an icon for starting the “spreadsheet” application on the second display / light sensor unit 300 </ b> B in the lower housing 41.

  As a result, as shown in FIG. 17B, the display of the window image of the “spreadsheet” application in the first display / light sensor unit 300A in the upper housing 40 is terminated, and the “spreadsheet” application is turned off. The icon associated with the execution file to be executed can be displayed on the second display / light sensor unit 300B in the lower housing 41. As shown in FIG. 12B, when displaying the window image with the icon color lightened, the process of restoring the icon color may be executed. Good.

(Advantages of data display / sensor device 100)
As described above, in the data display / sensor device 100 according to the present invention, the application from the indication direction detection unit 12 that detects the indication direction indicated by the user and the image display area arranged in the direction indicated by the indication direction. A display position determining unit 14 that determines the display / light sensor unit 300 that displays the window image displayed by the display, and a display display processing control unit 15 that displays the window image on the determined display / light sensor unit 300. ing.

  Thus, in the data display / sensor device 100, the user can display the window image on the desired display / light sensor unit 300 only by instructing the direction of the display / light sensor unit 300 to display the window image. . Therefore, the data display / sensor device 100 can display the window image of the activated application in the image display area desired by the user without forcing the user to perform complicated operations.

  The data display / sensor device 100 according to the present invention also includes an image number calculation unit 11 that calculates the number of images of an object covering at least a part of an image displayed on any of the display / light sensor units 300. And an execution process determination unit 13 that determines a process to be executed in the data display / sensor device 100 according to the type of image and the number of images calculated in the image number calculation unit 11.

  Thereby, the data display / sensor device 100 can cause the data display / sensor device 100 to execute a desired process only by changing the number of images of the object. For example, the data display / sensor device 100 can execute a desired process by changing the number of fingers of the user or the number of touch pens. Therefore, the data display / sensor device 100 can set a process to be executed in the data display / sensor device 100 with an easy operation.

  In addition, the data display / sensor device 100 according to the present invention detects the indication direction according to the scroll direction of the object.

  As a result, the user can set the display / light sensor unit 300 on which the image is displayed more intuitively, so that the user is unfamiliar with the operation of the data display / sensor device 100 such as a young person or an elderly person. The instruction direction can be set by an operation that can be easily understood even by a user.

(Operation example using command of main control unit 800)
In the description of the operation process of the main control unit 800 described above, the operation of the main control unit 800 has been described without using a command in order to facilitate understanding of the present invention. An operation of the main control unit 800 will be described while touching a command transmitted to the processing unit 700. In this section, the data display / sensor device 100 includes two display / light sensor units 300, and the execution associated with the icon displayed on the second display / light sensor unit 300B in the lower housing 41 is performed. An operation process in the case where a window image of an application activated by executing a file is displayed on the first display / light sensor unit 300A in the upper housing 40 will be described as an example.

  First, in the second display / light sensor unit 300B as the lower screen, the image number calculation unit 11 calculates the value of the “scan panel” field in order to calculate the number of images of an object covering at least a part of the icon. Specify “second display / light sensor section” (“001”), and specify “event” (“01”) or “all” (“10”) as the value of the “data acquisition timing” field. The command specifying both “partial image” and “coordinate” (“011”) as the value of the “data type” field is transmitted to the data processing unit 700.

  Thereby, the image number calculation unit 11 can receive the partial image and the coordinates of the partial image (coordinates in the entire image) in the second display / light sensor unit 300B from the data processing unit 700. Based on the coordinates, the number of images of the object covering at least a part of the icon can be calculated. When the number of images covering at least a part of the icon is one, the data display / sensor device 100 executes the execution file associated with the icon stored in the storage unit 901 as described above. To start the application.

  In addition, in order to detect the pointing direction indicated by the user in the second display / light sensor unit 300B which is the lower screen, the pointing direction detection unit 12 sets “second display / light sensor” as the value of the “scan panel” field. "Part" ("001"), "all" ("10") as the value of the "data acquisition timing" field, and "coordinate" ("001") as the value of the "data type" field Is transmitted to the data processing unit 700.

  As a result, the indication direction detection unit 12 can receive the change over time in the representative coordinates of the partial image in the second display / light sensor unit 300B from the data processing unit 700, and thus the moving direction of the object, that is, The direction indicated by the user can be detected.

  Then, the display processing control unit 15 transmits the window image of the application to be activated to the data processing unit 700 together with the command specifying “first display / light panel” (“010”) as the value of the “display panel” field. To do. Thereby, the window image of the application executed by the execution file associated with the selected icon can be displayed on the first display / light sensor unit 300A.

(Additional notes)
In the present embodiment, as shown in FIGS. 9A to 9C, a case where two display / light sensor units 300 are continuously provided in the vertical direction and the horizontal direction will be described as an example. However, three or more display / light sensor units 300 may be continuously provided in the vertical direction or the horizontal direction.

  When three or more display / light sensor units 300 are continuously provided in the vertical direction or the horizontal direction, an image is displayed on any display / light sensor unit 300 together with an instruction direction for displaying an image. It is preferable to include a parameter detection unit that detects a parameter for determining whether to perform the operation. This can be detected from the position coordinates of the image of the object and the magnitude of the vector detected based on the position coordinates of the image detected after the image. For example, when the pointing direction is determined by scrolling with the user's finger, it may be determined by the scroll amount per unit time (that is, scroll speed).

  More specifically, for example, when three or more display / light sensor units 300 are continuously provided in the vertical direction and the user designates an instruction direction on the bottom display / light sensor unit 300, parameter detection is performed. When the magnitude of the vector detected by the unit is larger than a predetermined threshold, the display position determination unit 14 determines the uppermost display / light sensor unit 300 as the display / light sensor unit 300 that displays an image. When the magnitude of the vector detected by the parameter detection unit is smaller than a predetermined threshold value, the display position determination unit 14 displays the image on the adjacent (ie, middle) display / photosensor unit 300. 300 is determined.

  In the case where four or more display / light sensor units 300 are continuously provided in the vertical direction or the horizontal direction, the threshold value may be set in more detail to cope with it.

  Accordingly, even if three or more display / light sensor units 300 are continuously provided in the vertical direction or the horizontal direction, the display position determination unit 14 can display an image on any display / light sensor unit 300. Can be determined.

  In the present embodiment, basically, the display / light sensor unit 300 is provided in each case, that is, the image display area in the display / light sensor unit 300 is physically divided. However, the present invention is not limited to this. For example, there may be a configuration in which one display / light sensor unit 300 is provided and the image display region is virtually divided into a plurality of portions in one display / light sensor unit 300. In addition, the display / light sensor unit 300 may be provided in each case, and the display / light sensor unit 300 may be virtually divided into a plurality of image display areas.

  Further, in order to prevent the direction unintended by the user from being detected as the indication direction, the indication direction detection unit 12 determines the indication direction only when the amount of change in position coordinates per unit time is greater than or equal to a predetermined amount. It is preferable to detect. That is, it is preferable that the pointing direction detection unit 12 detects the pointing direction only when the object is scrolled at a predetermined speed or higher.

  In the above-described embodiment, the application is started when the number of images of the object is one, the image is moved when the number is two, and the display of the window image is ended when the number is three. Although a case has been described as an example, of course, the number of images of the object can be arbitrarily set.

(Program and recording medium)
Finally, the circuit control unit 600, the data processing unit 700, and the main control unit 800 included in the data display / sensor device 100 may be configured by hardware logic. Alternatively, it may be realized by software using a CPU (Central Processing Unit) as follows.

  That is, the circuit control unit 600, the data processing unit 700, and the main control unit 800 are a CPU such as an MPU that executes instructions of a program that realizes each function, a ROM (Read Only Memory) that stores the program, and executes the program. A RAM (Random Access Memory) that expands into a possible format, and a storage device (recording medium) such as a memory that stores the program and various data are provided.

  The object of the present invention is not limited to the case where the program memory of the circuit control unit 600, the data processing unit 700, and the main control unit 800 is fixedly supported, but the program code of the above program (execution format program, intermediate code) Also, a recording medium in which a program or a source program is recorded is supplied to the data display / sensor device 100, and the data display / sensor device 100 reads and executes the program code recorded in the recording medium. Achievable.

  The recording medium is not limited to a specific structure or type. That is, the recording medium includes, for example, a tape system such as a magnetic tape and a cassette tape, a magnetic disk such as a floppy (registered trademark) disk / hard disk, and an optical disk such as a CD-ROM / MO / MD / DVD / CD-R. System, a card system such as an IC card (including a memory card) / optical card, or a semiconductor memory system such as a mask ROM / EPROM / EEPROM / flash ROM.

  The object of the present invention can be achieved even if the circuit control unit 600, the data processing unit 700, and the main control unit 800 (or the data display / sensor device 100) are configured to be connectable to a communication network. In this case, the program code is supplied to the circuit control unit 600, the data processing unit 700, and the main control unit 800 via the communication network. The communication network only needs to be able to supply program codes to the circuit control unit 600, the data processing unit 700, and the main control unit 800, and is not limited to a specific type or form. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication network, etc. may be used.

  The transmission medium constituting the communication network may be any medium that can transmit the program code, and is not limited to a specific configuration or type. For example, even with wired lines such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL (Asymmetric Digital Subscriber Line) line, infrared rays such as IrDA and remote control, Bluetooth (registered trademark), 802.11 wireless, HDR, mobile phone It can also be used by radio such as a telephone network, a satellite line, and a terrestrial digital network. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  The present invention has been specifically described above based on the embodiments. However, the present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims, and different implementations are possible. Embodiments obtained by appropriately combining the technical means disclosed in each form are also included in the technical scope of the present invention.

  The image display / image detection apparatus according to the present invention can be suitably applied to a mobile phone, a portable game machine, a computer, or the like provided with an image display monitor. Further, the present invention can be suitably applied to an image display device in which an image display area is virtually divided.

It is a block diagram which shows the principal part structure of the main control part in the data display / sensor apparatus which concerns on this invention. It is a figure which shows typically the cross section of a liquid crystal panel with a built-in sensor. It is a figure which shows typically the detection method which detects the position which touched on the liquid crystal panel with a built-in sensor, (a) is a schematic diagram which shows a mode that the position which the user touched is detected by detecting a reflected image. (B) is a schematic diagram which shows a mode that the position which the user touched is detected by detecting a shadow image. It is a block diagram which shows the structure of the data display / sensor apparatus based on this invention. It is a figure which shows an example of the frame structure of a command typically. It is a figure explaining an example of the value which can be specified to each field contained in a command, and its outline. It is a figure which shows the image data obtained as a result of scanning the liquid crystal panel with a built-in sensor, (a) is a result of scanning the whole liquid crystal panel with a built-in sensor when the object is not placed on the liquid crystal panel with a built-in sensor. (B) is image data obtained as a result of scanning the entire sensor-equipped liquid crystal panel when the user touches the sensor-equipped liquid crystal panel with a finger. It is a block diagram which shows the structure of a liquid crystal panel with a built-in sensor, and the structure of a peripheral circuit. It is a figure which shows typically the external appearance of the data display / sensor apparatus which concerns on this invention, (a) is a figure which shows the structure which consists of an upper housing | casing and a lower housing | casing, (b) is a left housing | casing and right side housing | casing. It is a figure which shows the structure which consists of a housing | casing, (c) is a figure which shows the structure provided with three display / light sensor parts. It is a figure which shows the definition of the instruction | indication direction in case a data display / sensor apparatus consists of three housings. It is a flowchart which shows the operation | movement at the time of displaying the window image of an application in the display / light sensor part which a user desires. It is a figure which shows the specific example which displays the window image of an application on the display / light sensor part which a user desires, (a) is a figure which shows the operation | movement at the time of executing an application, (b) is (a). It is a figure which shows the state which displayed the window image based on the operation | movement shown in FIG. It is a flowchart which shows the operation | movement at the time of moving and displaying an image on the display / light sensor part which a user desires. It is a figure which shows the specific example which moves and displays a window image on the display / light sensor part which a user desires, (a) It is a figure which shows the operation | movement which moves a window image, (b) is shown to (a). It is a figure which shows the state which moved and displayed the window image based on operation | movement. It is a figure which shows the specific example which moves and displays an icon on the display / light sensor part which a user desires, (a) It is a figure which shows the operation | movement which moves an icon, (b) is operation | movement shown to (a). It is a figure which shows the state which moved and displayed the icon based on. It is a flowchart which shows the operation | movement at the time of complete | finishing the display of the window image currently displayed on the display / light sensor part. It is a figure which shows the specific example which complete | finishes the display of the window image currently displayed on the display / light sensor part, (a) It is a figure which shows the operation | movement which complete | finishes the display of a window image, (b) is a figure to (a). It is a figure which shows the state which complete | finished the display of the window image based on the operation | movement shown. It is a figure which shows the conventional example which displays the window image of the executed application, (a) is a figure which shows the operation | movement which performs an application, (b) is the window displayed based on the operation | movement shown to (a) It is a figure which shows the operation | movement which moves an image, (c) is a figure which shows the state which moved and displayed the window image based on the operation | movement shown in (b).

Explanation of symbols

11 Image number calculation unit (calculation means)
12 Instruction direction detection unit 13 Execution process determination unit (process determination unit)
14 Display position determination unit (display area setting means)
DESCRIPTION OF SYMBOLS 15 Display processing control part 40 Upper housing | casing 41 Lower housing | casing 42 Left housing | casing 43 Right housing | casing 44 1st housing | casing 45 2nd housing | casing 46 3rd housing | casing 100 Data display / sensor apparatus (image display / image detection apparatus)
300 Display / Optical sensor part (planar member)
300A First display / light sensor unit 300B Second display / light sensor unit 300C Third display / light sensor unit 301 Liquid crystal panel with built-in sensor 600 Circuit control unit 700 Data processing unit 800 Main control unit

Claims (10)

An image display / image detection apparatus including one or more planar members for displaying an image and detecting an image of an object located nearby.
When a point on an icon displayed in any of a plurality of image display areas provided on the one or more planar members is designated by an object located in the vicinity of the planar member, the position of this point An indication direction detecting means for detecting an indication direction based on the coordinates and a position coordinate of a point indicated by an object located in the vicinity of the planar member next to the point;
A display area in which one of the other image display areas in the indicated direction with respect to the image display area in which the icon is displayed is set as an image display area for displaying a window displayed by the application associated with the icon Setting means;
Display processing control means for displaying a window displayed by the application associated with the icon in the image display area set in the display area setting means;
An image display / image detection apparatus comprising: An image display / image detection apparatus including one or more planar members for displaying an image and detecting an image of an object located in the vicinity thereof,
When a point on the first image displayed in any of the plurality of image display areas provided on the one or more planar members is indicated by an object located in the vicinity of the planar member, this Calculating means for calculating the number of images of an object located in the vicinity of the planar member indicating a point;
A process determining unit that determines a process to be executed in the image display / image detection device according to the type of the first image and the number of images calculated by the calculating unit;
Based on the position coordinates of the point on the first image instructed by the object and the position coordinates of the point instructed by the object located in the vicinity of the planar member next to the point, the indication direction is determined. Indicating direction detecting means for detecting;
The second image obtained by executing the process determined by the process determining means on any one of the other image display areas in the indicated direction with respect to the image display area on which the first image is displayed. Display area setting means for setting as an image display area for displaying,
Display processing control means for displaying the second image in the image display area set by the display area setting means;
An image display / image detection apparatus comprising:   When the first image is an icon and the number of images calculated by the calculation unit is a first value, the process determination unit performs a process to be executed in the image display / image detection apparatus. The image display / image detection apparatus according to claim 2, wherein the image display / image detection apparatus is determined to execute an application associated with the icon.   When the first image is a window displayed by the application and the number of images calculated by the calculation unit is a third value, the processing determination unit is configured to display the image display / image detection device. The image display / image detection apparatus according to claim 3, wherein the process to be executed is determined to end the display of the window.   When the number of images calculated by the calculation unit is the second value, the process determination unit performs the process to be executed in the image display / image detection apparatus by displaying the first image. The image display / image detection apparatus according to claim 2, wherein the image display / image detection apparatus is determined to be changed.   6. The pointing direction detecting unit according to claim 1, wherein the pointing direction detecting unit detects the pointing direction with reference to a locus of position coordinates of an image obtained by sequentially detecting an object located in the vicinity of the planar member. The image display / image detection device according to item.   7. The image display / image detection apparatus according to claim 1, further comprising an image display apparatus corresponding to each of the image display areas. An image display method in an image display / image detection apparatus including one or more planar members for displaying an image and detecting an image of an object located nearby,
When a point on an icon displayed in any of a plurality of image display areas provided on the one or more planar members is designated by an object located in the vicinity of the planar member, the position of this point An indication direction detection step for detecting an indication direction based on the coordinates and a position coordinate of a point indicated by an object located in the vicinity of the planar member next to the point;
A display area in which one of the other image display areas in the indicated direction with respect to the image display area in which the icon is displayed is set as an image display area for displaying a window displayed by the application associated with the icon Configuration steps;
A display processing control step of displaying a window displayed by the application associated with the icon in the image display region set in the display region setting step;
An image display method comprising:   A program for operating a computer included in the image display / image detection apparatus according to claim 1, wherein the program causes the computer to function as each unit.   A computer-readable recording medium in which the program according to claim 9 is recorded.

Images