JP2007058552A - Display device and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To carry out a high-level operation based on simultaneous contact in a plurality of positions while performing light emission and light reception simultaneously or alternately. <P>SOLUTION: An input/output display 20 performs display by a plurality of display elements arranged in matrix form, and performs output by light reception elements arranged adjacently to a plurality of display elements individually. Based on a light reception signal outputted by the light reception element, a position in contact with or close to a display surface is determined by an input information analysis part 14. When contact/adjacence in a plurality of positions is determined in the input information analysis part 14, predetermined processing is carried out by a control part 11 according to change of a plurality of contact/adjacent positions. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a display device and a display method suitable for application to, for example, a liquid crystal display or an EL (electro-luminescence) display, and more particularly to a display technology capable of receiving light in parallel with light emission.

  Conventionally, when a touch panel that can be operated by touching a display screen of a display device such as a television receiver is configured, a touch panel separate from the display device is stacked on the display screen. .

  As a configuration using a separate touch panel, for example, a transparent and thin input detection device is pasted on a screen. This is a touch sensor using a conductive film, and includes a pressure-sensitive sensor that detects pressure and a capacitive sensor that changes by contact with a human body. There is also an electromagnetic induction type in which a position is input using a special pen. These have a structure in which another special position detection panel is stacked on the surface of the display panel.

  When these detection panels are overlapped with the display panel, the principle of touch detection is simple. However, since some members are overlapped on the display panel, display quality is inevitably deteriorated. Further, since the detection method is mainly a method for detecting a change in capacitance, it is difficult to detect two or more inputs simultaneously.

  There is an optical type of touch panel system that does not add a panel to the surface. Light emitting elements (such as light emitting diodes) and phototransistors are arranged in combination on the top, bottom, left and right of the panel, and the position is detected by light shielding with a finger or the like. In such an optical system, display quality does not deteriorate, but a device installed around the display device becomes large and unsuitable for portable devices.

  In order to solve the disadvantages of these conventional touch panels, in recent years, it has been proposed that the screen of the display device functions as a touch panel without providing a separate touch panel. Patent Literature 1 discloses a display device that performs such light emission and light reception in parallel.

As an example of a display device that performs such light emission and light reception in parallel, for example, display (light emission) on a light emitting element for image display arranged on the display surface is intermittently performed, and the light emission is paused. In addition, it has been proposed that charges corresponding to light reception are accumulated in a light receiving element arranged adjacent to the light emitting element, and light is received by the light receiving element during a period in which display (light emission) is suspended.
Japanese Patent Laying-Open No. 2004-127272 (FIG. 5)

  By the way, the touch panel which has been put into practical use is basically for detecting contact at one place on the display screen, and it has been difficult to simultaneously detect contact at a plurality of places. Therefore, as a usage method, relatively simple operations such as displaying operation buttons on the screen and operating the buttons touched by the user are mostly, and advanced operations are difficult. In the case of the display device described in Patent Document 1, since the state of touching the display surface is detected as an image, it is possible to detect contact at a plurality of locations. There has been no development of operations.

  The present invention has been made in view of such a point, and is capable of performing advanced operations by simultaneous contact at a plurality of locations after light emission and light reception are performed simultaneously (or alternately) in parallel. Objective.

  The present invention performs display on a plurality of display elements arranged in a matrix, and touches or approaches the display surface based on a light reception signal output by a light receiving element arranged in proximity to each of the plurality of display elements. When a position is determined and contact or proximity at a plurality of locations is determined based on the determination, a predetermined process is performed according to a change in the contact position or proximity location at the plurality of locations.

  By doing in this way, various processing becomes possible according to the change of the contact position of a plurality of places, or a proximity position.

  According to the present invention, various types of processing can be performed according to changes in the contact position or proximity position of a plurality of locations. For example, an operation for changing a display image or an operation for changing an application can be directly instructed. Advanced operation by touching the screen becomes possible.

  Hereinafter, an embodiment of the present invention will be described with reference to FIGS.

  In this example, the present invention is applied to a display device configured as a liquid crystal display, and a light receiving element is arranged adjacent to each light emitting element constituting the liquid crystal display, and light emission (display) and light reception (reading) are performed. ) In parallel. Here, the display of this example capable of emitting light and receiving light in parallel is referred to as an I / O display because it is a display capable of simultaneously inputting (receiving light) and outputting (displaying) an image. Further, as will be described later, the I / O display of this example can detect not only a touch in a state where the screen is touched but also an object close to the screen. In some cases, proximity detection is also included, unless otherwise specified.

  FIG. 1 is a block diagram illustrating a configuration example of the display device of this example. The application program execution unit 11 performs processing for displaying an image corresponding to the application being executed by the application program execution unit 11, detects contact of the display panel, and performs processing according to the touched display location. It is like that. The application program execution unit 11 includes a communication unit 11a and can communicate via an external network, and the control unit 11b performs an application program execution process. An instruction to display an image from the application program execution unit 11 is sent to the display display signal processing circuit 12, and driving for displaying an image on the I / O display panel 20 is performed.

  The I / O display panel 20 is configured as a liquid crystal display, in which transparent electrodes are arranged on a transparent substrate such as a glass substrate, and a plurality of pixels (display elements) are matrixed in a display area (sensor area) 21. And a backlight (not shown) is arranged on the back surface. The backlight of this example uses, for example, an array of a plurality of light emitting diodes, and can turn on / off the backlight at a relatively high speed. On / off control of lighting with the backlight is performed in conjunction with display driving in the display display signal processing circuit 12. The liquid crystal display display is driven by the display display signal processing circuit 12 by applying a driving voltage signal to the pixel electrodes constituting the display.

  The I / O display panel 20 has a plurality of light receiving elements arranged separately from the display elements. That is, for example, light receiving elements are arranged in a matrix adjacent to each display pixel in the display area (sensor area) 21, and signal charges accumulated corresponding to the amount of light received by the light receiving elements are used as light receiving signals. It is configured to read by driving from the processing unit 13.

  The light reception signal (difference image signal described later) read and determined by the light reception signal processing unit 13 is sent to the input information analysis unit 14, and the contact state and the like are determined as an image. When this determination is made, a difference from the determination result of the previous frame is determined to detect a change in the contact state. In this case, the storage device 15 is used to store information on changes in the contact position from a predetermined frame period. In this example, determination is also made for simultaneous contact at a plurality of locations. The determination result is sent to the application program execution unit 11 as a predetermined message. The application program execution unit 11 performs predetermined processing according to the application being executed. An example of a specific processing state will be described later.

  Next, with reference to FIG. 2, an arrangement example of drivers of the I / O display panel 20 of this example will be described. As shown in FIG. 2, the I / O display panel 20 in which a transparent display area (sensor area) 21 is arranged at the center has a display horizontal driver 22 and a display vertical driver 23 on four end faces of the display area 21. A sensor horizontal driver 24 and a sensor vertical driver 25 are arranged. The display horizontal driver 22 and the display vertical driver 23 are supplied with display signals and control clocks as display data, and drive the display pixels arranged in a matrix in the display area 21. A reading clock is supplied to the sensor horizontal driver 24 and the sensor vertical driver 25, and a light reception signal read in synchronization with the clock is supplied to the light reception signal processing unit 13 through the light reception signal line. To do.

  FIG. 3 is a diagram showing one configuration of the pixels arranged in the display area 21. As a configuration for display provided in one pixel 31, here, a gate electrode 31h is disposed in the horizontal direction, a drain electrode 31i is disposed in the vertical direction, and a switching element 31a is provided at the intersection of both electrodes. The switching element 31a and the pixel electrode 31b are connected. The switching element 31a is controlled to be turned on / off by a signal obtained via the gate electrode 31h, and a display state at the pixel electrode 31b is set by a signal supplied via the drain electrode 31i.

  A light receiving sensor (light receiving element) 31c is arranged at a position adjacent to the pixel electrode 31b, and the power supply voltage VDD is supplied. The light receiving sensor (light receiving element) 31c is connected to a reset switch 31d and a capacitor 31e. After being reset by the reset switch 31d, the capacitor 31e accumulates charges corresponding to the amount of light received. A voltage proportional to the accumulated charge is supplied to the signal output electrode 31j via the buffer amplifier 31f and output to the outside at the timing when the readout switch 31g is turned on. The on / off of the reset switch 31d is controlled by a signal obtained from the reset electrode 31k, and the on / off of the readout switch 31g is controlled by a signal obtained from the readout control electrode 31k.

  FIG. 4 is a diagram showing a state in which image display (light emission) and light reception are performed in each frame period. In FIG. 4, the horizontal axis represents time, and the vertical axis represents the position of a scanning line (horizontal line) for displaying and receiving light. Here, rewriting of the display signal and reading of the received light signal are performed at the bottom of one screen. The scanning line is changed upward in order from the line, and finally the scanning of the uppermost line (first line) is performed. FIG. 4 shows the process of the nth frame, which is an arbitrary frame position, and the process of the (n + 1n) th frame, which is the next frame of the nth frame, and the same process is continuously performed.

  Here, one frame period is, for example, 1/60 seconds, and as shown in FIG. 4, the one frame period is divided into two equal parts, the first half and the second half, and the first half is a period during which the backlight is turned on. Yes, the second half is a period for turning off the backlight. The light reception signal is read out in each of the lighting period and the extinguishing period.

  Further, the backlight on period and the backlight off period are each divided into two equal parts. In the nth frame, the pixel electrode drive line G1 for display scans the lower half of the screen in the first half of the backlight on period and rewrites the display state of the line to an image of the frame period. In the second half of the backlight-on period, the scanning line is not changed and the rest period is set. In the first half of the backlight off period, the upper half of the screen is scanned to rewrite the display state of the line to the image of the frame period, and in the second half of the backlight off period, the scan line is changed. Without a rest period.

  As for the light receiving process, in the nth frame, the process RS1 for sequentially resetting the light receiving signals of all the lines is performed in the first half of the backlight on period, and the light receiving signals of all the lines are processed in the second half of the backlight on period. Processing RD1 is sequentially read, and the received light signals accumulated for a certain period by each sensor are read. Similarly, in the first half of the backlight off period, a process RS2 for sequentially resetting the light reception signals of all lines is performed, and in the second half of the backlight off period, a process RD2 for sequentially reading the light reception signals of all lines is performed, Read the received light signal accumulated for a certain period by each sensor.

  In this way, two readings are performed per frame, that is, reading the received light signal when the backlight is on and so-called self-emission, and reading the received light signal when the backlight is off and the light is turned off. The two readout signals of one frame are input to a frame memory (not shown) in the received light signal processing unit 13, and a difference is detected for each signal at each pixel position. The difference received light signal is sent to the input information analysis unit 14.

  FIG. 5 is a diagram showing an example of the shape of the display device of this example. Here, it is configured as a small and thin display device 10 that can be carried by the user, for example, and the user (operator) touches the display surface of the display area 21 of the I / O display panel 20 with fingers f1, f2, and the like. Thus, the operation can be performed.

  6 and 7 are examples of images detected from the light reception signal when the display surface of the display area 21 is touched with a finger. For example, as shown in FIG. 6, when two fingers touch different parts at the same time, the two contact portions 41 and 42 are detected. For example, as shown in FIG. 7, when four fingers touch different parts at the same time, the four contact portions 51, 52, 53, and 54 are detected. Depending on the detection sensitivity of the received light signal, for example, similar detection is possible even when a finger is brought close to the display surface.

  The detection of the contact position from the received light image as shown in FIGS. 6 and 7 is performed by the input information analysis unit 14. The input information analysis unit 14 detects a change in the contact position for each frame, and sends a message corresponding to the detected change state to the application program execution unit 11 side.

  The flowchart of FIG. 8 is a diagram showing an example of processing in the input information analysis unit 14 in each frame, and here shows processing for the Nth frame and processing for the next N + 1th frame. First, in N frames, an image based on the received light signal is acquired (step S11), the received light image is compared with the image of the immediately preceding frame period (step S12), and the difference between the received light images is calculated (step S13). Based on the difference, the movement of the contact position from the previous frame is determined, and the movement of the contact state is analyzed (step S14). In this case, when there are a plurality of contact positions, each of the plurality of contact positions is determined and analyzed. And it collates with the information of the change of the contact position from the predetermined frame period memorize | stored in the memory | storage device 15 (FIG. 1) (step S15), and the message regarding the detection of the contact position sent to the control part 11b is determined (step S16). ). If the received light image is acquired in step S11, the received light image is temporarily stored and used for processing the next frame (step S17).

  When the received light image of the next N + 1 frame is acquired (step S21), the received light image is compared with the image of the previous frame period temporarily stored in step S17 (step S22), and the difference between the two received light images is calculated ( In step S23), the movement of one or more contact positions from the previous frame is determined based on the difference, and the movement of the contact state is analyzed (step S24). And it collates with the information of the change of the contact position from before the predetermined frame period memorize | stored in the memory | storage device 15 (step S25), and the message regarding the detection of the contact position sent to the control part 11b of the application program execution part 11 is determined (step S25). Step S26).

  In this manner, one or a plurality of contact positions are determined for each frame period, and an application program that performs display on the application program execution unit 11 according to the determined change state of each position is executed. It becomes possible.

  Next, a processing example for changing the execution state of the application program based on detection of contact (or proximity) on the display screen will be described with reference to FIGS. 9 to 14. In the example of FIG. 9, the display image 21 a having a relatively small size is displayed in the display area 21, and the left and right edges of the display image 21 a are touched with the two fingers f1 and f2 of the user. This is an example in which a program of an application that performs display processing for changing the display size of the display image 21a by changing the touched position is executed.

  Here, it is assumed that the position touched by the fingers f1 and f2 is moved as indicated by arrows a and b from the state shown in FIG. FIG. 10 is an example of a received light image before the finger is moved, and two contact positions 61 and 62 are detected. FIG. 11 is an example of a received light image after moving a finger from the state of FIG. 10 (for example, an image after several tens of frames from FIG. 10), and the distance between the contact positions 61 ′ and 62 ′ is separated.

  By moving the fingers f1 and f2 in this manner, for example, as shown in FIG. 12, the movement of the fingers f1 and f2 indicated by the arrows a and b is determined, and the display size is changed in accordance with the movement. The display image 21b is expanded to be executed, and an application for changing the size of the image is executed. The process of changing from the display image 21a to the display image 21b is executed as a process of generating display data in the application program execution unit 11. Since such a process of changing the display image can be performed by changing the contact positions at a plurality of locations, an advanced application can be executed based on detection of the contact positions at the plurality of locations.

  The example of FIG. 13 is an operation example in which only the contact position of one finger f2 is moved to draw a circle as indicated by an arrow c in a state where the fingers f1 and f2 are in contact. By performing such an operation, for example, the display image may be rotated and displayed along a locus indicated by an arrow c. The rotation center of the image may be set at a position touched with the finger f1.

  In the example of FIG. 14, the display surface of the display area 21 is touched with three fingers f1, f2, and f3. In this state, by moving at least one of the three fingers f1, f2, and f3, it is possible to perform processing that changes the display image. For example, when three fingers f1, f2, and f3 are moved almost simultaneously toward a certain point and the center coordinates of the movement are obtained, a process of selecting an image displayed at the center coordinate position is performed. May be.

  Up to this point, the example of changing the display state (display shape, etc.) of the display image has been described. However, by touching (or approaching) a plurality of locations simultaneously with a finger, pen, etc., at least one position change among them is possible. It can be applied to those that perform various application processes. That is, what has been described above is a display device that inputs a user's intention by directly touching a screen with two or more places such as a finger or a pen and moving the position (arbitrary line, circle, etc.). As a process performed by the input, a process of changing the display image itself is possible. Further, the application itself to be displayed may be changed depending on the user operation state (that is, the contact state). Further, the operation of the application to be displayed may be changed. Further, the user's intention may be analyzed, and input reception may be performed according to the analyzed intention.

  In the above-described embodiment, a liquid crystal display has been described as an example of the display panel. However, the present invention can be applied to a display having any other configuration as long as it can incorporate a light receiving element.

  For example, it can be applied to an EL display. FIG. 15 is a diagram illustrating a configuration example of each pixel when a display that performs input (light reception) and output (display) is applied to an EL display. The light emitting element 41 of the organic EL display is shown as a light emitting diode, and a parasitic capacitance 42 is generated in the light emitting diode 41. Display data is supplied from the display data signal line 43 to the light emitting element 41 for image display via the switch SW1. Accordingly, the display period (light emission period) is set in a period during which the switch SW1 is turned on.

  Then, during the period in which the light emission of the light emitting element 41 is stopped, electric charges are accumulated in the parasitic capacitance 42 generated in the light emitting element 41 according to the amount of light incident on the surface of the display panel. The accumulated charge is read out to the reception data signal line 45 when the switch SW2 is turned on. Note that at the start of the light receiving period, it is necessary to turn on the reset switch SW3 for a moment to release the charge accumulated in the parasitic capacitor 42 during light emission. The switch SW <b> 2 is turned on by a signal obtained from the read line selection line 44.

  By configuring such an organic EL display type I / O display panel, it is possible to perform processing for detecting contact or proximity to the panel while displaying an image or the like. In the case of this configuration, for example, both processes can be performed by dividing a field period into two and setting a display period and a light reception period. By using the I / O display panel having such a configuration as the I / O display panel 20 shown in FIG. 1, processing similar to that of the above-described embodiment is possible.

  Further, in the description of the embodiments so far, it is assumed that an application to be performed based on contact detection at a plurality of locations is incorporated in advance in the display device having the configuration illustrated in FIG. An application program that changes the processing state in accordance with the change in the contact detection state of the location is created, and the program is installed in the display device having the configuration shown in FIG. 1 so that the above-described processing is performed. Good. The program may be acquired, for example, by communication in the communication unit 11a illustrated in FIG. 1, or may be distributed using some storage medium.

It is a block diagram which shows the structural example of the display apparatus by one embodiment of this invention. It is a block diagram which shows the example of the display panel by one embodiment of this invention. It is a connection diagram showing a pixel configuration example according to an embodiment of the present invention. It is a timing diagram which shows the example of a timing of the display and light reception by one embodiment of this invention. It is a perspective view which shows the example of an apparatus shape by one embodiment of this invention. It is explanatory drawing which shows the example of the light reception screen by one embodiment of this invention. It is explanatory drawing which shows the example of the light reception screen by one embodiment of this invention. It is a flowchart which shows the process example in each frame by one embodiment of this invention. It is explanatory drawing which shows the example of operation by one embodiment of this invention. It is explanatory drawing which shows the example of the light reception screen by one embodiment of this invention. It is explanatory drawing which shows the example of the light reception screen by one embodiment of this invention. It is explanatory drawing which shows the example of operation by one embodiment of this invention. It is explanatory drawing which shows the example of operation by one embodiment of this invention. It is explanatory drawing which shows the example of operation by one embodiment of this invention. It is a connection diagram which shows the example of a pixel structure by other embodiment of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Display apparatus, 11 ... Application program execution part, 11a ... Communication part, 11b ... Control part, 12 ... Display display signal processing circuit, 13 ... Light reception signal processing part, 14 ... Input information analysis part, 15 ... Memory | storage device, 20 ... I / O display panel, 21 ... display area (sensor area), 22 ... display horizontal driver, 23 ... display vertical driver, 24 ... sensor horizontal driver, 25 ... sensor vertical driver, 31, 32, 33 ... Pixel, 31a ... Switching element, 31b ... Pixel electrode, 31c ... Sensor element, 31d ... Reset switch, 31e ... Capacitor, 31f ... Buffer amplifier, 31g ... Reading switch, 41 ... Light emitting element, 42 ... Parasitic capacitance, 43 ... Display Data signal line 44 ... Read line selection line 45 ... Reception data signal line SW1, SW2 SW3 ... switch

Claims (6)

In a display device that performs image display and light reception simultaneously or alternately,
A plurality of display elements arranged in a matrix;
A plurality of light receiving elements arranged adjacent to the plurality of display elements and receiving light incident on the display surface;
A position detection unit that determines a position in contact with or close to the display surface based on a light reception signal received by the light receiving element;
A control unit that controls execution of a predetermined process according to a change in the contact position or proximity position of the plurality of locations when the position detection unit determines contact or proximity of the plurality of locations. Display device. The display device according to claim 1,
The predetermined process by the control of the control unit is a process of changing a display image to be displayed on the display element. The display device according to claim 1,
The display apparatus according to claim 1, wherein the predetermined process under the control of the control unit is a process of changing an application for generating a display image to be displayed on the display element. The display device according to claim 1,
The display apparatus according to claim 1, wherein the predetermined process under the control of the control unit is a process of changing an operation of an application that generates a display image to be displayed on the display element. The display device according to claim 1,
The display device according to claim 1, wherein the predetermined process by the control of the control unit is a process of analyzing an operation intention of an operator from a change in the contact position or the proximity position of the plurality of places. In a display method in which image display and light reception are performed simultaneously or alternately,
Display with a plurality of display elements arranged in a matrix,
Based on a light reception signal output by a light receiving element arranged in proximity to each of the plurality of display elements, a position in contact with or close to the display surface is determined,
A display method comprising: performing a predetermined process according to a change in the contact position or proximity position of the plurality of locations when the contact or proximity of the plurality of locations is determined in the determination.

Images