JP2006243927A - Display device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To cause a display device that allows the input of information using an optical sensor to properly control the sensitivity of the optical sensor according to external light. <P>SOLUTION: A calibration circuit 93 varies the requirements for driving the optical sensor according to output values of the optical sensor that vary depending on external light. In this way, the sensitivity of the optical sensor is appropriately controlled. The requirements for driving the optical sensor include at least either precharge time or exposure time. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a display device capable of inputting information from a display screen by light.

  2. Description of the Related Art In recent years, liquid crystal display devices widely used as display devices in various devices such as mobile phones and notebook computers are wired so that a plurality of scanning lines and a plurality of signal lines intersect with each other. A display portion including a pixel having a capacitor and an auxiliary capacitor, a driving circuit for driving a scanning line, and a driving circuit for driving a signal line are provided. The display unit is formed on a glass array substrate. With the recent development of integrated circuit technology and practical application of process technology, part of the drive circuit can be formed on the array substrate, and the entire liquid crystal display device is reduced in weight and size.

  By the way, a display device has been developed in which an optical sensor is arranged in a pixel and information can be input from a display screen by light (see, for example, Patent Document 1).

  In a display device having this type of light input function, for example, a photodiode is used as an optical sensor, and a capacitor is connected to the photodiode to change the amount of light received by the photodiode with respect to light incident from the display screen. Accordingly, the amount of charge of the capacitor is changed, and the voltage at both ends of the capacitor is detected to generate captured image data, thereby capturing the image.

As an application of such a display device, touch panel function to input information by detecting shadows made by objects such as fingers projected on the display screen, and light emitted from light emitting objects such as light pens are detected. Thus, there has been proposed one having a digitizer function for inputting information.
JP 2004-318819 A

  However, in a display device having a conventional light input function, when the surrounding environment is dark, a shadow created by an object approaching the display screen cannot be accurately identified, so that information input accuracy decreases. In addition, the accuracy of information input decreases even if the surrounding environment is too bright.

  This invention is made | formed in view of the above, The place made into the subject exists in controlling appropriately the sensitivity of an optical sensor according to external light.

  The display device according to the present invention displays an image on a display screen, and displays an image of an object close to the display screen using a photosensor, and drives the photosensor based on a statistic of an output value of the photosensor. Sensor driving means for changing the conditions.

  In the present invention, it is possible to appropriately control the sensitivity of the optical sensor by changing the driving condition of the optical sensor based on the output value of the optical sensor that varies according to the external light.

  In the display device, the driving condition is preferably at least one of an exposure time and a precharge time of the photosensor.

  In the display device, the display means displays an area for bringing an object close to the display screen, and the sensor driving means outputs an optical sensor arranged in an area other than the area on the display screen. The drive condition is changed based on the statistical value.

  In the present invention, since the driving condition is changed based on the output value of the optical sensor arranged in a region other than the region where the object is brought close to, it becomes difficult to be affected by the shadow when the object comes close. Thus, it is possible to accurately grasp fluctuations in the output of the optical sensor due to changes in the illuminance of external light.

  In the display device, the sensor driving means controls the driving condition so that an upper value including a center in a histogram of gradation values obtained by gradationizing the output value falls within a predetermined range. Is output.

  In the present invention, by controlling the driving conditions of the optical sensor so that the upper value including the center in the gradation histogram is within a predetermined range, the shadow generated when the object comes close has a low gradation. However, since it is difficult to be affected by such a low gradation value change, it is possible to accurately grasp the fluctuation of the output of the optical sensor due to the illuminance change of the external light.

  In the display device, the sensor driving unit may change the upper gradation value in accordance with a display screen and an object size.

  In the present invention, the upper gradation value can be changed according to the size of the display screen and the object, so that control using an appropriate gradation value according to the size of the display screen and the object can be performed. It becomes possible.

  In the display device, the sensor driving unit determines the control signal using a table indicating a relationship between an exposure time of a photosensor, a precharge voltage, and the predetermined range.

  In the present invention, by using a table that discretely shows the relationship between the exposure time, the precharge voltage, and the predetermined range of the optical sensor, the exposure time and the precharge voltage can be changed continuously. It takes less time to control.

  In the display device, the table is characterized in that the exposure time is maximized within an allowable range and the precharge voltage is changed.

  In the present invention, the S / N ratio is improved by changing the precharge voltage and maximizing the exposure time within an allowable range.

  In the display device, the sensor driving unit determines the control signal using the table by a linear search method.

  In the present invention, by using the linear search method, the exposure time and the precharge voltage can be obtained quickly and accurately so that the gradation value falls within a predetermined range.

  In the display device, the sensor driving unit determines the control signal using the table by a binary search method.

  In the present invention, by using the binary search method, it is possible to obtain the exposure time and the precharge voltage faster than the linear search method.

  In the display device, the sensor driving unit uses a gradation value after pixel thinning processing is performed for each region when the display screen is divided into a plurality of regions within a certain range.

  In the present invention, it is possible to reduce the processing load of changing the driving conditions of the optical sensor by using the gradation value after the pixel thinning process is performed for each predetermined region.

  According to the display device of the present invention, it is possible to more appropriately control the sensitivity of the optical sensor in accordance with the brightness of external light.

[First Embodiment]
FIG. 1 is a plan view illustrating a configuration of a display device according to an embodiment. The display device shown in FIG. 1 includes a display unit 2 formed on a glass array substrate 1, a flexible substrate 3 connected to the array substrate 1, a sensor IC (Integrated Circuit) 4 formed on the flexible substrate 3, and a display. IC 5, sensor IC 4 interface (I / F) 6, and display IC interface 7.

  The display unit 2 is wired so that a plurality of signal lines and a plurality of scanning lines intersect, and includes a pixel at each intersection. The display unit 2 includes a display function for displaying an image based on a video signal transmitted from the host-side CPU via the display interface I / F 7 and the display IC 5, and an external object close to the display screen. It has a light input function for taking images. The sensor IC 4 processes the captured image and transmits the processing result to the host-side CPU via the sensor interface I / F 6. The display IC 5 controls display processing.

  FIG. 2 is a cross-sectional view showing the configuration of the display unit 2. In the array substrate 1, a photosensor 8 and the like are formed in a pixel, and an insulating layer 9 is formed so as to cover it. A liquid crystal layer 11 is formed in a gap between the array substrate 1 and a glass counter substrate 12 disposed to face the array substrate 1. A backlight 13 is disposed outside the counter substrate 12. As shown in the figure, outside light that is not blocked by an object 20 such as a finger and light that is emitted from the backlight 13 and reflected by the object 20 enter the optical sensor 8.

  FIG. 3 is a circuit diagram illustrating a configuration of a pixel. In the display unit 2, red (R), blue (B), and green (G) pixels are regularly arranged, and each pixel serves as a display system 31 as a switching element 33, a liquid crystal capacitor LC, and an auxiliary capacitor CS. Is provided. In the figure, Gate (m) is a scanning line, Sig (n) is a signal line, and CS (m) is an auxiliary capacitance line. The switch element 33 is a MOS type, and has a gate connected to the scanning line, a source connected to the signal line, and a drain connected to the auxiliary capacitor CS and the liquid crystal capacitor LC. The other terminal of the auxiliary capacitor CS is connected to the auxiliary capacitor line.

  The video signal transmitted through the signal line from the CPU on the host side is given to the auxiliary capacitor CS and the liquid crystal capacitor LC via the switch element 33 when the switch element 33 is turned on by the scanning signal transmitted to the scanning line. And used for display.

  The display unit 2 includes an optical sensor 8, a sensor capacitor 37, an output control switch 34, a source follower amplifier 35, and a precharge control switch 38 for each of R, G, and B pixels as the optical sensor system 32. Here, a PIN photodiode is used as an example of the optical sensor 8.

  The optical sensor 8 and the sensor capacitor 37 are connected in parallel. The optical sensor 8 and the sensor capacitor 37 are connected to the red signal line Sig (n) via the source follower amplifier 35 and the output control switch 34, and are connected to the blue signal line Sig (n + 2) via the precharge control switch 38. Connected to.

  On / off of the output control switch 34 is controlled by a signal on the control line OPT (m), and on / off of the precharge control switch 38 is controlled by a signal on the control line CRT (m).

  Next, the operation of the optical sensor 8 will be described. For example, a voltage of 4V is precharged to the sensor capacitor 37 from the blue signal line through the precharge control switch 38. When a leak current occurs in the optical sensor 8 according to the amount of light incident on the optical sensor 8 during a predetermined exposure time, the potential of the sensor capacitor 37 changes. The sensor capacitance 37 is maintained at about 4V if the leakage current is small, and approaches 0V if the leakage current is large. On the other hand, after the red signal line is precharged to 5 V, the output control switch 34 is turned on to make the source follower amplifier 35 conductive to the red signal line. Since the sensor capacitor 37 is connected to the gate of the source follower amplifier 35, the source follower amplifier 35 is turned on if the residual voltage of the sensor capacitor 37 remains at 4V, for example, and the potential of the red signal line goes from 5V to 0V. Change. If the remaining voltage of the sensor capacitor 37 is 0V, the source follower amplifier 35 is turned off, and the potential of the red signal line remains at 5V and hardly changes.

  As shown in the circuit diagram of FIG. 4, the comparator 41 compares the potential of the red signal line with the reference voltage of the reference power supply 40, and outputs a high level signal when the potential of the signal line is greater than the reference voltage. When the potential of the signal line is lower than the reference voltage, a low level signal is output. The reference voltage is fixed.

  Thereby, the comparator 41 outputs a high level signal when the light sensor 8 detects light brighter than a predetermined value, and outputs a low level signal when light darker than the predetermined value is detected. Will do. The output of the comparator 41 is transmitted to the sensor IC 4.

  FIG. 5 is a timing chart showing the operation of the display device. In the drawing, for each scanning line, a timing chart of the display system is shown at the top and an imaging system is shown at the bottom. As shown in the figure, after the writing of the video signal to the pixel is completed in the display system, after a certain blank period (the sensor capacitor 37 is precharged during this period), the object on the display screen is displayed. Images are taken for a certain exposure time. This exposure time can be changed.

  In the display device of this embodiment, as shown in FIG. 6, the first to twelfth switches are displayed on the display screen as regions for bringing an object close to each other, and it is determined which switch the finger 20 touches. Assume that.

  FIG. 7 is a circuit block diagram showing the configuration of the sensor IC 4. The sensor IC 4 shown in FIG. 1 includes a level shifter 91, a gradation circuit 92, a calibration circuit 93, a DAC (Digital Analog Converter) 94, a counting circuit 95, an inter-frame difference processing circuit 96, an edge detection circuit 97, and a contact probability calculation circuit. 98 and RAM (Random Access Memory) 99. In the present embodiment, the calibration circuit 93 corresponds to the sensor driving means in the claims.

  The level shifter 91 adjusts the signal voltage for exchanging signals with the display unit 2.

  The gradation circuit 92 converts the binary image transmitted from the comparator 41 described with reference to FIG. 4 into a multi-gradation image. As a method of this conversion, for example, as shown in FIG. 8, by taking the sum of binary data consisting of 0 or 1 in a surrounding 12 pixel × 12 pixel square area for each pixel, 0 to 144 is obtained. Multitone values between are obtained. Alternatively, a multi-tone value between 0 and 255 may be obtained in a square area of 16 × 16 pixels. The suitability value should be determined in consideration of the arrangement efficiency of the memory area provided inside the sensor IC. Alternatively, the display unit 2 may output the captured image data as an analog signal, and the gradation circuit 92 may convert the data into a multi-gradation digital signal by an A / D converter. Note that even if the gradation image is thinned out in the same manner as in the first embodiment, the following processing has little influence. The suitability value should be determined in consideration of the arrangement efficiency of the memory area provided inside the sensor IC.

  The calibration circuit 93 counts the number of white pixels in the captured image and outputs a control signal so that 30 to 70% of all pixels are white. Upon receiving this control signal, the DAC 94 and the level shifter 91 adjust at least one of the exposure time and the precharge time. Specifically, when the number of output white pixels increases to 70%, the number of white pixels is pulled back to about 50% by shortening the exposure time or increasing the precharge voltage. On the other hand, when the number of output white pixels is reduced to 30%, the number of white pixels is pulled back to about 50% by increasing the exposure time or lowering the precharge voltage. Thus, by setting the number of white pixels to 30 to 70% of all pixels, the response of the photosensor can be made sensitive as shown in the graph of FIG.

  However, it is preferable that the operating point is changed based on a change in external light, but accurate reading cannot be performed if the operating point frequently changes due to the finger / hand approaching. In other words, when it is desired to detect the movement of the finger in the difference image, if a difference in the entire read image due to a change in the driving condition is detected, it becomes a significant noise.

  Therefore, instead of outputting the average value based on the output value of the photosensor for the entire screen, the drive condition is changed based on the output value of the photosensor arranged in an area other than the area where the switch is displayed on the display screen. Like that. This area is, for example, the upper part of the screen where no shadow is applied even when a finger approaches.

  FIG. 10 is a diagram illustrating a screen when an average value is output based on the output value of the optical sensor of the entire screen, and FIG. 11 is a diagram illustrating a gradation histogram at that time. A broken line is a histogram before a finger approaches, and a solid line is a histogram when a finger approaches.

  On the other hand, FIG. 12 is a diagram showing a screen when an average value is output based on the output value of the photosensor in an area other than the switch in the upper part of the screen, and FIG. 13 shows a gradation histogram at that time. FIG. Here, the output value of the optical sensor arranged in the area indicated by the broken line in FIG. 12 is used. The relationship between the solid line and the broken line in FIG. 13 is the same as that in FIG.

  As can be seen from FIGS. 11 and 13, when the finger approaches, the region having a low gradation value corresponding to the shadow of the finger increases, so the histogram shifts to the lower gradation value side. Become.

  Comparing the two, when the optical sensor of the entire screen is used, the average value 1 of the histogram before the finger approaches is an average value 2 when the finger approaches, and the gradation average value greatly decreases. To do. On the other hand, when only the optical sensor at the top of the screen is used, the sampling region of the gradation histogram is limited to the dotted line portion in FIG. In this way, since a shadow is hardly projected on the sampling area before and after the finger approaches, the shift of the gradation average value in this sampling area is reduced. Therefore, it is possible to avoid a sensitive reaction to the approach of a finger or hand, and it is possible to accurately read only changes in external light.

  FIG. 14 is a diagram illustrating a temporal change in the average gradation value before and after the finger approaches. The dotted line in the figure shows the change of the average value using the optical sensor of the entire screen, and the average value has changed greatly due to the finger approach. On the other hand, the solid line shows the time change of the average value in the sampling area at the top of the screen, and the average value does not change so much before and after the finger approaches. On the other hand, when the illuminance of outside light changes greatly, the gradation average of the dotted sampling area shifts in conjunction with the outside light, so that the calibration functions effectively.

  Considering that this calibration is performed inside the sensor IC, when calculating the average value for the entire screen, you must wait for the output value from the optical sensor for the entire screen. If the value is calculated, there is an advantage that it is not necessary to wait for the output value from the optical sensor of the entire screen.

  In the present embodiment, whether or not to perform calibration is determined by observing the “average value”, but is not limited to the “average value”. In order to determine whether or not the driving conditions of the optical sensor are appropriate, any statistic based on the output value of the image may be used. For example, the median value (median value) of the gradation histogram may be used. However, the circuit scale for obtaining the “average value” is smaller, and there is an advantage that the cost of the sensor IC can be reduced.

  It is desirable that the sampling area be large as long as no finger shadow is projected. Therefore, as shown in FIGS. 15 and 16, it is effective to make the sampling area asymmetric with respect to the screen center when the hand input by the user is the right hand and the left hand. This setting is set by the user when purchasing the terminal. For example, a right-handed user will often input with the right hand as shown in FIG. 15, so the right side of the screen is slightly more likely to be shaded and the left side of the screen is likely to be shaded. Since it is considered to be low, it is preferable to define a sampling area for calibration on the left side of the upper part of the screen as shown by the dotted line in FIG. On the other hand, since a left-handed user often inputs with his left hand as shown in FIG. 16, it is preferable to define a sampling area on the right side of the upper part of the screen as shown by a dotted line in FIG.

  The counting circuit 95 counts the number of white pixels for each switch displayed on the display screen, and holds these count values for each switch. Also, the difference value between the count value in the current nth frame and the count value in the past n−1th frame is calculated and held, and the maximum difference value for each switch is greater than or equal to a predetermined threshold value. In this case, a signal indicating that the finger has touched the switch is output as a probability calculation candidate. As shown in FIG. 17, when a finger touches a specific switch (fifth switch in FIG. 17), the gradation change between images to be subjected to difference occurs in conjunction with other switches. However, since the gradation change is most severe with the switch that is actually in contact with the finger, the accuracy of the determination can be increased by using the maximum difference value as a determination target. The output signal of the counting circuit 95 is sent to the contact probability calculation circuit 98.

  The inter-frame difference processing circuit 96 obtains a difference image obtained by taking the difference between the multi-tone image in the current frame and the past multi-tone image stored in the RAM 99, and binarizes the difference image to obtain the object. The area to be shown is extracted, the center of gravity of this area is calculated, and a signal indicating that the finger has touched the switch located at this center of gravity is output as a probability calculation candidate.

  When obtaining the difference image, for example, as shown in FIG. 18, the tone value of the multi-tone image one frame before is subtracted for each pixel from the tone value of the multi-tone image at an arbitrary time t1. In the figure, the gradation value at the coordinates (x, y) at time t is F (x, y, t). In addition, when the polarity of the driving voltage for driving the pixels of the display unit 2 changes every frame, in order to eliminate the influence of this polarity, a difference from the multi-tone image two frames before is taken. Also good. In this way, the data output from the sensor is reduced to once every two frames, so that there is an advantage that power consumption can be reduced and the battery can last longer. Alternatively, after taking the average value of the even-numbered frame, the odd-numbered frame, and the multi-tone image, the difference may be taken between the average values.

  The edge detection circuit 97 calculates the strength of the edge (the magnitude of the spatial change of the gradation) for the multi-gradation image of each frame, calculates the centroid of the edge whose gradation value is equal to or greater than a predetermined threshold, As a probability calculation candidate, a signal indicating that the finger has touched the switch located at the center of gravity is output. It is also effective to hold a multi-tone image at the time of completion of calibration in a memory and perform edge detection on a new multi-tone image subtracted from the latest multi-tone image. This is because it is possible to reduce imaging unevenness caused by variations in sensor characteristics and to cut out only the edge of a finger.

  When detecting an edge, for example, as shown in FIG. 19, for an arbitrary pixel, the gradation values of four pixels adjacent in the vertical and horizontal directions of the pixel are subtracted from the value obtained by multiplying the gradation value of the pixel by four. Then, a Laplacian filter is used as the gradation value of the pixel. In addition, a well-known filter such as a Sobel filter or a Roberts filter may be used. In addition, the pixel to be calculated is not limited to the pixel adjacent to the upper / lower / left / right sides of the arbitrary pixel, but may be a pixel adjacent to the upper / lower / left / right side. Alternatively, pixels separated by several pixels may be used. For example, when entering a dormitory with adult fingers (about 1 cm wide), it should be 5 mm or more. It is not necessary to make it larger than 1 cm. It is preferable that the five pixels used for the filtering process protrude from the shadow of the finger (such as a pointing member). When the shadow width of the indicating member is W, it is preferable that the pixel used for the filtering process be separated from the target pixel by W / 2 or more. It is not necessary to separate W or more.

  The edge detection circuit 97 may use the multi-tone image output from the gradation circuit 92 when detecting the edge, or the multi-tone image after the difference is obtained by the inter-frame difference processing circuit 96. It may be used. These processes may be performed using the gradation image after thinning.

  The contact probability calculation circuit 98 calculates a contact probability for each switch based on signals output from the counting circuit 95, the inter-frame difference processing circuit 96, and the edge detection circuit 97, respectively. For example, it is assumed that 10 points are given to each of the switches 95, 96, and 97 that the finger is in contact with, and 0 is given to the other switches. As an example, when a signal indicating that there is a high possibility that a finger has touched the fifth switch is received from each of the circuits 95, 96, 97, the fifth switch has a total of 30 points and the other switches have 0 points. Therefore, the probability is calculated as 30/30 × 100 (%) = 100% for the fifth switch and 0/30 × 100 (%) = 0% for the other switches. As another example, when a signal that the fifth switch is touched is received from the circuits 95 and 96 and a signal that the sixth switch is touched is received from the circuit 97, the fifth switch has a total of 20 points. 10 points and the other switches are 0 points, so the probability is calculated that the fifth switch is 67%, the sixth switch is 33%, and the other switches are 0%.

  If the host CPU that receives the output of the contact probability calculation circuit 98 is set to determine that the finger has touched the display screen only when the contact probability is 100%, the accuracy of the contact determination is extremely high. A system can be realized. On the other hand, if the host is set to determine that the finger has touched the display screen even when the contact probability is 67%, a system with good responsiveness can be realized. The former can be applied to applications that require reliability such as bank ATMs, and the latter can be applied to applications that do not require much reliability such as games.

  Therefore, according to the present embodiment, it is possible to appropriately control the sensitivity of the optical sensor by changing the driving condition of the optical sensor based on the output value of the optical sensor that varies according to the external light.

  According to the present embodiment, the influence of the shadow when the object comes close is changed by changing the driving condition of the light sensor based on the output value of the light sensor arranged in a region other than the region where the object is brought close to. Since it becomes difficult to receive, the fluctuation | variation of the output of the optical sensor by the illumination intensity change of external light can be grasped | ascertained correctly.

[Second Embodiment]
In this embodiment, only the function of the calibration circuit 93 is different from that of the first embodiment, and the other basic configuration is the same as that of the first embodiment. Therefore, only the calibration circuit 93 will be described here. It is assumed that the same parts as those in the first embodiment are not described repeatedly.

  In this embodiment, calibration is performed in real time. This is because it may not be sufficient to perform calibration only at the time of shipment or power-on. This is because, for example, the illuminance on the panel surface may change greatly due to a change in the illuminance of external light during use or a change in the angle of the hand holding the display even if the illuminance of the external light is constant.

  As shown in FIG. 20, the calibration circuit 93 according to the present embodiment sequentially receives 15 × 20 × 8-bit gradation data after the area gradation processing is performed on the captured image every 16 × 16 pixels. The By performing the thinning process in this way, the real-time processing load in the calibration circuit 93 is reduced.

  On the other hand, it is not preferable that the operating point change due to calibration occurs when the finger approaches / contacts. Therefore, the calibration circuit 93 creates a gradation histogram and outputs a control signal so that the upper value including the center in the gradation histogram falls within a predetermined range. In response to this control signal, the DAC 94 and the level shifter 91 adjust at least one of the exposure time and the precharge time.

  Next, it will be described that the calibration focusing on the upper value of the gradation histogram is more stable than the calibration focusing on the average value.

  FIG. 21 is a diagram showing a gradation histogram before a finger approaches, and FIG. 22 is a diagram showing a gradation histogram when a finger approaches. In each figure, the average value and the median value (median value) are shown. In FIG. 22, the broken line is the same histogram as that in FIG. 21, and the dashed arrow indicates the average value.

  As shown in these figures, when it is assumed that the switch near the center of the screen is pressed under external light, the amount of light incident on the optical sensor gradually decreases as the finger and palm approach the screen. The average value changes in a decreasing direction. If the operating conditions of the optical sensor are changed in response to this sensitively, the overall tone change accompanying the change of the operating point other than the finger movement component is added to the difference image as noise during the difference processing. End up.

  On the other hand, the higher gradation values from the center of the histogram have less influence. This is because the shadow of the finger has a low gradation value, so that the influence on the lower part of the gradation histogram is large, but the influence on the upper part from the center of the gradation histogram is small. As shown in FIG. 22, the average value shifts to a lower gradation, but the median value has a small shift amount, so it can be said that the stability is excellent.

  Of course, instead of the median value, for example, the same processing is performed using the gradation value at the upper third position of the gradation histogram or the gradation value at the upper quarter position of the gradation histogram. Also good. By using the upper gradation values, unnecessary calibration is not applied and the operational stability is improved even when the shadow of the finger approaching the entire screen due to the light source or how the hand is covered. it can. Only when the external light changes significantly, the upper gradation value of the gradation histogram changes. The upper gradation value used depends on how much shadow is applied to the screen when the finger approaches. For example, if the screen size is sufficiently larger than an object such as a finger or hand, a median value may be used. If the screen size is about the same size as that of a finger or hand, the gradation value is about 1/5 from the top. Better. As described above, it is desirable that the calibration circuit 93 can change the upper gradation value according to the display screen and the size of the object.

  In addition, if the sensor driving condition parameters such as the precharge voltage and the exposure time are continuously changed, the control takes time. Therefore, at least one of the exposure time or the precharge voltage of the photosensor and a predetermined gradation value are set. It is desirable to prepare a table that discretely shows the relationship with the range. At this time, the parameters in the table should have an overlapping part between adjacent ones. Otherwise, there will be a case where an infinite loop occurs when calibration is applied at the boundary. The table assumes that the exposure time is maximized within an allowable range and the precharge voltage is changed. As a table that satisfies these conditions, how to change the precharge voltage and the exposure time is determined as shown in the table of FIG.

  This table sets the relationship between the precharge voltage Vprc, the exposure time, and the minimum value and maximum value indicating the median value range. FIG. 24 is a graph of this table.

Both the precharge voltage and the exposure time are not changed little by little at regular intervals, but the intervals are set so that the time required for shifting to the appropriate condition is shortened. The reason why the exposure time is maximized within the allowable range and the precharge voltage is changed preferentially is that, as shown in FIG. 25, the longer the exposure time, the better the S / N ratio.

  Next, processing for controlling the exposure time and the precharge voltage in real time using such a table will be described.

  FIG. 26 is a flowchart showing a flow of processing for setting the exposure time and the precharge voltage by the linear search method. Here, as shown in the table of FIG. 23, a number N that is incremented by one in order from the top is assigned for each parameter relationship such as precharge voltage, exposure time, minimum value of median value, and maximum value. Further, as the number N increases, the predetermined range determined by the minimum value and the maximum value for the median value changes toward a higher gradation value, and the precharge voltage and the exposure time also change accordingly. Shall be set to

  The linear search process of FIG. 6 includes initial value setting processing (steps S1 and S2), a normal imaging loop (steps S3 to S5) when the measured median value is within a predetermined range, and the measured median value. Has a linear search loop (steps S6 to S9) in the case where is not within the predetermined range.

  In step S1, ½ of the maximum value Nmax of N is set to the initial value N.

  In step S2, the precharge voltage Vprc and exposure time corresponding to the Nth column are read from the table, these values are set by the calibration circuit 93, and a control signal based on these values is output.

  In step S <b> 3, the display unit 2 captures an image using the optical sensor 8, the gradation circuit 92 gradations the output value of the captured image, and the calibration circuit 93 measures the median value of the gradation histogram.

  In step S4, the calibration circuit 93 reads the median maximum value Dn corresponding to the Nth column of the table and compares it with the measured median value. If the measured median value is smaller than the maximum value, the process proceeds to step S5, and if greater, the process proceeds to step S6.

  In step S5, the calibration circuit 93 reads the median minimum value Cn corresponding to the Nth column of the table and compares it with the measured median value. If the measured median value is larger than the minimum value, the process returns to step S3, and if it is smaller, the process proceeds to step S8.

  In steps S6 and S7, N is incremented by one and the process returns to step S2 so that the measured median value falls within the range smaller than the maximum value. If it cannot be moved up, the process returns to step S3 without changing N.

  In steps S8 and S9, N is decremented by one and the process returns to step S2 so that the measured median value falls within the range larger than the minimum value. If it cannot be lowered, the process returns to step S3 without changing N.

  By such a linear search process, it is possible to set an appropriate precharge voltage and exposure time in real time. In this method, when a dark state suddenly changes to a bright state, or conversely, when a bright state suddenly changes to a dark state, it takes a maximum of N frames until an appropriate setting is made. Become.

  Next, a binary search method will be described as another method. FIG. 27 is a flowchart showing a flow of processing for setting the exposure time and the precharge voltage by the binary search method. Again, the table used is the one described above.

  The binary search process of FIG. 6 includes an initial value setting process (steps S21 to S22), a normal imaging loop when the measured median value is within a predetermined range (steps S24 to S26), and a clogging determination process. (Steps S27 to S28, Steps S32 to S33) and a binary search loop (Steps S29 to S31, Steps S34 to S36) when the measured median value is not within the predetermined range.

  In step S21, 1/2 of the maximum value Nmax of N is set to the initial value N.

  In step S22, the precharge voltage Vprc and exposure time corresponding to the column N are read from the table, these values are set by the calibration circuit 93, and a control signal based on these values is output.

  In step S <b> 24, the display unit 2 captures an image using the optical sensor 8, the gradation circuit 92 gradations the output value of the captured image, and the calibration circuit 93 measures the median value of the gradation histogram.

  In step S25, the calibration circuit 93 reads the median maximum value Dn corresponding to the Nth column of the table and compares it with the measured median value. If the measured median value is smaller than the maximum value, the process proceeds to step S26, and if larger, the process proceeds to step S27.

  In step S26, the calibration circuit 93 reads the median minimum value Cn corresponding to the Nth column of the table and compares it with the measured median value. If the measured median value is larger than the minimum value, the process returns to step S24, and if it is smaller, the process proceeds to step S32.

  In step S27, the current value of N is substituted into the variable L, and the value of Nmax is substituted into the variable R.

  In step S28, it is determined whether L = R. This is a determination of whether N has reached the end of the table. If L = R, N cannot be changed, and the process directly returns to step S24. On the other hand, if L and R are different, the process proceeds to step S29.

  In step S29, the calibration circuit 93 sets the value of (L + R) / 2 as a new value of N, reads and sets the precharge voltage and exposure time corresponding to this Nth column, and based on these values Output a control signal.

  In step S30, the display unit 2 captures an image using the optical sensor 8, the gradation circuit 92 gradations the output value of the captured image, and the calibration circuit 93 measures the median value of the gradation histogram.

  In step S31, it is determined whether or not the measured median value is within a range determined by the minimum value and the maximum value corresponding to the column of the number N. If the measured value is larger than the maximum value, L = N and the process returns to step S28. If the measured value is smaller than the minimum value, R = N and the process returns to step S28. If the measured value is within the range between the minimum value and the maximum value, it is in the normal range, and the process returns to the normal loop of step S24.

  On the other hand, in step S32, the current value of N is substituted into the variable R, and the value of Nmin is substituted into the variable L.

  In step S33, it is determined whether L = R. This is a determination of whether N has reached the end of the table. If L = R, N cannot be changed, and the process directly returns to step S24. On the other hand, if L and R are different, the process proceeds to step S34.

  In step S34, the calibration circuit 93 sets the value of (L + R) / 2 as a new N value, reads out and sets the precharge voltage and exposure time corresponding to the Nth column, and based on these values. Output a control signal.

  In step S <b> 35, the display unit 2 captures an image using the optical sensor 8, the gradation circuit 92 gradations the output value of the captured image, and the calibration circuit 93 measures the median value of the gradation histogram.

  In step S36, it is determined whether or not the measured median value is within a range determined by the minimum value and the maximum value corresponding to the number N column. If the measured value is larger than the maximum value, L = N and the process returns to step S33. If the measured value is smaller than the minimum value, R = N and the process returns to step S33. If the measured value is within the range between the minimum value and the maximum value, it is in the normal range, and the process returns to the normal loop of step S24.

By such a binary search process, an appropriate precharge voltage and exposure time can be set in real time. This method only requires a maximum time of log ₂ N frames until an appropriate setting is made, and can be processed at a higher speed than the linear search process described above.

  Therefore, according to the present embodiment, by controlling the driving conditions of the optical sensor so that the upper value including the center in the gradation histogram is within a predetermined range, the shadow that occurs when the object approaches When the gradation value is low, it becomes difficult to be affected by such a change in the gradation value, so that the fluctuation of the output of the optical sensor due to the change in the illuminance of the external light can be accurately grasped.

  According to the present embodiment, the exposure time and the precharge voltage are controlled by using a table that discretely shows the relationship between the exposure time, the precharge voltage, and the predetermined range of the photosensor. You can also prevent the control from taking time.

  According to the present embodiment, since the upper gradation value can be changed according to the size of the display screen and the object, control using an appropriate gradation value according to the size of the display screen and the object is performed. It can be performed.

  According to the present embodiment, the S / N can be improved by changing the precharge voltage and maximizing the exposure time within an allowable range.

  According to the present embodiment, by using the linear search method, the exposure time and the precharge voltage can be obtained quickly and accurately so that the gradation value falls within a predetermined range.

  According to the present embodiment, by using the binary search method, the exposure time and the precharge voltage can be obtained faster than the linear search method.

  According to the present embodiment, by using the gradation value after the pixel thinning process is performed for each predetermined region, it is possible to reduce the burden of the process of changing the driving condition of the optical sensor.

  In this embodiment, as shown in FIGS. 12, 15, and 16, it is also effective to limit the area of the optical sensor for calculating the median and the like. The area other than the switch display at the top of the screen is usually not subject to shadows from fingers or hands, so it is considered that there is almost no change in the gradation histogram due to the approach of the hand. This is because a change in the histogram occurs and a value such as a median value changes.

  In the present embodiment, an example in which the driving condition such as the precharge voltage is changed immediately after the change of median or the like is shown, but a predetermined waiting period may be provided. When the median begins to change, it is not yet known whether it is due to a change in external light or a finger approach. If it is due to a change in external light, the driving condition should be changed, and if it is a change due to the approach of a finger, it is not necessary to change the driving condition (because it will return to its original state after a while). If the median value or the like returns to the original value after a predetermined waiting time (for example, 1 second), the drive conditions are not changed. If the value of median or the like continues to change to the extent that the drive condition needs to be changed after a predetermined waiting time, the drive condition is really changed. In this way, excessive calibration during temporary finger approach can be drastically reduced. As for the change point of the sensor IC, the predetermined period is the time required for the finger contact determination (difference diagram in FIG. 17, the first peak when the finger approaches and the finger is detached. Time is slightly longer than the time between the second peaks (this is set to “within 0.5 seconds” or the upper limit is determined). Store in a register.

  In each of the above embodiments, the case where a human finger touches the display screen has been described as an example, but the present invention is not limited to this. The object that touches the display screen may be a light pen with a light source such as an LED or a white mascot. Alternatively, a metal piece having a metallic mirror surface, a pencil sack, or the like may be used.

[Third Embodiment]
In the first embodiment, the potential of the signal line is compared with the fixed reference voltage in the comparator 41, but in this embodiment, a variable reference voltage that changes in accordance with the output of the comparator instead of the fixed reference voltage. Is used. Since the basic configuration of the display device is the same as that of the first embodiment, only the comparator and the reference voltage will be described here, and redundant description of the same parts as those of the first embodiment will be omitted.

  As in the first embodiment, when the reference voltage of the comparator 41 is fixed, it is not possible to output a comparison result that appropriately corresponds to a change in the illuminance of ambient ambient light. In addition, if a reference power supply 40 is provided for each comparator 41 in order to provide a fixed reference voltage, the circuit scale becomes enormous.

  Therefore, in the present embodiment, as shown in FIG. 28, the output value from each comparator 41 provided for each pixel is parallel-serial converted by the parallel-serial conversion unit 42 and output, and this output value is output. Accordingly, the reference voltage of the comparator 41 is changed. This variable reference voltage value is determined by the arithmetic circuit 51 shown in the plan view of FIG.

  As shown in FIG. 29, the display device of this embodiment includes a display unit 2, an arithmetic circuit 51, an LCD gate driver circuit 52, a display circuit 53, and an optical input gate driver circuit 54 on an array substrate 1.

  The arithmetic circuit 51 varies the reference voltage of the comparator in the next frame based on the output value of the comparator 41 in one frame so that the comparator 41 can output an appropriate comparison result according to the illuminance of external light. To. For example, in a display device having a light input function for recognizing a shadow caused by a finger or a pen, when the bright is “1” and the dark is “0”, the surroundings are bright and the output value of each comparator 41 is When all are 1, the reference voltage value is adjusted so that “0” is generated in the output of the comparator. Conversely, when the surroundings are dark and the output value of each comparator 41 is “0”, the reference voltage value is adjusted so that “1” increases in the output of the comparator. This adjustment is continued until a value that can recognize a difference in brightness of a certain area can be recognized.

  The LCD gate driver circuit 52 performs on / off control of a switch for outputting a video signal of each pixel to the pixel electrode. The display circuit 53 outputs the transmitted video signal to the display unit 2. The light input gate driver circuit 54 performs on / off control of a switch for controlling the output of the light sensor of each pixel.

  Therefore, according to the present embodiment, by changing the reference voltage of the comparator 41 according to the output value of the comparator 41, the comparator 41 can output an appropriate comparison result according to the illuminance of ambient environmental light. it can. Further, it can be realized with almost no increase in the wiring of the panel.

It is a top view which shows the structure of the display apparatus in one embodiment. It is sectional drawing which shows the structure of the display part in the said display apparatus. It is a circuit diagram which shows the structure of the pixel with which the said display part is provided. It is a circuit diagram which shows the structure of the optical sensor system in the said pixel. It is a timing chart which shows operation | movement of the said display apparatus. It is a figure which shows the image pattern which the said display apparatus displays on a display screen. It is a circuit block diagram which shows the structure of IC for sensors in the said display apparatus. It is a figure which shows the example of the process in the gradation circuit in the said IC for sensors. It is a graph which shows the relationship between the ratio of a white pixel, and exposure time. It is a figure which shows a screen in the case of outputting an average value based on the output value of the optical sensor of the whole screen. It is a figure which shows the gradation histogram at the time of using the optical sensor of the whole screen. It is a figure which shows a screen in the case of outputting an average value based on the output value of the optical sensor in areas other than the switch at the top of the screen. It is a figure which shows the gradation histogram at the time of using the optical sensor in areas other than the switch of the screen upper part. It is a figure which shows the time change of the gradation average value before and behind a finger | toe approaches. It is a figure which shows a screen when the area | region for calibration is set to the area | region on the left side of the screen upper part. It is a figure which shows a screen when the area | region for a calibration is set to the area | region on the right side of the screen upper part. It is the figure which put together the characteristic of the time change of the number of white pixels for every switch. It is a figure which shows the example of the process in the difference processing circuit between frames in the said IC for sensors. It is a figure which shows the example of the process in the edge detection circuit in the said IC for sensors. It is a figure for demonstrating the process of pixel thinning-out in a gradation circuit. It is a figure which shows the average value and median value in the gradation histogram before a finger approach. It is a figure which shows the average value and median value in the gradation histogram after a finger approach. It is a figure which shows the table which set the relationship of the precharge voltage, exposure time, the minimum value of median value, and the maximum value. It is the figure which formed the table of FIG. 23 into a graph. It is a graph which shows the relationship between precharge time and exposure time. It is a flowchart which shows the flow of the process which sets exposure time and a precharge voltage by a linear search method. It is a flowchart which shows the flow of the process which sets exposure time and a precharge voltage by a 2-minute search method. It is a circuit diagram which shows another structure of the optical sensor system in the said display apparatus. It is a top view which shows the arrangement place of the arithmetic circuit which determines the variable reference voltage value for comparators.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Array substrate, 2 ... Display part 3 ... Flexible substrate 4 ... IC for sensors
5 ... IC for display
6 ... Sensor I / F
7 ... Display I / F
DESCRIPTION OF SYMBOLS 8 ... Optical sensor, 9 ... Insulating layer 11 ... Liquid crystal layer, 12 ... Opposite substrate 13 ... Backlight, 20 ... Object 31 ... Display system, 32 ... Optical sensor system 33 ... Switch element 34 ... Output control switch 35 ... Source follower amplifier 37 ... sensor capacitance 38 ... precharge control switch 40 ... reference power supply 41 ... comparator 42 ... parallel / serial converter 51 ... arithmetic circuit 52 ... LCD gate driver circuit 53 ... display circuit 54 ... light input gate driver circuit 91 ... level shifter 92 ... gradation circuit 93 ... calibration circuit 94 ... DAC
95 ... Counter circuit 96 ... Inter-frame difference processing circuit 97 ... Edge detection circuit 98 ... Contact probability calculation circuit 99 ... RAM

Claims (12)

Display means for displaying an image on a display screen and photographing an object close to the display screen using an optical sensor;
Sensor driving means for changing the driving conditions of the optical sensor based on the statistic of the output value of the optical sensor;
A display device comprising:   The display device according to claim 1, wherein the driving condition is at least one of an exposure time or a precharge time of an optical sensor. The display means displays an area for bringing an object close to the display screen,
The display device according to claim 1, wherein the sensor driving unit changes a driving condition based on a statistic of an output value of an optical sensor arranged in an area other than the area on the display screen.   The sensor driving means outputs a control signal for controlling the driving condition so that a higher gradation value including a center in a histogram of gradation values obtained by gradation of the output value falls within a predetermined range. The display device according to claim 1, wherein the display device is a display device.   The display device according to claim 4, wherein the sensor driving unit can change the upper gradation value according to a display screen and a size of an object.   6. The display device according to claim 4, wherein the sensor driving unit determines the control signal using a table indicating a relationship between an exposure time of an optical sensor, a precharge voltage, and the predetermined range.   7. The display device according to claim 6, wherein the table has a maximum exposure time within an allowable range and changes a precharge voltage.   The display device according to claim 6, wherein the sensor driving unit determines the control signal using the table by a linear search method.   The display device according to claim 6, wherein the sensor driving unit determines the control signal using the table by a binary search method.   10. The sensor driving unit according to claim 4, wherein the sensor driving unit uses a gradation value after pixel thinning processing is performed for each region when the display screen is divided into a plurality of regions within a certain range. A display device according to any one of the above.   The sensor driving means has a predetermined waiting period between the time when the output value of the optical sensor changes to the extent that the driving condition of the optical sensor should be changed and the time when the driving condition of the sensor is actually changed. The display device according to claim 1, further comprising a display device. 12. The display device according to claim 11, further comprising means for holding the statistics during the predetermined waiting period.