JP4303954B2 - Display device - Google Patents

Description

（１）の計算結果が100％を超える場合には、Y(m,n) = 100％とする。なお、式（１）は例であって、表示素子の特性、センサの特性、撮像対象の特性に応じて変形すべきである。要は最適撮像時間のばらつきを表示輝度で保証するようなものであればよい。
【００８７】
次に、撮像時間を例えば0.5msec刻みで変えながら、撮像時間10msec〜50msecまで81枚の撮像を行う（ステップＳ２６；本撮像）。
【００８８】
次に、81枚の撮像結果の平均値を計算し、最終的な撮像データを得る（ステップＳ２７）。例えば、図３７はステップＳ２７で得られる撮像データの一例を示している。このとき、撮像時間を同じ間隔で変えながら撮像を行うのではなく、図３７及び図３８に示すように30msec付近の撮像を多めに行うなどの重み付けを行って、例えば、81枚より少ない回数の撮像結果を用いて平均値を計算してもよい。全体の処理時間を短縮できる。あるいは、平均値を計算する際に、各撮像結果に重み付けをしてもよい。
【００８９】
図３５のような処理を行うことで、図３４に示すように撮像対象に黒密度のばらつきがあっても、図３９に示すように、部分的な黒つぶれやかすれのない良好な撮像結果が得られる。
【００９０】
このように、第２の実施形態では、撮像対象の黒密度を事前に（試行撮像によって）調べて、撮像対象に適した表示輝度をブロック単位で設定した状態で、センサ１２による撮像を行うため、部分的な黒つぶれやかすれのない、均一な画質の撮像画像が得られる。
【００９１】
本実施形態では、撮像対象の黒密度の分布の偏りに起因する撮像結果の黒つぶれを解決する手法を説明したが、センサのリーク電流の偏りやTFTの特性ばらつきの偏り等に起因する撮像結果の画質劣化の解決手段としても、同様の効果を奏する。
【００９２】
また、試行撮像の後の本撮像の動作を次のような手順で行ってもよい。
（R1）図３０のモードｍ１を行い、液晶の表示を全面赤表示にする。但し試行撮像に基づいて各ブロックごとに階調は変化させる。
（R2）モードｍ２を行い、全てのセンサ容量のプリチャージを行う。
（R3）モードｍ３，ｍ４，ｍ５を行い、赤信号線、緑信号線及び青信号線に属すする画素の撮像データも出力する。
（G1）モードｍ１を行い、液晶の表示を全面緑表示にする。但し試行撮像に基づいて各ブロックごとに階調は変化させる。
（G2）モードｍ２を行い、全てのセンサ容量のプリチャージを行う。
（G3）モードｍ３，ｍ４，ｍ５を行い、赤信号線、緑信号線及び青信号線に属すする画素の撮像データも出力する。
（B1）モードｍ１を行い、液晶の表示を全面青表示とする。但し試行撮像に基づいて各ブロックごとに階調は変化させる。
（B2）モードｍ２を行い、全てのセンサ容量のプリチャージを行う。
（B3）モードｍ３，ｍ４，ｍ５を行い、赤信号線、緑信号線及び青信号線に属すする画素の撮像データも出力する。
【００９３】
一見、表示を全面赤として撮像を行う際、緑画素及び青画素のセンサのデータは無意味と思い込みやすいがそうではない。とくに光源（液晶層の背面に設けるバックライトの光が拡散光であり撮像面にいろいろな角度から光が照射されるような場合で、かつ、図５のように液晶セルの観察面側にセンサアレイが配置される構成の場合に意義がある。従来の密着CMOSイメージセンサと異なり、センサは撮像対象と密着できず、ｄ（ガラス基板厚＋偏光板などの光学フィルム厚）程度（0.2乃至0.7mm）は離れる。バックライト光は確かに赤画素からのみ発するが、撮像対象での反射光は適当に拡散する。拡散範囲はｄ程度であり画素ピッチはｄ程度かそれより小さい。したがって、緑画素及び青画素のセンサにも撮像対象に基づく光が入射する。上述の（R1）〜(B3)の処理を行うことにより、水平方向により高解像度な撮像結果を得ることができる。以下撮像時間を変えながら複数回繰り返して平均化処理する部分は詳述した実施形態と同様なので省略する。
【００９４】
試行撮像では結局、各ブロックごとの平均階調を知れば足りることに着目すると、試行撮像においては、カウンタによる平均階調の計数結果のみブロックごとに出力するように構成してもよい。８個のデータ出力回路を停止して外部負荷駆動するための消費電力を節約することができる。
【００９５】
また、本実施形態ではSRAMを画素内に設け、▲１▼センサの微弱な電流の増幅、▲２▼撮像後データを出力するまでの間のデータ保持をさせているが、SRAMに限定されない。ソースフォロアで▲１▼の電流増幅をしてもよい。撮像後データを出力するまでの間、リークなどのためデータを保持し続けることができない場合は、撮像直後にデータを出力するような制御回路及びシーケンスを用意すればよい。
【００９６】
また、表示画面を区分してブロックごとに撮像時の表示輝度を調節する例を説明したが、対角画面寸法が比較的小さい表示装置（2インチ程度以下の携帯電話向け等）の場合や、センサやTFTの特性のチップ内ばらつきが小さい場合は、画面を分割せずに（分割数＝１として）適用してもよい。その場合さらに、試行撮像と本撮像を分けずに次のように行ってもよい。
【００９７】
すなわち、（１）まず、撮像時間をt0=10msec（どんな撮像対象でも殆どセンサに光リークが生じず、黒つぶれ画面が得られるような時間）で最初の撮像データ出力と、平均階調の計数を行う。最初の撮像データはコントロールIC５５等のメモリに保存する。
【００９８】
（２）t=t0+Δt（例えばΔt=0.5msec）として2回目の撮像をし、平均階調の計数を行う。
【００９９】
（３）平均階調が所定値未満であれば、撮像データを出力することなく、撮像時間をt=t+2Δtとして、撮像と平均階調の計数を行う。
【０１００】
（４）計数結果が所定値以上であれば撮像データを出力してコントロールIC５５等のメモリに加算する。
【０１０１】
（５）（２）〜（４）を平均階調が256階調程度になるまで撮像時間tを適当に増やしながら繰り返す。
【０１０２】
上述した（１）〜（５）によりメモリ上に完成する画像を上述の実施形態で得られるものと同等の高品位な撮像結果と考えることができる。しかも画像演算のためのメモリは1フレーム程度しか要しない。ハード資源に制約のある携帯電話などで特に有効である。
【０１０３】
（第３の実施形態）
第２の実施形態の場合、ブロックごとに表示輝度を設定するため、場合によっては、隣接するブロック同士が著しく表示輝度が異なる場合もありうる。
【０１０４】
図４０は第２の実施形態における各ブロックの表示輝度の一例を示す図であり、横軸はブロックの位置、縦軸は表示輝度を表している。理解がしやすいように、特定の1行に属するブロックを抜き出した。また、図４１は第２の実施形態における隣接する４ブロックの撮像画像を図式化した図である。これらの図に示すように、隣接するブロック同士の表示輝度が不連続に変化している。このため、隣接するブロック同士で輝度差が大きいと、白密度の多い撮像対象を撮像したときに、図４１のような色ムラが起きるおそれがある。たとえば同じ白地のはずなのに、撮像結果は市松状のブロックごとに白さが違って見えてしまうようなことが起こる。
【０１０５】
そこで、第３の実施形態では、各ブロック内の中央画素については、図３５と同様の処理により表示輝度を設定し、中央画素から徐々に表示輝度を変化させ、隣接ブロックの境界付近の輝度差があまり生じないようにする。
【０１０６】
図４２は図４０と同じ条件で撮像を行った場合の第３の実施形態の処理結果を示す図、図４３は第３の実施形態における隣接する４ブロックの撮影画像を図式化した図である。これらの図に示すように、隣接するブロック間で輝度差が大きく変化しなくなる。
【０１０７】
このように、第３の実施形態では、ブロックの中央画素から周辺にかけて徐々に輝度を変化させ、隣接するブロック間で輝度が大きく変化しないようにしたため、ブロック間の輝度差による撮像画像の色ムラがなくなる。
【０１０８】
（第４の実施形態）
第４の実施形態は、センサ１２による撮像結果を、予め用意した基準パターンとパターンマッチングするものである。
【０１０９】
図４４は本発明に係る表示装置の第４の実施形態の概略構成を示すブロック図である。図４４の表示装置は、図１９の構成に加えて、複数の基準パターンを格納する基準パターン格納部８６を備えている。
【０１１０】
図４５は基準パターン格納部に格納されている基準パターンの一例である。各基準パターン1a,1b,1c,2a,2b,2c,2d,3a,3b,3c,3dは、８画素×８画素のサイズであり、黒で示した部分がパターンを示している。なお、基準パターンのサイズや種類は図示したものに限定されない。
【０１１１】
図４６は本実施形態のコントロールIC５５が行う処理動作を示すフローチャートである。以下では、仮に図４７のような撮像対象をセンサ１２で撮像した結果、図４８のような撮像データが得られたとして、図４６のフローチャートの処理動作を説明する。
【０１１２】
本実施形態のコントロールIC５５は、センサ１２の撮像データを基準パターン格納部８６に格納されているすべての基準パターンと比較する（ステップＳ３１）。
【０１１３】
図４５の各基準パターンの上部に付された数字は、図４８の撮像データとの不一致画素数である。コントロールIC５５は、不一致画素数の少ない基準パターンをいくつか選択する（ステップＳ３２）。例えば、コントロールIC５５が図４５の４つの基準パターン1a,1b,1c,1dを選択したとする。
【０１１４】
次に、コントロールIC５５は、選択した基準パターンの明暗を反転させたパターン（図４９の反転パターンn1a,n1b,n1c,n1d）を生成し（ステップＳ３３）、この反転パターンを画素アレイ部１に順に表示させながら、センサ１２による撮像を繰返し行う（ステップＳ３４）。この場合、図４９の反転パターンn1a,n1b,n1c,n1dの白部分のみ光が透過するため、センサ１２の撮像データは図５０のようになる。撮像データr1aは基準パターン1aと反転パターンn1aに対応し、撮像データr1bは基準パターン1bと反転パターンn1bに対応し、撮像データr1cは基準パターン1cと反転パターンn1cに対応し、撮像データr1dは基準パターン1dと反転パターンn1dに対応する。
【０１１５】
次に、撮像データの中から、白色画素数がより少ない基準パターンをいくつか選択する（ステップＳ３５）。例えば、図５０の例では、２つの基準パターン1a,1bを選択する。
【０１１６】
次に、選択した基準パターンに基づいて、最終的な撮像結果を得る（ステップＳ３６）。ここでは、選択した基準パターンと最初に得られた撮像データとを平均化するなどして、図５１に示す最終的な撮像結果を得る。
【０１１７】
このように、第４の実施形態では、予め基準パターンを複数種類用意し、センサ１２による撮像データを基準パターンと比較して最終的な撮像データを生成するため、センサ１２の解像度をあまり高くすることなく、高品質の撮像データが得られる。特に、本実施形態は、形状が予めパターン化されている撮像対象を撮像する場合に特に効果が大きい。
【０１１８】
上述した各実施形態では、本発明に係る表示装置を液晶表示装置に適用した例について説明したが、本発明は、EL（Electroluminescense)表示装置やPDP（Plasma Display Panel）などの他の表示装置にも適用可能である。
【０１１９】
【発明の効果】
以上詳細に説明したように、本発明によれば、複数の撮影条件で画像取込みを行った結果に基づいて取込画像のデジタル画像データを生成するため、撮像部の特性ばらつき等の影響を受けることなく画像取込みを行うことができ、取込画像の品質を向上できる。また、複数回の画像取り込みおよび演算を行うにもかかわらず、多くの計算資源を要しない。
【０１２０】
また、本発明によれば、表示素子の表示輝度をブロックごとに設定した状態で撮像部による撮像を行うため、撮像対象に部分的な黒密度のばらつきがあっても、黒つぶれやかすれのない高品質の撮像画像が得られる。
【０１２１】
さらに、本発明は、撮像部による撮像結果を、予め用意した基準パターンと比較して、その比較結果に基づいて最終的な撮像データを生成するため、撮像部の解像度を上げなくても、撮像対象に忠実な撮像データが得られる。
【図面の簡単な説明】
【図１】本発明に係る表示装置の第１の実施形態の概略構成図。
【図２】画素アレイ部１の一部を詳細に示したブロック図。
【図３】図２の一部を詳細に示した回路図。
【図４】 SRAMの内部構成を示す回路図。
【図５】表示装置の断面図。
【図６】図１に示したロジックＩＣの内部構成を示すブロック図。
【図７】キャパシタに印加する電圧を切り替える例を示す図。
【図８】ＣＰＵの処理動作の一例を示すフローチャート。
【図９】名刺の画像を取り込む例を示す図。
【図１０】周囲８画素の平均を取る様子を示す図。
【図１１】図８の処理結果を示す画像の例を示す図。
【図１２】「Ｔ」の文字を含む画像の一例を示す図。
【図１３】図１２の点線行の画像取込みを行った結果を示す図。
【図１４】図１３の画像を取り込んだ結果として最終的に得られる図。
【図１５】各撮影条件での画像取込み結果を別個にメインメモリに格納する例を示す図。
【図１６】メインメモリの容量を削減する例を示す図。
【図１７】（ａ）は撮像対象の一例を示す図、（ｂ）は撮像結果の一例を示す図。
【図１８】本発明に係る表示装置の第２の実施形態の全体構成を示すブロック図。
【図１９】ガラス基板上の信号線駆動回路、走査線駆動回路、センサ制御回路及び信号処理出力回路と、制御回路基板上のコントロールIC及びメモリとの接続関係を示すブロック図。
【図２０】ガラス基板３１の詳細構成の一例を示すブロック図。
【図２１】走査線駆動回路３の詳細構成の一例を示す回路図。
【図２２】信号処理出力回路５４の詳細構成の一例を示すブロック図。
【図２３】同期信号発生回路７１の詳細構成の一例を示すブロック図。
【図２４】 P/S変換回路７２の詳細構成の一例を示すブロック図。
【図２５】デコーダの内部構成の一例を示す回路図。
【図２６】ラッチの内部構成の一例を示す回路図。
【図２７】出力バッファ７３の詳細構成を示すブロック図。
【図２８】画素アレイ部１の１画素分の詳細回路図。
【図２９】ガラス基板３１上の１画素分のレイアウト図。
【図３０】本実施形態の表示装置の動作を説明する図。
【図３１】モードｍ１の動作タイミング図。
【図３２】モードｍ２，ｍ３の動作タイミング図。
【図３３】モードｍ４，ｍ５の動作タイミング図。
【図３４】ブロック分割を説明する図。
【図３５】図１８のコントロールIC５５が行う処理動作の一例を示すフローチャート。
【図３６】撮像時間と平均階調との関係を示す図。
【図３７】ステップＳ７で得られる撮像データの一例を示す図。
【図３８】平均階調増分を説明する図。
【図３９】本実施形態の撮像結果の一例を示す図。
【図４０】第２の実施形態における各ブロックの表示輝度の一例を示す図。
【図４１】第２の実施形態における隣接する４ブロックの撮像画像を図式化した図。
【図４２】図４０と同じ条件で撮像を行った場合の第３の実施形態の処理結果を示す図。
【図４３】第３の実施形態における隣接する４ブロックの撮影画像を図式化した図。
【図４４】本発明に係る表示装置の第４の実施形態の概略構成を示すブロック図。
【図４５】基準パターン格納部に格納されている基準パターンの一例を示す図。
【図４６】本実施形態のコントロールIC５５が行う処理動作を示すフローチャート。
【図４７】撮像対象の一例を示す図。
【図４８】撮像結果の一例を示す図。
【図４９】反転パターンの一例を示す図。
【図５０】センサの撮像データの一例を示す図。
【図５１】最終的な撮像結果の一例を示す図。
【符号の説明】
１ 画素アレイ部
２ 信号線駆動回路
３ 走査線駆動回路
４ 検出回路＆出力回路
１１ 画素ＴＦＴ
１２ａ，１２ｂ センサ
１３ バッファ
２１ アレイ基板
２２ 紙面
２３ バックライト
２４ 対向基板
３１ ガラス基板
３２ 半導体基板
３３ ロジックＩＣ
５２ 制御回路基板
５３ センサ制御回路
５４ 信号処理出力回路
５５ コントロールIC
５６ メモリ
８６ 基準パターン格納部[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a display device having an image capturing function.
[0002]
The liquid crystal display device includes an array substrate on which signal lines, scanning lines, and pixel TFTs are arranged, and a drive circuit that drives the signal lines and the scanning lines. With the recent progress and development of integrated circuit technology, a process technology for forming a part of a drive circuit on an array substrate has been put into practical use. As a result, the entire liquid crystal display device can be made light and thin, and it is widely used as a display device for various portable devices such as mobile phones and notebook computers.
[0003]
By the way, a display device having an image capturing function in which a contact area sensor for capturing an image is arranged on an array substrate has been proposed (see, for example, Patent Documents 1 and 2).
[0004]
A conventional display device having this type of image capture function changes the charge amount of a capacitor connected to the sensor in accordance with the amount of light received by the sensor, and detects the voltage across the capacitor, thereby capturing the image. It is carried out.
[0005]
[Patent Document 1]
Japanese Patent Laid-Open No. 2001-292276
[Patent Document 2]
JP 2001-339640 A
[0006]
[Problems to be solved by the invention]
However, since the current flowing through the sensor is weak, it is difficult to accurately detect a change in the voltage across the capacitor due to the current, resulting in a large measurement error. For this reason, noise tends to appear in the captured image.
[0007]
In addition, when an SRAM or a buffer circuit is connected to the capacitor in order to detect the voltage across the capacitor, it is “0” or “1” depending on whether the threshold voltage of the transistors constituting the SRAM or the buffer circuit is exceeded. However, since the threshold voltages of the transistors vary, the determination criteria of “0” and “1” may be shifted. In addition, since the current flowing through the sensor also varies, there is a possibility that the determination criteria of “0” and “1” are shifted.
[0008]
The present invention has been made in view of these points, and an object of the present invention is to provide a display device capable of capturing an image without being affected by variations in light leakage and variations in electrical characteristics of transistors and the like. Is to provide.
[0009]
According to one aspect of the present invention, display elements formed in the vicinity of intersections of signal lines and scanning lines arranged in rows and columns,
At least one image sensor corresponding to each of the display elements, each receiving an incident light in a specified range and converting it into an electrical signal;
A charge accumulating unit that accumulates electric charges according to the electrical signal converted by the imaging unit;
A signal processing unit that generates digital image data corresponding to an image captured by the imaging unit based on accumulated charges of the charge storage unit in each of a plurality of imaging conditions;
A binary data generation unit that outputs binary data indicating whether or not the charge accumulated in the charge accumulation unit is equal to or greater than a predetermined threshold;
A shooting condition switching unit that switches the shooting conditions step by step in ascending or descending order,
If the logic of the binary data changes when the shooting condition switching unit switches the shooting condition by one step, the signal processing unit may change the value of the binary data of a plurality of pixels around the target pixel. Generating digital image data of the pixel of interest based on Display device Is provided.
[0012]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, a display device according to the present invention will be specifically described with reference to the drawings.
FIG. 1 is a schematic configuration diagram of a first embodiment of a display device according to the present invention, which is characterized by having an image capturing function. The display device shown in FIG. 1 includes a glass substrate 31 and a semiconductor substrate 32. On the glass substrate 31, a pixel array section 1 in which signal lines and scanning lines are arranged, a signal line driving circuit 2 for driving signal lines, a scanning line driving circuit 3 for driving scanning lines, and an image are captured. Is provided with a detection circuit & output circuit 4 for outputting at the same time. These circuits are formed by, for example, polysilicon TFTs. The signal line driving circuit 2 includes a D / A conversion circuit that converts digital pixel data into an analog voltage suitable for driving the display element. A known D / A conversion circuit is used. On the semiconductor substrate 32, a logic IC 33 for performing display control and image capture control is mounted. The glass substrate 31 and the semiconductor substrate 32 transmit and receive various signals through, for example, an FPC.
[0013]
FIG. 2 is a block diagram showing a part of the pixel array unit 1 in detail. 2 includes a pixel TFT 11 formed in the vicinity of each intersection of a signal line and a scanning line arranged in rows and columns, a liquid crystal capacitor C1 connected between one end of the pixel TFT 11 and the Cs line, and The auxiliary capacitor C2 and two image capturing sensors 12a and 12b provided for each pixel TFT11 are provided. The sensors 12a and 12b are connected to a power line and a control line (not shown).
[0014]
Although FIG. 2 shows an example in which two sensors 12a and 12b are provided for each pixel in order to increase the resolution of image capture, the number of sensors is not particularly limited.
[0015]
FIG. 3 is a circuit diagram showing a part of FIG. 2 in detail. As shown in FIG. 3, the sensors 12a and 12b have photodiodes D1 and D2 and sensor switching transistors Q1 and Q2, respectively. The photodiodes D1 and D2 output an electrical signal corresponding to the amount of received light. The sensor switching transistors Q1 and Q2 alternately select any one of the plurality of photodiodes D1 and D2 in one pixel.
[0016]
Each pixel includes two sensors 12a and 12b, a capacitor C3 shared by the two sensors 12a and 12b in the same pixel, a buffer 13 for storing binary data corresponding to the accumulated charge in the capacitor C3, and a buffer 13 A transistor Q3 that controls writing to the buffer, and a reset transistor Q4 that initializes the buffer 13 and the capacitor C3.
[0017]
The buffer 13 is composed of a static RAM (SRAM). For example, as shown in FIG. 4, two inverters IV1 and IV2 connected in series, an output terminal of a subsequent inverter IV2, and an input terminal of a previous inverter IV1 And an output transistor Q6 connected to the output terminal of the subsequent stage inverter.
[0018]
When the signal SPOLB is at a high level, the transistor Q5 is turned on and the two inverters IV1 and IV2 perform a holding operation. When the signal OUTi is at a high level, the retained data is output to the detection line.
[0019]
The display device of the present embodiment can perform a normal display operation, and can also capture an image similar to a scanner. When a normal display operation is performed, the transistor Q3 is set to an off state, and valid data is not stored in the buffer 13. In this case, the signal line voltage from the signal line driving circuit 2 is supplied to the signal line, and display according to the signal line voltage is performed.
[0020]
On the other hand, when image capture is performed, an image capture target (for example, paper surface) 22 is arranged on the upper surface side of the array substrate 21 as shown in FIG. Irradiates the paper surface 22 via The light reflected by the paper surface 22 is received by the sensors 12a and 12b on the array substrate 21, and an image is captured.
[0021]
The captured image data is stored in the buffer 13 as shown in FIG. 3, and then sent to the logic IC 33 shown in FIG. 1 via the detection line. The logic IC 33 receives the digital signal output from the display device of the present embodiment and performs arithmetic processing such as data rearrangement and noise removal in the data.
[0022]
FIG. 6 is a block diagram showing an internal configuration of the logic IC 33 shown in FIG. As illustrated in FIG. 6, the logic IC 33 includes a display control unit 41 that performs display control on the pixel array unit 1, an image capture control unit 42 that performs image capture control of the sensors 12 a and 12 b, and the entire logic IC 33. It has CPU43 which performs control, and main memory 44 which CPU43 uses for work.
[0023]
The image capture control unit 42 includes a buffer memory 45 that temporarily stores imaging data on the detection line in FIG. 3 and a control signal generation circuit 46 that generates a control signal for image capture. The CPU 43 performs image processing of the captured image based on the imaging data stored in the buffer memory and generates display image data.
[0024]
The display control unit 41 generates a buffer memory 47 for temporarily storing the display image data generated by the CPU 43, and a control signal for controlling the operation timing of the signal line driving circuit 2 and the scanning line driving circuit 3 in FIG. And a control signal generation circuit 48.
[0025]
When capturing an image, initial charges are accumulated in the capacitor C3 of each pixel in advance. When the sensors 12a and 12b capture a whitish image, a current flows through the sensors 12a and 12b, the charge of the capacitor C3 corresponding to the sensors 12a and 12b is discharged, and the voltage across the capacitor C3 decreases. On the other hand, when the sensors 12a and 12b capture black images, current does not flow so much in the sensors 12a and 12b, and the voltage across the capacitor C3 hardly changes.
[0026]
Therefore, the density of the captured image can be determined by detecting the voltage across the capacitor C3. In the present embodiment, the voltage across the capacitor C3 is temporarily stored in the buffer 13 made of SRAM. The buffer 13 determines “1” if the voltage across the capacitor C3 is equal to or higher than the threshold voltage of the first-stage inverter of the SRAM, and determines “0” if it is less than the threshold.
[0027]
However, since the variation in the light leakage current due to the sensors 12a and 12b is weak, the voltage across the capacitor C3 is likely to vary, and the threshold voltage of the transistors constituting the SRAM also varies. Even if it is taken in, it is determined as “1” or “0” depending on the case. Such variations appear as noise in the captured image. Compared to the current variation of photoelectric conversion elements formed on silicon wafers often used in commercially available scanners, photoelectric conversion elements formed on insulating substrates such as glass substrates used for display array substrates are current variations. Becomes larger. The latter has a large area and a low process temperature (restricted by the heat-resistant temperature of the substrate), so that uniform formation is difficult. Therefore, a variation compensation means specific to the display device is required in some form. In addition, it is desirable to be able to reproduce subtle gradations of the imaging target, but this is hindered by the above-described variation. Hereinafter, a description will be given of means and configuration capable of reducing noise or reproducing gradation display even though the sensor circuit includes a transistor having a large characteristic variation and a photoelectric conversion element having a variation in leak current.
[0028]
The CPU 43 shown in FIG. 6 captures images a plurality of times while changing the imaging conditions of the sensors 12a and 12b, and generates final captured image data based on the results of the plurality of image captures. Specifically, as shown in FIG. 7, a control signal for capturing an image in a state where the voltage Vprc applied to the capacitor C3 is changed in four ways and each voltage Vprc is applied to the capacitor C3 is displayed on the glass substrate. To supply. In addition, the digital data as a result of image capture output from the glass substrate is processed. Since the signal (digital pixel data, control clock, control signal) input to the glass substrate and the signal output from the glass substrate are both digital signals (digital signals based on the imaging result), the display control unit and the image shown in FIG. The capture control unit can be easily formed on a one-chip semiconductor. If there is no D / A conversion circuit on the glass substrate, an amplifier circuit (analog circuit) is required for the display control unit, and the one-chip configuration is advantageous compared to the high cost. In addition, with the recent progress in miniaturization and improvement in the degree of integration of semiconductor manufacturing processes, the CPU and main memory in FIG. 6 can be easily integrated into a single chip with a display control unit and an image capture control unit.
[0029]
FIG. 8 is a flowchart showing an example of processing operation of the CPU 43. First, the CPU 43 has a voltage Vprc = 3.5 V (a value relatively close to the threshold value of the first stage inverter of the SRAM) at one end of the capacitor C3 of FIG. 3 provided for each pixel. The initial charge is accumulated in the capacitor C3 (step S1).
[0030]
Next, the first image capture is performed (step S2). In this case, current flows in the sensors 12a and 12b that read the white portion of the image or gray near white, the initial charge of the capacitor C3 is discharged, and the voltage across the capacitor C3 decreases. On the other hand, since no current flows through the sensors 12a and 12b that read the black portions of the image, the voltage across the capacitor C3 hardly changes.
[0031]
In step S2, if the voltage across the capacitor C3 is higher than the threshold voltage of the first inverter of the SRAM, it is determined that the pixel is black. That is, first, in the captured image, only the black portion is extracted in step S2, the extracted pixel is determined as a black pixel value, and the other pixels are stored in the main memory 44 as white pixel values. (Step S3). Since the precharge voltage of the capacitor C3 is relatively close to the threshold voltage of the first stage inverter of the SRAM, if the portion of the image facing the sensor portion is more or less white, some leakage current occurs and the capacitor C3 Is likely to be lower than the threshold voltage of the first-stage inverter of the SRAM. Conversely, in this state, if the voltage of C3 continues to be above the threshold voltage of the inverter, it means that the corresponding image portion can be definitely judged to be black.
[0032]
For example, FIG. 9 is an example in which an image of a business card (black characters on a white background) is captured, and FIG. 9A shows the captured image obtained in steps S1 to S3. In step S3, since only the blackish pixels are detected as black, as shown in FIG. 9A, an image that is entirely whitish and has some characters blurred is obtained.
[0033]
Next, a voltage Vprc = 4V is applied to one end of the capacitor C3, initial charge is accumulated in the capacitor C3 (step S4), and a second image capture is performed (step S5). In this case, pixels that are slightly whitish than the first time may be determined to be black.
[0034]
When the second image capture is completed, pixels that are determined to be white for the first time and black for the second time are extracted, and the average value of the first pixel values of the eight pixels around the extracted pixels is calculated. Is the pixel value of the extracted pixel (step S6).
[0035]
FIG. 9B shows the captured image obtained in steps S4 to S6. Since an image that is slightly whitish than that in FIG. 9A is also determined to be black, an overall darker image than that in FIG. 9A is obtained.
[0036]
In this step S6, for example, if the pixel indicated by the hatched portion in FIG. 10 is an extraction pixel, the average value (G1 +... + G8) / 2 of the surrounding pixel values G1 to G8 is obtained as the pixel value of the extraction pixel. And If G1 to G8 are all white, the pixel value is white, but if G1 to G8 are several white and black, the pixel value is halftone.
[0037]
Next, a voltage Vprc = 4.5 V is applied to one end of the capacitor C3, initial charge is accumulated in the capacitor C3 (step S7), and a third image capture is performed (step S8). In this case, pixels that are slightly whitish than the second time may be determined to be black.
[0038]
FIG. 9C shows the captured image obtained in steps S7 to S9. Since an image that is slightly whitish than that in FIG. 9B is also determined to be black, an overall darker image than that in FIG. 9B is obtained.
[0039]
When the third image capture is completed, pixels that are determined to be white for the second time and black for the third time are extracted, and the average value of the first pixel values of the eight pixels around the extracted pixel is calculated. Is the pixel value of the extracted pixel (step S9).
[0040]
Next, a voltage Vprc = 5 V is applied to one end of the capacitor C3, initial charge is accumulated in the capacitor C3 (step S10), and a fourth image capture is performed (step S11). In this case, pixels that are slightly whitish than the third time may be determined to be black.
[0041]
FIG. 9D shows the captured image obtained in steps S10 to S12. Since an image slightly whitish than that in FIG. 9C is also determined to be black, an overall darker image than in FIG. 9C is obtained.
[0042]
When the fourth image capture is completed, pixels that are determined to be white for the third time and black for the fourth time are extracted, and the average value of the first pixel values of the eight pixels around the extracted pixel is calculated. Is the pixel value of the extracted pixel (step S12).
[0043]
The image obtained as a result of performing the process of step S12 is as shown in FIG. 11, and it can be seen that halftones can be expressed and noise can be removed.
[0044]
FIG. 12 is a diagram showing an example of an image including the character “T”, and FIG. 13 is a diagram showing the result of image capture of the dotted line in FIG. As shown in the figure, at the time of the first image capture, only the pixel P7 becomes “H”. Therefore, at this time, only the pixel P7 is determined to be black, and a black pixel value is assigned to the pixel P7.
[0045]
Next, when the second image capture is performed, the pixel P9 is newly set to “H”. Accordingly, the average value of the previous pixel values of the surrounding eight pixels (in this case, all white pixel values) is set as the pixel value of the pixel P9.
[0046]
Next, when the third image capture is performed, the pixel P4 newly becomes “H”. Therefore, the average value of the previous pixel values of the surrounding eight pixels (in this case, all white pixel values) is set as the pixel value of the pixel P4.
[0047]
Next, when the fourth image capture is performed, all the pixels P1 to P15 become “H”. Accordingly, the pixel values of all the remaining pixels P1 to P3, P5, P6, P8, P9, and P11 to P15 are determined based on the average value of the previous pixel values of the surrounding eight pixels.
[0048]
When the process shown in FIG. 8 is performed on all the lines shown in FIG. 12 by such a method, an image as shown in FIG. 14 is finally obtained. As can be seen from FIG. 14, noise during image capture can be removed, and even intermediate colors can be reproduced.
[0049]
In the present embodiment, as shown in the flowchart of FIG. 8, image capturing is performed a plurality of times (the greater the number of times, the higher the accuracy of image reproduction) by changing the shooting conditions, and based on the image capturing results of each time. In order to determine the final captured image, it is necessary to store the image capture results of each time. For example, as shown in FIG. 15, if the result of each image capture is stored in the main memory 44, the required memory capacity increases. Considering the application to small information terminals such as mobile phones, which require strong downsizing of the entire set, arithmetic processing that can be performed with limited calculation resources is desirable. An example of the calculation resource is a memory for holding data for the CPU to perform calculations.
[0050]
For this reason, in this embodiment, a buffer memory 45 is provided in the image capture control unit 42, the image capture result for one time is stored in the buffer memory 45, and the image capture result is transferred to the main memory 44. . The CPU 43 performs one process using the data in the main memory 44 according to the flowchart of FIG. 8 and stores the processing result in another storage area of the main memory 44. Meanwhile, the buffer memory 45 stores the next image capture result. Thereafter, by repeating the same operation, a final captured image is obtained.
[0051]
In this case, as shown in FIG. 16, the main memory 44 stores only one image capture result, so the capacity of the main memory 44 can be reduced.
[0052]
As described above, in this embodiment, the final captured image is determined based on the result of capturing images a plurality of times while changing the shooting conditions. Therefore, variations in the characteristics of the sensors 12a and 12b and the threshold value of the SRAM are determined. An image can be captured without being affected by voltage variations and the like, and a captured image with less noise and reproducible up to a halftone can be obtained.
[0053]
In the above-described embodiment, the example in which the voltage applied to the capacitor C3 is changed as the plurality of shooting conditions has been described. Instead of changing the voltage applied to the capacitor C3, the time for capturing an image is changed for each shooting condition. Also good. Alternatively, the transmittance of the liquid crystal may be changed. Although the specific example of the variation of conditions was shown in FIG. 9, the variation with the same meaning is possible for others.
[0054]
In addition, the voltage applied to the capacitor C3 may be changed and the time for image capture may be changed. In this case, the number of shooting conditions can be increased.
[0055]
(Second Embodiment)
The density of the imaging target is not necessarily uniform, and the density of black varies depending on the location. For example, when the characters “Toshiba Matsushita Display” in FIG. 17A are imaged by a sensor, an imaging result as shown in FIG. 17B is obtained. As shown in the figure, the character “East” has a higher black density than other characters, and is therefore crushed in black. On the other hand, the letters “L” and “I” are almost black because the density of black is low.
[0056]
In this way, high density black characters are crushed black because it is difficult for multiple reflected light to enter the interface of the surrounding white paper surface / glass substrate. This is because the multiple reflected light is incident more and the black line width becomes narrower.
[0057]
In view of this, the second embodiment described below is characterized in that image capture is performed taking into account partial black density variations of the imaging target. At this time, unlike a simple sensor array, the fact that it is integrated in a display device is actively used to compensate for variations in characteristics of sensors and the like by adjusting the luminance of each pixel.
[0058]
FIG. 18 is a block diagram showing the overall configuration of the second embodiment of the display device according to the present invention. The display device of FIG. 18 includes a glass substrate 31 on which a pixel array unit 1 and a part of a drive circuit are formed, and a control circuit substrate 52 connected to the glass substrate 31 with a flexible cable (FPC) 51. .
[0059]
On the glass substrate 31, the pixel array unit 1 in which the pixel TFT 11 and the image reading sensor 12 are arranged, a signal line driving circuit 2 that drives a signal line, a scanning line driving circuit 3 that drives a scanning line, A sensor control circuit 53 that controls the sensor 12 and a signal processing output circuit 54 that outputs the imaging result of the sensor 12 are formed. Each circuit on the glass substrate 31 is formed by, for example, polysilicon TFT.
[0060]
On the control circuit board 52, a control IC 55 for controlling each circuit on the glass substrate 31, a memory 56 for storing image data and the like, and various DC voltages used in the glass substrate 31 and the control circuit board 52 are output. A power supply circuit 57 is mounted. A CPU may be provided separately from the control IC 55, the memory 56 and the power supply circuit 57 may be integrated with the control IC 55, or discrete components may be mounted on the control circuit board 52.
[0061]
FIG. 19 shows a connection relationship between the signal line drive circuit 2, the scanning line drive circuit 3, the sensor control circuit 53, and the signal processing output circuit 54 on the glass substrate 31, and the control IC 55 and the memory 56 on the control circuit board 52. It is a block diagram.
[0062]
As shown in the drawing, a display control unit 41, an image capture control unit 42, and a CPU are built in the control IC 55. The display control unit 41 transmits digital pixel data and control signals such as a synchronization signal and a clock signal to the signal line driving circuit 2 and the scanning line driving circuit 3. The image capture control unit 42 transmits a control signal to the sensor control circuit 53 and the signal processing output circuit 54, and synchronizes the imaging data from the signal processing output circuit 54 and the position of the imaging data as necessary. Receive a signal. A buffer memory 47 and a control signal generation circuit 48 are provided inside the display control unit 41, and a buffer memory 45 and a control signal generation circuit 46 are also provided inside the image capture control unit 42.
[0063]
FIG. 20 is a block diagram showing a detailed configuration of the glass substrate 31. The pixel array unit 1 of the present embodiment has a display resolution of 320 pixels in the horizontal direction × 240 pixels in the vertical direction. The pixel is divided into red, blue, and green portions in the horizontal direction, and a signal line is provided for each. The total number of signal lines is 320 × 3 = 960, and the total number of scanning lines is 240.
[0064]
The scanning line driving circuit 3 includes a 240-stage shift register 61, a SHUT (malfunction prevention circuit) 62, a level shifter 63, a multiplexer (MUX circuit) 64, and a buffer 65.
[0065]
The signal processing output circuit 54 includes 960 precharge circuits 66, a 3-select decoder 67, a 320-stage shift register 68, and 8 output buffers 69.
[0066]
FIG. 21 is a circuit diagram showing a detailed configuration of the scanning line driving circuit 3. The characteristic part in FIG. 21 is that a MUX circuit 64 is provided after the level shifter 63. The MUX circuit 64 switches whether the scanning lines are turned on line by line or all the scanning lines are turned on simultaneously. All the scanning lines are simultaneously turned on in order to accumulate initial charges in the capacitor C3 that stores the imaging result of the sensor 12.
[0067]
Thus, by providing the MUX circuit 64, a dedicated TFT for switching whether or not to store the initial charge in the capacitor C3 becomes unnecessary, and the circuit scale can be reduced.
[0068]
FIG. 22 is a block diagram showing a detailed configuration of the signal processing output circuit 54. The signal processing output circuit 54 of FIG. 22 includes a synchronization signal generation circuit 71 that outputs a synchronization signal, and eight P / S conversion circuits 72 that convert imaging data supplied from 120 signal lines into one serial data. And an output buffer 73 for buffering the serial data output from each P / S conversion circuit 72, and a counter 74 for detecting the average gradation of the imaging data. Here, the “average gradation” means the average of the gradation of the output data over a plurality of pixels. When an image with 256 gradations is finally formed, if 5 out of 10 pixels are white and the remaining 5 pixels are black, the average gradation is 256 [gradation] x 5 [pixel] / 10 [ Pixel] = 128 [gradation].
[0069]
FIG. 23 is a block diagram showing a detailed configuration of the synchronization signal generating circuit 71. As shown in FIG. 23 includes a NAND gate 75 and a clock-controlled D type F / F 76, and an output buffer 73 is connected to the subsequent stage of the D type F / F 76. Only a combinational circuit such as a NAND gate formed on an insulating substrate may cause a phase variation with respect to output data due to variations in TFT characteristics, and may not serve as a synchronization signal. Therefore, it is preferable to reduce the phase difference from the clock on the insulating substrate by providing a D-type F / F controlled by the clock on the insulating substrate as shown in FIG.
[0070]
FIG. 24 is a block diagram showing a detailed configuration of the P / S conversion circuit 72. The P / S conversion circuit 72 in FIG. 24 includes a 3-input 1-output decoder 77, a latch 78, and a 40-stage shift register 79. The decoder 77 includes a circuit as shown in FIG. The latch 78 includes a circuit as shown in FIG. The clock used for the control of the shift register is made common with the clock used for the control of the D-type F / F in FIG.
[0071]
FIG. 27 is a block diagram showing a detailed configuration of the output buffer 73. As shown, a plurality of buffers (inverters) 80 are connected in cascade. The downstream buffer increases the TFT channel width of each inverter and secures the necessary external load (flexible cable (FPC) 51, etc.) driving force.
[0072]
FIG. 28 is a detailed circuit diagram for one pixel of the pixel array section 1, and FIG. 29 is a layout diagram for one pixel on the glass substrate 31. As shown in the figure, one pixel is composed of three RGB subpixels 81r, 81g, and 81b, and each subpixel has a pixel TFT11 and a display control TFT82 that controls whether or not electric charge is accumulated in the auxiliary capacitor Cs. The image capturing sensor 12, the capacitor C3 for storing the imaging result of the sensor 12, the SRAM 83 for storing binary data corresponding to the stored charge of the capacitor C3, and the initialization for storing the initial charge in the capacitor C3 It has a TFT 84 and a data holding TFT 85 of the SRAM 83. Here, the luminance of each pixel is gradation controlled by the difference between the pixel electrode potential determined based on the charge accumulated in the auxiliary capacitor Cs and the potential of the common electrode formed on the counter substrate.
[0073]
When the capacitor C3 is initialized, the pixel TFT 11 and the initialization TFT 84 are turned on. When an analog voltage (analog pixel data) for setting the luminance of the display element is written to the auxiliary capacitor Cs, the pixel TFT 11 and the display control TFT 82 are turned on. When data retention (refresh) of the SRAM 83 is performed, the initialization TFT 84 and the data retention TFT 84 are turned on. When the imaging data stored in the SRAM 83 is supplied to the signal line, the pixel TFT 11 and the data holding TFT 85 are turned on.
[0074]
FIG. 30 is a diagram for explaining the operation of the display device of this embodiment. When performing normal display, the operation in mode m1 is performed. On the other hand, when the image is captured by the sensor 12, first, the operation of the mode m1 is performed to set the luminance of all pixels to a predetermined value. Next, in the mode m2, the capacitors C3 of all the pixels are precharged (initial charge accumulation). Next, the red component image for one screen is captured in mode m3. Next, the image of the green component for one screen is taken in mode m4. Finally, the image of the blue component for one screen is captured in mode m5.
[0075]
31 to 33 are operation timing charts of modes m1 to m5. Hereinafter, the operation timings of the modes m1 to m5 will be described in order with reference to these drawings.
[0076]
In the mode m1, as shown at times t1 to t2 in FIG. 31, the scanning line driving circuit 3 sequentially drives the scanning lines, and the signal line driving circuit 2 applies analog signals to the signal lines for each horizontal line in accordance with the timing. Pixel data is displayed by supplying pixel data. The analog pixel data is obtained by converting digital pixel data output from the control IC 55 by a known D / A conversion circuit. The D / A converter circuit is formed as a signal line driver circuit on a glass substrate by a known technique (Japanese Patent Laid-Open No. 2000-305535, etc.).
[0077]
In mode m2, as shown at time t3 in FIG. 32, the scanning line driving circuit 3 drives all the scanning lines at the same timing. A precharge voltage (5 V) is accumulated in the sensor capacitor C3 of all pixels. At time t4, both the initialization TFT 84 and the data holding TFT 85 are turned on, and the SRAM 83 performs a refresh operation. Even if the precharge of the sensor capacitor C3 is not completed between the time t3 and the time t4, the precharge voltage of all the sensor capacitors C3 is made equal to the power supply voltage (5V) of the SRAM by the refresh operation of the SRAM 83.
[0078]
In mode m3, as shown at times t5 to t6 in FIG. 32, red component imaging data is supplied to the signal line for each horizontal line. The red component imaging data supplied to each signal line is converted into serial data by a P / S conversion circuit 72 shown in FIG. 22, and is output to the outside through eight data lines.
[0079]
In mode m4, as shown at time t7 in FIG. 33, green imaging data is supplied to the signal line for each horizontal line. In mode m5, as shown at time t8 in FIG. 33, blue imaging data is supplied to the signal line for each horizontal line.
[0080]
The control IC 55 in FIG. 18 divides the display area of 320 pixels × 240 pixels into block units of 40 pixels × 30 pixels as shown in FIG. 34 (8 blocks in the horizontal direction × 8 blocks in the vertical direction are formed). The image is taken by the sensor 12 with the display brightness set individually for each block. This is one of the features of the present invention. Unlike conventional CMOS image sensors, not only sensors are formed, but also the brightness control means for each pixel is actively used during imaging to compensate for in-plane variations in sensors and TFT characteristics, and Higher quality (ensure uniformity, etc.) can be achieved.
[0081]
FIG. 35 is a flowchart showing an example of processing operations performed by the control IC 55 of FIG. First, while performing pixel display of each block so that each block has a predetermined reference luminance (for example, 80% of the maximum luminance), imaging is performed a plurality of times while changing the imaging time (step S21). Here, for example, imaging is performed nine times (trial imaging) while switching the imaging time from 10 msec to 50 msec in units of 5 msec.
[0082]
Next, based on the trial imaging results (by interpolation of the graph of imaging time vs. average gradation), the average gradation in the block is approximately the median value (in the case of 256 gradations, 128 gradations for each gradation). Imaging time t (m, n) such that (value) is obtained (step S22). t (m, n) varies depending on variations in sensor leakage current, TFT characteristics, light reflection characteristics of the imaging target, color of the imaging target, density distribution of lines such as graphics and characters, and the like.
[0083]
As shown in FIG. 36, since the average gradation changes greatly by changing the imaging time, the optimum imaging time is obtained for each block in step S22 described above.
[0084]
Next, it is determined for each block whether or not the obtained imaging time t (m, n) is less than a reference time (for example, 30 msec) (step S23). For blocks less than the reference time, the display luminance Y is set lower than the reference luminance (for example, 80% of the maximum luminance) (step S24), and for blocks longer than the reference time, the display luminance Y is set to be equal to or higher than the reference luminance ( Step S25). That is, the variation in the optimum imaging time is compensated by the brightness of the display element (the amount of light irradiated to the imaging target). Such a compensation method is not an extension of the technology of the conventional CMOS image sensor. Note that changing the imaging time for each block is not practical because of complicated control.
[0085]
More specifically, for example, the display luminance Y of each block is set based on the following equation (1). However, m represents a row and n represents a column.
[0086]
[Expression 1]

When the calculation result of (1) exceeds 100%, Y (m, n) = 100%. Expression (1) is an example, and should be modified according to the characteristics of the display element, the characteristics of the sensor, and the characteristics of the imaging target. The point is that the display brightness can guarantee the variation in the optimum imaging time.
[0087]
Next, 81 images are taken from the imaging time of 10 msec to 50 msec while changing the imaging time in increments of 0.5 msec, for example (step S26; main imaging).
[0088]
Next, the average value of the 81 imaging results is calculated to obtain final imaging data (step S27). For example, FIG. 37 shows an example of the imaging data obtained in step S27. At this time, instead of performing imaging while changing the imaging time at the same interval, weighting such as performing imaging in the vicinity of 30 msec as shown in FIG. 37 and FIG. 38 is performed, for example, the number of times less than 81 The average value may be calculated using the imaging result. Overall processing time can be reduced. Alternatively, each imaging result may be weighted when calculating the average value.
[0089]
By performing the processing as shown in FIG. 35, even if there is a variation in black density in the imaging target as shown in FIG. 34, an excellent imaging result without partial blackout or blurring is obtained as shown in FIG. can get.
[0090]
As described above, in the second embodiment, the black density of the imaging target is checked in advance (by trial imaging), and imaging is performed by the sensor 12 in a state where the display luminance suitable for the imaging target is set in units of blocks. Thus, a captured image with uniform image quality without partial blackout or blurring can be obtained.
[0091]
In the present embodiment, the method for solving the blackout of the imaging result due to the uneven distribution of the black density of the imaging target has been described. However, the imaging result due to the unevenness of the leakage current of the sensor, the unevenness of the TFT characteristic variation, or the like. The same effect can be obtained as a means for solving the image quality degradation.
[0092]
Further, the main imaging operation after the trial imaging may be performed in the following procedure.
(R1) The mode m1 in FIG. 30 is performed, and the liquid crystal display is entirely red. However, the gradation is changed for each block based on trial imaging.
(R2) The mode m2 is performed, and all sensor capacitors are precharged.
(R3) Modes m3, m4, and m5 are performed, and imaging data of pixels belonging to the red signal line, the green signal line, and the blue signal line are also output.
(G1) The mode m1 is performed, and the entire liquid crystal display is displayed in green. However, the gradation is changed for each block based on trial imaging.
(G2) The mode m2 is performed, and all sensor capacitors are precharged.
(G3) Modes m3, m4, and m5 are performed, and imaging data of pixels belonging to the red signal line, the green signal line, and the blue signal line are also output.
(B1) The mode m1 is performed, and the liquid crystal display is entirely blue. However, the gradation is changed for each block based on trial imaging.
(B2) The mode m2 is performed, and all sensor capacitors are precharged.
(B3) The modes m3, m4, and m5 are performed, and imaging data of pixels belonging to the red signal line, the green signal line, and the blue signal line are also output.
[0093]
At first glance, it is easy to assume that the sensor data of the green pixel and the blue pixel is meaningless when imaging with the display being entirely red. Especially when the light source (backlight provided on the back side of the liquid crystal layer is diffused light and the image pickup surface is irradiated with light from various angles, and as shown in FIG. Unlike the conventional contact CMOS image sensor, the sensor cannot be in close contact with the object to be imaged, and is about d (glass substrate thickness + optical film thickness such as polarizing plate) (0.2 to 0.7). The backlight light is emitted only from the red pixel, but the reflected light from the imaging target is appropriately diffused, the diffusion range is about d, and the pixel pitch is about d or smaller. Light based on the imaging target also enters the pixel and blue pixel sensors, and by performing the above-described processes (R1) to (B3), a higher-resolution imaging result can be obtained in the horizontal direction. Change Portion averaging process is repeated a plurality of times will be omitted because it is similar to the embodiment described in detail.
[0094]
Focusing on the fact that it is only necessary to know the average gradation for each block in the trial imaging, in the trial imaging, only the result of counting the average gradation by the counter may be output for each block. Power consumption for driving the external load by stopping the eight data output circuits can be saved.
[0095]
In this embodiment, an SRAM is provided in a pixel, and (1) a weak current of the sensor is amplified and (2) data is held until data is output after imaging. However, the present invention is not limited to the SRAM. The current amplification of (1) may be performed by the source follower. In the case where data cannot be held due to a leak or the like until data is output after imaging, a control circuit and sequence that outputs data immediately after imaging may be prepared.
[0096]
In addition, an example of adjusting the display brightness at the time of imaging for each block by dividing the display screen has been described, but in the case of a display device with a relatively small diagonal screen size (for mobile phones of about 2 inches or less), When the variation in the chip of the characteristics of the sensor or TFT is small, it may be applied without dividing the screen (the number of divisions = 1). In that case, the trial imaging and the main imaging may be performed as follows without separating them.
[0097]
That is, (1) First, when the imaging time is t0 = 10 msec (the time that almost no light leaks to the sensor for any imaging target and a blackened screen is obtained), the first imaging data output and the average gradation count I do. The first imaging data is stored in a memory such as the control IC 55.
[0098]
(2) The second imaging is performed with t = t0 + Δt (for example, Δt = 0.5 msec), and the average gradation is counted.
[0099]
(3) If the average gradation is less than the predetermined value, imaging and counting of the average gradation are performed with the imaging time t = t + 2Δt without outputting the imaging data.
[0100]
(4) If the counting result is equal to or greater than a predetermined value, the imaging data is output and added to a memory such as the control IC 55.
[0101]
(5) Repeat (2) to (4) while appropriately increasing the imaging time t until the average gradation reaches about 256 gradations.
[0102]
The image completed on the memory according to the above (1) to (5) can be considered as a high-quality imaging result equivalent to that obtained in the above-described embodiment. Moreover, only about one frame is required for memory for image calculation. This is especially effective for mobile phones with limited hardware resources.
[0103]
(Third embodiment)
In the case of the second embodiment, since the display brightness is set for each block, in some cases, the display brightness may be significantly different between adjacent blocks.
[0104]
FIG. 40 is a diagram illustrating an example of the display brightness of each block according to the second embodiment. The horizontal axis represents the block position, and the vertical axis represents the display brightness. To make it easier to understand, a block belonging to a specific line was extracted. FIG. 41 is a diagram schematically showing captured images of four adjacent blocks in the second embodiment. As shown in these figures, the display luminance between adjacent blocks changes discontinuously. For this reason, if the luminance difference between adjacent blocks is large, color unevenness as shown in FIG. 41 may occur when an imaging target with a high white density is imaged. For example, although the image should be the same white background, the imaging result may appear to be different in white for each checkered block.
[0105]
Therefore, in the third embodiment, for the central pixel in each block, the display luminance is set by the same processing as in FIG. 35, the display luminance is gradually changed from the central pixel, and the luminance difference near the boundary between adjacent blocks is set. So that it does n’t happen too much.
[0106]
FIG. 42 is a diagram showing a processing result of the third embodiment when imaging is performed under the same conditions as in FIG. 40, and FIG. 43 is a diagram schematically showing captured images of four adjacent blocks in the third embodiment. . As shown in these figures, the luminance difference does not change greatly between adjacent blocks.
[0107]
As described above, in the third embodiment, since the luminance is gradually changed from the central pixel to the periphery of the block and the luminance is not greatly changed between adjacent blocks, the color unevenness of the captured image due to the luminance difference between the blocks. Disappears.
[0108]
(Fourth embodiment)
In the fourth embodiment, the imaging result of the sensor 12 is pattern-matched with a reference pattern prepared in advance.
[0109]
FIG. 44 is a block diagram showing a schematic configuration of the display device according to the fourth embodiment of the present invention. The display device of FIG. 44 includes a reference pattern storage unit 86 that stores a plurality of reference patterns in addition to the configuration of FIG.
[0110]
FIG. 45 shows an example of a reference pattern stored in the reference pattern storage unit. Each of the reference patterns 1a, 1b, 1c, 2a, 2b, 2c, 2d, 3a, 3b, 3c, and 3d has a size of 8 pixels × 8 pixels, and a black portion indicates a pattern. Note that the size and type of the reference pattern are not limited to those illustrated.
[0111]
FIG. 46 is a flowchart showing the processing operation performed by the control IC 55 of this embodiment. Hereinafter, the processing operation of the flowchart of FIG. 46 will be described assuming that imaging data as shown in FIG. 48 is obtained as a result of imaging the imaging target as shown in FIG. 47 by the sensor 12.
[0112]
The control IC 55 of the present embodiment compares the imaged data of the sensor 12 with all the reference patterns stored in the reference pattern storage unit 86 (step S31).
[0113]
The number given to the upper part of each reference pattern in FIG. 45 is the number of inconsistent pixels with the imaging data in FIG. The control IC 55 selects some reference patterns with a small number of mismatched pixels (step S32). For example, assume that the control IC 55 selects the four reference patterns 1a, 1b, 1c, and 1d shown in FIG.
[0114]
Next, the control IC 55 generates a pattern (inverted patterns n1a, n1b, n1c, n1d in FIG. 49) obtained by inverting the brightness of the selected reference pattern (step S33), and this inverted pattern is sequentially applied to the pixel array unit 1. While displaying, the imaging by the sensor 12 is repeated (step S34). In this case, since light passes through only the white portions of the inversion patterns n1a, n1b, n1c, and n1d in FIG. 49, the imaged data of the sensor 12 is as shown in FIG. The imaging data r1a corresponds to the reference pattern 1a and the inverted pattern n1a, the imaging data r1b corresponds to the reference pattern 1b and the inverted pattern n1b, the imaging data r1c corresponds to the reference pattern 1c and the inverted pattern n1c, and the imaging data r1d is the reference It corresponds to the pattern 1d and the inverted pattern n1d.
[0115]
Next, some reference patterns having a smaller number of white pixels are selected from the imaging data (step S35). For example, in the example of FIG. 50, two reference patterns 1a and 1b are selected.
[0116]
Next, a final imaging result is obtained based on the selected reference pattern (step S36). Here, the final imaging result shown in FIG. 51 is obtained by, for example, averaging the selected reference pattern and the first obtained imaging data.
[0117]
As described above, in the fourth embodiment, a plurality of reference patterns are prepared in advance, and the final image data is generated by comparing the image data obtained by the sensor 12 with the reference pattern. Therefore, high quality imaging data can be obtained. In particular, this embodiment is particularly effective when imaging an imaging target whose shape is patterned in advance.
[0118]
In each of the above-described embodiments, the example in which the display device according to the present invention is applied to a liquid crystal display device has been described. However, the present invention is applied to other display devices such as an EL (Electroluminescense) display device and a PDP (Plasma Display Panel). Is also applicable.
[0119]
【The invention's effect】
As described above in detail, according to the present invention, the digital image data of the captured image is generated based on the result of capturing the image under a plurality of shooting conditions, and thus is affected by variations in characteristics of the imaging unit. The image can be captured without any problem, and the quality of the captured image can be improved. In addition, a large amount of calculation resources are not required despite the multiple image captures and calculations.
[0120]
In addition, according to the present invention, since the imaging unit performs imaging with the display brightness of the display element set for each block, even if there is a partial black density variation in the imaging target, there is no blackout or blurring. A high-quality captured image can be obtained.
[0121]
Furthermore, the present invention compares the imaging result by the imaging unit with a reference pattern prepared in advance, and generates final imaging data based on the comparison result, so that imaging can be performed without increasing the resolution of the imaging unit. Imaging data faithful to the object can be obtained.
[Brief description of the drawings]
FIG. 1 is a schematic configuration diagram of a first embodiment of a display device according to the present invention.
FIG. 2 is a block diagram showing a part of the pixel array unit 1 in detail.
FIG. 3 is a circuit diagram showing a part of FIG. 2 in detail.
FIG. 4 is a circuit diagram showing an internal configuration of SRAM.
FIG. 5 is a cross-sectional view of a display device.
6 is a block diagram showing an internal configuration of the logic IC shown in FIG. 1. FIG.
FIG. 7 is a diagram showing an example of switching a voltage applied to a capacitor.
FIG. 8 is a flowchart illustrating an example of processing operation of a CPU.
FIG. 9 is a diagram showing an example of capturing a business card image.
FIG. 10 is a diagram showing an average of eight surrounding pixels.
11 is a diagram showing an example of an image showing the processing result of FIG. 8;
FIG. 12 is a diagram showing an example of an image including a character “T”.
13 is a diagram showing a result of image capture of the dotted line in FIG. 12;
14 is a diagram finally obtained as a result of capturing the image of FIG. 13;
FIG. 15 is a diagram showing an example in which image capture results under each shooting condition are stored separately in the main memory.
FIG. 16 is a diagram showing an example of reducing the capacity of the main memory.
17A is a diagram illustrating an example of an imaging target, and FIG. 17B is a diagram illustrating an example of an imaging result.
FIG. 18 is a block diagram showing an overall configuration of a second embodiment of a display device according to the present invention.
FIG. 19 is a block diagram showing a connection relationship between a signal line driving circuit, a scanning line driving circuit, a sensor control circuit, and a signal processing output circuit on a glass substrate, and a control IC and a memory on the control circuit board.
20 is a block diagram showing an example of a detailed configuration of a glass substrate 31. FIG.
FIG. 21 is a circuit diagram showing an example of a detailed configuration of a scanning line driving circuit 3;
22 is a block diagram showing an example of a detailed configuration of a signal processing output circuit 54. FIG.
23 is a block diagram showing an example of a detailed configuration of a synchronization signal generation circuit 71. FIG.
24 is a block diagram showing an example of a detailed configuration of a P / S conversion circuit 72. FIG.
FIG. 25 is a circuit diagram showing an example of an internal configuration of a decoder.
FIG. 26 is a circuit diagram showing an example of an internal configuration of a latch.
27 is a block diagram showing a detailed configuration of an output buffer 73. FIG.
FIG. 28 is a detailed circuit diagram for one pixel of the pixel array section 1;
29 is a layout diagram for one pixel on a glass substrate 31. FIG.
FIG. 30 is a diagram illustrating an operation of a display device of an embodiment.
FIG. 31 is an operation timing chart of mode m1.
FIG. 32 is an operation timing chart of modes m2 and m3.
FIG. 33 is an operation timing chart of modes m4 and m5.
FIG. 34 is a diagram illustrating block division.
35 is a flowchart showing an example of a processing operation performed by the control IC 55 of FIG.
FIG. 36 is a diagram showing a relationship between imaging time and average gradation.
FIG. 37 is a diagram showing an example of imaging data obtained in step S7.
FIG. 38 is a diagram illustrating average gradation increments.
FIG. 39 is a diagram showing an example of an imaging result of the present embodiment.
FIG. 40 is a diagram showing an example of display luminance of each block in the second embodiment.
FIG. 41 is a diagram schematically showing captured images of four adjacent blocks in the second embodiment.
FIG. 42 is a diagram showing a processing result of the third embodiment when imaging is performed under the same conditions as in FIG.
FIG. 43 is a diagram schematically showing captured images of four adjacent blocks in the third embodiment.
FIG. 44 is a block diagram showing a schematic configuration of a display device according to a fourth embodiment of the present invention.
FIG. 45 is a diagram illustrating an example of a reference pattern stored in a reference pattern storage unit.
FIG. 46 is a flowchart showing the processing operation performed by the control IC 55 of the present embodiment.
FIG. 47 is a diagram showing an example of an imaging target.
FIG. 48 is a diagram showing an example of an imaging result.
FIG. 49 is a diagram showing an example of a reversal pattern.
FIG. 50 is a diagram illustrating an example of imaging data of a sensor.
FIG. 51 is a diagram showing an example of a final imaging result.
[Explanation of symbols]
1 Pixel array section
2 signal line drive circuit
3 Scanning line drive circuit
4 Detection circuit & output circuit
11 pixel TFT
12a, 12b sensor
13 buffers
21 Array substrate
22 pages
23 Backlight
24 Counter substrate
31 glass substrate
32 Semiconductor substrate
33 Logic IC
52 Control circuit board
53 Sensor control circuit
54 Signal processing output circuit
55 Control IC
56 memory
86 Reference pattern storage

Claims (4)

Display elements formed in the vicinity of intersections of signal lines and scanning lines arranged vertically and horizontally;
At least one image sensor corresponding to each of the display elements, each receiving an incident light in a specified range and converting it into an electrical signal;
A charge accumulating unit that accumulates electric charges according to the electrical signal converted by the imaging unit;
A signal processing unit that generates digital image data corresponding to an image captured by the imaging unit based on accumulated charges of the charge storage unit in each of a plurality of imaging conditions;
A binary data generation unit that outputs binary data indicating whether or not the charge accumulated in the charge accumulation unit is equal to or greater than a predetermined threshold;
A shooting condition switching unit that switches the shooting conditions step by step in ascending or descending order,
If the logic of the binary data changes when the shooting condition switching unit switches the shooting condition by one step, the signal processing unit may change the value of the binary data of a plurality of pixels around the target pixel. And generating digital image data of the pixel of interest based on the display . The imaging condition switching unit switches the initial charge amount accumulated in the charge accumulation unit step by step in ascending or descending order,
The charge accumulation unit accumulates a remaining charge obtained by subtracting a charge corresponding to an amount of light received by the imaging unit from the initial charge amount for each of the plurality of initial charge amounts. The display device according to 1.   The display device according to claim 1, wherein the photographing condition switching unit switches the imaging time in the imaging unit in ascending order or descending order in a stepwise manner.   If the logic of the binary data changes when the imaging condition switching unit switches the initial charge amount by one step, the signal processing unit is configured to capture the previous image of eight pixels around the target pixel. The display device according to claim 1, wherein an average value of the binary data is digital image data of the pixel of interest.

Images