JP2004159273A - Display device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To capture an image without being affected by a fluctuation in optical leak or a fluctuation in the electrical characteristics of a transistor or the like. <P>SOLUTION: A display device is composed of a glass substrate 31 and a semiconductor substrate 32 and on the semiconductor substrate 32, a logic IC 33 is packaged for performing display control and image capture control. The logic IC 33 has a display control part 41 for controlling display onto a pixel array part 1, an image capture control part 42 for controlling image capturing by sensors 12a, 12b, a CPU 43 for controlling the entire logic IC 33, and a main memory 44 to be utilized for work by the CPU 43. Based upon the result of capturing images a plurality of times while changing image pickup conditions, a final captured image is determined. Thus, the image is captured without being affected by a fluctuation in the characteristics of the sensors 12a, 12b or a fluctuation in a threshold voltage of an SRAM, and a captured image which has little noise and is reproducible to a halftone is obtained. <P>COPYRIGHT: (C)2004,JPO

Description

（１）の計算結果が１００％を超える場合には、Ｙ（ｍ，ｎ） ＝ １００％とする。なお、式（１）は例であって、表示素子の特性、センサの特性、撮像対象の特性に応じて変形すべきである。要は最適撮像時間のばらつきを表示輝度で保証するようなものであればよい。
【００８７】
次に、撮像時間を例えば０．５ｍｓｅｃ刻みで変えながら、撮像時間１０ｍｓｅｃ〜５０ｍｓｅｃまで８１枚の撮像を行う（ステップＳ２６；本撮像）。
【００８８】
次に、８１枚の撮像結果の平均値を計算し、最終的な撮像データを得る（ステップＳ２７）。例えば、図３７はステップＳ２７で得られる撮像データの一例を示している。このとき、撮像時間を同じ間隔で変えながら撮像を行うのではなく、図３７及び図３８に示すように３０ｍｓｅｃ付近の撮像を多めに行うなどの重み付けを行って、例えば、８１枚より少ない回数の撮像結果を用いて平均値を計算してもよい。全体の処理時間を短縮できる。あるいは、平均値を計算する際に、各撮像結果に重み付けをしてもよい。
【００８９】
図３５のような処理を行うことで、図３４に示すように撮像対象に黒密度のばらつきがあっても、図３９に示すように、部分的な黒つぶれやかすれのない良好な撮像結果が得られる。
【００９０】
このように、第２の実施形態では、撮像対象の黒密度を事前に（試行撮像によって）調べて、撮像対象に適した表示輝度をブロック単位で設定した状態で、センサ１２による撮像を行うため、部分的な黒つぶれやかすれのない、均一な画質の撮像画像が得られる。
【００９１】
本実施形態では、撮像対象の黒密度の分布の偏りに起因する撮像結果の黒つぶれを解決する手法を説明したが、センサのリーク電流の偏りやＴＦＴの特性ばらつきの偏り等に起因する撮像結果の画質劣化の解決手段としても、同様の効果を奏する。
【００９２】
また、試行撮像の後の本撮像の動作を次のような手順で行ってもよい。
（Ｒ１）図３０のモードｍ１を行い、液晶の表示を全面赤表示にする。但し試行撮像に基づいて各ブロックごとに階調は変化させる。
（Ｒ２）モードｍ２を行い、全てのセンサ容量のプリチャージを行う。
（Ｒ３）モードｍ３，ｍ４，ｍ５を行い、赤信号線、緑信号線及び青信号線に属すする画素の撮像データも出力する。
（Ｇ１）モードｍ１を行い、液晶の表示を全面緑表示にする。但し試行撮像に基づいて各ブロックごとに階調は変化させる。
（Ｇ２）モードｍ２を行い、全てのセンサ容量のプリチャージを行う。
（Ｇ３）モードｍ３，ｍ４，ｍ５を行い、赤信号線、緑信号線及び青信号線に属すする画素の撮像データも出力する。
（Ｂ１）モードｍ１を行い、液晶の表示を全面青表示とする。但し試行撮像に基づいて各ブロックごとに階調は変化させる。
（Ｂ２）モードｍ２を行い、全てのセンサ容量のプリチャージを行う。
（Ｂ３）モードｍ３，ｍ４，ｍ５を行い、赤信号線、緑信号線及び青信号線に属すする画素の撮像データも出力する。
【００９３】
一見、表示を全面赤として撮像を行う際、緑画素及び青画素のセンサのデータは無意味と思い込みやすいがそうではない。とくに光源（液晶層の背面に設けるバックライトの光が拡散光であり撮像面にいろいろな角度から光が照射されるような場合で、かつ、図５のように液晶セルの観察面側にセンサアレイが配置される構成の場合に意義がある。従来の密着ＣＭＯＳイメージセンサと異なり、センサは撮像対象と密着できず、ｄ（ガラス基板厚＋偏光板などの光学フィルム厚）程度（０．２乃至０．７ｍｍ）は離れる。バックライト光は確かに赤画素からのみ発するが、撮像対象での反射光は適当に拡散する。拡散範囲はｄ程度であり画素ピッチはｄ程度かそれより小さい。したがって、緑画素及び青画素のセンサにも撮像対象に基づく光が入射する。上述の（Ｒ１）〜（Ｂ３）の処理を行うことにより、水平方向により高解像度な撮像結果を得ることができる。以下撮像時間を変えながら複数回繰り返して平均化処理する部分は詳述した実施形態と同様なので省略する。
【００９４】
試行撮像では結局、各ブロックごとの平均階調を知れば足りることに着目すると、試行撮像においては、カウンタによる平均階調の計数結果のみブロックごとに出力するように構成してもよい。８個のデータ出力回路を停止して外部負荷駆動するための消費電力を節約することができる。
【００９５】
また、本実施形態ではＳＲＡＭを画素内に設け、▲１▼センサの微弱な電流の増幅、▲２▼撮像後データを出力するまでの間のデータ保持をさせているが、ＳＲＡＭに限定されない。ソースフォロアで▲１▼の電流増幅をしてもよい。撮像後データを出力するまでの間、リークなどのためデータを保持し続けることができない場合は、撮像直後にデータを出力するような制御回路及びシーケンスを用意すればよい。
【００９６】
また、表示画面を区分してブロックごとに撮像時の表示輝度を調節する例を説明したが、対角画面寸法が比較的小さい表示装置（２インチ程度以下の携帯電話向け等）の場合や、センサやＴＦＴの特性のチップ内ばらつきが小さい場合は、画面を分割せずに（分割数＝１として）適用してもよい。その場合さらに、試行撮像と本撮像を分けずに次のように行ってもよい。
【００９７】
すなわち、（１）まず、撮像時間をｔ０＝１０ｍｓｅｃ（どんな撮像対象でも殆どセンサに光リークが生じず、黒つぶれ画面が得られるような時間）で最初の撮像データ出力と、平均階調の計数を行う。最初の撮像データはコントロールＩＣ５５等のメモリに保存する。
【００９８】
（２）ｔ＝ｔ０＋Δｔ（例えばΔｔ＝０．５ｍｓｅｃ）として２回目の撮像をし、平均階調の計数を行う。
【００９９】
（３）平均階調が所定値未満であれば、撮像データを出力することなく、撮像時間をｔ＝ｔ＋２Δｔとして、撮像と平均階調の計数を行う。
【０１００】
（４）計数結果が所定値以上であれば撮像データを出力してコントロールＩＣ５５等のメモリに加算する。
【０１０１】
（５）（２）〜（４）を平均階調が２５６階調程度になるまで撮像時間ｔを適当に増やしながら繰り返す。
【０１０２】
上述した（１）〜（５）によりメモリ上に完成する画像を上述の実施形態で得られるものと同等の高品位な撮像結果と考えることができる。しかも画像演算のためのメモリは１フレーム程度しか要しない。ハード資源に制約のある携帯電話などで特に有効である。
【０１０３】
（第３の実施形態）
第２の実施形態の場合、ブロックごとに表示輝度を設定するため、場合によっては、隣接するブロック同士が著しく表示輝度が異なる場合もありうる。
【０１０４】
図４０は第２の実施形態における各ブロックの表示輝度の一例を示す図であり、横軸はブロックの位置、縦軸は表示輝度を表している。理解がしやすいように、特定の１行に属するブロックを抜き出した。また、図４１は第２の実施形態における隣接する４ブロックの撮像画像を図式化した図である。これらの図に示すように、隣接するブロック同士の表示輝度が不連続に変化している。このため、隣接するブロック同士で輝度差が大きいと、白密度の多い撮像対象を撮像したときに、図４１のような色ムラが起きるおそれがある。たとえば同じ白地のはずなのに、撮像結果は市松状のブロックごとに白さが違って見えてしまうようなことが起こる。
【０１０５】
そこで、第３の実施形態では、各ブロック内の中央画素については、図３５と同様の処理により表示輝度を設定し、中央画素から徐々に表示輝度を変化させ、隣接ブロックの境界付近の輝度差があまり生じないようにする。
【０１０６】
図４２は図４０と同じ条件で撮像を行った場合の第３の実施形態の処理結果を示す図、図４３は第３の実施形態における隣接する４ブロックの撮影画像を図式化した図である。これらの図に示すように、隣接するブロック間で輝度差が大きく変化しなくなる。
【０１０７】
このように、第３の実施形態では、ブロックの中央画素から周辺にかけて徐々に輝度を変化させ、隣接するブロック間で輝度が大きく変化しないようにしたため、ブロック間の輝度差による撮像画像の色ムラがなくなる。
【０１０８】
（第４の実施形態）
第４の実施形態は、センサ１２による撮像結果を、予め用意した基準パターンとパターンマッチングするものである。
【０１０９】
図４４は本発明に係る表示装置の第４の実施形態の概略構成を示すブロック図である。図４４の表示装置は、図１９の構成に加えて、複数の基準パターンを格納する基準パターン格納部８６を備えている。
【０１１０】
図４５は基準パターン格納部に格納されている基準パターンの一例である。各基準パターン１ａ，１ｂ，１ｃ，２ａ，２ｂ，２ｃ，２ｄ，３ａ，３ｂ，３ｃ，３ｄは、８画素×８画素のサイズであり、黒で示した部分がパターンを示している。なお、基準パターンのサイズや種類は図示したものに限定されない。
【０１１１】
図４６は本実施形態のコントロールＩＣ５５が行う処理動作を示すフローチャートである。以下では、仮に図４７のような撮像対象をセンサ１２で撮像した結果、図４８のような撮像データが得られたとして、図４６のフローチャートの処理動作を説明する。
【０１１２】
本実施形態のコントロールＩＣ５５は、センサ１２の撮像データを基準パターン格納部８６に格納されているすべての基準パターンと比較する（ステップＳ３１）。
【０１１３】
図４５の各基準パターンの上部に付された数字は、図４８の撮像データとの不一致画素数である。コントロールＩＣ５５は、不一致画素数の少ない基準パターンをいくつか選択する（ステップＳ３２）。例えば、コントロールＩＣ５５が図４５の４つの基準パターン１ａ，１ｂ，１ｃ，１ｄを選択したとする。
【０１１４】
次に、コントロールＩＣ５５は、選択した基準パターンの明暗を反転させたパターン（図４９の反転パターンｎ１ａ，ｎ１ｂ，ｎ１ｃ，ｎ１ｄ）を生成し（ステップＳ３３）、この反転パターンを画素アレイ部１に順に表示させながら、センサ１２による撮像を繰返し行う（ステップＳ３４）。この場合、図４９の反転パターンｎ１ａ，ｎ１ｂ，ｎ１ｃ，ｎ１ｄの白部分のみ光が透過するため、センサ１２の撮像データは図５０のようになる。撮像データｒ１ａは基準パターン１ａと反転パターンｎ１ａに対応し、撮像データｒ１ｂは基準パターン１ｂと反転パターンｎ１ｂに対応し、撮像データｒ１ｃは基準パターン１ｃと反転パターンｎ１ｃに対応し、撮像データｒ１ｄは基準パターン１ｄと反転パターンｎ１ｄに対応する。
【０１１５】
次に、撮像データの中から、白色画素数がより少ない基準パターンをいくつか選択する（ステップＳ３５）。例えば、図５０の例では、２つの基準パターン１ａ，１ｂを選択する。
【０１１６】
次に、選択した基準パターンに基づいて、最終的な撮像結果を得る（ステップＳ３６）。ここでは、選択した基準パターンと最初に得られた撮像データとを平均化するなどして、図５１に示す最終的な撮像結果を得る。
【０１１７】
このように、第４の実施形態では、予め基準パターンを複数種類用意し、センサ１２による撮像データを基準パターンと比較して最終的な撮像データを生成するため、センサ１２の解像度をあまり高くすることなく、高品質の撮像データが得られる。特に、本実施形態は、形状が予めパターン化されている撮像対象を撮像する場合に特に効果が大きい。
【０１１８】
上述した各実施形態では、本発明に係る表示装置を液晶表示装置に適用した例について説明したが、本発明は、ＥＬ（Ｅｌｅｃｔｒｏｌｕｍｉｎｅｓｃｅｎｓｅ）表示装置やＰＤＰ（Ｐｌａｓｍａ Ｄｉｓｐｌａｙ Ｐａｎｅｌ）などの他の表示装置にも適用可能である。
【０１１９】
【発明の効果】
以上詳細に説明したように、本発明によれば、複数の撮影条件で画像取込みを行った結果に基づいて取込画像のデジタル画像データを生成するため、撮像部の特性ばらつき等の影響を受けることなく画像取込みを行うことができ、取込画像の品質を向上できる。また、複数回の画像取り込みおよび演算を行うにもかかわらず、多くの計算資源を要しない。
【０１２０】
また、本発明によれば、表示素子の表示輝度をブロックごとに設定した状態で撮像部による撮像を行うため、撮像対象に部分的な黒密度のばらつきがあっても、黒つぶれやかすれのない高品質の撮像画像が得られる。
【０１２１】
さらに、本発明は、撮像部による撮像結果を、予め用意した基準パターンと比較して、その比較結果に基づいて最終的な撮像データを生成するため、撮像部の解像度を上げなくても、撮像対象に忠実な撮像データが得られる。
【図面の簡単な説明】
【図１】本発明に係る表示装置の第１の実施形態の概略構成図。
【図２】画素アレイ部１の一部を詳細に示したブロック図。
【図３】図２の一部を詳細に示した回路図。
【図４】ＳＲＡＭの内部構成を示す回路図。
【図５】表示装置の断面図。
【図６】図１に示したロジックＩＣの内部構成を示すブロック図。
【図７】キャパシタに印加する電圧を切り替える例を示す図。
【図８】ＣＰＵの処理動作の一例を示すフローチャート。
【図９】名刺の画像を取り込む例を示す図。
【図１０】周囲８画素の平均を取る様子を示す図。
【図１１】図８の処理結果を示す画像の例を示す図。
【図１２】「Ｔ」の文字を含む画像の一例を示す図。
【図１３】図１２の点線行の画像取込みを行った結果を示す図。
【図１４】図１３の画像を取り込んだ結果として最終的に得られる図。
【図１５】各撮影条件での画像取込み結果を別個にメインメモリに格納する例を示す図。
【図１６】メインメモリの容量を削減する例を示す図。
【図１７】（ａ）は撮像対象の一例を示す図、（ｂ）は撮像結果の一例を示す図。
【図１８】本発明に係る表示装置の第２の実施形態の全体構成を示すブロック図。
【図１９】ガラス基板上の信号線駆動回路、走査線駆動回路、センサ制御回路及び信号処理出力回路と、制御回路基板上のコントロールＩＣ及びメモリとの接続関係を示すブロック図。
【図２０】ガラス基板３１の詳細構成の一例を示すブロック図。
【図２１】走査線駆動回路３の詳細構成の一例を示す回路図。
【図２２】信号処理出力回路５４の詳細構成の一例を示すブロック図。
【図２３】同期信号発生回路７１の詳細構成の一例を示すブロック図。
【図２４】Ｐ／Ｓ変換回路７２の詳細構成の一例を示すブロック図。
【図２５】デコーダの内部構成の一例を示す回路図。
【図２６】ラッチの内部構成の一例を示す回路図。
【図２７】出力バッファ７３の詳細構成を示すブロック図。
【図２８】画素アレイ部１の１画素分の詳細回路図。
【図２９】ガラス基板３１上の１画素分のレイアウト図。
【図３０】本実施形態の表示装置の動作を説明する図。
【図３１】モードｍ１の動作タイミング図。
【図３２】モードｍ２，ｍ３の動作タイミング図。
【図３３】モードｍ４，ｍ５の動作タイミング図。
【図３４】ブロック分割を説明する図。
【図３５】図１８のコントロールＩＣ５５が行う処理動作の一例を示すフローチャート。
【図３６】撮像時間と平均階調との関係を示す図。
【図３７】ステップＳ７で得られる撮像データの一例を示す図。
【図３８】平均階調増分を説明する図。
【図３９】本実施形態の撮像結果の一例を示す図。
【図４０】第２の実施形態における各ブロックの表示輝度の一例を示す図。
【図４１】第２の実施形態における隣接する４ブロックの撮像画像を図式化した図。
【図４２】図４０と同じ条件で撮像を行った場合の第３の実施形態の処理結果を示す図。
【図４３】第３の実施形態における隣接する４ブロックの撮影画像を図式化した図。
【図４４】本発明に係る表示装置の第４の実施形態の概略構成を示すブロック図。
【図４５】基準パターン格納部に格納されている基準パターンの一例を示す図。
【図４６】本実施形態のコントロールＩＣ５５が行う処理動作を示すフローチャート。
【図４７】撮像対象の一例を示す図。
【図４８】撮像結果の一例を示す図。
【図４９】反転パターンの一例を示す図。
【図５０】センサの撮像データの一例を示す図。
【図５１】最終的な撮像結果の一例を示す図。
【符号の説明】
１ 画素アレイ部
２ 信号線駆動回路
３ 走査線駆動回路
４ 検出回路＆出力回路
１１ 画素ＴＦＴ
１２ａ，１２ｂ センサ
１３ バッファ
２１ アレイ基板
２２ 紙面
２３ バックライト
２４ 対向基板
３１ ガラス基板
３２ 半導体基板
３３ ロジックＩＣ
５２ 制御回路基板
５３ センサ制御回路
５４ 信号処理出力回路
５５ コントロールＩＣ
５６ メモリ
８６ 基準パターン格納部[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a display device having an image capturing function.
[0002]
The liquid crystal display device includes an array substrate on which signal lines, scanning lines, and pixel TFTs are arranged in rows, and a driving circuit that drives the signal lines and scanning lines. With the recent advancement and development of integrated circuit technology, a process technology for forming a part of a drive circuit on an array substrate has been put to practical use. As a result, the entire liquid crystal display device can be made lighter, thinner and smaller, and is widely used as a display device for various portable devices such as a mobile phone and a notebook computer.
[0003]
Meanwhile, there has been proposed a display device having an image capturing function in which a contact area sensor for capturing an image is arranged on an array substrate (for example, see Patent Documents 1 and 2).
[0004]
Conventional display devices equipped with this type of image capture function change the amount of charge in a capacitor connected to the sensor in accordance with the amount of light received by the sensor, and detect the voltage across the capacitor to capture the image. It is carried out.
[0005]
[Patent Document 1]
JP 2001-292276 A
[Patent Document 2]
JP 2001-339640 A
[0006]
[Problems to be solved by the invention]
However, since the current flowing through the sensor is weak, it is difficult to accurately detect a change in the voltage across the capacitor due to the current, and the measurement error increases. For this reason, noise tends to appear in the captured image.
[0007]
When an SRAM or a buffer circuit is connected to the capacitor to detect the voltage across the capacitor, whether the threshold voltage of a transistor forming the SRAM or the buffer circuit is exceeded is “0” or “1”. Is determined, the thresholds of the transistors vary, and thus the criteria for “0” and “1” may be shifted. In addition, since the current flowing through the sensor also varies, the criterion of “0” and “1” may be shifted.
[0008]
The present invention has been made in view of such a point, and an object of the present invention is to provide a display device capable of capturing an image without being affected by variations in light leakage and variations in electrical characteristics of transistors and the like. Is to provide.
[0009]
[Means for Solving the Problems]
In order to solve the above-described problems, the present invention provides a display element formed near each intersection of signal lines and scanning lines arranged in rows and columns, and at least one display element corresponding to each of the display elements. An imaging unit that receives incident light in a specified range and converts it into an electric signal, a charge storage unit that accumulates a charge corresponding to the electric signal converted by the imaging unit, and a plurality of imaging conditions. And a signal processing unit that generates digital image data corresponding to an image captured by the imaging unit based on the charge stored in the charge storage unit.
[0010]
Further, according to the present invention, at least one display element is provided corresponding to each of the display elements formed near each intersection of signal lines and scanning lines arranged in rows and columns, and each of the display elements is designated. An imaging unit that receives incident light in a range and converts the light into an electric signal; a charge accumulation unit that accumulates a charge corresponding to the electric signal converted by the imaging unit; and a binary corresponding to the accumulated charge of the charge accumulation unit An image data storage unit for storing data and, for each block composed of a plurality of adjacent display elements, detecting an average gradation of an object to be imaged in the block based on binary data stored in the image data storage unit An average gradation detection unit, based on the detection result of the average gradation detection unit, a luminance setting unit that individually sets the display luminance of the display element when performing imaging with the imaging unit for each of the blocks, The imaging An imaging time control unit that controls switching of the imaging time in a plurality of ways, and individual imaging that is switched and controlled by the imaging time control unit with each of the blocks set to the display luminance set by the luminance setting unit. Image data generating means for generating digital image data corresponding to a captured image based on an imaging result of the imaging unit over time.
[0011]
Also, at least one display element is provided in the vicinity of each intersection of the signal lines and the scanning lines arranged in rows and columns, and at least one display element is provided corresponding to each of the display elements, and each of the display elements has an incident light in a designated range. An imaging unit that receives light and converts it into an electric signal; a charge accumulation unit that accumulates electric charges according to the electric signal converted by the imaging unit; and stores binary data corresponding to the accumulated charge of the charge accumulation unit. An imaging data storage unit, a reference pattern storage unit that stores a plurality of reference patterns indicating a display mode of a block including a plurality of display elements used when imaging is performed by the imaging unit, and an imaging result of the imaging unit A reference pattern selection unit for selecting a plurality of types of reference patterns from the reference pattern storage unit that are similar to the reference pattern; While displaying the turn at the display device, provided on the basis of a result of repeated imaging at the imaging unit, an image data generation means for generating a digital image corresponding to the captured image, a.
[0012]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, a display device according to the present invention will be specifically described with reference to the drawings.
FIG. 1 is a schematic configuration diagram of a first embodiment of a display device according to the present invention, which is characterized by having an image capturing function. 1 includes a glass substrate 31 and a semiconductor substrate 32. On a glass substrate 31, a pixel array unit 1 in which signal lines and scanning lines are arranged in a row, a signal line driving circuit 2 for driving signal lines, a scanning line driving circuit 3 for driving scanning lines, and an image And a detection circuit & output circuit 4 for outputting the data. These circuits are formed by, for example, a polysilicon TFT. The signal line driving circuit 2 includes a D / A conversion circuit that converts digital pixel data into an analog voltage suitable for driving a display element. A known D / A conversion circuit is used. A logic IC 33 that performs display control and image capture control is mounted on the semiconductor substrate 32. The glass substrate 31 and the semiconductor substrate 32 transmit and receive various signals via, for example, an FPC.
[0013]
FIG. 2 is a block diagram showing a part of the pixel array unit 1 in detail. The pixel array unit 1 in FIG. 2 includes a pixel TFT 11 formed near each intersection of a signal line and a scanning line arranged vertically and horizontally, a liquid crystal capacitor C1 connected between one end of the pixel TFT 11 and a Cs line, and It has an auxiliary capacitor C2 and two image capture sensors 12a and 12b provided for each pixel TFT 11. The sensors 12a and 12b are connected to a power line and a control line (not shown).
[0014]
FIG. 2 shows an example in which two sensors 12a and 12b are provided for each pixel in order to increase the resolution of image capture, but the number of sensors is not particularly limited.
[0015]
FIG. 3 is a circuit diagram showing a part of FIG. 2 in detail. As shown in FIG. 3, the sensors 12a and 12b have photodiodes D1 and D2 and sensor switching transistors Q1 and Q2, respectively. The photodiodes D1 and D2 output an electric signal according to the amount of received light. The sensor switching transistors Q1 and Q2 alternately select one of the plurality of photodiodes D1 and D2 in one pixel.
[0016]
Each pixel includes two sensors 12a and 12b, a capacitor C3 shared by the two sensors 12a and 12b in the same pixel, a buffer 13 for storing binary data corresponding to the charge stored in the capacitor C3, and a buffer 13 And a reset transistor Q4 for initializing the buffer 13 and the capacitor C3.
[0017]
The buffer 13 is composed of a static RAM (SRAM), for example, as shown in FIG. 4, two inverters IV1 and IV2 connected in series, an output terminal of the succeeding inverter IV2, and an input terminal of the preceding inverter IV1. And an output transistor Q6 connected to the output terminal of the subsequent inverter.
[0018]
When the signal SPOLB is at a high level, the transistor Q5 is turned on, and the two inverters IV1 and IV2 perform a holding operation. When the signal OUTi is at a high level, the held data is output to the detection line.
[0019]
The display device of the present embodiment can perform a normal display operation, and can also perform image capture similar to a scanner. When a normal display operation is performed, the transistor Q3 is turned off, and no valid data is stored in the buffer 13. In this case, a signal line voltage is supplied from the signal line driving circuit 2 to the signal line, and display according to the signal line voltage is performed.
[0020]
On the other hand, when performing image capture, an image capture target (for example, a paper surface) 22 is disposed on the upper surface side of the array substrate 21 as shown in FIG. 5, and light from the backlight 23 is transmitted to the opposite substrate 24 and the array substrate 21. And irradiate the paper surface 22 via. The light reflected on the paper surface 22 is received by the sensors 12a and 12b on the array substrate 21, and an image is captured.
[0021]
The captured image data is stored in the buffer 13 as shown in FIG. 3, and then sent to the logic IC 33 shown in FIG. 1 via a detection line. The logic IC 33 receives digital signals output from the display device of the present embodiment and performs arithmetic processing such as data rearrangement and removal of noise in the data.
[0022]
FIG. 6 is a block diagram showing the internal configuration of the logic IC 33 shown in FIG. As shown in FIG. 6, the logic IC 33 includes a display control unit 41 that controls display on the pixel array unit 1, an image capture control unit 42 that controls image capture of the sensors 12a and 12b, and an entire logic IC 33. It has a CPU 43 that performs control and a main memory 44 that the CPU 43 uses for work.
[0023]
The image capture control unit 42 has a buffer memory 45 for temporarily storing image data on the detection lines in FIG. 3 and a control signal generation circuit 46 for generating a control signal for image capture. The CPU 43 performs image processing of the captured image based on the image data stored in the buffer memory to generate image data for display.
[0024]
The display control unit 41 generates a buffer memory 47 for temporarily storing image data for display generated by the CPU 43 and a control signal for controlling operation timing of the signal line driving circuit 2 and the scanning line driving circuit 3 in FIG. And a control signal generating circuit 48.
[0025]
When an image is captured, an initial charge is stored in the capacitor C3 of each pixel in advance. When the sensors 12a and 12b capture a whitish image, a current flows through the sensors 12a and 12b, and the charge of the capacitor C3 corresponding to the sensors 12a and 12b is discharged, and the voltage across the capacitor C3 decreases. On the other hand, when the sensors 12a and 12b take in a dark image, little current flows through the sensors 12a and 12b, and the voltage across the capacitor C3 hardly changes.
[0026]
Therefore, by detecting the voltage between both ends of the capacitor C3, the density of the captured image can be determined. In the present embodiment, the voltage between both ends of the capacitor C3 is temporarily stored in the buffer 13 composed of the SRAM. The buffer 13 determines “1” when the voltage across the capacitor C3 is equal to or higher than the threshold voltage of the first inverter of the SRAM, and determines “0” when the voltage is lower than the threshold.
[0027]
However, since the light leak currents due to the sensors 12a and 12b have very small variations, the voltage between both ends of the capacitor C3 tends to vary, and the threshold voltage of the transistors constituting the SRAM also varies. Even if it is taken in, it may be determined as “1” or “0” in some cases. Such variations appear as noise in the captured image. Compared to the current variation of photoelectric conversion elements formed on silicon wafers often used in commercial scanners, etc., the photoelectric conversion elements formed on insulating substrates such as glass substrates used for display device Becomes larger. The latter has a large area and a low process temperature (constrained by the heat-resistant temperature of the substrate), so that uniform formation is difficult. Therefore, a dispersion compensation means specific to the display device is required in some form. In addition, it is desirable to be able to reproduce a delicate gradation of the imaging target, but this is hindered by the above-mentioned variation. In the following, a description will be given of a means and configuration capable of reducing noise or reproducing a gray scale display even though the sensor circuit includes a transistor having a large characteristic variation and a photoelectric conversion element having a leakage current variation.
[0028]
The CPU 43 shown in FIG. 6 performs image capturing a plurality of times while changing the imaging conditions of the sensors 12a and 12b, and generates final captured image data based on the results of the image capturing a plurality of times. Specifically, as shown in FIG. 7, a control signal for performing image capture while changing the voltage Vprc applied to the capacitor C3 in four ways and applying each voltage Vprc to the capacitor C3 is transmitted to the glass substrate. To supply. In addition, it performs arithmetic processing of digital data as a result of image capture output from the glass substrate. Since both the signals (digital pixel data, control clock, control signal) input to the glass substrate and the signals output from the glass substrate are digital signals (digital signals based on imaging results), the display control unit and the image shown in FIG. The capture control unit can be easily formed on a one-chip semiconductor. If there is no D / A conversion circuit on the glass substrate, an amplification circuit (analog circuit) is required for the display control unit, and the one-chip configuration is more advantageous than an increase in cost. In addition, with the recent progress in miniaturization and improvement in the degree of integration of the semiconductor manufacturing process, the CPU and the main memory in FIG. 6 can be easily integrated into a single chip with the display control unit and the image capture control unit.
[0029]
FIG. 8 is a flowchart showing an example of the processing operation of the CPU 43. First, the CPU 43 applies a voltage Vprc = 3.5 V (a value relatively close to the threshold value of the first-stage inverter of the SRAM. The threshold value of the inverter is half of the power supply voltage (5 V)) at one end of the capacitor C3 of FIG. (Variation about 2.5 V) is applied, and the initial charge is stored in the capacitor C3 (step S1).
[0030]
Next, the first image capture is performed (step S2). In this case, a current flows through the sensors 12a and 12b which read a white portion of the image or gray close to white, the initial charge of the capacitor C3 is discharged, and the voltage across the capacitor C3 decreases. On the other hand, since no current flows through the sensors 12a and 12b that read the black portion of the image, the voltage across the capacitor C3 hardly changes.
[0031]
In step S2, when the voltage across the capacitor C3 is higher than the threshold voltage of the first inverter of the SRAM, it is determined that the pixel is black. That is, first, only the black portion in the captured image is extracted in step S2, the extracted pixels are determined as black pixel values, and the other pixels are stored in the main memory 44 as white pixel values. (Step S3). Since the precharge voltage of the capacitor C3 is relatively close to the threshold voltage of the first-stage inverter of the SRAM, some leakage current occurs if the image portion facing the sensor section is somewhat white, and the capacitor C3 Is more likely to be lower than the threshold voltage of the first-stage inverter of the SRAM. Conversely, in this state, the fact that the voltage of C3 continues to be higher than the threshold voltage of the inverter means that the corresponding image portion can be definitely determined to be black.
[0032]
For example, FIG. 9 shows an example in which an image of a business card (black letters on a white background) is captured, and FIG. 9A shows the captured images obtained in steps S1 to S3. In step S3, since only the most black pixels are detected as black, an image which is entirely whitish and has slightly blurred characters is obtained as shown in FIG. 9A.
[0033]
Next, a voltage Vprc = 4 V is applied to one end of the capacitor C3 to accumulate initial charges in the capacitor C3 (step S4), and a second image capture is performed (step S5). In this case, even pixels slightly whitish than the first time may be determined as black.
[0034]
When the second image capturing is completed, the pixels determined to be white in the first time and black in the second time are extracted, and the average value of the first pixel values of eight pixels around the extracted pixel is calculated. Is the pixel value of the extracted pixel (step S6).
[0035]
FIG. 9B shows the captured image obtained in steps S4 to S6. Since an image that is slightly whitish than that in FIG. 9A is also determined to be black, an image that is entirely darker than that in FIG. 9A is obtained.
[0036]
In step S6, for example, assuming that a pixel indicated by a hatched portion in FIG. 10 is an extracted pixel, an average value (G1 +... + G8) / 2 of pixel values G1 to G8 of eight pixels around the extracted pixel is defined as a pixel value of the extracted pixel. I do. If G1 to G8 are all white, the pixel value is white, but if some of G1 to G8 are white and black, the pixel value is halftone.
[0037]
Next, a voltage Vprc = 4.5 V is applied to one end of the capacitor C3 to accumulate initial charges in the capacitor C3 (step S7), and a third image capture is performed (step S8). In this case, even pixels slightly whitish than the second time may be determined to be black.
[0038]
FIG. 9C shows the captured image obtained in steps S7 to S9. Since an image that is slightly whitish than that in FIG. 9B is also determined to be black, an image that is entirely darker than that in FIG. 9B is obtained.
[0039]
When the third image capturing is completed, the pixels determined to be white in the second time and black in the third time are extracted, and the average value of the first pixel values of eight pixels around the extracted pixel is calculated. Is the pixel value of the extracted pixel (step S9).
[0040]
Next, a voltage Vprc = 5 V is applied to one end of the capacitor C3 to accumulate initial charges in the capacitor C3 (step S10), and a fourth image capture is performed (step S11). In this case, even pixels slightly whitish than the third time may be determined to be black.
[0041]
FIG. 9D shows the captured image obtained in steps S10 to S12. Since an image that is slightly whitish than that in FIG. 9C is also determined to be black, an image that is entirely darker than that in FIG. 9C is obtained.
[0042]
When the fourth image capture is completed, the pixels determined to be white in the third time and black in the fourth time are extracted, and the average value of the first pixel values of eight pixels around the extracted pixel is calculated. Is the pixel value of the extracted pixel (step S12).
[0043]
The image obtained as a result of performing the processing in step S12 is as shown in FIG. 11, and it can be seen that the image can be expressed up to the halftone and the noise can be removed.
[0044]
FIG. 12 is a diagram illustrating an example of an image including the character “T”, and FIG. 13 is a diagram illustrating a result obtained by capturing an image of a dotted line in FIG. As shown, at the time of the first image capture, only the pixel P7 becomes “H”. Therefore, at this time, only the pixel P7 is determined to be black, and the pixel P7 is assigned a black pixel value.
[0045]
Next, when the second image capturing is performed, the pixel P9 is newly set to “H”. Therefore, the average value of the previous pixel values (all the white pixel values in this case) of the eight surrounding pixels is set as the pixel value of the pixel P9.
[0046]
Next, when the third image capturing is performed, the pixel P4 is newly set to “H”. Therefore, the average value of the previous pixel values (all the white pixel values in this case) of the eight surrounding pixels is set as the pixel value of the pixel P4.
[0047]
Next, when the fourth image capture is performed, all the pixels P1 to P15 become “H”. Therefore, the pixel values of all the remaining pixels P1 to P3, P5, P6, P8, P9, P11 to P15 are determined based on the average value of the previous pixel values of the surrounding eight pixels.
[0048]
When the processing of FIG. 8 is performed on all the lines of FIG. 12 by such a method, an image as shown in FIG. 14 is finally obtained. As can be seen from FIG. 14, it is possible to remove noise at the time of capturing an image and to reproduce even an intermediate color.
[0049]
In the present embodiment, as shown in the flowchart of FIG. 8, image capturing is performed a plurality of times (the greater the number of times, the higher the accuracy of image reproduction is improved), and the image capturing is performed based on the result of each image capturing. In order to determine the final captured image, it is necessary to store the result of each image capture. For example, as shown in FIG. 15, if the results of each image capture are stored in the main memory 44, the required memory capacity increases. Considering application to a small information terminal such as a mobile phone that strongly demands a reduction in the size of the entire set, it is desirable to perform arithmetic processing that can be performed with limited calculation resources. An example of the computational resource is a memory for holding data for the CPU to perform computation.
[0050]
For this reason, in the present embodiment, a buffer memory 45 is provided in the image capture control unit 42, the result of one image capture is stored in the buffer memory 45, and the image capture result is transferred to the main memory 44. . The CPU 43 performs one process using the data of the main memory 44 according to the flowchart of FIG. 8 and stores the processing result in another storage area of the main memory 44. During that time, the buffer memory 45 stores the result of the next image capture. Thereafter, by repeating the same operation, a final captured image is obtained.
[0051]
In this case, as shown in FIG. 16, since only one image capture result is stored in the main memory 44, the capacity of the main memory 44 can be reduced.
[0052]
As described above, in the present embodiment, the final captured image is determined based on the result of performing the image capturing a plurality of times while changing the photographing conditions, and thus the characteristic variation of the sensors 12a and 12b and the threshold value of the SRAM are determined. An image can be captured without being affected by variations in voltage or the like, and a captured image with little noise and which can be reproduced up to a halftone can be obtained.
[0053]
In the above-described embodiment, an example in which the voltage applied to the capacitor C3 is changed as a plurality of shooting conditions has been described. However, instead of changing the voltage applied to the capacitor C3, the time for image capture is changed for each shooting condition. Is also good. Alternatively, the transmittance of the liquid crystal may be changed. Although a specific example of the variation of the condition is shown in FIG. 9, other variations with the same purpose are possible.
[0054]
Further, while changing the voltage applied to the capacitor C3, the time during which the image is captured may be changed. In this case, the number of shooting conditions can be further increased.
[0055]
(Second embodiment)
The density of the imaging target is not always uniform, and the density of black differs depending on the location. For example, when the image of “Toshiba Matsushita Display” in FIG. 17A is captured by the sensor, an imaging result as shown in FIG. 17B is obtained. As shown in the figure, the character "East" has a higher density of black than other characters, and is therefore blackened out. On the other hand, the characters “レ” and “イ” have almost black lines because the density of black is low.
[0056]
The reason why the characters with high black density are blackened is that multiple reflection light at the surrounding white paper surface / glass substrate interface is difficult to enter, and conversely, the lines of the characters with low black density disappear. This is because the multiple reflection light is additionally incident and the black line width is reduced.
[0057]
Therefore, a second embodiment described below is characterized in that an image is captured in consideration of a partial variation in black density of an imaging target. At this time, unlike a mere sensor array, by utilizing the fact that it is integrated into a display device, characteristic variations of sensors and the like are compensated by adjusting the luminance of each pixel.
[0058]
FIG. 18 is a block diagram showing the overall configuration of the second embodiment of the display device according to the present invention. The display device shown in FIG. 18 includes a glass substrate 31 on which the pixel array unit 1 and a part of a driving circuit are formed, and a control circuit substrate 52 connected to the glass substrate 31 by a flexible cable (FPC) 51. .
[0059]
On a glass substrate 31, a pixel array unit 1 in which pixel TFTs 11 and image reading sensors 12 are arranged in a row, a signal line driving circuit 2 for driving signal lines, a scanning line driving circuit 3 for driving scanning lines, A sensor control circuit 53 that controls the sensor 12 and a signal processing output circuit 54 that outputs an imaging result of the sensor 12 are formed. Each circuit on the glass substrate 31 is formed by, for example, a polysilicon TFT.
[0060]
On the control circuit board 52, a control IC 55 for controlling each circuit on the glass substrate 31, a memory 56 for storing image data and the like, and various DC voltages used in the glass substrate 31 and the control circuit board 52 are output. A power supply circuit 57 is mounted. Note that a CPU may be provided separately from the control IC 55, the memory 56 and the power supply circuit 57 may be integrated with the control IC 55, or discrete components may be mounted on the control circuit board 52.
[0061]
FIG. 19 shows a connection relationship between the signal line driving circuit 2, the scanning line driving circuit 3, the sensor control circuit 53, and the signal processing output circuit 54 on the glass substrate 31, and the control IC 55 and the memory 56 on the control circuit substrate 52. It is a block diagram.
[0062]
As illustrated, the display IC 41, the image capture controller 42, and the CPU are built in the control IC 55. The display control unit 41 transmits digital pixel data and control signals such as a synchronization signal and a clock signal to the signal line driving circuit 2 and the scanning line driving circuit 3. The image capture control unit 42 transmits a control signal to the sensor control circuit 53 and the signal processing output circuit 54, and performs synchronization for specifying the position of the imaging data from the signal processing output circuit 54 and, if necessary, the imaging data. Receive the signal. A buffer memory 47 and a control signal generation circuit 48 are provided inside the display control unit 41, and a buffer memory 45 and a control signal generation circuit 46 are also provided inside the image capture control unit 42.
[0063]
FIG. 20 is a block diagram showing a detailed configuration of the glass substrate 31. The pixel array unit 1 of the present embodiment has a display resolution of 320 pixels in the horizontal direction × 240 pixels in the vertical direction. The pixel is divided into red, blue, and green portions in the horizontal direction, and a signal line is provided for each of them. The total number of signal lines is 320 × 3 = 960, and the total number of scanning lines is 240.
[0064]
The scanning line driving circuit 3 includes a 240-stage shift register 61, a SHUT (malfunction prevention circuit) 62, a level shifter 63, a multiplexer (MUX circuit) 64, and a buffer 65.
[0065]
The signal processing output circuit 54 includes 960 precharge circuits 66, a three-selection decoder 67, a 320-stage shift register 68, and eight output buffers 69.
[0066]
FIG. 21 is a circuit diagram showing a detailed configuration of the scanning line driving circuit 3. A characteristic part in FIG. 21 is that a MUX circuit 64 is provided at a stage subsequent to the level shifter 63. The MUX circuit 64 switches between turning on the scanning lines one by one or turning on all the scanning lines simultaneously. The reason why all the scanning lines are turned on at the same time is to store the initial charge in the capacitor C3 that stores the imaging result of the sensor 12.
[0067]
By providing the MUX circuit 64 in this way, a dedicated TFT for switching whether or not to store the initial charge in the capacitor C3 becomes unnecessary, and the circuit scale can be reduced.
[0068]
FIG. 22 is a block diagram showing a detailed configuration of the signal processing output circuit 54. A signal processing output circuit 54 in FIG. 22 includes a synchronization signal generation circuit 71 that outputs a synchronization signal, and eight P / S conversion circuits 72 that convert imaging data supplied from 120 signal lines into one serial data. And an output buffer 73 for buffering the serial data output from each P / S conversion circuit 72, and a counter 74 for detecting the average gradation of the imaging data. Here, “average gradation” refers to an average of gradations of output data over a plurality of pixels. When an image of 256 gradations is finally to be formed, if 5 out of 10 pixels are white and the remaining 5 pixels are black, the average gradation is 256 [gradation] x 5 [pixels] / 10 [ Pixel] = 128 [gradation].
[0069]
FIG. 23 is a block diagram showing a detailed configuration of the synchronization signal generation circuit 71. 23 includes a NAND gate 75 and a clock-controlled D-type F / F 76, and an output buffer 73 is connected to a stage subsequent to the D-type F / F 76. With only a combinational circuit such as a NAND gate formed on an insulating substrate, a phase variation with respect to output data becomes large due to a variation in TFT characteristics, and the function of a synchronization signal may not be achieved. Therefore, it is preferable to provide a D-type F / F controlled by a clock on the insulating substrate as shown in FIG. 23 to reduce the phase difference from the clock on the insulating substrate.
[0070]
FIG. 24 is a block diagram showing a detailed configuration of the P / S conversion circuit 72. The P / S conversion circuit 72 in FIG. 24 includes a decoder 77 having three inputs and one output, a latch 78, and a shift register 79 having 40 stages. The decoder 77 is configured by a circuit as shown in FIG. The latch 78 is configured by a circuit as shown in FIG. By making the clock used for controlling the shift register common to the clock used for controlling the D-type F / F in FIG. 23, the phase difference between the data and the synchronization signal can be reduced.
[0071]
FIG. 27 is a block diagram showing a detailed configuration of the output buffer 73. As shown, a plurality of buffers (inverters) 80 are connected in cascade. The later the buffer, the larger the channel width of the TFT constituting each inverter to secure the necessary external load (such as the flexible cable (FPC) 51) driving force.
[0072]
FIG. 28 is a detailed circuit diagram of one pixel of the pixel array unit 1, and FIG. 29 is a layout diagram of one pixel on the glass substrate 31. As shown in the figure, one pixel includes three sub-pixels 81r, 81g, and 81b of RGB. Each sub-pixel has a pixel TFT 11 and a display control TFT 82 that controls whether or not to accumulate charges in the auxiliary capacitance Cs. , An image capture sensor 12, a capacitor C3 for storing an image pickup result of the sensor 12, an SRAM 83 for storing binary data corresponding to the charge stored in the capacitor C3, and an initialization for storing an initial charge in the capacitor C3. It has a TFT 84 and a data holding TFT 85 of the SRAM 83. Here, the brightness of each pixel is controlled in gradation by the difference between the pixel electrode potential determined based on the charge stored in the auxiliary capacitance Cs and the potential of the common electrode formed on the counter substrate.
[0073]
When the initialization of the capacitor C3 is performed, the pixel TFT 11 and the initialization TFT 84 are turned on. When writing an analog voltage (analog pixel data) for setting the luminance of the display element to the auxiliary capacitance Cs, the pixel TFT 11 and the display control TFT 82 are turned on. When data holding (refreshing) of the SRAM 83 is performed, the initialization TFT 84 and the data holding TFT 84 are turned on. When supplying the image data stored in the SRAM 83 to the signal line, the pixel TFT 11 and the data holding TFT 85 are turned on.
[0074]
FIG. 30 is a diagram for explaining the operation of the display device of the present embodiment. When performing a normal display, the operation in the mode m1 is performed. On the other hand, when an image is captured by the sensor 12, the operation of the mode m1 is performed first, and the brightness of all pixels is set to a predetermined value. Next, in mode m2, the capacitors C3 of all the pixels are precharged (accumulation of initial charges). Next, an image of a red component for one screen is captured in mode m3. Next, an image of a green component for one screen is captured in a mode m4. Finally, an image of a blue component for one screen is captured in the mode m5.
[0075]
FIGS. 31 to 33 are operation timing diagrams of the modes m1 to m5. Hereinafter, the operation timings of the modes m1 to m5 will be sequentially described with reference to these drawings.
[0076]
In the mode m1, as shown from time t1 to time t2 in FIG. 31, the scanning line driving circuit 3 sequentially drives the scanning lines, and in accordance with the timing, the signal line driving circuit 2 applies analog signals to the signal lines for each horizontal line. Pixel data is supplied to perform pixel display. The analog pixel data is obtained by converting digital pixel data output from the control IC 55 by a known D / A conversion circuit. The D / A conversion circuit is formed into a thin film as a signal line driving circuit on a glass substrate by a known technique (Japanese Patent Laid-Open No. 2000-305535).
[0077]
In the mode m2, as shown at time t3 in FIG. 32, the scanning line driving circuit 3 drives all the scanning lines at the same timing. A precharge voltage (5 V) is accumulated in the sensor capacitors C3 of all pixels. At time t4, both the initialization TFT 84 and the data holding TFT 85 are turned on, and the SRAM 83 performs a refresh operation. Even if the precharge of the sensor capacitor C3 is not completed between the time t3 and the time t4, the precharge voltage of all the sensor capacitors C3 is made equal to the power supply voltage (5 V) of the SRAM by the refresh operation of the SRAM 83.
[0078]
In the mode m3, as shown at times t5 to t6 in FIG. 32, the imaging data of the red component is supplied to the signal line for each horizontal line. The image data of the red component supplied to each signal line is converted into serial data by a P / S conversion circuit 72 shown in FIG. 22 and output to the outside through eight data lines.
[0079]
In the mode m4, as shown at time t7 in FIG. 33, green imaging data is supplied to the signal line for each horizontal line. In the mode m5, as shown at time t8 in FIG. 33, blue imaging data is supplied to the signal line for each horizontal line.
[0080]
The control IC 55 in FIG. 18 divides the display area of 320 pixels × 240 pixels into blocks of 40 pixels × 30 pixels as shown in FIG. 34 (eight horizontal blocks × eight vertical blocks are formed). Then, imaging is performed by the sensor 12 in a state where the display luminance is individually set for each block. This is one of the features of the present invention. Unlike a conventional CMOS image sensor, not only a sensor is formed, but also brightness control means for each pixel is positively utilized at the time of imaging, thereby compensating for in-plane variations in sensor and TFT characteristics, and improving the imaging screen. Higher quality (such as ensuring uniformity) can be achieved.
[0081]
FIG. 35 is a flowchart showing an example of the processing operation performed by the control IC 55 of FIG. First, while performing pixel display of each block so that each block has a predetermined reference luminance (for example, 80% of the maximum luminance), the image pickup time is changed and image pickup is performed a plurality of times (step S21). Here, for example, nine times of imaging (trial imaging) are performed while switching the imaging time from 10 msec to 50 msec in units of 5 msec.
[0082]
Next, based on the trial imaging results (by interpolation of a graph of imaging time versus average gray level), for each block, the average gray level in the block is approximately the median value (128 gray levels in the case of 256 gray levels). Is obtained (step S22). t (m, n) varies variously depending on leak current variation of the sensor, variation of TFT characteristics, light reflection characteristics of the imaging target, color of the imaging target, density distribution of lines of figures, characters, and the like.
[0083]
As shown in FIG. 36, by changing the imaging time, the average gradation greatly changes. Therefore, in the above-described step S22, the optimal imaging time is obtained for each block.
[0084]
Next, it is determined for each block whether or not the obtained imaging time t (m, n) is shorter than a reference time (for example, 30 msec) (step S23). The display luminance Y is set lower than the reference luminance (for example, 80% of the maximum luminance) for blocks less than the reference time (step S24), and the display luminance Y is set to the reference luminance or more for blocks longer than the reference time (step S24). Step S25). That is, the optimal variation in the imaging time is compensated for by the luminance of the display element (the amount of light applied to the imaging target). Such a compensation method is not an extension of the conventional CMOS image sensor technology. Changing the imaging time for each block is complicated and impractical.
[0085]
More specifically, for example, the display luminance Y of each block is set based on the following equation (1). Here, m represents a row, and n represents a column.
[0086]
(Equation 1)

If the calculation result of (1) exceeds 100%, Y (m, n) = 100%. Expression (1) is an example, and should be modified according to the characteristics of the display element, the characteristics of the sensor, and the characteristics of the imaging target. In short, any method can be used as long as the variation in the optimal imaging time is guaranteed by the display luminance.
[0087]
Next, while changing the imaging time at intervals of, for example, 0.5 msec, 81 images are captured from the imaging time of 10 msec to 50 msec (step S26; main imaging).
[0088]
Next, an average value of the 81 imaging results is calculated to obtain final imaging data (step S27). For example, FIG. 37 illustrates an example of the imaging data obtained in step S27. At this time, instead of performing the imaging while changing the imaging time at the same interval, weighting is performed such as performing more imaging near 30 msec as shown in FIGS. The average value may be calculated using the imaging result. Overall processing time can be reduced. Alternatively, when calculating the average value, each imaging result may be weighted.
[0089]
By performing the processing as shown in FIG. 35, even if the imaging target has a variation in black density as shown in FIG. 34, as shown in FIG. 39, a good imaging result without partial blackout or blurring can be obtained. can get.
[0090]
As described above, in the second embodiment, the black density of the imaging target is checked in advance (by trial imaging), and the imaging by the sensor 12 is performed in a state where the display luminance suitable for the imaging target is set for each block. Thus, a captured image of uniform image quality without partial blackout or blurring can be obtained.
[0091]
In the present embodiment, the method of solving the blackening of the imaging result due to the uneven distribution of the black density of the imaging target has been described. However, the imaging result caused by the unevenness of the leak current of the sensor, the unevenness of the characteristic variation of the TFT, and the like. The same effect can be obtained as a solution for the image quality deterioration.
[0092]
Further, the operation of the main imaging after the trial imaging may be performed in the following procedure.
(R1) The mode m1 in FIG. 30 is performed, and the display of the liquid crystal is entirely displayed in red. However, the gradation is changed for each block based on the trial imaging.
(R2) Perform mode m2 to precharge all sensor capacitances.
(R3) Modes m3, m4, and m5 are performed, and imaging data of pixels belonging to the red signal line, the green signal line, and the blue signal line are also output.
(G1) The mode m1 is performed so that the display of the liquid crystal is entirely green. However, the gradation is changed for each block based on the trial imaging.
(G2) Perform mode m2 to precharge all sensor capacitances.
(G3) Perform the modes m3, m4, and m5, and also output the imaging data of the pixels belonging to the red signal line, the green signal line, and the blue signal line.
(B1) The mode m1 is performed, and the liquid crystal display is entirely displayed in blue. However, the gradation is changed for each block based on the trial imaging.
(B2) Perform mode m2 to precharge all sensor capacitances.
(B3) The modes m3, m4, and m5 are performed, and the imaging data of the pixels belonging to the red, green, and blue signal lines are also output.
[0093]
At first glance, when imaging is performed with the display being entirely red, it is easy to assume that the sensor data of the green and blue pixels is meaningless, but this is not the case. In particular, in the case where the light from the light source (backlight provided on the back side of the liquid crystal layer is diffused light and the image pickup surface is irradiated with light from various angles, and as shown in FIG. This is significant in the case of a configuration in which an array is arranged.Unlike a conventional contact CMOS image sensor, the sensor cannot adhere to an object to be imaged and has a thickness of about d (glass substrate thickness + optical film thickness such as a polarizing plate) (0.2). The backlight light is emitted only from the red pixels, but the reflected light from the object to be imaged is appropriately diffused, the diffusion range is about d, and the pixel pitch is about d or smaller. Therefore, light based on the object to be imaged is also incident on the sensors of the green and blue pixels, and by performing the above-described processes (R1) to (B3), an imaging result with higher resolution in the horizontal direction can be obtained. Portion averaging process is repeated a plurality of times while changing the following imaging time is omitted because it is similar to the embodiment described in detail.
[0094]
Paying attention to the fact that it is sufficient to know the average gray level of each block in the trial imaging, the trial imaging may be configured to output only the counting result of the average gray level by the counter for each block. Power consumption for driving the external load by stopping the eight data output circuits can be saved.
[0095]
Further, in the present embodiment, an SRAM is provided in the pixel to (1) amplify a weak current of the sensor and (2) retain data until outputting data after imaging, but the present invention is not limited to the SRAM. The current follow-up (1) may be performed by the source follower. If data cannot be held due to leakage or the like until data is output after imaging, a control circuit and a sequence that outputs data immediately after imaging may be prepared.
[0096]
Also, an example has been described in which the display screen is divided and the display brightness at the time of imaging is adjusted for each block. However, in the case of a display device having a relatively small diagonal screen size (such as for a mobile phone of about 2 inches or less), When the variation in the characteristics of the sensor or the TFT in the chip is small, the present invention may be applied without dividing the screen (assuming the number of divisions = 1). In this case, the trial imaging and the main imaging may be performed as follows without separating the imaging.
[0097]
That is, (1) First, the imaging time is set to t0 = 10 msec (light leakage does not almost occur in the sensor for any imaging target and a black-out screen is obtained), and the first imaging data output and the average gradation count are performed. I do. The first imaging data is stored in a memory such as the control IC 55.
[0098]
(2) The second imaging is performed with t = t0 + Δt (for example, Δt = 0.5 msec), and the average gradation is counted.
[0099]
(3) If the average gradation is less than the predetermined value, the imaging and the counting of the average gradation are performed with the imaging time set to t = t + 2Δt without outputting the imaging data.
[0100]
(4) If the counting result is equal to or more than the predetermined value, the imaging data is output and added to the memory such as the control IC 55.
[0101]
(5) Steps (2) to (4) are repeated while appropriately increasing the imaging time t until the average gradation becomes about 256 gradations.
[0102]
An image completed on the memory by the above-described (1) to (5) can be considered as a high-quality imaging result equivalent to that obtained in the above-described embodiment. In addition, only about one frame is required for a memory for image calculation. This is particularly effective for mobile phones with limited hardware resources.
[0103]
(Third embodiment)
In the case of the second embodiment, since the display luminance is set for each block, the display luminance may be significantly different between adjacent blocks in some cases.
[0104]
FIG. 40 is a diagram showing an example of the display luminance of each block in the second embodiment, where the horizontal axis represents the position of the block and the vertical axis represents the display luminance. Blocks belonging to a specific row have been extracted for easy understanding. FIG. 41 is a diagram schematically illustrating captured images of four adjacent blocks in the second embodiment. As shown in these figures, the display luminance of adjacent blocks changes discontinuously. Therefore, if the luminance difference between adjacent blocks is large, color unevenness as shown in FIG. 41 may occur when an imaging target having a high white density is imaged. For example, although the white background should be the same, the imaging result may look different for each checkered block.
[0105]
Therefore, in the third embodiment, for the central pixel in each block, the display luminance is set by the same processing as in FIG. 35, the display luminance is gradually changed from the central pixel, and the luminance difference near the boundary of the adjacent block is set. Should not occur much.
[0106]
FIG. 42 is a diagram showing a processing result of the third embodiment when imaging is performed under the same conditions as in FIG. 40, and FIG. 43 is a diagram schematically showing captured images of four adjacent blocks in the third embodiment. . As shown in these figures, the luminance difference between adjacent blocks does not change significantly.
[0107]
As described above, in the third embodiment, the luminance is gradually changed from the central pixel to the periphery of the block so that the luminance does not largely change between adjacent blocks. Disappears.
[0108]
(Fourth embodiment)
In the fourth embodiment, the imaging result of the sensor 12 is subjected to pattern matching with a reference pattern prepared in advance.
[0109]
FIG. 44 is a block diagram showing a schematic configuration of the fourth embodiment of the display device according to the present invention. The display device in FIG. 44 includes a reference pattern storage unit 86 that stores a plurality of reference patterns in addition to the configuration in FIG.
[0110]
FIG. 45 is an example of a reference pattern stored in the reference pattern storage unit. Each of the reference patterns 1a, 1b, 1c, 2a, 2b, 2c, 2d, 3a, 3b, 3c, 3d has a size of 8 pixels × 8 pixels, and the portion shown in black indicates the pattern. The size and type of the reference pattern are not limited to those illustrated.
[0111]
FIG. 46 is a flowchart showing a processing operation performed by the control IC 55 of the present embodiment. In the following, the processing operation of the flowchart of FIG. 46 will be described assuming that image data as illustrated in FIG. 48 is obtained as a result of imaging the imaging target as illustrated in FIG. 47 by the sensor 12.
[0112]
The control IC 55 of the present embodiment compares the image data of the sensor 12 with all the reference patterns stored in the reference pattern storage unit 86 (Step S31).
[0113]
The number given above each reference pattern in FIG. 45 is the number of pixels that do not match the imaging data in FIG. The control IC 55 selects some reference patterns having a small number of mismatched pixels (step S32). For example, it is assumed that the control IC 55 has selected the four reference patterns 1a, 1b, 1c, 1d in FIG.
[0114]
Next, the control IC 55 generates a pattern (inversion patterns n1a, n1b, n1c, n1d in FIG. 49) in which the selected reference pattern is inverted in brightness (step S33). While displaying the image, the image pickup by the sensor 12 is repeatedly performed (step S34). In this case, since only the white portions of the inverted patterns n1a, n1b, n1c, and n1d in FIG. 49 transmit light, the image data of the sensor 12 is as shown in FIG. The image data r1a corresponds to the reference pattern 1a and the reverse pattern n1a, the image data r1b corresponds to the reference pattern 1b and the reverse pattern n1b, the image data r1c corresponds to the reference pattern 1c and the reverse pattern n1c, and the image data r1d corresponds to the reference pattern. It corresponds to the pattern 1d and the inverted pattern n1d.
[0115]
Next, some reference patterns having a smaller number of white pixels are selected from the imaging data (step S35). For example, in the example of FIG. 50, two reference patterns 1a and 1b are selected.
[0116]
Next, a final imaging result is obtained based on the selected reference pattern (step S36). Here, the final imaging result shown in FIG. 51 is obtained by, for example, averaging the selected reference pattern and the initially obtained imaging data.
[0117]
As described above, in the fourth embodiment, a plurality of types of reference patterns are prepared in advance, and image data obtained by the sensor 12 is compared with the reference patterns to generate final image data. Thus, high-quality image data can be obtained. In particular, the present embodiment is particularly effective when imaging an imaging target whose shape is patterned in advance.
[0118]
In each of the embodiments described above, an example in which the display device according to the present invention is applied to a liquid crystal display device has been described. Is also applicable.
[0119]
【The invention's effect】
As described in detail above, according to the present invention, digital image data of a captured image is generated based on a result of image capturing under a plurality of image capturing conditions, and thus the digital image data is affected by a variation in characteristics of an image capturing unit. It is possible to capture an image without the need to improve the quality of a captured image. Also, despite the fact that image capture and calculation are performed a plurality of times, much computational resources are not required.
[0120]
Further, according to the present invention, imaging is performed by the imaging unit in a state where the display luminance of the display element is set for each block. Therefore, even if there is partial variation in black density in the imaging target, there is no blackout or blurring. High quality captured images can be obtained.
[0121]
Furthermore, the present invention compares the imaging result of the imaging unit with a reference pattern prepared in advance, and generates final imaging data based on the comparison result. Imaging data faithful to the target can be obtained.
[Brief description of the drawings]
FIG. 1 is a schematic configuration diagram of a first embodiment of a display device according to the present invention.
FIG. 2 is a block diagram showing a part of the pixel array unit 1 in detail.
FIG. 3 is a circuit diagram showing a part of FIG. 2 in detail;
FIG. 4 is a circuit diagram showing an internal configuration of an SRAM.
FIG. 5 is a cross-sectional view of a display device.
FIG. 6 is a block diagram showing an internal configuration of the logic IC shown in FIG. 1;
FIG. 7 is a diagram showing an example of switching a voltage applied to a capacitor.
FIG. 8 is a flowchart illustrating an example of a processing operation of a CPU.
FIG. 9 is a diagram illustrating an example of capturing an image of a business card.
FIG. 10 is a diagram illustrating a state in which an average of eight surrounding pixels is obtained.
FIG. 11 is a view showing an example of an image showing the processing result of FIG. 8;
FIG. 12 is a diagram illustrating an example of an image including a character “T”.
FIG. 13 is a diagram showing the result of image capture of the dotted line in FIG. 12;
FIG. 14 is a view finally obtained as a result of capturing the image of FIG. 13;
FIG. 15 is a diagram showing an example in which image capture results under each shooting condition are separately stored in a main memory.
FIG. 16 is a diagram showing an example of reducing the capacity of a main memory.
17A is a diagram illustrating an example of an imaging target, and FIG. 17B is a diagram illustrating an example of an imaging result.
FIG. 18 is a block diagram showing an overall configuration of a display device according to a second embodiment of the present invention.
FIG. 19 is a block diagram showing a connection relationship between a signal line driver circuit, a scanning line driver circuit, a sensor control circuit, and a signal processing output circuit on a glass substrate, and a control IC and a memory on the control circuit substrate.
FIG. 20 is a block diagram showing an example of a detailed configuration of a glass substrate 31.
FIG. 21 is a circuit diagram showing an example of a detailed configuration of a scanning line driving circuit 3.
FIG. 22 is a block diagram showing an example of a detailed configuration of a signal processing output circuit 54.
FIG. 23 is a block diagram showing an example of a detailed configuration of a synchronization signal generation circuit 71.
FIG. 24 is a block diagram showing an example of a detailed configuration of a P / S conversion circuit 72.
FIG. 25 is a circuit diagram showing an example of the internal configuration of a decoder.
FIG. 26 is a circuit diagram showing an example of the internal configuration of a latch.
FIG. 27 is a block diagram showing a detailed configuration of an output buffer 73.
FIG. 28 is a detailed circuit diagram of one pixel of the pixel array unit 1;
FIG. 29 is a layout diagram of one pixel on a glass substrate 31;
FIG. 30 illustrates an operation of the display device of the embodiment.
FIG. 31 is an operation timing chart of a mode m1.
FIG. 32 is an operation timing chart in modes m2 and m3.
FIG. 33 is an operation timing chart of modes m4 and m5.
FIG. 34 is a diagram illustrating block division.
FIG. 35 is a flowchart illustrating an example of a processing operation performed by the control IC 55 in FIG. 18;
FIG. 36 is a diagram illustrating a relationship between an imaging time and an average gradation.
FIG. 37 is a view showing an example of imaging data obtained in step S7.
FIG. 38 is a diagram illustrating an average gradation increment.
FIG. 39 is a view showing an example of an imaging result of the embodiment.
FIG. 40 is a diagram illustrating an example of display luminance of each block according to the second embodiment.
FIG. 41 is a diagram schematically illustrating captured images of four adjacent blocks according to the second embodiment.
FIG. 42 is a diagram showing a processing result of the third embodiment when imaging is performed under the same conditions as in FIG. 40;
FIG. 43 is a diagram schematically illustrating captured images of four adjacent blocks according to the third embodiment.
FIG. 44 is a block diagram showing a schematic configuration of a fourth embodiment of the display device according to the present invention.
FIG. 45 is a view showing an example of a reference pattern stored in a reference pattern storage unit.
FIG. 46 is a flowchart showing a processing operation performed by the control IC 55 of the embodiment.
FIG. 47 is a diagram illustrating an example of an imaging target.
FIG. 48 is a view showing an example of an imaging result.
FIG. 49 is a diagram showing an example of a reverse pattern.
FIG. 50 is a diagram showing an example of imaging data of a sensor.
FIG. 51 is a diagram showing an example of a final imaging result.
[Explanation of symbols]
1 Pixel array section
2 Signal line drive circuit
3 Scan line drive circuit
4 Detection circuit & output circuit
11 pixel TFT
12a, 12b sensor
13 Buffer
21 Array board
22 space
23 Backlight
24 Counter substrate
31 glass substrate
32 Semiconductor substrate
33 Logic IC
52 control circuit board
53 Sensor control circuit
54 signal processing output circuit
55 Control IC
56 memories
86 Reference pattern storage

Claims (16)

A display element formed near each intersection of signal lines and scanning lines arranged in rows and columns;
An image pickup unit that is provided at least one by one corresponding to each of the display elements and receives an incident light in a specified range and converts the incident light into an electric signal,
A charge accumulation unit that accumulates charges according to the electric signal converted by the imaging unit;
A signal processing unit that generates digital image data corresponding to an image captured by the imaging unit based on the charge stored in the charge storage unit under each of a plurality of imaging conditions. Display device provided with. A binary data generation unit that outputs binary data indicating whether or not the charge stored in the charge storage unit is equal to or greater than a predetermined threshold;
The display device according to claim 1, wherein the signal processing unit generates the digital image data based on the binary data obtained under each of the plurality of shooting conditions. A photographing condition switching unit for switching the photographing conditions in a stepwise ascending order or a descending order; wherein the signal processing unit changes the logic of the binary data when the photographing condition switching unit switches the photographing conditions by one stage. 3. The display device according to claim 2, wherein, in the case where the digital image data of the target pixel is generated, digital image data of the target pixel is generated based on values of the binary data of a plurality of pixels around the target pixel. The imaging condition switching unit switches the initial charge amount stored in the charge storage unit in a stepwise or ascending order,
The charge accumulating unit accumulates, for each of a plurality of types of the initial charge amounts, remaining charges obtained by subtracting charges corresponding to a light reception amount in the imaging unit from the initial charge amounts. 4. The display device according to 3. The display device according to claim 3, wherein the imaging condition switching unit switches the imaging time of the imaging unit in an ascending order or a descending order in a stepwise manner. When the logic of the binary data changes when the photographing condition switching unit switches the initial charge amount by one stage, the signal processing unit determines whether or not eight pixels around the target pixel have been captured last time. The display device according to claim 3, wherein an average value of the binary data is set as digital image data of the pixel of interest. A temporary storage unit for temporarily storing the binary data under one shooting condition;
A first area for storing the binary data stored in the temporary storage unit; and a second area for storing the digital image data corresponding to the binary data stored in the first area. The display device according to claim 2, further comprising a work storage unit. The display element, the imaging unit, the charge storage unit and the binary data generation unit are formed on the same insulating substrate,
The display device according to claim 7, wherein the signal processing unit, the temporary storage unit, and the working storage unit are formed on a semiconductor substrate separate from the insulating substrate. A display element formed near each intersection of signal lines and scanning lines arranged in rows and columns;
An imaging unit provided at least one for each of the display elements, each of which receives incident light in a designated range and converts the incident light into an electric signal;
A charge accumulation unit that accumulates charges according to the electric signal converted by the imaging unit;
An image data amplifying unit that amplifies binary data according to the charge stored in the charge storage unit;
An average gray level detection unit that detects an average gray level of the imaging target based on the binary data amplified by the imaging data amplification unit;
A brightness setting unit that sets a display brightness of the display element when performing imaging with the imaging unit based on a detection result of the average gradation detection unit;
An imaging time control unit configured to switch and control the imaging time of the imaging unit in a plurality of ways;
Digital image data corresponding to a captured image based on imaging results of the imaging unit at individual imaging times that are switched and controlled by the imaging time control unit in a state in which the display luminance is set by the luminance setting unit. And an image data generating means for generating the image data. 10. The display device according to claim 9, wherein the display brightness setting based on the first-half average gradation is performed for each block including a plurality of adjacent display elements. An imaging time detection unit that detects an imaging time of the imaging unit required for the average gradation detected by the average gradation detection unit to reach a predetermined reference gradation value for each block; The display device according to claim 9, wherein: sets the luminance of the display element based on a detection result of the imaging time detection unit. The display device according to claim 9, wherein the image data generation unit weights an imaging result of the imaging unit at each imaging time. A scanning line driving circuit capable of switching control whether to sequentially drive the scanning lines or to drive all the scanning lines simultaneously,
13. The display device according to claim 9, wherein the scanning line driving circuit drives all the scanning lines at the same time when initial charge is stored in the charge storage unit. A data number measuring unit for measuring the number of “0” or “1” included in the serial data,
The display device, wherein the average gray level detecting unit detects an average gray level based on a measurement result of the data number measuring unit. The brightness setting unit sets the brightness of the display element located at the center of the block, and gradually changes the brightness of the display element other than the center position of the block according to the distance from the center position. The display device according to any one of claims 10 to 14, wherein A display element formed near each intersection of signal lines and scanning lines arranged in rows and columns;
An imaging unit provided at least one for each of the display elements, each of which receives incident light in a designated range and converts the incident light into an electric signal;
A charge accumulation unit that accumulates charges according to the electric signal converted by the imaging unit;
An image data amplifying unit that amplifies binary data according to the charge stored in the charge storage unit;
A reference pattern storage unit that stores a plurality of reference patterns indicating a display mode of a block including a plurality of display elements, which is used when imaging is performed by the imaging unit.
A reference pattern selection unit that selects a plurality of types of reference patterns approximate to the imaging result of the imaging unit from the reference pattern storage unit,
In a state where a reference pattern obtained by inverting the brightness of each of the reference patterns selected by the reference pattern selection unit is displayed on the display element, based on a result of repeatedly performing imaging in the imaging unit, a captured image is formed. A display device, comprising: image data generating means for generating a corresponding digital image.

Images