WO2012018090A1 - Area sensor and display device - Google Patents

Abstract

Light sensors (8a, 8b) are provided with: an I layer (9i) constituting a light receiving part; and a light blocking member (14) which blocks light and is formed in a layer above the I layer (9i) constituting the light receiving part. In the light sensor (8a), an opening (14a) of the light blocking member (14) is formed at a predetermined amount of offset to the left side of the I layer (9i) constituting the light receiving part, in order to partially overlap the I layer (9i) in plan view in such a way that light components from the upper left side in the drawing can be received and light components from the upper right side in the drawing can be blocked. In the light sensor (8b), the opening (14a) of the light blocking member (14) is formed at a predetermined amount of offset to the right side of the I layer (9i) constituting the light receiving part, in order to partially overlap the I layer (9i) in plan view in such a way that the light components from the upper right side in the drawing can be received and the light components from the upper left side in the drawing can be blocked. It is therefore possible to realize a display device and an area sensor capable of estimating distance with higher precision and also enabling relatively free placement of a light receiving element having different directionality characteristics.

Description

Area sensor and display device

The present invention relates to an area sensor provided with a light receiving element (light sensor) and a display device provided with such an area sensor.

Some display devices such as liquid crystal display devices have a touch panel (area sensor) function that can detect the touched position when the surface of the display device is touched with an input pen or a human finger. An integrated display device has been developed.

Conventional touch panel integrated display devices have a resistive film method (a method in which an input position is detected by contact between an upper conductive substrate and a lower conductive substrate when pressed) and a capacitive method (in which the touched area is touched). The method of detecting the input position by detecting a change in capacitance) has been the mainstream.

However, such touch panel integrated display devices have disadvantages such as an increase in thickness and the addition of a separate manufacturing process. For example, a touch panel integrated liquid crystal display device in which a sensor) is provided for each pixel (or for each of a plurality of pixels) in an image display region of the liquid crystal display device has been developed.

According to such a configuration, the liquid crystal display device is provided with a pixel transistor for each pixel. And since the said light receiving element is formed between the both board | substrates with which the said liquid crystal display device was equipped, without increasing the thickness of the said liquid crystal display device, the said light receiving element and the said pixel transistor are used. Since it can be formed in the same process, it is not necessary to add a separate manufacturing process.

For this reason, a display device that includes a light receiving element and an area sensor that detects an input position from the outside is attracting attention.

In addition, in an area sensor having a light receiving element, a floating state (non-contact) of an object to be detected such as an input pen or a human finger, which is difficult to realize with a resistive touch panel or a capacitive touch panel The position in the state) can be detected.

Furthermore, in an area sensor provided with a light receiving element, the distance of the detection object in a float state from the display surface of the display device can also be detected.

For example, Patent Document 1 describes an information input device including a light receiving unit configured by a photodiode capable of obtaining distance information of a target object such as a hand with high resolution and extracting a three-dimensional shape of the target object. ing.

FIG. 19 is a diagram showing a schematic configuration of the information input device.

As shown in the figure, the light emitted from the light emitting means 105 is reflected by the target object 106 and forms an image on the light receiving surface of the reflected light extracting means 108 by the light receiving optical system 107 such as a lens. ing.

The reflected light extraction unit 108 includes a first light receiving unit 109, a second light receiving unit 110, and a difference calculation unit 111, and detects the intensity distribution of reflected light reflected by the target object 106, that is, a reflected light image. . The first light receiving means 109 and the second light receiving means 110 are set to receive light at different timings, and the light emitting means 105 emits light when the first light receiving means 109 is receiving light, The timing control unit 112 controls the operation timing so that the light emitting unit 105 does not emit light when the second light receiving unit 110 is receiving light.

Due to such a configuration, the first light receiving means 109 receives the reflected light of the light from the light emitting means 105 by the target object 106 and other external light such as sunlight and illumination light, while the second light receiving means 109 receives the second light. The light receiving means 110 receives only external light. Since the timings at which the light is received are different, but are close to each other, fluctuations in external light during this period can be ignored.

Therefore, if the difference between the image received by the first light receiving means 109 and the image received by the second light receiving means 110 is taken, only the component of the reflected light from the target object 106 is extracted from the light from the light emitting means 105. Can do. The difference calculation unit 111 calculates and outputs the difference between the images received by the first light receiving unit 109 and the second light receiving unit 110.

The reflected light extraction means 108 sequentially outputs the reflected light amount of each pixel of the reflected light image. The output from the reflected light extraction means 108 is amplified by the amplifier 113, converted into digital data by the A / D converter 114, stored in the memory 115, and stored in the memory 115 at an appropriate timing. Data is read out and processed by the feature information generation means 116. These controls are performed by the timing control means 112.

The reflected light from the target object 106 decreases significantly as the distance between the target object 106 and the first light receiving means 109 increases. When the surface of the target object 106 scatters light uniformly, the amount of received light per pixel of the reflected light image decreases in inverse proportion to the square of the distance between the target object 106 and the first light receiving means 109. Therefore, when the target object 106 is placed in front of the information input device, the reflected light from the background becomes small enough to be ignored, and a reflected light image from only the target object 106 can be obtained.

Specifically, when a hand is brought in front of the information input device, a reflected light image from the hand is obtained. At this time, the amount of light received per pixel of the reflected light image is The amount of reflected light received by the unit light receiving unit corresponding to the one pixel is obtained. The amount of reflected light is affected by the properties of the target object 106 (such as specularly reflecting, scattering, and absorbing light), the orientation of the surface of the target object 106, the distance of the target object 106, etc. When the entire object 106 is an object that uniformly scatters light, the amount of reflected light has a close relationship with the distance to the target object 106. A hand or the like is an object in which the entire target object 106 scatters light uniformly, so the reflected light image when the hand is put out shows the distance of the hand, the inclination of the hand (the distance is partially different), etc. reflect. Therefore, by extracting such feature information, various information can be input / generated.

In the configuration of Patent Document 1 described above, by using a logarithmic amplifier as the amplifier 113, the amount of received light per pixel of the reflected light image can be made inversely proportional to the distance to the target object 106, and the dynamic range can be increased. It is described that an information input device that can be used effectively, can obtain distance information with high resolution, and can extract the three-dimensional shape of the target object 106 can be realized.

FIG. 20 is a diagram showing a schematic configuration of the display device described in Patent Document 2.

As shown in the figure, the pixels 140 have a matrix arrangement that is continuously arranged in the vertical and horizontal directions. The display area 120 in one pixel 140 is formed in a rectangular shape, and one set of the optical sensors 130R and 130L is arranged in the horizontal direction below the display area 120.

Further, the light shielding member 150 is arranged as hatched in FIG. In other words, the light shielding member 150 has an opening 152 that matches the shape of the display area 120 and a rectangular opening 154 that matches the optical sensors 130R and 130L.

As shown in the drawing, the optical sensors 130R and 130L are alternately arranged, and the optical sensor 130R and the optical sensor 130L are arranged in this order from the left in FIG. A light shielding member 150 is provided so as to straddle the sensor 130L.

On the other hand, in the place where the optical sensor 130L and the optical sensor 130R are arranged in this order from the left in FIG. 20, an opening 154 of the light shielding member 150 is provided so as to straddle the optical sensor 130L and the optical sensor 130R.

The light shielding member 150 is provided so that the boundary between the optical sensor 130R and the optical sensor 130L in each pixel 140 and the center line 138 of the rectangular opening 154 in the light shielding member 150 coincide with each other.

That is, in the above configuration, in order to give different directional characteristics to the optical sensor 130R and the optical sensor 130L in each pixel 140, one light shielding member is provided so as to straddle part of the optical sensor 130R and part of the optical sensor 130L. In this configuration, 150 openings 154 are provided.

Further, in the display device described in Patent Document 2, when a detection object such as a finger is present from the display panel 100 to the driver seat side or the passenger seat side in a relatively bright external state, the detection target is detected. A shadow of the object is generated, that is, a part that becomes darker than the background is generated, and the detected object is detected by the optical sensors 130R and 130L.

On the other hand, when the outside is relatively dark, such as at night or when traveling in a tunnel, the irradiation light from the backlight (not shown) is reflected by the detected object and enters the optical sensors 130R and 130L. , The object to be detected is detected by the optical sensors 130R and 130L.

Because of such a configuration, the portion specified by the optical sensor 130R among the portions where the light amount is reduced or increased compared to the background portion, whether the outside is bright or dark, is the R image, the optical sensor. An image specified by 130L can be an L image.

FIG. 21 is a diagram illustrating a case where an object to be detected such as a finger is shown as a sphere, and the object to be detected approaches the display panel 100.

As shown in the figure, for example, when a finger approaches the display panel 100 from the passenger seat side, the finger passes through points (a), (b), and (c).

Then, as the finger approaches the display panel 100 (as the finger moves from (a) to (c)), the R image and the L image approach each other, and the finger touches the display panel 100. As a result, the R image and the L image almost overlap each other.

That is, as the finger approaches the display panel 100, the parallax between the R image and the L image decreases.

Therefore, in Patent Document 2, the parallax between the R image and the L image is used, and the finger performs a touch operation on the display panel 100 when the parallax is smaller than a preset threshold. It is described that it is possible to realize a display device that can be discriminated.

Japanese Patent Laid-Open No. 10-177449 (released on June 30, 1998) Japanese Patent Laying-Open No. 2009-9098 (released on January 15, 2009) Japanese Patent Laid-Open No. 9-27969 (published January 28, 1997)

However, in Patent Document 1, the amount of received light per pixel of the reflected light image is inversely proportional to the distance to the target object 106 in estimating the distance of the target object 106, that is, the reflection of the target object 106. Only strength is used.

Since the reflection intensity of the target object 106 varies depending on the reflectance of the target object 106, for example, if the target object 106 is located at the same distance from the first light receiving unit 109, the target object 106 is separated by the same distance. Nevertheless, the reflection characteristics of the hand differ depending on race (skin color), hand skin thickness, blood circulation, etc. Therefore, in order to estimate the distance with high accuracy, every time the user changes There is a problem that optimization (adjustment) of the information input device is required.

Also, the configuration of the above-mentioned Patent Document 1 has a problem that it is difficult to estimate the distance because it cannot cope with a momentary fluctuation in external light.

In Patent Document 2, as shown in FIG. 20, a part of the optical sensor 130R and the optical sensor 130L are provided so that the optical sensor 130R and the optical sensor 130L in each pixel 140 have different directivity characteristics. It is the structure which provided the opening part 154 of the one light shielding member 150 so that one part may be straddled.

Therefore, the optical sensor 130R, the optical sensor 130L, and the light shielding member 150 are arranged so that the boundary between the optical sensor 130R and the optical sensor 130L in each pixel 140 and the center line 138 of the rectangular opening 154 in the light shielding member 150 coincide. Is provided in stripes at regular intervals.

According to the above configuration, in each pixel 140, since one common opening 154 is formed for the two photosensors 130R and 130L, the two photosensors 130R and 130L have different directivity characteristics. Therefore, the arrangement positions of the optical sensors 130R and 130L with respect to the arrangement position of the opening 154 are limited.

Therefore, it is difficult to freely arrange the two optical sensors 130R and 130L independently in each pixel 140.

The present invention has been made in view of the above problems, and is an area sensor that can estimate a distance with higher accuracy and can relatively freely arrange light receiving elements having different directivity characteristics. An object is to provide a display device.

In order to solve the above-described problem, the area sensor of the present invention has a surface in which a plurality of first and second light receiving elements that detect the amount of incident light are combined to form a plurality. The first light receiving element includes a first light receiving portion and a first light blocking member that is formed in an upper layer than the first light receiving portion and blocks the light. The second light receiving element includes a second light receiving part and a second light shielding member that is formed in an upper layer than the second light receiving part and blocks the light. In the first light receiving element, The light component from the first direction can be received, and the light component from the second direction that is opposite to the direction of the reflected light when the light from the first direction is specularly reflected by the surface can be blocked. As described above, the opening of the first light-shielding member is in plan view with respect to the first light-receiving portion. The second light receiving element is formed to be shifted by a predetermined amount so as to partially overlap, and the second light receiving element can receive a light component from the second direction. The opening of the second light blocking member is located on the second direction side so as to partially overlap the second light receiving unit in plan view so that light components from the direction can be blocked. It is characterized by being formed with a fixed amount of displacement.

Conventionally, in order to form a light receiving element having two different directional characteristics, an opening of one light shielding member is provided so as to straddle a part of each of the two light receiving elements formed adjacent to each other.

Therefore, the light receiving element and the light shielding member are provided at regular intervals so that the boundary between the two light receiving elements coincides with the center line of the opening of the one light shielding member.

Because of this configuration, it was difficult to arrange the two light receiving elements independently and freely.

On the other hand, according to the configuration of the present invention, in order to give directivity to the first light receiving element, the opening of the first light shielding member is partially separated from the first light receiving unit in a plan view. So that the second light-receiving element has directivity, the opening of the second light-shielding member is formed in the second direction. The light receiving portion is formed so as to be shifted by a predetermined amount on the second direction side so as to partially overlap in plan view.

Due to such a configuration, the arrangement position of the second light receiving element does not affect the directivity of the first light receiving element, and the arrangement position of the first light receiving element is the second position. This does not affect the directivity of the light receiving element.

That is, in the above configuration, since the directivity can be controlled independently for each light receiving element, there is no particular limitation on the arrangement positions of the first light receiving element and the second light receiving element. It will be.

Therefore, according to the above configuration, the first light receiving element and the second light receiving element having different directivity characteristics do not necessarily have to be provided at adjacent positions. Therefore, the light receiving elements having different directivity characteristics are compared. An area sensor that can be freely arranged can be realized.

In order to solve the above problems, a display device of the present invention includes the area sensor, and is formed in a matrix on the surface on which the first light receiving element and the second light receiving element are formed. In addition, a plurality of pixels for displaying an image are provided, and the first light receiving element and the second light receiving element are provided according to the position of the pixel.

According to the above configuration, a display device in which the area sensor is integrated can be realized. Therefore, information can be input using the area sensor while viewing the image of the display device.

The first light receiving element and the second light receiving element are provided according to the position of the pixel. For example, the first light receiving element and the second light receiving element are 1 When the first light receiving element is provided in one of the two adjacent pixels, the second light receiving element is provided in the other, or the light receiving elements are not provided in all the pixels. An example is a case where a light receiving element is provided only in a pixel relating to color, but is not limited thereto.

As described above, in the area sensor of the present invention, the first light receiving element includes the first light receiving portion and the first light shielding member that is formed in an upper layer than the first light receiving portion and blocks the light. The second light receiving element includes a second light receiving portion and a second light shielding member that is formed in an upper layer than the second light receiving portion and blocks the light. In the light receiving element, the light component from the first direction can be received, and the light from the first direction is reflected from the second direction which is opposite to the reflected light direction when the light is specularly reflected by the surface. The opening of the first light shielding member is shifted by a predetermined amount toward the first direction so as to partially overlap the first light receiving part in plan view so that the light component can be shielded. In the second light receiving element, the light component from the second direction is received. The second light-shielding member has an opening that partially overlaps the second light-receiving part in plan view so that the light component from the first direction can be shielded. 2 is configured to be shifted by a predetermined amount in the direction of 2.

Moreover, the display device of the present invention includes the area sensor as described above, and is formed in a matrix on the surface on which the first light receiving element and the second light receiving element are formed. A plurality of pixels for displaying an image are provided, and the first light receiving element and the second light receiving element are provided according to the position of the pixel.

Therefore, it is possible to estimate the distance with higher accuracy and to realize an area sensor and a display device in which light receiving elements having different directivity characteristics can be arranged relatively freely.

It is a figure which shows schematic structure of the optical sensor with which the area sensor integrated liquid crystal display device of one embodiment of this invention was equipped. It is a top view of the photosensor shown in FIG. 1 is a diagram showing a schematic configuration of an area sensor integrated liquid crystal display device according to an embodiment of the present invention; FIG. The example of formation of the optical sensor provided in the area sensor integrated liquid crystal display device of one embodiment of this invention is shown. It is a figure which shows the schematic circuit structure in each pixel of the area sensor integrated liquid crystal display device of one embodiment of this invention. It is a schematic block diagram which shows the structure of the area sensor integrated liquid crystal display device of one embodiment of this invention. FIG. 6 is a diagram for explaining a case where a parallax is calculated by performing correlation calculation using a binarized left image and right image in the area sensor integrated liquid crystal display device according to the embodiment of the present invention. . In the area sensor integrated liquid crystal display device according to the embodiment of the present invention, a parallax is calculated by matching a 256-tone left image and a right image for each line using a normalized correlation method. It is a figure for demonstrating. To describe a case where a parallax is calculated by performing correlation calculation for each line using a left image and a right image in which an edge is detected in an area sensor integrated liquid crystal display device according to an embodiment of the present invention. FIG. In the area sensor-integrated liquid crystal display device according to the embodiment of the present invention, the left image and the right image in which the edge illustrated in FIGS. 9B and 9C is detected are normalized and correlated. It is a figure for demonstrating the case where matching is performed using FIG. In the area sensor integrated liquid crystal display device according to one embodiment of the present invention, a case where an edge direction image of a left image and a right image is obtained and parallax is obtained by correlation calculation including a term related to the edge direction is described. FIG. FIG. 6 is a diagram for explaining a method for obtaining a height h from a parallax d and an incident angle θ in an area sensor integrated liquid crystal display device according to an embodiment of the present invention. It is a figure for demonstrating the flow which calculates | requires the correlation function for every pixel in the area sensor integrated liquid crystal display device of one embodiment of this invention. It is a figure which shows R (j, k) in FIG. It is a figure which shows the reference table for calculating | requiring the value of Wdir which is the term regarding the direction of the edge in FIG. It is a figure for demonstrating the flow which calculates | requires the maximum value of the correlation function for every line in the area sensor integrated liquid crystal display device of one embodiment of this invention. It is a figure which shows R (j, k) and parallax z (k) in FIG. It is a figure which shows schematic structure of the optical sensor provided with the optical element which restrict | limits the incident angle within the fixed range with which the area sensor integrated liquid crystal display device of one embodiment of this invention is equipped. It is a figure which shows schematic structure of the conventional information input device. It is a figure which shows schematic structure of the conventional display apparatus of patent document 2. As shown in FIG. In the conventional display device shown in FIG. 20, a detected object such as a finger is shown as a sphere, and the detected object approaches the display panel.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, relative arrangements, and the like of the component parts described in this embodiment are merely one embodiment, and the scope of the present invention should not be construed as being limited thereto.

[Embodiment 1]
In this embodiment, an area sensor integrated liquid crystal display device including a light receiving element (light sensor) will be described as an example. However, the present invention is not limited thereto, and the area sensor is provided. For example, the present invention can be applied to a display device such as an organic EL display device or an area sensor including a light receiving element.

First, the configuration of the area sensor integrated liquid crystal display device 1 according to the present embodiment will be described with reference to FIG.

The area sensor integrated liquid crystal display device 1 (also referred to simply as the liquid crystal display device 1) shown in FIG. 3 is provided with a photosensor for each pixel of the liquid crystal display panel 2, although not shown. The image of the detected object 7 (finger) placed on the display surface 2a side of the liquid crystal display panel 2 is extracted and its position is detected.

Further, a backlight 3 for irradiating the liquid crystal display panel 2 with light is provided on the back surface 2b side of the liquid crystal display panel 2.

The liquid crystal display panel 2 includes an active matrix substrate 4 in which a large number of pixels are arranged in a matrix, and a counter substrate 5 disposed so as to face the active matrix substrate 4, and further between these two substrates 4 and 5. The liquid crystal layer 6 is sandwiched between the two. In this embodiment, the VA mode liquid crystal display panel 2 is used. However, the display mode of the liquid crystal display panel 2 is not particularly limited, and any display mode such as a TN mode or an IPS mode can be applied. it can.

Although not shown, a front-side polarizing plate and a back-side polarizing plate are provided outside the liquid crystal display panel 2 so as to sandwich the liquid crystal display panel 2.

Each of the polarizing plates serves as a polarizer, and in the present embodiment in which the liquid crystal material in the liquid crystal layer 6 is a vertical alignment type, the polarization direction of the front-side polarizing plate and the polarization direction of the back-side polarizing plate are By arranging them in a crossed Nicols relationship, a normally black mode liquid crystal display device can be realized.

Although not shown, the active matrix substrate 4 includes a TFT element which is a switching element for driving each pixel, a pixel electrode electrically connected to the TFT element, the optical sensor, an alignment film, and the like. Is formed.

On the other hand, although not shown, the counter substrate 5 is formed with a color filter layer, a counter electrode, an alignment film, and the like. The said color filter layer can be comprised from the coloring part which has each color of red (R), green (G), and blue (B), and a black matrix, for example.

In the present embodiment, as the backlight 3, in addition to the visible light for display, the detected object 7 is detected with higher accuracy regardless of the external brightness without affecting the image quality of the display image. Therefore, a backlight having a function of uniformly emitting infrared light that is invisible light is used. Since the infrared light is invisible light, it can be appropriately modulated and pulsed as necessary.

In the present embodiment, since the infrared light emitted from the backlight 3 is reflected by the detected object 7 and detects the amount of incident light that is incident on the optical sensor, the light is used. On the region where the sensor is formed, a black matrix made of carbon black or the like provided on the counter substrate 5 is positioned.

The black matrix made of carbon black or the like blocks visible light and transmits infrared light. Therefore, light that passes through the black matrix and enters the photosensor is only infrared light. The amount of infrared light reflected by the detection object 7 and incident on the optical sensor can be detected.

When the backlight 3 is a backlight that does not contain infrared light and emits visible light for display, the backlight 3 is provided on the counter substrate 5 on the region where the photosensor is formed. The opening of the black matrix made of carbon black or the like is positioned, and the position of the detection object 7 can be detected as follows.

When the outside is relatively bright, the amount of incident light from the outside is different between the location where the detected object 7 exists and the location where the detected object 7 does not exist, so the position of the detected object 7 is not used without using a backlight. Can be detected.

On the other hand, when the outside is relatively dark, there is no difference in the amount of incident light from the outside between the place where the detected object 7 exists and the place where the detected object 7 does not exist, so the light emitted from the backlight is detected. The position of the detected object 7 can be detected by utilizing the fact that the object 7 is reflected.

Hereinafter, based on FIG. 1 and FIG. 2, the optical sensor provided in the liquid crystal display device 1 of the present embodiment will be described in detail.

FIG. 1 is a diagram showing a schematic configuration of an optical sensor provided in the liquid crystal display device 1 of the present embodiment.

FIG. 1A shows an optical sensor 8a (first light receiving element) having a directional characteristic in the upper left direction in the drawing provided in the active matrix substrate 4, and FIG. An optical sensor 8b (second light receiving element) having a directivity characteristic is shown in the upper right direction in the figure provided on the matrix substrate 4.

As shown in FIG. 1A, in the present embodiment, the active matrix substrate 4 provided with a photosensor (light receiving element) having a high sensing speed is relatively easy to manufacture. As shown in FIG. 1, a PIN diode having a structure in which the P + layer 9p +, the I layer 9i (first light receiving unit / second light receiving unit), and the N + layer 9n + do not overlap each other is used. However, the present invention is not limited to this.

Further, as the optical sensor, any optical sensor may be used as long as it passes different current values according to the amount of light received in the light receiving unit provided in the optical sensor. You can also.

Hereinafter, a process for forming the optical sensors 8a and 8b on the active matrix substrate 4 will be described.

In the present embodiment, a glass substrate is used as a substrate for constituting the active matrix substrate 4 not shown in FIGS. 1A and 1B, but the present invention is not limited to this. Alternatively, a quartz substrate, a plastic substrate, or the like can be used.

In the glass substrate, although not shown in detail, the light emitted from the backlight 3 is applied to the surface on which the pixel TFT and the optical sensors 8a and 8b, which will be described later, are formed, and the pixel TFT and the optical sensor 8a. A light-shielding film is formed to block incident on 8b. A base coat film is formed on the light shielding film so as to cover the entire surface of the light shielding film and the glass substrate.

As the base coat film, a film made of an insulating inorganic material such as a silicon oxide film, a silicon nitride film, or a silicon oxynitride film, or a laminated film appropriately combining them can be used. A silicon oxide film was used. Further, these films can be formed by depositing by LPCVD, plasma CVD, sputtering, or the like.

And on the upper surface of the base coat film, optical sensors 8a and 8b shown in FIGS. 1A and 1B are formed.

The process for forming the optical sensors 8a and 8b on the base coat film is as follows.

First, on the base coat film in the region where the light shielding film is formed, a non-single-crystal semiconductor thin film that will later become the polycrystalline semiconductor film 9 is formed by LPCVD, plasma CVD, sputtering, or the like. .

The non-single-crystal semiconductor thin film includes amorphous silicon, polycrystalline silicon, amorphous germanium, polycrystalline germanium, amorphous silicon / germanium, polycrystalline silicon / germanium, amorphous silicon / carbide, Crystalline silicon carbide or the like can be used. In this embodiment mode, amorphous silicon is used.

Subsequently, the non-single-crystal semiconductor thin film is crystallized to form a polycrystalline semiconductor film 9. In crystallization, a laser beam, an electron beam, or the like can be used. In this embodiment mode, crystallization is performed using a laser beam.

Next, the polycrystalline semiconductor film 9 is patterned by photolithography according to the formation region of the light shielding film.

Then, an intrinsic semiconductor layer serving as a light receiving portion or an I layer 9i that is a semiconductor layer having a relatively low impurity concentration is formed in the center of the polycrystalline semiconductor film 9 in the region where the optical sensors 8a and 8b are formed. On both sides, a P + layer 9p + that is a semiconductor layer having a relatively high P-type impurity concentration and an N + layer 9n + that is a semiconductor layer having a relatively high N-type impurity concentration are formed.

Next, a gate insulating film 10 made of a silicon oxide film or the like is formed, and the polycrystalline semiconductor film 9 is covered with the gate insulating film 10.

Then, contact holes penetrating the gate insulating film 10 are formed on the P + layer 9p + and the N + layer 9n +, respectively.

Then, a conductive film is formed on the entire surface by sputtering or the like.

As the conductive film, for example, a conductive film made of aluminum or the like can be used. However, the conductive film is not limited to this, and is selected from Ta, W, Ti, Mo, Al, Cu, Cr, Nd, and the like. An element, or an alloy material or a compound material containing the element as a main component may be used, and if necessary, a laminated structure may be formed by appropriately combining them. Alternatively, the conductive film may be formed using a semiconductor film typified by polycrystalline silicon or the like doped with an impurity such as phosphorus or boron. Note that in this embodiment mode, aluminum is used for the conductive film.

The conductive film is patterned into a desired shape by etching by photolithography, and becomes metal electrodes (wiring) 11a and 11b electrically connected to the P + layer 9p + and the N + layer 9n +, respectively.

Then, a transparent insulating layer 12 is formed on the entire surface so as to cover the gate insulating film 10 and the conductive film. In the present embodiment, an organic interlayer insulating film made of an acrylic resin is used as the transparent insulating layer 12.

Thereafter, a shield electrode layer 13 made of a transparent conductive film such as ITO (Indium Tin Oxide) or IZO (Indium Zinc Oxide) is formed into a P + layer 9p +, as shown in FIGS. 1 (a) and 1 (b). It is formed on transparent insulating layer 12 so as to overlap with I layer 9i and N + layer 9n + in plan view.

Then, a light shielding member 14 having an opening 14 a is formed on the shield electrode layer 13.

In the present embodiment, as shown in FIG. 1A, an optical sensor 8a having a directional characteristic is formed in the upper left direction in the drawing, and therefore the light component from the upper left direction in the drawing is received. The light component from the upper right direction in the figure, which is the direction opposite to the direction of the reflected light when the light from the upper left direction in the figure is specularly reflected by the surface on which the optical sensor 8a is formed, can be shielded. As described above, the opening 14a of the light shielding member 14 is located in the left direction in the drawing so as to partially overlap the I layer 9i of the polycrystalline semiconductor film 9 which is the light receiving portion of the optical sensor 8a in plan view. A fixed amount is formed.

2 (a) is a plan view of the optical sensor 8a shown in FIG. 1 (a).

Note that the predetermined amount to be shifted to the left (the width of the opening 14a of the light shielding member 14 formed on the n + layer 9n + in FIG. 2A) is the same as that of FIG. It is preferable that it is 5% or more and 50% or less of the lateral width which is the width in the left-right direction.

On the other hand, as shown in FIG. 1B, since the optical sensor 8b having the directivity characteristic is formed in the upper right direction in the figure, the light component from the upper right direction in the figure can be received. The light shielding member can shield the light component from the upper left direction in the figure, which is the direction opposite to the direction of the reflected light when the light from the upper right direction is specularly reflected by the surface on which the optical sensor 8b is formed. 14 openings 14a are formed by shifting a predetermined amount in the right direction in the drawing so as to partially overlap the I layer 9i of the polycrystalline semiconductor film 9 which is a light receiving portion of the optical sensor 8b in plan view. ing.

2 (b) is a plan view of the optical sensor 8b shown in FIG. 1 (b).

The predetermined amount to be shifted to the right (the width of the opening 14a of the light shielding member 14 formed on the p + layer 9p + in FIG. 2B) is the same as that of FIG. 2B of the opening 14a of the light shielding member 14. It is preferable that it is 5% or more and 50% or less of the lateral width which is the width in the left-right direction.

In the present embodiment, the description has been given of the case where the opening 14a of the light shielding member 14 is formed by shifting a predetermined amount with respect to the I layer 9i of the polycrystalline semiconductor film 9. However, the present invention is not limited to this. However, it may be formed shifted by a predetermined amount with respect to a portion where a non-permeable film such as a conductive film is not formed on the I layer 9i, which is a substantial light receiving portion of the optical sensor.

In the present embodiment, as already described above, the infrared light emitted from the backlight 3 is reflected by the detection object 7, and the incident light quantity of the infrared light incident on the optical sensors 8a and 8b is set. Since the detection configuration is used, a black matrix made of carbon black or the like provided on the counter substrate 5 is positioned on the region where the optical sensors 8a and 8b are formed.

Since the black matrix made of carbon black or the like blocks visible light and transmits infrared light, the light transmitted through the black matrix and incident on the optical sensors 8a and 8b is only infrared light. In the present embodiment, the material of the light shielding member 14 is not particularly limited as long as it has a function of blocking infrared light, and is a material capable of blocking infrared light. For example, a metal material can be used.

In the present embodiment, aluminum is used as the light shielding member 14, but the light shielding member 14 is not limited to this, and any material having a high reflectance in the wavelength region of the infrared light may be used. Is possible.

On the other hand, when considering the influence of oblique light incident directly on the optical sensors 8a and 8b without using a black matrix made of carbon black or the like, or when using an area sensor alone including the optical sensors 8a and 8b, or as described above. When the backlight 3 is a backlight that does not contain infrared light and emits visible light for display, etc., the light wavelength range of light sensitive to the optical sensors 8a and 8b is cut as the light shielding member 14. It is necessary to select materials that can be used.

More specifically, in the case where polycrystalline silicon is used as the light receiving part of the optical sensors 8a and 8b as in the present embodiment, the light shielding member 14 absorbs a wavelength region of light of 1100 nm or less, or A reflective material may be selected.

As shown in FIG. 2, in the present embodiment, in order to provide the optical sensor 8a with directivity, the opening 14a of the light shielding member 14 is replaced with the polycrystalline semiconductor film 9 which is a light receiving portion of the optical sensor 8a. In order to partially overlap with the I layer 9i in plan view, the I layer 9i is shifted by a predetermined amount in the left direction in the figure. The opening 14a is formed by shifting a predetermined amount in the right direction in the drawing so as to partially overlap the I layer 9i of the polycrystalline semiconductor film 9 which is a light receiving portion of the optical sensor 8b in plan view. .

Because of this configuration, the arrangement position of the optical sensor 8a does not affect the directivity of the optical sensor 8b, and the arrangement position of the optical sensor 8b does not affect the directivity of the optical sensor 8a.

That is, in the above configuration, since the directivity characteristics can be controlled independently for each of the optical sensors 8a and 8b, the arrangement position of the optical sensors 8a and 8b is not particularly limited.

Therefore, according to the above configuration, the optical sensors 8a and 8b having different directivity characteristics do not necessarily have to be provided at adjacent positions. Therefore, the optical sensors 8a and 8b having different directivity characteristics can be arranged relatively freely. Can do.

In the present embodiment, the first direction is the upper left direction in FIG. 1 and the second direction is the upper right direction in FIG. 1, but photosensors having different directivity characteristics can be formed. The first direction and the second direction are not limited to the above directions as long as parallax can be obtained from the image of the detection object obtained using the optical sensor.

FIG. 4 shows a formation example of the optical sensors 8a and 8b provided in the liquid crystal display device 1 of the present embodiment.

As described above, the optical sensors 8a and 8b have a configuration in which the directivity characteristics can be controlled independently for each of the optical sensors 8a and 8b. Therefore, the arrangement positions of the optical sensors 8a and 8b are not particularly limited. As in the prior art, the optical sensors 8a and 8b do not necessarily have to be provided at adjacent positions, so that the optical sensors 8a and 8b having different directivity characteristics can be arranged relatively freely.

Therefore, as shown in FIG. 4, when the pixel P (1, 1) is provided with the photosensor 8a having directivity in the upper left direction in the figure, the right direction of the pixel P (1, 1). Light having directivity characteristics in the upper right direction in the figure is P (1,2) and P (2,1) which is the nearest neighbor pixel in the lower direction of the pixel P (1,1). The sensor 8b is provided, and the pixel provided with the photosensor 8b is disposed in the nearest pixel in the vertical and horizontal directions of the pixel provided with the photosensor 8a, and the pixel provided with the photosensor 8b. The optical sensors 8a and 8b can be arranged in a staggered manner so that the pixel provided with the optical sensor 8a is arranged at the nearest pixel in the direction.

Further, the present invention is not limited to this, and although not shown, for example, the pixels relating to a specific color are not provided, for example, the photosensors 8a and 8b are provided in one pixel or the photosensors 8a and 8b are not provided in all pixels. It is also possible to provide the optical sensors 8a and 8b only in the case.

Further, in one pixel, the optical sensor 8a and the optical sensor 8b are arranged apart from each other in the vertical and horizontal ends with the display region interposed therebetween, and the optical sensors 8a and 8b are arranged in a plurality of pixels. It can also be provided for each.

That is, in the present embodiment, a configuration in which either one of the optical sensor 8a or the optical sensor 8b is provided for each pixel is used, but the present invention is not limited to this, and detection of an object to be detected is performed. Therefore, the optical sensors 8a and 8b can be provided in consideration of the accuracy of parallax, which will be described in detail later, from the images of the detected objects detected by the optical sensors 8a and 8b having directional characteristics different from those required for the purpose. .

FIG. 5 is a diagram showing a schematic circuit configuration in each pixel of the liquid crystal display device 1 of the present embodiment.

As shown in the drawing, an optical sensor circuit comprising an optical sensor 8a or an optical sensor 8b, a boosting capacitor 16, and a transistor 17 serving as a sensor output (output amplifier) is provided below each pixel. The scanning signal line GLn to which a scanning signal is supplied from a scanning signal line driving circuit (not shown) (see FIG. 6) is provided above the area where the photosensor circuit is provided in each pixel. Further, a data signal line SLn to which a data signal is supplied from a data signal line driving circuit (not shown) (see FIG. 6) is formed so as to intersect, and in the vicinity of a position where the scanning signal line GLn and the data signal line SLn intersect. The pixel TFT 15 is formed.

FIG. 5 shows an example of four adjacent pixels P (n−1, n−1) · P (n−1, n) · P (n) in the liquid crystal display device 1 having n × n pixels shown in FIG. , N−1) · P (n, n) is shown as an example.

In the present embodiment, in order to lengthen the decay time of the charge charged in the liquid crystal capacitor Clc formed by the pixel electrode, the liquid crystal layer 6 and the counter electrode (Vcom), in parallel with the liquid crystal capacitor Clc. The auxiliary capacitor Cs is provided, and one end of the auxiliary capacitor Cs is electrically connected to the drain electrode of the pixel TFT 15 and the pixel electrode of the liquid crystal capacitor Clc, and the other end is in the corresponding pixel. It is electrically connected to a storage capacitor bus line CSn formed in parallel with the scanning signal line GLn.

Hereinafter, the optical sensor circuit will be described in more detail.

The optical sensor circuit is configured as a 1T (abbreviation of transistor) type circuit using only one transistor 17 that plays a role of sensor output, and the transistor 17 functions as a source follower transistor (voltage follower transistor). .

The drain electrode of the transistor 17 is connected to the AMP power supply bus line Vsn (n is a natural number indicating the pixel column number), and the source electrode is connected to the photosensor output bus line Von. The AMP power supply bus line Vsn and the optical sensor output bus line Von are connected to a sensor readout circuit (not shown) (see FIG. 6). The AMP power supply bus line Vsn receives the power supply voltage VDD from the sensor readout circuit. Applied.

Further, the gate electrode of the transistor 17 is connected to an optical sensor 8a or an optical sensor 8b made of a PIN diode and one end of a boosting capacitor 16 is connected.

Further, the anode electrode (metal electrode (wiring) 11a in FIG. 1A and FIG. 1B) of the PIN diode provided in the optical sensors 8a and 8b is a sensor scanning signal line drive circuit not shown. (See FIG. 6) is connected to a wiring Vrstn (n is a natural number indicating the row number of the pixel) to which a reset signal RST is sent, and the other end of the boosting capacitor 16 is a sensor scanning signal line drive circuit (not shown). 6) is connected to the wiring Vrwn to which the optical sensor row selection signal RWS is sent. The photosensor row selection signal RWS has a role of selecting a specific row of photosensor circuits arranged in a matrix and outputting a detection signal from the photosensor circuit in the specific row.

Next, the operation of the optical sensor circuit will be described with reference to FIG.

First, in order to reset the potential of the gate electrode of the transistor 17, a high level reset signal RST is sent to the wiring Vrstn from a sensor scanning signal line drive circuit (not shown) (see FIG. 6). As a result, during the period in which the high level reset signal RST is sent to the wiring Vrstn, the forward bias is applied to the PIN diodes provided in the optical sensors 8a and 8b, so that the boosting capacitor 16 is charged and the gate of the transistor 17 is charged. The potential of the electrode gradually rises and finally reaches the initialization potential.

When the reset signal RST is lowered to a low level after the potential of the gate electrode of the transistor 17 reaches the initialization potential, the cathode electrodes of the PIN diodes provided in the photosensors 8a and 8b (FIG. 1 (a) and Since the potential of the metal electrode (wiring) 11b) in FIG. 1B is higher than the potential of the anode electrode, a reverse bias is applied to the PIN diode.

In this state, when the optical sensors 8a and 8b are irradiated with light, a photocurrent due to reverse bias flows through the PIN diode according to the intensity of the light. As a result, the charge held in the boosting capacitor 16 is discharged through the wiring Vrstn, so that the potential of the gate electrode of the transistor 17 gradually decreases, and finally the detection potential corresponding to the intensity of light. Go down.

Subsequently, the detection result is read. At this time, a high-level row selection signal RWS is applied to the other end of the boosting capacitor 16 from a sensor scanning signal line drive circuit (not shown) (see FIG. 6) via the wiring Vrwn. As a result, the potential of the gate electrode of the transistor 17 is pushed up through the boosting capacitor 16, so that the potential of the gate electrode of the transistor 17 becomes a potential obtained by adding the high level potential of the row selection signal RWS to the detection potential. .

As described above, when the potential of the gate electrode of the transistor 17 is raised, the threshold voltage for turning on the transistor 17 is exceeded, so that the transistor 17 is turned on. As a result, a voltage controlled at an amplification factor according to the potential level of the gate electrode of the transistor 17, that is, according to the light intensity, is output as a detection signal from the source electrode of the transistor 17, and output from the optical sensor. It is sent to a sensor readout circuit (see FIG. 6) not shown via the bus line Von.

Thus, a detection signal having a level corresponding to the intensity of light received by the optical sensors 8a and 8b is generated, and the detection signal is generated for each pixel provided with the optical sensors 8a and 8b. Is done. Therefore, the detection operation can be performed on the object to be detected arranged close to the liquid crystal display device 1.

In this embodiment, the AMP power supply bus line Vsn and the optical sensor output bus line Von are provided for each pixel separately from the data signal line SLn. However, the data signal line SLn and the AMP power supply are provided. The bus line Vsn can be shared to increase the aperture ratio. When the sensor circuit is provided for each of a plurality of pixels, the AMP power supply bus line Vsn is shared with the data signal line SLn-1, and the optical sensor output bus line Von is shared with the data signal line SLn. You can also

In the present embodiment, the optical sensors 8a and 8b, the pixel TFT 15, the auxiliary capacitor Cs, the boosting capacitor 16, the pixel electrode, and the transistor 17 are formed by the same process. However, the present invention is not limited to this. Absent.

Hereinafter, a schematic circuit configuration provided in the liquid crystal display device 1 will be described with reference to FIG.

FIG. 6 is a schematic block diagram showing the configuration of the liquid crystal display device 1.

As shown, the liquid crystal display device 1 includes a liquid crystal display panel 2, a scanning signal line driving circuit 18, a data signal line driving circuit / sensor reading circuit 19, a sensor scanning signal line driving circuit 20, a control circuit 21, and sensing. An image processing unit 22 and an interface circuit 28 are provided.

The scanning signal line driving circuit 18, the data signal line driving circuit / sensor reading circuit 19, and the sensor scanning signal line driving circuit 20 may have a separately created LSI attached to the liquid crystal display panel 2, or the liquid crystal display panel 2. It may be formed monolithically on top.

The scanning signal line driving circuit 18 generates a scanning signal for selectively scanning the pixels P (n, n) row by row by using a scanning signal line (not shown). The data signal line driving circuit / sensor reading circuit 19 supplies a data signal to each pixel P (n, n) using a data signal line (not shown).

The sensor scanning signal line driving circuit 20 selects and drives the optical sensor circuit row by row, and the data signal line driving circuit / sensor reading circuit 19 uses an AMP power supply bus line (not shown). A power supply voltage VDD having a constant potential is supplied to the photosensor circuit, and a detection signal of an object to be detected is read from the photosensor circuit using a photosensor output bus line (not shown).

The control circuit 21 supplies necessary power supply voltages and control signals to the scanning signal line driving circuit 18, the data signal line driving circuit / sensor reading circuit 19, the sensor scanning signal line driving circuit 20, and the sensing image processing unit 22.

The sensing image processing unit 22 includes a left image frame memory 23 for storing image data of an object to be detected obtained from the optical sensor 8a having a directivity in the upper left direction and an optical sensor 8b having the directivity in the upper right direction. And a right image frame memory 24 for storing detected image data.

Further, the correlation between the left image obtained from the left image frame memory 23 and the right image obtained from the right image frame memory 24 is calculated, and the parallax between the two images and the height (Z coordinate) of the detected object are obtained. The required parallax and height detection circuit 25 (detection circuit) is provided.

Further, reference image generation for generating a reference image based on the left image obtained from the left image frame memory 23, the right image obtained from the right image frame memory 24, and the parallax obtained from the parallax and height detection circuit 25. A circuit 26 and a coordinate extraction circuit 27 that extracts XY coordinates from the reference image are provided.

Then, each information obtained by the sensing image processing unit 22 can be taken out to a CPU or the like not shown via the interface circuit 28.

FIG. 7 is a diagram for explaining a case where a parallax is calculated by performing a correlation calculation using the binarized left image and right image.

7A shows a case where a finger is present as an object to be detected on the liquid crystal display panel 2. FIG.

FIG. 7B shows the image data of the detected object obtained from the optical sensor 8a having the directivity in the upper left direction in black when it is less than a certain threshold, and white (dot pattern) when it is greater than or equal to a certain threshold. On the other hand, FIG. 7C shows the left image binarized so that the image data of the detected object obtained from the optical sensor 8b having the directivity in the upper right direction is less than a certain threshold value. The right image that is binarized so as to be white (dot pattern) above a certain threshold is shown in black.

As shown in FIGS. 7B and 7C, the left image is shifted leftward from the actual finger position, and the right image is shifted from the actual finger position. It is shifted to the right. Therefore, there is a parallax between the left image and the right image.

As shown in FIG. 7D, in order to obtain the parallax, a normalized correlation method is used to set one of the left image and the right image as a reference, and the reference image as 1 The left image and the right image are matched for each line while shifting pixel by pixel, and the correlation value R becomes maximum when the seven pixels are shifted. The parallax corresponds to seven pixels, and the liquid crystal display panel 2 The parallax distance can be calculated based on the pixel pitch.

On the other hand, FIG. 8 is a diagram for explaining a case where the parallax is calculated by matching the left image and the right image of 256 gradations for each line using the normalized correlation method.

As shown in the drawing, when the correlation calculation is performed using the left image and the right image of 256 gradations, and the parallax is calculated, the peak value of the correlation value R is broad, so the parallax is calculated with high accuracy. It is difficult.

In addition, when a 256-gradation image is used for the above-described parallax calculation, the amount of information is large compared to the case where a binarized image is used, and the amount of calculation processing in the correlation calculation is increased by the amount of information. It becomes enormous, and it becomes necessary to provide hardware and software capable of processing a huge amount of calculation at high speed.

FIG. 9 is a diagram for explaining a case where the parallax is calculated by performing a correlation operation for each line using the left image and the right image in which the edge is detected.

FIG. 9A shows a case where a finger is present as an object to be detected on the liquid crystal display panel 2.

FIG. 9B shows the left image of the detected object obtained from the optical sensor 8a having the directivity in the upper left direction using a known filter such as a Sobel filter or a Roberts filter to detect the detected object. It is the image which detected the outline of the finger | toe which is a thing, ie, an edge.

On the other hand, (c) of FIG. 9 filters the right image of the detected object obtained from the optical sensor 8b having the directivity characteristic in the upper right direction by using a known filter such as a Sobel filter or Roberts filter, and detects the detected object. It is an image (edge image) in which the contour line of a finger as an object, ie, an edge is detected.

As shown in FIGS. 9B and 9C, the left image is shifted leftward from the actual finger position, and the right image is shifted from the actual finger position. It is shifted to the right. Therefore, there is a parallax between the left image and the right image.

FIG. 10 illustrates a case where parallax is obtained by performing matching using the normalized correlation method on the left image and the right image in which the edges illustrated in FIGS. 9B and 9C are detected. It is a figure for doing.

10A shows a left image in which an edge is detected, FIG. 10B shows a right image in which an edge is detected, and FIG. 10C shows the right image as a reference. The left image and the right image are matched one line at a time while shifting the reference right image one pixel at a time, and when the seven pixels are shifted, the left image and the right image are detected objects. It shows that the outlines of the fingers just match.

As shown in FIG. 10D, when the correlation value R is obtained while performing matching using the left image and the right image in which the edge is detected, the correlation value R is illustrated in FIG. Compared with the case where a binarized image is used, no ghost is generated, so that the parallax can be calculated with relatively high accuracy.

FIG. 11 is a diagram for explaining a case where parallax is obtained by obtaining an edge direction image of a left image and a right image and performing a correlation calculation including a term relating to the edge direction.

FIG. 11A shows a case where a finger is present as an object to be detected on the liquid crystal display panel 2.

FIG. 11B shows the left image of the detected object obtained from the optical sensor 8a having the directivity in the upper left direction using a known filter such as a Sobel filter or a Roberts filter to detect the detected object. This is an image in which the contour line of a finger, that is, an edge, is detected, and the direction of the edge is also detected. In addition, in order to show the direction of an edge, in FIG.11 (b), the number from 1 to 8 is described by the direction.

On the other hand, (c) of FIG. 11 filters the right image of the detected object obtained from the optical sensor 8b having the directivity characteristic in the upper right direction by using a known filter such as a Sobel filter or Roberts filter, and detects the detected object. This is an image in which the contour line of a finger, that is, an edge, is detected, and the direction of the edge is also detected. In addition, in order to show the direction of an edge, the number of 1-8 is described by (c) of FIG. 11 with the direction.

In addition, in order to obtain the parallax by the correlation calculation including the term related to the edge direction, the following (Formula 1) was used as the correlation function.

In the above (Formula 1), ILi is the i-th edge intensity of the left image, and IRi + j is the i-th edge intensity shifted to the left by j pixels of the right image.

Wdiri is a COS value of an angle formed between the direction of the i-th edge of the left image and the direction of the i-th edge shifted to the left by j pixels of the right image. Therefore, for example, if the directions of the two edges are the same, wdiri is 1, and if the directions of the two edges are different by 90 degrees, wdiri is 0, and if the directions of the two edges are 180 degrees, wdiri is -1.

As described above, when performing the correlation calculation, the left edge of the left image and the left edge of the right image, the right edge of the left image, and the right edge of the right image are weighted by wdiri, which is a term related to the edge direction. If the two match, a high correlation value R can be obtained.

Therefore, when the correlation calculation is performed using the above (Equation 1) to obtain the parallax, the parallax can be obtained with high accuracy.

FIG. 12 shows the parallax d obtained by the method as described above and the incident angle of the reflected light from the detected object (finger) 7 to the optical sensors 8a and 8b, that is, the I layer 9i in the optical sensors 8a and 8b. It is a figure for demonstrating the method of calculating | requiring the height h from the I layer 9i to the to-be-detected object (finger) 7 from the angle (theta) which a perpendicular and the said reflected light make.

The parallax d, the incident angle θ, and the height h shown in FIG. 12 are Tanθ = (d / 2) / h. When this equation is arranged with respect to the height h, h = (d / 2) X (1 / Tanθ).

Therefore, the height h from the I layer 9i (not shown) to the detected object (finger) 7 can be obtained from the parallax d obtained for each line and the incident angle θ which is a set value.

In the present embodiment, the incident angle θ is a straight line connecting the perpendicular from the center of the I layer 9 i in the optical sensors 8 a and 8 b and the center of the I layer 9 i and the center of the opening 14 a of the light shielding member 14. However, the present invention is not limited to this.

The reliability information (reliability) obtained from the parallax and height detection circuit 25 shown in FIG. 6 can be obtained by the following (Equation 2), and the square root of the maximum value Rjmax of the correlation value R is obtained. Normalized by edge strength.

If the reliability is lower than a certain value, information on the height (Z coordinate) can be not displayed.

The correlation between the left image obtained from the left image frame memory 23 and the right image obtained from the right image frame memory 24 by the parallax and height detection circuit 25 shown in FIG. A flow for performing the calculation and a flow for obtaining the parallax between the two images will be described in detail.

Here, as shown in FIG. 11, a case will be described in which an edge direction image of a left image and a right image is obtained, and parallax is obtained by correlation calculation including a term related to the edge direction.

FIG. 13 is a diagram for explaining a flow for obtaining a correlation function for each pixel. FIG. 14 is a diagram illustrating R (j, k) in FIG. 13. HSZ is the number of pixels in the horizontal direction. Indicates the number of pixels in the vertical direction. FIG. 15 is a reference table for obtaining the value of Wdir, which is a term related to the edge direction.

As shown in FIG. 13, at the start of the flow, R (j, k) = 0, k = 0, j = 0, i = 0 are set as initial values.

In S101, the reference table shown in FIG. 15 is referred to determine Wdir, which is a COS value determined by the angle formed by the edge direction of a pixel in the left image and the edge direction of a pixel in the right image.

As shown in FIG. 15, the direction of the edge is described by a number from 1 to 8, and the edge direction in a pixel in the left image is the same as the edge direction in a pixel in the right image. , COS0, and Wdir becomes 1.

Then, in S102, it is determined whether i + j is 0 or more and HSZ-1 or less. If so, Prij is set to PR (i + j, k), otherwise Prij is set to 0.

In S103, first, a correlation function between the upper left pixel (0, 0) of the left image and the upper left pixel (0, 0) of the right image is calculated. In S104, i is determined to be equal to or higher than HSZ. Until the upper left pixel (0, 0) of the left image and all the pixels (0, 0) to (0, HSZ) of the uppermost row of the right image -1) is calculated.

On the other hand, if it is determined in S104 that i is equal to or higher than HSZ, the pixel (0, 1) obtained by shifting the left image by one column in the same row and all the pixels (0, 0) in the uppermost row of the right image. A correlation function with (0, HSZ-1) is calculated.

Then, until it is determined in S105 that j is equal to or higher than HSZ, the left image is shifted by one column in the same row while shifting the left image pixels (0, 2) to (0, HSZ-1) and one of the right image. A correlation function with all the pixels (0, 0) to (0, HSZ-1) in the top row is calculated.

On the other hand, if it is determined in S105 that j is equal to or higher than HSZ, the pixels (1, 0) to (1, HSZ-1) obtained by shifting the left image by one row and all the pixels in the second row of the right image ( Correlation functions with (1, 0) to (1, HSZ-1) are calculated.

In S106, the correlation function between all the left image pixels and all the right image pixels is calculated in the same manner as described above while shifting the left image line by line until it is determined that k is equal to or greater than VSZ.

Then, when it is determined in S106 that k is equal to or greater than VSZ, the above flow ends.

As described above, in this embodiment, the correlation function for each line of the left image and the right image can be obtained.

Hereinafter, the flow for obtaining the parallax by obtaining the maximum value of the correlation function obtained as described above will be described based on FIG. 16 and FIG.

FIG. 16 is a diagram for explaining a flow for obtaining the maximum value of the correlation function for each line, and FIG. 17 is a diagram showing R (j, k) and parallax z (k) in FIG. .

As shown in FIG. 16, with the start of the flow, z (k) = 0, k = 0, and i = 0 are set as initial values.

If it is determined in S201 that k (z) <R (j, k), the upper left pixel (0, 0) of the left image and all of the uppermost row of the right image The maximum value of the correlation function with the pixels (0, 0) to (0, HSZ-1) is obtained.

In step S202, each pixel (0, 1) to (0, HSZ-1) of the left image and the right image are shifted while shifting the left image by one column in the same row until j is determined to be equal to or higher than HSZ. The maximum value of the correlation function with all the pixels (0, 0) to (0, HSZ-1) in the top row is obtained.

If it is determined in S201 that k (z) ≧ R (j, k), the left image is shifted by one column within the same row without obtaining the maximum value of the correlation function as described above.

If it is determined in S202 that j is equal to or higher than HSZ, each pixel (1, 0) to (1, HSZ-1) obtained by shifting the left image by one row and all of the second row of the right image The maximum value of the correlation function with the pixels (1, 0) to (1, HSZ-1) is obtained.

Then, in S203, the maximum value of the correlation function between all the left image pixels and all the right image pixels is set in the same manner as described above while shifting the left image line by line until it is determined that k is equal to or greater than VSZ. Ask.

As described above, the parallax z (k) can be obtained by obtaining the maximum value of the correlation function for each line.

[Embodiment 2]
Next, a second embodiment of the present invention will be described based on FIG. The present embodiment is different from the first embodiment in that an optical element that limits the incident angle to the optical sensors 8a and 8b within a certain range is provided, and other configurations are the same as in the first embodiment. 1 as described above. For convenience of explanation, members having the same functions as those shown in the drawings of the first embodiment are given the same reference numerals, and descriptions thereof are omitted.

FIG. 18 is a diagram showing a schematic configuration of an optical sensor provided with an optical element that limits the incident angle within a certain range.

As shown in FIG. 18A, in the optical sensor 8c having directivity in the upper left direction in the drawing, it is an upper layer from the I layer 9i that is a light receiving portion, and the opening 14a of the light shielding member 14 An optical element 29a that restricts the incident angle to the I layer 9i that is the light receiving portion within a certain range is provided at a position overlapping in plan view.

On the other hand, as shown in FIG. 18B, in the optical sensor 8d having directivity characteristics in the upper right direction in the drawing, it is an upper layer than the I layer 9i that is the light receiving portion, and the opening portion of the light shielding member 14 An optical element 29b that limits the incident angle to the I layer 9i that is the light receiving portion within a certain range is provided at a position overlapping with 14a in plan view.

According to the above configuration, the liquid crystal display device 1 can improve the blur of incident light incident on the I layer 9i that is a light receiving unit, and can estimate the distance of the detection object with high accuracy. Can be realized.

In the area sensor of the present invention, the first image of the detected object detected by the plurality of first light receiving elements and the second image of the detected object detected by the plurality of second light receiving elements. Is detected at the same timing, performs a correlation operation between the first image and the second image, and based on the calculated parallax between the first image and the second image, It is preferable that a detection circuit for detecting the height of the detected object from the surface is provided.

According to the configuration, the first object of the detected object is based on the parallax of the image of the detected object detected by the first light receiving element and the second light receiving element having different directivity characteristics. The height from the surface on which the light receiving element and the second light receiving element are formed can be detected.

Therefore, it is possible to realize an area sensor that can estimate the distance with higher accuracy than the method of estimating the distance of the detected object using only the reflection intensity of the detected object.

Further, according to the above configuration, it is possible to estimate the distance even when instantaneous fluctuations occur in the external light.

In the area sensor of the present invention, each of the first image and the second image is a binarized image, and the parallax is calculated by a correlation operation of the binarized image. Is preferred.

For example, when a 256-gradation image is used for calculating the parallax, the amount of information is large compared to the case where the binarized image is used, and the amount of calculation processing in the correlation calculation is larger by the amount of information. Therefore, it becomes necessary to provide hardware and software that can process a huge amount of calculation at high speed.

On the other hand, according to the above configuration, since the binarized image is used for the calculation of the parallax, the amount of calculation processing in the correlation calculation can be reduced.

In the area sensor of the present invention, the first image and the second image are edge images obtained by extracting contour lines of the detected object, and the parallax is calculated by correlation calculation of the edge images. It is preferable.

For example, when a 256-gradation image is used for the above-described parallax calculation, the portion where the pattern between images is the best match and the correlation value (similarity) is the largest is broad, so the accurate parallax is calculated. In some cases, it is impossible to estimate the distance of the detected object with high accuracy.

In addition, when a binarized image is used for the above-described parallax calculation, a ghost peak occurs on both sides of a portion where the pattern between images most closely matches and the correlation value (similarity) is maximum. The parallax calculation accuracy is reduced.

On the other hand, according to the above configuration, since the edge image obtained by extracting the contour line of the detection object is used for the calculation of the parallax, the correlation value (similarity) having the highest matching pattern between the images is maximized. Since the portion to be shown can be obtained relatively sharply, an area sensor that can calculate an accurate parallax and can estimate the distance of the detected object with high accuracy can be realized.

In the area sensor of the present invention, the first image and the second image are edge images obtained by extracting contour lines of the detected object, and the parallax is obtained by correlation calculation including a term related to the direction of the edge. Is preferably calculated.

According to the above configuration, for example, when the direction of the edge of the detection object in the first image matches the direction of the edge of the detection object in the second image, the correlation value (similarity) The parallax is calculated by a correlation calculation including a term related to the edge direction set so as to increase.

Therefore, since the portion where the pattern between images most closely matches and the correlation value (similarity) is maximum can be obtained relatively sharply, accurate parallax can be calculated, and the distance of the detected object with high accuracy can be calculated. An area sensor that can perform estimation can be realized.

In the area sensor of the present invention, a reference image generation circuit that generates a reference image based on the first image, the second image, and the parallax between the first image and the second image. And a coordinate extraction circuit for extracting coordinates from the reference image.

According to the above configuration, not only the distance of the detection object, that is, the coordinates relating to the height of the detection object from the surface on which the first light receiving element and the second light receiving element are formed, The coordinates on the surface on which the first light receiving element and the second light receiving element are formed can also be extracted from the reference image.

Therefore, for example, the area sensor can be used as information input means for three-dimensional coordinates (XYZ coordinates).

In the area sensor of the present invention, a position that is an upper layer than the first light receiving unit and the second light receiving unit and overlaps the opening of the first light shielding member and the opening of the second light shielding member in plan view It is preferable that an optical element for limiting an incident angle to the first light receiving unit and the second light receiving unit within a certain range is provided.

According to the above configuration, the incident angle to the first light receiving unit and the second light receiving unit at a position overlapping the opening of the first light blocking member and the opening of the second light blocking member in plan view. Since an optical element that restricts the light intensity within a certain range is provided, it is possible to improve blurring of incident light incident on the first light receiving unit and the second light receiving unit, and further, high accuracy. An area sensor capable of estimating the distance of the object to be detected can be realized.

In the display device of the present invention, an active matrix substrate having a surface on which the first light receiving element and the second light receiving element are formed, and the first light receiving element and the second light receiving element are formed. It is preferable that a counter substrate disposed so as to be opposed to the surface being provided, and a liquid crystal layer sandwiched between the active matrix substrate and the counter substrate.

According to the above configuration, a liquid crystal display device in which the area sensor is integrated can be realized. Therefore, information can be input using the area sensor while viewing the image of the liquid crystal display device.

The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and the present invention can be obtained by appropriately combining technical means disclosed in different embodiments. Embodiments are also included in the technical scope of the present invention.

The present invention can be applied to an area sensor including a light receiving element (light sensor) and a display device including such an area sensor.

1 Area sensor integrated liquid crystal display device (display device)
2 LCD panel 3 Backlight 7 Finger (object to be detected)
8a, 8c Optical sensor (first light receiving element)
8b / 8d optical sensor (second light receiving element)
9i light receiving part (first light receiving part, second light receiving part)
14 light shielding member 14a opening 25 parallax and height detection circuit 26 reference image generation circuit 27 coordinate extraction circuit

Claims (9)

1. An area sensor having a plurality of surfaces formed by combining a first light receiving element and a second light receiving element that detect the amount of incident light.
   The first light receiving element includes a first light receiving portion and a first light shielding member that is formed in an upper layer than the first light receiving portion and blocks the light.
   The second light receiving element includes a second light receiving portion and a second light shielding member that is formed above the second light receiving portion and blocks the light.
   In the first light receiving element, the light component from the first direction can be received, and the second direction is opposite to the direction of the reflected light when the light from the first direction is specularly reflected by the surface. The opening of the first light shielding member is on the first direction side so as to partially overlap the first light receiving portion in plan view so that the light component from the direction can be shielded. It is formed by shifting a predetermined amount,
   In the second light receiving element, the opening of the second light shielding member is configured to receive the light component from the second direction and to shield the light component from the first direction. An area sensor characterized by being formed by being shifted a predetermined amount toward the second direction so as to partially overlap the two light receiving portions in plan view.
2. The first image of the detected object detected by the plurality of first light receiving elements and the second image of the detected object detected by the plurality of second light receiving elements are detected at the same timing. ,
   The correlation between the first image and the second image is calculated, and the height of the detected object from the surface is calculated based on the calculated parallax between the first image and the second image. The area sensor according to claim 1, further comprising: a detection circuit for detecting
3. The first image and the second image are binarized images, respectively.
   The area sensor according to claim 2, wherein the parallax is calculated by a correlation calculation of the binarized image.
4. The first image and the second image are edge images obtained by extracting outlines of the detection object,
   The area sensor according to claim 2, wherein the parallax is calculated by a correlation calculation of the edge image.
5. The first image and the second image are edge images obtained by extracting outlines of the detection object,
   The area sensor according to claim 2, wherein the parallax is calculated by a correlation calculation including a term relating to the direction of the edge.
6. A reference image generation circuit that generates a reference image based on the first image, the second image, and the parallax between the first image and the second image;
   The area sensor according to claim 2, further comprising a coordinate extraction circuit that extracts coordinates from the reference image.
7. It is an upper layer than the first light receiving part and the second light receiving part,
   In a position overlapping the opening of the first light shielding member and the opening of the second light shielding member in plan view,
   7. The optical element according to claim 1, further comprising an optical element that limits an incident angle to the first light receiving unit and the second light receiving unit within a certain range. area sensor.
8. The area sensor according to any one of claims 1 to 7, comprising:
   On the surface on which the first light receiving element and the second light receiving element are formed, a plurality of pixels that are formed in a matrix and display an image are provided.
   The display device, wherein the first light receiving element and the second light receiving element are provided in accordance with the position of the pixel.
9. An active matrix substrate having a surface on which the first light receiving element and the second light receiving element are formed;
   A counter substrate disposed to face a surface on which the first light receiving element and the second light receiving element are formed;
   The display device according to claim 8, further comprising a liquid crystal layer sandwiched between the active matrix substrate and the counter substrate.

Images