JP2011259089A - Image display device, electronic equipment, image display system, image acquisition method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress an influence which appears on an image being caused by a diffraction phenomenon generated at a light transmission section when imaging an object on a display surface side through a minute light transmission section of a display section by using an imaging device disposed on a rear surface.SOLUTION: An image display device 1A includes a diffraction correction section 100 that suppresses an influence appears on image information, acquired by an imaging device 20 through a plurality of the light transmission sections of an image display section 10, due to a diffraction effect of a light transmission section, by approaching from a side of a signal processing. By limiting a processing object only to a part of signal components composing an image, the diffraction correction section 100 performs a processing at a higher speed than in a processing of all the signal components. For example, image information of a specific area can be defined as a processing object. Further, only one signal of a plurality of signals representing image information can be defined as a processing object. By limiting a processing object only to a signal component having a strong correlation with luminance information (typically a green component) or only to luminance information, a speed of a correction processing can be increased while avoiding a reduction of a correction effect.

Description

  The present invention relates to an image display device, an electronic apparatus, an image display system, an image acquisition method, and a program. More specifically, the present invention relates to a mechanism for imaging an object on the display surface side by disposing an imaging device on the back side of the image display unit.

  Attempts to add functions other than image display to an image display device by combining an image display device and an imaging device are being actively made (see, for example, Patent Documents 1 and 2).

JP 2005-176151 A JP-A-2005-010407

  In the technique disclosed in Patent Document 1, a plurality of openings having microlenses are provided between pixels constituting an image display unit of an image display device, and light passing through the plurality of openings is captured by a plurality of cameras. To do. Here, the face of the user who looks at the image display device is imaged from a plurality of different angles, and the obtained plurality of images are processed to generate an image capturing the user from the front.

  In the technique disclosed in Patent Document 2, for example, as shown in FIGS. 15 and 16, an image is picked up by an image pickup device based on light that has passed through one light transmitting portion provided in a plurality of pixels. To do.

  In the technique disclosed in Patent Document 1, when the light transmission part is very small, a diffraction phenomenon occurs in the light transmission part. As a result, the image formed on the imaging device is blurred, and the captured image becomes clear due to the influence. It may be missing.

  Further, in the technique disclosed in Patent Document 1, it is necessary to provide a microlens at the opening, and a high-precision microlens is required to accurately connect an image to the imaging device. This increases the manufacturing cost of the image display device. In addition, the front face of the user is not captured, but a front image is created from a plurality of images captured from different angles, so that a so-called CG (computer graphics) image is provided to the other party instead of a real image. Therefore, the feeling of strangeness is actually great.

  Similarly to the technique disclosed in Patent Document 1, the technique disclosed in Patent Document 2 also causes a diffraction phenomenon to occur in the light transmitting part when the light transmitting part is very small. A blur may occur, and the captured image may lack clarity due to the influence thereof. In addition, since an image is picked up by the image pickup device based on the light that has passed through one light transmission portion provided in the plurality of pixels, it is difficult to collect a sufficient amount of light on the image pickup device.

  The present invention has been made in view of the above circumstances, and when a subject on the display surface side is imaged by an imaging device disposed on the back surface through a minute light transmission portion of the display portion, a diffraction phenomenon occurs in the light transmission portion. The main purpose is to provide a mechanism that can suppress the effects that appear in the image.

  It is another object of the present invention to provide a mechanism that can be manufactured at low cost and can easily acquire a user's image that faces the image display unit.

  It is another object of the present invention to provide a mechanism capable of concentrating a sufficient amount of light on the imaging apparatus.

  According to one embodiment of the present invention, a plurality of pixels including a display element are arranged, and an imaging unit (or imaging device: hereinafter the same in this section) that captures an image can be disposed on the back side. And an image display unit in which a plurality of light transmission units are provided in a corresponding region, and image information acquired by the imaging unit through the plurality of light transmission units, the influence that appears in the image information due to a diffraction effect by the light transmission unit A diffraction correction unit that performs a diffraction correction process for suppressing the above. The influence on the image due to the diffraction phenomenon occurring in the light transmission part is suppressed by coping with the aspect of signal processing.

  Here, the diffraction correction unit aims at speeding up the processing by processing only a part of the signal components constituting the image as a target of the diffraction correction processing, compared with the case of processing all the signal components. Although how to determine “part of the signal component” can take various forms, in the present invention, first, a method that targets only image information of a specific region extracted from image information acquired by the imaging unit By reducing the area to be processed, the processing speed is increased. In this case, by processing the area of a predetermined specific object (a person, a person's face, a custom-made part of a person's face, etc.) as a specific area, the sensory correction effect is not deteriorated. It is better to increase the processing speed.

  More preferably, the diffraction correction processing is performed on at least one (but not all) of a plurality of signals representing the image information acquired by the imaging unit. In this case, the diffraction correction process is performed only on the signal component (typically the green component) that has a strong correlation with the luminance information or only the luminance information, thereby speeding up the correction process while avoiding a reduction in the correction effect. It is good to plan.

  Preferably, a wavelength distribution measurement unit that measures the wavelength distribution of external light is further provided, and the diffraction correction processing is performed with reference to the wavelength distribution of external light measured by the wavelength distribution measurement unit.

  Preferably, the diffraction phenomenon is suppressed from the side surface of the light transmission part itself. For example, it is preferable to make the arrangement state (the size, shape, distribution (arrangement pitch), etc., of the light transmission part) of the light transmission part in the light transmission region corresponding to the imaging part non-uniform. Alternatively, the light transmission part may be constituted by the first light transmission part and the second light transmission part, and the second light transmission part may be arranged so as to surround the first light transmission part.

  More preferably, the light passing through the light transmission part is disposed between the image display part and the imaging part so that the light passing through the light transmission part can be reliably collected in the imaging part. It is good to concentrate.

  According to one aspect of the present invention, an imaging unit (or imaging device: hereinafter the same in this section) is arranged on the back side of the image display unit, and the subject on the display surface side is imaged through the light transmission unit. It is possible to easily acquire the image of the user who is responding.

  At this time, the influence on the image due to the diffraction phenomenon occurring in the light transmitting portion can be suppressed by coping with the aspect of signal processing. At this time, by processing only the image information of the specific area instead of the entire area of the image, the processing time can be shortened compared with the case of processing all the signal components.

  Preferably, if only a part of a plurality of signal components constituting the image is subjected to diffraction correction processing, the processing time can be shortened compared with the case where all signal components are processed. In this case, if only the luminance signal component or the signal component having a strong correlation with the luminance (for example, the green component) is subjected to the diffraction correction processing, the processing time can be shortened and the processing is performed for all the signal components. Correction processing can be performed with the same (equivalent) accuracy.

  More preferably, if the countermeasure from the side surface of the light transmission portion itself arranged in the light transmission region is also taken, it is possible to prevent the diffraction phenomenon from occurring in the light transmission portion in advance. As a result, the influence on the image due to the diffraction phenomenon can be more reliably suppressed.

  Preferably, if a condensing unit is arranged between the image display unit and the imaging unit, light that has passed through the light transmission unit is reliably condensed on the imaging unit, so that a sufficient amount of light is applied to the imaging unit. Can be condensed. In order to accurately tie an image to the imaging surface of the imaging unit, a high-precision microlens is not required, and the manufacturing cost of the image display device is not increased, and can be manufactured at a low cost.

FIG. 1 is a conceptual diagram of an image display device and an image display system according to the first embodiment. FIG. 2 is a schematic diagram of the most typical arrangement of a plurality of pixels constituting the image display unit. FIG. 3 is a conceptual diagram illustrating the relationship between the arrangement position of the imaging device and the displayed image. FIG. 4 is a diagram illustrating a captured image by the image display device. FIG. 5 is a diagram illustrating details of the image display unit. FIG. 6 is a schematic diagram for explaining the influence of the diffraction phenomenon on the captured image. FIG. 7 is a diagram illustrating an example of an image captured by placing a glass plate in front of the imaging device. FIG. 8 is a block diagram of the image display apparatus according to the first embodiment. FIG. 9 is a block diagram of the image display system of the first embodiment. FIG. 10 is a conceptual diagram for obtaining the diffracted light intensity and the MTF from the light transmission part (opening part). FIG. 11 is a diagram (part 1) schematically illustrating an example of the shape of the light transmission portion. FIG. 12 is a diagram (part 2) schematically illustrating an example of the shape of the light transmission portion. FIG. 13 is a conceptual diagram of an image display device and an image display system according to the second embodiment. FIG. 14 is a block diagram of an image display apparatus according to the second embodiment. FIG. 15 is a block diagram of an image display system according to the second embodiment. FIG. 16 is a diagram illustrating a first example of diffraction correction processing. FIG. 17 is a diagram illustrating a second example of diffraction correction processing. FIG. 18 is a diagram illustrating a third example of diffraction correction processing. FIG. 19 is a diagram illustrating a fourth example of diffraction correction processing. FIG. 20 is a diagram illustrating a fifth example of diffraction correction processing. FIG. 21 is a diagram illustrating a first modification (No. 1 & 2) of the light transmission portion. FIG. 22 is a diagram illustrating a first modification (No. 3) and a second modification of the light transmission portion. FIG. 23 is a diagram illustrating a third modified example (No. 1 & 2) of the light transmissive portion. FIG. 24 is a diagram illustrating an example of an electronic apparatus to which the image display device of this embodiment is applied. FIG. 25 is a diagram illustrating a first modification of the image display device. FIG. 26 is a diagram illustrating a first modification of the image display system. FIG. 27 is a diagram illustrating a second modification of the image display device. FIG. 28 is a diagram illustrating a second modification of the image display system. FIG. 29 is a diagram illustrating a third modification (No. 1) of the image display device and the system. FIG. 30 is a diagram illustrating a third modification (No. 2) of the image display device and the system. FIG. 31 is a diagram illustrating a fourth modification of the image display device. FIG. 32 is a diagram illustrating a fourth modification of the image display system.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. When distinguishing each functional element according to the embodiment, an uppercase English reference such as A, B,... Is added and described, and when not particularly described, this reference is omitted. To describe. The same applies to the drawings. Various numerical values and materials in each embodiment are examples, and are not limited thereto.

The description will be made in the following order.
1. Basic concept (overall outline, diffraction correction processing, wavelength distribution measurement, light transmission region, imaging device, image display unit)
2. First embodiment (overall outline, arrangement position of imaging device, cross-sectional structure of image display unit, configuration corresponding to diffraction correction, principle of countermeasures for diffraction phenomenon)
3. Second embodiment (first embodiment + wavelength distribution measurement process)
4). Diffraction correction processing (first example: method for reducing processing area)
5). Diffraction correction processing (2nd example: 1st example + all signals)
6). Diffraction correction processing (third example: first example + all signals + color conversion)
7). Diffraction correction processing (fourth example: first example + specific color signal → signal correlated with luminance information)
8). Diffraction correction processing (fifth example: first example + luminance information, summary of first to fifth examples)
9. Modified example of light transmission part First modified example (random arrangement)
Second modification (double annular structure)
3rd modification (cross-girder shape, L-shape, etc.)
Ten. Alternative to electronic equipment monitoring devices
11． Summary (first modification: position detection, second modification: 3D image display + position detection, third modification: TV conference system, fourth modification: digital mirror)

<Basic concept>
[Overview]
In the image display device with an image pickup device (hereinafter simply referred to as “image display device”) of the present embodiment, an image pickup device (or image pickup unit: the same applies hereinafter) is disposed on the back of the image display unit, and the image display unit A plurality of minute light transmission portions are provided in the light transmission region corresponding to the imaging device. Then, the subject on the display surface side is imaged through each light transmitting portion.

  The light that has passed through the light transmission part is condensed on the imaging device. Since the imaging device is arranged on the back side of the image display unit, it is possible to accurately capture the face, eyes, movements, etc. of the user facing the display with the imaging device. By providing the image display unit (display panel of the display device) with a light transmitting part that allows light to reach the back surface and installing the imaging device at the corresponding position, the face and eyes of the user facing the display Since the operation and the like can be accurately grasped by imaging with the imaging device, the added value of the display device can be increased easily and inexpensively.

  Although not essential, the light condensing unit is disposed between the image display unit and the imaging device so that the light that has passed through the light transmitting unit is reliably condensed on the imaging device. By arranging the condensing unit, a high-precision minute lens is not required to accurately connect the image to the imaging device, and the manufacturing cost of the image display device is not increased, and the imaging device has a sufficient amount of light. Light can be collected.

  Here, in the image display device according to the present embodiment, when the aperture of the light transmission portion constituting the light transmission region is small, when an image is captured through the aperture, a so-called diffraction phenomenon occurs, resulting in blurring of the image formed on the imaging device. May occur or lack of clarity.

  As a countermeasure, in the present embodiment, first, from the aspect of signal processing, the influence of the diffraction phenomenon generated in the light transmission portion on the captured image is corrected. Preferably, the occurrence of diffraction phenomenon is suppressed from the side surface (for example, shape, size, distribution, etc.) of the light transmission part itself.

[Diffraction correction processing]
From the aspect of signal processing, the image display apparatus and the image display system of the present embodiment are configured to include a diffraction correction unit in order to correct the influence of the diffraction phenomenon that has occurred in the light transmission unit. Such a configuration is referred to as a “configuration of the first embodiment”.

  The diffraction correction unit corrects diffraction generated in the light transmission unit constituting the light transmission region with respect to the captured image information and other image data acquired via the imaging device. Incidentally, “other image data” is image data to be displayed on an image display unit obtained by imaging a user or the like with an imaging device, for example. Hereinafter, the captured image information and “other image data” may be collectively referred to as “image data”.

For example, part or all of the light transmission part (opening thereof) is periodically provided along the first direction and the second direction of the image display part. At this time, when the length of the light transmission part along the first direction is L _tr-1 and the pitch of the pixels along the first direction is P _px-1 , the line aperture ratio L in the first direction _tr-1 / P _px-1 may have to be satisfied to "L _tr-1 / P _px-1 ≧ 0.5", more preferably "L _tr-1 / P _px-1 ≧ It is better to satisfy “0.8”. The upper limit value of the line aperture ratio L _tr-1 / P _px-1 is not particularly limited as long as a light transmission part can be formed. Here, the length L _tr-1 of the light transmission part along the first direction is the length per one cycle of the line segment corresponding to the shape when the light transmission part is projected in the first direction. , And the pixel pitch P _px-1 along the first direction means the length per one period of the pixels along the first direction.

Further, when the length of the light transmission part along the second direction is L _tr-2 and the pixel pitch along the second direction is P _px-2 , the line aperture ratio L _{tr in} the second direction is set. ₋₂ / P _px-2 should satisfy “L _tr−2 / P _px−2 ≧ 0.5”, and more preferably “L _tr−2 / P _px−2 ≧ 0”. .8 "should be satisfied. The upper limit value of the line aperture ratio L _tr-2 / P _px-2 is not particularly limited as long as a light transmission part can be formed. Here, the length L _tr-2 of the light transmission part along the second direction is the length per cycle of the line segment corresponding to the shape when the light transmission part is projected in the second direction. The pixel pitch P _px-2 along the second direction means the length per one period of the pixels along the second direction.

  The first direction and the second direction may be orthogonal to each other, and in some cases, may intersect at an angle other than 90 degrees. In the latter case, a part or all of the light transmission part is periodically provided not only in the first direction and the second direction of the image display part but also in the third direction, the fourth direction,. In such a case, the length of the light transmission part along at least two directions in each direction and the pitch of the pixels along these at least two directions are related to each other (specifically, In terms of 0.5 times or more, more preferably 0.8 times or more).

  As a diffraction correction process in the diffraction correction unit, an MTF (absolute value), which is an amplitude (absolute value), is used using a response function (indicating spatial frequency characteristics) that evaluates the sharpness (resolution) of an image acquired by the imaging apparatus. Modulation transfer function) may be used as an index value. As is well known, the response function is obtained by Fourier transforming a point image or line image intensity distribution. The influence of the reduction in resolution (the degree of blurring of the image) that appears in the image by imaging with the imaging device through the light transmission part corresponds to the arrangement pattern of the light transmission part (micro opening). For example, the influence of resolution reduction (the degree of image blur) can be evaluated by the magnitude of the high-frequency component in the MTF.

  Therefore, if the inverse Fourier transform is performed on the captured image with reference to the arrangement pattern of the light transmitting portion, the image in which the influence of the resolution reduction appearing in the image is eliminated can be restored by imaging with the imaging device through the light transmitting portion. . That is, the image information acquired by the imaging device is Fourier transformed, and the Fourier transformed information is subjected to inverse Fourier transform using a response function (in this case, MTF) corresponding to the arrangement state of the light transmission unit, thereby correcting diffraction. Processing should be done.

  For example, it is preferable to perform MTF inverse transform processing (image restoration processing) calculated based on the arrangement pattern (shape, size, distribution, etc.) of the light transmitting portion on the image information. In the MTF inverse transform process, the MTF inverse transform process is performed on the signal component of the captured image using the MTF shape data. In this case, a storage unit may be provided in the diffraction correction unit, and MTF shape data representing an arrangement pattern such as the shape, size, and distribution of the light transmission unit may be stored in the storage unit in advance. The response function may be specified by applying a Fourier transform method that performs Fourier analysis on the image (a signal representing the image).

  The diffraction correction processing technique shown here is merely an example of a technique for correcting the influence of the reduction in resolution (the degree of blurring of the image) that appears in the image by taking an image with the imaging device through the light transmission unit. Techniques may be applied.

  For example, it is conceivable that the diffraction correction unit is provided in an electronic device equipped with an image display device or an image display device as a circuit including a CPU having an input / output unit and a memory. Further, when the image display device and the image pickup device are configured separately (detachable), it is conceivable to provide a diffraction correction unit in the image pickup device (an example of an electronic device) disposed on the back surface of the image display device. In an image display system in which a peripheral device such as a personal computer (electronic computer) is connected to the image display device, it is conceivable that the peripheral device has a function of a diffraction correction unit.

  In performing MTF inverse transform processing on image information, a method for processing each signal component (for example, R, G, B color image information) or a part of the signal component is processed. A method for reducing the correction processing time can be considered.

  In the former case, since all the signal components constituting the image information are subjected to the MTF inverse transform process, a processing load is applied, but the correction effect is higher than that of the latter. On the other hand, in the latter case, since the MTF inverse transform process is performed on a part of the signal components constituting the image information, the correction effect is lower than the former, but the processing load is reduced.

  In the latter case, in this embodiment, first, a method of processing a signal component of a part of the region instead of the signal component of the entire region of the captured image acquired by the imaging device (referred to as a “processing area reduction method”). ) To reduce the correction processing time. That is, the MTF inverse transform process is performed on some signal components from the side of the processing area.

  In the processing area reduction method, a partial region of an image to be subjected to MTF inverse transform processing is referred to as a “specific region”, and a region obtained by excluding the “specific region” from the entire region of the captured image is referred to as a “background region”. . The captured image is separated into a “specific region” and a “background region”, and only the “specific region” is subjected to the MTF reverse conversion process. Thereafter, the image of the “specific region” that has undergone diffraction correction (its signal component) and the image of that “background region” that has not been subjected to diffraction correction processing (its signal component) are combined to complete the processing.

  The “specific area” separated (extracted) from the entire area of the captured image has a smaller area than the entire captured image, so the amount of calculation is less than when processing the entire area of the captured image. Since the correction processing time is shortened, the processing can be speeded up.

  Various viewpoints can be considered as to how to set the “specific region”. For example, a method of defining from the side of a place such as the center or corner of a captured image (referred to as a “place-oriented method”) and a method of prescribing a specific (notable) object such as a user. Can be broadly divided into methods (called “subject-oriented methods”). In the latter case, compared to the case where the diffraction correction process is performed on the entire screen, for example, the diffraction correction process is performed only on the human image, so that high-speed processing is possible.

  In the place-oriented method, a place to be extracted from image information acquired by the imaging apparatus may be determined in advance, or some candidate places may be presented so that the user can select. In the place-oriented technique, it is further conceivable to define a shape (for example, a rectangle or a circle) when extracting the “specific region” from the captured image. The shape to be extracted may be determined in advance, or some candidate shapes may be presented so that the user can select them.

  In the diffraction correction process, as described below, as an example, fast Fourier transform (FFT) and its inverse transform (IFFT) are executed. Considering the ease of FFT processing and IFFT processing, it is preferable that the region shape when extracting the “specific region” is a rectangle, and only the method of determining “rectangle” in advance may be used.

  In the method emphasizing the subject, it is conceivable to adopt a method (referred to as “a method emphasizing a specific part”) that is defined from the side of a part of the subject. For example, when trying to sharpen a person image from a blurred image (unclear image with degraded resolution) by the light transmission part, even if the whole person is not clear, the entire face, or even the eyes, nose, There is no sense of incongruity even when only a specific part representing a facial feature such as a mouth is sharpened. By doing so, the area ratio of the image to be processed becomes smaller, so that the processing can be performed at a higher speed, and the sensory correction effect is not deteriorated although the correction effect is actually inferior.

  In the method of emphasizing the subject (including the method of emphasizing a specific part), when extracting the specified subject region from the entire image, a method of faithfully extracting along the contour can be adopted, but FFT processing or IFFT Considering the ease of processing, it is preferable that the region shape when extracting the portion of the subject (including the case of a portion of the subject) is a rectangle.

  Subject-oriented methods (including a method that emphasizes a specific part) require processing to extract (specify) a subject and a part of a specific part from the entire image, and generally the processing load becomes heavy. On the other hand, the place-oriented method allows the “specific region” extracted from the entire image to have a predetermined shape, and thus the extraction process is simple.

  In the case of adopting a method that emphasizes a subject (including a method that emphasizes a specific part), it is also conceivable to cause the “specific region” to follow the movement of the subject (including the specific part).

  It is conceivable to combine a place-oriented technique and a subject-oriented technique (including a technique that emphasizes a specific part). For example, the location-oriented method is first adopted and the central portion of the captured image is set as a processing target, for example. ) To follow the “specific area”. Even when the subject protrudes from the center of the screen, it is possible to reliably execute correction processing that places importance on the subject (including a specific part) to be noted.

  More preferably, in the present embodiment, a method of processing at least one (not all) representing each signal component (referred to as a “specific signal emphasis method”) is also used. That is, when performing the diffraction correction process, only an arbitrary signal component is corrected using the corresponding color light or luminance or the point spread function of the arbitrary color light. Thereafter, the signal component that has been subjected to diffraction correction and the remaining signal component that has not been subjected to diffraction correction processing are combined to complete the processing. Since the number of signal components subject to correction processing is smaller than the number of all signal components representing an image, the amount of calculation is reduced compared to the case where all signal components are subject to diffraction correction processing, and correction processing is performed. Since time is shortened, processing can be speeded up.

  In the case of a method emphasizing a specific signal, for example, it is possible to consider processing that focuses on one of the colors (for example, R, G, B), but even in that case, in particular, the correlation between the luminance signal component and the luminance signal component It is preferable to perform processing while paying attention to (relatively strong) color image information. Further, the signal component of the image information to be processed and the component of the MTF shape data used for the inverse transformation are not necessarily the same type, and those having a strong correlation between them can be used. The speed of correction processing can be increased while maintaining a substantial (appearance) correction effect at the same level as when all signals are targeted.

  For example, it is generally known that human visual characteristics are particularly sensitive to brightness (brightness and darkness), and that the luminance component has a strong correlation with the green component of the image. It is conceivable to perform an MTF inverse transform process using the sensitivity to the luminance of. That is, the MTF inverse conversion process is performed only for the luminance component of the captured image.

  According to such a method focused on human visual characteristics, it was confirmed that the restored image is comparable in terms of blur compared to the image restored to all the signal components constituting the image information. . Furthermore, the processing time can be shortened compared with the case where the restoration processing is performed on all the signal components. That is, the diffraction correction process can be speeded up.

  When performing a process focusing on the luminance component, the MTF shape data used for the inverse transformation may adopt a method that uses the MTF shape data faithfully corresponding to the luminance component, or corresponds to the green component having a strong correlation with the luminance component. A method may be used in which the MTF shape data to be substituted is used. In the latter case, when performing the diffraction correction process, the correction processing time is shortened by correcting only the luminance signal component using the point spread function of green light. This is because the human component of visual characteristics occupies a large proportion of the luminance component because it is the green component, so that the MTF shape data of the green component may be substituted for the MTF inverse conversion processing of the luminance component. Since the MTF shape data of the green component obtained by imaging is substituted, it is not necessary to obtain the MTF shape data of the luminance component, and the processing load is reduced.

  In the present embodiment, the user can select a desired one from these various diffraction correction processing methods in terms of processing speed and correction accuracy.

[Wavelength distribution measurement]
The image display device and the image display system of the present embodiment can be configured to further include a wavelength distribution measurement unit that measures the wavelength distribution of external light. Such a configuration is referred to as a “configuration of the second embodiment”.

  By adopting the configuration of the second embodiment, in the diffraction correction process, not only the shape, size, and distribution of the light transmission part but also the wavelength distribution of the external light can be considered, and the accuracy of the MTF reverse conversion process is improved. Therefore, an optimum image can be obtained regardless of outside light (external illumination environment). In addition, it is possible to improve the accuracy of image information acquired via the imaging device (for example, improve the accuracy of color information).

  The wavelength distribution measuring unit can be constituted by a light receiving device such as a photosensor, for example. The control of the wavelength distribution measuring unit may be provided in, for example, an image display device or an electronic device equipped with the image display device as a control circuit including a CPU having an input / output unit and a memory. Further, in an image display system in which a peripheral device such as a personal computer (electronic computer) is connected to the image display device, it is conceivable that the peripheral device has a function of controlling the wavelength distribution measuring unit.

[Light transmission area]
The light transmission region provided in the image display unit of the image display device of the present embodiment may be formed in a portion corresponding to a portion where the imaging device is provided. In the light transmissive region, a plurality of light transmissive portions in which minute openings through which light representing the subject image on the display surface side passes are preferably provided.

  Here, the light transmission region may be a first configuration example in which a light transmission portion is provided in a plurality of pixels. The light transmission region may be a second configuration example in which a light transmission part is provided around at least one or more (preferably at least two) pixels. In this case, the light transmission part is May be provided all around the pixel, or may be provided on a part of the periphery of the pixel (specifically, on two or more consecutive sides of the side corresponding to the boundary of the pixel). In the latter case, a light transmission region is provided over a length of 1/4 times or more of the entire circumference of the pixel (in the case of two consecutive sides, it is 1/2 times the length of each side). It is preferable.

  In such a configuration, the light that has passed through the light transmission portions provided in the plurality of pixels is collected on the imaging device, or passes through the light transmission portions provided around at least one pixel. The collected light is collected on the imaging device. Therefore, a high-precision microlens is not required to accurately connect the image to the imaging device, and the manufacturing cost of the image display device with the imaging device is not increased, and a sufficient amount of light is condensed on the imaging device. Can be made.

  In the case of the first configuration example, the light transmission portion is provided in a plurality of pixels. For example (but not limited to), it is preferable that the light transmission portion is provided in three or more pixels. The outer shape of the light transmitting portion is essentially arbitrary, and examples thereof include a rectangle such as a rectangle or a square.

  In the case of the second configuration example, the light transmission region is provided with a light transmission part around at least one or more pixels. For example (but not limited to), the light transmission region is provided around three or more pixels. It is desirable that the outer shape of the light transmission portion is essentially arbitrary. For example, an “L” shape (in which the light transmission part is provided on two consecutive sides corresponding to the pixel boundary), and a “U” shape (the light transmission part corresponds to the pixel boundary) 3), a “B” shape (a shape in which the light transmission part is provided on all sides corresponding to the pixel boundaries), and a cross-shaped shape ( For example, the light transmitting portion is provided on all the sides corresponding to the boundaries of the pixels, and is provided in common between adjacent pixels. Alternatively, a configuration in which a light transmission part is provided around a pixel group including a projection image of a lens provided in the imaging device may be considered.

  Here, in the image display device of the present embodiment, when the light transmission region is composed of a plurality of minute light transmission parts, for example, the shape, size, distribution (arrangement pitch and arrangement position) of the light transmission part itself is also included. It is preferable to apply (devise) a diffraction phenomenon suppressing method in advance so as to suppress the occurrence of the diffraction phenomenon. “Small” means that when the diffraction phenomenon suppression method of the present embodiment is not applied, the sizes (sizes) of the openings of the plurality of light transmission parts are such that light (electromagnetic waves) passes through each of the openings. It means that the diffraction effect is generated and the problem of blur is generated.

  The first diffraction phenomenon suppression method for the light transmission part itself makes the arrangement state of the plurality of light transmission parts in the light transmission region non-uniform (random). Since the size, shape, distribution, etc. of the light transmission part are uniform (same), the problem of blur due to the diffraction effect appears strongly, so that blur does not occur (so that it does not easily occur) from that viewpoint. The purpose. By optimizing the size, shape, and distribution (arrangement pitch) of the light transmitting portion, it is possible to reliably suppress the occurrence of diffraction phenomenon.

  Specifically, the size of the plurality of light transmission parts is random [Case A], the shape of the plurality of light transmission parts is random [Case B], and the distribution of the plurality of light transmission parts is random [ At least one of the three types of cases C] is employed. That is, [Case A], [Case B], and [Case C] may be employed independently, or any combination thereof may be employed.

  Specifically, the form of [Case A] is such that each of the at least two light transmission parts adjacent to one light transmission part in the horizontal direction and / or the vertical direction is different from the size of the one light transmission part. It is.

  Specifically, the form of [Case B] is different from the shape of the one light transmission part in each of at least two light transmission parts adjacent to the one light transmission part in the horizontal direction and / or the vertical direction. It is a form.

  The form of [Case C] is a form in which the distribution is random depending on the arrangement pitch and the arrangement positional relationship of the plurality of light transmission parts, and specifically, in the horizontal direction and / or the vertical direction with respect to one light transmission part. Arrangement position relationship between at least two adjacent light transmission portions is the same as that of one light transmission portion with respect to the light transmission portion adjacent to the one light transmission portion in the horizontal direction and / or the vertical direction. It is a form different from the relationship. In other words, with respect to the arrangement pitch between the two light transmission portions, each of at least two arrangement pitches adjacent in the horizontal direction and / or the vertical direction with respect to the one arrangement pitch of interest is different from the one arrangement pitch. is there.

  As a second diffraction phenomenon suppressing method for the light transmission part itself, a form of [Case D] in which the light transmission part has a double annular structure (double hollow structure) is adopted. Specifically, the light transmission part may be configured by a first light transmission part and a second light transmission part, and the second light transmission part may be disposed so as to surround the first light transmission part. By optimizing the size, shape and arrangement of the first and second light transmission parts and the positional relationship between the first light transmission part and the second light transmission part, it is ensured that a diffraction phenomenon will occur. Can be suppressed.

  [Case D] can be combined with [Case A], [Case B], and [Case C] alone, or [Case A], [Case B], and [Case C] in any combination. It can also be combined with the embodiment.

  By applying these diffraction phenomenon suppression methods in the light transmission region, it is possible to suppress the occurrence of the diffraction phenomenon due to the light transmission portion. By combining with diffraction correction processing from the side of signal processing, it is possible to more reliably suppress blurring and resolution problems due to diffraction phenomena.

[Imaging device]
In the image display device of the present embodiment, the imaging device may be arranged on the back side of the image display unit, but is preferably arranged in the center of the image display unit. There may be one imaging device or a plurality of imaging devices. As the imaging device, for example, a known and commercially available solid-state imaging device including a CCD element and a CMOS sensor may be used.

  In the image display device of the present embodiment, it is preferable to provide a condensing unit that condenses the light that has passed through the light transmission part of the light transmission region on the imaging device between the image display unit and the imaging device. A well-known lens can be mentioned as a condensing part. Specifically, the lens can be composed of either a biconvex lens, a plano-convex lens, or a meniscus convex lens, or can be composed of a reflecting mirror or a Fresnel lens, or a combination of these various convex lenses. It is also possible to combine a concave lens and these various convex lenses.

  As the imaging device, a well-known and commercially available solid-state imaging device such as a video camera or a web camera can be used. In these cases, the light collecting unit and the imaging device are integrated.

  In the image display device of the present embodiment, a color filter may not be disposed in the optical path of light that enters the image display unit, passes through the light transmission unit, exits from the image display unit, and enters the light collection unit. It is also preferable not to arrange an imaging system such as a microlens.

[Image display area]
The image display unit used in the image display apparatus according to the present embodiment only needs to be able to form a light transmission part in the gap between the pixels (display part thereof), and display an electro-optical element whose luminance changes depending on the applied voltage or flowing current. Any device may be used as long as it is used as an element.

  For example, a liquid crystal display element is a typical example of an electro-optical element whose luminance changes depending on an applied voltage, and an organic electroluminescence (Organic Electro Luminescence, Organic EL, Organic) A typical example is a light emitting diode (OLED) element. The organic EL display device using the latter organic EL element is a so-called self-luminous display device using a self-luminous light-emitting element (self-luminous element) as a pixel display element. On the other hand, the liquid crystal display element that constitutes the liquid crystal display device controls the passage of light from the outside (light from the front or back surface, or external light in the case of the front surface), and the pixel includes a light emitting element. is not.

  For example, in recent years, interest in organic EL display devices has increased as flat panel display devices (FP display devices). At present, a liquid crystal display (LCD) dominates as an FP display device, but a member such as a backlight or a polarizing plate is required instead of a self-luminous device. Therefore, there are problems such as an increase in the thickness of the FD display device and insufficient luminance. On the other hand, the organic EL display device is a self-luminous device, and does not require a member such as a backlight in principle, and has many advantages compared to an LCD, such as being thin and having high luminance. In particular, an active matrix organic EL display device in which a switching element is arranged in each pixel can keep current consumption low by holding each pixel in a hold state, and relatively large screen and high definition are relatively easy. Because it is possible to do so, development is progressing in each company, and it is expected to become the mainstream of next-generation FP display devices.

  As a self-luminous display device, in addition to an organic EL display device, a plasma display panel (PDP), a field emission display (FED), a surface conduction electron-emitting device display device (SED: Surface-conduction Electron-emitter Display) and the like can be applied to the image display unit of the present embodiment.

  However, in the image display device of the present embodiment, the light emitting element of the image display unit is preferably a self-luminous light emitting element, and more preferably an organic EL element. When the light emitting element is composed of an organic EL element, the organic layer (light emitting portion) constituting the organic EL element includes a light emitting layer made of an organic light emitting material. Specifically, for example, a hole transport layer and a light emitting layer are provided. And electron transport layer, a hole transport layer and a light emitting layer that also serves as an electron transport layer, a hole injection layer, a hole transport layer, a light emitting layer, an electron transport layer, and an electron injection layer It can consist of a structure.

  When the electron transport layer, the light-emitting layer, the hole transport layer, and the hole injection layer are “tandem units”, the organic layer is formed by stacking the first tandem unit, the connection layer, and the second tandem unit 2 It may also have a tandem structure of stages, and may also have a tandem structure of three or more stages in which three or more tandem units are stacked. In these cases, an organic layer that emits white light as a whole can be obtained by changing the luminescent color of each tandem unit to red, green, and blue.

  By optimizing the thickness of the organic layer, for example, the light emitted in the light emitting layer is resonated between the first electrode and the second electrode, and a part of this light is transmitted to the outside through the second electrode. It can also be set as the structure which radiate | emits.

  In the image display device of the present embodiment, when the light emitting element of the image display unit is an organic EL element, the first substrate, the driving circuit provided on the first substrate, the interlayer insulating layer covering the driving circuit, and the interlayer insulating layer The light emitting part provided, the protective layer provided on the light emitting part, the light shielding layer provided on the protective layer, and the second substrate covering the protective layer and the light shielding layer may be provided.

  Further, each pixel includes a drive circuit and a light emitting portion, and the light shielding layer is provided with an opening, and the opening, the portion of the protective layer located below the opening, and the interlayer insulating layer The light transmission part is comprised by the part, and it can be set as the form by which the imaging device is arrange | positioned on the surface side of the 1st board | substrate which does not oppose the 2nd board | substrate.

Here, examples of the pixel arrangement include a stripe arrangement, a diagonal arrangement, a delta arrangement, and a rectangle arrangement. Further, as the first substrate and the second substrate, a high strain point glass substrate, a soda glass (Na ₂ O · CaO · SiO ₂ ) substrate, a borosilicate glass (Na ₂ O · B ₂ O ₃ · SiO ₂ ) substrate, Stellite (2MgO · SiO ₂ ) substrate, lead glass (Na ₂ O · PbO · SiO ₂ ) substrate, various glass substrates with an insulating film formed on the surface, quartz substrate, quartz substrate with an insulating film formed on the surface, surface A silicon substrate having an insulating film formed thereon, polymethyl methacrylate (polymethyl methacrylate, PMMA), polyvinyl alcohol (PVA), polyvinyl phenol (PVP), polyethersulfone (PES), polyimide, polycarbonate, polyethylene terephthalate (PET) Organic polymers (flexible plastic film composed of polymer material) And plastic sheets, the form of polymeric material such as a plastic substrate) may be mentioned.

The drive circuit may be composed of, for example, one or a plurality of thin film transistors (TFTs). As a constituent material of the interlayer insulating _{layer, SiO 2, BPSG, PSG,} BSG, AsSG, PbSG, SiON, SOG ( spin on glass), low-melting glass, SiO ₂ based materials such glass paste; insulation, such as polyimide; SiN-based materials Resins can be used alone or in appropriate combination. When the pixel is composed of an organic EL element, the light emitting unit is as described above. As a material constituting the protective film, it is preferable to use a material that is transparent to the light emitted from the light emitting portion, is dense, and does not transmit moisture. Specifically, for example, amorphous silicon (α-Si), Amorphous silicon carbide (α-SiC), amorphous silicon nitride (α-Si _1-x N _x ), amorphous silicon oxide (α-Si _1-y O _y ), amorphous carbon (α-C), amorphous silicon oxide / silicon nitride (Α-SiON) and Al ₂ O ₃ can be mentioned. The light shielding film (black matrix) may be made of a known material. You may provide a color filter as needed.

  The image display unit includes a plurality of pixel units including display elements (light emitting elements). Here, when the number of pixel units is represented by (M, N), VGA (640, 480), S- VGA (800, 600), XGA (1024, 768), APRC (1152, 900), S-XGA (1280, 1024), U-XGA (1600, 1200), HD-TV (1920, 1080), Q- In addition to XGA (2048, 1536), some of the image display resolutions such as (1920, 1035), (720, 480), (854, 480), (1280, 960) can be exemplified. It is not limited to the value of.

  In an image display unit that performs color display, one pixel unit includes, for example, a red pixel that displays a red (R) color component, a green pixel that displays a green (G) color component, and a blue (B) color component. It is composed of three types of blue pixels to be displayed. Alternatively, in addition to these three types of pixels, pixels that display white light for improving luminance, pixels that display complementary colors to expand the color reproduction range, and pixels that display yellow to expand the color reproduction range In order to expand the color reproduction range, it may be composed of four or more types of pixels such as pixels displaying yellow and cyan.

  The image display device of the present embodiment is an example of an electronic device, and it is only necessary to be able to arrange the imaging device on the back side of the display panel portion. The imaging device may be detachable or fixedly fixed. It may be installed.

  The image display device can be used, for example, as a substitute for a monitor device constituting a personal computer, or as a substitute for a monitor device incorporated in a notebook personal computer. Furthermore, it can be used as a substitute for a mobile phone, a PDA (Personal Digital Assistant), a monitor device incorporated in a game machine, or a conventional television receiver.

<First Embodiment>
[Overview]
1 to 2 are views for explaining the concept of the image display device and the image display system according to the first embodiment. Here, FIG. 1 (1) is a conceptual diagram of an image display device, and FIG. 1 (2) is a conceptual diagram of an image display system. FIG. 1 (1-1) is a conceptual diagram of the image display device viewed from the front, and FIG. 1 (1-2) is a conceptual diagram of the image display device viewed from the side. FIG. 1 (2-1) is a conceptual diagram of the image display system viewed from the front, and FIG. 1 (2-2) is a conceptual diagram of the image display system viewed from the side. FIG. 2 is a diagram schematically showing the most typical arrangement of a plurality of pixels constituting the image display unit.

  As illustrated in FIG. 1A, the image display device 1 </ b> A according to the first embodiment includes an image display unit 10, an imaging device 20 disposed on the back side of the image display unit 10, and light formed on the image display unit 10. The light-transmitting unit 30 and the light condensing unit 21 that condenses the light that has passed through the light-transmitting unit 30 on the imaging device 20. The imaging device 20 may be configured to be detachable from the back surface of the image display device 1. A portion corresponding to the imaging device 20 of the image display unit 10 is a light transmission region 12. For example, at least a portion of the image display unit 10 corresponding to the effective imaging region of the imaging device 20 is the light transmission region 12. When the light transmission region 12 is narrower than the effective imaging region of the imaging device 20, the actual imaging region in the imaging device 20 is narrowed.

The imaging device 20 is disposed on the back side of the image display unit 10, more specifically, on the center of the back side of the image display unit 10, and the number of the imaging device 20 provided is one. Here, the imaging device 20 and the light collecting unit 21 are formed of a well-known and commercially available video camera in which they are integrated, that is, provided with a CCD element. Or you may use the camera which consists of the condensing part and imaging device optimized so that it may arrange | position to the back side of the image display part 10. FIG.

  The image display device 1A is used as an alternative to a monitor device constituting a personal computer, for example. That is, as shown in FIG. 1 (2), in the image display system 2A of the first embodiment, a peripheral device 70A such as a main body of a personal computer (electronic computer) is connected to the image display device 1A_2. The peripheral device 70A (the same applies to other peripheral devices 70 described later) is an example of an electronic device. The image display device 1A_2 functions as a monitor device for the peripheral device 70A. The image display device 1A_2 is obtained by removing some functional units from the image display device 1A. The removed functional part is incorporated in the peripheral device 70A. The peripheral device 70 </ b> A, the image display unit 10, and the imaging device 20 are connected by cables 72 and 73.

  As the pixel 11 of the image display unit 10, a self-luminous light emitting element, specifically, an organic EL element is used. The image display unit 10 includes a color display XGA type organic EL display device. That is, when the number of pixel units is represented by (M, N), it is (1024, 768).

  As shown in FIG. 2, one pixel unit is composed of three pixels: a red light emitting pixel 11R that emits red light, a green light emitting pixel 11G that emits green light, and a blue light emitting pixel 11B that emits blue light. . Note that the outer edge of the pixel is indicated by a dotted line (the same applies to other examples described later).

  The image display unit 10 includes a plurality of pixels 11 (11R, 11G, and 11B) including display elements, and the plurality of pixels 11 in the light transmission region 12 of the image display unit 10 are provided with a light transmission unit 30. ing. In this example, the light transmission part 30 is different for each pixel 11, but this is not essential, and the light transmission part 30 may straddle a plurality of pixels 11. In this example, the light transmission unit 30 is provided for each pixel 11 (for all the pixels 11 in the light transmission region 12). However, this is not essential, and at least the plurality of pixels 11 in the light transmission region 12 are provided. It is only necessary that the light transmission part 30 is provided. For example, some pixels 11 in the light transmission region 12 such as every two pixels may not be provided with the light transmission part 30.

  Although not limited, the light transmission part 30 is provided in, for example, 6 × 3 = 18 pixels 11. One light transmitting portion 30 is provided for one pixel. The condensing unit 21 condenses the light that has passed through the light transmitting unit 30 in the 6 × 3 = 18 pixels 11 on the imaging device 20. The shape of each light transmission part 30 is a rectangle.

  Although not shown, the image display unit 10 is provided with a scanning signal supply IC that drives each scanning line and a video signal supply IC that supplies a video signal. A scanning line control circuit is connected to the scanning signal supply IC, and a signal line control circuit is connected to the video signal supply IC. A color filter is not disposed in the optical path of light that enters the image display unit 10, passes through the light transmission unit 30, exits from the image display unit 10, and enters the light collection unit 21. There is no image system.

[Imaging device placement position]
3 to 4 are conceptual diagrams illustrating the relationship between the arrangement position of the imaging device and the displayed image. Here, FIG. 3 (1) shows the case of the image display device 1 of the first embodiment, and FIG. 3 (2) shows the image display device 1X of the comparative example in which the imaging device is fixed outside the image display unit. This case is shown.
FIG. 4 is a diagram illustrating a captured image by the image display device 1A.

  As shown in FIG. 3B, when the imaging device 20X is fixed to the outside of the image display unit 10X, the imaging device 20X captures an image of the user of the image display device 1X from an oblique direction. When the image is displayed on the image display unit 10X, the image displayed on the image display unit 10X displays a user's image taken from an oblique direction. Therefore, it is impossible to accurately display the user's face, and it is not possible to accurately determine where in the image display unit 10X the user is gazing. Furthermore, when the user approaches the image display unit 10X, there is a high possibility that the user will be outside the imaging range.

  On the other hand, as shown in FIG. 3 (1), in the image display device 1A of the first embodiment, since the imaging device 20 is disposed at the center on the back side of the image display unit 10, the image display device The imaging device 20 can capture the user 1A from the front. When the image is displayed on the image display unit 10, the image display unit 10 displays the image of the user captured from the front. The Therefore, the user's face can be accurately displayed, and it is possible to easily and accurately determine where in the image display unit 10 the user is gazing. Further, even when the user approaches the image display unit 10, the user can be imaged.

  In the image display device 1 </ b> A, the light that has passed through the light transmission units 30 (the plurality of light transmission units 30) provided in each of the plurality of pixels 11 in the light transmission region 12 is collected on the imaging device 20. Therefore, a high-precision microlens is not required to accurately connect an image to the imaging device 20, and the manufacturing cost of the image display device 1A is not increased, and a sufficient amount of light is condensed on the imaging device 20. Can be made.

  For example, FIG. 4A shows a state in which an observer (a subject for the imaging apparatus 20) is pointing with a pen while looking at a display image on the image display unit 10 as an example. An imaging device 20 is provided on the back surface of the image display device 1A (image display unit 10), and as shown in FIG. 4 (2), the face and eyes of the observer facing the display surface, the hand, The pen can be imaged. Thereby, for example, the line of sight of the observer can be detected from the captured image. Further, since the corresponding indication point of the image display unit 10 can be specified from the orientation of the hand or pen, a so-called pointer function can be easily added to the image display device 1A. Furthermore, in addition to the pointer function, the user's face, eyes, hand movements, peripheral brightness, and the like can be known from the captured image, so various information is obtained from the image display device 1A and sent to various systems. The added value of the image display apparatus 1A can be increased.

[Cross-sectional structure of image display section]
FIG. 5 is a diagram for explaining the details of the image display unit 10. Here, FIG. 5A is a schematic partial cross-sectional view of the image display unit 10. FIG. 5B is a chart summarizing the detailed configuration of the light emitting elements of the image display unit 10.

  The image display unit 10 includes a first substrate 40, a drive circuit including a plurality of TFTs provided on the first substrate 40, an interlayer insulating layer 41 that covers the drive circuit, and an organic layer provided on the interlayer insulating layer 41. 63 (light emitting portion), a protective layer 64 provided on the organic layer 63, a light shielding layer 65 provided on the protective layer 64, a second layer 67 covering the protective layer 64 and the light shielding layer 65.

  Each pixel 11 includes a drive circuit and a light emitting unit, and the light shielding layer 65 is provided with an opening 65A, and the opening 65A and a portion of the protective layer 64 positioned below the opening 65A, The light transmitting portion 30 is configured by the second electrode 62 portion, the interlayer insulating layer 41 portion, and the like. The condensing unit 21 and the imaging device 20 are disposed on the side of the first substrate 40 that does not face the second substrate 67.

  More specifically, a drive circuit is provided on the first substrate 40 made of soda glass. The TFT constituting the driving circuit includes a gate electrode 51 formed on the first substrate 40, a gate insulating film 52 formed on the first substrate 40 and the gate electrode 51, and a semiconductor layer formed on the gate insulating film 52. The portion of the semiconductor layer located between the source / drain region 53 and the source / drain region 53 and above the gate electrode 51 is composed of a corresponding channel formation region 54. In the illustrated example, the TFT is a bottom gate type, but may be a top gate type.

The gate electrode 51 of the TFT is connected to a scanning line (not shown). The interlayer insulating layer 41 (41A, 41B) covers the first substrate 40 and the drive circuit. The first electrode 61 constituting the organic EL element, SiO _X and SiN _Y, is provided on the interlayer insulating layer 41B made of a polyimide resin. The TFT and the first electrode 61 are electrically connected via a contact plug 42, a wiring 43, and a contact plug 44 provided in the interlayer insulating layer 41A. In the drawing, one TFT is shown for one organic EL element driving unit.

  On the interlayer insulating layer 41, an insulating layer 45 having an opening 46 and having the first electrode 61 exposed at the bottom of the opening 46 is formed. The insulating layer 45 is excellent in flatness, and is made of an insulating material having a low water absorption rate, specifically, a polyimide resin in order to prevent the organic layer 63 from being deteriorated by moisture and maintain the light emission luminance. An organic layer 63 including a light emitting layer made of an organic light emitting material is formed over the portion of the insulating layer 45 surrounding the opening 46 from above the portion of the first electrode 61 exposed at the bottom of the opening 46. The organic layer 63 is composed of, for example, a stacked structure of a light-emitting layer that also serves as a hole transport layer and an electron transport layer, but is represented by one layer in the drawing.

An insulating protective layer 64 made of amorphous silicon nitride (α-Si _1-x N _x ) is provided on the second electrode 62 based on the plasma CVD method for the purpose of preventing moisture from reaching the organic layer 63. It has been. A light shielding layer 65 made of black polyimide resin is formed on the protective layer 64, and a second substrate 67 made of soda glass is disposed on the protective layer 64 and the light shielding layer 65. The protective layer 64, the light shielding layer 65, and the second substrate 67 are bonded by an adhesive layer 66 made of an acrylic adhesive.

  The first electrode 61 is used as an anode electrode, and the second electrode 62 is used as a cathode electrode. Specifically, the first electrode 61 is made of a light reflecting material composed of aluminum (Al), silver (Ag), or an alloy thereof having a thickness of 0.2 μm to 0.5 μm, and the second electrode 62. Is made of a transparent conductive material such as ITO or IZO having a thickness of 0.1 μm, or a metal thin film (semi-transparent metal thin film) having a thickness of about 5 nm such as silver (Ag) or magnesium (Mg). The second electrode 62 is not patterned and is formed in a single sheet. In some cases, an electron injection layer (not shown) made of LiF having a thickness of 0.3 nm may be formed between the organic layer 63 and the second electrode 62.

  In summary, the detailed configuration of the light emitting elements of the image display unit 10 in the image display apparatus 1A of the present embodiment is as shown in FIG.

  From the first substrate 40 to the protective layer 64 and the light shielding layer 65 are display element substrates. The second substrate 67 functions as a sealing substrate. Here, the opening 65A forming the light transmitting portion 30 provided in the second substrate 67 is in a portion where the pixel electrode (first electrode 61), TFT (including the gate electrode 51) and wiring of the display element substrate are absent. In addition, a protective layer 64 that functions as a sealant, a second electrode 62 that functions as an EL common electrode, an insulating layer 45 that functions as a pixel separation film, an interlayer insulating layer 41 (41A, 41B) that functions as a planarization insulating film, The source / drain regions 53 and the gate insulating film 52 are light transmissive. Accordingly, external light incident from the display surface (second substrate 67) side can reach the back surface (first substrate 40 side) through the opening 65A.

  The imaging device 20 installed on the rear surface of the panel is installed so that the imaging surface is close to the rear surface of the panel where the light transmission unit 30 (opening 65A) is located (in this example, the light collecting unit 21 is interposed). Therefore, the external light incident from the display surface side is imaged by a lens (not shown) of the imaging device 20 and enters a solid-state image sensor such as a CCD or a CMOS, so that a subject existing on the display surface side is imaged. be able to.

[Configuration for diffraction correction]
6 to 9 are diagrams for explaining the diffraction phenomenon and countermeasures. Here, FIG. 6 is a schematic diagram for explaining the influence of the diffraction phenomenon on the captured image. FIG. 7 is a diagram illustrating an example of an image obtained by imaging by placing a glass plate in front of the imaging device 20. FIG. 8 is a block diagram of the image display apparatus 1A according to the first embodiment in which the influence of the diffraction phenomenon is corrected (compensated) from the signal processing surface. FIG. 9 is a block diagram of the image display system 2A of the first embodiment in which the influence of the diffraction phenomenon is corrected (compensated) from the signal processing surface.

  In the image display apparatus 1A of the first embodiment, an organic EL element is used as a display element, and the imaging device 20 is installed on the back surface (first substrate 40) side only by providing the light transmission unit 30, A subject existing on the display surface (second electrode 62) side can be imaged.

  Such a simple configuration is difficult, if not impossible, with a so-called LCD, and a structure that transmits light including visible wavelengths is even more difficult. On the other hand, the image display apparatus 1A of the first embodiment only provides the light transmission unit 30, and can image a subject on the display surface side from the back surface with a simple configuration.

  By the way, generally, when an image is taken through a small light transmission opening, a so-called diffraction phenomenon occurs in the light transmission opening. That is, as shown in FIG. 6, when the imaging device 20 is installed on the back of an object (image display unit 10) provided with minute light transmission units 30 at a predetermined pitch, and the image is transmitted through the light transmission unit 30, the light transmission is performed. The part 30 acts as an optical slit. For this reason, the same image as the image “C” appears as an image “A” and an image “B” at an equal pitch due to the diffraction phenomenon, resulting in blurring of the image.

  FIG. 7A shows an image obtained by imaging a transparent glass plate that is not provided with a light transmission portion in front of the imaging device 20. FIG. 7B shows an image obtained by arranging and imaging a transparent glass plate provided with a light transmission part having a shape, size, and distribution in front of the imaging device 20. Blur is observed in the image shown in FIG. On the other hand, no blur is observed in the image shown in FIG.

  The intensity and distribution of the diffracted light depends on the shape, size, distribution, and wavelength of incident light (external light) of the light transmission part. If the blur due to diffraction is small, there is no need to correct (compensate) the diffraction for the captured image, but if a high-quality captured image is required, the effect of the diffraction phenomenon is corrected (compensated). ) Is preferable.

  FIG. 8 shows an image display apparatus 1A according to the first embodiment that corrects the influence of the diffraction phenomenon from the signal processing surface.

  The image display device 1A according to the first embodiment includes a control unit 90 that controls the operation of the entire device, a display timing controller 92 that controls the display operation of the image display unit 10, and the imaging operation of the imaging device 20 (imaging unit 20a). An imaging timing controller 94 is controlled.

  The control unit 90 supplies display data, timing signals, and the like to the display timing controller 92. The control unit 90 supplies an imaging timing signal, a shutter control signal, a gain control signal, and the like to the imaging timing controller 94.

  The display timing controller 92 includes a signal line control circuit (not shown), and the signal line control circuit supplies display data and a horizontal timing signal to the image display unit 10 via a video signal supply IC (not shown). The display timing controller 92 includes a scanning line control circuit (not shown), and the scanning line control circuit supplies a vertical timing signal to the image display unit 10 via a scanning signal supply IC (not shown).

  The image display device 1 </ b> A of the first embodiment includes a diffraction correction unit 100 that performs correction (compensation) of diffraction generated in the light transmission unit 30 on image information acquired via the imaging device 20. When the imaging device 20 (an example of an electronic device) is separated from the image display device 1A, the diffraction correction unit 100 (also an imaging timing controller 94 in some cases) is connected to the imaging device as shown by a dashed line in the figure. Arrangement on the 20 side is also conceivable. The diffraction correction unit 100 includes an MTF shape storage unit 102 and an MTF inverse conversion unit 104. The MTF inverse transform unit 104 has an image storage memory (not shown), and stores captured image data supplied from the imaging device 20 in the image storage memory. The processing operation of the diffraction correction unit 100 is controlled by the control unit 90.

  The control unit 90 receives various instructions from the user and controls the operations of the diffraction correction unit 100 and other functional units. For example, as will be described in detail later, various procedures can be considered as the diffraction correction processing of this embodiment, but since they have superiority and inferiority in each aspect of processing speed and correction accuracy, which method is adopted, It is preferable to be able to select appropriately according to a user's instruction.

  The MTF shape storage unit 102 stores MTF shape data such as the size, shape, and distribution of the light transmission unit 30. For example, the MTF shape storage unit 102 stores the wavelength of external light for each of red (R), green (G), and blue (B) and the MTF shape data of the light transmitting unit 30 obtained by the two-dimensional FFT. deep.

  Image information acquired via the imaging device 20 is sent to the MTF inverse transform unit 104 constituting the diffraction correction unit 100. The MTF inverse conversion unit 104 converts the wavelength of external light for each of red (R), green (G), and blue (B) and the MTF shape data of the light transmission unit 30 obtained by two-dimensional FFT into an MTF shape storage unit. The data is read from 102, MFT reverse conversion is performed, the original image is restored, and the image is sent to the controller 90. At this time, in order to simplify the calculation and shorten the processing time, processing not focusing on all signal components but focusing on some signal components may be performed. Details of this point will be described later.

  The control unit 90 performs various detections from the restored image obtained by the diffraction correction unit 100 (the MTF inverse transform unit 104), such as detecting the user's line of sight and detecting the movement of the user's hand, and displays an image. This is reflected in the display on the part 10.

  The image display in the image display unit 10 is performed under the control of the control unit 90. That is, display data, timing signals, and the like are sent from the control unit 90 to the display timing controller 92, and display data and horizontal timing signals are sent from the display timing controller 92 to a signal line control circuit (not shown). Is sent to a scanning line control circuit (not shown). Then, the image display unit 10 displays an image based on a known method. On the other hand, an imaging timing signal, a shutter control signal, a gain control signal, and the like are sent from the control unit 90 to the imaging timing controller 94, and these signals are sent from the imaging timing controller 94 to the imaging device 20. Operation is controlled.

  FIG. 9 shows an image display system 2A of the first embodiment that corrects the influence of the diffraction phenomenon from the signal processing surface. 8 is different from the image display device 1A shown in FIG. 8 in that the control unit 90 and the diffraction correction unit 100 are removed from the image display device 1A to obtain the image display device 1A_2. It is incorporated in the peripheral device 70A (an example of an electronic device).

  That is, the control unit 90 may be provided in the image display device 1A, or may be provided in a peripheral device 70A such as a personal computer connected to the image display device 1A_2. The diffraction correction unit 100 may also be provided in the image display device 1A, or may be provided in a peripheral device 70A such as a personal computer connected to the image display device 1A_2.

  The control functions of the diffraction correction unit 100 and the control unit 90 (particularly the function of controlling the diffraction correction unit 100) in the image display device 1A and the image display system 2A can also be realized by software. It is also possible to extract the stored recording medium as an invention. These points reflect the measurement result of the wavelength distribution of the external light, thereby improving the accuracy of the image information acquired through the imaging device (for example, improving the accuracy of the color information) and improving the accuracy of the MTF inverse conversion process. The same applies to the plan.

  That is, in this embodiment, the structure of the control configuration for performing the diffraction correction processing, the wavelength distribution measurement processing, and the control processing related to these processing is not limited to being configured by a hardware processing circuit, but is a program code that realizes the function. Based on the above, it can be realized by software using a mechanism of an electronic computer (computer). In the mechanism executed by software, the processing procedure and the like can be easily changed without changing hardware.

  The program may be provided by being stored in a computer-readable storage medium, or may be provided by distribution via wired or wireless communication means.

  The program is provided as a file describing a program code for realizing each function of diffraction correction processing, wavelength distribution measurement processing, and control processing related to these processing, but in this case, the program is not limited to being provided as a batch program file. Instead, it may be provided as an individual program module according to the hardware configuration of the system configured by a computer.

  Specific means of each part (including functional blocks) for realizing each function of diffraction correction processing, wavelength distribution measurement processing, and control processing related to these processing is hardware, software, communication means, and a combination thereof. Other means can be used, as will be apparent to those skilled in the art. Moreover, the functional blocks may be combined and integrated into one functional block. Also, software that causes a computer to execute program processing is installed in a distributed manner according to the combination.

  Although the mechanism performed by software can flexibly cope with parallel processing and continuous processing, the processing time becomes longer as the processing becomes complicated, so that a reduction in processing speed becomes a problem. On the other hand, when constructed with a hardware processing circuit, even if the processing is complicated, a reduction in processing speed is prevented, and an accelerator system is constructed that achieves high speed to obtain high throughput.

[Countermeasure principle of diffraction phenomenon]
10 to 12 are diagrams for explaining a technique for taking measures against the influence of the diffraction phenomenon from the aspect of signal processing. Here, FIG. 10 is a conceptual diagram for obtaining the diffracted light intensity and the MTF from the light transmitting part 30 (opening 65A). FIGS. 11-12 is a figure which shows typically an example of the shape of the light transmissive part 30 (different from FIG. 2).

The diffraction distribution P _diff can be calculated by Expression (1) when the pattern shape, size, distribution, and wavelength of incident light (external light) of the light transmitting unit 30 are determined. In double integration, integration is performed from −∞ to + ∞ with respect to x and y.

Here, P _at (x, y) is a two-dimensional pattern on the xy plane of the light transmitting portion 30 (see FIG. 10A), λ is the wavelength of incident light (external light), and θ _x , Θ _y are diffraction angles in the x and y directions. Here, in order to simplify the calculation, the value of the wavelength λ of the incident light (external light) is constant at 525 nm.

Since Equation (1) is a two-dimensional Fourier transform of P _at (x, y), it can be calculated at high speed by using a fast Fourier transform (FFT). Although P _diff (k _x , k _y ) includes phase information, the imaging device actually detects the diffracted light intensity H _diff (k _x , k _y ). As shown in Equation (2), the diffracted light intensity H _diff (k _x , k _y ) is equal to the square of the absolute value of P _diff (k _x , k _y ) (see FIG. 10 (2)).

  Here, assuming that the spatial resolution of the imaging device 20 is modulated by the diffracted light, an MTF (Modulation Transfer Function) is calculated from the equation (3) (see FIG. 10 (3)). “FFT []” means performing a fast Fourier transform, and “IFFT []” means performing a fast inverse Fourier transform.

Here, f _x, f _y represent the x and y directions of the spatial frequencies in the image sensor of each constituting the imaging device 20. And between the image I _{cam (x, y)} obtained through the light transmission part 30 obtained by the imaging device 20 and the original image I _{real (x, y)} when the light transmission part 30 is not passed. The relationship shown in the equation (4) is established. The image I _{cam (x, y)} is, for example, imaging R data R _{cam (x, y)} if it is a red (R) color component, and imaging G data G _{cam (x, y} ) if it is a green (G) color component. ₎ , If it is a blue (B) color component, the imaged B data B _{cam (x, y)} is used.

That is, in the spatial frequency domain, the image I _{cam (x, y)} is a product of the original image I _{real (x, y)} and MTF. Therefore, in order to obtain the original image I _{real (x, y)} from the image I _{cam (x, y} ), the processing based on the equation (5) may be performed. In other words, the image information is subjected to MTF inverse conversion processing calculated based on the shape, size, distribution, and wavelength of incident light (external light) (collectively “MTF shape data”). That's fine. The original image _{Ireal (x, y)} is, for example, a red original image _{Rreal (x, y)} if it is a red (R) color component, and a green original image _{Greal (x, y)} if it is a green (G) color component _{. y)} If it is a blue (B) color component, the blue original image B _{real (x, y)} is used.

  Here, if the size, shape, and distribution of the light transmission part 30 are determined, the MTF may be scaled by the wavelength of the incident light (external light) obtained by Fourier transforming the two-dimensional pattern of the light transmission part 30. It can be easily obtained, and the original image can be easily restored from the relationship shown in Expression (5).

  For example, the shape of the light transmission part 30 is illustrated in FIG. 11 (1), (2) and FIG. Part or all of the light transmission unit 30 is provided periodically along the first direction (horizontal direction) and the second direction (vertical direction) of the image display unit 10.

  In the example shown in FIG. 11 (1), the light transmission part 30H extending along the first direction (horizontal direction) below the pixel is composed of three pixels (11R, 11G, 11B). The light transmission part 30V provided across the two pixel units and extending in the second direction (vertical direction) is provided in each of the pixels 11R, 11G, and 11B, and is provided between the pixels. It has been. In the example shown in FIG. 11 (2), the light transmission part 30H and the light transmission part 30 are connected. In the example shown in FIG. 12, the light transmission part 30H is provided across one pixel unit composed of three pixels (11R, 11G, 11B). Unlike, it consists of two parts.

When the length of the light transmission part 30 along the first direction is L _tr−1 and the pitch of the pixels 11 along the first direction is P _px−1 , the line aperture ratio L _{tr in} the first direction. ₋₁ / P _px−1 satisfies “L _tr−1 / P _px−1 ≧ 0.5”, and the length of the light transmission part 30 along the second direction is represented by L _tr−2 , second When the pitch of the pixels 11 in the direction of P is P _px-2 , the line aperture ratio L _tr-2 / P _px-2 in the _{second direction} is “L _tr-2 / P _px-2 ≧ 0. Satisfy 5 ”. This can be explained from the definition of MTF.

The MTF squares the diffraction distribution obtained from the two-dimensional pattern P _at (x, y) on the xy plane of the light transmission part 30 as shown in the equation (6) derived from the equations (2) and (3). The result is subjected to a fast Fourier transform, and the result is further squared.

From the so-called Wiener Kinchin's theorem, the Fourier transform of the autocorrelation function is equal to the power spectrum. Conditions under which the autocorrelation function does not have a so-called uncorrelated point in the spatial frequency domain (that is, a point where it becomes 0) are “L _tr-1 / P _px-1 ≧ 0.5”, “L _tr-2 / P _px-2 ≧ 0.5 ”. When there is no point at which the MTF is 0, the equation (5) has no singular point, so that it is easy to reproduce the original image. Therefore, it is requested that the values of the line opening ratio L _tr-1 / P _{px-1 in the first} direction and the line opening ratio L _tr-2 / P _px-2 in the second direction are 0.5 or more. It is preferable to satisfy.

Second Embodiment
13 to 15 are diagrams for explaining an image display device and an image display system according to the second embodiment. Here, FIG. 13 is a diagram illustrating the concept of the image display device and the image display system according to the second embodiment. FIG. 13 (1) is a conceptual diagram of an image display device, and FIG. 13 (2) is a conceptual diagram of an image display system. FIG. 13 (1-1) is a conceptual diagram of the image display device viewed from the front, and FIG. 13 (1-2) is a conceptual diagram of the image display device viewed from the side. FIG. 13 (2-1) is a conceptual diagram of the image display system viewed from the front, and FIG. 13 (2-2) is a conceptual diagram of the image display system viewed from the side. FIG. 14 is a block diagram of an image display device 1B of the second embodiment that enables wavelength distribution measurement. FIG. 15 is a block diagram of an image display system 2B according to the second embodiment that enables wavelength distribution measurement.

  As shown in FIG. 13A, the image display device 1B of the second embodiment includes a wavelength distribution measurement unit 110 that measures the wavelength distribution of external light in addition to the image display device 1A of the first embodiment. Yes. The wavelength distribution measuring unit 110 may be configured to be detachable from the image display device 1. In this example, the wavelength distribution measuring unit 110 is arranged at the upper part of the panel of the image display unit 10, but this is an example, and as long as it is a place where the wavelength distribution of external light can be measured, the wavelength distribution measuring unit 110 is located at any location. You may arrange.

  As shown in FIG. 13B, the image display system 2B of the second embodiment further includes a wavelength distribution measurement unit 110 that measures the wavelength distribution of external light in addition to the image display system 2A of the first embodiment. ing. Information acquired by the wavelength distribution measurement unit 110 is supplied to the diffraction correction unit 100. The wavelength distribution measuring unit 110 may be disposed at any location in the vicinity of the image display device 1B_2 as long as it is a location where the wavelength distribution of external light can be measured.

  The wavelength distribution measuring unit 110 includes, for example, a set of a photo sensor to which a red filter is attached, a photo sensor to which a green filter is attached, and a photo sensor to which a blue filter is attached.

  As shown in FIG. 14, in the image display apparatus 1 </ b> B, the control unit 90 in the apparatus controls the measurement operation of the wavelength distribution measurement unit 110. Information acquired by the wavelength distribution measurement unit 110 is supplied to the diffraction correction unit 100. As shown in FIG. 15, in the image display system 2B, the control unit 90 incorporated in the peripheral device 70B controls the measurement operation of the wavelength distribution measurement unit 110.

As described in the first embodiment, the expression (1) for _obtaining the diffraction distribution P _diff (k _x , k _y ) in the xy plane of the light transmitting unit 30 includes the wavelength λ of the incident light (external light). It is. Therefore, by measuring the wavelength distribution of external light, the MTF of each wavelength can be adapted according to the external environment, and correction and compensation for diffraction can be performed more accurately, resulting in a higher quality captured image. Obtainable.

  By providing the wavelength distribution measuring unit 110, the wavelength distribution of external light (light spectrum) can be obtained. And the wavelength distribution for each primary color (red, green, blue) of the captured image can be obtained by multiplying the obtained wavelength distribution of the external light and the spectral spectrum of the imaging device 20. Then, by weighting the image subjected to MTF inverse transform processing for each wavelength with the wavelength distribution of the captured image, correction and compensation for diffraction can be performed more accurately.

  In the second embodiment, by adopting such a configuration, it is possible not only to correct and compensate for diffraction more accurately, but also to improve the accuracy of image information acquired via the imaging device 20. (For example, the accuracy of color information can be improved).

  Although not directly related to the mechanism of this embodiment, when the blur due to diffraction is small, it is not necessary to perform a process of correcting (compensating) diffraction for the captured image. However, even in such a case, by further including a wavelength distribution measuring unit 110 that measures the wavelength distribution of external light, for example, the diffraction correction unit 100 can convert the image acquired by the imaging device 20 into the wavelength distribution of the captured image. Can be weighted. As a result, it is possible to improve the accuracy of the image information acquired via the imaging device 20.

<Diffraction correction processing: first example (processing area reduction processing)>
FIG. 16 is a diagram (flow chart) illustrating a first example of diffraction correction processing by the diffraction correction unit 100. For easy understanding, a schematic diagram of an image corresponding to the processing status is also shown. The diffraction correction process of the first example applies a process area reduction technique.

  As an example, R, G, and B color image data (imaging R data, imaging G data, and imaging B data: collectively imaging RGB data) as imaging data from the imaging device 20 are sent to the MTF inverse conversion unit 104 of the diffraction correction unit 100. It shall be supplied (S10). The MTF inverse transform unit 104 stores the captured image data supplied from the imaging device 20 in the image storage memory.

  The MTF inverse transform unit 104 separates the captured image into a “specific region” and a “background region”, and extracts a “specific region”. For example, the “subject-oriented method” is applied, the “specific region” is set as a partial region of the subject (for example, the user's face region), and the face region is extracted by separating the face into other regions. . As a face area extraction algorithm, for example, a known face detection method such as Haar-like feature classification may be used. Incidentally, it is assumed that a rectangular area including the specified face is extracted. Of course, the extracted face area has a smaller area than the entire captured image.

  Further, in the extraction of the “specific region”, a region having a specific shape or color is extracted, or, for example, based on the difference between a plurality of captured images captured at different times, the change of the captured image (for example, movement of the subject) ) May be detected, and the region in which the change has occurred may be extracted.

  The MTF inverse transform unit 104 performs the MTF inverse transform process only on the extracted specific region to remove the influence of diffraction (S30). For example, in a video phone, the facial expression and facial image quality between callers are more important than the background, and even an image from which only the face portion has been diffracted can be communicated without feeling uncomfortable.

  The MTF inverse transform unit 104 synthesizes the extracted image (its signal component) of the diffraction-corrected face area (specific area) and the background area image (its signal component) that has not been subjected to diffraction correction processing. Complete (S40).

  The extracted area (specific area: face area in this example) separated (extracted) from the entire captured image has a smaller area than the entire captured image. In comparison, the amount of calculation is reduced, and the correction processing time is shortened, so that the processing can be speeded up. For example, when the area of the specific region is 1 / X times the entire region, the processing speed is “α · X” times. α is a value less than 1, and is based on the process of extracting the specific area from the entire area (S20) and the process of synthesizing the image of the specific area after the diffraction correction process and the image of the unprocessed background area (S40). This is overhead.

<Diffraction correction processing: second example (for all signal components)>
FIG. 17 is a diagram (flow chart) for explaining a second example of diffraction correction processing by the diffraction correction unit 100. The diffraction correction process of the second example is a method of processing each signal component constituting the image information acquired by the imaging device 20.

  The MTF inverse transform unit 104 performs the MTF inverse transform process only on the extracted specific region to remove the influence of diffraction (S30). At this time, the MTF reverse conversion unit 104 performs an MTF reverse conversion process that is calculated based on the shape, size, and distribution of the light transmission unit 30 and the wavelength of incident light (external light) based on the equation (5). It is applied separately for R, G, and B color image data.

For example, the MTF inverse transform unit 104 obtains an MTF for each of R, G, and B colors based on the equations (2) and (3) (S130). Specifically, the diffracted light intensity H _diff (k _x , k _y ) for each of R, G, and B is detected, and the MTF is calculated for each of R, G, and B based on Expression (3). The MTFs for R, G, and B are MTF (red component), MTF (green component), and MTF (blue component), respectively.

The MTF inverse transform unit 104 performs fast Fourier transform on R, G, and B imaging data (imaging R data, imaging G data, and imaging B data) acquired by the imaging device 20 through the light transmission unit 30 and performs FFT [ ] Is obtained (S140). Specifically, the imaging R data R _{cam (x, y)} , the imaging G data G _{cam (x, y)} , the imaging B data B _{cam (x, y)} obtained by imaging with the imaging device 20 through the light transmission unit 30 _. by performing a fast Fourier transform on each of _{y), FFT [R cam (} x, y)], FFT [G cam (x, y)], obtaining the _{FFT [B cam (x, y} )].

The MTF inverse transform unit 104 divides the FFT [] obtained in step S140 by the MTF of the corresponding color obtained in step S130 for each of R, G, and B colors (S150). Specifically, FFT [R _{cam (x, y)} ] / MTF (red component), FFT [G _{cam (x, y)} ] / MTF (green component), FFT [B _{cam (x, y)} ] / Obtain MTF (blue component).

The MTF inverse transform unit 104 performs fast inverse Fourier transform on the division result obtained in step S150 for each of R, G, and B colors _, and restores the original image I _{real (x, y)} for each color (S160). Specifically, the original red image R _{real (x, y)} is acquired by IFFT [FFT [R _{cam (x, y)} ] / MTF (red component)] according to the equation (5), and IFFT [FFT [G _{cam (x, y)} ] / MTF (green component)] to obtain a green original image _{Greal (x, y)} , and IFFT [FFT [B _{cam (x, y)} ] / MTF (blue component)] The original image B _{real (x, y)} is acquired.

The MTF inverse transform unit 104 restores the red original image R _{real (x, y)} , the green original image G _{real (x, y)} , the blue original image B _{real (x,} Based on _y) , a color restoration image is acquired for the specific region (S180).

  The MTF inverse transform unit 104 synthesizes the extracted image (its signal component) of the specific region that has undergone diffraction correction and the image (its signal component) of the background region that has not been subjected to diffraction correction processing to complete the processing (image display) (Displayed on the unit 10) (S40).

  In the diffraction correction process of the second example, the RTF, G, and B signal components constituting the image information are subjected to the MTF inverse transform process, so that a processing load is applied, but compared to a process that focuses on some signal components. The correction effect is high, and the unclearness appearing in the captured image due to the diffraction phenomenon can be accurately corrected.

<Diffraction correction process: third example (all signal components are subject to color conversion)>
FIG. 18 is a diagram (flow chart) for explaining a third example of diffraction correction processing by the diffraction correction unit 100. The diffraction correction process of the third example is a method of converting the color space of the imaging data of each component constituting the image information acquired by the imaging device 20 and processing the converted image data for each data. Hereinafter, the difference from the second example will be mainly described. The processing steps are shown in the 200s, and the same processing steps as in the second example are shown in the 10s and 1s as in the second example.

  The MTF inverse transform unit 104 performs the MTF inverse transform process only on the extracted specific region to remove the influence of diffraction (S30). At this time, the MTF inverse transform unit 104 converts the captured R data, captured G data, and captured B data in the RGB color space to other color spaces (for example, the XYZ color space or the Yuv color space) for the extracted image of the specific region. (S210).

  In this example, the color space conversion process is performed after first extracting the image information of the specific area, but conversely, the color space conversion process is performed for all the image areas acquired by the image display unit 10. The image information of the specific area may be extracted from the image. In the latter case, the processing amount is increased and the conversion processing time is increased as compared with the case where the color space conversion processing is performed only on the image information of the specific region. In consideration of this point, the former method is adopted in this example.

  In step S210, the MTF inverse conversion unit 104 acquires the MTF inverse conversion process calculated based on the shape, size, distribution, and wavelength of incident light (external light) of the light transmission unit 30 based on Expression (5). This is applied to each image data after color conversion. It is preferable that the color conversion includes at least a signal component representing luminance information. FIG. 18A shows a case where the RGB color space is converted into the XYZ color space, and FIG. 18B shows a case where the RGB color space is converted into the Yuv color space. Below, it demonstrates as what converted from RGB color space to XYZ color space.

For example, the MTF inverse transform unit 104 obtains the MTF for each X, Y, Z image data based on the equations (2) and (3) (S230). Specifically, the diffracted light intensity H _diff (k _x , k _y ) for each of X, Y, and Z is detected, and the MTF is calculated for each of X, Y, and Z based on Expression (3). The MTFs for X, Y, and Z are MTF (X component), MTF (Y component), and MTF (Z component), respectively.

The MTF inverse transform unit 104 performs fast Fourier transform on the image data (transformed X data, transformed Y data, transformed Z data) for each of X, Y, and Z after color space conversion, and calculates FFT [] for each of X, Y, and Z. Obtain (S240). Specifically, FFT is performed by performing fast Fourier transform on each of the transformed X data X _{cam (x, y)} , transformed Y data Y _{cam (x, y)} , and transformed Z data X _{cam (x, y).} [X _{cam (x, y)} ], FFT [Y _{cam (x, y)} ], FFT [Z _{cam (x, y)} ] are obtained.

For each of X, Y, and Z, the MTF inverse transform unit 104 divides the FFT [] obtained in step S240 by the corresponding signal component MTF obtained in step S230 (S250). Specifically, FFT [X _{cam (x, y)} ] / MTF (X component), FFT [Y _{cam (x, y)} ] / MTF (Y component), FFT [Z _{cam (x, y)} ] / MTF (Z component) is obtained.

The MTF inverse transform unit 104 performs fast inverse Fourier transform on the division result obtained in step S250 for each of X, Y, and Z _, and restores the original image I _{real (x, y)} for each of X, Y, and Z (S260). . Specifically, according to the equation (5), an X original image X _{real (x, y)} is acquired by IFFT [FFT [X _{cam (x, y)} ] / MTF (X component)], and IFFT [FFT [Y _{cam (x, y)} ] / MTF (Y component)] to obtain the Y original image Y _{real (x, y)} , and IFFT [FFT [Z _{cam (x, y)} ] / MTF (Z component)] to obtain Z The original image Z _{real (x, y)} is acquired.

The MTF inverse transform unit 104 restores the X original image X _{real (x, y)} , the Y original image Y _{real (x, y)} , and the Z original image Z _{real (x, y)} reconstructed for each of X, Y, and Z in step S260. Is returned to the original image in the RGB color space, that is, converted into image data in the RGB color space (S270). Thereby, the red original image R _{real (x, y)} , the green original image G _{real (x, y)} , and the blue original image B _{real (x, y)} are restored.

The MTF inverse transform unit 104 restores the red original image R _{real (x, y)} , the green original image G _{real (x, y)} , the blue original image B _{real (x,} Based on _y) , a color restoration image for the specific area is acquired (S280). The color conversion image in step S270 may be omitted and the color restored image may be acquired directly.

  The MTF inverse transform unit 104 synthesizes the extracted image (its signal component) of the specific region that has undergone diffraction correction and the image (its signal component) of the background region that has not been subjected to diffraction correction processing, thereby completing the processing (S40). ).

  The diffraction correction process of the third example involves a color conversion process and thus requires a processing load. However, since the diffraction correction process is performed separately for the luminance component and the color component, the correction coefficient is made different between the luminance component and the color component. There is an advantage that can be. Further, since the MTF inverse transform process is performed on each signal constituting the image information, the correction effect is higher than the process focusing on a part of the signals, and the blurring appearing in the captured image due to the diffraction phenomenon can be accurately corrected. For example, when an image is represented by signals of three (for example, R, G, B) color components, the correction speed is about 0.8 times (0.2 is the color) by performing the correction processing of the third example. Loss due to conversion process).

<Diffraction correction processing: fourth example (specific signal emphasis processing: green with strong correlation with luminance signal component)>
FIG. 19 is a diagram (flow chart) for explaining a fourth example of diffraction correction processing by the diffraction correction unit 100. The diffraction correction process of the fourth example is a method of applying a specific signal-oriented technique for processing at least one (but not all) of a plurality of signal components constituting image information acquired by the imaging device 20.

  In the following, the processing is focused on only one of a plurality (for example, R, G, B, X, Y, Z, Y, u, v after color conversion) constituting the image information. Hereinafter, the difference from the second example will be mainly described. The processing steps are shown in the 300s, and the same processing steps as in the second example are shown in the 10s and 1s as in the second example.

  The MTF inverse conversion unit 104 performs MTF inverse conversion processing calculated based on the shape, size, distribution, and wavelength of incident light (external light) of the light transmission unit 30 based on Expression (5) as R, G , B is applied only to one of the colors, that is, the diffraction correction processing is applied only to one of the colors. At this time, it is preferable to perform processing while paying attention to the green image information having a relatively strong correlation with the luminance signal component, that is, the diffraction correction processing is performed only for green having a strong correlation with the luminance signal component.

For example, the MTF inverse transform unit 104 obtains the M color MTF based on the equations (2) and (3) (S330). Specifically, the G color diffracted light intensity H _diff (k _x , k _y ) is detected, and the G color MTF (green component) is calculated based on Equation (3).

The MTF inverse transform unit 104 performs fast Fourier transform on the G color imaging G data acquired by the imaging device 20 through the light transmission unit 30 to obtain G color FFT [] (S340). Specifically, FFT [G _{cam (x, y)} ] is obtained by performing fast Fourier transform on the _captured G data G _{cam (x, y)} obtained by imaging with the imaging device 20 through the light transmission unit 30. Ask.

The MTF inverse transform unit 104 divides the FFT [G _{cam (x, y)} ] obtained in step S340 by the M color MTF (green component) obtained in step S330 for the G color, thereby _{obtaining the} FFT [G _{cam. (x, y)} ] / MTF (green component) is obtained (S350).

The MTF inverse transform unit 104 performs fast inverse Fourier transform on the division result obtained in step S350 for the G color to restore the original image G _{real (x, y)} for the G color (S360). Specifically, the green original image G _{real (x, y)} is acquired by IFFT [FFT [G _{cam (x, y)} ] / MTF (green component)] according to the equation (5).

The MTF inverse transform unit 104, the green original image G _{real (x, y)} restored for the G color in step S360, the unprocessed R color imaging R data R _{cam (x, y)} and the B color imaging B, respectively. Based on the data B _{cam (x, y)} , a color restored image is acquired (S380).

  In the diffraction correction process of the fourth example, the MTF inverse transform process is performed for at least one of R, G, and B (but not all colors) constituting the image information, so the second and third examples are processed for all signal components. However, the processing load is reduced, and the diffraction correction processing can be speeded up. For example, when an image is represented by signals of three (for example, R, G, B) color components, the correction speed is tripled by performing processing focusing on one color.

  In the fourth example, it is conceivable to perform processing while paying attention to the R color and the B color, but preferably all signal components are processed by paying attention to the green image information having a relatively strong correlation with the luminance signal component. It has been confirmed that a correction effect comparable to that of the second and third examples of processing can be obtained. For example, the strongest color component in external light may be detected by the wavelength distribution measurement unit, or the strongest color component in the captured image may be detected from the signal of the captured image, and processing may be performed on the color component. A well-balanced diffraction correction process was realized in terms of both the correction effect and the processing load. In comparison with the fifth example, which will be described later, since the color conversion process is not involved, the processing load is lighter than in the fifth example.

  In addition, the combined use of the “processing area reduction method” and the “specific signal emphasis method” adds the effect (processing speed α · X times) of the diffraction correction processing to the extracted specific region only. Can significantly reduce the processing time. Further, for example, in the “processing area reduction method”, a region of a specific color (for example, blue) or a region having a large amount of a specific color component (for example, a region having the largest B component among the three RGB primary colors) is referred to as a “specific region”. If the MTF inverse transform process is performed on the specific color component for the area, the process can be performed at a high speed and the correction effect of the area can be secured.

When the processing procedure of the fourth example shown in FIG. 19 is applied to the third example, processing steps corresponding to the color conversion processing (S210, S270) are added, and processing is performed with attention paid to the luminance signal component. Good. In this case, for MTF, MTF (Y component) is specified, and fast inverse Fourier transform is performed on FFT [Y _{cam (x, y)} ] / MTF (Y component). In comparison with the fifth example which will be described later, since the fast inverse Fourier transform is executed faithfully for the luminance signal component Y, the processing load is heavier than that of the fifth example, but the fidelity is in that the focus is on the luminance signal component. Therefore, the correction effect is higher than in the fifth example. For example, when an image is represented by three (for example, R, G, B) color component signals, the correction speed is not tripled, but is about 2.5 times (0.5 is the loss due to the color conversion process). )become.

<Diffraction correction processing: fifth example (specific signal emphasis processing: luminance signal component)>
FIG. 30 is a diagram (flow chart) for explaining a fifth example of the diffraction correction processing by the diffraction correction unit 100. The diffraction correction process of the fifth example converts the color space of the imaging data of each component constituting the image information acquired by the imaging device 20, and at least one (but not all) of the plurality of image data after color conversion. ). In the following, it is assumed that the process focuses on only one of a plurality (for example, X, Y, Z, Y, u, v) of color-converted image data. Hereinafter, the difference from the third example will be mainly described. The processing steps are shown in the 400s, and the same processing steps as those in the third example are shown in the 10s and 1s as in the third example.

  The MTF inverse conversion unit 104 performs an MTF inverse conversion process calculated based on the shape, size, distribution, and wavelength of incident light (external light) of the light transmission unit 30 based on the equation (5), in step S410. Is applied only to any one of the plurality of image data after color conversion obtained in step 4, that is, the diffraction correction processing is performed only for any one signal component. In this case, preferably, processing is performed by paying attention to the luminance signal component, that is, the luminance signal component is extracted from the plurality of image data after color conversion, and the green image having a relatively strong correlation with the luminance signal component. With reference to the information, only the luminance signal component is subjected to diffraction correction processing. In the following description, it is assumed that the RGB color space is color-converted to the XYZ color space.

For example, the MTF inverse transform unit 104 obtains the MTF of the luminance signal component Y based on the equations (2) and (3) (S430). Specifically, the G color diffracted light intensity H _diff (k _x , k _y ) is detected, and the G color MTF (green component) is calculated based on Equation (3).

The MTF inverse transform unit 104 performs FFT on the luminance signal component Y image data (transformed Y data) extracted from the color-converted XYZ data to obtain FFT [] of the luminance signal component Y (S440). Specifically, FFT [Y _{cam (x, y)} ] is obtained by performing fast Fourier transform on the transformed Y data Y _{cam (x, y)} .

The MTF inverse transform unit 104 divides the FFT [Y _{cam (x, y)} ] obtained in step S440 for the luminance signal component Y by the M color MTF (green component) obtained in step S430, thereby obtaining the FFT [ Y _{cam (x, y)} ] / MTF (green component) is obtained (S450).

The MTF inverse transform unit 104 performs fast inverse Fourier transform on the division result obtained in step S450 for the luminance signal component Y to restore the original image Y _{real (x, y)} for the luminance signal component Y (S460). Specifically, the green original image Y _{real (x, y)} is acquired by IFFT [FFT [Y _{cam (x, y)} ] / MTF (green component)] according to the equation (5).

The MTF inverse transform unit 104, the original image Y _{real (x, y)} restored for the luminance signal component Y in step S460, the unprocessed transformed X data X _{cam (x, y)} and the transformed Z data Z _{cam (x , y)} is returned to the original image in the RGB color space, that is, converted into image data in the RGB color space (S470). Thereby, the red original image R _{real (x, y)} , the green original image G _{real (x, y)} , and the blue original image B _{real (x, y)} are restored.

In step S470, the MTF inverse transform unit 104 restores the red original image R _{real (x, y)} , the green original image G _{real (x, y)} , and the blue original image B _{real (x,} Based on _y) , a color restoration image is acquired (S480). The color conversion image in step S470 may be omitted and the color restored image may be directly acquired.

  The diffraction correction process of the fifth example is accompanied by a color conversion process, and thus a processing load is applied. However, since the MTF inverse conversion process is performed on at least one (but not all) of each signal component after color conversion, the process is performed on all signal components. The processing load is reduced as compared with the case of doing so. Therefore, although the correction effect is inferior to the case where all signal components are processed, the overall processing load tends to be reduced, and the diffraction correction processing can be speeded up.

  In the fifth example, if attention is paid to the luminance signal component and diffraction correction processing is performed only on the luminance signal component with reference to the green image information having a relatively strong correlation with the luminance signal component, the processing is performed on all signal components. It was confirmed that a correction effect comparable to that in the case of performing is obtained. The diffraction correction process is well balanced in terms of both the correction effect and the processing load.

In the fifth example, the MTF (G component) is specified for the MTF, and the fast inverse Fourier transform is performed on the FFT [Y _{cam (x, y)} ] / MTF (green component), so attention is paid to the luminance signal component. In this respect, the fidelity is inferior to that of the fourth example, but the processing load is lighter than that of the fourth example because MTF (green component) is substituted without obtaining MTF (Y component). For example, when an image is represented by signals of three (for example, R, G, and B) color components, the correction speed is 2.8 times although the correction speed is not tripled by the correction processing of the fifth example. Degree (there is an improvement of 0.3 over the fourth example).

[Summary of first to fifth examples]
As can be understood from the above description, in the diffraction correction processing of the present embodiment, various procedures such as those listed in the first to fifth examples can be considered, but these are aspects of processing speed and correction accuracy. There is superiority or inferiority. Therefore, it is preferable that the user can appropriately select which method is adopted.

<Modification of light transmission part>
FIGS. 31 to 33 are diagrams illustrating modifications of the light transmission unit 30 in the light transmission region 12. Here, FIGS. 31 to 32 (1) show a first modification, FIG. 32 (2) shows a second modification, and FIG. 33 shows a third modification. In FIGS. 31 to 32 (1), in order to simplify the drawings, each of the four light transmission parts 30 adjacent to one light transmission part 30 in each light transmission part 30 has this light transmission. It is not shown to be different from the size of the portion 30.

[First Modification]
When the light transmission part 30 is very small, a diffraction phenomenon occurs in the light transmission part 30, resulting in blurring of the image formed on the imaging device 20 and lack of clarity. Therefore, in the first modification and the second modification, the light transmission portion itself is devised so as to prevent the diffraction phenomenon from occurring in advance with respect to the size, shape, distribution (arrangement pitch, arrangement position relationship, etc.), and the like. Keep it. The first modified example (also a second modified example described later) is based on such a viewpoint.

  In the first modification shown in FIGS. 31 to 32 (1), the state of the light transmission part 30 of the light transmission region 12 is made random, specifically, at least the size, shape, and distribution of the light transmission part 30 One is random.

  When the size of the plurality of light transmission parts 30 is random, the size of each of the at least two light transmission parts 30 adjacent to the one light transmission part 30A of interest in the horizontal direction or the vertical direction is set to be light transmission. It is desirable to make it different from the size of the portion 30A. That is, the two light transmission portions 30 arranged in the horizontal direction and / or the vertical direction of the light transmission portion 30A, more preferably, the three light transmission portions 30 adjacent to the light transmission portion 30A, and even more preferably the light transmission The size of the four light transmission parts 30 adjacent to the part 30A may be different from the size of the light transmission part 30A.

  In the example shown in FIG. 31, as an example, 6 × 3 = 18 light transmission portions 30 are provided in the light transmission region 12, and one light transmission portion 30 of interest is set as the light transmission portion 30A, with respect to the light transmission portion 30A. Two light transmission parts 30 adjacent in the horizontal direction are referred to as a light transmission part 30B and a light transmission part 30C, and two light transmission parts 30 adjacent in the vertical direction are referred to as a light transmission part 30D and a light transmission part 30E. Other than the light transmission part 30A adjacent to the light transmission part 30B in the horizontal direction is a light transmission part 30f, and the two light transmission parts 30 adjacent in the vertical direction are a light transmission part 30g and a light transmission part 30h.

  As schematically shown in FIG. 31 (1), in all of the 18 light transmission parts 30, each of the other four light transmission parts 30 adjacent to one light transmission part 30 has this light transmission. The configuration is different from the size of the portion 30. In FIG. 31 (1), focusing on the light transmission part 30A, two light transmission parts 30B and 30C arranged adjacent to each other in the horizontal direction of the light transmission part 30A, and two arranged next to each other in the vertical direction. Each of the two light transmission portions 30D and 30E is different in size from the light transmission portion 30A. Focusing on the light transmission part 30B, the two light transmission parts 30A and 30f arranged adjacent to the light transmission part 30B in the horizontal direction and the two light transmission parts 30g arranged adjacent to each other in the vertical direction. , 30h are different in size from the light transmitting portion 30B. As a result, it was possible to avoid the occurrence of a diffraction phenomenon in the light transmission region 12.

  When the shape of the plurality of light transmission parts 30 is random, the shape of each of the at least two light transmission parts 30 adjacent to the one light transmission part 30A of interest in the horizontal direction or the vertical direction is changed to the one light. It is desirable to make it different from the shape of the transmission part 30A. That is, the two light transmission portions 30 arranged in the horizontal direction and / or the vertical direction of the light transmission portion 30A, more preferably, the three light transmission portions 30 adjacent to the light transmission portion 30A, and even more preferably the light transmission The shape of the four light transmission parts 30 adjacent to the part 30A may be different from the shape of the light transmission part 30A.

  For example, as schematically shown in FIG. 31 (2), in all of the 18 light transmission parts 30, each of the other four light transmission parts 30 adjacent to one light transmission part 30 has this. The configuration is different from the shape of the light transmitting portion 30. In FIG. 31 (2), focusing on the light transmitting portion 30A, two light transmitting portions 30B and 30C disposed adjacent to the light transmitting portion 30A in the horizontal direction, and 2 disposed adjacent to the vertical direction. Each of the two light transmission portions 30D and 30E is different in shape from the light transmission portion 30A. Focusing on the light transmission part 30B, the two light transmission parts 30A and 30f arranged adjacent to the light transmission part 30B in the horizontal direction and the two light transmission parts 30g arranged adjacent to each other in the vertical direction. , 30h are different in shape from the light transmitting portion 30B. Also by this, it was possible to avoid the occurrence of the diffraction phenomenon in the light transmission part 30.

  When the distribution of the plurality of light transmission parts 30 is random, the arrangement positional relationship between at least two light transmission parts 30 adjacent in the horizontal direction or the vertical direction with respect to one light transmission part 30A of interest is the light transmission. It is desirable that the light transmission part 30 adjacent to the part 30A in the horizontal direction and / or the vertical direction is different from the arrangement positional relationship in the same relation as the light transmission part 30A. In other words, each of the at least two arrangement pitches adjacent in the horizontal direction and / or the vertical direction with respect to the one arrangement pitch of interest is different from the one arrangement pitch. That is, the two arrangement pitches P arranged in the horizontal direction of the arrangement pitch PHA focusing on the horizontal direction may be different from the arrangement pitch PHA, and the two arrangement pitches PVA focusing on the vertical direction may be different. The arrangement pitch P may be different from the arrangement pitch PVA.

  In the example shown in FIG. 32A, as an example, 5 × 5 = 25 light transmission portions 30 are provided in the light transmission region 12, and one arrangement pitch P with the light transmission portion 30A of interest in the horizontal direction is arranged. A pitch PHA is used, and two arrangement pitches P that are adjacent to the arrangement pitch PHA in the horizontal direction are an arrangement pitch PHB and an arrangement pitch PHC. An arrangement pitch PHf other than the arrangement pitch PHA horizontally adjacent to the arrangement pitch PHB is used. Further, one arrangement pitch P with the light transmitting portion 30A of interest in the vertical direction is an arrangement pitch PVA, and two arrangement pitches P adjacent to the arrangement pitch PVA in the vertical direction are an arrangement pitch PVB and an arrangement pitch PVC. . Further, two arrangement pitches P that are adjacent to the arrangement pitch PVB in the vertical direction are an arrangement pitch PVg and an arrangement pitch PVh.

  As schematically shown in FIG. 32 (1), in all of the 25 light transmitting portions 30, each of the other two arrangement pitches PH adjacent to one arrangement pitch P in the horizontal direction is this arrangement pitch. Different from P, each of the other two arrangement pitches PV adjacent to one arrangement pitch P in the vertical direction is different from this arrangement pitch P. In FIG. 32 (1), focusing on the light transmitting portion 30A, each of the two arrangement pitches PHB and PHC arranged adjacent to each other in the horizontal direction of the arrangement pitch PHA is different from the arrangement pitch PHA. Each of the two arrangement pitches PVB and PVC arranged adjacent to each other in the vertical direction of the PVA is different from the arrangement pitch PVA. Further, when paying attention to the light transmitting portion 30B, each of the two arrangement pitches PHA and PHf arranged adjacent to each other in the horizontal direction of the arrangement pitch PHB is different from the arrangement pitch PVB, and is perpendicular to the arrangement pitch PVb. Each of the two arrangement pitches PVa and PVc arranged adjacent to each other is different from the arrangement pitch PVb. Also by this, it was possible to avoid the occurrence of the diffraction phenomenon in the light transmission region 12.

What is necessary is just to process the opening part 65A which comprises the light transmissive part 30 so that such a structure and structure may be obtained. Here, the minimum value and the minimum shape of the light transmission part 30 depend on the minimum processing dimension (for example, F: 0.5 μm) in the photolithography technique and the etching technique for providing the light transmission part 30. Therefore, the size of the light transmitting portion 30, to be defined by the aggregate of this unit the rectangular area F ² (or shapes derived on photolithography rectangular shape of the area F ²⁾ was defined as 1 unit, The shape of the light transmission part 30 is also defined by the aggregate of the units.

  The configuration and structure of the light transmission part in the first modification can be applied to the light transmission part 30 in the first and second embodiments.

[Second Modification]
In the second modification, the light transmission part has a double annular structure (double hollow structure). Specifically, the arrangement of a plurality of pixels constituting the image display unit 10 in the image display device 1 is schematically shown in FIG. 32 (2). The light transmission unit 30 includes the first light transmission unit 30A and the second light transmission unit 30A. The second light transmission unit 30B is configured to include the light transmission unit 30B and surround the first light transmission unit 30A. In FIG. 32 (2), the first light transmission unit 30A and the second light transmission unit 30B are hatched for clarity of the first light transmission unit 30A and the second light transmission unit 30B. By optimizing the size, shape, and arrangement of the first light transmission part 30A and the second light transmission part 30B and the positional relationship between the first light transmission part 30A and the second light transmission part 30B, the diffraction phenomenon is caused. It was possible to reliably suppress the occurrence.

  The configuration and structure of the light transmission part in the second modification can be applied to the light transmission part 30 in the first to second embodiments. Furthermore, you may combine a 1st modification and a 2nd modification.

[Third Modification]
In the third modification, the light transmission part 30 is formed in a cross-girder shape or an “L” shape. That is, as shown schematically in FIG. 33 (1) and (2), the arrangement of the plurality of pixels 11 (11R, 11G, and 11B) constituting the image display unit 10 is an image display with an imaging device of the third modified example. The apparatus is arranged on the back side of the image display unit 10, an image display unit 10 including a plurality of pixels 11 including display elements, a light transmission region 12 (light transmission unit 30) provided in the image display unit 10. The image capturing device 20 and the light condensing unit 21 that condenses the light that has passed through the light transmitting unit 30 on the image capturing device 20 are provided.

  For example, in the example shown in FIG. 33 (1), the light transmission part 30 is provided around all of the pixels 11 and has a cross-like shape. That is, the light transmission part 30 is provided on all the sides corresponding to the boundaries of the pixels, and is provided in common between the adjacent pixels 11. In the example shown in FIG. 33 (2), the light transmission portion 30 is provided in a part of the periphery of the pixel 11 and has an “L” shape. In other words, the light transmission portions 30 are provided on two consecutive sides among the sides corresponding to the boundaries of the pixels 11.

  In the third modification, the light transmission unit 30 is provided around at least one or more pixels 11. Specifically, the light transmission unit 30 is provided around 6 × 3 = 18 pixels 11.

  Except for the above points, the image display device with an imaging device can have the same configuration and structure as the image display device 1 of the first and second embodiments, and thus detailed description thereof is omitted.

  In the third modification, the light that has passed through the light transmission unit 30 provided around at least one pixel 11 is collected on the imaging device 20. Therefore, a high-precision microlens is not required to accurately tie an image to the imaging device 20, and the manufacturing cost of the image display device with the imaging device is not increased, and a sufficient amount of light is supplied to the imaging device 20. It can be condensed.

<Replacement of electronic device monitoring device>
FIG. 34 is a diagram illustrating an example of an electronic apparatus to which the image display device 1 of the present embodiment is applied. The image display device 1A and the image display device 1B are not limited to replacement of a monitor device that constitutes a personal computer, for example, and can be used as an alternative to a monitor device of various electronic devices. For example, it can be used as an alternative to a monitor device incorporated in a notebook personal computer (see FIG. 34 (1)). It can be used as an alternative to a mobile phone (see FIG. 34 (2)), although not shown, a PDA (personal digital assistant), a monitor device incorporated in a game machine, a conventional television receiver or the like. it can. In any case, the image display unit 10 is provided with a light transmission region 12 in which a light transmission unit 30 (not shown) is formed, and an imaging device 20 is provided on the back surface opposite to the display surface side.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. Various changes or improvements can be added to the above-described embodiment without departing from the gist of the invention, and embodiments to which such changes or improvements are added are also included in the technical scope of the present invention.

  Further, the above embodiments do not limit the invention according to the claims (claims), and all combinations of features described in the embodiments are not necessarily essential to the solution means of the invention. Absent. The embodiments described above include inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. Even if some constituent requirements are deleted from all the constituent requirements shown in the embodiment, as long as an effect is obtained, a configuration from which these some constituent requirements are deleted can be extracted as an invention.

  Furthermore, the image display device 1 and the image display system 2 described in the above embodiment can be further modified, and further inventions can be extracted. The following are some variations. In addition, although all are shown by the modification with respect to 1st Embodiment, it is not restricted to this, It can apply also to other embodiment.

[First Modification: Position Detection]
The first modification shown in FIGS. 35 to 36 includes a position detection unit 71 </ b> C that obtains position information of a subject based on image information acquired via the imaging device 20. The position detector 71C may be provided in the image display device 1C or may be provided in the peripheral device 70C.

  Since the function and effect of the configuration including the diffraction correction unit 100 described above can be enjoyed, the influence of the blur due to the diffraction phenomenon by the light transmission unit 30 can be suppressed, and the position information of the subject can be obtained with high accuracy. Examples of the subject include an observer (user) 's hand, finger, eyeball, and a rod-shaped object held by the user in his / her hand.

  If position information of a subject (for example, a hand, a finger, an eyeball, a rod-like object (for example, a pen or a pencil)) is continuously obtained in time series by the position detection unit 71C, the movement of the subject can be obtained. For this reason, it is possible to perform various processes corresponding to the movement of the subject (for example, moving the image up and down, moving left and right, closing the screen, opening another screen, etc. on the monitor device of the personal computer). The relationship between the movement of the subject and various processes may be registered in the position detection unit 71C.

  In some cases, the position detection unit 71C uses a well-known algorithm or software to determine the shape of the subject (for example, a shape expressed by the shape of the body or hand, a shape expressed by a combination of fingers, or a gesture) from the position information of the subject. Thus, various processes corresponding to the shape of the subject can be performed. If the direction in which the subject is directed is obtained by the position detection unit 71C, various processes corresponding to the direction in which the subject is directed can be performed.

[Second Modification: Three-dimensional image display + position detection]
In the second modification shown in FIGS. 37 to 38, a plurality (typically two) of imaging devices 20 are arranged on the back side of the image display unit 10, and images are based on image information from the imaging devices 20. The position detection unit 71D obtains the distance from the display unit 10 to the user. The position detection unit 71D may be provided in the image display device 1D or may be provided in the peripheral device 70D.

  Since the function and effect of the configuration including the diffraction correction unit 100 described above can be enjoyed, the influence of the blur due to the diffraction phenomenon by the light transmission units 30_1 and 30_2 can be suppressed, and the position information of the observer can be acquired with high accuracy.

  The position information of the observer can be the position data of both eyes of the observer, or the distance data from the image display unit 10 to the observer. Further, the position information of the observer can be obtained based on the eyes of the observer of the image data captured through the plurality of imaging devices 20_1 and 20_2. The position information of the observer can be configured to be displayed on the image display unit 10, so that the optimum three-dimensional image observation position is clearly shown to the observer so that the observer can easily observe the three-dimensional image. The user can be instructed or guided to the optimum three-dimensional image observation position. Or it can be set as the structure which optimizes the image displayed on the image display part 10 based on an observer's positional information.

[Third Modification: TV Conference System]
In the third modification shown in FIGS. 39 to 50, the mechanism of the above embodiment is applied to a so-called video teleconference system (video telephone device). The third modification further includes an information sending unit 80 for sending image information acquired via the imaging device 20, and a display control unit 82 for displaying an image based on image information inputted from the outside on the image display unit 10. I have. Image information acquired via the imaging device 20 is sent to the outside by the information sending unit 80, and an image based on the image information input from the outside is displayed on the image display unit 10 by the display control unit 82.

  The information sending unit 80 and the display control unit 82 may be provided in the image display device 1E or in the peripheral device 70E. In FIG. 39, for convenience, the information sending unit 80 and the display control unit 82 are shown on the pedestal portion of the image display device 1F (the main body). The same technique shown in the figure also applies other modifications described later.

  According to the third modified example, since the imaging device 20 is arranged on the back side of the image display unit 10, the user's face located in front of the image display unit 10 can be imaged, and the image display unit 10. Since the other user's face projected on the screen faces the user, there is no sense of incongruity that the lines of sight of the conventional TV phone system do not match each other. Since the function and effect of the configuration including the diffraction correction unit 100 described above can be enjoyed, the image of the user's face and the like in a state in which the influence of the blur due to the diffraction phenomenon by the light transmission unit 30 is suppressed is the image display unit of the other party 10 is projected.

[Fourth Modification: Digital Mirror]
The fourth modification shown in FIGS. 51 to 52 causes the image display device 1 of the above embodiment to function as a so-called digital mirror.

  In the fourth modification, an image information storage unit 86 that stores image information acquired via the imaging device 20, and image information and images acquired via the imaging device 20 (including the acquired status) are also included. A display control unit 88 that displays an image based on the image information stored in the information storage unit 86 on the image display unit 10 is further provided. The image information storage unit 86 and the display control unit 88 may be provided in the image display device 1F or may be provided in the peripheral device 70F.

  According to the fourth modification, the comparison result between the past and the current user can be displayed in separate windows in the image display unit 10. Since the function and effect of the configuration including the diffraction correction unit 100 described above can also be enjoyed, an image of the user's face and the like in a state where the influence of blur due to the diffraction phenomenon by the light transmission unit 30 is suppressed is displayed on the image display unit 10. Projected.

  DESCRIPTION OF SYMBOLS 1 ... Image display apparatus, 2 ... Image display system, 10 ... Image display part, 100 ... Diffraction correction part, 102 ... MTF shape memory | storage part, 104 ... MTF reverse conversion part, 11 ... Pixel, 110 ... Wavelength distribution measurement part, 12 DESCRIPTION OF SYMBOLS ... Light transmission area | region, 20 ... Imaging device, 20a ... Imaging part, 21 ... Condensing part, 30 ... Light transmission part, 70 ... Peripheral device, 90 ... Control part

Claims (20)

An image display in which a plurality of pixels including a display element are arranged, an imaging unit that captures an image on the back side can be arranged, and a plurality of light transmission units are provided in a region corresponding to the imaging unit And
Diffraction correction processing is performed to suppress the effect of appearing in the image information due to the diffraction effect by the light transmitting portion on the image information of the specific area extracted from the image information acquired by the imaging unit through the plurality of light transmitting portions, A diffraction correction unit that synthesizes image information that has not been subjected to the diffraction correction processing of the background region excluding the specific region and image information that has been subjected to the diffraction correction processing for the specific region;
An image display device comprising: The diffraction correction unit performs the diffraction correction process by performing a Fourier transform on the image information, and performing an inverse Fourier transform on the Fourier transformed information using a response function corresponding to an arrangement state of the light transmission unit. 2. The image display device according to 1. The image display device according to claim 1, wherein the diffraction correction unit extracts an image of a predetermined region in the image information acquired by the imaging unit. The image display device according to claim 1, wherein the diffraction correction unit extracts an image of a predetermined region of a specific object in the image information acquired by the imaging unit. The image display apparatus according to claim 4, wherein the diffraction correction unit sets the specific object as one of a person, a person's face, and a custom-made part of the person's face. The image display device according to any one of claims 1 to 5, wherein the diffraction correction unit performs the diffraction correction processing on each of a plurality of signals representing the image information. The image according to any one of claims 1 to 5, wherein the diffraction correction unit performs the diffraction correction processing on at least one (but not all) of a plurality of signals representing the image information. Display device. The image display apparatus according to claim 7, wherein the diffraction correction unit performs the diffraction correction processing on a signal having a correlation with luminance information among a plurality of signals representing the image information. The diffraction correction unit performs color space conversion on a plurality of signals representing the image information so that at least a signal component representing luminance information is included, and a signal component representing the luminance information among the plurality of converted signals. The image display device according to claim 7, wherein the diffraction correction process is performed. The diffraction correction unit performs color space conversion on a plurality of signals representing the image information so that at least a signal component representing luminance information is included, Fourier-transforms the signal component representing the luminance information, and Fourier-transformed information, The image display apparatus according to claim 7, wherein the diffraction correction process is performed by performing an inverse Fourier transform using a response function corresponding to an arrangement state of the light transmission unit for a component having a correlation with luminance information. It has a wavelength distribution measurement unit that measures the wavelength distribution of external light,
The image display device according to any one of claims 1 to 10, wherein the diffraction correction unit performs the diffraction correction processing with reference to a wavelength distribution of external light measured by the wavelength distribution measurement unit. The image display device according to any one of claims 1 to 11, wherein an arrangement state of the light transmission portions is non-uniform. The light transmission part is composed of a first light transmission part and a second light transmission part,
The image display device according to any one of claims 1 to 11, wherein the second light transmission unit is disposed so as to surround the first light transmission unit. 14. The image display device according to claim 1, further comprising a condensing unit that condenses light that has passed through the light transmission unit on the imaging unit. The image display device according to any one of claims 1 to 14, wherein the display element is a light emitting element. An imaging unit that captures an image;
A plurality of pixels including display elements, an image display unit in which the imaging unit is disposed on the back side, and a plurality of light transmission units are provided in a region corresponding to the imaging unit;
Diffraction correction processing is performed to suppress the effect of appearing in the image information due to the diffraction effect by the light transmitting portion on the image information of the specific area extracted from the image information acquired by the imaging unit through the plurality of light transmitting portions, A diffraction correction unit that synthesizes image information that has not been subjected to the diffraction correction processing of the background region excluding the specific region and image information that has been subjected to the diffraction correction processing for the specific region;
With electronic equipment. An image display in which a plurality of pixels including a display element are arranged, an imaging unit that captures an image on the back side can be arranged, and a plurality of light transmission units are provided in a region corresponding to the imaging unit A diffraction correction unit that performs a diffraction correction process that suppresses an influence appearing in the image information due to a diffraction effect by the light transmission unit, on the image information acquired by the imaging unit through the plurality of light transmission units of the unit,
The diffraction correction unit is
Image information of a specific area from image information acquired by the imaging unit through the plurality of light transmission units,
Diffraction correction processing is performed to suppress the influence that appears in the image information due to the diffraction effect by the light transmitting portion on the extracted image information of the specific region,
An electronic apparatus that synthesizes image information that has not been subjected to the diffraction correction process in a background area excluding the specific area and image information that has been subjected to the diffraction correction process for the specific area. An imaging device for capturing an image;
A plurality of pixels including a display element are arranged, the image pickup device can be arranged on the back side, and an image display portion provided with a plurality of light transmission portions in a region corresponding to the image pickup device is provided. An image display device,
Diffraction correction processing is performed to suppress the effect of appearing in the image information due to the diffraction effect by the light transmission portion on the image information of the specific region extracted from the image information acquired by the imaging device through the plurality of light transmission portions, A diffraction correction unit that synthesizes image information that has not been subjected to the diffraction correction processing of the background region excluding the specific region and image information that has been subjected to the diffraction correction processing for the specific region;
An image display system comprising: An image display in which a plurality of pixels including a display element are arranged, an imaging unit that captures an image on the back side can be arranged, and a plurality of light transmission units are provided in a region corresponding to the imaging unit Display an image on the
Image information is acquired by imaging the subject on the display surface side with the imaging unit through the plurality of light transmission units,
Extract image information of a specific area from the acquired image information,
Diffraction correction processing is performed to suppress the influence that appears in the image information due to the diffraction effect by the light transmitting portion on the extracted image information of the specific region,
An image acquisition method for synthesizing image information that has not been subjected to the diffraction correction process in the background area excluding the specific area and image information that has been subjected to the diffraction correction process for the specific area. An image display unit in which a plurality of pixels including a display element are arranged, an imaging unit that captures an image on the back side can be arranged, and a plurality of light transmission units are provided in a region corresponding to the imaging unit A step of extracting an image of a specific area from image information acquired by imaging a subject on the display surface side by the imaging unit through the plurality of light transmission units;
A step of performing a diffraction correction process for suppressing an influence appearing in the image information due to a diffraction effect by the light transmission unit on the extracted image information of the specific region;
Combining the image information that has not been subjected to the diffraction correction processing of the background region excluding the specific region with the image information that has been subjected to the diffraction correction processing for the specific region;
A program that causes a computer to execute.