JP2012103753A - Image processing method, image processor and correction processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To correct an image based on the degree of aberration in consideration of the fact that the degree of aberration that occurs on the retina of a viewer varies according to the viewer's pupil diameter, in order to solve the problem in which the sharpness of an image is deteriorated due to aberration that occurs on a viewer's retinal images when the viewer views the image displayed on a display device.SOLUTION: An image processing method involves an aberration estimation process 206 in which to estimate the amount of aberration occurring on the eyes of a viewer using information on the viewer's pupil diameter, a correction amount acquisition process 208 in which to convert the estimated aberration amount obtained in the aberration estimation process to the amount of correction for an input image and a correction process 217 in which to perform the correction based on the correction amount obtained in the correction amount acquisition process.

Description

  The present invention relates to an image processing method that improves the image quality of an image perceived by an observer by performing an image correction process in consideration of the aberration of the eye of the observer.

  Humans perceive an object by forming an optical image of the object on the retina with the crystalline lens of the eye and recognizing a signal obtained by the visual nerve cell on the retina as an image in the brain. Since the lens is not suppressed in aberrations unlike a lens used for taking a photograph, the retinal image includes a certain amount of aberration. Among aberrations, it is known that chromatic aberration cannot be suppressed by a single lens made of a material having dispersion such as a crystalline lens. Therefore, compared to green light which is easy to focus on the retina, red light and blue light are not accurately focused and are so-called “blurred” images.

  The reason that the influence of aberration is hardly felt in the perceived image is that correction is performed when a signal obtained by the visual nerve cell on the retina is recognized as an image in the brain.

  A display device and an image processing method that improve the image quality in consideration of the characteristics of the eyes of the observer are disclosed in Patent Document 1.

  The vehicle display device disclosed in Patent Document 1 has a line-of-sight detection function and a function of comparing a retinal image of a driver's eye with a photographing result outside the vehicle. The retinal image is generated by applying the optical transfer function of the driver's eye recorded in advance to the photographing result outside the vehicle. The purpose is to alert the driver by highlighting a point that is difficult to visually recognize using these functions. Further, as an optical transfer function of the driver's eyes, an aberration inherent to the driver is also illustrated.

JP-A-6-233306

  In general, the aberration of a single lens is the largest for light rays that have passed through the periphery of the lens.

  The same applies to the eye lens. The effect of aberration is greater when the pupil of the eye is wide open than when the eye pupil is small open.

  As a result, the appearance (image quality) of the image perceived without being corrected in the brain is affected.

  In the case of a display device such as a liquid crystal television set, when the room where the display device is installed is dark, the pupil of the observer who is watching the display device opens up and aberrations are strongly generated. The image quality perceived for light that is difficult to match will be reduced.

  As in the method described in Patent Document 1, a method for correcting a display image based on a gaze direction and gaze point information obtained by using gaze detection is known. However, since aberrations that affect image blur are not caused by changes in the line-of-sight direction, such information cannot be used for correction.

  Further, Patent Document 1 also describes a means for recording aberrations of the eyes of the driver (observer of the display device). However, there is no consideration for changes in the pupil diameter of the driver, and therefore, correction of aberrations that change with the pupil diameter cannot be handled.

The image processing method of the present invention includes:
An aberration estimation step for estimating aberrations occurring in the eyes of the observer of the display image displayed in the image display device;
A correction amount acquisition step of converting the aberration estimation amount obtained by the aberration estimation step into a correction amount of the input image;
A correction step of performing a correction process on the image information according to the correction amount obtained by the correction amount acquisition step;
It is characterized by having.

  According to the present invention, the observer perceives the display image displayed on the display device by estimating the aberration generated in the eye of the observer and performing correction for suppressing the influence of the aberration on the image information. Image appearance (image quality) can be improved.

1 is a configuration diagram of a display device according to Embodiment 1. FIG. FIG. 6 is a diagram illustrating a flow of image processing performed by a video control unit of the display device according to the first embodiment. It is the figure which described the relationship between the variable of Numerical formula 1, and a physical quantity. It is the figure which showed the relationship between the retinal image brightness | luminance in Numerical formula 2, and a pupil diameter. It is a figure explaining the relationship between MTF and correction amount regarding blue and green. 6 is a configuration diagram of a display device according to Embodiment 2. FIG. FIG. 10 is a diagram illustrating a flow of image processing performed by a video control unit of a display device according to a second embodiment. It is a figure explaining the relationship between the pupil diameter assumed in Example 2, and a pulse. FIG. 10 is a configuration diagram of a display device according to a third embodiment. It is a block diagram of the small camera for pupil diameter measurement. FIG. 10 is a diagram illustrating a flow of image processing performed by a calculation / memory module of a display device according to a third embodiment. FIG. 10 is a diagram illustrating a flow of image processing performed by a video control unit of a display device according to a fourth embodiment.

  First, matters common to the embodiments of the present invention will be described. In the aberration estimation step constituting the present invention, the aberration occurring in the observer's eye is estimated by obtaining the pupil diameter. Here, aberrations affected by changes in pupil diameter are handled. Such aberrations include “spherical aberration” that occurs due to differences in the focal lengths of light having different pupil passing positions, and “longitudinal chromatic aberration” that is caused by differences in the focal lengths of light having different wavelengths.

  When spherical aberration occurs, the optical image is totally blurred and the sharpness is lowered. In the correction for spherical aberration, correction is performed so that a blurred image is brought closer to the focused image.

  On the other hand, in the case of longitudinal chromatic aberration, an image of blue light whose focal length is shortened is more blurred than other colors. In the case of a farsighted observer, the blue light with a shorter focal length is more easily focused, and the red light is blurred. Depending on the shape of the crystalline lens, a plurality of color lights may be greatly blurred. In the correction for longitudinal chromatic aberration, correction is performed for a color that is more blurred than a color that is easily focused. When there are a plurality of colors that are likely to be blurred, correction is performed on the plurality of colors.

  The acquisition of the pupil diameter is performed by direct measurement or estimation from information other than the pupil diameter. As the direct measurement, for example, a region including the pupil when the observer is looking at the screen is photographed, and the size of the pupil is obtained from the image. Estimating from information other than the pupil diameter is a function of the relationship between parameters affecting the pupil diameter and the pupil diameter, such as brightness of the display screen, ambient light, biological reactions such as heart rate and sweating, room atmosphere, etc. Or by creating a table. Parameters such as time and temperature do not directly affect the pupil diameter, but affect parameters affecting the pupil diameter, such as ambient light and perspiration, and can be used for estimation in a similar manner.

  In the correction amount acquisition process and the correction process that constitutes the present invention, image processing that can offset the deterioration is applied to the display image so as not to cause deterioration in image quality (suppression of high-frequency components or change in gradation) due to aberrations. Apply. Specifically, in order to cancel out suppression of the high frequency component, high frequency enhancement is performed using a high frequency component enhancement filter, or gradation conversion having an effect opposite to the change in gradation is performed.

  However, since a psychological correction such as color constancy is performed when a change in gradation or high frequency component is perceived, the correction process is not executed based only on the aberration value. Also, due to psychological correction, a similar perceptual effect may be obtained even if different processing is performed. Regarding the enhancement of high frequency components, psychologically similar effects can be expected even if the high frequency component enhancement filter and the edge enhancement filter are strictly different processes. Since the effect of psychological correction can be estimated by subjective evaluation experiments, accurate correction is possible by discounting the effect from the correction amount obtained from the aberration or by directly obtaining the correction amount from the subjective evaluation experimental value. .

  A first embodiment of the present invention will be described with reference to FIG. The display device of the present invention includes a display screen 101, a remote controller 102, a control signal receiving unit 103, a peripheral light amount measuring unit 104, an observer position ranging unit 105, a video control unit 106, and a video input / output terminal 107.

  A flow of processing in the display device of the present invention will be described. The display program control signal issued from the remote controller 102 is received by the control signal receiving unit 103. A moving image signal to be displayed is input to the video input / output terminal 107 in accordance with the display program control signal.

  The input moving image signal is divided into frames and transmitted to the video control unit 106 in order. In synchronism with the input of one frame in the moving image to the video control unit 106, the illuminance around the display device (peripheral light amount measurement value) is obtained by a simple photometer provided in the peripheral light amount measurement unit 104. Measured and transmitted to the video controller 106.

  Similarly, the distance to the observer is measured by the observer position ranging unit 105 and transmitted to the video control unit 106. The observer position ranging unit 105 is equipped with a measuring device that applies multi-point ranging, which is generally known for the autofocus function of a single-lens reflex camera. Specifically, the distance is measured with respect to the distance measuring point selected in advance on the setting screen, and the average value of the distance is set as the distance to the observer.

  After the image control unit 106 performs correction processing based on the image processing method proposed in the present invention using the image information included in one frame, the peripheral light amount measurement value, and the observer distance, Then, color conversion processing and gradation conversion processing performed in a normal display device are performed.

  The flow of image processing executed by the video control unit 106 will be described with reference to FIG. The image data 201 input to the video control unit 106 is a color image composed of three color signals of RED, GREEN, and BLUE. One frame is an R plane composed of a RED signal, a G plane composed of a GREEN signal, and a BLUE signal. It is comprised from three color image planes (RGB surface) of the B surface which consists of these. In order to remove the influence of the gamma correction already applied to the image data, the image data 201 composed of such RGB planes has a gradation so that the pixel value of the image data is proportional to the display luminance by the linearization processing step 216. Convert. Next, an average luminance estimated value 203 (hereinafter abbreviated as display screen luminance) on the display screen is calculated by the average on-screen luminance estimation step 202. In calculating the display screen luminance 203, an average pixel value in the G plane of the image data 201 is obtained, and a luminance value corresponding to the average pixel value is obtained from the luminance characteristics of the display screen. The reason why the average pixel value on the G plane is used for the estimation is that the spectral sensitivity characteristic of the human eye with respect to the brightness is almost the same as the spectral luminance characteristic of the green pixel used in the display device.

[Aberration estimation process]
Next, in an aberration estimation step 206, the pupil diameter is estimated using the average retinal image luminance of the observer, and the chromatic aberration of the eye is estimated.
An aberration estimation value 207 is calculated from the display screen luminance 203, the peripheral light amount measurement value 204, and the observer distance 205. In the aberration estimation step 206, first, the average luminance of the retinal image with the observer's pupil open (hereinafter abbreviated as retinal image luminance) is estimated by Equation 1.

  Here, Α is the viewing angle, Φ is the area of the display screen, D is the distance from the observer to the display screen, P and Q are variables relating to luminance, and S and T are variables relating to the ratio.

  The relationship between Formula 1 and the physical quantity will be described with reference to FIG. The luminance obtained by Equation 1 is obtained by calculating the average value of the luminances of the regions that fall within the visual field when the virtual diffuse reflection surface 308 exists around the display screen 301. Since the display screen 301 and the retina 304 are in an imaging relationship, it is based on the fact that the average luminance value obtained is almost the same as the average luminance value on the retina.

  A more detailed description of the parameters of Equation 1 will be given. FIG. 3 shows the relationship between the display screen 301 and the eyes 302 of the observer. The observer's eye 302 is roughly composed of a crystalline lens 303 and a retina 304, and the display screen 301 and the retina 304 are in an imaging relationship. The variable の in Equation 1 is the viewing angle 305, the variable Φ is the display screen area 306, and the variable D is the distance 307 from the observer to the display screen.

  The variable P is the average brightness of the virtual diffuse reflection surface 308 that is supposed to exist around the display screen 301 and can be obtained as being illuminated by the ambient light amount measurement value 204 by the illumination 309. The variable Q is the display screen brightness 203 estimated with respect to the display screen 301.

  A variable S is a luminance correction ratio by image formation. As described in Non-Patent Document 1, it is known that if the optical system has sufficiently suppressed aberration, the luminance of the object and the luminance of the image coincide with each other, but the aberration of the crystalline lens is suppressed. Since the transmittance is low, luminance correction is necessary. Here, a value obtained in advance by an illumination simulation for the crystalline lens is used as the luminance correction ratio.

  The variable T is a constant corresponding to the installation environment of the display device. When the display device is installed on a white wall, the diffuse reflection surface actually exists around the display device. Therefore, when T = 1, the case where the display device is installed away from the wall If the color is close to black, T is brought close to 0. This value is set by the observer when the display device is installed.

  Next, the pupil diameter is calculated from the retinal image luminance obtained by Equation 1 using the function R given by Equation 2.

Here, B is the retinal image luminance estimated according to Equation 1, R _MIN and R _MAX are the minimum and maximum values of the pupil diameter, B _MIN is the maximum value of the retinal image luminance estimated when the pupil diameter is R _MAX , B _MAX is a minimum value of the retinal image luminance estimated when the pupil diameter is R _MIN . R _MIN , R _MAX , B _MIN , and B _MAX can be easily determined by experiment.

In order to deepen the understanding of Equation 2, changes in the estimated pupil diameter when the retinal image brightness is increased from 0 will be described with reference to FIG. If retinal image brightness is less than B _MIN, because the image is dark ugly state, pupil diameter such that the light that reaches the retina is maximum is adjusted to the maximum value R _MAX. When the retinal image luminance is between B _{MIN and} B _MAX , the pupil diameter is adjusted so that the amount of light in the retinal image is constant. The amount of light of the retinal image is approximately given as (retinal image luminance) × (pupil area) × (retinal area) / (focal length of the lens), but (retinal area) and (focal length of the lens) are Since there is no change, the pupil diameter is adjusted so that (retinal image luminance) × (pupil area) becomes a constant constant. In Equation 2, so that the pupil diameter becomes the reference light amount in the case of the maximum, and the constant and ^{_{B MIN × ΠR MAX 2/4}} . Needless to say, a constant is determined based on the minimum pupil, or a function that smoothly connects between the minimum and maximum pupils instead of the constant is (retinal image brightness) × (pupil area). It is good as a standard. If retinal image brightness is B _MAX or more, because it is state image is too bright, the pupil diameter such that the light that reaches the retina is minimized is adjusted to a minimum value R _MIN.

  If the pupil diameter can be estimated, MTF (MODULATION TRANSFER FUNCTION), which is an index representing the reproducibility of stripes formed on the display screen, is obtained by optical simulation by using statistically obtained lens shape / refractive index distribution information. It becomes possible to calculate. The aberration estimation value 207 estimated in the aberration estimation step 206 is physically the same as the axial chromatic aberration generally used in optical design, but the MTF of different colors is suitable for image correction performed in the subsequent steps. A function F (Ν) (Expression 3) representing the ratio is used.

  Here, MTFΛ, R (Ν) is the MTF of the eye that occurs when the wavelength of the incident light is Λ and the pupil diameter is R, and is the result obtained by the above simulation. The wavelength of the incident light of the MTF is 550 and 470 NM for each numerator denominator. These numerical values assume the wavelength at which the spectral luminance of the green and blue pixels of the display screen is maximized. In the subsequent process, such a wavelength was selected in order to perform color correction using green and blue pixels, but when dealing with pixels of other colors or when the spectral luminance distribution is special, the wavelength is selected. Or an average with the MTF for a nearby wavelength may be used.

  Since Formula 1-3 requires an optical simulation in the middle of the calculation, the aberration estimation step 206 does not perform the calculation, but creates an LUT (LOOK-UP TABLE) by concatenating the three formulas in advance and instantaneously estimates the aberration. The value 207 can be calculated.

[Correction amount acquisition process]
In the next correction amount acquisition step 208, the correction amount 209 is calculated using the aberration estimated value 207. The correction amount 209 obtained here is a numerical value related to the spatial frequency amplification of the B surface of the image data 201 and is given by Expression 4.

  The variables ΝMIN and ΝMAX are the minimum and maximum values of the spatial frequency band to be corrected, and N is the number of sample points in the spatial frequency band. Since knowledge of the correction process is required to understand the contents of Equation 4, it will be described together with the next step.

[Correction process]
In the next correction step 210, the correction processing of the image data 201 is performed using the correction amount 209. The correction step 210 includes steps of B-plane extraction 211, spatial frequency component separation 212, high-frequency component amplification 213, and image composition 214. In the B surface extraction step 211, the B surface to be corrected is extracted from the RGB surfaces of the image data 201. In the spatial frequency component separation step 212, the B surface is separated into a low frequency component and a high frequency component by filtering. In the present embodiment, a low frequency component is extracted using a Gaussian filter, and a high frequency component is extracted by subtracting the low frequency component from the original data. The cut-off frequency will be described later.

  In the high frequency component amplification step 213, the correction amount 209 is integrated with respect to the high frequency component on the B surface. In the image synthesis 214, the high frequency component processed in the high frequency component amplification step 213 and the low frequency component separated in the spatial frequency component separation step 212 are synthesized to create a corrected B-plane. The corrected image data 215 is generated by combining with the R and G planes.

  Here, the contents of Expression 4 and how to select the cutoff frequency in the spatial frequency component separation step 212 will be described with reference to FIG. FIG. 5A shows a green MTF 501 corresponding to the case where the incident light wavelength is 550 NM (green light) and the pupil diameter is maximum, and the case where the incident light wavelength is 470 NM (blue light) and the pupil diameter is maximum. A blue MTF 502 is schematically shown. Due to the influence of chromatic aberration, blue light is blurred compared with green light, so that it can be confirmed that a difference occurs in high frequency components. In the correction step 210, the high frequency component of the light intensity distribution (that is, the pixel value on the B surface) formed by the blue light is amplified so that the high frequency component is about the same for green light and blue light.

  Ideally, the spatial frequency components on the B surface should be finely separated and the function F given by Equation 3 should be integrated for each frequency component. Absent. Therefore, as in the spatial frequency component separation step 212, the spatial frequency is separated into two and the high frequency components are uniformly amplified. A blue MTF 503 after amplification is schematically shown in FIG. Thus, the slope of the high frequency component becomes gentler than that of the blue MTF 502, and the difference from the green MTF 501 becomes smaller.

The function G given by Expression 4 is an amplification factor that is uniformly integrated with the high-frequency component, and uses the average (synergistic average) value of the MTF ratio given by Expression 3. The spatial frequency band for which the average is obtained is designated by G variables Ν _MIN and Ν _MAX , and the number of sample points in the spatial frequency band is designated by N. For both Ν _MIN and 両 者_{MAX, it} is necessary to select a numerical value in the vicinity of the cutoff frequency 504, but if noise appears in the amplified image, a low-frequency numerical value is selected.

  As can be seen from the shape of the blue MTF 503 after amplification, a discontinuous step occurs in the vicinity of the cut-off frequency 504, which may cause image quality degradation. For this reason, as the cutoff frequency 504, a low frequency component that is easily captured by the eye is avoided, and a high frequency that is about a fraction of the Nyquist frequency at the time of display or a frequency at which MTF is sufficiently suppressed is selected. In addition, as is obvious, if the spatial frequency component separation step 212 divides into three or more instead of two of low frequency and high frequency, a step generated at the frequency of the divided boundary can be suppressed.

  The post-correction image data 215 is subjected to image processing in a normal display device such as color temperature adjustment and gamma adjustment in a pre-display processing step 217. The obtained display image data 218 becomes the output of the video controller 106.

  The display image data 218 is displayed on the display screen 101 and presented to the observer.

  As described above, the first embodiment of the present invention obtains chromatic aberration information generated in the eyes of the observer by estimating the pupil diameter from information such as the illuminance of the ambient light, and corrects the chromatic aberration of the observer. As described above, the display device includes a color correction function that emphasizes the high-frequency component of the B surface.

A second embodiment of the present invention will be described.
In the first embodiment, the amount of peripheral light of the display device is measured to obtain the average luminance of the retinal image, and the pupil diameter is estimated.However, the situation in which the pupil diameter increases is not necessarily determined only by the peripheral light amount, and the psychological state of the observer Etc. are also affected. The second embodiment is characterized in that the pupil diameter is estimated using the observer's heartbeat and time information at the time of observation. Furthermore, assuming a hyperopic observer who has not performed visual acuity correction, color correction is performed on the R plane as a second feature.

  The configuration of the display device shown in FIG. 6 is almost the same as the configuration of the first embodiment, except that a high-performance microphone 608, an Internet connection terminal 609, and a built-in computer 610 are provided.

  Since the normal processing flow is the same as in the first embodiment, only the differences will be described below.

  In the pupil diameter estimation according to the second embodiment, the output values of the peripheral light amount measurement unit 604 and the observer position ranging unit 605 are abnormal when explicitly instructed by the remote controller 602 or due to the presence of an obstacle. It becomes effective when judged.

  The moving image signal input to the video input / output terminal 607 is divided for each frame and transmitted to the video control unit 606 in order. The pulse information and the current time are acquired from the built-in computer 610 in synchronization with the input of one frame in the moving image to the video control unit 606.

  Acquisition of pulse information is executed by a heart sound analysis program recorded on the hard disk of the built-in computer 610. The heart sound analysis program uses, as a sound, a component (hereinafter referred to as a pulse signal) relating to the pulse of the observer who has registered a heart sound waveform in advance from the sound signal in the vicinity of the display device installation location that is constantly sent from the high-sensitivity microphone 608. Extracted by recognition processing. The observer's heart waveform can be registered on the adjustment screen of the display device, measured while sitting for 1 minute in front of the display device, and recorded on the hard disk of the built-in computer 610. The extracted pulse signal is buffered on the hard disk of the built-in computer 610 for the past one minute. The pulse information transmitted to the video control unit 606 in synchronization with one frame is a value obtained by examining the difference between the heartbeat periods 1 second and 10 seconds before the synchronization signal from the buffered pulse signal.

  The current time is obtained from the clock function of the built-in computer 610. Since the Internet connection terminal 609 can synchronize with a time server on the Internet, the clock function always returns an accurate time.

  The video control unit 606 performs image conversion proposed in the present invention using image information, pulse information, and current time included in one frame after color conversion processing and gradation conversion processing performed in a normal display device are performed. Color correction processing based on the processing method is executed. The processed image signal is displayed on the display screen 601.

  The flow of image processing executed by the video control unit 606 will be described with reference to FIG. First, the image data 701 input to the video control unit 606 is subjected to image processing in a normal display device such as color temperature adjustment and gamma adjustment in a display preprocessing step 702.

  Next, the aberration estimation step 706 is executed using the pulse information 704 or the time information 705, and the aberration estimated value 707 is estimated. Which information to use between the pulse and the time is instructed in advance using the remote controller 602. Similarly, information regarding the visual acuity of the observer and the presence / absence of glasses / contact lenses is also instructed using the remote controller 602. Such information affects the type of color to be corrected. If the observer's visual acuity is normal, or is farsighted but wearing glasses, it will be omitted because it will be processed on the B surface as in the first embodiment, and the observer is farsighted and not wearing glasses. Only mention.

  The aberration estimation step 706 is the same as that of the first embodiment except for the pupil diameter estimation method and the type of color to be corrected.

  A case where the pupil diameter is estimated using the pulse information 704 will be described. It is generally known that when an observer sees an image that induces excitement, the pupil temporarily opens and then returns to the original as the video gets used. Such a state of excitement is accompanied by a temporary rise in pulse, and can be detected using the pulse information 704.

  The relationship between the pupil diameter assumed in the present invention and the pulse will be described with reference to FIG. As described above, the pulse information 704 is the difference between the heartbeat periods 1 second and 10 seconds ago, but when creating a graph with the horizontal axis as the elapsed time from the start of measurement and the vertical axis as the pulse information 704, As shown in FIG. During the time when the surprise occurs, the value of the pulse fluctuates in the negative direction because the pulse period is rapidly shortened. After that, it swings slightly in the positive direction and then gradually approaches 0. Therefore, the time when the value falls below the threshold 801 can be set as the excited state detection time 802. On the other hand, with respect to the change in pupil diameter, as shown in FIG. 8B, it is assumed that the pupil diameter 803 at the time of release is reached at the excited state detection time 802 and then the normal pupil diameter 805 is returned after the relaxation time 804 has elapsed. .

  The threshold value 801 can be determined by a subjective evaluation experiment, and other parameters can be actually measured. On the program, the relationship between the elapsed time and the pupil diameter as shown in FIG. 8B is held in the LUT, and the pupil diameter is returned according to the LUT after the excited state is detected.

  A case where the pupil diameter is estimated using the time information 705 will be described. This function assumes a situation in which the ambient brightness changes periodically according to the time. In this situation, it is not necessary to measure the ambient light illuminance as in the first embodiment, and the pupil diameter can be estimated even if it is difficult to always measure the ambient light illuminance. Assume that a change in brightness corresponding to the time is recorded in advance and an LUT is created in advance. At the time of estimation, first, a peripheral light amount measurement value is obtained by LUT from the current time. Next, the pupil diameter is calculated by giving the peripheral light amount measurement value, the distance from the observer to the display screen, and the display screen brightness in Equations 1 and 2. However, it is assumed that the distance from the observer to the display screen and the display screen brightness are not measured values but average values.

  When the estimated aberration value 707 is obtained from the pupil diameter, the estimated aberration value is calculated by Equation 3 as in the first embodiment, but the wavelength of the denominator MTF is set to 630 NM at which the spectral luminance of the red pixel is maximized. As described above, in the present embodiment, it is assumed that the observer is viewing the display screen without spectacles in hyperopia. Since the focus is easily achieved on the blue light side where the focal length is short and the red light is likely to be blurred, the correction target color is set to red. If it is desired to enhance the correction effect, both the colors may be corrected simultaneously by changing the ratio of blue and red according to the visual acuity.

  In the next correction amount acquisition step 708, a correction amount 709 is calculated using the estimated aberration value 707. The correction amount 709 is a value obtained by converting the correction amount in the first embodiment given by Equation 4 into a parameter for edge enhancement processing executed in the correction step 710. The conversion function to the edge enhancement processing parameter is created in advance by a subjective evaluation experiment. Specifically, several images that have been subjected to the correction of the first embodiment and some images that have been subjected to edge enhancement processing on the R plane are generated, and a plurality of sets with high similarity are extracted from them. Since some combinations of the correction amount and the edge enhancement processing parameter of the first embodiment can be made, the relationship between both can be converted into a function by function fitting.

  In the correction step 710, correction processing is performed on the pre-display image data 701 using the correction amount 709. The correction process 710 includes R-plane extraction 711, edge enhancement processing 712, and image composition 713. The difference from the first embodiment is that the processing related to the spatial frequency is the edge enhancement processing 712, and therefore only the edge enhancement processing will be described.

  The edge enhancement process 712 is a linear filter process using the filter given by Equation 5.

  The parameter Α is a parameter representing the degree of enhancement, and the correction amount 709 uses the parameter Α. Similar to the high-frequency component enhancement processing performed in the first embodiment, the high-frequency component of the image can be enhanced. Since the calculation load is smaller than the high-frequency component enhancement processing of the first embodiment, the processing can be performed at high speed. The filter given by Equation 5 is a 3 × 3 filter, but a filter of a larger size may be used to enhance the enhancement performance.

  Display image data 714 as a result of the correction process 710 is displayed on the display screen 701 and presented to the observer.

  In this way, the display device according to the second embodiment of the present invention acquires chromatic aberration information generated in the eyes of a hyperopic observer by estimating the pupil diameter from information such as the observer's heartbeat and current time. And a correction function for performing edge enhancement processing on the red component so as to correct the chromatic aberration of the observer.

  A third embodiment of the present invention will be described. In the embodiment described so far, the pupil diameter of the observer is estimated from the peripheral information, but this embodiment is characterized in that the pupil diameter is directly measured. In addition, a small display screen such as a mobile phone screen is assumed, and correction is performed only by tone conversion, not processing with a large calculation load such as frequency enhancement.

  The display device shown in FIG. 9 includes a touch panel 901, a pupil diameter measuring small camera 902, operation keys 903, a calculation / memory module 904, a wireless LAN module 905, and an external storage module 906.

  The flow of processing in the third embodiment will be described. A standby screen is normally displayed on the touch panel 901. In order to display an image, the corresponding item on the display screen 901 is clicked with a finger, or the mouse cursor displayed on the touch panel 901 is moved over the corresponding item with the operation key 903 and clicked. At this time, the location of the image data (external server or external storage module 906) is also specified. After the instruction is completed, image data is acquired and sent to the calculation / memory module 904. The calculation / memory module 904 is a module including a main memory, a cache memory, a small MPU, and a GPU, and image data is read into the main memory.

  In synchronism with the reading of the image data into the main memory, an image near the observer's eyes (hereinafter referred to as an eye measurement image) is acquired by the small pupil diameter measurement camera 902, and the eye is measured by the calculation / memory module 904. An image is sent. The small pupil diameter measurement camera 902 includes two lenses 1001 and 1002 as shown in FIG. With these lenses, one eye of the observer is simultaneously photographed from different angles of view.

  Next, the calculation / memory module 904 calculates the pupil diameter from the eye measurement image, and executes correction. Image processing executed in the calculation / memory module 904 will be described with reference to FIG. First, the pupil diameter information 1103 is acquired from the eye measurement image 1101 held in the main memory of the calculation / memory module 904 by the pupil diameter measurement step 1102.

  The pupil diameter measurement step 1102 includes three stages of processing: pupil position detection 1104, observer distance acquisition 1105, and pupil diameter detection 1106. In the pupil position detection 1104, a circular pattern is searched from two images obtained by the lenses 1001 and 1002 shown in FIG. 10, and the position on the image of the center of the circular pattern is set as the pupil position. Simultaneously with the pupil position, the pupil diameter on the image is also obtained. In the observer distance acquisition 1105, the distance Y from the lens surface to the pupil is calculated from the pupil position in the two images according to Equation 6.

  The relationship between the parameter in Equation 6 and the physical quantity is shown using FIG. D is the distance 1004 between the lens and the sensor surface 1003, X is the distance 1005 between the lenses, and P and Q are the X of the pupil position when the intersections 1006 and 1007 of the optical axes of the lenses 1001 and 1002 and the sensor surface are the origin. The coordinates are 1008 and 1009. However, it is assumed that the X coordinate is parallel to a straight line connecting the intersections 1006 and 1007 between the optical axis and the sensor surface.

  In the pupil diameter detection 1106, the pupil diameter H is calculated according to Equation 7 from the pupil diameter H ′ on the image obtained by the pupil position detection 1104 and the distance Y from the lens surface to the pupil obtained by the observer distance acquisition 1105. To do.

  Here, D is a distance 1004 between the lens and the sensor surface 1003, and U is a distance between pixels on the sensor surface 1003.

  For convenience of explanation, the observer distance acquisition 1105 and the pupil diameter detection 1106 are divided and calculated, but it is possible to calculate them at the same time. Further, since the state in which the observer sees the screen is almost determined, it is desirable to create a table instead of calculating with a function when programming.

  In the aberration estimation step 1107, an estimated chromatic aberration value 1108 for the pupil diameter information 1103 is obtained from the LUT. Although the estimated chromatic aberration value in the other embodiments is the ratio of MTF, in this embodiment, a different method is adopted in order to avoid processing with a large calculation load. As a secondary effect of chromatic aberration, it is known that the brightness of the peak value of point images of different colors decreases. The aberration estimation value in this embodiment is the ratio of the brightness sensation amount given by Equation 8, and the LUT uses a table that shows the correspondence between the pupil diameter and the ratio of the brightness sensation amount.

SEN _{T, R} is a sensory amount related to the brightness of the color type T when the pupil diameter is R. The amount of sensation is obtained by subjective evaluation. Specifically, the subject's pupil diameter is changed by changing the brightness of the room's illumination, and the subject's pupil diameter is stable. It can be obtained by an evaluation experiment.

  The processing after the correction amount acquisition step 1109 is the same as that of the first embodiment if the processing related to spatial frequency is replaced with the gradation conversion 1114. The gradation conversion 1114 is a process of uniformly integrating the correction amount 1110 for all the pixels on the B surface.

  The display image data 1118 is displayed on the touch panel 901 and presented to the observer.

  As described above, the third embodiment of the present invention obtains chromatic aberration information generated in the eyes of the observer by measuring the pupil diameter with a camera, and corrects the chromatic aberration of the observer so as to correct the chromatic aberration of the observer. A display device having a color correction function for adjusting a tone.

  A fourth embodiment of the present invention will be described. The present embodiment is characterized by handling not a chromatic aberration which is a difference between light of each color but a general aberration influenced by the pupil diameter. As such aberrations, spherical aberrations, higher order aberrations and the like are known, which all cause blurring of the optical image.

  The configuration of the display device and the flow of processing in this embodiment are almost the same as those in the first embodiment, and a description thereof will be omitted. Since part of the flow of processing in the video control unit is different, a description will be given with reference to FIG.

  In the aberration estimation step 1206, the process is the same as that in the first embodiment until the pupil diameter is obtained. When obtaining the estimated aberration value 1207 after obtaining the pupil diameter, Equation 9 is used instead of Equation 3.

Here, MTFΛ, R (Ν) is the MTF of the eye that occurs when the wavelength of the incident light is Λ and the pupil diameter is R. P is a value selected in advance for the pupil diameter when no blur occurs. Q is an estimated pupil diameter. The wavelength is 550 NM for both MTFs. It is known that the light intensity at this wavelength is close to the sense amount of brightness. By the function given by Equation 9, a ratio for correcting the aberration generated in light having a wavelength of 550 NM can be obtained.

  The correction amount 1209 is obtained by substituting the function given by Equation 9 into Equation 4.

  In the correction step 1210, spatial frequency separation 1211, high frequency component amplification 1212, and image composition 1213 are performed on all RGB planes. These processes are the same as those in the first embodiment.

  In the present embodiment, since correction is performed based on Equation 9, the aberration is not accurately corrected for light having a wavelength other than 550 NM. Based on the same equation as Equation 9, for example, the correction amount of 470 NM, which is a typical wavelength of blue light, and 630 NM of red light, is calculated, and each is applied to the B surface and the R surface, so that aberration can be more accurately Can be corrected.

  As described above, in the fourth embodiment of the present invention, by estimating the pupil diameter from information such as the illuminance of ambient light, aberration information generated in the observer's eyes is acquired, and the observer's aberration is corrected. As described above, the display device includes a correction function that emphasizes high-frequency components.

DESCRIPTION OF SYMBOLS 101 Display screen 102 Remote controller 103 Control signal light-receiving part 104 Peripheral light quantity measuring part 105 Observer position ranging part 106 Image | video control part 107 Image | video input / output terminal 201 Image data 202 Average on-screen brightness | luminance estimation process 203 Display screen brightness | luminance 204 Measured value 205 Observer distance 206 Aberration estimation step 207 Aberration estimation value 208 Correction amount acquisition step 209 Correction amount 210 Correction step 211 B surface extraction 212 Spatial frequency separation 213 High frequency component amplification 214 Image composition 215 Corrected image data 216 Linearization step 217 Display Preprocessing Step 218 Display Image Data 301 Display Screen 302 Observer Eye 303 Crystal Lens 304 Retina 305 Viewing Angle 306 Display Screen Area 307 Distance from Viewer to Display Screen 308 Virtual Diffuse Reflection Surface 309 Illumination 50 Green MTF
502 Blue MTF
503 Blue MTF after amplification
504 Cut-off frequency 601 Display screen 602 Remote controller 603 Control signal light receiving unit 604 Peripheral light measurement unit 605 Observer position ranging unit 606 Video control unit 607 Video input / output terminal 608 Heart sound measurement unit 609 Internet connection terminal 610 Built-in computer 701 Image data 702 Pre-display processing step 703 Post-display pre-processing image data 704 Pulse information 705 Time information 706 Aberration estimation step 707 Aberration estimation value 708 Correction amount acquisition step 709 Correction amount 710 Correction step 711 R surface extraction 712 Edge enhancement 713 Image composition 714 For display Image data 801 Threshold 802 Excited state detection time 803 Release pupil diameter 804 Relaxation time 805 Normal pupil diameter 901 Display screen 902 Pupil diameter measurement small camera 903 Operation key 904 Calculation / memo Re-module 905 Wireless LAN module 906 External storage module 907 Button battery 908 Appearance of device 909 Internal module layout 1001, 1002 Lens 1003 Sensor surface 1004 Distance between lens and sensor surface 1005 Distance between lenses 1006, 1007 Intersection of optical axis and sensor surface 1008 , 1009 X coordinate of pupil position 1010 Eye of observer 1011 Distance from lens surface to pupil 1101 Eye measurement image 1102 Pupil diameter measurement process 1103 Pupil diameter information 1104 Pupil position detection 1105 Observer distance acquisition 1106 Pupil diameter detection 1107 Aberration estimation process 1108 Estimated aberration value 1109 Correction amount acquisition step 1110 Correction amount 1111 Image data 1112 Color correction step 1113 B-side extraction 1114 Tone conversion 1115 Image composition 1116 Corrected image data 1117 Display preprocessing step 1118 Display image data 1201 Image data 1202 Display screen average luminance estimation step 1203 Display screen luminance 1204 Peripheral light quantity measurement value 1205 Observer distance 1206 Aberration estimation step 1207 Aberration estimation value 1208 Correction amount acquisition step 1209 Correction amount 1210 Correction step 1211 Spatial frequency separation 1212 High frequency component amplification 1213 Image composition 1214 Image data after correction 1215 Linearization step 1216 Pre-display processing step 1217 Display image data

Claims (11)

In an image processing method for performing correction processing on input image information,
An aberration estimation step for estimating aberrations occurring in the eyes of the observer of the display image displayed in the image display device;
A correction amount acquisition step of converting the aberration estimation amount obtained by the aberration estimation step into a correction amount of the input image;
A correction step of performing a correction process on the image information according to the correction amount obtained by the correction amount acquisition step;
An image processing method comprising:   The image processing method according to claim 1, wherein in the aberration estimation step, pupil diameter information of the observer is acquired.   The image processing method according to claim 1, wherein in the aberration estimation step, pupil diameter information of the observer is estimated.   The estimation of the pupil diameter information is performed by calculating the illuminance information of the observer's ambient light, the observer's heartbeat information, the distance information from the image display device to the observer, and the average luminance information of the display screen displaying the display image. 4. The image processing method according to claim 3, wherein the image processing method is performed using any one or more of time information for displaying the display image.   The correction amount obtained by the correction amount acquisition step relates to any one or more of spatial frequency component processing, edge enhancement processing, and gradation processing, according to any one of claims 1 to 4. The image processing method as described. In an image processing apparatus that performs correction processing on input image information,
An aberration estimating means for estimating an aberration occurring in an eye of an observer of a display image displayed in the image display device;
A correction amount acquisition unit that converts the aberration estimation amount obtained by the aberration estimation unit into a correction amount of an input image;
Correction means for performing correction processing on the image information in accordance with the correction amount obtained by the correction amount acquisition means;
An image processing apparatus comprising:   The image processing apparatus according to claim 6, wherein the aberration estimation unit acquires pupil diameter information of the observer.   The image processing apparatus according to claim 6, wherein the aberration estimation unit estimates pupil diameter information of the observer.   The estimation of the pupil diameter information is performed by calculating the illuminance information of the observer's ambient light, the observer's heartbeat information, the distance information from the image display device to the observer, and the average luminance information of the display screen displaying the display image. The image processing apparatus according to claim 8, wherein the image processing apparatus is performed using any one or more of time information for displaying the display image.   The correction amount obtained by the correction amount acquisition unit relates to one or more of spatial frequency component processing, edge enhancement processing, and gradation processing, according to any one of claims 6 to 9. The image processing apparatus described.   An image information correction processing program that causes a computer to execute the image processing method according to any one of claims 1 to 5.