JP2011139158A - Image processor and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To suppress generation of a false color even when a white thin line is contained in an image when making aberration correction of the displayed image. <P>SOLUTION: An image processor is provided with: a means (702) to discriminate whether or not interest pixels at the time of processing the image are white thin line pixels composing high-luminance thin lines included in the image; a means (702) to perform smoothing processing for the interest pixels when they are discriminated to be the white thin line pixels; a means (706) to convert the positions of the interest pixels subjected to smoothing processing based on information which shows a shift amount of the focusing position resulting from color aberration; and a means (707) to perform interpolation processing of the converted interest pixels, and to reconstruct color information in the predetermined display position. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing technique for correcting chromatic aberration generated in a display optical system.

  Conventionally, a mixed reality, so-called MR (Mixed Reality) technique is known as a technique for seamlessly combining the real world and the virtual world in real time. Furthermore, as one of the MR techniques, a technique using a video see-through HMD (Head Mounted Display, hereinafter referred to as HMD) is known. In the case of the HMD, a subject that substantially matches the subject observed from the pupil position of the wearer is captured by a video camera or the like, and CG (Computer Graphics) is superimposed and displayed on the captured image. Thereby, the HMD wearer can observe the mixed reality image.

  As described above, in the HMD, a subject is imaged by a charge coupled device such as a CCD to obtain digital image data of the subject, and an MR image (mixed reality image) on which a CG image is superimposed is mounted via a display device such as a liquid crystal. Display to the person. For this reason, it is important to correct aberrations that occur in an optical system for imaging and display (imaging optical system and display optical system). Due to distortion of the lens, the displayed image will be a barrel-shaped image or a pincushion-type image, or due to the chromatic aberration of magnification of the lens, red, blue or green color blur will occur at the boundary of the subject. This is because the quality of the image may be affected.

  In general, correction for aberrations occurring in an imaging optical system and a display optical system includes an optical correction approach and an electronic correction approach using signal processing. Of these, devices that are worn on the head, such as HMDs, are indispensable to reduce size and weight. Therefore, electronic devices that use signal processing rather than optical correction approaches that inevitably increase in size and weight. The corrective approach is selected. This is because when an approach based on electronic correction is selected, there is an advantage that a cost reduction effect can be obtained by adopting an inexpensive lens or reducing the number of lenses.

  In electronic correction using such signal processing, aberration correction is usually realized by two processes. The first process is to convert each pixel of the distorted image to the ideal position based on the correspondence between the image position obtained by the ideal optical system and the image position affected by the actual aberration. This is a coordinate conversion process for converting coordinates. The second process is an interpolation process in which the color information of each pixel is reconstructed on the lattice based on each pixel subjected to coordinate conversion.

  Among these, in the coordinate conversion process, by performing coordinate conversion processing for each pixel, it is possible to correct distortion, which is image distortion, and by performing coordinate conversion processing for each color constituting the pixel, A certain chromatic aberration of magnification is corrected.

  Here, each pixel whose aberration has been corrected by the coordinate conversion process must finally exist on a specified lattice point regardless of whether it is an imaging system or a display system. For this reason, in the pixel interpolation process, when a pixel that is not on the grid point is generated by the coordinate conversion process, the color information of the pixel is reconstructed on the grid point. Specifically, using linear interpolation and high-order bicubic interpolation (bicubic) used in general resolution conversion methods, weighting is performed according to the positional relationship (distance) between the pixel and neighboring grid points, Color information on each grid point is calculated.

  Various proposals have been made for aberration correction using such signal processing. For example, Patent Document 1 below proposes a configuration that realizes aberration correction with higher image quality by using cubic convolution (BiCubic) for the pixel interpolation process.

JP 2005-11269 A JP-A-5-328106 JP 2003-102025 A

  However, conventional aberration correction using signal processing has the following problems. That is, in the aberration correction of the display optical system, when the original image includes a high-luminance white thin line, a false color is generated by executing the interpolation process. The false color is not conspicuous in a natural image, but has a characteristic that the generation of the false color becomes remarkable in an image artificially created by a computer graphics (CG) such as a wire frame or an OSD (On Screen Display) function. . In addition, since a pixel at a position where no pixel originally exists is reconstructed according to the positional relationship with the grid point, false color occurs regardless of the amount of aberration correction, and the periphery of the image It occurs not only at the center but also at the center. For this reason, it can be said that suppressing the generation of false colors in aberration correction is a particularly important issue in display in HMD or the like.

  As a technique for suppressing the occurrence of false colors, for example, the above-mentioned Patent Document 2 and Patent Document 3 disclose a configuration that suppresses the generation of false colors generated in moire removal or an imaging system using a low-pass filter. Has been. However, since both of the configurations disclosed in Patent Documents 2 and 3 are techniques for processing for the purpose of preserving the edge portion, when applied to the interpolation process, the false color generated in the white thin line is changed. The result will be encouraging.

  The present invention has been made in view of the above problems, and an object of the present invention is to suppress the occurrence of false colors even when a white thin line is included in the image when the displayed image is corrected for aberration. To do.

In order to achieve the above object, an image processing apparatus according to the present invention comprises the following arrangement. That is,
An image processing apparatus for processing an image to be displayed,
Determining means for determining whether or not the pixel of interest when processing the image is a white thin line pixel that is included in the image and that forms a white thin line that is higher than a predetermined luminance value and thinner than a predetermined width;
When the determination means determines that the pixel is a white thin line pixel, it operates to perform a smoothing process on the pixel of interest, and when the determination unit determines that the pixel is not a white thin line pixel, A calculation means for calculating color information of the target pixel by operating so as not to perform the smoothing process on the pixel;
Conversion means for converting the position of the pixel of interest for which color information has been calculated by the calculation means, based on information indicating a shift amount of an imaging position caused by chromatic aberration occurring in a display optical system for displaying the image; ,
The color information of the target pixel converted by the conversion unit is reproduced by the color information of the pixel that is in the vicinity of the target pixel and that corresponds to the display position when displaying an image in the display optical system. And interpolation means for performing interpolation processing of the target pixel.

  According to the present invention, in correcting aberrations in an image to be displayed, generation of false colors can be suppressed even when white thin lines are included in the image.

It is a figure which shows the system configuration | structure of MR system with which the image processing method is performed by 1st Embodiment. It is a figure which shows the function structure of MR system. It is a figure which shows the hardware constitutions of the image composition apparatus which comprises MR system. It is a figure which shows the hardware constitutions of HMD which comprises MR system. It is a figure explaining a distortion aberration and magnification chromatic aberration. It is a figure which shows the example of 1 structure of a display optical system. It is a figure which shows the function structure of the display system aberration correction part of HMD. It is a figure which shows the flow of the aberration correction process in a display system aberration correction part. It is a functional block diagram of the adaptive smoothing process part which comprises a display system aberration correction part. It is a figure which shows the various matrices used in a display system aberration correction part. A flow of white thin line detection processing in adaptive smoothing processing is shown. It is a figure which shows the flow of the smoothing process in an adaptive smoothing process. It is a figure which shows an example of the aberration correction table used when performing a coordinate transformation process in a display system aberration correction part. It is a figure which shows the flow of a coordinate transformation process. It is a schematic diagram which shows the concept of the cubic interpolation which is an example of a coordinate transformation process. It is a figure which shows the flow of the interpolation process in a display system aberration correction part. It is a schematic diagram for explaining the effect of false color suppression by the false color generation mechanism and the aberration correction processing in the display system aberration correction unit. It is a schematic diagram for explaining the effect of false color suppression by the false color generation mechanism and the aberration correction processing in the display system aberration correction unit. It is a figure which shows the flow of the white thin line detection process in the adaptive smoothing process by the display system aberration correction part by which the image processing method which concerns on 2nd Embodiment is performed. It is a figure which shows the flow of the smoothing process in an adaptive smoothing process. It is a figure which shows the function structure of the display system aberration correction part by which the image processing method which concerns on 3rd Embodiment is performed. It is a figure which shows the flow of the aberration correction process in a display system aberration correction part. It is a figure which shows the example of the interpolation curve used in the adaptive interpolation process by a display system aberration correction part. It is a figure which shows the flow of the adaptive interpolation process in a display system aberration correction part. It is a figure which shows the function structure of the display system aberration correction part by which the image processing method which concerns on 4th Embodiment is performed. It is a figure which shows the flow of the aberration correction process in a display system aberration correction part. It is a figure which shows the example of the interpolation curve used in the adaptive interpolation process by a display system aberration correction part. It is a figure which shows the flow of the adaptive interpolation process in a display system aberration correction part.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
<1. System configuration of MR system>
FIG. 1 is a diagram showing a system configuration of an MR system 100 in which the image processing method according to the first embodiment of the present invention is executed.

  In mixed reality, which is a technology that seamlessly fuses the real world and the virtual world in real time, so-called MR technology, a display device with an imaging function is used. Hereinafter, the display device with an imaging function is abbreviated as HMD. However, the display device with an imaging function may be a hand-held type such as binoculars, and is not limited to a head-mounted type.

  In the HMD, a CG image generated based on three-dimensional position and orientation information such as the position and direction of the wearer is superimposed on the captured image of the real space obtained from the wearer's viewpoint acquired by the imaging unit, and displayed on the display unit. It is configured. With this configuration, the HMD wearer can experience a mixed reality feeling as if an object drawn with CG exists in the observed real space.

  As shown in FIG. 1, the MR system 100 using the HMD can generally be configured by a controller 102 and an image composition device 103 in addition to the HMD 101.

  Among these, the HMD 101 includes an imaging unit for acquiring an image of the real space observed by the wearer as described above. In addition, a captured image in the real space captured by the imaging unit, an output image from the image composition device 103, or a composite image in which a CG image generated by the image composition device 103 is superimposed on the captured image in the real space is displayed as a display image. A display unit for providing to the wearer; Further, the HMD 101 has a function of communicating with the controller 102. The HMD 101 may be configured to be driven by receiving power supply from the controller 102 or may be configured to be driven by a battery.

  On the other hand, the image composition device 103 connected to the controller 102 includes a CG rendering unit that generates a CG image, and an image composition unit that synthesizes the captured image of the real space and the CG image.

  The image composition device 103 is configured to communicate with the HMD 101 via the controller 102, and the composite image synthesized by the image composition unit is transmitted to the HMD 101 via the controller 102 and displayed on the display unit.

  On the other hand, the controller 102 (image processing apparatus) includes various functions including a function of converting the resolution of the image and a function of converting the color space, and an aberration correction processing function of the display optical system according to the image processing method according to the present embodiment. A processing function and a transmission format conversion function are provided.

  In the MR system of FIG. 1, the image composition device 103 and the controller 102 are configured as separate hardware, but the present invention is not limited to such a configuration. For example, the function possessed by the controller 102 and the function possessed by the image composition device 103 may be integrated to constitute a dedicated image composition device.

  In the MR system of FIG. 1, each device is connected by wire, but the present invention is not limited to this, and a part or all of them may be connected wirelessly.

  Or you may comprise so that a part or all of the function of the controller 102 may be taken in into the HMD101 side. In the following MR system, the combination of the functions of the controller 102 and the HMD 101 will be described as the HMD 101 (image processing apparatus).

<2. MR System Functional Configuration>
FIG. 2 is a diagram illustrating a functional configuration of the MR system 200 in which the image processing method according to the first embodiment is executed.

  In FIG. 2, reference numeral 201 denotes a video see-through type HMD. The HMD 201 includes an imaging unit 203 that images a real space, and a display unit 208 that displays an MR image on which a CG image is superimposed. In addition, an I / F 206 for performing communication with an image composition device 211 that performs generation and composition of CG images, a three-dimensional position and orientation sensor 205 that outputs position and orientation information of the HMD 201, and an imaging system aberration correction unit 204 And a display system aberration correction unit 207. Hereinafter, each part of the HMD 201 will be described.

  The imaging unit 203 images a real space that is substantially coincident with the line-of-sight position of the HMD wearer. The imaging unit 203 includes two sets of right-eye and left-eye imaging devices capable of generating a stereo image, an imaging optical system, and a signal processing circuit.

  The imaging system aberration correction unit 204 corrects aberrations that occur in the imaging optical system. The content of the aberration correction process is basically the same for both the imaging system and the display system.

  The three-dimensional position / orientation sensor 205 is a sensor for obtaining the position / orientation information of the HMD wearer, and generates information used by a position / orientation information generation unit 213 described later. As the three-dimensional position and orientation sensor 205, for example, a magnetic sensor or a gyro sensor (acceleration, angular velocity) is used. Note that the three-dimensional position / orientation sensor 205 is not necessarily an indispensable device in the case of an MR system that acquires position / orientation information only from a captured image.

  Reference numeral 206 denotes an I / F for transmitting a captured image captured by the imaging unit 203 to the image synthesis apparatus 211 and transmitting a synthesized MR image to the HMD 201. The I / F 206 functions as an interface when performing data communication between the HMD 201 and the image composition device 211. The same applies to the I / F 212 provided on the image composition device 211 side. However, since both of the I / Fs 206 and 212 are required to have real-time properties, it is desirable to adopt a communication standard that enables large-capacity transmission. In the case of a wired system, it is desirable to use an optical fiber such as a USB or IEEE 1394 metal wire or Gigabit Ethernet (registered trademark). In the case of a wireless system, it is desirable to use high-speed wireless communication conforming to IEEE 802.11 wireless LAN, IEEE 802.15 wireless PAN standard, or the like. In the present embodiment, an optical fiber is used for a wired system, and an UWB (Ultra Wide Band) is used for a wireless system. The transmission band of the optical fiber is several Gbps, and UWB is several hundred Mbps.

  The display system aberration correction unit 207 corrects aberrations that occur in the display optical system. Details of the display system aberration correction unit 207 will be described later.

  The display unit 208 displays the synthesized MR image. Similar to the imaging unit 203, it is composed of two sets of display devices for right eye and left eye and a display optical system. As a display device, for example, a small liquid crystal display or a retinal scan type device using MEMS (Micro Electro Mechanical System) can be used. The configuration of the display optical system will be described later.

  On the other hand, the image composition device 211 generates position / orientation information based on the captured image received from the HMD 201, or generates a CG image based on the position / orientation information, and performs synthesis with the captured image. The image composition device 211 is configured by a device having a high-performance arithmetic processing function or a graphic display function such as a personal computer or a workstation. Hereinafter, each unit of the image composition device 211 will be described.

  Reference numeral 212 denotes an I / F on the image processing apparatus side. Since the function of the I / F 212 is the same as that of the I / F 206 in the HMD 201, the description thereof is omitted here.

  A position and orientation information generation unit 213 generates position and orientation information indicating the position and orientation of the HMD wearer from the received captured image. Position / orientation information is generated by extracting a marker or a feature point instead of the marker from the captured image. Note that the position / orientation information generation unit 213 is configured to supplementarily use a captured image from an objective viewpoint (not shown) and information from the three-dimensional position / orientation sensor 205 attached to the HMD 201, and a marker is included in the captured image. It is possible to cope with the case where there is no feature point.

  Reference numeral 214 denotes a CG image rendering unit that forms a virtual space in which each virtual object is arranged in the virtual space, using data related to each virtual object stored in the content DB 215. The CG image drawing unit 214 generates a virtual image (CG image) when viewed from the viewpoint of the HMD wearer in the formed virtual space. In addition, since the process which produces | generates the image of the virtual space at the time of seeing from the viewpoint of the HMD wearing person of a predetermined position and orientation is known, detailed description is abbreviate | omitted here.

  A content DB 215 stores virtual image content, and is a DB (database) that holds data related to each virtual object constituting the virtual space. The data related to the virtual object includes, for example, data indicating the arrangement / position / attitude of the virtual object and its operation rule. If the virtual object is composed of polygons, normal vector data of each polygon, its color data, position data of each vertex constituting the polygon, and the like are included. In addition, when texture mapping is performed on a virtual object, texture data is also included.

  Reference numeral 216 denotes an MR image synthesis unit that synthesizes a captured image and a CG image to generate an MR image. The MR image generated here is sent to the HMD 201 via the I / F 212 and displayed on the display unit 208 of the HMD 201.

  Under the functional configuration as described above, the captured image captured by the imaging unit 203 is processed according to the following flow. That is, the imaging system aberration correction unit 204 corrects the aberration of the imaging optical system and transmits the corrected optical system to the image processing apparatus 202. Then, the image processing apparatus 202 calculates the position and orientation information of the HMD 201 based on the marker and the feature amount in the received captured image. Thereafter, a CG image generated based on the calculated position and orientation information is superimposed on the captured image, and an MR image that is a composite image is generated. The generated MR image is sent to the HMD 201 via the I / F 212.

  In the HMD 201, similarly to the imaging optical system, the display system aberration correction unit 207 corrects the aberration of the display optical system and then displays it on the display unit 208. Through such a processing flow, the MR image whose aberration has been corrected is guided to the pupil of the HMD wearer.

<3. Hardware configuration of image synthesizer>
Next, the hardware configuration of the image composition device 211 will be described. FIG. 3 is a block diagram illustrating a hardware configuration of the image composition device 211. In FIG. 3, reference numeral 301 denotes a CPU, which executes processing based on programs and data stored in a RAM 302 and a ROM 303, thereby controlling the entire computer and realizing various functions of the image composition apparatus 211. .

  The RAM 302 provides an area for temporarily storing programs and data loaded from the external storage device 306, data received from the outside (the HMD 201 in this embodiment) via the I / F 307, and the like. A work area used when the CPU 301 executes various processes is also provided. Note that the RAM 302 can provide these areas as appropriate. The ROM 303 stores computer setting data, a boot program (BIOS), and the like.

  An operation unit 304 includes a keyboard and a mouse. A user of the computer inputs various instructions to the CPU 301 by operating the operation unit 304. A display unit 305 includes a CRT, a liquid crystal screen, and the like. A processing result by the CPU 301 or a graphics board (not shown) is displayed on the display unit 305 as an image or a character.

  Reference numeral 306 denotes an external storage device, which includes a large-capacity information storage device represented by a hard disk drive device. The external storage device 306 stores an OS (operating system) and programs and data for causing the CPU 301 to execute various processes. These programs and data are appropriately loaded into the RAM 302 under the control of the CPU 301 and executed by the CPU 301.

  Reference numeral 307 denotes an I / F, which corresponds to the I / F 212 shown in FIG. The I / F 307 functions as an interface for performing data communication with the HMD 201. A bus 308 connects the above-described units.

<4. HMD hardware configuration>
Next, the hardware configuration of the HMD 201 will be described. FIG. 4 is a block diagram showing a hardware configuration of the HMD 201. As shown in FIG. In FIG. 4, 401 is an imaging unit, which corresponds to the imaging unit 203 of FIG. Reference numeral 402 denotes a display unit, which corresponds to the display unit 208 in FIG. Reference numeral 403 denotes a RAM, which provides a work area used by the CPU 406 for performing various processes and an area for temporarily storing data received from the outside (here, the image processing apparatus 202) via the I / F 206. .

  Reference numeral 404 denotes a ROM which stores programs and data for causing the CPU 406 to execute various processes to be described later performed by the HMD 201. Reference numeral 405 denotes a three-dimensional position / orientation sensor, which corresponds to the three-dimensional position / orientation sensor 205 shown in FIG. Reference numeral 406 denotes a CPU which executes a program for performing initial settings of the HMD 201 and controlling various devices. Reference numeral 407 denotes an I / F, which corresponds to the I / F 206 in FIG.

  Reference numeral 408 denotes an aberration correction LSI, which corresponds to the imaging system aberration correction unit 204 and the display system aberration correction unit 207 of FIG. Although an ASIC that is a dedicated integrated circuit is assumed here, it may be realized by describing functions in a software manner using a DSP that is a signal processor. Reference numeral 409 denotes a bus connecting the above-described units.

<5. Explanation of distortion and lateral chromatic aberration>
Next, distortion and lateral chromatic aberration will be described. FIG. 5 is a diagram for explaining distortion and lateral chromatic aberration. 5A shows a state without distortion, FIG. 5B shows a state in which distortion has occurred, and FIG. 5C shows a state in which lateral chromatic aberration has occurred in addition to the distortion.

  In FIG. 5C, among the three primary colors of RGB, Green is represented by a solid line, Red is represented by a broken line, and Blue is represented by a one-dot chain line. As described above, the aberration occurs for each color because the refractive index of the lens differs depending on each wavelength of Red, Green, and Blue. In the imaging optical system, a phenomenon occurs in which the red image is connected to the outside and the blue image is connected to the inside with respect to the green image, and even in the case of a black-and-white subject, color blur (color shift) occurs at the edge of the image. Arise. In the case of a color image subject, a similar color blur occurs at an edge portion where the color changes such as a boundary region.

  In imaging with a real lens, when a figure as shown in FIG. 5A is captured, the image is distorted as shown in FIG. 5C, and the imaging position (magnification) varies depending on the color. . In general, image distortion in a single color is called “distortion aberration”, and a magnification difference due to a color difference is called “magnification chromatic aberration”.

  The refraction of light becomes more conspicuous as the wavelength is shorter, and in the imaging optical system that images a convex lens, the red color shifts outward, but in the display optical system that is an enlargement optical system, the blue color shifts outward. In the electronic aberration correction processing, correction is performed so as to shift in the direction opposite to the direction of deviation. For example, in the case of correcting aberrations of the display optical system, it is preferable that an image is formed so that blue is arranged on the inner side and red is arranged on the outer side, so that the aberration of the display optical system is offset and color misregistration of each color does not occur. An image will be provided at the pupil position.

<6. Explanation of optical system of display unit>
Next, the optical system of the display unit 402 of the HMD 201 will be described. FIG. 6 is a diagram showing an example of the display optical system. In FIG. 6, reference numeral 601 denotes a display panel. The display panel 601 is composed of a TFT liquid crystal or an organic EL device. In the case of the TFT liquid crystal, light using a backlight (not shown) as a light source is irradiated through a filter of each color. Become. On the other hand, in the case of an organic EL device, since it is self-luminous, a backlight is not necessary. A color image presented to the HMD wearer is formed on the display panel 601.

  Reference numeral 602 denotes a free-form surface prism for enlarging the light from the small display panel 601 and guiding it to the pupil. In the case of a free-form surface prism, the display panel 601 can be made thinner and smaller than when a simple lens is used.

  Reference numeral 603 denotes an image formation point of an image formed on the display panel. By placing the pupil at this imaging position, the HMD wearer can observe the image on the display panel 601 as an enlarged display image on a large screen.

  In general, various aberrations generated in the display optical system can be suppressed by a plurality of lens groups, but in order to realize a small HMD, it is indispensable to simplify the display optical system and to reduce the size and weight. Therefore, as described above, in the HMD 201, various aberrations are corrected electronically.

<7. Functional configuration of display system aberration correction unit>
Next, a functional configuration of the display system aberration correction unit 207 of the HMD 201 will be described. FIG. 7 is a block diagram illustrating a functional configuration of the display system aberration correction unit 207. In FIG. 7, reference numeral 701 denotes a buffer for storing image data composed of pixels having RGB color components. Reference numeral 702 denotes an adaptive smoothing processing unit that reads an image stored in the buffer 701, detects a pixel to be smoothed, and performs smoothing processing on the detected pixel. Details of the adaptive smoothing processing unit 702 will be described later.

  A color separation unit 703 separates the image output from the adaptive smoothing processing unit 702 into RGB color components. Reference numeral 704 denotes an aberration correction table, which stores a correction value for calculating a coordinate value after conversion for a “target pixel” (pixel to be processed). The aberration correction table 704 stores coordinate values after conversion of a specific color and difference values with respect to other colors other than the specific color when the specific color is used as a reference for capacity reduction. To do. Details of the table configuration and stored values of the aberration correction table 704 will be described later.

  A correction value selection unit 705 selects a correction value for each color, and reads a correction value for calculating conversion coordinates corresponding to the coordinate value of the target pixel from the aberration correction table 704.

  Reference numeral 706 denotes a coordinate calculation unit that calculates coordinate values after conversion of each color component of the target pixel based on the correction value selected by the correction value selection unit 705. The coordinate calculation unit 706 calculates the converted coordinate value by interpolation calculation using various interpolation equations. Details of processing in the coordinate calculation unit 706 will be described later.

  Reference numeral 707 denotes an interpolation processing unit that performs an interpolation process for reconstructing the color information of the pixel of interest after conversion into an interpolation position near the pixel of interest. Here, the luminance value at the interpolation position is calculated for each color of Green, Red, and Blue. The “interpolation position” here refers to the position of a display pixel (on a grid point) on the display panel. When converting the image size of the input image and the output image, that is, resolution conversion, the interpolation processing unit 707 performs the processing together. Details of the processing in the interpolation processing unit 707 will be described later.

  Reference numeral 708 denotes an adaptive sharpening processing unit that sharpens white thin lines dulled by the smoothing processing in the adaptive smoothing processing unit 702 for each color image after interpolation processing. The adaptive sharpening processing unit 708 determines whether or not the pixel is a smoothing target pixel, and performs sharpening processing on the pixel that is determined to be the smoothing target pixel. The outline of the sharpening process in the adaptive sharpening processing unit 708 will be described later.

  Reference numeral 709 denotes a color combining unit that combines the luminance values of the respective color components in the display pixel based on the new luminance values obtained for the respective color components. For example, when there is 8-bit input data for each color, the color combination unit 709 outputs pixel data of a total of 24 bits for each 8-bit input by combining the converted pixels.

  Although not shown in FIG. 7, the display system aberration correction unit 207 may include a preprocessing unit that converts input image data into an image format that can be subjected to aberration correction processing. As the image format conversion, for example, when the pixel information of each pixel of the input image is composed of a coordinate value, a luminance value, and a color difference value, the pixel information is converted into the coordinate value, the luminance value, and each color component ( For example, a process of reconfiguring each color of RGB) is performed. Further, a post-processing unit that performs image color correction, brightness value adjustment, and subsequent conversion to an image format may be added to the display system aberration correction unit 207 as necessary.

  Based on the functional configuration described above, the display system aberration correction unit 207 performs smoothing processing on the white thin lines detected by the adaptive smoothing processing unit 702 on each input RGB plane image. Then, correction for canceling out aberrations of the display optical system is performed by performing coordinate conversion processing and interpolation processing on the smoothed image. Furthermore, sharpening processing for realizing sharpening of the edge is performed on the aberration-corrected image as necessary, so that the edge of the white fine line is eliminated. Further, the respective color image planes subjected to the sharpening process are combined and displayed as a color image.

  The display system aberration correction unit 207 performs processing according to such a flow, so that it is possible to display an image in which the false color generated in the white thin line is suppressed. In realizing false color suppression, the sharpening process by the adaptive sharpening processing unit 708 is not an essential process, but a process aimed at providing a more crisp and preferable image.

<8. Flow of aberration correction processing by display system aberration correction unit>
Next, the flow of aberration correction processing in the display system aberration correction unit 207 will be described. FIG. 8 is a flowchart showing the flow of aberration correction processing in the display system aberration correction unit 207. In step S801, it is determined whether or not pre-processing is to be performed for aberration correction. The preprocessing here is, for example, conversion processing to an image format to which chromatic aberration correction can be applied. If pre-processing is to be performed, the process proceeds to step S802. If not, the process proceeds to step S803. In step S802, as preprocessing, conversion processing to an image format to which chromatic aberration correction can be applied is performed.

  In step S803, a white thin line to be smoothed is detected, and a smoothing process is performed on the detected white thin line. The details of the adaptive smoothing process will be described later using another flow.

  In step S804, the color information of each pixel composed of the three primary colors RGB is separated for each color plane (color component) in the smoothed image. Here, the separation is performed based on the three primary colors of RGB. However, when the aberration correction table itself is composed of correction values indicating the amount of misregistration of other color components, the separation is performed using colors other than RGB. (For example, complementary colors such as CMYK).

  In step S805, the converted coordinate value indicates where the image formation point is shifted for each color of each pixel, or where the pixel should be arranged to lead to an ideal image formation point. Is calculated. Details of the coordinate conversion process will be described later using another flow.

  In step S806, the luminance value of each color component is reconstructed at the new configuration position (“interpolation position”) based on the pixel information of the pixel of interest and the pixel information of the reference pixel obtained in step S805. Perform interpolation processing. It should be noted that the “reference pixel” is a pixel in which the coordinate-converted target pixel is not located at the display position (on the grid point) on the display panel (that is, “interpolation pixel” that requires interpolation processing) It is assumed that the pixel is in the vicinity of the interpolation pixel and is in the display position on the display panel. That is, the luminance value of each color component of the interpolation pixel is reconstructed at the position of the reference pixel (at the interpolation position). Details of the interpolation processing will be described later using another flow.

  In step S807, sharpening processing is performed on the pixels that have been subjected to smoothing, in order to sharpen white thin lines that have been blunted by the smoothing processing in the image that has undergone the interpolation processing. Specifically, matrix calculation using a filter such as a Laplacian filter, which will be described later, is performed on the pixels that are to be smoothed. Since the white thin line detection process is performed on the image before the coordinate conversion process, it is determined whether or not the pixel is a pixel to be smoothed in the image after the coordinate conversion process. It is assumed that the determination is made based on the result and the correction value used in the coordinate conversion process.

  In step S808, color combination processing for recombining color information of a new pixel is performed based on the color information of each pixel that has undergone coordinate conversion processing and interpolation processing for each color component.

  In step S807, it is determined whether post-processing such as color correction or image format conversion is performed after the aberration correction processing. If it is determined that post-processing is to be performed, the process proceeds to step S810. If it is determined that post-processing is not to be performed, the aberration correction processing is terminated. In step S810, post-processing is performed on the aberration-corrected image.

<9. Explanation of Adaptive Smoothing Processing and Adaptive Sharpening Processing>
Next, details of the adaptive smoothing process (step S803) and the adaptive sharpening process (step S807) executed in the display system aberration correction unit 207 will be described.

<9.1 Functional Configuration of Adaptive Smoothing Processing Unit and Adaptive Sharpening Processing Unit>
First, functional configurations of the adaptive smoothing processing unit 702 that executes adaptive smoothing processing and the adaptive sharpening processing unit 708 that executes adaptive sharpening processing will be described. FIG. 9 is a functional block diagram of the adaptive smoothing processing unit 702.

  In FIG. 9, reference numeral 901 denotes a white thin line detection unit that detects a white thin line included in a displayed image. A white area for reading a region necessary for white thin line detection from the buffer 701 storing the display image and determining whether or not the target pixel is a part of a pixel forming the white thin line using a reference matrix described later. Perform thin line detection processing. The detection result is used in the subsequent adaptive sharpening process (step S803). Details of the white thin line detection process will be described later.

  Reference numeral 902 denotes a timing adjustment buffer. Reference numeral 903 denotes a smoothing filter that performs smoothing. The smoothing filter 903 is configured by a matrix of M rows × N columns. The smoothing filter 903 weights the target pixel with a specified coefficient, thereby realizing a smoothing process. .

  A control unit 904 controls the smoothing process. The degree of smoothing is controlled by changing the shape, size, and coefficient of the smoothing filter.

  A selector 905 selects an output from the smoothing filter 903 (pixels subjected to the smoothing process) and an output from the timing adjustment buffer 902 (pixels not subjected to the smoothing process) based on the detection result of the white thin line. It is.

  On the other hand, the adaptive sharpening processing unit 708 also has basically the same functional configuration as the adaptive smoothing processing unit 702. However, the adaptive sharpening processing unit 708 uses a sharpening filter instead of the smoothing filter 903. Further, the pixel to be sharpened by the sharpening filter is a pixel derived from the detection result of the white thin line transmitted from the adaptive smoothing processing unit 702 and the correction value used in the coordinate conversion process.

<9.2 Matrix Used in Adaptive Smoothing and Adaptive Sharpening>
Next, the matrix used in the adaptive smoothing process and the adaptive sharpening process will be described. FIG. 10 is a diagram illustrating a configuration of a reference matrix and a smoothing filter used in the adaptive smoothing process, and a sharpening filter used in the adaptive sharpening process.

  FIG. 10A shows an example of a reference matrix for detecting a white thin line. Although the reference matrix is shown here as a 3 × 5 matrix, the reference matrix is not limited to this size. The white thin line detection unit 901 determines whether the pixel of interest at the center of the reference matrix is a white thin line pixel that forms part of the white thin line or a non-applicable pixel that does not form a white thin line. Judgment based on. That is, the “white thin line pixel” refers to a pixel to be smoothed (and a pixel to be sharpened).

  When the white thin line is configured with a width of 3 pixels, it is necessary to refer to an area wider than 3 pixels in order to determine that the pixel outside the white thin line is not a white pixel. That is, in the case of the reference matrix, it is possible to detect up to a width of 3 pixels in the vertical line and up to a width of 1 pixel in the horizontal line. The size and shape of the reference matrix are arbitrarily changed by the control unit 904 according to the line width of the white thin line to be detected.

  FIG. 10B shows an example of a matrix of the smoothing filter 903. According to the matrix shown in FIG. 10 (b), smoothing is performed using a value obtained by dividing the sum of the luminance value of the pixel of interest by two and the luminance value of the surrounding eight pixels by 10 as the luminance value of the pixel of interest. Processing is performed. The filter coefficient (weighting coefficient) shown here is an example and is not limited to this value. In order to further enhance the smoothing effect, the value of the weighting coefficient of the pixel of interest may be made relatively smaller than the surrounding coefficients. Conversely, when it is desired to weaken the smoothing effect, the value of the weighting coefficient of the pixel of interest may be made relatively larger than the surrounding coefficients. In addition, the degree of smoothing can be increased by increasing the size of the reference matrix. Further, if the newly calculated luminance value after the smoothing process is not different from the original luminance value by a certain threshold or more, a process of adopting the original luminance value may be added.

  FIG. 10C shows an example of a sharpening filter matrix. According to the matrix shown in FIG. 10C, a value obtained by subtracting the sum of the luminance values of the surrounding eight pixels from the value obtained by multiplying the luminance value of the target pixel by 10 is divided by 2, and a new luminance value of the target pixel is obtained. A sharpening process is performed to obtain a value. The filter coefficient (weighting coefficient) shown here is an example and is not limited to this value. For example, only the diagonal pixel of the target pixel may be referred to like a Laplacian filter.

<9.3 White Thin Line Detection Processing Flow in Adaptive Smoothing Processing>
Next, the flow of the white thin line detection process in the adaptive smoothing process (step S803) will be described. FIG. 11 is a flowchart showing the flow of white thin line detection processing. Using the above-described reference matrix, it is determined whether or not the pixel of interest is a white thin line pixel constituting a part of the white thin line.

  In step S1101, the luminance value of each pixel in the reference matrix is acquired. In step S1102, it is determined whether the pixel of interest in the reference matrix is white. Whether the color is white is determined by determining whether each luminance value of RGB is equal to or greater than an arbitrary threshold value. For example, in the case of 8-bit data for each color, if each color component has a value of 192 or more, it is determined as white. If it is determined that the target pixel is white, the process proceeds to step S1103. If it is determined that the target pixel is not white, the process proceeds to step S1106.

  In step S1103, it is determined whether a pixel having the same luminance value as the target pixel exists in the pixels other than the target pixel in the reference matrix. This is because if the pixel of interest is a pixel that forms part of a white thin line, there should be pixels with the same or substantially the same luminance value in the matrix. If it is determined that there is a pixel having the same luminance value, the process proceeds to step S1104. If it is determined that no pixel exists, that is, the target pixel is a white isolated pixel, the process proceeds to step S1106.

  In step S1104, it is determined whether there is a pixel with a low luminance value having a value equal to or less than an arbitrary threshold value in the reference matrix. For example, in the case of 8-bit data for each color, if each color component has a luminance value smaller than 96, it can be determined that the difference in contrast between the white thin line and its periphery is large. If the contrast is large, false colors are likely to be noticeable during the interpolation processing by aberration correction, so that it can be determined that smoothing processing is necessary. On the other hand, when the luminance value of the outer periphery (background portion) of the white thin line is relatively large, the false color at the time of interpolation processing is not noticeable. For this reason, it can be judged that smoothing processing is unnecessary. Therefore, if there is a pixel having a luminance value equal to or less than an arbitrary threshold, the process proceeds to step S1105, and if not, the process proceeds to step S1106.

  In step S1105, in response to satisfying the conditions from step S1102 to step S1104, it is determined that the target pixel is a white thin line pixel (a pixel to be smoothed) that forms part of the white thin line, and white The thin line detection process ends.

  On the other hand, in step S1106, in response to not satisfying any of the conditions from step S1102 to step S1104, it is determined that the pixel is a non-corresponding pixel that is not to be smoothed, and the white thin line detection process ends.

<9.4 Flow of Smoothing Processing in Adaptive Smoothing Processing>
Next, the flow of the smoothing process in the adaptive smoothing process (step S803) will be described. FIG. 12 is a flowchart showing the flow of the smoothing process. In step S1201, the luminance value of each pixel in the matrix of the smoothing filter 903 at the time of the filter operation in the smoothing process is acquired.

  In step S1202, the detection result of the white thin line detection process described in the flowchart of FIG. 11 is grasped. In step S <b> 1203, it is determined whether or not a white thin line pixel constituting a part of the white thin line exists in the matrix of the smoothing filter 903. When it exists, it progresses to step S1204, and when it does not exist, smoothing processing is complete | finished.

  In step S1204, in response to the fact that the smoothing filter 903 matrix includes white thin line pixels that constitute a part of the white thin line, a filter operation is performed.

<10. Explanation of coordinate conversion processing and interpolation processing>
Next, coordinate conversion processing (step S805) and interpolation processing (step S806) executed in the display system aberration correction unit 207 will be described.

<10.1 Explanation of Aberration Correction Table>
First, the aberration correction table 704 used for the coordinate conversion process will be described. FIG. 13 is a diagram illustrating an example of the aberration correction table 704. As illustrated in FIG. 13, the aberration correction table 704 stores coordinates before conversion of a specific pixel and coordinates after conversion (or a difference value between the coordinates after conversion). By specifying the XY coordinates of a specific pixel based on the pixel of interest, the reference color (G in this embodiment) is the coordinate after conversion, and the colors other than the reference color (R and B) are the reference colors A difference value from the converted coordinates can be obtained. That is, the R converted coordinates can be obtained from the stored difference value 1301 (Gx−Rx) and the G converted coordinates Gx. Further, the converted coordinates of B can be obtained from the stored difference value 1302 (Bx−Gx) and the converted coordinates Gx of G.

  In this embodiment, coordinates before coordinate conversion are also configured as an aberration correction table. However, the present invention is not limited to this, and coordinates before coordinate conversion are associated with addresses at the time of memory access. Also good. With this configuration, it is not necessary to secure the pre-conversion coordinates in the memory area, and the memory size can be further reduced.

<10.2 Coordinate transformation process flow>
FIG. 14 is a flowchart showing the flow of coordinate conversion processing. Based on the distortion aberration information, chromatic aberration information, and resolution information of the lens with the display center coordinates (x0, y0) in the two-dimensional coordinate system as the origin, the amount of displacement in each pixel in the two-dimensional coordinate system is obtained. Then, each pixel is coordinate-converted in the x-axis direction and the y-axis direction based on the positional deviation amount.

  In this embodiment, in order to reduce the size of the aberration correction table, the relationship between the pre-conversion coordinates and the post-conversion coordinates does not hold for all the pixels in the image, but has only sampled representative points. The pixels between and are obtained by interpolation calculation.

  In step S1401, the coordinates of the target pixel are designated. In step S1402, the coordinates of a specific pixel in the aberration correction table corresponding to the target pixel are designated. Here, the coordinates of a specific pixel in the aberration correction table corresponding to the pixel of interest are designated after step S1401, but the present invention is not limited to this. For example, the order of steps S1401 and S1402 may be switched so as to be performed before step S1401.

  In step S1403, a correction value necessary for obtaining a coordinate value after coordinate conversion of the target pixel is acquired from the aberration correction table. In step S1404, using the correction value acquired from the aberration correction table in step S1403, the coordinate value after conversion for the target pixel is acquired. Specifically, if the distortion is a distortion, the amount of displacement of the pixel, and if it is chromatic aberration, the amount of displacement for each color component is used to calculate the coordinates after conversion of each color component. Get the later coordinate value. Note that the pre-conversion coordinates indicated in the stored aberration correction table are sampled representative values, and values between them are obtained by interpolation calculation. In this interpolation calculation, in addition to linear interpolation, a cubic interpolation curve, an interpolation curved surface, a spline curve, an approximate polynomial, or the like is used. However, the interpolation calculation is not particularly limited to such an interpolation expression.

  In step S1405, it is determined whether the coordinate conversion process has been executed for all pixels. If it is determined that the process has been completed for all pixels, the coordinate conversion process is terminated. On the other hand, if it is determined that the coordinate conversion process has not been performed on all the pixels, the process returns to step S1401 to continue the coordinate conversion process for correcting the aberration of the corresponding pixel.

<10.3 Outline of interpolation processing>
Next, the interpolation process will be described. First, an outline of the interpolation process will be described. FIG. 15 is a schematic diagram showing the concept of cubic interpolation processing. In the interpolation processing in aberration correction, the image quality is improved by using a cubic interpolation formula as interpolation processing for reconstructing the color information of the pixel of interest after coordinate conversion into the interpolation position. Here, the concept of general cubic interpolation processing will be described. The interpolation processing may use linear interpolation or may use a higher-order interpolation formula.

  Interpolation processing is to obtain pixel information (RGB luminance values) of an interpolation position in the two-dimensional coordinate system based on the pixel position of the pixel of interest subjected to coordinate conversion and the luminance value of each color component by interpolation calculation. In the example of FIG. 15, the x coordinate sx and the y coordinate sy of the interpolation pixel 1501 are specified in the interpolation calculation, and first, normalized distances px and py between the reference pixel and the interpolation pixel are obtained. Then, using the obtained distances px and py, a weighting coefficient is obtained for each coordinate of x and y. The cubic function used for the calculation is an interpolation curve based on the above-mentioned cubic equations. If bicubic, equations (1) and (2) are Lagrange, and (3) and (4) are applicable cubic interpolation equations. It becomes.

  Bicubic interpolation curve (bicubic interpolation from surrounding 16 points)

  Lagrange interpolation curve (bicubic interpolation from surrounding 16 points)

  The weighting coefficient of the reference pixel is calculated by substituting the values of px and py into the cubic interpolation formula for the coordinates of x and y. Note that px and py are normalized. In normal resolution conversion, the grid is square and has a constant value. However, since each interpolation pixel converted by coordinate conversion does not form a square, normalization is performed with the interval between reference pixels set to 1. Hereinafter, in order to simplify the explanation, it is assumed that the reference pixel exists on the normalized axis.

  In FIG. 15, a portion surrounded by a dotted line around the interpolation pixel 1501 is a normalized one region. That is, the distances px and py between the four reference pixels and the interpolation pixels existing on the y and y + 1 axes and the x and x + 1 axes are both smaller than 1. Conversely, the 12 pixels around it take a value between 1 and 2. The cubic interpolation formula used when calculating the weight coefficients of the nearest four pixels is (1) for bicubic and (3) for Lagrange. The cubic interpolation formula used for calculating the weighting coefficients of the outer 12 pixels is (2) for bicubic and (4) for Lagrange. After obtaining these weighting factors in the x and y directions, the luminance value of the reference pixel is multiplied by the x and y weighting factors, and the value added to the luminance values of the surrounding 16 pixels is a new value at the interpolation position. It becomes a luminance value.

  In actual processing, not all the nearest four pixels are present in or on the normalized grid, and conversely, the neighboring 12 pixels are not limited to taking a normalized value of 1 or more. In either case, it varies depending on the aberration correction amount.

<10.4 Flow of interpolation processing>
Next, the flow of interpolation processing will be described. FIG. 16 is a flowchart showing the flow of interpolation processing. Here, an interpolation process using the cubic interpolation formula will be described.

  In step S1601, coordinates are designated for the position of the interpolation pixel to be newly interpolated. In step 1602, 16 reference pixels in the vicinity of the interpolation pixel are designated.

  In step S1603, the coordinates of 16 neighboring pixels that are reference pixels are acquired. This coordinate is a normalized value. In step S1604, the distance between the interpolation pixel and each reference pixel is calculated. The distance is prepared as a value normalized by the interval between representative points. In the present embodiment, since the target is a two-dimensional space, the values of the x coordinate and the y coordinate are obtained.

  In step S1605, the weighting coefficient of each reference pixel is obtained by substituting the distance calculated in step S1604 into an interpolation curve or an interpolation line. Here, it is assumed that a cubic interpolation formula is adopted, but a linear interpolation (bilinear) algorithm may be adopted. In step S1606, the value of each reference pixel and the product of the weighting coefficients in the x and y coordinates are added to calculate the luminance value at the interpolation position.

  In step S1607, it is determined whether the above processing has been executed for all pixels. If it is determined that the above process has been executed for all pixels, the interpolation process is terminated. On the other hand, if it is determined that there is a pixel that has not been subjected to the above process, the process returns to step S1601 to execute the above process for the pixel.

<11. Effect of aberration correction processing by display system aberration interpolation unit>
Next, the effect of suppressing false color generation by the aberration correction processing in the display system aberration correction unit 207 will be described. In the description, first, a false color generation mechanism when aberration correction processing is performed on a white thin line included in an image will be described.

  17A and 17B are schematic diagrams for explaining the mechanism of false color generation and the effect of suppressing false color generation by the aberration correction processing in the display system aberration correction unit 207. Here, in order to simplify the description, a white thin line having a line width of one pixel will be described. Even when the line width is composed of a plurality of pixels, false colors are generated according to the same mechanism. Therefore, the description of the effect of suppressing false color generation by the aberration correction processing in the display system aberration correction unit 207 is the same.

  17A shows an example of a white thin line before aberration correction, and FIG. 17A shows an example of a white thin line reproduced by interpolation processing that realizes aberration correction. FIG. 17A (b) shows a line of a single color (R, G, or B). FIG. 17A (c) shows an example of a false color pattern generated by overlapping the single-color white thin lines shown in FIG. 17A (b).

  FIG. 17B (d) shows an example of a white thin line before aberration correction to which the smoothing process is applied in the display system aberration correction unit 207, and FIG. 17B (e) shows an interpolation after the smoothing process is applied. An example of a white thin line reproduced by processing is shown. Further, (f) of FIG. 17B shows a pattern in which the generation of false colors is suppressed even when the single-color white thin lines shown in (e) of FIG. 17B overlap each other. Further, (g) of FIG. 17B shows an example of a monochromatic white thin line when the sharpening process is applied to (e) of FIG. 17B.

  In correcting chromatic aberration, coordinate conversion processing and interpolation processing are performed for each monochrome image plane of each RGB color constituting the image. The false color is generated due to a local luminance difference between the case where the interpolated pixel is expressed by a plurality of pixels and the case where it is expressed by a single pixel.

  (A) of FIG. 17A is an example of the white thin line in the original image before aberration correction. Here, it is assumed that a white thin line is configured by a single pixel and a single pixel.

  FIG. 17A (b) shows a white thin line reproduced by performing coordinate conversion processing and interpolation processing on the white thin line shown in FIG. 17A (a). Since reverse correction that cancels the influence of aberration is applied, the straight line is reconstructed as a curved line or an oblique line. In the line reconstructed by the interpolation processing, as shown in (b), a place constituted by a single pixel and a place constituted by a plurality of pixels are alternately repeated.

  Here, in the case of being configured with a single pixel and the case of being configured with a plurality of pixels, although the overall luminance value is preserved, the luminance value locally differs for each pixel. That is, even if a uniform white thin line is expressed in the original image, the luminance value is increased or decreased in the image after the interpolation processing, and luminance unevenness appears on the display image visually recognized by the user. When general interpolation processing is applied to the pixel configuration at the time of aberration correction, such luminance unevenness occurs.

  In an image to which interpolation processing is applied, it is rare to be represented by a single pixel, but even when it is composed of multiple pixels, the value of one pixel is large, The pixel can be regarded as a single pixel.

  FIG. 17A (c) shows an example of a false color pattern generated when the white thin line is reconstructed by the display optical system by applying aberration correction, and each color of RGB periodically appears on the white thin line. Is shown.

  Furthermore, when aberration correction is performed on the white thin line in FIG. 17A (a), pixels are generated by coordinate conversion processing and interpolation processing for each RGB color component, and therefore, as shown in FIG. 17A (b). As described above, the white thin lines of each color of RGB have their own luminance unevenness. When the white thin lines are reconstructed by combining and synthesizing these color thin lines, the intensity of each color (unevenness in luminance) is further shifted, so that a periodic false color appears on the white thin lines. For example, when R and G are composed of a plurality of pixels and B is composed of a single pixel at a certain point on the white thin line, the luminance value of B is higher than that of R and G when compared with the luminance value. In the synthesized pixel, the B color appears strongly on the white thin line.

  Such false colors occur in an image that has undergone interpolation processing, and occur regardless of the amount of aberration correction of the display optical system, and also occur in the central portion as well as the outer peripheral portion of the image. To do.

  On the other hand, (d) of FIG. 17B shows an example in which smoothing processing is applied to a white thin line in an original image before aberration correction. A white thin line with a single luminance and a line composed of a single pixel, the luminance value of the original white thin line portion is reduced by the smoothing process, and the luminance value is also distributed to the pixels adjacent to the white thin line. It becomes.

  (E) of FIG. 17B has shown the example of the monochrome white thin line by which the interpolation process was carried out. The false color is generated due to the strength of the local pixel value on the white thin line reconstructed by the interpolation process. Therefore, the generation of the false color is made inconspicuous by reducing the strength of the pixel value. It is possible.

  In other words, smoothing processing by a smoothing filter is applied to reduce the luminance value of a pixel represented by a single pixel, and at the same time, distribute the luminance value to the peripheral pixels of the white thin line to preserve the overall luminance value However, it is possible to reduce the luminance unevenness of each color. As a result, the generation of false colors is suppressed even in the combined and synthesized white thin line. In addition, the smoothing process in this embodiment shall be applied independently for every color.

  FIG. 17B (f) shows a pattern in which the occurrence of false colors is suppressed even if the single color white thin lines overlap each color component. When white smooth lines are reconfigured by applying smoothing processing to each color component, the maximum brightness value of the white thin lines is slightly reduced, but white thin lines with few false colors with reduced brightness unevenness are generated as display images. It becomes possible to do.

  When detecting the white thin line pixel to be smoothed, only the pixel whose target pixel is white and the pixel adjacent to the white pixel has low luminance is defined as a white thin line pixel. That is, even if the target pixel is white, if the luminance value of the adjacent pixel is a relatively large value, it is determined as a non-relevant pixel. This is because the occurrence of false colors is not noticeable when the luminance value of the pixel adjacent to the target pixel is relatively large.

  Such white thin line pixels to be smoothed are generally image elements that do not exist in a natural image, such as CG images artificially generated for MR or VR use, characters inserted for annotation purposes, Included in the image displaying the icon. Further, when the thin line is not white, for example, if the line is green or red, periodic luminance unevenness occurs but false color does not occur.

  (G) of FIG. 17B has shown the example which performed the sharpening process with respect to the white thin line to which the smoothing process was applied. Although the overall luminance is almost preserved only by applying the smoothing process, the edge portion is seen as a dull line by human eyes. Therefore, by applying sharpening processing to the pixels that make up the white thin line and increasing the lowered luminance value, the line itself is visually recognized as thick, but the white thin line with high sharpness that is visually recognized as white is white. Can be provided as a display image.

  As described above, in this embodiment, when correcting chromatic aberration caused by the display optical system by signal processing, it is determined whether or not the target pixel is a white thin line pixel included in the display image. . Then, a smoothing process is performed on the target pixel determined to be a white thin line pixel before the interpolation process. This makes it possible to suppress the occurrence of false colors that occur during interpolation processing.

  Further, in the present embodiment, a sharpening process is applied to the pixels to be smoothed to increase the sharpness of the white thin lines. This makes it possible to display white lines with high visibility and preserved edges.

  As described above, in the present embodiment, the display image quality is improved by combining the processing with a relatively small processing load such as the white thin line detection processing, the smoothing processing, and the sharpening processing. For this reason, it is possible to provide a balanced chromatic aberration correction function capable of realizing high image quality, high processing speed, and downsizing of the apparatus.

[Second Embodiment]
A second embodiment of the present invention will be described. In the image processing method according to the present embodiment, the image processing method according to the first embodiment applies the smoothing process to all the pixels determined to be white thin line pixels, whereas the white thin line It is characterized in that the smoothing process is applied only to the outer peripheral pixels excluding the central part of the pixels. In the case of the image processing method, the effect does not change if the line width of the white thin line is within 2 pixels, but if the line width is 3 pixels or more, the luminance value at the center of the white thin line is prevented from decreasing. There is an effect that can be. Hereinafter, the image processing method according to the present embodiment will be described focusing on this point.

<1. Flow of white thin line detection process in adaptive smoothing process>
FIG. 18 is a flowchart showing a flow of white thin line detection processing in adaptive smoothing processing. In step S1801, the luminance value of each pixel in the reference matrix is acquired. In step S1802, it is determined whether the target pixel is white. The processing content is the same as that in step S1102 of FIG. If the target pixel is white, the process proceeds to step S1803. If the target pixel is not white, the process proceeds to step S1808.

  In step S1803, it is determined whether a pixel having the same luminance value as the target pixel exists in the pixels other than the target pixel in the reference matrix. The processing content is the same as that in step S1103 in FIG. If it is determined that there is a pixel having the same luminance value, the process proceeds to step S1804. If it is determined that there is no pixel, that is, the target pixel is an isolated pixel, the process proceeds to step S1808.

  In step S1804, it is determined whether or not there is a pixel with a low luminance value equal to or less than an arbitrary threshold in the reference matrix. The contents of this process are the same as in step S1104 in FIG. If there is a pixel having a luminance value equal to or less than an arbitrary threshold, the process proceeds to step S1805, and if not, the process proceeds to step S1808.

  In step S1805, it is determined whether the outer peripheral pixel of the target pixel is also white. When not only the pixel of interest but also its peripheral pixels are white, it can be determined that the pixel of interest is a pixel located at the center of the white thin line. If it is determined that the outer peripheral pixels are also white, the process proceeds to step S1806, and if any of the pixels is not white, the process proceeds to step S1807.

  In step S1806, the target pixel is a pixel that constitutes a part of the white thin line, and is a central pixel that is not located on the outer periphery of the white thin line (a “white thin line central pixel that is not subject to smoothing”). Judge and end the process.

  In step S1807, it is determined that the pixel of interest is a pixel corresponding to the outer peripheral portion of the white thin line (“white thin line outer peripheral pixel” to be smoothed), and the process ends.

  In step S1808, since any of the conditions from step S1802 to step S1804 is not satisfied, it is determined that the pixel does not become a smoothing target, and the process ends.

<2. Flow of smoothing process in adaptive smoothing process>
Next, the flow of the smoothing process in the adaptive smoothing process will be described. FIG. 19 is a flowchart showing the flow of the smoothing process in the adaptive smoothing process. The processing contents in step S1901 and step S1902 are the same as the processing contents in step S1201 and step S1202 of FIG.

  In step S1903, whether the target pixel is a part of pixels constituting the white thin line and is an outer peripheral pixel of the white thin line adjacent to the non-corresponding pixel (that is, the outer peripheral pixel of the white thin line to be subjected to the smoothing process). ) Determine whether or not. If it is determined that the pixel is a white thin line peripheral pixel, the process proceeds to step S1904, and if it is determined that the pixel is not a white thin line peripheral pixel (that is, if it is determined that the pixel is a white thin line central pixel or a non-applicable pixel). The process ends.

  In step S1904, the pixel of interest is smoothed by performing the filter operation described above, and the process ends.

  As described above, in the first embodiment, the white thin line is reconfigured by combining the smoothing process and the sharpening process. However, according to the present embodiment, the smoothing process is performed only on the peripheral pixel of the white thin line. By applying, it becomes possible to have a configuration in which sharpening processing is not performed. In this case, although the sharpness is somewhat reduced, it is possible to suppress the occurrence of false color while maintaining the brightness value of the white fine line (the brightness value of the peak at the center of the white fine line) as in the first embodiment. It becomes.

  In the case of this embodiment, the width of the applicable white thin line is limited, but the adaptive sharpening process can be omitted, so that the processing load can be reduced compared to the first embodiment. . Note that the image processing method described in the first embodiment and the image processing method described in this embodiment may be combined in accordance with the line width.

[Third Embodiment]
A third embodiment of the present invention will be described. In the first embodiment, the smoothing process and the subsequent interpolation process are realized as separate processes. On the other hand, in the image processing method according to the present embodiment, when performing interpolation processing on white thin line pixels to be smoothed, an interpolation curve having a high smoothing effect is adopted, and the pixels are processed in the same process. It is characterized in that smoothing and interpolation are realized. Hereinafter, the image processing method according to the present embodiment will be described in detail focusing on this point.

<1. Functional configuration of display system aberration correction unit>
First, a display system aberration correction unit that realizes the image processing method according to the present embodiment will be described. FIG. 20 is a block diagram illustrating a functional configuration of the display system aberration correction unit 2000 that realizes the image processing method according to the present embodiment. A buffer 2001, a color separation unit 2003, an aberration correction table 2004, and a correction value selection unit 2005 illustrated in FIG. 20 are the same as the corresponding functional blocks described with reference to FIG. 7 in the first embodiment. In addition, for each functional block of the coordinate calculation unit 2006, the adaptive sharpening processing unit 2008, and the color combining unit 2009 shown in FIG. 20, the corresponding functions described with reference to FIG. 7 in the first embodiment. Same as block. For this reason, description of these functional blocks is omitted here.

  Reference numeral 2002 denotes a white thin line detection unit that detects white thin lines included in a display image. The white thin line detection unit 2002 reads an area necessary for white thin line detection from the buffer 2001 that stores the display image, and uses the reference matrix described above to form a white thin line pixel in which the target pixel forms part of the white thin line. It is determined whether or not.

  Reference numeral 2007 denotes an adaptive interpolation processing unit that performs an interpolation process to reconstruct the color information of the pixel of interest after conversion into an interpolation position near the pixel of interest. In the adaptive interpolation processing unit 2007, when the target pixel is a white thin line pixel based on the detection result in the white thin line detection unit 2002, the interpolation process is performed after selecting a high smoothing effect as an interpolation formula. To do. Details of the interpolation processing will be described later.

  With the above configuration, the effect of smoothing that suppresses the generation of false colors is realized at the stage of interpolation processing, and the scale of the processing circuit for realizing the display system aberration correction unit 2000 can be reduced. It becomes.

<2. Flow of aberration correction processing by display system aberration correction unit>
Next, the flow of aberration correction processing by the display system aberration correction unit 2000 will be described. FIG. 21 is a flowchart showing the flow of aberration correction processing by the display system aberration correction unit 2000. The processing contents of steps S2101, 2102, 2104, 2105, and 2107 to 2110 are the same as the processing contents of the corresponding steps described with reference to FIG. 8 in the first embodiment, and thus description thereof is omitted here. To do.

  In step S2103, a white thin line pixel constituting a part of the white thin line is detected as a pixel to be subjected to an interpolation process having a smoothing effect. The content of the processing is almost the same as the processing content described with reference to FIG. 11 in the first embodiment. In the white thin line detection process in the first embodiment, it is determined whether the pixel is a part of a pixel constituting the white thin line or a non-corresponding pixel. It is also determined whether or not the pixel is a non-corresponding pixel adjacent to the fine line (“white fine line adjacent pixel”).

  In step S2106, interpolation is performed to reconstruct the luminance value of each color component at a new configuration position (interpolation position) based on the pixel information of the pixel of interest and the pixel information of the reference pixel obtained in step S2105. Process. At this time, if it is determined in step S2103 that the pixel of interest is a white thin line pixel, an interpolation curve having a smoothing effect is applied. Details of the adaptive interpolation processing will be described below.

<3. Explanation of adaptive interpolation processing>
Next, the adaptive interpolation process (step S2106) executed in the display system aberration correction unit 2000 will be described.

<3.1 Interpolation curve explanation>
First, the interpolation curve used in the adaptive interpolation process (step S2106) will be described. FIG. 22 is a diagram illustrating an example of an interpolation curve used in the adaptive interpolation process (step S2106). In FIG. 22, 2201 (dashed line) is an interpolation curve indicating the bicubic filter shape. Similarly, 2202 (thin solid line) indicates a Lagrangian interpolation curve, 2203 (dotted line) indicates a bilinear straight line, and 2204 (solid line) indicates a B-SPLINE curve (solid line). The numerical value on the axis is a value normalized by the pixel interval. Note that bicubic and Lagrange, which are relational expressions between surrounding reference pixels and interpolation pixels, are as described above.

  As a filter shape for realizing the interpolation processing, a curve having a larger value between 0 and 1 (a curve is located on the upper side) than bilinear which is linear interpolation (primary interpolation) has a high sharpening effect. In general, when bilinear which is linear interpolation is used, it is known that an interpolated pixel is an average of four surrounding pixels, resulting in a dull image in which edge components are lost. On the other hand, bicubic can perform a third-order calculation in which the surrounding 16 pixels are weighted to obtain a high-quality image that leaves the edge component.

  That is, in terms of the sharpening effect, bicubic 2201 is the highest, followed by Lagrange 2202, bilinear 2203, and B-SPLINE curve 2204. On the other hand, in terms of the smoothing effect, the order is reversed.

  In normal interpolation processing, bicubic 2201 is used to achieve high image quality, and when it is desired to obtain a smoothing effect, other interpolation formulas, interpolation curves / straight lines shown in the figure depending on the degree of smoothing. Is adopted. In the interpolation process for white thin line pixels determined to be part of the pixels constituting the white thin line, an interpolation curve having different smoothing effects is selected depending on the number of white pixels included in the reference matrix, Generate interpolated pixels.

  For example, if all four pixels near the interpolation position in the reference matrix are white thin line pixels, it is assumed that the interpolation pixel is a white thin line center pixel, so a Lagrangian curve having a smoothing effect is selected. To do.

<3.2 Flow of adaptive interpolation processing>
Next, a detailed process flow of the adaptive interpolation process (step S2106) will be described. FIG. 23 is a flowchart showing a detailed process flow of the adaptive interpolation process. Here, it is assumed that bicubic is used as a commonly used interpolation curve.

  Since the processing contents of steps S2301 to 2304 are the same as the processing contents of the corresponding steps described with reference to FIG. 17 in the first embodiment, description thereof is omitted here.

  In step S2305, it is determined whether or not the target pixel included in the reference matrix is a white thin line pixel constituting a part of the white thin line, and the detection result of the white thin line detection process is grasped.

  In step S2306, it is determined whether or not white pixels forming a white thin line are included in the four pixels near the interpolation position. If it is determined that it is included, the process proceeds to step S2307, and if it is determined that it is not included, the process proceeds to step S2310.

  In step S2307, it is further determined whether or not white four-line adjacent pixels are included in the four neighboring pixels. If it is determined that it is included, the process proceeds to step S2309. If it is determined that it is not included, the process proceeds to step S2308.

  In step S2308, an interpolation process having a smoothing effect is performed. While normal interpolation processing is performed by the bicubic described above, here, for example, interpolation processing by Lagrange or linear interpolation is performed.

  In step S2309, an interpolation process having an intensity smoothing effect is performed. For example, the interpolation process using the B-SPLINE curve described above is performed.

  In step S2310, normal interpolation processing is performed. Here, interpolation processing by bicubic is performed.

  In step S2311, it is determined whether the above process has been executed for all pixels. If it is determined that the above process has been executed for all pixels, the process ends. If it is determined that there is a pixel that has not been subjected to the above process, the process returns to step S2301, and the above process is performed for the corresponding pixel.

  As is apparent from the above description, the present embodiment is configured to selectively switch to an interpolation curve having a high smoothing effect during interpolation processing. As a result, smoothing and interpolation processing can be realized on the same process. In addition, by adopting a configuration realized on the same process as in this embodiment, when the same cubic interpolation formula is adopted, switching can be performed only by changing the coefficient, so that the aberration correction function can be performed relatively easily. It can be realized. For this reason, compared with the said 1st and 2nd embodiment, it becomes possible to reduce the circuit scale of a hardware structure.

[Fourth Embodiment]
A fourth embodiment of the present invention will be described. In the third embodiment, the smoothing process and the subsequent interpolation process are realized by the same process. On the other hand, in the image processing method according to the present embodiment, the sharpening process and the interpolation process are realized on the same process by adopting an interpolation curve having a high sharpening effect when the white thin line pixel is interpolated. There is a feature in that. Hereinafter, the image processing method according to the present embodiment will be described in detail with a focus on this point.

<1. Functional configuration of display system aberration correction unit>
First, a display system aberration correction unit that realizes the image processing method according to the present embodiment will be described. FIG. 24 is a block diagram illustrating a functional configuration of a display system aberration correction unit 2400 that realizes the image processing method according to the present embodiment. Note that the processing contents of the buffer 2401, the adaptive smoothing processing unit 2402, the color separation unit 2403, and the aberration correction table 2404 shown in FIG. 24 are the corresponding functions described with reference to FIG. 7 in the first embodiment. The processing contents of the block are the same. The processing contents of each functional block of the correction value selection unit 2405, the coordinate calculation unit 2406, and the color combination unit 2408 are also the processing contents of the corresponding functional blocks described with reference to FIG. 7 in the first embodiment. Is the same. Therefore, description of these functional blocks is omitted here.

  Reference numeral 2407 denotes an adaptive interpolation processing unit that performs an interpolation process for reconstructing the color information of the pixel of interest after conversion into an interpolation position near the pixel of interest. In the adaptive interpolation processing unit 2407, based on the detection result of the white thin line detection unit, when the target pixel is a white thin line pixel, after selecting a high sharpening effect as an interpolation formula to be used during the interpolation process, Perform interpolation processing.

  By adopting such a configuration, the sharpening process that should be realized after performing the smoothing process that suppresses the occurrence of false colors is realized at the stage of the interpolation process, and the display system aberration correction unit 2400. It is possible to reduce the scale of the processing circuit for realizing the above.

<2. Flow of aberration correction processing by display system aberration correction unit>
Next, the flow of aberration correction processing by the display system aberration correction unit 2400 will be described. FIG. 25 is a flowchart showing the flow of aberration correction processing by the display system aberration correction unit 2400. Since the processing contents of steps S2501 to 2505 and 2507 to 2509 are the same as the processing contents of the corresponding steps described with reference to FIG. 8 in the first embodiment, description thereof is omitted here. Note that in the adaptive smoothing process in step S2503, it is also determined whether or not the pixel is adjacent to the white fine line described in the white thin line detection process of the third embodiment.

  In step S2506, interpolation is performed to reconstruct the luminance value of each color component at a new configuration position (interpolation position) based on the pixel information of the pixel of interest and the pixel information of the reference pixel obtained in step S2505. Process. At this time, if it is determined in step S2503 that the pixel of interest is a white thin line pixel, an interpolation curve having a high sharpening effect is applied.

<3. Explanation of adaptive interpolation processing>
Next, the adaptive interpolation process (step S2506) executed in the display system aberration correction unit 2400 will be described.

<3.1 Interpolation curve explanation>
First, the interpolation curve used in the adaptive interpolation process (step S2506) will be described. FIG. 26 is a diagram illustrating an example of an interpolation curve used in the adaptive interpolation process (step S2506). In FIG. 26, reference numeral 2601 (dashed line) denotes an interpolation curve indicating a bicubic filter shape. Similarly, 2602 (dotted line) indicates a bilinear straight line, and 2603 and 2604 (both solid lines) indicate curves having a high sharpening effect. The numerical value on the axis is a value normalized by the pixel interval, as in the third embodiment. Bicubic and Lagrange, which are relational expressions between surrounding reference pixels and interpolation pixels, are as described above. Here, an equation of a curve having a high sharpening effect is shown.

  Sharpening curve 2603 showing medium sharpness

  Sharpening curve 2604 showing the sharpness of the intensity

  Note that the interpolation curve having a high sharpening effect shown here is a unique curve calculated so that the sharpening effect is higher than that of bicubic. Other than the curves shown here, as described above, any curve can be used as long as it is a larger value between 0 and 1 (the curve is located on the upper side) than bicubic. It doesn't matter.

  The sharpening effect has the highest intensity sharpening curve 2604, followed by the medium sharpening curve 2603, bicubic 2601, and bilinear 2602 in this order.

  In order to achieve high image quality in normal interpolation processing, when using bicubic 2601 to obtain a sharpening effect, other interpolation formulas and interpolations shown in FIG. 26 according to the degree of smoothing processing are used. Adopt a curve.

  In interpolation processing for white thin line pixels that are determined to be pixels that form part of the white thin line, an interpolation curve with a different sharpening effect is selected according to the number of white pixels included in the reference matrix. Perform interpolation processing. For example, when all four pixels in the vicinity of the interpolation pixel in the reference matrix are white thin line pixels, it is assumed that the interpolation pixel is a white thin line center pixel. Therefore, a sharpening curve 2603 having a sharpening effect is obtained. select.

<3.2 Flow of adaptive interpolation processing>
Next, a detailed process flow of the adaptive interpolation process (step S2506) will be described. FIG. 27 is a flowchart showing the flow of adaptive interpolation processing. Here, bicubic is used as a normal interpolation curve.

  Since the processing contents from step S2701 to S2707 are the same as the processing contents of each step described with reference to FIG. 23 in the third embodiment, description thereof is omitted here.

  In step S2708, an interpolation process having a sharpening effect is performed. When bicubic is used as normal interpolation processing, for example, interpolation processing using a sharpened curve 2603 is performed here.

  In step S2309, an interpolation process having an intensity sharpening effect is performed. Specifically, an interpolation process using a sharpening curve 2604 is performed. In step S2310, normal interpolation processing is performed. Here, interpolation processing is performed using bicubic.

  In step S2711, it is determined whether the above processing has been executed for all pixels. If it is determined that the above process has been executed for all pixels, the process ends. On the other hand, if it is determined that there is a pixel that has not been subjected to the above process, the process returns to step S2701 to execute the above process on the corresponding pixel.

  As is apparent from the above description, the present embodiment is configured to selectively switch to an interpolation curve having a high sharpening effect during interpolation processing. As a result, the sharpening process and the interpolation process can be realized on the same process. Further, as in the present embodiment, by adopting a configuration realized by the same process, when the same cubic interpolation formula is adopted, it can be realized only by changing the coefficient. The circuit scale of the configuration can be reduced.

[Fifth Embodiment]
In the first to fourth embodiments, the case where the image processing method according to the present invention is applied to all images displayed on the HMD in the MR system has been described.

  However, whether or not the display image generated by the image processing apparatus 202 includes a wire frame or a character configured by a CG image can be grasped in advance by the image processing apparatus 202 that generates a white thin line. Therefore, information regarding whether or not an image including a white thin line and a similar white pixel has been generated may be transmitted from the image processing apparatus 202 to the HMD 201 to switch application of adaptive smoothing processing. .

  Further, when characters for warning display and information display can be displayed by OSD or the like inside the HMD 201, the HMD 201 determines the presence or absence of such display and switches the application of adaptive smoothing processing. May be.

  Further, in the first to fourth embodiments, no particular mention was made of variations of the display optical system. However, the optical member such as a lens or prism may be a display optical system that employs an up / down or left / right asymmetric system, or may be a display optical system that generates a combination of a plurality of lenses or an image plane multiple times. There may be.

  In the first to fourth embodiments, the address conversion is described as an example of the coordinate conversion process. However, the coordinate conversion process based on the numerical conversion using the resolution conversion method and the approximate polynomial described in the prior art is adopted. May be.

  In the first to fourth embodiments, the case where the aberration correction is performed in the HMD that employs the magnifying optical system of the eyepiece has been described. However, the application destination of the aberration correction is not limited to the HMD, but a digital camera, a mobile phone with a camera and a digital video camera, and a rear projection type television that have a display optical system and need to correct the aberration. Or EVF mounted on a camera or the like.

  Needless to say, the configurations described in the first to fourth embodiments may be used in combination with each other. Furthermore, it is easy for those skilled in the art to configure a new system by appropriately combining various techniques in the above embodiments. Therefore, a system based on such various combinations also belongs to the category of the present invention.

[Other Embodiments]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (13)

An image processing apparatus for processing an image to be displayed,
Determining means for determining whether or not the pixel of interest when processing the image is a white thin line pixel that is included in the image and that forms a white thin line that is higher than a predetermined luminance value and thinner than a predetermined width;
When the determination means determines that the pixel is a white thin line pixel, it operates to perform a smoothing process on the pixel of interest, and when the determination unit determines that the pixel is not a white thin line pixel, A calculation means for calculating color information of the target pixel by operating so as not to perform the smoothing process on the pixel;
Conversion means for converting the position of the pixel of interest for which color information has been calculated by the calculation means, based on information indicating a shift amount of an imaging position caused by chromatic aberration occurring in a display optical system for displaying the image; ,
The color information of the target pixel converted by the conversion unit is reproduced by the color information of the pixel that is in the vicinity of the target pixel and that corresponds to the display position when displaying an image in the display optical system. An image processing apparatus comprising: interpolation means for performing interpolation processing on the target pixel.   The image processing apparatus further includes sharpening processing means for performing sharpening processing on a target pixel determined to be a white thin line pixel by the determination means among the target pixels interpolated by the interpolation means. The image processing apparatus according to claim 1.   The interpolation unit performs the interpolation process using interpolation curves having different sharpening effects between a target pixel determined to be a white thin line pixel by the determination unit and a target pixel determined not to be a white thin line pixel. The image processing apparatus according to claim 1, wherein the image processing apparatus performs the processing.   The interpolation curve used in the interpolation process when the determination unit determines that the pixel is a white thin line pixel is compared with the interpolation curve used in the interpolation process when the determination unit determines that the pixel is not a white thin line pixel. The image processing apparatus according to claim 3, wherein the interpolation processing curve has a high sharpening effect.   In the case where the calculating unit determines that the pixel is a white thin line pixel in the determining unit, and the target pixel is a white thin line outer peripheral pixel that is a pixel constituting the outer peripheral portion of the white thin line, The image processing apparatus according to claim 1, wherein smoothing processing is performed. An image processing apparatus for processing an image to be displayed,
Determining means for determining whether or not the pixel of interest when processing the image is a white thin line pixel included in the image and constituting a white thin line higher than a predetermined luminance value and thinner than a predetermined width;
Conversion means for converting the position of the pixel of interest based on information indicating a shift amount of an imaging position caused by chromatic aberration generated in a display optical system for displaying the image;
The color information of the target pixel converted by the conversion unit is reproduced by the color information of the pixel that is in the vicinity of the target pixel and that corresponds to the display position when displaying an image in the display optical system. And interpolation means for performing interpolation processing of the target pixel,
The interpolation means uses the interpolation curves having different smoothing effects when the pixel of interest is determined to be a white thin line pixel by the determination means and when it is determined not to be a white thin line pixel. An image processing apparatus that performs processing.   The interpolation curve used in the interpolation process when the determination unit determines that the pixel is a white thin line pixel is compared with the interpolation curve used in the interpolation process when the determination unit determines that the pixel is not a white thin line pixel. The image processing apparatus according to claim 6, wherein the interpolation processing curve has a high smoothing effect.   The image processing apparatus further includes sharpening processing means for performing sharpening processing on a target pixel determined to be a white thin line pixel by the determination means among the target pixels interpolated by the interpolation means. The image processing apparatus according to claim 7.   The determination unit is a pixel in which the pixel of interest is higher than a predetermined luminance value, a pixel having the same luminance value as the pixel of interest exists around the pixel of interest, and is around the pixel of interest. The image processing apparatus according to claim 1, wherein when there is a pixel having a luminance value equal to or less than a predetermined threshold, the image processing apparatus determines that the pixel is a white thin line pixel. An image processing method in an image processing apparatus for processing an image to be displayed,
Judgment means for judging whether or not the pixel of interest when processing the image is a white fine line pixel constituting a white fine line included in the image that is higher than a predetermined luminance value and thinner than a predetermined width. Process,
When it is determined that the calculation unit is a white thin line pixel in the determination step, the calculation unit operates to perform a smoothing process on the target pixel, and when it is determined that the calculation unit is not a white thin line pixel in the determination step. Calculating a color information of the target pixel by operating so as not to perform a smoothing process on the target pixel;
The conversion means converts the position of the target pixel for which color information has been calculated in the calculation step based on information indicating a shift amount of an imaging position caused by chromatic aberration occurring in a display optical system for displaying the image. Conversion process to
The color information of the pixel of interest converted in the conversion step by the interpolation means is the pixel color information corresponding to the display position when the display optical system displays an image in the vicinity of the pixel of interest. An image processing method comprising: an interpolation step of performing interpolation processing of the target pixel by reproducing by the above. An image processing method in an image processing apparatus for processing an image to be displayed,
Judgment means for judging whether or not the pixel of interest when processing the image is a white fine line pixel constituting a white fine line included in the image that is higher than a predetermined luminance value and thinner than a predetermined width. Process,
A converting step for converting the position of the pixel of interest based on information indicating a shift amount of an imaging position caused by chromatic aberration generated in a display optical system for displaying the image;
The color information of the pixel of interest converted in the conversion step by the interpolation means is the pixel color information corresponding to the display position when the display optical system displays an image in the vicinity of the pixel of interest. And an interpolation step for performing interpolation processing of the target pixel by reproducing by
The interpolation step uses the interpolation curves having different smoothing effects depending on whether the target pixel is determined to be a white thin line pixel or not to be a white thin line pixel in the determination step. An image processing method characterized by performing processing.   The program for making a computer perform each process of the image processing method of Claim 10 or 11.   The storage medium which memorize | stored the program for making a computer perform each process of the image processing method of Claim 10 or 11.