JP2017050700A - Image processor, imaging apparatus and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make excellent magnification chromatic aberration corrections while reducing a computational complexity.SOLUTION: An image processor has: processing means 111 of performing correction processing, associated with magnification chromatic aberrations of an optical system, on an input image generated by imaging through the optical system; detection means 116 of detecting magnification chromatic aberration quantities of the optical system; calculation means 117 of calculating a first correction value corresponding to a detection result of the detection means as a correction value to be used for the correction processing; and storage means 117a of holding a second correction value prepared as the correction value without reference to the detection result of the detection means. The processing means divides the input image into a plurality of correction object regions, selects one of the first correction value and second correction value for each of the correction object regions, and performs the correction processing.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for performing image processing for improving image quality on an image generated by imaging.

  When an image is acquired using an imaging device such as a digital camera, the lateral chromatic aberration, which is one of the various aberrations of the imaging optical system, causes a fringe of a color different from the original subject color to the edge portion of the subject image. Cause image quality to deteriorate. The magnitude (amount) of such lateral chromatic aberration depends on the incident angle of the light beam from the subject to the optical system, and in particular, the lateral chromatic aberration for a light beam having a large incident angle from a short-distance subject increases.

  In Patent Document 1, correction of lateral chromatic aberration in an image (captured image) obtained by imaging is determined using a lateral chromatic aberration amount based on a design value of the optical system and a lateral chromatic aberration amount detected in the captured image. A method of performing using is disclosed.

JP 2012-070053 A

  In the method disclosed in Patent Document 1, the magnification chromatic aberration correction amount is detected in all regions in the captured image. However, since detection of the amount of chromatic aberration of magnification in a captured image requires calculation for each pixel and color, if the amount of chromatic aberration of magnification is detected in all regions, the amount of calculation becomes enormous, and the calculation processing unit in the imaging apparatus is burdened. large. When the burden on the arithmetic processing unit is increased, the continuous shooting performance of the imaging apparatus is reduced and the power consumption is increased.

  The present invention provides an image processing apparatus, an imaging apparatus, and an image processing program that can perform good lateral chromatic aberration correction while reducing the amount of calculation.

  An image processing apparatus according to one aspect of the present invention detects processing for correcting a chromatic aberration of magnification of an optical system, and processing means for performing correction processing on the chromatic aberration of magnification of the optical system for an input image generated by imaging through the optical system. A detection means for calculating, a calculation means for calculating a first correction value corresponding to a detection result by the detection means as a correction value used in the correction process, and a correction value prepared without depending on the detection result by the detection means. Storage means for holding the second correction value. The processing means divides the input image into a plurality of correction target areas, and performs correction processing by selecting one of the first correction value and the second correction value for each correction target area.

  Note that an imaging apparatus including an imaging unit that performs imaging and the image processing apparatus that performs image processing on an input image generated by the imaging unit also constitutes another aspect of the present invention.

  An image processing program according to another aspect of the present invention is a computer program that causes a computer to perform image processing including correction processing related to lateral chromatic aberration of an optical system with respect to an input image generated by imaging through the optical system. . The image processing performs detection processing for detecting the amount of chromatic aberration of magnification of the optical system, calculates a first correction value corresponding to the detection result by the detection processing as a correction value used in the correction processing, and performs detection processing as the correction value. Preparing a second correction value that does not depend on the detection result of, dividing the input image into a plurality of correction target areas, and selecting one of the first correction value and the second correction value for each correction target area Correction processing is performed.

  In the present invention, correction processing is performed by selecting one of the first correction value corresponding to the detection result of the magnification chromatic aberration amount and the second correction value prepared without depending on the detection result for each correction target region. . As a result, the chromatic aberration of magnification is corrected well while reducing the amount of calculation compared to the case where the amount of chromatic aberration of magnification is detected or the first correction value is calculated in all of the plurality of correction target areas over the entire input image. Result image can be obtained.

1 is a block diagram illustrating a configuration of an imaging apparatus that is Embodiment 1 of the present invention. FIG. 6 is a diagram illustrating the size of a processed image obtained by processing a captured image in the first embodiment. FIG. 3 is a block diagram illustrating a configuration of a region matching unit according to the first embodiment. FIG. 3 is a diagram illustrating logic of a frame control unit according to the first embodiment. FIG. 3 is a diagram illustrating a logic of an area counting unit according to the first embodiment. FIG. 3 is a diagram illustrating logic of a ratio counting unit according to the first embodiment. FIG. 3 is a diagram illustrating logic of a first selection unit according to the first embodiment. FIG. 6 is a diagram illustrating logic of a second selection unit in the first embodiment. FIG. 4 is a diagram illustrating a definition of a region for detecting a magnification chromatic aberration amount in the first embodiment. FIG. 3 is a block diagram illustrating a configuration of a detection execution determination unit according to the first embodiment. The figure which shows the distance map data in Example 1, and a detection implementation determination result. FIG. 3 is a diagram illustrating a chromatic aberration amount for each image height region in the first embodiment. FIG. 6 is a diagram illustrating a relationship between a subject distance and a defocus amount in the first embodiment. 9 is a flowchart illustrating an image processing program that is Embodiment 2 of the present invention.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 shows a configuration of a digital camera (hereinafter simply referred to as a camera) as an image pickup apparatus that is Embodiment 1 of the present invention. An imaging optical system (referred to as an optical system in the figure) 121 is configured by a lens that is provided integrally with the camera or exchangeable with the camera.

  The imaging system (imaging means) 100 includes an image sensor such as a CMOS sensor, an A / D converter, a timing generation unit, and the like. The imaging system 100 photoelectrically converts the subject image formed by the optical system 121 by an image sensor that operates according to the timing signal from the timing generation unit, and converts the analog signal output from the image sensor to an A / D converter. Is converted into a digital signal. Thereby, a captured image (hereinafter referred to as image data) as digital data is generated. In this embodiment, the pixel array of the image sensor is a Bayer array composed of R / Gr / Gb / B. However, a pixel array other than the Bayer array may be used.

  Reference numerals 102 to 105 and 109 to 120 denote components constituting a digital signal processing unit as an image processing apparatus. The digital signal processing unit in this embodiment is configured as one LSI, and performs image processing, which will be described later, as hardware. However, the digital signal processing unit may be configured by a CPU dedicated to image processing, and image processing may be performed by software (computer program). Reference numerals 106 to 108 denote processing data generated in the digital signal processing unit.

  The transmission system 101 transmits the image data generated by the imaging system 100 to the sensor correction unit 102. A means for rearranging the pixel readout order of the sensor to the correction order by the sensor correction unit 102 may be mounted here.

  The sensor correction unit 102 corrects an image quality deterioration factor caused by physical characteristics of components of the imaging system 100 when the imaging system 100 generates image data. For example, the S / N of the image sensor is improved, a pixel defect or a sensitivity defect is corrected, or an individual difference of an amplifier that reads a signal from the image sensor is corrected. The image data that has been processed by the sensor correction unit 102 is temporarily stored as RAW data (denoted as RAW Image in the figure) 107 as pre-development data, as well as a WB detection unit 103 and an image reduction processing unit. It is sent to 104.

  The WB detection unit 103 is equipped with a block integration circuit (not shown) for evaluating white balance in RAW data, and temporarily stores a white balance evaluation value 106 that is the integration result. The image reduction processing unit 104 resizes (reduces) the RAW data to the requested processing size of the distance map generation unit 105. When the requested image size of the distance map generation unit 105 is equivalent to the RAW data from the sensor correction unit 102, the image reduction processing unit 104 does not need to perform resizing. The required processing size (resolution) is determined by the use of a distance map, which will be described later, generated by the distance map generation unit 105.

  The distance map generation unit (distance acquisition unit) 105 uses the resized RAW data from the image reduction processing unit 104, and the distance (subject distance) for each subject in one or more subjects shown in the RAW data. And a distance map 108 is generated. The distance map 108 corresponds to information related to the subject distance. The distance map generation unit 105 uses, for example, a DFD (Depth From Defocus) method to generate the distance map 108. DFD is a method for estimating a subject distance from information on blur (defocus amount) in an image.

  When the DFD method is used, the distance map generation unit 105 performs a process of applying a band pass filter (BPF) to a plurality (two) of RAW data and correlating their output powers. At this time, in the RAW data, a low power is obtained in a region where a low-contrast subject is captured, while a high power is obtained in a region where a high-contrast subject (a subject causing an isolated point or pixel saturation) is captured. Even in this area, the reliability of the estimated subject distance is low. For this reason, the distance map generation unit 105 may be provided with a reliability determination unit that determines the reliability (certainty) of the calculation result for each region of the obtained distance map 108.

  The reliability determination unit detects a low-contrast region and a high-contrast region from the RAW data from the image reduction processing unit 104 using threshold values, and the reliability of the estimated subject distance of each of the detected low-contrast region and high-contrast region is detected. Find the degree. Then, an area whose reliability is lower than a predetermined value is set as an invalid data area (for example, an area defined by 0 value).

  The distance map generation unit 105 temporarily stores data of the generated distance map 108. The WB evaluation value 106, the RAW data 107, and the distance map 108 may be temporarily stored in a storage device (not shown) such as a DRAM, and read from the storage device as appropriate at the start of subsequent processing.

  Here, the distance map 108 is a set of numerical values indicating the subject distance for each image area in the image data (RAW data in the present embodiment) obtained by imaging, and is generated with a resolution suitable for its use. Uses include imaging scene determination for improving the performance of distance measurement and subject tracking, image processing (Computational Photography) outside the camera attached to an image data file, and the like. When subject tracking is used, the resolution may be coarse so that the spatial context of the subject can be grasped.

  However, it is not appropriate from the viewpoint of imaging processing throughput and resource burden to change the resolution of the distance map for each use and handle the distance map of various resolutions. For this reason, in this embodiment, a region matching unit 119 is provided. The region matching unit 119 includes a plurality of distance regions having different subject distances in the distance map 108 and a plurality of chromatic aberration detection regions (that is, correction targets for correcting the chromatic aberration of magnification) in which the amount of chromatic aberration of magnification is detected by the magnification chromatic aberration detection unit 116 described later. And (region) are matched (corresponding) to each other. Specifically, the distance map 108 is subdivided, that is, resolution conversion is performed so that the distance area corresponds to the chromatic aberration detection area on a one-to-one basis.

  The detection execution determination unit 120 is a magnification chromatic aberration detection unit according to the subject distance in an arbitrary distance region among a plurality of distance regions whose resolution is converted to correspond to the chromatic aberration detection region by the region matching unit 119 in the distance map 108. Whether or not the magnification chromatic aberration amount is detected by 116 is determined. The threshold for determining whether or not to detect the amount of chromatic aberration of magnification is the amount of color misregistration for each subject distance by actually imaging the reference object (subject) multiple times while changing the subject distance in the assembly process of the optical system 121. May be determined by measuring each individual optical system 121. The detection execution determination unit 120 outputs flag information indicating the result of whether or not the detection of the magnification chromatic aberration amount for each chromatic aberration detection region is performed to the magnification chromatic aberration detection unit 116.

  Each of the WB correction units 109 and 114 performs white balance correction corresponding to the WB evaluation value 106 on the RAW data from the sensor correction unit 102.

  The synchronization processing units 110 and 115 perform demosaic processing on the RAW data that has been subjected to white balance correction by the WB correction units 109 and 114, respectively. That is, the synchronization processing unit 115 outputs three image data (hereinafter, collectively referred to as RGB image data) obtained by synchronizing the demosaic data of the three colors R, G, and B.

  In the present embodiment, the WB correction units 109 and 114 and the synchronization processing units 114 and 115 are configured separately, but they may be the same circuit.

  The magnification chromatic aberration detection unit (detection means) 116 has a magnification in a chromatic aberration detection region in which the flag information from the detection execution determination unit 120 of the RGB image data from the synchronization processing unit 115 indicates that the amount of magnification chromatic aberration is detected. The amount of chromatic aberration is detected. Then, a correction amount (first correction value: hereinafter referred to as a calculated correction amount) for correcting this is calculated according to the detected amount of lateral chromatic aberration. Here, for the detection of the amount of chromatic aberration of magnification, it is desirable to use the same image data as the correction target of the chromatic aberration of magnification using the detection result. The lateral chromatic aberration is corrected by the lateral chromatic aberration correcting unit 111 (to be described later) on the RGB image data that has been subjected to white balance correction by the WB correcting unit 109. For this reason, the magnification chromatic aberration detection unit 116 detects the amount of magnification chromatic aberration for the RGB image data on which the white balance correction is performed by the WB correction unit 114.

  The correction coefficient processing unit 117 (calculation unit) prepares a correction coefficient to be used by the magnification chromatic aberration correction unit 111. Here, a representative value as a function of the image height (hereinafter referred to as a representative calculated correction amount) is calculated from the calculated correction amounts in the plurality of chromatic aberration detection regions obtained by the magnification chromatic aberration detection unit 116. The lateral chromatic aberration detection unit 116 needs to merge the correction amount based on the design value of the optical system 121 in the region that does not have the calculation correction amount, according to the result of the detection execution determination unit 120. A representative design correction amount, which is a representative value of the correction amount based on the design value of the optical system 121 at this time (second correction value: hereinafter referred to as design correction amount), is held in a memory (storage means) 117a in FIG. If you do.

  The representative design correction amount is a design correction amount held (prepared) as a function of the image height in a representative optical state (subject distance, aperture value, zoom position, etc.) of the optical system 121 at the time of imaging. It is this correction coefficient processing unit 117 that obtains the design correction amount under the optical conditions at the time of photographing from the representative design correction amount by interpolation calculation.

  Based on the flag information output from the detection execution determination unit 120, the correction coefficient processing unit 117 selects a calculated correction amount for a region where the magnification chromatic aberration amount is to be detected. On the other hand, a design correction amount is selected for a region where the detection of the magnification chromatic aberration amount is not performed. In this embodiment, switching of the correction amount in an arbitrary image height direction is equivalent to the resolution of a chromatic aberration amount detection area described later. Therefore, when generating a correction amount in an arbitrary image height direction, one of the calculated correction amount and the design correction amount is selected as the use correction amount at each image height, and the use correction amount used by the magnification chromatic aberration correction unit 111 is determined. To do. The correction coefficient processing unit 117 sends the use correction amount to the magnification chromatic aberration correction unit (processing unit) 111 in FIG.

  The lateral chromatic aberration correction unit 111 uses the use correction amount (calculated correction amount or design correction amount) received from the correction coefficient processing unit 117 for the RGB image data from the synchronization processing unit 110 to perform color correction in the image height direction. A correction process for a shift (for example, a shift between the R image and the B image with respect to the G image) is performed. The RGB image data in which the lateral chromatic aberration is corrected in this way is sent to the monochromatic aberration correction unit 112.

  The monochromatic aberration correction unit 112 corrects aberrations other than lateral chromatic aberration, such as distortion, on the RGB image data after lateral chromatic aberration correction. Further, the false color suppression unit 113 generates false color generated in the high luminance region and high frequency region and color noise generated in the dark region or low contrast region with respect to the RGB image data after the aberration correction by the monochromatic aberration correction unit 112. Perform suppression processing. Then, the processed RGB image data is sent to a development processing unit (recording processing unit) and a display processing unit.

  FIGS. 2A and 2B show the size relationship between the image data generated by imaging and the distance map generated by the distance map generation unit 105 in this embodiment. As shown in FIG. 2A, the size (number of pixels) of the image data in this embodiment is horizontal M pixels × vertical N pixels. On the other hand, as shown in FIG. 2B, the distance map 108 has a size of horizontal m pixels × vertical n pixels (m <M, n <N). In this embodiment, the distance map 108 does not need a resolution (resolution) equivalent to that of the image data. For example, even if the size of the image data is horizontal 6000 pixels × vertical 4000 pixels (24M pixels), the subject distance only needs to be obtained for each chromatic aberration detection region obtained by dividing the 24M pixels into a plurality of pixels, and therefore 100 pixels in both horizontal and vertical directions. The size may be about.

  FIG. 2C shows the number of divisions when the image data is divided into a plurality of chromatic aberration detection areas. One chromatic aberration detection region includes a plurality of pixels of image data. In this embodiment, the image data is divided into chromatic aberration detection areas of horizontal H area × vertical V area.

  FIG. 9 shows the direction in which the lateral chromatic aberration is detected in this embodiment. The chromatic aberration of magnification appears as a difference in image magnification for each color (wavelength) in the image height direction with the optical axis position of the optical system 121 on the imaging surface as the center (starting point). Therefore, by detecting the shift amount of the edge portion in the image height direction of another color image with respect to the specific color image (for example, the shift amount of the edge portion in the R and B images with respect to the edge portion in the G image). The amount of lateral chromatic aberration can be obtained. In this embodiment, in order to reduce the amount of calculation, as shown in FIG. 9, the radiation direction from the center of the optical axis is divided into eight regions, and the upper and lower (N, S), which are the radiation directions representing each region, Left and right (E, W) and diagonal (NE, SE, SW, NW) are taken as the detection direction of the lateral chromatic aberration. With respect to the detection direction of the lateral chromatic aberration set in this way, the arrangement direction of the plurality of chromatic aberration detection regions shown in FIG. 2C is directed to the horizontal and vertical directions as indicated by thin lines in FIG. For this reason, in this embodiment, it is determined in which detection direction each chromatic aberration detection region exists, and magnification chromatic aberration is detected in the chromatic aberration detection region in the detected direction in the determined region.

  FIG. 3 shows the configuration of the region matching unit 119 in the present embodiment. In the present embodiment, as described above, since the resolution (number of divisions) of the chromatic aberration detection region is higher than the resolution (resolution) of the distance map 108, the region matching unit 119 performs resolution conversion of the distance map 108. The data of the distance map 108 is stored in a map storage unit on a DRAM (not shown) and is appropriately read out by a DMA controller (hereinafter referred to as DMAC).

  Reference numerals 301 to 309 denote constituent blocks of the area matching unit 119. Reference numeral 301 denotes an area coefficient block that counts distance areas (frames) in the distance map 108, and 302 denotes an area coefficient block that counts chromatic aberration detection areas (frames) in the image data.

  A frame control block 309 sends a data transfer request signal sig_300 to the DMAC 300 in response to the arrival of a start event from a CPU (not shown) mounted on the camera (hereinafter referred to as camera CPU). Also, the frame control block 309 sends a status signal sig_315 to the area counting block 301 so that the counting operation by the area counting block 301 is started by the start event. The enable signal sig_313 is also sent to the area counting block 302 so that the counting operation is started.

  Upon receiving the data transfer request signal sig_300, the DMAC 300 sends the map data sig_302 of the attention line and the map data sig_301 of the next line read from the map storage unit to the data buffer 303.

  The data buffer 303 sequentially writes the map data sig_302 of the target line into the buffer 3030 having the FIFO structure and the map data sig_301 of the next line into the buffer 3031 having the FIFO structure. The data buffer 303 sequentially reads the map data, and sends the map data sig_304 of the target line to the selection block A306 and the map data sig_305 of the next line to the selection block A307. Further, the data buffer 303 sends a data update signal sig_ 303 indicating the sending timing (update timing) of the map data to the selection blocks A 306 and 307 and the frame control block 309.

  The frame control block 309 that has received the data update signal sig_303 sends a progress request status signal sig_314 to the area counting block 302 and the ratio counting block 304. The progress request status signal sig_314 is enabled from the time when the data update signal sig_303 is output until the time when the transmission completion timing signal sig_307 is output to the ratio counting block 304. The progress request status signal sig_314 is also output to the detection execution determination unit 120.

  The area counting block 302 sends a counter reset signal sig_317 to the ratio counting block 304 every time the line unit is counted. The counter reset signal sig_317 is also sent to the ratio counting block 305 as a count update signal. The ratio counting blocks 304 and 305 are configured to match (correlate) one distance area after the above-described resolution conversion of the distance map 108 and one chromatic aberration amount detection area, and a chromatic aberration detection area corresponding to the one distance area. Set the ratio of the number of. Then, selection signals sig_306 and sig_308 are sent to selection blocks A306 and 307 and selection block B308, respectively, at the time of switching the chromatic aberration detection area to be matched.

  Since the boundary for each chromatic aberration detection area and the distance area unit on the distance map are different, there is a situation where the pixel data of interest and the next pixel data on the corresponding distance map are mixed at the data switching point of the chromatic aberration detection area. Occur. The ratio counting blocks 304 and 305 calculate the pixel-of-interest center of gravity on the rear-side pixel (pixel having a resolution equivalent to that of the chromatic aberration detection region) at the pixel-of-interest switching point. Select Map Data as it is. On the other hand, if not, the selection signals sig_306 and sig_308 are output so as to select the next pixel data (next distance map data).

  The selection block A306 is a horizontal data selection selector for the target line data on the distance map, and the selection block A307 is a horizontal data selection selector for the next line data. Then, signals sig_309 and sig_310 indicating the results are sent to the selection block B308. The selection block B is a data selection selector that selects data in the vertical direction with respect to the map data of the attention and next line selected by the selection blocks A 306 and 307. Then, the signal sig_311 indicating the selection result, that is, the distance map data after resolution conversion is output to the detection execution determination unit 120.

  4 to 8 show the logic of each block described above of the region matching unit 119. FIG. Each logic represents a typical operation in a steady state (one-frame processing performed by batch sequential scanning). In addition, the circuit of each component block operates in synchronization with a common synchronous clock (so that in-phase transfer is established).

  FIG. 4 shows the logic of the frame control block 309. The reset signal reg_3091 is set by the camera CPU, and returns the state holding element (DFF) to the initial state by the signal value “1”. The trigger signal reg — 3092 is set by the camera CPU, and starts the alignment operation by the region alignment unit 119 when the signal value is “1”. The event of trigger signal reg — 3092 (setting of signal value “1”) is controlled exclusively with the issuance of reset signal reg — 3091.

  When receiving the trigger signal reg_3092 when the reset signal reg_3091 is released, the frame control block 309 sends a data transfer request signal sig_300 to the DMAC 300. The trigger signal reg_3092 is a pulse signal, and the data transfer request signal sig_300 is also output as a pulse signal. The frame control block 309 that has output the data transfer request signal sig_300 causes the enable signal sig_313 to be transmitted to the area counting block 302 to transition to the state of the signal value “1” and holds it.

  When the data transfer request signal sig_300 is transmitted, the data buffer unit 303 transmits the data update signal sig_303 to the frame control block 309. The data update signal sig_303) is a pulse signal and is output with a width corresponding to one cycle of the synchronous clock.

  The data update signal sig_303 and the enable signal sig_313 are changed by holding the progress request status signal sig_314 to the state of the signal value “1” by AND logic. The signal value “1” of the enable signal sig_ 313 indicates the enable state of the count operation of the area counting block 302. The signal value “1” of the progress request status signal sig_314 indicates the update of the count value of the area count block 302 and the update of the count value of the ratio count block 304.

  The data update signal sig_307 is a pulse signal indicating that the transmission of one data to the detection execution determination unit 120 is completed. Upon arrival of the transmission completion timing signal sig_307, the frame control block 309 changes the progress request status signal sig_314 to the signal value “0” state and holds it. At the same time, the frame control block 309 issues a data transfer request signal sig_300 to request the next processing data in the distance map 108 from the DMAC 300. Also, the frame control block 309 that has received the data update signal sig_307 sends a frame update signal sig_316 to the area count block 301 to notify the update of the frame count for the distance map 108.

  FIG. 5 shows the logic of the area counting blocks 301 and 302. In the figure, reference numerals without parentheses indicate signals or setting values relating to the area counting block 301, and reference numerals with parentheses indicate signals or setting values relating to the area counting block 302. The area counting block 301 manages the progress of processing for the distance map 108, and the area counting block 302 manages the progress of processing for the magnification chromatic aberration detection area.

  In the area counting block 301, the H size setting value reg_3011 sets the size (number of pixels) in the horizontal direction of the distance map 108, and the V size setting value reg_3012 sets the size (number of pixels) in the vertical direction of the distance map 108. To do.

  In the state where the status signal sig_315 is a signal value “0”, the DFFs 501 and 502 of the area counting block 301 are in a reset state, and the count process is not performed. When the status signal sig_315 is in the signal value “1”, the area counting block 301 performs a count process. The DFF 501 of the area counting block 301 counts the number of processed pixels in the horizontal direction. When the frame update signal sig_316 is a signal value “1”, the count value of the DFF 501 is incremented. Further, the DFF 502 of the area counting block 301 counts the number of processed pixels in the vertical direction. When the count results of the number of processed pixels in the horizontal and vertical directions reach the H size setting value reg_301 and the V size setting value reg_3012, respectively, the area counting block 301 sends an output signal sig_312 having a signal value “1” to the frame control block 309. To do.

  On the other hand, in the area counting block 302, the H area setting value reg_3021 sets the number of areas in the horizontal direction of the magnification chromatic aberration detection area, and the V image number setting value reg_3022 sets the number of areas in the vertical direction of the chromatic aberration detection area. . When the enable signal sig_313 is in the state of the signal value “1” and the progress request status signal sig_314 is in the state of the signal value “1”, the count value of the DFF 501 is incremented. When the count value of the DFF 501 becomes equal to the H region setting value reg_3021, the DFF 502 is incremented. When the count results in the horizontal and vertical directions reach the set values reg_3021, reg_3022, the area counting block 301 sends an output signal sig_318 having a signal value “1” to the frame control block 309. In the area counting block 302, in response to the count value H in the horizontal direction reaching a value equivalent to the area setting value reg_3021, a signal indicating the arrival of the signal value “1” of the progress request status signal sig_314 is output as a counter reset signal. It outputs as sig_317. The counter reset signal sig_317 is sent to the ratio counting blocks 304 and 305.

  FIG. 6 shows the logic of the ratio counting blocks 304 and 305. In the figure, reference numerals without parentheses indicate signals or setting values relating to the ratio counting block 304, and reference numerals with parentheses indicate signals or setting values relating to the ratio counting block 305.

  In the ratio counting block 304, the H ratio setting value reg_3041 sets the ratio of the number of pixels in the horizontal direction of the image data corresponding to one horizontal pixel of the distance map 108. For example, when one pixel of the distance map 108 corresponds to the number of image data pixels of 2.4 in the horizontal direction, the H ratio setting value reg_3041 is set to a value corresponding to 2.4. An area including the number of image data pixels corresponding to 2.4 is equivalent to a chromatic aberration detection area. The hardware can be implemented as a fixed point. In that case, where to set the decimal point in many bits, the required specification of the desired resolution (how much the value equivalent to 2.4 is held) at the time of circuit implementation It may be determined in view of the above.

  The ratio counting block 304 updates the calculation of the ratio every time the signal value of the progress request status signal sig_314 becomes “1”. The DFF 601 accumulates a value of 1.0 each time processing for one chromatic aberration detection region is completed. The DFF 601 has a resolution below the decimal point, and updates the integrated value to a result value obtained by subtracting the H ratio set value reg_3041 from the current integrated value when the integrated value exceeds the H ratio set value reg_3041. In addition, when the progress request status signal sig_314 arrives, the DFF 601 transmits a signal (1 bit) at the place of 0.5 in the updated integrated value as the selection signal sig_306. Furthermore, the DFF 601 resets the integrated value when the signal value “1” of the counter reset signal sig_317 is input.

  The operation of the ratio counting block 305 is the same as the operation of the ratio counting block 305. The V ratio set value reg_3051 sets the ratio of the number of chromatic aberration detection areas in the vertical direction corresponding to one vertical pixel in the distance map 108. This V ratio set value reg_3051 may be the same value as the H ratio set value reg_3041 when the distance map 108 and the pixels of the image data are arranged in a square lattice. However, in the ratio counting block 305, the DFF 601 updates the integrated value according to the input of the counter reset signal sig_317. The DFF 601 resets the integrated value in response to the input of the output signal sig_318.

  FIG. 7 shows the logic of the selection blocks A 306 and 307. In the figure, reference numerals without parentheses indicate signals related to the selection block A306, and reference numerals with parentheses indicate signals regarding the selection block A307.

  As described above, the selection block A 306 performs processing on the map data of the target line in the distance map 108, and the selection block A 307 performs processing on the map data of the next line in the distance map 108. Data update in these selection blocks A306 and 307 is performed at the input timing of the data update signal sig_303. In the selection block A306, the DFF 702 holds the target pixel data of the map data of the target line, and the DFF 701 stores the next pixel data of the map data of the target line. On the other hand, in the selection block A307, the DFF 702 holds the target pixel data of the map data of the next line, and the DFF 701 holds the next pixel data of the map data of the next line. In the selection blocks A 306 and 307, when the selection signal sig_306 has the signal value “1”, the next pixel data held by the DFF 701 is selected, and when the selection signal sig_306 has the signal value “0”, the DFF 702 holds. The target pixel data is selected. Since this selection is a horizontal switching selection of the distance area on the distance map corresponding to the chromatic aberration detection area of interest, the instruction of the signal sig_306 common to the attention line and the next line is followed.

  FIG. 8 shows the logic of the selection block B308. The distance information on the target line is input to the DFF 801 as the output signal sig_309 of the selection block A306 at the input timing of the signal sig_303. Distance information on the next line is input to the DFF 802 as the output signal sig_310 of the selection block A307 at the input timing of the signal sig_303. The selection of the outputs of DFF 801 and DFF 802 is selected in the state of the output signal sig_308 of the ratio counting block 305. In FIG. 8, if the signal sig_308 is “0” value, the distance information of the line of interest is selected, and if it is “1” value, the distance information of the next line is selected.

  The output from the region matching unit 119 is distance map data having a resolution that matches the resolution of the chromatic aberration detection region in which the magnification chromatic aberration amount is detected by the magnification chromatic aberration detection unit 116. FIG. 11 shows an example of the process of matching the distance map data with the resolution of the chromatic aberration detection area. The actual distance map is multi-value information, and FIGS. 11A to 11C show the multi-value information as monochrome image data. In these figures, the closer the subject distance is, the closer the color is to white, and the farther the distance is, the closer the color is to black.

  FIG. 11A shows a distance map 108. In this figure, the number of pixels (resolution) of the distance map 108 is set to horizontal 10 pixels × vertical 7 pixels. However, this is an example for facilitating understanding, and an actual distance map may have a larger number of pixels. In FIG. 11B, a plurality of distance regions in the distance map after resolution conversion by the region matching unit 119 are superimposed on the distance map shown in FIG. One area surrounded by a white frame represents one distance area after resolution conversion.

  FIG. 11C shows a distance map (signal sig_311 in FIG. 3) after resolution conversion in the region matching unit 119 as image data. One distance area of this distance map corresponds to one chromatic aberration detection area.

  FIG. 10 shows the configuration of the detection execution determination unit 120.

  A signal sig_314 in FIG. 3 is a status signal indicating that the signal sig_311 in FIG. 3 is valid. The detection execution determination unit 120 takes in the signal sig_311 when the signal sig_314 has a “1” value. These signals sig_311 and sig_314 are taken in as internal signals of the detection execution determination unit 120 via the reception-side I / F control block 1201 of the detection execution determination unit 120 shown in FIG. The comparison of the subject distance (multi-valued data) and the set value reg_1000 in an arbitrary distance area taken into the receiving-side I / F control block 1201 is performed in the comparator 1206 in FIG. The set value (threshold value) reg_1000 measures the amount of color misregistration for each subject distance when the reference object is actually imaged while changing the subject distance in the assembly process of the optical system 121 as described above. This is the value determined. The comparator 1206 outputs a value of 1 as a determination result (comparison result) for a distance area (first distance area) determined that the subject distance is smaller than the set value reg_1000, that is, a short distance. This is temporarily stored in the temporary buffer 1204. In this embodiment, the temporary buffer 1204 has a capacity capable of holding a determination result in units of frames (for one image). Actually, the capacity may be set smaller than the frame unit in consideration of the processing speed of the magnification chromatic aberration detection unit 116 in the subsequent stage.

  The comparator 1206 outputs a value of 0 as a determination result for a distance area (second distance area) determined that the subject distance is greater than the set value reg_1000, that is, a long distance, and this is temporarily output. The data is temporarily stored in the buffer 1204. FIG. 11D shows binary data indicating whether or not the magnification chromatic aberration amount is detected for each chromatic aberration detection region. A chromatic aberration detection area where the magnification chromatic aberration amount is detected is indicated by white (1 value), and a chromatic aberration detection area where the magnification chromatic aberration amount is not detected is indicated by black (0 value). This binary data is output from the detection execution determination unit 120.

  In this way, each time the distance map data (sig_311) from the region matching unit 119 is input to the comparator 1206, the determination result is temporarily stored in the temporary buffer 1204. A data transfer request signal sig_1002 from the magnification chromatic aberration detection unit 116 at the subsequent stage of the detection execution determination unit 120 is input to the arbiter 1202 via the transmission-side I / F control unit 1205. The arbiter 1202 writes the received data from the previous stage when the reception of data from the previous area matching unit 119 of the detection execution determination unit 120 and the data transfer request from the subsequent magnification chromatic aberration detection unit 116 collide. Prioritize and arbitrate to hold (hold) the data transfer request from the succeeding stage.

  In addition, the arbiter 1202 sends a data processing request to the RAM_I / F control unit 1203 each time the received data is stored in the temporary buffer 1204 and the transmission data corresponding to the data transfer request is read from the temporary buffer 1204.

  In response to the data processing request, the RAM I / F control unit 1203 issues an address signal to the temporary buffer 1204, issues a read / write status, issues an enable signal, and the like. Thus, transmission / reception of data to / from the temporary buffer 1204 is controlled. The data of the determination result read from the temporary buffer 1204 is sent to the subsequent stage as data sig_1000 and data valid status sig_1001 via the transmission side I / F control unit 1205.

  It should be noted that a frame counter that recognizes the absolute position of the processing region may be mounted in the detection execution determination unit 120 to set information on the optical axis center position and image height threshold information. Then, at the image height lower than the image height threshold, the output of the comparator 1206 may be discarded and the determination result may be output so as not to detect the magnification chromatic aberration amount.

  The lateral chromatic aberration detection unit 116 performs white balance correction and receives the synchronized RGB image data and the determination result from the detection execution determination unit 120, and the determination result indicates detection of the amount of lateral chromatic aberration. The magnification chromatic aberration amount is detected only for the region. Then, the above-described calculation correction amount is calculated based on the detected magnification chromatic aberration amount. The lateral chromatic aberration detection unit 116 sends the calculated correction amount and the determination result from the detection execution determination unit 120 to the correction coefficient processing unit 117.

  The correction coefficient processing unit 117 receives the calculated correction amount, the determination result from the detection execution determination unit 120, and the design correction amount described above.

  The amount of lateral chromatic aberration in design of the optical system 121 (hereinafter also referred to as “design chromatic aberration amount”) changes according to the optical state such as the subject distance, zoom position, aperture value, and the like, and also changes according to the image height. In FIGS. 12A and 12B, h1 to h8 denote design chromatic aberration amounts of the eight image height regions (FIG. 12B when the total image height in an arbitrary optical state is divided into eight image height regions. ) Shows the representative value of the color misregistration amount). In FIG. 12A, reference numeral 1200 denotes the optical axis center position. FIG. 12B shows an approximate curve generated by approximating the representative value of the color misregistration amount for each image height region by spline interpolation or the like. At the time of actual imaging, after an optical state (optical condition) of the optical system 121 is determined, an approximate curve for the color shift amount for each image height region is generated, and magnification chromatic aberration correction is performed for the color shift amount represented by this approximate curve. The unit 111 performs correction.

  The correction coefficient processing unit 117 selects one of the calculated correction amount and the design correction amount according to the determination result from the detection execution determination unit 120 and sends it to the magnification chromatic aberration correction unit 111 as a use correction amount.

  As described above, in this embodiment, a distance map is generated, and a process of matching (corresponding) the distance area in the distance map to the chromatic aberration detection area in the captured image (RGB image data) is performed. Then, it is determined whether or not to detect the magnification chromatic aberration amount (that is, to generate a calculated correction amount) for each chromatic aberration detection region. Then, the correction processing is performed using the calculated correction amount in the chromatic aberration detection region determined to detect the magnification chromatic aberration amount, and the design correction amount is used in the chromatic aberration detection region determined not to detect the magnification chromatic aberration amount. Perform correction processing. As a result, it is possible to obtain a result image in which the chromatic aberration of magnification is favorably corrected while reducing the calculation amount compared to the case where the calculation correction amount is calculated in all of the plurality of chromatic aberration detection regions over the entire captured image (input image). Can do.

  In this embodiment, the case where the DFD method is used for generating the distance map has been described, but the distance map may be generated using other methods.

  The relationship between the distance between the imaging unit and the subject is calculated as a defocus amount (image-side movement amount). In order to treat the defocus amount as the object distance, processing for converting the defocus amount into the object side distance is required. FIG. 13 shows the relationship between the subject distance and the defocus amount.

When the lateral magnification of the optical system 121 is M (= Y / X), the focal length is f, the subject distance is (A + f), and the image distance is (B + f),
M = Y / X
= F / A = B / f (1)
It is. When (1) is multiplied by (fA),
f / A · (fA) = B / f · (fA) (2)
So
f ² = A · B (3)
It becomes.
If the object movement amount is a and the corresponding defocus amount is b,
(A + a) (B + b) = f ² (4)
It becomes.
From the equations (3) and (4),
(A + a) = (A · B) / (B + b) (5)
Is obtained. Thus, the relationship between the subject distance (A + a) and the defocus amount b is expressed by the equation (5). When the distance A (the distance from the front focal length to the object plane) is deleted from the right side of the equation (5),
1+ (a / A) = B / (B + b) (6)
It becomes. Equation (6) is equivalent to a value obtained by normalizing the subject distance by the distance A, and is easier to handle than handling a direct distance. Since ratio calculation (division) is required, even if it is configured to set a reasonable number of arithmetic digits, such as treating it as infinity after a certain distance, clip the value when it exceeds that Good.

  Note that a predetermined region near the optical axis including the optical axis of the optical system 121 (1200 in FIG. 12) is defined as a specific correction target region, and in such a region, the design correction value is set regardless of the detected subject distance. It is good also as an apparatus structure which selects.

  In the first embodiment, the case where the image processing apparatus (digital signal processing unit) is configured as hardware has been described. However, in the second embodiment of the present invention, the computer (CPU) performs image processing using software (computer program). The case will be described. FIG. 14 shows a flowchart of the image processing program.

  In step S201, the computer divides image data generated (acquired) by imaging into a plurality of chromatic aberration detection regions (correction target regions). Next, in step S202, the computer generates a distance map using the DFD method described in the first embodiment.

Next, in step S203, as described in the first embodiment with reference to FIG. 11C, the computer adjusts (corresponds to) the distance area of the distance map and the chromatic aberration detection area. Perform resolution conversion. next,
In step S204, the computer binarizes the distance map after the resolution conversion into a short-distance region that is closer than the threshold and a long-distance region that is farther than the threshold.
Specifically, the chromatic aberration detection area corresponding to the short distance area is determined as “1” as the area for detecting the chromatic aberration of magnification, and the chromatic aberration detection area corresponding to the long distance area is not detected for the chromatic aberration of magnification. The area is determined to be “0”. As a result, the determination result (binary data) shown in FIG.

  Next, in step S205, the computer switches between performing and not performing detection of the chromatic aberration of magnification according to the determination result in step S204. For the chromatic aberration detection region determined to be “1” as a result of step S204, the computer performs a detection process for detecting the amount of lateral chromatic aberration in step S206. Further, in step S207, the computer calculates the calculated correction amount (first correction value) described in the first embodiment as the use correction amount for the chromatic aberration detection area from the detected magnification chromatic aberration amount. Thereafter, the process proceeds to step S209.

  On the other hand, for the chromatic aberration detection region determined to be “0” in step S204, the computer performs the process of step S208. In step S208, a design correction amount (second correction value) is selected as a use correction amount for the chromatic aberration detection region, and an optical state at the time of imaging from a memory (storage means) (not shown) in which the design correction amount is held. A design correction amount corresponding to the image height of the chromatic aberration detection area is read out. Thereafter, the process proceeds to step S209.

  In step S209, for each color detection region, the computer corrects the chromatic aberration of magnification for the image data using the calculated correction amount calculated in step S207 or the design correction amount read in step S208 as the use correction amount.

In the present embodiment, the case where the computer mounted on the imaging apparatus executes image processing (magnification chromatic aberration correction processing) has been described. However, an image processing apparatus such as a personal computer may perform image processing on the image data acquired from the imaging apparatus according to the image processing program shown in FIG.
(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  Each embodiment described above is only a representative example, and various modifications and changes can be made to each embodiment in carrying out the present invention.

121 Optical systems 102 to 105, 109 to 120 Digital signal processing unit (image processing apparatus)
116 Chromatic Aberration Detection Unit 111 Chromatic Aberration Correction Unit 117 Correction Coefficient Acquisition Unit 117a Memory

Claims (9)

Processing means for performing a correction process on the chromatic aberration of magnification of the optical system for an input image generated by imaging through the optical system;
Detecting means for detecting a chromatic aberration of magnification of the optical system;
Calculation means for calculating a first correction value according to a detection result by the detection means as a correction value used in the correction processing;
Storage means for holding a second correction value prepared as the correction value regardless of the detection result of the detection means;
The processing means includes
Dividing the input image into a plurality of correction target areas;
An image processing apparatus that performs the correction process by selecting one of the first correction value and the second correction value for each correction target region. A distance acquisition means for acquiring information on a distance for each subject in the imaging;
The image processing apparatus according to claim 1, wherein the processing unit selects one of the first correction value and the second correction value for each correction target region according to the information regarding the distance. The distance acquisition unit generates a distance map indicating a plurality of distance regions having different distances in the input image,
The processing means performs a process of causing the plurality of correction target areas and the plurality of distance areas to correspond to each other, and then selects one of the first and second correction values for each correction target area. The image processing apparatus according to claim 2. The processing means includes
Dividing the plurality of distance regions into a first distance region in which the distance is a first distance and a second distance region in which the distance is a second distance farther than the first distance;
The first correction value is selected for the correction target area corresponding to the first distance area, and the second correction value is selected for the correction target area corresponding to the second distance area. The image processing apparatus according to claim 2, wherein the image processing apparatus is an image processing apparatus. The processing means includes
Determine the reliability of the distance,
5. The second correction value is selected for the correction target region when the reliability of the distance in the correction target region is lower than a predetermined value. 6. Image processing apparatus.   6. The method according to claim 2, wherein the processing unit selects the second correction value for a specific correction target region among the plurality of correction target regions regardless of the distance. 7. The image processing apparatus according to item.   The image processing apparatus according to claim 6, wherein the specific correction target area is a predetermined area including an optical axis of the optical system. Imaging means for performing the imaging;
An image processing apparatus comprising: the image processing apparatus according to claim 1 that performs image processing on an input image generated by the image capturing unit. A computer program for causing a computer to perform image processing including correction processing related to lateral chromatic aberration of the optical system with respect to an input image generated by imaging through an optical system,
The image processing is
A detection process for detecting the amount of chromatic aberration of magnification of the optical system is performed,
As a correction value used in the correction process, a first correction value according to a detection result by the detection process is calculated,
As the correction value, a second correction value that does not depend on the detection result of the detection process is prepared,
Dividing the input image into a plurality of correction target areas;
An image processing program that performs the correction process by selecting one of the first correction value and the second correction value for each correction target area.