JP2013127819A - Image processing apparatus and method thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve processing efficiency in development processing, and to obtain processing result intended by a user.SOLUTION: When applying development processing to a photographic image, demosaicing processing is applied to the photographic image (S102), and a display image is generated from the photographic image to which the demosaicing processing has been applied (S103). Then, an instruction indicating a reference region on the display image is input (S104), and an adjustment instruction of the development processing parameters of the photographic image is input (S106). Then, an image region having a high correlation with the reference region is extracted as a similar region (S109). Then, the development processing parameters are applied to the photographic image before the demosaicing processing, which corresponds to the reference region and the photographic image before the demosaicing processing, which corresponds to the similar region to generate the display images of the reference region and the similar region (S107, S110).

Description

  The present invention relates to image processing of captured images.

  In recent years, digital cameras have made remarkable progress, and various functional aspects such as high pixels, high sensitivity, and camera shake correction have been advanced. In addition, the price of digital single-lens reflex cameras has been reduced, and further popularization of digital cameras has been promoted.

  One advantage of a digital camera over a film camera is that the image is handled as digital data, so that it can be easily retouched after shooting. A digital camera converts light focused on an image sensor such as a CCD or CMOS into an electric signal, and saves a photographed image processed in the camera as digital data in a recording medium. A data compression format such as JPEG is generally used as the data format. In recent years, it is also possible to store digital data in the form of shooting data with a higher degree of freedom of retouching.

  The data of the captured image is data obtained by digitizing a signal output from the image sensor (hereinafter referred to as RAW data), and demosaicing / demosaicing or the like is not performed. Therefore, it cannot be displayed as a normal image as it is.

  Here, RAW data will be briefly described. Many imaging apparatuses such as digital cameras and digital video cameras obtain a subject's color information by placing a filter of a specific color in front of a photoelectric conversion device such as a CCD or CMOS. This method is called a single plate type. FIG. 8 is a diagram showing a Bayer array known as a typical color filter array used in a single-plate digital camera or digital video camera. In the case of a single-plate imaging device, signals of other colors cannot be obtained from an element having a filter of a specific color. Therefore, signals of other colors are obtained by interpolation from signals of neighboring elements. This interpolation process is called demosaicing.

  Demosaicing will be described by taking as an example the case where the color filter has the Bayer arrangement shown in FIG. First, a sensor signal mixed with RGB obtained from a photoelectric conversion device is divided into three planes, that is, R, G, and B planes. FIG. 9 is a diagram showing the result of dividing the sensor signal into planes, FIG. 9 (a) shows the R plane, FIG. 9 (b) shows the G plane, and FIG. 9 (c) shows the B plane. In each plane, substantially zero is inserted into a pixel whose value is unknown (a pixel corresponding to an element in which a filter of a color other than the color is arranged).

  FIG. 10 shows a convolution filter, and demosaicing is performed by interpolating each plane with the filter. FIG. 10A shows an R and B plane interpolation filter, and FIG. 10B shows a G plane interpolation filter.

  FIG. 11 is a diagram showing the G-plane demosaicing. FIG. 11 (a) shows the state of the G plane before demosaicing, where the center is a pixel whose value is unknown and zero is inserted. FIG. 11 (b) shows the state of the G plane after demosaicing, and the average value of the adjacent pixels in the vertical and horizontal directions is assigned to the pixel whose central value is unknown. As for the R and B planes, similarly to the G plane, pixels whose values are unknown are interpolated by the values of surrounding pixels.

  Note that there are various methods other than the above method for demosaicing, for example, not the average value of pixels adjacent vertically and horizontally, but the average value of pixels adjacent vertically and adaptively, or There is also a method of interpolating to the average value of the values of pixels adjacent to the left and right.

  The RAW data is temporarily stored in a recording medium at the time of shooting, and thereafter, image processing such as demosaicing processing is performed by software that operates on a personal computer (PC). The image after image processing can be displayed or converted into a general-purpose data format such as JPEG and stored in a recording medium. In other words, processing for using RAW data in a digital camera can be compared to shooting processing and development processing in a film camera.

  If RAW data is used, user adjustment of image processing corresponding to development (hereinafter referred to as development processing) becomes possible, and the degree of freedom of retouching is increased. In addition, since RAW data has a large number of bits per pixel and is losslessly compressed, development processing with little deterioration in image quality is possible.

  Software for performing development processing (hereinafter referred to as development software) generally includes a display function interface for displaying an image after development processing and an adjustment function interface for adjusting development processing parameters. The development processing parameters that can be adjusted by the user include edge enhancement processing parameters, blurring processing parameters, color adjustment parameters, demosaicing parameters, and the like. The user adjusts the development processing parameter while observing the displayed image, and updates the display image with the adjusted development processing parameter. The user repeats this procedure to determine development processing parameters so as to obtain a desired display image.

  Most digital cameras that can use RAW data are high-end devices such as digital single-lens reflex cameras. High end machines generally have a large amount of RAM data because of the large number of imaging pixels. For this reason, the computation load of the development processing is large, and the time until the user adjusts the development processing parameters and displays the image subjected to the image processing again becomes long. The technique of Patent Document 1 improves processing efficiency by performing image processing on an image of a partial region of a captured image and displaying the image.

  However, since the technique of Patent Document 1 determines development processing parameters with reference to a partial area of a captured image (hereinafter referred to as a reference area), the development processing after adjusting the development processing parameters is performed in an area other than the reference area. It is unclear how it will affect. In other words, when the development processing after adjusting the development processing parameters is applied to the entire image, an image intended by the user cannot be obtained outside the reference region even if an image as intended by the user is obtained in the reference region. There is a case.

JP 2004-040559 A

  An object of the present invention is to improve the processing efficiency in development processing and to obtain a processing result intended by a user.

  The present invention has the following configuration as one means for achieving the above object.

  The image processing according to the present invention generates a display image from a RAW image, inputs an instruction indicating a reference area in the display image, and an instruction of a development processing parameter for the reference area. An image area generated by extracting an image area having a high correlation with the reference area in the RAW image as a similar area, and applying the development processing parameters to the RAW image corresponding to the reference area and the RAW image corresponding to the similar area Is displayed on the display image.

  According to the present invention, the processing efficiency in the development process is improved, and the processing result intended by the user can be obtained.

1 is a block diagram illustrating a configuration example of an image processing apparatus. FIG. 3 is a diagram showing an outline of processing by a development processing application according to the first embodiment. 6 is a flowchart for explaining development processing according to the first embodiment. The flowchart explaining calculation of correlation. FIG. 6 is a diagram showing an outline of processing by a development processing application according to a second embodiment. 9 is a flowchart for explaining development processing according to the second embodiment. The figure which shows an example of a test chart. The figure which shows a payer arrangement | sequence. The figure explaining plane division | segmentation and zero insertion. The figure which shows the example of a filter used for demozaking. The figure explaining demosaicing. The figure explaining "ramen noise".

  Hereinafter, image processing according to an embodiment of the present invention will be described in detail with reference to the drawings.

[Device configuration]
FIG. 1 is a block diagram illustrating a configuration example of an image processing apparatus.

  The CPU 101 executes an operating system (OS) and various programs stored in the ROM of the main memory 102 and the hard disk drive (HDD) 105 using the RAM of the main memory 102 as a work memory. Each component is controlled via a system bus 1114 such as a PCI (peripheral component interconnect) bus. Further, various programs including a development processing application described later are executed.

  The CPU 101 accesses the HDD 105 via the system bus 1114 and the serial ATA interface (SATA I / F) 103. Further, a network 1113 such as a local area network (LAN) is accessed via the network I / F 104.

  In the following description, it is assumed that the development processing application, image data, and the like are read from the HDD 105, but can be read from a server on the network 1113.

  Further, the CPU 101 displays a user interface and processing results of processing to be described later on the monitor 107 via the graphic accelerator 106, and inputs user instructions via the keyboard 1111 and mouse 1112 connected to the keyboard / mouse controller 110. .

  Further, the CPU 101 outputs the image data to the printer 109 via a USB (Universal Serial Bus) controller 108, and prints an image instructed by the user, for example.

[Image processing]
FIG. 2 is a diagram showing an outline of processing by the development processing application of the first embodiment. Development processing is roughly divided into processing 1, processing 2, and processing 3.

  Although details will be described later, in processing 1, the RAW data 114 of the reference area 121 when adjusting the development processing parameters specified on the display image 113 is acquired. Next, in process 2, a correlation is calculated between the reference area 121 and another area. Then, in process 3, an image area having a high correlation is displayed.

  FIG. 3 is a flowchart for explaining the developing process according to the first embodiment, which is a process executed by the CPU 101.

  The CPU 101 acquires the RAW data 111 to be developed (S101), and generates an image 112 obtained by demosaicing the RAW data 111 (S102). For example, a 3CCD or a single-panel digital camera that sequentially captures RGB images need not be demosaiced, and the processing in step S102 may be omitted.

  Next, the CPU 101 performs image processing 130 on the demosaiced image 112 to generate a display image 113 and displays it on the monitor 107 (S103). Then, a user instruction indicating the reference area 121 on the display image 113 is acquired (S104), and reference RAW data 114 corresponding to the reference area 121 is acquired from the RAW data 111 (S105).

  Next, the CPU 101 inputs a user adjustment instruction for the development processing parameter (S106), and develops the reference RAW data 114 using the adjusted development processing parameter. Then, the image of the reference area 121 after the development processing is displayed on the display image 113 (S107). At this time, it is preferable to display the reference area 121 so that the user can easily recognize it, for example, by displaying a frame in the reference area 121.

  That is, steps S101 to S107 correspond to process 1.

  Next, the CPU 101 sets a region other than the reference region 121 as the correlation detection target region 122, and acquires the RAW data 115 corresponding to the correlation detection target region 122 from the RAW data 111 (S108). Then, the correlation between the RAW data 115 and the reference RAW data 114 is calculated in each area other than the reference area 121 (S109).

  That is, steps S108 and S109 correspond to process 2.

  Next, the CPU 101 uses the development processing parameters adjusted in the reference area 121 to develop RAW data in an area 116 (hereinafter referred to as a similar area) having a high correlation (similarity) with the reference area 121. Then, the image of the similar region 116 after the development processing is displayed on the display image 113 (S110). At this time, it is preferable to display the similar region 116 so that the user can easily recognize the similar region 116, such as displaying a frame in the similar region 116.

  That is, step S110 corresponds to process 3.

  Next, the CPU 101 determines whether or not the development process parameter adjustment has been completed, for example, when the user has pressed the end button to instruct the end (S111). If not completed, the process returns to step S104. .

[Calculation of correlation]
FIG. 4 is a flowchart for explaining the calculation of correlation (S109).

  The CPU 101 determines an area for calculating the correlation (hereinafter referred to as a comparison area) (S201). Although it is desirable to calculate the correlation over the entire area of the RAW data 115 to be detected for correlation, the correlation may be calculated at a predetermined pixel interval. Or you may limit the area | region which calculates a correlation as needed, such as a user specifying the area | region which calculates a correlation. The size of the area for calculating the correlation is the same as that of the reference area 121.

  Next, the CPU 101 extracts the RAW data in the comparison area from the RAW data 115 corresponding to the correlation detection area, and sets it as comparison RAW data (S202). For example, when the reference area 121 is 5 × 5 pixels, the comparison area is also 5 × 5 pixels.

Next, the CPU 101 calculates the correlation between the reference RAW data 114 and the comparative RAW data (S203). The calculation of the correlation is performed by equation (1).
Corr = Σ _i Σ _j Iref (i, j) Icomp (i, j)… (1)
Where Corr is the correlation value,
i is an index representing the pixel position in the vertical direction,
j is an index indicating the pixel position in the horizontal direction,
Iref is the value of pixel (i, j) of the reference RAW data,
Icomp is the pixel (i, j) value of the comparative RAW data.

Equation (2) or other calculation methods may be used for calculating the correlation. Further, when the imaging device is not a single-plate type, for example, a method of recording Bayer array RGB data as RAW data, the correlation may be calculated only for pixels corresponding to G.
Corr = Σ _i Σ _j {Iref (i, j)-Iref _ave } {Icomp (i, j)-Icomp _ave }… (2)
Where Corr is the correlation value,
Iref _ave is the average value of the reference RAW data,
Icomp _ave is the average value of comparative RAW data.

  Next, the CPU 101 determines whether or not the correlation of the area for which the correlation is to be calculated has been calculated (S204), and repeats the processing from step S201 to S203 until there is no area for which the correlation calculation has not been completed.

  When the calculation of the correlation is completed, the CPU 101 determines the region where the correlation value exceeds the predetermined value as the similar region 116 (S205). Note that the similar area 116 is not necessarily the same size as the reference area 121. That is, if adjacent comparison regions are both similar regions, the region obtained by combining them is set as the similar region 116.

  Although not shown in FIG. 2, the CPU 101 displays an image of the reference area 121 and an image of the similar area 116 that have been developed using the adjusted development processing parameters on the display image 113. Therefore, the user can know whether or not the adjusted development processing parameter is appropriate for the reference area 121. In addition, it is possible to visually evaluate the similar region 116 having a high correlation with the reference region 112 and how the development processing parameter affects (or does not affect) the image in the similar region 116. At that time, since the CPU 101 does not perform image processing on an area having a low correlation with the reference area 112, the processing efficiency is higher than when processing the RAW data 111 (entire image) using the development processing parameter. Can be improved.

  For example, when edge enhancement is performed on an image region having a high spatial frequency such as a human hair (reference region), there is little need to confirm the processing effect of an image region having a relatively low spatial frequency such as an overhead sky (dissimilar region). By presenting the reference area 121 and the similar area 116 and displaying the development processing result of the area, the processing load on the CPU 101 can be reduced.

  The merit of detecting the similar region 116 based on comparison of RAW data, not comparison of developed images, will be described. Therefore, first, image quality deterioration due to development processing will be described.

  A pattern that does not exist in an actual subject may occur as image quality degradation due to development processing. Moire is one of the image quality degradations caused by development processing. In addition to moire, image quality degradation called “ramen noise” may occur depending on the development processing method. `` Ramen noise '' means that when a subject with dense lines as shown in Fig. 12 (a) is photographed, the fine lines are spiraled at the pixel level as shown in Fig. 12 (b) in the developed image. It is connected image quality degradation.

  Such image quality degradation due to development processing is difficult to detect from an image after development. This is because, when an image after development is observed, it cannot be distinguished whether the subject has a pattern similar to “ramen noise” or whether “ramen noise” has occurred due to the development process. If the similar region 116 is detected from the RAW data instead of detecting the similar region 116 from the developed image, the above problem is solved. That is, if the RAW data is similar at the time, the result of the development processing is also similar.

  The image processing according to the second embodiment of the present invention will be described below. Note that the same reference numerals in the second embodiment denote the same parts as in the first embodiment, and a detailed description thereof will be omitted.

  In the first embodiment, the example in which the development processing parameter is adjusted for one captured image has been described. In the second embodiment, a method for adjusting development processing parameters common to a plurality of captured images will be described. When using development processing parameters common to a plurality of captured images, the similar region is not limited to one captured image, but is included in other captured images.

  FIG. 5 is a diagram showing an outline of processing by the development processing application of the second embodiment. The development processing is roughly divided into processing 1, processing 2, and processing 3.

  Processes 1 and 3 are the same as those in the first embodiment. In the process 2, a correlation is calculated between a captured image 132 that is included in a plurality of captured images to be developed and includes a reference region 121 (hereinafter referred to as a reference image).

  FIG. 6 is a flowchart for explaining the developing process according to the second embodiment, which is a process executed by the CPU 101.

  Process 1 from steps S101 to S107 is the same as the process of the first embodiment shown in FIG.

  Next, the CPU 101 selects a captured image (hereinafter referred to as a comparative image) that is included in the plurality of captured images to be developed and is different from the reference image, and acquires the RAW data 131 of the comparative image (S301). Then, the correlation between the RAW data 131 and the reference RAW data 114 is calculated in each area of the comparison image (S302). The comparison image may be selected based on, for example, user input.

  That is, steps S301 and S302 correspond to process 2.

  Next, the CPU 101 develops the RAW data in the similar area 116 using the development processing parameters adjusted in the reference area 121. Then, the image of the similar region 116 after the development processing is displayed on the display image 132 displayed in another window (S303). At this time, it is preferable to display the similar region 116 so that the user can easily recognize the similar region 116, such as displaying a frame in the similar region 116. In addition, when there are a plurality of comparison images, display images 132 indicating the similar regions 116 are displayed in a window corresponding to the number of comparison images, or the user can operate the scroll bar or the like to observe the plurality of comparison images. That's fine.

  That is, step S303 corresponds to process 3.

  Next, the CPU 101 determines whether or not the development process parameter adjustment has been completed, for example, when the user has pressed the end button to instruct the end (S111). If not completed, the process returns to step S104. .

  Although a detailed description is omitted, it goes without saying that the development processing application also executes processing of similar regions in the same image of the first embodiment.

[Modification]
In the second embodiment, for example, a comparative image designated by the user is acquired, and the development processing parameter adjusted in the reference image is applied to the similar region 116 having a high correlation with the reference region 121 of the comparative image. Here, the comparison image may be RAW data designated in advance, such as a test chart shown in FIG. The test chart shown in FIG. 7 is an example of an image having a low spatial frequency at the center of the image and a high spatial frequency at the periphery of the image. There are test charts in which patterns of various spatial frequencies are recorded, and the characteristics of image processing can be grasped at a glance by taking and developing test charts.

  By using such a test chart, the user can know a spatial frequency region similar to the reference region 121 from the test chart when adjusting the development processing parameter. In other words, general-purpose development processing parameters can be obtained by adjusting development processing parameters while referring to the similar region 116 in the test chart. That is, since it is possible to grasp how the area in the normal photographed image such as a landscape photograph or a portrait corresponds to the area of the test chart, general-purpose development processing parameters can be adjusted efficiently.

[Other embodiments]
Note that the present invention can be applied to a system composed of a plurality of devices (for example, a computer, an interface device, a reader, a printer, etc.), or a device (for example, a copier, a facsimile machine, a control device) composed of a single device. Etc.).

  Another object of the present invention is to supply a recording medium or a recording medium recording a computer program for realizing the functions of the above embodiments to a system or apparatus. This can also be achieved by the computer (CPU or MPU) of the system or apparatus executing the computer program. In this case, the software read from the recording medium itself realizes the functions of the above embodiments, and the computer program and the computer-readable recording medium storing the computer program constitute the present invention. .

  Further, the above functions are not only realized by the execution of the computer program. That is, according to the instruction of the computer program, the operating system (OS) and / or the first, second, third,... This includes the case where the above function is realized.

  The computer program may be written in a memory of a device such as a function expansion card or unit connected to the computer. That is, it includes the case where the CPU of the first, second, third,... Device performs part or all of the actual processing according to the instructions of the computer program, thereby realizing the above functions.

  When the present invention is applied to the recording medium, the recording medium stores a computer program corresponding to or related to the flowchart described above.

Claims (16)

Image processing means for generating a display image from a RAW image;
An input means for inputting an instruction indicating a reference area in the display image, and an instruction of a development processing parameter for the reference area;
Extraction means for extracting an image area having a high correlation with the reference area in the RAW image as a similar area from the RAW image,
An image processing apparatus comprising: a display unit configured to display an image generated by applying the development processing parameter to a RAW image corresponding to the reference area and a RAW image corresponding to the similar area on the display image.   2. The image processing apparatus according to claim 1, wherein the extraction unit takes the correlation only for pixels corresponding to a G plane of the RAW image. Image processing means for generating a display image from a RAW image;
An input means for inputting an instruction indicating a reference area in the display image, and an instruction of a development processing parameter for the reference area;
Extraction means for extracting an image area having a high correlation with the reference area in the RAW image as a similar area from a pre-designated image;
An image processing apparatus comprising: a display unit configured to display an image generated by applying the development processing parameter to a RAW image corresponding to the reference area and a RAW image corresponding to the similar area on the display image.   4. The image processing apparatus according to claim 3, wherein the predesignated image is a test chart having a lower spatial frequency at a central portion of the image and a higher spatial frequency at a peripheral portion of the image. Image processing means for generating a display image from a RAW image;
An input means for inputting an instruction indicating a reference area in a display image generated from a reference image which is a RAW image, and an instruction of a development processing parameter for the reference area;
Extraction means for extracting, as a similar region, an image region having a high correlation with the reference region in the reference image that is the RAW image, from a comparison image that is a RAW image different from the reference image,
An image generated by applying the development processing parameter to the RAW image corresponding to the reference area is displayed on the display image of the reference image, and the development processing parameter is applied to the RAW image corresponding to the similar area. An image processing apparatus comprising: display means for displaying the generated image on the display image of the comparison image.   6. The image processing apparatus according to claim 1, wherein the display unit displays a frame in the reference area.   7. The image processing apparatus according to claim 1, wherein the display unit displays a frame in the similar region.   8. The image processing apparatus according to claim 1, wherein the image processing unit generates the display image by demosaicing the RAW image.   9. The image processing apparatus according to claim 1, wherein the display unit does not apply the development processing parameter to a RAW image excluding the reference area and the similar area.   10. The development processing parameter includes at least one of an edge enhancement processing parameter, a blurring processing parameter, a color adjustment parameter, and a parameter related to the demosaicing. The described image processing apparatus.   11. The image processing apparatus according to claim 1, wherein the extraction unit combines adjacent similar regions. Generate display image from RAW image,
Input an instruction indicating a reference area in the display image, and an instruction of a development processing parameter for the reference area,
From the RAW image, an image area having a high correlation with the reference area in the RAW image is extracted as a similar area,
An image processing method, wherein an image generated by applying the development processing parameter to a RAW image corresponding to the reference area and a RAW image corresponding to the similar area is displayed on the display image. Generate display image from RAW image,
Input an instruction indicating a reference area in the display image, and an instruction of a development processing parameter for the reference area,
From the image specified in advance, an image area having a high correlation with the reference area in the RAW image is extracted as a similar area,
An image processing method for displaying an image generated by applying the development processing parameter to a RAW image corresponding to the reference area and a RAW image corresponding to the similar area on the display image. Generate display image from RAW image,
An instruction indicating a reference area in a display image generated from a reference image which is a RAW image, and an instruction of a development processing parameter for the reference area are input.
From a comparison image that is a RAW image different from the reference image, an image region having a high correlation with the reference region in the reference image that is the RAW image is extracted as a similar region,
An image generated by applying the development processing parameter to the RAW image corresponding to the reference area is displayed on the display image of the reference image, and the development processing parameter is applied to the RAW image corresponding to the similar area. An image processing method for displaying a generated image on a display image of the comparison image.   12. A program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 11.   16. A computer-readable recording medium in which the program according to claim 15 is stored.

Images