JP2018022324A - Image processing apparatus, image processing method and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus which can correct a portion of whiteout in a digital image.SOLUTION: A neighboring pixel group extraction unit 1 extracts a neighboring pixel group formed of the target pixel and peripheral pixels surrounding the target pixel in a digital image. A luminance value calculation unit 2 calculates the luminance value of each pixel in the neighboring pixel group. An average luminance value calculation unit 3 calculates an average luminance value Yav of the neighboring pixel group. A luminance difference cumulative value calculation unit 4 calculates a luminance difference cumulative value Dyac of the difference between the luminance value of the target pixel and the luminance value of each peripheral pixel. A multiplier coefficient generation unit 5 generates a multiplier coefficient k whose value is less than 1 in a case where it is determined that the target pixel is in the whiteout state on the basis of the average luminance value Yav and luminance difference cumulative value Dyac and a specific condition that the target pixel is the pixel located on the upper part from the upper end of the frame to the position in the prescribed vertical direction is satisfied. An RGB correction unit 6 multiplies the multiplier coefficient k by R and G data in the target pixel.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program that can correct a portion that is overexposed in an image.

  In recent years, HDR (High Dynamic Range) image standards have been developed, and the content of HDR images is increasing. Since an HDR image has a wide dynamic range, a bright part is rarely over-exposed. However, general SDR (Standard Dynamic Range) image contents are still mainstream. An SDR image has a narrow dynamic range, and is generally photographed by assigning a large number of gradations to a person or the like, and a bright part may be in a state of overexposure.

JP 2007-133582 A

  In many cases, the content of an SDR image has a bright portion in a whiteout state when an image is captured or when content is created based on the captured image. It is easy to reduce overexposure by adjusting the exposure at the time of photographing with the imaging device. However, it is not easy to reduce the whiteout of the content of the digital image once created.

  An object of the present invention is to provide an image processing apparatus, an image processing method, and an image processing program capable of correcting a portion that is overexposed in a digital image.

  The present invention provides a neighboring pixel group extraction unit that extracts a neighboring pixel group including a pixel of interest in a digital image composed of RGB data of three primary colors and a surrounding pixel surrounding the pixel of interest, and each pixel in the neighboring pixel group. Based on the luminance value calculation unit that calculates the luminance value and the luminance value of each pixel calculated by the luminance value calculation unit, an average luminance value that is an average value of the luminance values of the pixels in the neighboring pixel group is calculated. Based on the average luminance value calculation unit and the luminance value of each pixel calculated by the luminance value calculation unit, the absolute value of the difference between the luminance value of the target pixel and the luminance value of each of the surrounding pixels is accumulated. A luminance difference cumulative value calculating unit for calculating a luminance difference cumulative value; and, based on the average luminance value and the luminance difference cumulative value, the target pixel is determined to be in a whiteout state; When the pixel of interest satisfies a specific condition that is a pixel located at an upper portion from the upper end of the frame of the digital image to a predetermined vertical position, a multiplication coefficient having a value less than 1 is generated, and the specific condition is satisfied A multiplication coefficient generation unit that generates a multiplication coefficient having a value of 1 when there is not, and an RGB correction unit that multiplies the R and G data in the pixel of interest by the multiplication coefficient generated by the multiplication coefficient generation unit, An image processing apparatus is provided.

  The present invention extracts a neighboring pixel group consisting of a pixel of interest in a digital image composed of RGB data of three primary colors and surrounding pixels surrounding the pixel of interest, calculates a luminance value of each pixel in the neighboring pixel group, Based on the calculated luminance value of each pixel, an average luminance value, which is an average value of the luminance values of the pixels in the neighboring pixel group, is calculated, and based on the calculated luminance value of each pixel, the target pixel The absolute value of the difference between the luminance value of each of the surrounding pixels and the luminance value of each of the surrounding pixels is accumulated to calculate a luminance difference accumulated value. Based on the average luminance value and the luminance difference accumulated value, the target pixel is overexposed. And whether or not the pixel of interest satisfies a specific condition that is a pixel located at an upper portion from an upper end of a frame of the digital image to a predetermined vertical position, When a certain condition is satisfied, a multiplication coefficient having a value of less than 1 is generated. When the specific condition is not satisfied, a multiplication coefficient having a value of 1 is generated, and the multiplication coefficient is multiplied by the R and G data in the target pixel. An image processing method is provided.

  The present invention includes a step of extracting, in a computer, a neighboring pixel group including a pixel of interest in a digital image composed of RGB data of three primary colors and peripheral pixels surrounding the pixel of interest, and luminance of each pixel in the neighborhood pixel group A step of calculating a value, a step of calculating an average luminance value that is an average value of the luminance values of the pixels in the neighboring pixel group based on the calculated luminance value of each pixel, and a step of calculating each of the calculated pixels. Based on the luminance value, a step of calculating an absolute value of a difference between the luminance value of the target pixel and the luminance value of each of the surrounding pixels to calculate a luminance difference cumulative value; and the average luminance value and the luminance difference cumulative value And the pixel of interest is in a state of whiteout, and the pixel of interest is from the upper end of the frame of the digital image to a predetermined vertical position. A step of determining whether or not a specific condition that is a pixel located at an upper part is satisfied; a multiplication coefficient having a value of less than 1 when the specific condition is satisfied; and a value of 1 when the specific condition is not satisfied And a step of multiplying the multiplication coefficient by the R and G data in the pixel of interest.

  According to the image processing apparatus, the image processing method, and the image processing program of the present invention, it is possible to correct an overexposed portion in a digital image.

It is a block diagram which shows the image processing apparatus of one Embodiment. It is a block diagram which shows the specific structural example of the neighboring pixel group extraction part 1 in FIG. It is a figure which shows an example of a neighborhood pixel group. It is a figure which shows an example of the digital image which correct | amends a whiteout with the image processing apparatus of one Embodiment, an image processing method, and an image processing program. It is a figure which shows an example of the multiplication coefficient which the multiplication coefficient production | generation part 5 in FIG. 1 produces | generates. It is a flowchart which shows the procedure by the image processing method of one Embodiment, and the process performed with the image processing program of one Embodiment.

  Hereinafter, an image processing apparatus, an image processing method, and an image processing program according to an embodiment will be described with reference to the accompanying drawings.

  In FIG. 1, a pixel P of a digital image composed of RGB data of three primary colors is input to the neighboring pixel group extraction unit 1. Specifically, the pixel values and pixel coordinates of R, G, and B of the pixel P are input to the neighboring pixel group extraction unit 1. The pixel values of R, G, and B are 8 bits, for example, and have values from 0 to 255.

  The neighboring pixel group extraction unit 1 sequentially includes the pixels P in order from the leftmost pixel P of one line constituting the frame of the digital image to the rightmost pixel P, and from the uppermost line to the lowermost line. Entered.

  As illustrated in FIG. 2, the neighboring pixel group extraction unit 1 includes line memories 11 and 12 that are one-line delay devices and D flip-flops (DFF) 13 to 18 that are one-pixel delay devices. The line memory 11 delays the input pixel P by one line, and the line memory 12 delays the output of the line memory 11 by one line.

  The DFF 13 delays the input pixel P by one pixel, and the DFF 14 delays the output of the DFF 13 by one pixel. The DFF 15 delays the output of the line memory 11 by one pixel, and the DFF 16 delays the output of the DFF 15 by one pixel. The DFF 17 delays the output of the line memory 12 by one pixel, and the DFF 18 delays the output of the DFF 17 by one pixel.

  The neighboring pixel group extraction unit 1 extracts a neighboring pixel group including nine pixels P of three pixels in the horizontal direction and three pixels in the vertical direction shown in FIG. 3 with the configuration shown in FIG. A hatched pixel P located in the center in FIG. 3 is defined as a target pixel P (i, j). The target pixel P (i, j) and the surrounding pixels P (i−1, j−1), P (i, j−1), P (i + 1, j−1) surrounding the target pixel P (i, j), P (i−1, j), P (i + 1, j), P (i−1, j + 1), P (i, j + 1), and P (i + 1, j + 1) as a whole are neighboring pixel groups.

  The neighboring pixel group extraction unit 1 may use 25 pixels P of 5 pixels in the horizontal direction and 5 pixels in the vertical direction as a neighboring pixel group. The area of the neighboring pixel group extracted by the neighboring pixel group extraction unit 1 is not particularly limited, but may be 9 square or 25 pixels in a square shape.

  If the pixel P located at the corner of one frame is the target pixel P (i, j) and there is no part of the peripheral pixel, the non-existing peripheral pixel may be interpolated by a known method.

  Returning to FIG. 1, the neighborhood pixel group extracted by the neighborhood pixel group extraction unit 1 is input to the luminance value calculation unit 2. The luminance value calculation unit 2 calculates the luminance value Y of each pixel P of each neighboring pixel group using a conversion equation of the BT.709 standard expressed by the following equation (1).

  Y = 0.2126 × R + 0.7152 × G + 0.0722 × B (1)

  The luminance value Y of the pixel P of each neighboring pixel group is input to the average luminance value calculation unit 3 and the luminance difference cumulative value calculation unit 4. The average luminance value calculation unit 3 calculates an average luminance value Yav that is an average value of the luminance values Y of the pixels P in the neighboring pixel group based on the inputted luminance values Y of the pixels P of the neighboring pixel group.

  The luminance difference cumulative value calculation unit 4 uses the following formula (2) based on the input luminance value Y of each pixel P of the neighboring pixel group to calculate the luminance value Y of the target pixel P (i, j) and the surrounding area. The absolute value of the difference from the luminance value Y of each pixel is accumulated to calculate the luminance difference accumulated value Dyac. In equation (2), Y (i, j) represents the luminance value Y of the target pixel P (i, j), and Y (i + p, j + q) represents the luminance value Y of the surrounding pixels.

  The average luminance value Yav and the luminance difference cumulative value Dyac are input to the multiplication coefficient generation unit 5. The multiplication coefficient generator 5 receives the pixel coordinates of the target pixel P (i, j). Only the number of lines (that is, the value of j) indicating the vertical position of the pixel of interest P (i, j) may be input to the multiplication coefficient generation unit 5.

  The multiplication coefficient generator 5 generates the multiplication coefficient k as follows. As an example, assume that one frame of a digital image is in a state as shown in FIG. If the digital image is a full HD image, one frame has 1080 lines. As shown in FIG. 4, the area from the upper end of the frame to a predetermined vertical position is often empty, and the empty part may be in a state of being overexposed.

  If the number of lines in one frame is H, it can be assumed that the H / 4 region at the top of the frame is empty. It may be set as appropriate to determine how much of the vertical region of one frame is assumed to be empty, and is not limited to H / 4 at the top of the frame.

  If the brightness is very high, the image is flat, and no pattern is present, it can be regarded as a whiteout state. Therefore, as an example, when the average luminance value Yav exceeds 220 and the luminance difference cumulative value Dyac is less than 60, the multiplication coefficient generation unit 5 determines that the image is in a whiteout state.

  The multiplication coefficient generation unit 5 is in a whiteout state where the value of j of the pixel of interest P (i, j) is less than H / 4, the average luminance value Yav exceeds 220, and the luminance difference cumulative value Dyac is less than 60. When a specific condition is satisfied, a multiplication coefficient k having a value less than 1 is generated.

  It is preferable that the multiplication coefficient generation unit 5 generates a multiplication coefficient k that is the smallest at the upper end of the frame and increases in order from the upper end toward a predetermined vertical position (H / 4 position). As an example, the multiplication coefficient generation unit 5 generates a multiplication coefficient k based on Expression (3).

  k = 0.7 + 0.3 × j / (H / 4) (3)

  The multiplication coefficient generation unit 5 sets a value when the specific condition is not satisfied, that is, when the value of j is H / 4 or more, or even if the value of j is less than H / 4, it is not in a whiteout state. Produces a multiplication factor k of 1.

  Assuming that the H / 4 region at the top of the frame is in a state of whiteout, the multiplication coefficient generator 5 generates a multiplication coefficient k as shown in FIG. The multiplication coefficient generation unit 5 generates a multiplication coefficient k in which the value of j increases linearly from 0 to 269, using Expression (3), and generates a multiplication coefficient whose value is 1 after 270.

  According to Equation (3), the multiplication coefficient k changes linearly, but the multiplication coefficient k may have a characteristic that changes nonlinearly.

  Returning to FIG. 1, the multiplication coefficient k is input to the RGB correction unit 6. The target pixel P (i, j) is input to the RGB correction unit 6 from the neighboring pixel group extraction unit 1. The RGB correction unit 6 multiplies the multiplication coefficient k by the R and G data of the pixel of interest P (i, j) and outputs a correction pixel P ′.

  If the RGB correction unit 6 multiplies the R and G data of the pixel of interest P (i, j) by a multiplication coefficient k having a value less than 1, the whiteout state at the pixel of interest P (i, j) is corrected. If the RGB correction unit 6 multiplies the R and G data of the target pixel P (i, j) by the multiplication coefficient k having a value of 1, the target pixel P (i, j) is not corrected and the original state is maintained. Will be.

  If the digital image is a moving image, the image processing apparatus shown in FIG. 1 repeats the above operation for each frame. Output a frame with corrected state.

  With the above configuration, according to the image processing apparatus of the present embodiment, it is possible to correct an overexposed portion at the top of a frame in a digital image. According to the image processing apparatus of the present embodiment, there is no influence on the 3H / 4 region other than the H / 4 region at the top of the frame.

  In the case of the digital image shown in FIG. 4, not only the sky but also a mountain exists in the H / 4 region at the top of the frame. The image processing apparatus according to the present embodiment corrects only a blank portion that is overexposed by the above configuration and operation. According to the image processing apparatus of the present embodiment, when the sky portion at the top of the frame is in a whiteout state, the sky color is corrected so that the bluish color increases, and the original sky color can be reproduced. it can.

  If the multiplication coefficient generator 5 generates the multiplication coefficient k using Expression (3), the degree of correction is increased as it approaches the upper end of the frame, and the degree of correction is decreased as it approaches a predetermined vertical position. Can do. Therefore, it is possible to avoid an unnatural change in color at the boundary between the H / 4 region at the top of the frame and the remaining 3H / 4 region.

  The image processing apparatus illustrated in FIG. 1 may be configured by hardware, may be configured by software, or may be configured by combining hardware and software.

  The image processing apparatus shown in FIG. 1 may be composed of a central processing unit (CPU) and a memory of a microcomputer, or an integrated circuit. The image processing apparatus shown in FIG. 1 may be functionally realized by a computer program (image processing program) stored in the memory being executed by the CPU.

  Processing executed by the image processing program according to the embodiment will be described with reference to FIG. FIG. 6 also shows a procedure according to the image processing method of the embodiment.

  In FIG. 6, when the image processing program is executed and processing is started, the CPU captures a pixel P (pixel value and pixel coordinate) of a digital image composed of RGB data of three primary colors in step S1. In step S2, the CPU extracts a neighboring pixel group including a pixel of interest P (i, j) and peripheral pixels surrounding the pixel of interest P (i, j).

  In step S3, the CPU calculates the luminance value Y of each pixel in the neighboring pixel group. In step S4, the CPU calculates the average luminance value Yav of the neighboring pixel group based on the calculated luminance value Y of each pixel. In step S5, the CPU calculates the luminance difference cumulative value Dyac of the neighboring pixel group based on the calculated luminance value Y of each pixel. The order of step S4 and step S5 may be reversed or may be executed simultaneously.

In step S6, the CPU is in a state in which the target pixel P (i, j) is overexposed,
It is determined whether or not the pixel of interest P (i, j) satisfies a specific condition that is a pixel located at the upper part from the upper end of the frame to a predetermined vertical position. If the specific condition is satisfied (YES), the CPU generates a multiplication coefficient k having a value less than 1 using Equation (3) in step S7. If the specific condition is not satisfied (NO), the CPU generates a multiplication coefficient k having a value of 1 in step S8.

  In step S9, the CPU multiplies the R and G data of the pixel of interest P (i, j) by the multiplication coefficient k. If the R and G data are multiplied by the multiplication coefficient k generated in step S7, the whiteout state at the target pixel P (i, j) is corrected. If the multiplication coefficient k generated in step S8 is multiplied by the R and G data, the pixel of interest P (i, j) is not corrected and the original state is maintained. In step S10, the CPU determines whether or not the processing in steps S1 to S9 has been executed for all the pixels in the frame.

  If the process has not been executed for all the pixels in the frame (NO), the CPU repeats the processes of steps S1 to S9. If the process is executed for all the pixels in the frame (YES), the CPU ends the process. If the digital image is a moving image, the CPU repeats the processes of steps S1 to S10 for each frame.

  According to the image processing method and the image processing program of the present embodiment, the same effects as those obtained by the image processing apparatus of the present embodiment described above can be obtained. The image processing program may be stored in a non-transitory storage medium, or distributed via a communication line such as the Internet.

  The present invention is not limited to the embodiment described above, and various modifications can be made without departing from the scope of the present invention. The digital image to be processed by the image processing apparatus, the image processing method, and the image processing program according to the present embodiment is not limited to a full HD image, and may be a standard resolution SD image or 4K. It may be a higher resolution image than full HD. In addition, although the SDR image in which whiteout is likely to occur is taken as an example of the digital image to be processed, it is not limited to the SDR image.

DESCRIPTION OF SYMBOLS 1 Neighborhood pixel group extraction part 2 Luminance value calculation part 3 Average luminance value calculation part 4 Luminance difference cumulative value calculation part 5 Multiplication coefficient generation part 6 RGB correction part

Claims (4)

A neighboring pixel group extraction unit that extracts a neighboring pixel group including a pixel of interest in a digital image composed of RGB data of three primary colors and a surrounding pixel surrounding the pixel of interest;
A luminance value calculation unit for calculating the luminance value of each pixel in the neighboring pixel group;
An average luminance value calculating unit that calculates an average luminance value that is an average value of the luminance values of the pixels in the neighboring pixel group based on the luminance value of each pixel calculated by the luminance value calculating unit;
Based on the luminance value of each pixel calculated by the luminance value calculation unit, the absolute value of the difference between the luminance value of the target pixel and the luminance value of each of the surrounding pixels is accumulated to calculate a luminance difference accumulated value. A luminance difference cumulative value calculation unit;
Based on the average luminance value and the accumulated luminance difference value, it is determined that the target pixel is in a whiteout state, and the target pixel is located from the upper end of the frame of the digital image to a predetermined vertical position. A multiplication coefficient generating unit that generates a multiplication coefficient having a value of less than 1 when a specific condition that is a pixel located at the top is satisfied, and that generates a multiplication coefficient having a value of 1 when the specific condition is not satisfied;
An RGB correction unit that multiplies the R and G data in the pixel of interest by the multiplication coefficient generated by the multiplication coefficient generation unit;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the multiplication coefficient generation unit generates a multiplication coefficient that is smallest at the upper end and increases in order from the upper end toward the predetermined vertical position. . Extracting a neighboring pixel group consisting of a pixel of interest in a digital image composed of RGB data of three primary colors and peripheral pixels surrounding the pixel of interest;
Calculating a luminance value of each pixel in the neighboring pixel group;
Based on the calculated luminance value of each pixel, an average luminance value that is an average value of the luminance values of the pixels in the neighboring pixel group is calculated,
Based on the calculated luminance value of each pixel, the absolute value of the difference between the luminance value of the target pixel and the luminance value of each of the surrounding pixels is accumulated to calculate a luminance difference cumulative value,
Based on the average luminance value and the accumulated luminance difference value, the pixel of interest is in a whiteout state, and the pixel of interest is located at an upper portion from the upper end of the frame of the digital image to a predetermined vertical position. Determine whether or not a specific condition is satisfied,
Generating a multiplication coefficient having a value less than 1 when the specific condition is satisfied;
Generating a multiplication coefficient having a value of 1 when the specific condition is not satisfied;
An image processing method, wherein the multiplication coefficient is multiplied by R and G data in the pixel of interest. On the computer,
Extracting a neighboring pixel group consisting of a pixel of interest in a digital image composed of RGB data of three primary colors and surrounding pixels surrounding the pixel of interest;
Calculating a luminance value of each pixel in the neighboring pixel group;
Calculating an average luminance value that is an average value of the luminance values of the pixels in the neighboring pixel group based on the calculated luminance value of each pixel;
Based on the calculated luminance value of each pixel, the absolute value of the difference between the luminance value of the pixel of interest and the luminance value of each of the surrounding pixels is accumulated to calculate a luminance difference accumulated value;
Based on the average luminance value and the accumulated luminance difference value, the pixel of interest is in a whiteout state, and the pixel of interest is located at an upper portion from the upper end of the frame of the digital image to a predetermined vertical position. Determining whether or not a specific condition that is a pixel to be satisfied is satisfied;
Generating a multiplication coefficient having a value of less than 1 when the specific condition is satisfied, and generating a multiplication coefficient having a value of 1 when the specific condition is not satisfied;
Multiplying R and G data in the pixel of interest by the multiplication factor;
An image processing program for executing

Images