JP2017204787A - Image processing apparatus, control method thereof, imaging apparatus, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus which can execute detection processing on a plurality of images with different points of view.SOLUTION: An image processing apparatus includes: a first image input unit 101 for acquiring a plurality of first images with different points of view; and a second image input unit 104 for acquiring a second image with single point of view. The first image detection unit 102 executes detection processing on the input first images. The detection processing on the acquired images can be set individually in accordance with detection items. The first image detection unit 102 determines whether the detection processing is to be executed on an image before synthesis or to be executed on an image after synthesis. A synthesizing unit 103 generates a synthesized image from the input first images. A second image detection unit 105 acquires the synthesized image or the second image, for detection processing. An image processing unit 106 generates an output image from the image synthesized by the synthesizing unit 103 or the second image, in accordance with a detection result.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing technique using a parallax image.

  An application example of image processing using parallax images is refocus processing. The parallax images are a plurality of images obtained by photographing the same subject from a plurality of viewpoints. For example, by devising the arrangement of lenses, microlens arrays, imaging elements, etc., an imaging apparatus having a configuration including a plurality of optical systems as a single piece of equipment is used. The imaging device disclosed in Patent Literature 1 can simultaneously obtain a parallax image by dividing the exit pupil of the photographing lens into a plurality of regions. One microlens and a plurality of divided photoelectric conversion units are formed, and light passing through different pupil partial areas of the exit pupil of the photographing lens is received through each microlens. A parallax image corresponding to the divided pupil partial region is generated from each light reception signal. Patent Document 2 discloses that a picked-up image is generated by adding all signals received by the divided photoelectric conversion units.

  The captured parallax image is equivalent to light field (Light Field) data, which is spatial distribution and angle distribution information of light intensity. In many cases, the refocus processing is performed immediately after the captured parallax image is read. In image processing after the refocus image is generated, it is common to perform processing equivalent to a normal image.

  On the other hand, as one of the utilization methods related to the refocus technique and the parallax image, there is a correction referred to as a previous blur correction. When the same subject is photographed from a plurality of viewpoints, the position of the subject that is not in focus may be different for each parallax image. In this case, a function has been proposed that allows a user operation to select which parallax image to use depending on the positional relationship with the main subject. This is called front blur correction because the subject in front is called front blur. The user sets an optimal correction amount using the parallax image as an input. The correction amount is an amount (weighting amount) for setting a weight for the input parallax image. By combining the final image using the set weight, it is possible to continuously adjust the position of the previous blur.

  By the way, a system that handles parallax images often places importance on compatibility with an existing system that handles a single image. A single image is an image taken from a single viewpoint (hereinafter also referred to as a single image). When the parallax image format is provided in the same format as the single image, the synthesis result of the parallax images is also an image in the same format. Therefore, subsequent image processing can be performed as common processing. In other words, a system that uses the same image processing, performs image composition in the case of parallax images, and treats the result of synthesis equivalent to a single image, so that the same image output can be obtained for both parallax images and single images. Can be designed. There is a great practical advantage in that a system that handles only a single image can be expanded while using parallax images and refocusing technology.

US Pat. No. 4,410,804 JP 2001-083407 A

  A problem in blur correction before handling a parallax image lies in processing relating to detection such as face detection. In normal image processing, processing such as detecting the face of a subject and changing the gain or the like according to the detection result is executed. For example, the adjustment process including the overall brightness is performed so that the detected image area of the face of the subject has an appropriate brightness. In this case, the output result varies greatly depending on whether or not a face is detected. When performing processing that is common to existing image processing on a parallax image, face detection is performed on the result of synthesizing the parallax image (synthesized image), and image processing is executed using the detection result. . Since the synthesis result differs depending on the adjustment result, the face detection result may be different. That is, even if the same parallax image is input, the output result may be greatly different in the process of the user continuously adjusting the composition ratio. Specifically, when the user adjusts the same image while being aware of the position of the previous blur by an operation, the overall brightness may change greatly with a certain adjustment value as a boundary. This results in an image that is natural and far from the adjustment result intended by the user.

  An object of the present invention is to provide an image processing apparatus capable of executing detection processing for a plurality of images having different viewpoints.

  An image processing apparatus according to an embodiment of the present invention includes an acquisition unit that acquires a plurality of first image data and second image data captured from different viewpoints, and the acquired plurality of first image data. Synthesizing means for synthesizing, detection means for performing detection processing on the first and second image data, detection results from the detecting means, and image data synthesized by the synthesizing means, and the second Image processing means for performing image processing on the image data. When the first image data is acquired, the detection means performs a first detection process on the first image data before being synthesized by the synthesis means, and detects the first detection process. When the result is output to the image processing means and the second image data is acquired, the image processing means performs a second detection process on the second image data before performing the image processing, The detection result of the second detection process is output to the image processing means.

  ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus which can perform the detection process with respect to several images from which a viewpoint differs can be provided.

1 is a typical configuration diagram of an image processing apparatus according to an embodiment of the present invention. It is a flowchart which shows operation | movement of the 1st image input part in embodiment of this invention. It is a flowchart which shows operation | movement of the 1st image detection part in embodiment of this invention. It is a flowchart which shows operation | movement of the synthetic | combination part in embodiment of this invention. It is a flowchart which shows operation | movement of the 2nd image input part in embodiment of this invention. It is a flowchart which shows operation | movement of the 2nd image detection part in embodiment of this invention. It is a flowchart which shows operation | movement of the image process part in embodiment of this invention. It is a figure which shows the example of a screen for adjustment with respect to the parallax image in embodiment of this invention.

  Hereinafter, representative examples of the embodiments of the present invention will be described. FIG. 1 is a block diagram illustrating a configuration example of an image processing apparatus according to an embodiment of the present invention.

  The image processing apparatus includes a first image input unit 101 that inputs a parallax image from an external device or an imaging unit. For example, in the case of an imaging device, a plurality of parallax image signals can be acquired from the imaging unit. A parallax image is an image with parallax between a plurality of images, and is an image taken from different viewpoints. In this specification, an image taken from a certain viewpoint is referred to as a “viewpoint image”. A parallax image is composed of a plurality of viewpoint images. Specifically, the imaging device includes a plurality of microlenses and a photoelectric conversion unit divided into a plurality corresponding to each microlens. Light that passes through different pupil partial areas of the exit pupil of the photographing lens is received by each of the plurality of photoelectric conversion units via each microlens and is photoelectrically converted into an electric signal. Thereby, a plurality of viewpoint images are generated from the outputs of the photoelectric conversion units corresponding to the plurality of pupil partial regions, and are input to the first image input unit 101. For example, the first viewpoint image is acquired from the output of the first photoelectric conversion unit, and the second viewpoint image is acquired from the output of the second photoelectric conversion unit. The first image detection unit 102 performs detection processing on a plurality of input viewpoint images. The synthesizing unit 103 synthesizes a plurality of input viewpoint images according to an adjustment value set by a user operation.

  A single image is input to the second image input unit 104. A single image is a viewpoint image taken from a single viewpoint. For example, in the case of an imaging device, it is possible to acquire an image output from the photoelectric conversion unit of the imaging element that constitutes the imaging unit. In the case of a pupil division type imaging device, image data can be acquired from an output obtained by combining the outputs of the plurality of divided photoelectric conversion units by addition. Specifically, an imaging device having a configuration in which two photoelectric conversion units correspond to one microlens is assumed. In this case, from the output signal of the first image (A image) output from the first photoelectric conversion unit and the output signal of the second image (B image) output from the second photoelectric conversion unit, an addition image ( (A + B image) output signal is obtained. The added image (A + B image) is a composite image having different viewpoints with respect to the image acquired from the A image signal and the image acquired from the B image signal. That is, the A + B image is an image having a parallax with respect to the A image and the B image, but when the image is processed alone, it is regarded as equivalent to the case of a single image. In the present embodiment, detection processing is performed by distinguishing between the image synthesized by the imaging unit and the image synthesized by the synthesis unit 103.

  The second image detection unit 105 performs detection processing on the input image or the synthesis result (composite image) by the synthesis unit 103. The image processing unit 106 performs predetermined image processing using the detection results of the first image detection unit 102 and the second image detection unit 105. Note that the image processing apparatus is provided with a system control unit (such as a CPU that executes a program) that controls each unit, an operation unit, a known circuit unit according to the application, and the like, but detailed description thereof is omitted. Hereinafter, functions of each component will be described in order.

  First, the first image input unit 101 acquires a plurality of viewpoint images constituting a parallax image. For example, the viewpoint image has a data structure of a Bayer array, and the previous blur correction is realized by combining the two images with a combination ratio corresponding to the adjustment value input by the user. The image generation method is not limited to a specific method. Multiple viewpoint images can be acquired at the same time by a method of shooting by combining a plurality of single-lens reflex cameras having a general imaging optical system or a method of shooting by a single device including a plurality of imaging optical systems. .

  The viewpoint image input process will be described with reference to FIG. First, data of two viewpoint images are input to the first image input unit 101 (S201). Data of the first viewpoint image A is read and developed on the memory (S202). Next, the image data of the second viewpoint image B is read and developed on the memory (S203). An adjustment value corresponding to the user operation is acquired (S204). This adjustment value is a value indicating the composition ratio, and is a result of the user performing an adjustment operation on a UI (user interface) screen or the like. An operation unit and a UI screen that receive user operation instructions will be described later with reference to FIG.

  Next, the first image detection unit 102 performs detection processing on a plurality of input viewpoint images. A detection process for a plurality of viewpoint images will be described with reference to FIG. The first image detection unit 102 reads a plurality of viewpoint images (S301), and determines whether to detect the viewpoint image or to detect the image after the composition unit 103 combines (S302). . This determination is made based on the set value. As a result of the determination, if it is determined to detect a plurality of viewpoint images, the process proceeds to S303. If it is determined that detection is to be performed after combining by the combining unit 103, the process ends.

  In S303, the first image detection unit 102 performs detection processing on the first viewpoint image A, and in S304, performs detection processing on the second viewpoint image B. A plurality of detection processes are executed, and the detection stage differs depending on the detection item. In the present embodiment, face detection will be described as an example. Assume a function of adjusting the overall brightness so that the brightness of the face area is appropriate when the face of the subject is present in the image. In this case, a setting for performing face detection on the viewpoint image is performed in advance. Face detection processing is performed according to the set value. In addition to this, for example, there is a process of detecting dust (unnecessary image portion) in an image and performing interpolation using peripheral pixels to remove it. An unnecessary image portion can be detected as a portion in which the difference between the luminance or hue of the portion and the luminance or hue in the surrounding area is equal to or greater than a threshold value. In this case, the first image detection unit 102 determines that it is more appropriate to perform the process on the image after the synthesis by the synthesis unit 103 rather than the viewpoint image. Settings for performing the detection process after synthesis are performed in advance.

  The first image detection unit 102 performs detection processing on each viewpoint image, and then combines a plurality of detection results into one detection result (S305). In this embodiment, since face detection is performed, an OR (logical sum) operation is performed on the detected area, and a face image is detected when a face image is detected in one of the viewpoint images. . The combined detection result is temporarily stored in the memory for use in subsequent image processing (S306).

  Next, the synthesis unit 103 synthesizes a plurality of viewpoint images. As for the synthesis, the viewpoint image is subjected to processing according to a general weighted synthesis algorithm. With reference to FIG. 4, a synthesis process for a plurality of viewpoint images will be described.

  Two viewpoint images are input to the combining unit 103 (S401). The synthesizing unit 103 reads the adjustment value designated by the user's operation (S402). The adjustment value has been acquired in S204 of FIG. 2, and is read from the memory. Then, a weighted addition process is performed by the process of adding after multiplying the composition ratio corresponding to the adjustment value for each viewpoint image (S403). A composite image reflecting the set area and adjustment value is generated. The composition unit 103 outputs a composite image.

  Taking the previous blur correction as an example, the user sets optimal adjustment values for a plurality of viewpoint images. The previous blur correction amount corresponding to this adjustment value is a weighting amount for each input viewpoint image. The composition unit 103 determines a composition ratio from the weighting amount, and generates a composite image by executing weighted addition processing. Thereby, the position of the previous blur of the image is adjusted based on the adjustment value input by the user, and an image desired by the user can be obtained.

  Next, processing of a single image will be described. In the case of a single image or a composite image obtained by combining with an image sensor, the second image input unit 104 reads image data. S501 in FIG. 5 shows image data reading processing. The read image data is output to the second image detection unit 105 and processed.

  The second image detection unit 105 performs detection processing on the synthesized image by the synthesizing unit 103 or the image from the second image input unit 104. The detection process will be described with reference to FIG. The second image detection unit 105 reads target image data (S601), and determines whether the image is a synthesized image synthesized by the synthesis unit 103 (S602). If the target image is a synthesized image by the synthesizing unit 103, the process proceeds to S603. If the target image is not a synthesized image by the synthesizing unit 103, the process proceeds to S604.

  The second image detection unit 105 checks whether the image is a detection target image (S603). When it is determined that the image is a detection target image, the process proceeds to S604, and a detection process is performed. However, in the case of a synthesized image by the synthesizing unit 103, the first image detecting unit 102 may have already completed detection processing (such as face detection) depending on preset contents. In that case, no detection process is performed. Since this differs for each item, the second image detection unit 105 performs detection processing based on the set value, as in the case of the first image detection unit 102. The face detection processing is performed by the first image detection unit 102. For example, it is assumed that the second image detection unit 105 is set to perform detection processing for dust removal of an image. In this case, the dust removal detection process is executed at the timing of S604. When the second image detection unit 105 performs the detection process, the detection result is stored in the memory for use in the image process (S605).

  The image processing unit 106 performs image processing using the detection result. Image processing will be described with reference to FIG. First, the image processing unit 106 reads image data (S701). Next, the image processing unit 106 reads the detection result detected by the first image detection unit 102, the detection result detected by the second image detection unit 105, or both detection results (S702). The image processing unit 106 performs image processing based on the input image data and the detection result (S703). The image processing unit 106 outputs the processed image data (S704). The output image is displayed on a screen of a display device, for example, or recorded on a recording medium.

  In the present embodiment, face detection and dust removal processing will be exemplified and will be briefly described. In the face detection process, information on a face area in which one or more faces are detected in the image is input to the image processing unit 106 as a detection result. The image processing unit 106 calculates the average luminance of the detected face area, and applies a plus or minus gain to the entire image when the average luminance is not within the predetermined range. That is, the process of changing the brightness so that the face area is within the range where the exposure is appropriate is performed by gain adjustment. For example, assume that the appropriate exposure range is equivalent to 8 bits and the setting is 100 to 140, and the calculated average brightness of the face area is 50. The image processing unit 106 adjusts the average luminance of the face area to 100 by multiplying the entire image by a gain. As a result, the average luminance of the face area can be set within the set range. This is processing that can be automatically corrected, for example, when the face is extremely dark in an image photographed in backlight.

  In the image dust removal process, the image processing unit 106 detects a black point whose pixel value in the image is equal to or less than a threshold value. Strictly speaking, a point that is extremely different in hue and luminance from the surrounding pixels is detected, and for example, a point having a radius of 2 pixels or less is detected. The detection result is information on the center position and radius of one or more dusts (image portions). The image processing unit 106 removes dust from the image by interpolating each dust from the surrounding pixels and replacing the dust area. As a result, when dust or the like remaining on the image sensor appears in the image, it can be automatically detected and removed. The image processing unit 106 performs image processing using such a detection result, and then outputs an image that is the processing result (S704).

As described above, in this embodiment, detection is performed in two modes in the detection process for image data. In the detection processing in the first mode, the first image detection unit 102 performs detection processing (for example, face detection) on image data before synthesis when the synthesis unit 103 synthesizes data of a plurality of viewpoint images. ). In the detection process in the second mode, the second image detection unit 105 executes the following process.
-Detection processing for single viewpoint image data (for example, face detection and dust image portion detection)
A detection process for the image data synthesized by the synthesis unit 103 (for example, detection of a dust image portion).
The detection result is sent to the image processing unit 106 according to the mode. Therefore, the timing of the image data detection process can be individually set according to the detection item.

  FIG. 8 illustrates a front blur correction UI screen when the user adjusts the composition ratio. Although there is an embodiment in which a plurality of viewpoint images are adjusted on a UI screen that handles a single image, this embodiment shows an example in which a user performs a viewpoint image adjustment operation while viewing a dedicated UI screen. First, the viewpoint image data is read, and a dialog screen 801 is displayed on the display unit. On the dialog screen 801, a cancel button 803, an enlarged display area 802, a thumbnail display area 804, an adjustment slider 805, and a save button 806 are arranged. The user performs a desired adjustment by a touch operation using a touch panel or an input operation using a pointing device such as a mouse or a keyboard. The adjustment value data instructed by the user through the operation unit is stored in the memory and used when correcting the previous blur of the image.

  In the present embodiment, detection processing for a plurality of viewpoint image data and detection processing for single viewpoint image data and synthesized image data can be executed at a preset timing. According to the present embodiment, in a system capable of image processing of a plurality of viewpoint images having a parallax and a single viewpoint image, it is possible to adjust a synthesis ratio of the plurality of viewpoint images and obtain an output result intended by the user. it can.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 101 1st image input part 102 1st image detection part 103 Composition part 104 2nd image input part 105 2nd image detection part 106 Image processing part

Claims (14)

Obtaining means for obtaining a plurality of first image data and second image data taken from different viewpoints;
Combining means for combining the plurality of acquired first image data;
Detection means for performing detection processing on the first and second image data;
An image processing unit that obtains a detection result from the detection unit and performs image processing on the image data combined by the combining unit and the second image data;
The detection means includes
When the first image data is acquired, a first detection process is performed on the first image data before being synthesized by the synthesis unit, and a detection result of the first detection process is used as the image process. Output to the means,
When the second image data is acquired, a second detection process is performed on the second image data before the image processing unit performs the image process, and a detection result of the second detection process is obtained. An image processing apparatus that outputs to the image processing means. Obtaining means for obtaining a plurality of first image data and second image data taken from different viewpoints;
Combining means for combining the plurality of acquired first image data;
Detection for executing detection processing in the first mode in which the combining unit combines the plurality of first image data, and detection processing in the second mode performed on the second image data Means,
Image data obtained by obtaining detection results from the detection means and synthesized by the synthesis means, and image processing means for performing image processing on the second image data,
The detection means includes
In the first mode, a first detection process is performed on the first image data before being synthesized by the synthesizing unit, and a detection result of the first detection process is output to the image processing unit. ,
In the second mode, the second detection process is performed on the second image data before the image processing unit performs the image processing, and the detection result of the second detection process is used as the image processing unit. An image processing apparatus that outputs to The acquisition means includes
First acquisition means for acquiring the plurality of first image data output from the image sensor;
The image processing apparatus according to claim 1, further comprising: a second acquisition unit configured to acquire the second image data synthesized by the image sensor. When the detection unit performs the second detection process on the image data combined by the combination unit, the detection unit outputs a detection result of the second detection process to the image processing unit,
4. The image processing unit according to claim 1, wherein the image processing unit acquires the detection results of the first and second detection processes and performs image processing on the image data combined by the combining unit. The image processing apparatus according to item. An operation unit that receives an input of an adjustment value used when the combining unit combines the images;
The said synthetic | combination means determines the synthetic | combination ratio with respect to a said some 1st image data from the said adjustment value, and produces | generates the image data synthesized | combined. Image processing device.   The image processing apparatus according to claim 5, wherein the synthesizing unit performs a process of correcting blur of an image using the adjustment value.   The detection means does not perform the process of detecting the face area of the subject on the image synthesized by the synthesizing means when the process of detecting the face area of the subject is performed on the first image data. The image processing apparatus according to claim 1, wherein:   When the detection unit detects the face area of the subject, the image processing unit calculates the luminance of the face area of the subject and adjusts the gain to change the brightness of the image. The image processing apparatus according to claim 7. The detection means detects an image portion in which a difference in luminance or hue between a region to be detected in the image and its surrounding region is equal to or greater than a threshold,
9. The image processing unit according to claim 1, wherein the image processing unit performs a process of interpolating and replacing a pixel value of the image portion detected by the detection unit using a pixel value of a surrounding area. The image processing apparatus according to item 1.   An imaging device comprising the image processing device according to claim 1. An image sensor having a plurality of microlenses and a plurality of photoelectric conversion units corresponding to each microlens,
The acquisition unit acquires the plurality of first image data from signals output from the plurality of photoelectric conversion units, and combines the signals output from the plurality of photoelectric conversion units from the second image. Data is acquired, The imaging device of Claim 10 characterized by the above-mentioned. A control method executed by an image processing apparatus that acquires a plurality of first image data and second image data captured from different viewpoints and performs image processing,
A detection step of performing detection processing on the first and second image data obtained by the detection means;
A combining step of combining the plurality of first image data obtained by the combining unit;
An image processing unit that obtains a detection result from the detection unit and performs image processing on the image data combined by the combining unit and the second image data; and
In the detection step,
When the first image data is acquired, the detection unit performs a first detection process on the first image data before being synthesized by the synthesis unit, and a detection result of the first detection process. Is output to the image processing means,
When the second image data is acquired, the detection unit performs a second detection process on the second image data before the image processing unit performs image processing, and the second detection process. A detection method for outputting an image processing device to the image processing means. A control method executed by an image processing apparatus that acquires a plurality of first image data and second image data captured from different viewpoints and performs image processing,
In the detection mode in the first mode in which the detection unit combines the acquired plurality of first image data with the combination unit, and in the second mode to be performed on the second image data A detection process for executing the detection process;
A combining step in which the combining means combines the plurality of acquired first image data;
An image processing unit having an image processing step of obtaining a detection result from the detection unit and performing image processing on the image data combined by the combining unit and the second image data;
In the first mode of the detection step, the detection means performs a first detection process on the first image data before being synthesized by the synthesis means, and a detection result of the first detection process Is output to the image processing means,
In the second mode of the detection step, the detection means performs a second detection process on the second image data before the image processing means performs image processing, and the second detection process. A detection method for outputting an image processing device to the image processing means. A program for causing a computer of an image processing apparatus to execute each step according to claim 12 or 13.

Images