JP6441552B2 - Image processing apparatus, imaging apparatus, display apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing device, an imaging device, a display device, an image processing method, and an image processing program.

  Various image processing technologies have been developed to enhance the sharpness and sharpness of still images or moving images when they are captured by an imaging device such as a digital camera or digital video camera or displayed on a display. ing. Examples of the image processing technique include a contour enhancement process that increases sharpness and a saturation enhancement process that increases sharpness. Furthermore, there is an image processing technique using depth information as image processing for adjusting the sense of depth. For example, in Patent Document 1, by controlling the depth correction gain based on the depth data (information) indicating the three-dimensional positional relationship between the imaging device and the subject and the depth level indicating the adjustment level of the depth sensation. An image processing apparatus that can freely adjust the depth of an image is disclosed.

JP 2008-33897 A

  However, the image processing apparatus described in Patent Document 1 does not consider the case where the setting of the imaging apparatus that captured the input image, for example, the focal length, the aperture, or the like is changed. There is a problem that the depth of the image is insufficient. For example, when zooming up by changing the focal length, even if the actual positional relationship between the subject and the imaging device has not changed significantly, the angle of view becomes narrower and the subject is enlarged, resulting in an image that approaches the subject. . In addition, the depth of field becomes shallow, and an image with a large blur amount of a subject existing outside the depth of field is obtained. In other words, although the sense of depth changes as the subject and composition of the input image change according to the change in the focal length, the image processing apparatus described in Patent Document 1 does not consider this, so the processing result image A feeling of depth becomes insufficient.

  The present invention has been made in view of such circumstances, and an image processing apparatus and an imaging apparatus capable of generating an image that allows the user to feel the depth even when the setting of the imaging apparatus is changed. An object is to provide a display device, an image processing method, and an image processing program.

(1) The present invention has been made to solve the above-described problems, and one aspect of the present invention refers to setting information of an imaging apparatus when an image is captured and depth information corresponding to the image. An image processing intensity determining unit that determines an image processing intensity for each region into which the image is divided, and an image processing unit that performs image processing on the image data of the image according to the image processing intensity. An image processing apparatus.

(2) Moreover, the other aspect of this invention comprises the imaging part which images a to-be-photographed object, and the above-mentioned image processing apparatus, The said image is the image which the said imaging part imaged, The said setting information is The image pickup apparatus is setting information when the image pickup unit picks up the image.

(3) Moreover, the other aspect of this invention is a display apparatus characterized by including the above-mentioned image processing apparatus and the image display part which displays the image image-processed by the said image processing apparatus.

(4) According to another aspect of the present invention, image processing is performed for each area obtained by dividing the image with reference to setting information of the imaging device when the image is captured and depth information corresponding to the image. An image processing method comprising: a first step of determining an intensity; and a second step of performing image processing on image data of the image according to the image processing intensity.

(5) According to another aspect of the present invention, the computer refers to setting information of an imaging device when an image is captured and depth information corresponding to the image, and for each area obtained by dividing the image. An image processing program for functioning as an image processing intensity determining unit that determines image processing intensity and an image processing unit that performs image processing on image data of the image according to the image processing intensity.

ADVANTAGE OF THE INVENTION According to this invention, even if the setting of the imaging device which imaged the input image changes, it becomes possible to produce | generate the image which a user can feel depth.

1 is a schematic block diagram of an image processing apparatus 100 according to a first embodiment of the present invention. It is an example of the graph which shows the correspondence of the focal distance and intensity correction value in the embodiment. It is another example of the graph which shows the correspondence of the focal distance and intensity correction value in the embodiment. It is a graph which shows the correspondence of F value and intensity | strength correction value in the 1st modification of the embodiment. It is a figure which shows an example of the correspondence of the depth information value and intensity | strength base value in the embodiment. It is a figure which shows an example of the correspondence of the intensity | strength correction value in the same embodiment, and the image processing intensity when the depth information value is the maximum. It is a figure which shows an example of the correspondence of the depth information value and image processing intensity | strength S in the embodiment. It is a figure which shows the example of the input image which the image data Gi in the embodiment represents. It is a figure which shows the example of the depth information D corresponding to the image data Gi of FIG. 4 is a flowchart illustrating an example of processing of the image processing apparatus 100 according to the embodiment. It is a schematic block diagram which shows the structure of the image processing apparatus 100a in the 2nd Embodiment of this invention. It is a schematic block diagram of the depth information analysis part 105 in the embodiment. It is the figure which showed an example of the frequency distribution of the depth information value in the embodiment. It is a figure which shows an example of the correspondence of the depth information value and intensity | strength base value in the embodiment. It is a figure which shows another example of the correspondence of the depth information value and intensity | strength base value in the embodiment. It is a figure which shows another example of the correspondence of the depth information value and intensity | strength base value in the embodiment. It is a schematic block diagram of the imaging device 200 in the 4th Embodiment of this invention. It is the figure which showed the relationship between the distance Z and the parallax d. It is a schematic block diagram of the display apparatus 300 in the 5th Embodiment of this invention.

[First Embodiment]
Hereinafter, a first embodiment of the present invention will be described in detail with reference to the drawings. Note that the expressions in the drawings are exaggerated for easy understanding, and may differ from actual ones. FIG. 1 is a schematic block diagram of an image processing apparatus 100 according to the first embodiment. The image processing apparatus 100 receives image data Gi of the input image, depth information D corresponding to the input image, and imaging device setting information P of the imaging device when the input image is captured. The image processing device 100 performs image processing on the image data Gi based on the depth information D and the imaging device setting information P, and outputs the processing result as output image data Go.

  Here, the image data Gi and the output image data Go are data composed of values such as the gradation values of the pixels constituting the image. The depth information D is information including depth information values corresponding to each pixel of the input image represented by the image data Gi. The depth information value is a value representing the distance from the imaging device that captured the input image to the subject represented by the pixel corresponding to the depth information value in the input image. That is, as the depth information value of a certain pixel increases, the distance from the imaging device to the subject represented by the pixel increases. For example, in the case of depth information in which each depth information value is represented by 8 bits, the depth information value 0 is the most distant subject and the depth information value 255 is the most distant subject.

  Further, the imaging device setting information P is information indicating the setting of the imaging device when the input image represented by the image data Gi is captured, and in this embodiment, represents the focal length of the lens of the imaging device that captured the input image. Information. As will be described later, the imaging apparatus setting information P may be information representing, for example, an F value, ISO sensitivity, exposure time (shutter speed), in-focus position, and the like. The image data Gi may be data representing a moving image or data representing a still image. The image data Gi may be input as a signal such as a video signal, or may be input as a data file such as a JPEG (Joint Photographic Experts Group) file or an MPEG (Moving Picture Experts Group) file.

The image processing apparatus 100 includes an image processing intensity determining unit 101 and an image processing unit 102. The image processing intensity determination unit 101 refers to the depth information D corresponding to the input image and the imaging device setting information P of the imaging device when the input image is captured, for each area into which the image is divided (in this embodiment, The image processing intensity S is determined for each pixel. The image processing unit 102 performs image processing on the image data Gi of the input image according to the image processing strength S.
The image processing intensity determination unit 101 includes an intensity correction value determination unit 103 and an image processing intensity calculation unit 104.

  The intensity correction value determination unit 103 determines an intensity correction value according to a predetermined rule based on the input imaging device setting information P, and outputs it to the image processing intensity calculation unit 104. Here, the intensity correction value is a value representing the degree of correction when the image processing intensity calculation unit 104 in the subsequent stage calculates the image processing intensity. Details of the processing of the intensity correction value determination unit 103 will be described later.

  Based on the intensity correction value determined by the intensity correction value determination unit 103 and the depth information D input from the outside of the image processing apparatus 100, the image processing intensity calculation unit 104 performs image processing intensity for each pixel of the image data Gi. S is calculated. The image processing intensity calculation unit 104 outputs the calculated image processing intensity S to the image processing unit 102. Details of the processing of the image processing intensity calculation unit 104 will be described later.

  The image processing unit 102 performs image processing on the image data Gi in accordance with the image processing strength S that is output information of the image processing strength calculation unit 104. The image processing performed by the image processing unit 102 may be any of contour enhancement processing, contrast enhancement processing, saturation enhancement processing, blurring processing, noise removal processing, and the like. The image processing unit 102 outputs an image generated by performing image processing to the outside of the image processing apparatus 100 as output image data Go. Details of the processing of the image processing unit 102 will be described later.

<Details of Processing of Strength Correction Value Determination Unit 103>
Next, details of the processing of the intensity correction value determination unit 103 will be described. The intensity correction value determination unit 103 calculates an intensity correction value based on setting information P input from the outside of the image processing apparatus 100 and outputs the intensity correction value to the image processing intensity calculation unit 104.

(Calculation method of intensity correction value based on focal length)
FIG. 2 is an example of a graph showing a correspondence relationship between the imaging apparatus setting information P (lens focal length) and the intensity correction value in the present embodiment. A method in which the intensity correction value determination unit 103 determines the intensity correction value based on the focal length of the lens will be described with reference to FIG.

  In the figure, the horizontal axis represents the focal length, and the vertical axis represents the intensity correction value. Further, in the figure, the minimum focal length fmin, the maximum focal length fmax, and the focal length fp are shown as an example of the focal length indicated by the imaging device setting information P input to the intensity correction value determination unit 103. The intensity correction value corresponding to the minimum focal distance fmin is the minimum intensity correction value Cfmin, and the intensity correction value corresponding to the maximum focal distance fmax is the maximum intensity correction value Cfmax. The intensity correction value corresponding to the focal length fp is the intensity correction value Cf1.

  The intensity correction value determination unit 103 stores an LUT (Look Up Table) corresponding to the graph shown in FIG. 2 in advance, and stores the LUT in association with the focal length indicated by the input imaging device setting information P. The read value is read out, and this value is used as the intensity correction value. That is, if the focal length indicated by the input imaging device setting information P is fp, the intensity correction value is Cf1, and if the focal length indicated by the input imaging device setting information P is fmin, the intensity correction value is Cfmin. If the focal length indicated by the input imaging device setting information P is fmax, the intensity correction value is Cfmax. The intensity correction value determination unit 103 may store an arithmetic expression representing the graph of FIG. 2 instead of the LUT, and calculate the intensity correction value using the arithmetic expression.

  Note that it is desirable that the minimum focal length and the maximum focal length of the imaging apparatus that captured the input image are within the ranges indicated by the minimum focal length fmin and the maximum focal length fmax. That is, it is desirable that an imaging apparatus that captures an input image includes an optical system that can change the focal length in the range from the minimum focal length fmin to the maximum focal length fmax or in a narrower range.

  The example shown in FIG. 2 shows a characteristic that the intensity correction value increases linearly as the focal length increases. Considering that the change in the focal length due to the zooming of the optical system is linear, such a correspondence relationship is most preferable. However, the present invention is not limited to this. For example, as shown in FIG. 3, nonlinear characteristics may be used in which the intensity correction value is significantly increased as the focal length is increased. In the figure, it is shown that the intensity correction value is the same as that in FIG. 2 at the focal lengths fmin and fmax, and the intensity correction value Cf2 is smaller than the intensity correction value Cf1 at the focal length fp.

  As described above, when the relationship between the focal length and the intensity correction value is monotonously increased, it is preferable because the change in the focal length corresponds to the image processing result. Here, the monotonic increase in the present embodiment is a monotonic increase in a broad sense including the case where the slope is 0. For example, when the focal length is sufficiently large or small, the intensity is corrected so as not to be overemphasized. The value may be saturated.

With the above method, the intensity correction value determination unit 103 calculates the intensity correction value based on the imaging apparatus setting information P, that is, the focal length of the lens, and outputs the intensity correction value to the image processing intensity calculation unit 104.
In general, as the focal length increases, the depth of field decreases, and the subject at the out-of-focus position is blurred and the blur area increases. In the blurred area, the outline of the subject becomes more ambiguous, and the contrast and saturation decrease. Thus, as described above, the image processing intensity set by the image processing intensity calculating unit 104 described later is increased by increasing the intensity correction value as the focal length increases. Accordingly, when the depth of field becomes shallow due to zooming up (increasing the focal length) and the amount of blur outside the depth of field becomes large, the contour enhancement processing, contrast enhancement, and saturation in the image processing unit 102 are performed. The intensity of image processing such as correction processing can be increased, and the change in blur amount due to the change in focal length can be suppressed.

  As will be described later, since the image processing intensity calculation unit 104 described later increases the image processing intensity as the depth information value increases, the intensity correction value determination unit 103 determines the image processing intensity for the magnitude change of the depth information value. The intensity correction value is determined so that the relationship between the magnitude change and the relation between the magnitude change in the image processing intensity with respect to the magnitude change in the focal length are the same.

(Calculation method of intensity correction value based on F value (first modification))
Further, the imaging device setting information P may be an F value representing an aperture. Hereinafter, as a first modification of the present embodiment, a case where the imaging apparatus setting information P is an F value and the intensity correction value determination unit 103 calculates an intensity correction value based on the F value will be described. FIG. 4 is a graph showing the correspondence between the F value and the intensity correction value in the first modification of the present embodiment. In the figure, the horizontal axis represents the F value, and the vertical axis represents the intensity correction value. Further, in the figure, a minimum F value Fmin, a maximum F value Fmax, and an F value Fp as an example of the F value indicated by the imaging device setting information P input to the intensity correction value determination unit 103 are illustrated. The intensity correction value corresponding to the minimum F value Fmin is the maximum intensity correction value CFmax, and the intensity correction value corresponding to the maximum F value Fmax is the minimum intensity correction value CFmin. The intensity correction value corresponding to the F value Fp is the intensity correction value CF1.

The intensity correction value determination unit 103 according to the present modification stores an LUT corresponding to the graph shown in FIG. 4 in advance, and the LUT stores the LUT in association with the F value indicated by the input imaging device setting information P. The value is read out and used as an intensity correction value.
Note that it is desirable that the minimum F value and the maximum F value of the imaging apparatus that captured the input image are within the ranges indicated by the minimum F value Fmin and the maximum F value Fmax. That is, it is desirable that an imaging apparatus that captures an input image includes an optical system whose aperture is variable in a range from the minimum F value Fmin to the maximum F value Fmax or in a narrower range.

  The example shown in FIG. 4 shows a characteristic that the intensity correction value decreases nonlinearly as the F value increases. For the same focal length, there is a relationship that the depth of field becomes deeper as the F value increases (the lens aperture is reduced), so the intensity correction value decreases as the F value increases. In this case, considering the relationship that the F value becomes √2 when the aperture area of the lens is halved (the brightness is halved), the non-linear correspondence as shown in FIG. 4 is most preferable. However, the present invention is not limited to this. For example, the intensity correction value may be linearly reduced as the F value increases.

  Thus, if the relationship between the F value and the intensity correction value is monotonically decreased, it is preferable because the change in the F value corresponds to the image processing result. Here, the monotonic decrease in the present embodiment is a monotonic decrease in a broad sense including the case where the slope is 0. For example, when the F value is sufficiently large or small, the image processing intensity parameter is not overemphasized. May be saturated.

  In general, the smaller the F value, the shallower the depth of field, and the greater the area where an out-of-focus subject is blurred. Conversely, the greater the F value, the deeper the depth of field and the more focused subject areas. Therefore, as described above, as the F value increases, the intensity correction value is decreased, so that the image processing intensity set by the subsequent image processing intensity calculation unit 104 is reduced. Therefore, when the lens aperture is opened (F value is decreased) and the depth of field becomes shallow and the amount of blur outside the depth of field increases, the contour enhancement processing, contrast enhancement and saturation correction processing are performed. The image processing intensity such as the above can be increased, and the influence of the change in the blur amount due to the change of the aperture (F value) can be suppressed.

  As will be described later, since the image processing intensity calculation unit 104 described later increases the image processing intensity as the depth information value increases, the intensity correction value determination unit 103 determines the image processing intensity for the magnitude change of the depth information value. The intensity correction value is determined so that the relationship between the magnitude change and the relation between the magnitude change in the image processing intensity with respect to the magnitude change in the F value are reversed.

(Calculation method of intensity correction value based on ISO sensitivity (second modification))
Further, the imaging device setting information P may be ISO sensitivity. Hereinafter, a case where the imaging apparatus setting information P is ISO sensitivity and the intensity correction value determination unit 103 calculates an intensity correction value based on the ISO sensitivity will be described as a second modification of the present embodiment.
In general, in an imaging apparatus, when the ISO sensitivity value is changed, the amplification degree of an electric signal when photoelectrically converting light irradiated to the imaging element is adjusted. Therefore, for example, when the amount of incident light to the image sensor is small, increasing the ISO sensitivity value increases the amplification factor when amplifying the electrical signal, and enables a bright image to be captured.

  In this modification, the intensity correction value determination unit 103 determines the intensity correction value so that the intensity correction value increases linearly as the ISO sensitivity value increases. For example, the intensity correction value determination unit 103 uses linear characteristics similar to the correspondence relationship between the focal length and the intensity correction value shown in FIG. Considering that the change of the ISO sensitivity value is linear, such a linear characteristic is most preferable.

  In general, when an electric signal is amplified, a noise signal is also amplified at the same time. Therefore, as the ISO sensitivity value increases, the noise signal is also amplified, and image quality deterioration becomes conspicuous. Therefore, as described above, as the ISO sensitivity value increases, the intensity correction value is increased to increase the image processing intensity set by the subsequent image processing intensity calculating unit 104. Accordingly, when the ISO signal is increased to increase the noise signal and the image is deteriorated, the image processing intensity of image processing such as blurring processing and noise removal processing is further increased, and the image due to the increase in ISO sensitivity. Can be prevented.

  As will be described later, since the image processing intensity calculation unit 104 described later increases the image processing intensity as the depth information value increases, the intensity correction value determination unit 103 determines the image processing intensity for the magnitude change of the depth information value. The intensity correction value is determined so that the relationship between the magnitude change and the relation between the magnitude change in image processing intensity with respect to the magnitude change in ISO sensitivity are the same.

  There may be a plurality of types of image processing in the image processing unit 102 described later. In that case, separate intensity correction values and image processing intensities may be calculated for each image processing. For example, a case where the image processing unit 102 performs noise removal processing and contour enhancement processing will be described. As described above, since the noise signal is amplified when the ISO sensitivity value increases, the image processing intensity is increased, and the noise amount is reduced by the noise removal processing. However, since the edge enhancement process makes noise conspicuous, it is not desirable to increase the image processing intensity as in the noise removal process. Therefore, the intensity correction value determination unit 103 calculates the first intensity correction value for the noise removal process from the relationship as shown in FIG. 3, and the second intensity correction value for the edge enhancement process as shown in FIG. Calculate from In this way, calculating a plurality of intensity correction values and image processing intensities is preferable because an appropriate image processing intensity can be calculated for the contents of the image processing.

(Calculation method of intensity correction value based on shutter speed (third modification))
Further, the imaging apparatus setting information P may be a shutter speed (exposure time). Hereinafter, a case where the imaging apparatus setting information P is the shutter speed and the intensity correction value determination unit 103 calculates the intensity correction value based on the shutter speed will be described as a third modification of the present embodiment.
In general, in an imaging apparatus, it is possible to adjust the exposure time to the image sensor by changing the shutter speed value. If the shutter speed value is halved, the exposure time can be halved. . For example, when a moving subject is imaged as a still image, the moving subject may be blurred due to a long exposure time. By reducing the shutter speed value and shortening the exposure time, the moving subject is prevented from being blurred. It becomes possible.

  In this modification, the intensity correction value determination unit 103 determines the intensity correction value so that the intensity correction value decreases nonlinearly as the shutter speed value increases. The intensity correction value determination unit 103 uses a non-linear characteristic such as the correspondence relationship between the F value and the intensity correction value shown in FIG. In consideration of changing the shutter speed value in a non-linear manner, such a non-linear correspondence is most preferable.

  In general, when the shutter speed value is reduced (exposure time is shortened), the amount of incident light is reduced, and the captured image becomes dark. Therefore, in order to capture an image with appropriate brightness, an electric signal is amplified by the above-described ISO sensitivity or the like, or each pixel value of the image after imaging is gained with a gain. When such a process is performed, the noise signal is also amplified at the same time, or the pixel value on which the noise is added is brightened, so that the image quality deterioration becomes conspicuous. Therefore, as described above, if the intensity correction value is decreased as the shutter speed value is increased, the image processing intensity set by the subsequent image processing intensity calculation unit 104 is decreased. Accordingly, when the shutter speed value is small and noise is conspicuous, the image processing intensity can be made stronger, and when the shutter speed value is large, the image processing intensity can be weakened.

  In the above description, the intensity correction value is calculated based on the shutter speed value. However, exposure time information may be used as the imaging device setting information P. As described above, since the correspondence between the shutter speed and the exposure time is linear, the intensity increases as the exposure time becomes longer (value increases) as in the nonlinear correspondence between the shutter speed and the intensity correction value. The correspondence relationship may be such that the correction value decreases nonlinearly.

  Further, as will be described later, the image processing intensity calculation unit 104 described later increases the image processing intensity as the depth information value increases, so the intensity correction value determination unit 103 determines the image processing intensity for the magnitude change of the depth information value. The intensity correction value is determined such that the relationship between the magnitude change and the relation between the magnitude change in the image processing intensity with respect to the magnitude change in the shutter speed or the exposure time are reversed.

(Calculation method of intensity correction value based on in-focus position (fourth modification))
Further, the imaging device setting information P may be information on the focal point position. Hereinafter, as a fourth modification of the present embodiment, a case where the imaging apparatus setting information P is in-focus position information and the intensity correction value determination unit 103 calculates an intensity correction value based on the in-focus position information. explain.
Here, the in-focus position information refers to two-dimensional position information on the input image where the subject at the in-focus position exists, and depth information values representing distance information from the imaging device to the subject. The depth information value may be acquired as a configuration in which the depth information D is input to the intensity correction value determination unit 103, or an approximate value may be calculated from the operation information of the optical system of the imaging apparatus, that is, the lens. May be.

  When calculating the intensity correction value based on the in-focus position information, for example, when the input image includes a human subject in the foreground and a landscape such as a mountain in the foreground, the two-dimensional position of the in-focus is on the far side. If it is on the subject, the focus is on the distant view. Therefore, the blur amount of the near-field human subject is large. At this time, by calculating so that the intensity correction value on the foreground side with a large depth information value is small and the intensity correction value on the foreground side with a small depth information value is large, a larger image is obtained in a foreground human subject with a large blur amount With the processing intensity, it is possible to reduce the image processing intensity in a distant subject in focus. Here, when the correspondence relationship between the depth information value of the subject and the intensity correction value is represented by a graph, if the vertical axis is the intensity correction value and the horizontal axis is the depth information value, the intensity correction value is the depth information value at the in-focus position. It can be expressed by a V-shaped characteristic that is minimized.

  In the first embodiment and the first to third modifications, the intensity correction value determination unit 103 determines the intensity correction value for each input image (for each frame in the case of a moving image). In the modification, the intensity correction value is determined for each pixel of the input image.

The method for calculating the intensity correction value based on the information on the focal length, F value, ISO sensitivity, shutter speed, or in-focus position has been described above, but the present invention is not limited to this. For example, since a plurality of imaging device setting information P may be input to the intensity correction value determination unit 103, the intensity correction value may be calculated based on the plurality of imaging device setting information P.
For example, it may be calculated based on information on both the focal length and the F value. In this case, a value obtained by adding the intensity correction value Cf1 calculated based on the focal length fp shown in FIG. 2 and the intensity correction value CF1 calculated based on the F value Fp shown in FIG. Can be a value. Further, an average value or a multiplication value of the intensity correction value Cf1 and the intensity correction value CF1 may be used, or a value obtained by changing the weights of the intensity correction value Cf1 and the intensity correction value CF1 may be used.

Further, for example, the calculation may be performed based on four of the imaging apparatus setting information P, that is, focal length, F value, ISO sensitivity, and shutter speed. In this case, as in the above example, the final intensity correction value may be an addition value, an average value, or a multiplication value of four intensity correction values calculated based on the respective imaging device setting information P. Further, a value obtained by adding the respective weights may be used.
As described above, the intensity correction value can be calculated by freely combining a plurality of imaging device setting information P.

  Further, depth of field information may be used as the imaging device setting information P. For example, by inputting a plurality of imaging device setting information such as focal length, F value, allowable confusion circle diameter and the like and depth information D to the intensity correction value determining unit 103, the object scene is based on the input information. Depth can be calculated. In this case, taking into consideration that the amount of blur of the subject that exists outside the depth of field increases as the depth of field decreases, the intensity correction is performed as the depth of field increases (value increases). A correspondence relationship having linear or non-linear characteristics such that the value is small can be obtained. As described above, as the depth of field value increases, the intensity correction value decreases so that the image processing intensity set by the subsequent image processing intensity calculation unit 104 decreases. Accordingly, when the depth of field value is small and the amount of blur is large, the image processing strength can be made stronger, and when the depth of field value is large, the image processing strength can be weakened.

<Details of Processing of Image Processing Intensity Calculation Unit 104>
The image processing intensity calculation unit 104 first determines an initial image processing intensity (intensity base value) for each pixel of the input image based on the depth information D. Subsequently, the image processing intensity calculation unit 104 corrects the intensity base value based on the intensity correction value determined by the intensity correction value determination unit 103 to determine the image processing intensity S. A method for determining the intensity base value based on the depth information D will be described below with reference to FIG.

  FIG. 5 is a diagram illustrating an example of a correspondence relationship between the depth information value and the intensity base value. In the figure, the horizontal axis is the depth information value that is the depth information D, and the vertical axis is the intensity base value. In the figure, there is shown a characteristic that the intensity base value increases linearly as the depth information value increases. The image processing intensity calculation unit 104 first determines an intensity base value based on this characteristic. Considering the continuity of the subject, it is preferable that the intensity base value has a monotonically increasing relationship with the depth information value.

Further, in the figure, when the depth information value is at the maximum value (for example, 255) of the possible range, the intensity base value is the maximum β2, and the depth information value is the minimum value (for example, 0) of the possible range. It is shown that the intensity base value is the minimum β1.
Next, a method of determining the image processing intensity S by correcting the intensity base value determined based on the characteristics shown in FIG. 5 based on the intensity correction value that is output information of the intensity correction value determining unit 103 will be described. .

Hereinafter, as an example, a case where the intensity correction value is calculated based on the characteristics shown in FIG. 2 will be described with reference to FIG.
FIG. 6 is a diagram illustrating an example of a correspondence relationship between the intensity correction value and the image processing intensity when the depth information value is maximum. In the figure, the horizontal axis is the intensity correction value, and the vertical axis is the image processing intensity when the depth information value is maximum. Note that the horizontal axis of the figure is the same as the intensity correction value calculated based on the characteristics shown in FIG. 2, and is indicated by the same symbol. In addition, the vertical axis in the figure is the image processing intensity that is newly set in the maximum value of the depth information shown in FIG. 5, and is determined according to the intensity correction value that is output information of the intensity correction value determination unit 103. Is done.

  In FIG. 6, when the intensity correction value input to the image processing intensity calculation unit 104 is 0, it is indicated that the image processing intensity at the maximum value of the depth information remains the intensity base value β2 illustrated in FIG. ing. In addition, when the intensity correction value is Cfmax, the image processing intensity at the maximum value of the depth information is βmax, and when the intensity correction value is Cf1, the image processing intensity at the maximum value of the depth information is βf1. ing.

The example shown in FIG. 6 shows a characteristic in which the image processing intensity at the maximum value of depth information increases linearly as the intensity correction value increases. However, the present invention is not limited to this, and it may be a non-linear characteristic in which the image processing intensity at the maximum value of the depth information suddenly increases as the intensity correction value increases.
When the intensity correction value is small, the image processing intensity at the maximum value of the depth information is determined to be linearly smaller, contrary to the case where the intensity correction value is large. Further, it may be a non-linear characteristic such that the image processing intensity at the maximum value of the depth information suddenly decreases as the intensity correction value decreases.

  When the intensity correction value Cf1 is input to the image processing intensity calculation unit 104, the image processing intensity at the maximum value of the depth information is βf1 based on the characteristics of FIG. At this time, the correspondence relationship between the depth information value and the intensity base value shown in FIG. 5 is updated to the correspondence relationship between the depth information value and the image processing strength S shown in FIG. In FIG. 7, the horizontal axis is the depth information value, and the vertical axis is the image processing intensity S. In the drawing, the image processing intensity at the maximum value of the depth information is βf1, and the intensity base value that was β2 in FIG. 5 is updated to the image processing intensity S that is βf1 in FIG. In this way, the intensity base value is corrected based on the intensity correction value, and the correspondence relationship between the depth information value and the image processing intensity S is set. The image processing intensity calculation unit 104 calculates the image processing intensity S in each pixel by applying the input depth information D to the correspondence relationship between the set depth information value and the image processing intensity S.

  As one aspect, the image processing intensity calculation unit 104 stores in advance an LUT in which the correspondence relationship between the depth information value and the intensity base value as shown in FIG. 5 is as in the example of FIG. The intensity base value corresponding to the depth information value in each pixel indicated by the depth information is read from the LUT. The image processing intensity calculation unit 104 adds the intensity correction value input from the intensity correction value determination unit 103 to the read intensity base value to calculate the image processing intensity S of each pixel. As another aspect, the image processing intensity calculation unit 104 stores in advance an LUT representing the correspondence between the depth information value and the image processing intensity S for each intensity correction value, and the input intensity correction value In response to this, an LUT is selected, and the image processing intensity S associated with the input depth information value is read from the selected LUT.

  By the above method, the image processing intensity calculation unit 104 calculates the image processing intensity S so that the image processing intensity S is maximized at the maximum value of depth information and the image processing intensity S is minimized at the minimum value of depth information. To do. Furthermore, the intensity base value is set to increase as the depth information value increases between the minimum value and the maximum value of the depth information, and the image processing intensity S is calculated so as to increase as the intensity base value increases. To do. Further, the image processing intensity S is calculated such that the larger the intensity correction value is, the larger the value is.

<Details of Processing of Image Processing Unit 102>
Image data Gi and depth information D input to the image processing apparatus 100 will be described with reference to FIGS. 8 and 9. FIG. 8 is a diagram illustrating an example of an input image represented by image data Gi input to the image processing apparatus 100. In the figure, an image region 8b of a human subject existing in the foreground and an image region 8a of a background subject existing in the distant view are shown.

  FIG. 9 is a diagram illustrating an example of the depth information D corresponding to the image data Gi of FIG. In each pixel in the figure, the depth information is represented by a gray scale from white to black, and the depth information is smaller as the color becomes white. In the drawing, depth information 9a corresponding to the image area 8a of the background subject existing in the distant view and depth information 9b corresponding to the image area 8b of the person subject existing in the foreground are shown. The depth information 9b of the person subject is smaller than the depth information 9a of the background subject, indicating that the person subject exists in the foreground.

  When a scene as shown in FIG. 8 is input as the image data Gi and the corresponding depth information is input as FIG. 9, the main subject is a human subject placed in the foreground, and the other areas are background subjects. . The outline of a background subject located in a distant view becomes unclear due to light scattering or the like. On the other hand, a human subject located in a near view is clearly imaged. The area of the foreground human subject is the area most watched by the user, and the noise is more conspicuous, so that it is easily recognized as image quality degradation.

Therefore, the processing in the image processing unit 102 increases the strength of the contour emphasis process as the distance to the subject (depth information) increases, thereby preventing the foreground subject from being emphasized and preventing the distant view subject from being blurred. Can be emphasized.
For example, the image processing unit 102 performs the contour emphasis process so as to enhance the contour of the input image represented by the image data Gi as the value of the image processing strength S determined by the image processing strength calculation unit 104 is larger. The contour enhancement process is a process of converting the brightness value so that, for example, the difference in brightness value between adjacent pixels becomes larger. At this time, the outline is enhanced as the difference in brightness value increases. These can be realized by a spatial filter in consideration of neighboring pixels in the vicinity of 4 or 8 neighborhoods. For example, in the case of considering four neighborhoods in which the pixel value of the target pixel is P0 and the pixel values of peripheral pixels are P1, P2, P3, and P4, the pixel value P after image processing is P = P0 + (P0 × 4-P1) -P2-P3-P4) * S. Here, S is the image processing intensity S of the target pixel calculated by the image processing intensity calculation unit 104.

Thus, the image processing unit 102 can generate a clearer image by enhancing the contour of the input image as the image processing strength S increases. As a result, the image processing apparatus 100 can improve the sense of depth of the image by making the distant view of the image clear and making it easier for the user who viewed the image to feel the sense of distance between the foreground and the distant view. . As a result, the image processing apparatus 100 can make the user feel the depth.
At this time, for example, when the focal length increases, the intensity correction value increases and the image processing intensity S also increases. Therefore, since the image processing unit 102 performs stronger edge enhancement when the focal length increases and the amount of blur outside the depth of field increases, it is possible to suppress blurring of the subject and blurring of the outline. .

  Further, in a distant view, the contrast decreases due to light scattering. Therefore, the processing in the image processing unit 102 may perform contrast correction that enhances contrast using a tone curve or the like. However, a human subject in the foreground is captured with sufficient contrast and sufficient gradation, and if excessive contrast correction is performed, the number of gradations is reduced and the image quality deteriorates.

  Therefore, the image processing unit 102 may perform contrast correction processing so that the contrast of the image data Gi increases as the image processing strength S increases. Here, the contrast correction process is, for example, a process of correcting the brightness value to be larger if the brightness value is large, and correcting to decrease the brightness value if the brightness value is small. is there. At this time, the larger the correction amount, the stronger the contrast correction process. These can be realized by an LUT that defines a corrected value for an input value.

  As a result, the image processing unit 102 can generate an image with improved contrast. As a result, the image processing apparatus 100 can make the sense of distance between the near view and the distant view easier by clarifying the distant view of the image, so that the sense of depth of the image can be improved. As a result, the image processing apparatus 100 can make the user who sees the image feel the depth. As described above, when contrast enhancement is performed in the image processing unit 102, as in the case of contour enhancement, the amount of blur increases according to the focal length, and it is possible to suppress a decrease in contrast.

  Also, it is preferred that the image has higher saturation due to the memory color and vividness, and it feels like the real thing. However, a person's skin or the like needs to be a skin color. If excessive saturation enhancement is performed, a sense of incongruity occurs and image quality deteriorates. Therefore, the image processing unit 102 may perform saturation correction on the image data Gi based on the image processing intensity S determined by the image processing intensity calculating unit 104. Here, saturation correction processing is performed by multiplying or adding saturation in a HSV (Hue, Saturation, Value) space composed of three components of hue, saturation, and brightness, or by linearly transforming input pixel values using a matrix. It can be realized by doing. Accordingly, the image processing unit 102 can change the saturation based on the depth information D. As a result, the image processing unit 102 can generate an image with enhanced saturation without a sense of incongruity by enhancing the saturation of the distant view of the image.

  As a result, the image processing apparatus 100 can enhance the sense of depth of the image by increasing the saturation of the distant view of the image and making the distant view clearer so that the sense of distance between the foreground and the distant view can be easily felt. Can be made. As a result, the image processing apparatus 100 can make the user who sees the image feel the depth. As described above, when the saturation correction process is performed in the image processing unit 102, as in the case of edge enhancement, the amount of blur increases according to the focal length, and a decrease in saturation can be suppressed.

Even in the first modification described above, when the image processing unit 102 performs contour enhancement processing, contrast enhancement, and saturation correction processing, it is possible to suppress the influence of the change in the amount of blur due to the change of the aperture (F value). it can.
In the second modification described above, when the image processing unit 102 performs the blurring process or the noise removal process, it is possible to suppress the influence of the image deterioration due to the change in ISO sensitivity. In the third modification described above, when the image processing unit 102 performs the blurring process or the noise removal process, it is possible to suppress the influence of the image deterioration due to the change in the shutter speed.

(Processing procedure of the first embodiment)
Hereinafter, the processing procedure in the present embodiment will be described with reference to the flowchart of FIG. FIG. 10 is a flowchart illustrating an example of processing of the image processing apparatus 100 according to the present embodiment. First, the intensity correction value determination unit 103 calculates an intensity correction value from the imaging device setting information P (step S101). Next, the image processing intensity calculation unit 104 determines the image processing intensity S based on the depth information D and the intensity correction value, and sets a correspondence relationship between the depth information D and the image processing intensity S (step S102). . Next, the image processing intensity calculation unit 104 calculates the image processing intensity S at each pixel by applying the depth information D at each pixel to the set correspondence (step S103). Finally, the image processing unit 102 performs image processing on the image data Gi of each pixel based on the image processing intensity S at each pixel determined by the image processing intensity calculation unit 104 (step S104). Above, the process of this flowchart is complete | finished.

When the image represented by the image data Gi is a moving image, the process shown in FIG. 10 is performed on each frame of the moving image. Further, the imaging apparatus setting information P may have a value for each frame, or may have a value for each predetermined number of frames.
As described above, the image processing apparatus 100 according to the present embodiment calculates the intensity correction value based on the imaging apparatus setting information P. The image processing apparatus 100 determines the image processing intensity S according to the depth information D and the intensity correction value, and determines the image processing intensity S for each pixel. Then, the image processing apparatus 100 performs predetermined image processing on the pixel with the image processing intensity S determined for each pixel.

  Thereby, the image processing apparatus 100 changes the image processing intensity S in accordance with the imaging apparatus setting information P within the range of the depth information D, so that the imaging apparatus setting information such as the imaging apparatus setting information changes. The influence of the change in the setting can be suppressed and the image area of the distant view can be made clear. As a result, it is possible to suppress the influence of the change in the setting of the imaging device on the distance between the near view and the distant view that the user sees from the image. That is, the image processing apparatus 100 can generate an image that allows the user to feel the depth.

<Second Embodiment>
Hereinafter, a second embodiment of the present invention will be described in detail with reference to the drawings. In the first embodiment, the method of determining the image processing intensity S based on the depth information D and the imaging device setting information P and performing image processing in the image processing apparatus 100 of FIG. 1 has been described. The image processing apparatus 100a in this embodiment includes a depth information analysis unit 105 as shown in FIG. 11, and the image processing intensity calculation unit 104a includes the depth information analysis unit 105 in addition to the intensity correction value and the depth information D. The image processing intensity S is determined using the result of the analysis.

  FIG. 11 is a schematic block diagram showing the configuration of the image processing apparatus 100a in the present embodiment. In the image processing apparatus 100a, the depth information analysis unit 105 is added to the image processing intensity determination unit 101 in the image processing apparatus 100, and the image processing intensity calculation unit 104 is changed to the image processing intensity calculation unit 104a. The unit 101a is provided. Elements that are the same as those in FIG. 1 are denoted by the same reference numerals (102, 103), and detailed description thereof is omitted.

The image processing apparatus 100a inputs the depth information D to the depth information analysis unit 105, and inputs the analysis result of the depth information analysis unit 105 to the image processing intensity calculation unit 104a.
The depth information analysis unit 105 extracts information related to the frequency distribution of the depth information D based on the depth information D corresponding to the image data Gi. Specifically, for example, the depth information analysis unit 105 extracts a plurality of depth information values according to a predetermined rule based on the frequency distribution of the depth information D corresponding to one input image data. Here, the plurality of depth information values are different depth information D. Then, the depth information analysis unit 105 outputs information on the frequency distribution of the extracted depth information D, for example, a plurality of depth information values, to the image processing intensity calculation unit 104a. Details of the processing of the depth information analysis unit 105 will be described later.

  The image processing intensity calculation unit 104a determines the image processing intensity S for each pixel of the image data Gi based on the output information of the depth information analysis unit 105, the output information of the intensity correction value determination unit 103, and the depth information D. To do. The image processing intensity calculation unit 104 a outputs the determined image processing intensity S to the image processing unit 102.

<Details of Processing of Depth Information Analysis Unit 105>
Details of the processing of the depth information analysis unit 105 will be described below.
FIG. 12 is a schematic block diagram of the depth information analysis unit 105 in the present embodiment. The depth information analysis unit 105 includes a frequency distribution calculation unit 111 and a depth information extraction unit 112. The frequency distribution calculation unit 111 calculates the frequency distribution of the depth information of the depth information D input from the outside of the image processing apparatus 100a, and outputs the calculated frequency distribution to the depth information extraction unit 112. The depth information extraction unit 112 extracts a plurality of depth information values from the depth information D in the frequency distribution input from the frequency distribution calculation unit 111 within a range where the frequency is equal to or higher than the threshold value α. Then, the depth information extraction unit 112 outputs the extracted plurality of depth information values to the image processing intensity calculation unit 104a.

  In the following description, a case where the depth information extraction unit 112 extracts two depth information values of a first depth information value and a second depth information value as a plurality of depth information values will be described as an example. Here, the first depth information value and the second depth information value will be described with reference to FIG. FIG. 13 is a diagram illustrating an example of a frequency distribution of depth information values. In the figure, the horizontal axis represents the depth information value, and the vertical axis represents the frequency. In the same figure, depth information Dobj with the maximum frequency is shown. In the same figure, the minimum value of the depth information D (hereinafter referred to as the range minimum value) Dmin in the range where the frequency is equal to or higher than the threshold value α, and the maximum value of the depth information D (hereinafter referred to as Dmax) (referred to as the maximum value of the range). At this time, the depth information extraction unit 112 sets the range minimum value Dmin as the first depth information value, and sets the range maximum value Dmax as the second depth information value.

  Next, processing of the depth information analysis unit 105 will be described with reference to FIG. For example, the depth information analysis unit 105 extracts the first depth information and the second depth information using the frequency distribution of the depth information D shown in FIG. Here, when the maximum value and the minimum value are extracted by the normal method, even if the frequency is smaller than the predetermined threshold value α, the maximum value and the minimum value are extracted, and noise exists in the depth information. May cause mis-extraction.

  In the present embodiment, the depth information analysis unit 105 extracts the range minimum value Dmin as the first depth information in the frequency distribution of one depth information. Further, the depth information analysis unit 105 extracts the range maximum value Dmax as second depth information in one distribution of depth information. The depth information analysis unit 105 outputs the extracted first depth information and second depth information to the image processing intensity calculation unit 104a.

Thereby, the depth information analysis part 105 can extract 1st depth information and 2nd depth information stably.
Further, the depth information analysis unit 105 may estimate the position of the main subject and extract the depth information of the main subject. For example, it is possible to estimate that the main subject exists in the depth information (mode) Docj having the largest frequency value among the maximum values of the depth information, and the depth information can be extracted.

  Further, the depth information analysis unit 105 divides the image data Gi into an arbitrary region (for example, a region divided into four parts in the vertical direction and four parts in the horizontal direction for a total of 16 parts), and calculates the frequency distribution of the depth information for each region. The position of the subject may be estimated. In this case, for example, in the case of a subject that is biased to a specific area, the subject may not appear as a maximum value in the frequency distribution of the entire depth information, but the subject may appear as a maximum value in the frequency distribution of the divided area. is there. Thereby, the depth information analysis unit 105 can estimate the depth information of each subject in the image data Gi. At the same time, it is possible to estimate the approximate position of each subject on image coordinates depending on which region the subject belongs to among the regions obtained by dividing the image data Gi.

  The depth information analysis unit 105 outputs the plurality of depth information extracted in this way to the image processing intensity calculation unit 104a. In addition, the depth information analysis unit 105 can output position information on the image coordinates of each subject to the image processing intensity calculation unit 104a.

<Details of Processing of Image Processing Intensity Calculation Unit 104a>
The depth information values that are actually input as the depth information D are distributed in a part of the range that the gradation value of the depth information can take as shown in FIG. Therefore, there is a possibility that the effect of image processing cannot be obtained sufficiently. Therefore, the image processing intensity calculation unit 104a determines the intensity base value using a plurality of depth information calculated by the depth information analysis unit 105.

  Hereinafter, an example in which two depth information values of a first depth information value and a second depth information value are used as a plurality of depth information values will be described. The image processing intensity calculation unit 104a changes the intensity base value in a desired range between the first depth information value and the second depth information value. Here, in the example of this embodiment, the range minimum value Dmin is the first depth information value, and the range maximum value Dmax is the second depth information value.

  FIG. 14 is a diagram illustrating an example of a correspondence relationship between the depth information value and the intensity base value. In the figure, the horizontal axis is the depth information value, and the vertical axis is the intensity base value. In the figure, the intensity base value is β1 at the range minimum value Dmin, and the intensity base value is β2 at the range maximum value Dmax. Further, in the figure, in the range from the range minimum value Dmin to the range maximum value Dmax, there is shown a characteristic that the intensity base value increases linearly as the depth information value increases. When the depth information value is Dmin or less, the intensity base value is constant at β1, and when the depth information value is Dmax or more, the intensity base value is constant at β2.

  For example, the image processing intensity calculation unit 104a determines the intensity base value based on the characteristics shown in FIG. Specifically, when changing the intensity base value from β1 to β2, the image processing intensity calculation unit 104a sets β1 as the range minimum value Dmin and β2 as the range maximum value Dmax, and the depth information value therebetween The intensity base value at is set continuously. Thereby, the image processing intensity calculation unit 104a sets the correspondence between the depth information value and the intensity base value. After the intensity base value is determined by the above method, the image processing intensity S is determined by correcting the intensity base value based on the intensity correction value in the same manner as described in the details of the processing of the image processing intensity calculation unit 104. To do.

(For videos)
The above-described image processing apparatus 100a has been described with respect to the image processing of the image data Gi corresponding to one still image. However, the image processing apparatus 100a can be applied to a moving image, and the same effect can be obtained. In the processing with a moving image, the image processing intensity calculation unit 104a considers the extraction result of the depth information analysis unit 105 in consideration of the result of the temporally previous frame with respect to the analysis target frame.

  For example, when the depth information analysis unit 105 analyzes the depth information D in a certain target frame, the maximum range value is Dmax1, the minimum range value is Dmin1, and the depth information of the main subject is Dobj1. The case will be described. In the following case, as a result of the depth information analysis unit 105 analyzing the depth information D in the frame immediately before the target frame, the maximum range value is Dmax0, the minimum range value is Dmin0, and the depth information of the main subject is Dobj0. This is the case.

  In this case, the depth information analysis unit 105 sets (Dmax1 + Dmax0) / 2 as the maximum range value to be finally output and (Dmin1 + Dmin0) / 2 as the minimum range value to be finally output. (Dobj1 + Dobj0) / 2 is calculated as depth information. Then, the depth information analysis unit 105 outputs the calculated parameters to the image processing intensity calculation unit 104a.

Thereby, the image processing intensity calculation unit 104a can reduce the variation between the frames of each parameter due to the noise included in the depth information D, and can reduce the flicker of the image indicated by the output image data Go.
Each parameter calculated by the depth information analysis unit 105 is not limited to an average value, and may be a median value or may be calculated by a separate calculation method for each parameter. Specifically, for example, the depth information analysis unit 105 calculates the minimum range value to be finally output as an average value of the minimum range values in the immediately preceding frame and the target frame, and the depth information of the main subject to be finally output is You may calculate with the median of the depth information of the main subject in the immediately preceding frame and the target frame.

  Also, the frame that takes the average of each parameter of the target frame is not limited to the immediately preceding frame, but may be a frame that is a plurality of frames before, a frame that is after the target frame, or a frame that is different from the target frame. Moreover, although the depth information analysis part 105 took the average in two frames, you may take an average not only in this but in three or more frames. Furthermore, when the scene of the image does not change greatly between frames, the delay of image processing can be reduced by using the analysis results of the target frame for the last few frames.

  That is, the depth information analysis unit 105 calculates information related to the frequency distribution of depth information in the target frame and information related to the frequency distribution of at least one depth information in a frame before the target frame. Here, the information on the frequency distribution is information calculated from the frequency distribution, and is, for example, the range minimum value, the range maximum value, the depth information of the main subject, or the intermediate depth information. The plurality of depth information calculated by the depth information analysis unit 105 may be output to the image processing intensity calculation unit 104a.

  It should be noted that the depth information analysis unit 105 may determine that the scene has changed when the average brightness, contour strength, color distribution, or the like of the image data has changed significantly. In this case, the depth information analysis unit 105 may calculate each parameter using only the calculation result of the target frame without using the calculation result of the previous frame. As a result, the image processing apparatus 100 does not refer to the parameter of the previous frame when the scene changes, so that the image corresponding to the depth information D of the current scene is not affected by the parameter of the previous scene. Image processing can be performed with the processing intensity S.

(When using the depth value of the main subject)
In the above description, a method of determining the image processing intensity S based on a plurality of depth information values (for example, the range minimum value Dmin, the range maximum value Dmax) extracted based on the depth information D and the intensity correction value will be described. However, the depth information analysis unit 105 may extract the depth information value of the main subject in the foreground and use the value to determine the image processing strength S by the image processing strength calculation unit 104a. Here, the image data Gi is an image centered on the main subject, and image processing that emphasizes the main subject can be performed by performing image processing on the difference in depth information D from the main subject.

  FIG. 15 is a diagram illustrating an example of a correspondence relationship between the depth information value and the intensity base value when image processing with higher intensity is performed on a subject located behind the main subject. In the figure, the horizontal axis is the depth information value, and the vertical axis is the intensity base value. Further, in the drawing, it is shown that the intensity base value is β1 in the depth information Dobj of the main subject, the intensity base value is β2 at the maximum range value Dmax. Further, in the figure, there is shown a characteristic that the intensity base value increases linearly as the depth information value increases in the range from the depth information Doobj of the main subject to the maximum range value Dmax.

As described above in the description of the processing of the depth information analysis unit 105, the depth information Dobj and the range maximum value Dmax of the main subject are extracted from the frequency distribution of the depth information D, and the extracted depth information Dobj and the range maximum value Dmax of the main subject are extracted. Is output to the image processing intensity calculation unit 104a.
When the intensity base value is changed from β1 to β2, the image processing intensity calculation unit 104a sets the intensity base value so that the depth information D becomes β1 when the main subject's depth information Dob becomes β1 and the range maximum value Dmax becomes β2. The intensity base value corresponding to the depth information D between them is set continuously.

That is, in this case, the depth information Doobj of the main subject extracted by the depth information analysis unit 105 is the first depth information value, and the maximum range value Dmax is the second depth information value. The image processing intensity calculation unit 104a then increases the intensity base value as the intensity base value becomes the largest in the second depth information value, the intensity base value becomes the smallest in the first depth information, and the depth information becomes larger. Is determined to be large.
The intensity base value determined in this way is corrected based on the intensity correction value by the same method as that described above, and the final image processing intensity S is determined.

Thereby, the image processing intensity calculation unit 104a sets the image processing intensity according to the depth information Doobj of the main subject of the image data.
Here, when the main subject is a person, the depth information analysis unit 105 may acquire depth information Dobj of the main subject by combining face recognition techniques. For example, when a face is detected by face recognition, depth information D corresponding to the detected face area is set as the depth information Dobj of the main subject. At this time, the depth information analysis unit 105 sets the depth information D corresponding to the detected face area as an average value of the predetermined area. Thereby, the depth information analysis part 105 can reduce the influence of the noise etc. which are contained in the depth information D by taking an average value.

  Furthermore, the image processing intensity calculation unit 104a may increase the image processing intensity S toward the range maximum value Dmax, starting from the depth information value between the depth information Dobj of the main subject and the range maximum value Dmax. Here, main subjects such as people and animals have thickness. Therefore, the depth information Dobj of the main subject calculated by the depth information analysis unit 105 is any one of the depth information D corresponding to each pixel included in the image area of the main subject.

  Therefore, in order to start image processing from a subject located behind the main subject, the image processing intensity calculation unit 104a starts from a depth information value that is larger than the calculated depth information Doobj of the main subject by a predetermined value. As an example, the intensity base value may be increased toward the range maximum value Dmax. At this time, for example, the image processing intensity calculation unit 104a may set the intensity base value using the correspondence relationship between the depth information value and the intensity base value as shown in FIG. Also in this case, the set intensity base value is corrected based on the intensity correction value by a method similar to the process described above, and the final image processing intensity S is determined.

  FIG. 16 is a diagram illustrating an example of a correspondence relationship between the depth information value and the intensity base value when image processing with higher intensity is performed on a subject located behind the main subject. In the figure, the horizontal axis is the depth information value, and the vertical axis is the intensity base value. In the figure, D0 is depth information (hereinafter referred to as intermediate depth information) located between the depth information Doobj of the main subject and the range maximum value Dmax. In the drawing, it is shown that the intensity base value is β1 in the intermediate depth information D0, and the intensity base value is β2 in the range maximum value Dmax. Further, in the figure, in the range from the intermediate depth information D0 to the maximum range value Dmax, there is shown a characteristic that the intensity base value increases linearly as the depth information value increases.

  The depth information analysis unit 105 calculates the range maximum value Dmax and the intermediate depth information D0. For example, the depth information analysis unit 105 calculates the intermediate depth information D0 by adding a predetermined value to the depth information Dobj of the main subject. The maximum range value Dmax is extracted by the method described in the details of the processing of the depth information analysis unit 105. Then, the depth information analysis unit 105 outputs the range maximum value Dmax and the intermediate depth information D0 to the image processing intensity calculation unit 104a.

  The depth information analysis unit 105 multiplies the depth information Dobj of the main subject by a predetermined value, sets the average value of the depth information Dobj of the main subject and the range maximum value Dmax, or the depth information Dobj of the main subject and the range. The intermediate depth information D0 may be calculated by, for example, setting a value that is mixed with the maximum value Dmax at a predetermined ratio.

When changing the intensity base value from β1 to β2, the image processing intensity calculation unit 104a sets β1 as the intermediate depth information D0 and β2 as the range maximum value Dmax, and sets the intensity base value for the depth information value therebetween. Set continuously.
That is, in this case, the intermediate depth information D0 calculated by the depth information analysis unit 105 is the first depth information, and the range maximum value Dmax is the second depth information. Then, the image processing intensity calculation unit 104a has the largest intensity base value in the second depth information, the smallest intensity base value in the first depth information, and the first depth information and the second depth information. The intensity base value is linearly increased as the depth information value increases in the range.
The final image processing intensity S is determined by correcting the intensity base value determined in this way based on the intensity correction value by the same method as that described above.

  Note that the image processing intensity calculation unit 104a may select the characteristics shown in FIG. 5, the characteristics shown in FIG. 14, the characteristics shown in FIG. 15, and the characteristics shown in FIG. For example, the image processing intensity calculation unit 104a determines the image processing intensity S using the characteristics of FIG. 5 when there is no subject in the foreground or when there is no person as the subject. Further, the image processing intensity calculation unit 104a uses the characteristics shown in FIG. 16 when a subject or person in the foreground exists as a subject. Further, the image processing intensity calculation unit 104a uniformly sets the image processing intensity S in advance regardless of the depth information D when the parallax range is partially biased, for example, in the case of an image of only a near view or an image of only a distant view. It may be a predetermined value. As a result, the image processing apparatus 100 switches the determination method of the image processing intensity S according to the scene, and thus can perform appropriate image processing according to the scene.

  Based on the image processing intensity S determined by the above method, the image processing unit 102 performs image processing on the image data Gi, so that the image processing intensity calculation unit 104a can detect the object located behind the main object. The image processing intensity S corresponding to the depth information D and the imaging device setting information P is determined for the image region or the image region of the subject located further from the back position by a predetermined value than the main subject. be able to. At this time, the image processing unit 102 performs the image processing with the image processing intensity S corresponding to the difference in the depth information D from the main subject. Therefore, the main subject is emphasized, and the main subject and the background are displayed to the user who viewed the image. This makes it easier to feel a sense of distance. That is, the image processing apparatus 100a can generate an image that can make the user feel the depth.

  In the present embodiment, a method using the image processing apparatus 100a including the depth information analysis unit 105 and the image processing intensity calculation unit 104a is shown as an example for the sake of explanation, but the present invention is not limited to this. The image processing apparatus 100 of FIG. 1 can be used, and in this case, the image processing intensity calculation unit 104 can perform the processing of this embodiment based on the depth information D.

[Third Embodiment]
Hereinafter, the third embodiment of the present invention will be described in detail. In each of the above-described embodiments, the image processing apparatus 100 or the image processing apparatus 100a increases the image processing intensity as the depth information value increases (as the distant view increases), and further increases the intensity according to the imaging apparatus setting information P such as the focal length. The image processing intensity was corrected by changing the correction value. Furthermore, by performing contour enhancement processing, contrast correction, saturation correction, and the like on each pixel of the image data Gi according to the corrected image processing intensity, it is possible to generate an image that can make the user feel the depth. Indicated.

  In the present embodiment, for example, the determination method of the image processing intensity S of the image processing intensity calculation unit 104 or the image processing intensity calculation unit 104a in the image processing apparatus 100 or the image processing apparatus 100a is changed, or the image processing unit 102 Change the image processing content or both. Thereby, it is possible to perform an image effect corresponding to the depth information D and the imaging device setting information P on the image data Gi.

Hereinafter, as an example, a method of producing an image effect by blurring processing using the image processing apparatus 100a illustrated in FIG. 11 will be described.
Here, the blurring process is a process of blurring an image by performing a filter process using an average value filter or a Gaussian filter on the target pixel and its peripheral pixels on the input image.
First, for example, the image processing intensity S is determined in the same manner as the method described in the second embodiment.

  That is, the depth information analysis unit 105 extracts the first depth information value and the second depth information value, and the image processing intensity is maximized in the second depth information value, and image processing is performed in the first depth information value. It sets so that intensity | strength base value may become large, so that intensity | strength becomes the smallest and a depth information value becomes large between a 1st depth information value and a 2nd depth information value. Further, the intensity correction value determination unit 103 calculates the intensity correction value, and the image processing intensity S is determined by correcting the intensity base value so that the image processing intensity increases as the intensity correction value increases.

  Then, based on the depth information D input from the outside of the image processing apparatus 100a, the image processing intensity S in each pixel of the image data Gi is determined. The image processing intensity calculation unit 104 a outputs the determined image processing intensity S to the image processing unit 102. Next, the image processing unit 102 performs a blurring process on the image data Gi with an intensity corresponding to the input image processing intensity S. That is, the blurring process is performed so that the intensity of the blurring process increases as the image processing intensity S increases. Therefore, the intensity of the blurring process gradually increases from the first depth information to the second depth information, and an image blurred as the distant view can be generated. Then, the image processing unit 102 outputs the blurred image data as output image data Go to the outside of the image processing apparatus 100a.

  Accordingly, the image processing apparatus 100a according to the present embodiment determines the image processing intensity S based on the depth information D and the imaging apparatus setting information P, and performs image processing according to the image processing intensity S, thereby obtaining the depth information and It is possible to generate an image with a blurring effect according to the imaging device setting information. For example, when the focal length is increased, the effect of reducing the depth of field can be provided by performing the blurring process, and an impression that the focal length is further increased can be provided.

  In the above description, the intensity base value is determined so that the image processing intensity increases as the depth information value increases, that is, the distant view increases. Further, the image processing intensity increases as the intensity correction value increases. Intensity S was determined. However, the method for determining the image processing intensity S is not limited to this. That is, the relationship between the magnitude of the imaging device setting information P and the magnitude of the intensity correction value in the case of the first embodiment, the first modification of the first embodiment, and the fourth modification is reversed. Good.

  In the above description, the blurring process is described as an example of the image process, but the present invention is not limited to this. For example, the above-described edge enhancement processing, contrast correction, or saturation correction may be performed as image processing, processing for changing brightness (luminance), contour enhancement processing, processing for converting an image from color to gray scale, or the like. It may be used.

  Specifically, for example, by setting so that the depth information value is large, that is, the saturation decreases with increasing distance, the foreground subject is colored, and the distant subject is black and white. An image effect that enhances the image quality becomes possible. Also, based on the information on the in-focus position, the subject with the in-focus position (zoom-up position) is set to lower the saturation as the subject has a larger difference between the depth information value of the in-focus position and the depth information value. It is possible to produce an image that enhances the attractiveness of the image.

Further, by combining the contour extraction process and the gray scale conversion process, an image called a line drawing in which only the contour portion of the subject is left can be obtained. Using this, for example, by changing the image processing intensity based on the depth of field information and making the subject outside the depth of field a line drawing, it is possible to produce an image effect that emphasizes only the subject within the depth of field It becomes.
In addition to the image processing as described above, various image processing such as monochrome and sepia can be applied, and the image corresponding to the depth information D and the imaging device setting information P can be obtained by the above processing. Production is possible.

  As a result, the image processing apparatus 100a described in the present embodiment determines the image processing intensity S based on the depth information D and the imaging apparatus setting information P, and performs image processing according to the image processing intensity S. It is possible to generate an image with a presentation effect according to the information D and the imaging device setting information P.

  In the present embodiment, a method using the image processing apparatus 100a including the depth information analysis unit 105 and the image processing intensity calculation unit 104a is shown as an example for the sake of explanation, but the present invention is not limited to this. The image processing apparatus 100 of FIG. 1 can be used, and in this case, the image processing intensity calculation unit 104 may perform the processing of the present embodiment based on the depth information D.

  Note that the blocks of the image processing intensity calculation unit 104, the image processing intensity calculation unit 104a, the image processing unit 102, the intensity correction value determination unit 103, and the depth information analysis unit 105 described in the above embodiment are FPGA (Field-). You may comprise by hardware, such as Programmable Gate Array, or the software processed by the microcomputer (not shown) etc. which exist in the image processing apparatus 100 or the image processing apparatus 100a.

[Fourth Embodiment]
Hereinafter, a fourth embodiment of the present invention will be described in detail with reference to the drawings. FIG. 17 is a schematic block diagram of an imaging apparatus 200 according to the fourth embodiment. Elements common to those in FIG. 1 are denoted by the same reference numerals (101 to 104), and a specific description thereof is omitted. The imaging device 200 includes an imaging unit control unit 201, an imaging unit 202, an imaging unit 203, an image processing device 100b, an image display unit 205, and an image storage unit 206. In addition, the configuration of the image processing apparatus 100b in FIG. 17 is obtained by adding a parallax calculation unit 204 to the configuration of the image processing apparatus 100 according to the first embodiment in FIG.

The imaging unit control unit 201 outputs a control signal for controlling the imaging units 202 and 203. Here, the control signal is imaging apparatus setting information P such as a focal length, F value, ISO sensitivity, and shutter speed, and is output to the imaging unit 202, the imaging unit 203, and the intensity correction value determination unit 103.
The two imaging devices 202 and 203 are arranged side by side or vertically so that their optical axes are substantially parallel. The imaging units 202 and 203 include an optical system such as a lens module and an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). In addition, the imaging units 202 and 203 change the focal length of the lens, the aperture, the ISO sensitivity of the image sensor, the shutter speed, the subject to be focused, and the like according to the imaging device setting information P output from the imaging unit control unit 201. To do.

The imaging unit 202 images a subject and outputs first image data G1 obtained by imaging to the parallax calculation unit 204 and the image processing unit 102. Similarly, the imaging unit 203 images a subject and outputs the second image data G <b> 2 obtained by the imaging to the parallax calculation unit 204.
The parallax calculation unit 204 calculates the parallax corresponding to each pixel of the input standard image based on the input standard image and the input reference image having a different viewpoint from the input standard image, and determines the depth information D based on the calculated parallax. . Here, the parallax is the amount of subject displacement between two images. Specifically, for example, the parallax calculation unit 204 uses the first image data G1 input from the imaging unit 202 and the second image data G2 input from the imaging unit 203 to perform the first image by block matching. A parallax value corresponding to each pixel of the data G1 is calculated.

  Here, the block matching performs the following processing. Pixel matching is performed on a pixel in one image data by scanning the other image data in the horizontal direction. Pixel matching is performed in units of blocks centering on the target pixel, and SAD (Sum of Absolute Difference) that calculates the sum of absolute value differences of pixels in the block is calculated to determine a block that minimizes the SAD value. Thus, the pixel on the other captured image corresponding to the target pixel of the one captured image is obtained. In addition to the SAD calculation method, there are calculation methods such as block matching by SSD (Sum of Squared Intensity Difference), graph cut, and DP (Dynamic Programming) matching. When the corresponding pixel is obtained, the parallax value of the pixel can be calculated.

  Here, the relationship between the distance Z to the captured subject and the parallax d is expressed by d = f × B / Z. f is a focal length of the imaging units 202 and 203, and B is a baseline length which is a distance between the imaging unit 202 and the imaging unit 203. From the relational expression between the distance Z and the parallax d, there is a correlation between the distance Z and the parallax d. In the present embodiment, the parallax calculation unit 204 of the image processing apparatus 100b calculates the depth information D based on the parallax d.

FIG. 18 is a diagram illustrating the relationship between the distance Z and the parallax d. As shown in the figure, the distance Z and the parallax d are in an inversely proportional relationship and not a linear relationship. Therefore, the parallax calculation unit 204 converts the parallax d so that the relationship with the distance is linear, and uses the converted value as the depth information D. Specifically, for example, the parallax calculation unit 204 calculates the reciprocal (1 / d) of the calculated parallax d and sets the reciprocal of the calculated parallax d as the depth information D. Then, the parallax calculation unit 204 outputs the calculated depth information D to the image processing intensity calculation unit 104.
Note that the parallax conversion need not be in a completely linear relationship with the distance, but may be a conversion close to that.

  Further, the parallax calculation unit 204 may use the calculated parallax d as it is as the depth information D. In this case, the depth information D is smaller as the distance to the subject is longer, and is larger as the distance is shorter. For this reason, the image processing intensity calculation unit 104 may increase the image processing intensity S as the value of the depth information D decreases. Accordingly, the image processing intensity calculation unit 104 can increase the image processing intensity S as the depth information D indicates the back.

  The image processing intensity calculation unit 104 determines the image processing intensity S based on the depth information D input from the parallax calculation unit 204 and the intensity correction value input from the intensity correction value determination unit 103. The image processing unit 102 performs image processing on the first image data G <b> 1 input from the imaging unit 202 in accordance with the image processing intensity S determined by the image processing intensity calculation unit 104. The image processing unit 102 displays the output image data Go obtained by the image processing on the image display unit 205 or stores it in the image storage unit 206.

  As described above, the imaging apparatus 200 according to the present embodiment calculates the depth information D from the two image data. Further, the intensity correction value is calculated based on the imaging device setting information P. The imaging apparatus 200 determines the image processing intensity S based on the depth information D and the intensity correction value, and determines the image processing intensity S for each pixel of the first image data G1. The image capturing apparatus 200 performs image processing determined in advance for each pixel with the image processing intensity S determined for each pixel, so that even when the image capturing apparatus setting information changes, the image area of the distant view is displayed. It can be clear. As a result, the user who viewed the image can easily feel the distance between the near view and the distant view, and the imaging apparatus 200 can generate an image that allows the user to feel the depth.

  In the present embodiment, the imaging apparatus 200 including two imaging units has been described. However, a similar effect can be obtained by a method of calculating parallax information from a plurality of images. For example, the horizontal direction from the first imaging position is obtained. This can be realized by moving to and performing the second imaging. Further, a depth information calculation unit to be described later may be provided instead of the parallax calculation unit 204.

Note that the imaging apparatus 200 may include a depth information analysis unit 105 and an image processing intensity calculation unit 104a, and the image processing method described in the second embodiment can be applied. Thereby, when a moving image is imaged by the imaging apparatus 200, fluctuations between frames of each parameter due to noise included in the depth information D can be reduced, and flickering of an image indicated by the output image data Go can be reduced. it can. Furthermore, by performing image processing with image processing intensity corresponding to the difference in depth information D from the main subject, it is possible to emphasize the main subject and make it easier for the user to feel the distance between the main subject and the background. An image can be generated.
The imaging apparatus 200 can apply the image processing method described in the third embodiment. As a result, it is possible to generate an image with a presentation effect according to the depth information and the imaging device setting information.

[Fifth Embodiment]
Hereinafter, a fifth embodiment of the present invention will be described in detail with reference to the drawings. FIG. 19 is a schematic block diagram of a display device 300 according to the fifth embodiment. In addition, the same code | symbol (101-104,205) is attached | subjected to the element which is common in FIG. 1 or FIG. 17, and the specific description is abbreviate | omitted. The display device 300 includes an image processing device 100 c and an image display unit 205. The configuration of the image processing apparatus 100c in FIG. 19 is a configuration in which a depth information calculation unit 301 is added to the configuration of the image processing apparatus 100 of the first embodiment in FIG.

  The depth information calculation unit 301 calculates depth information D from image data Gi input from the outside of the own device. Here, any conventional estimation method may be used for the calculation of the depth information D. For example, the depth information calculation unit 301 generates the depth information D by generating a 3D image from the 2D image by color information, vanishing point analysis, region division, object extraction, and the like. The depth information calculation unit 301 outputs the calculated depth information D to the image processing intensity calculation unit 104.

Note that the imaging device setting information P may be included in the image data Gi. For example, a JPEG format image stores focal length, F value, ISO sensitivity, shutter speed, and the like as Exif (Exchangeable image file format) data. Information added to the image data in this way can be used as the imaging device setting information P.
The image processing unit 102 has the same function as the image processing unit 102 of the first embodiment, but differs in the following points. The image processing unit 102 causes the image display unit 205 to display an output image obtained by image processing.

  Thereby, the display apparatus 300 calculates the depth information D based on the image data Gi. Further, the intensity correction value is calculated based on the imaging device setting information P. The display device 300 determines the image processing intensity S based on the depth information D and the intensity correction value, and determines the image processing intensity S for each pixel of the image data Gi. The display device 300 performs image processing predetermined for each pixel with the image processing intensity S determined for each pixel, so that the image area of the distant view can be displayed even when the imaging device setting information changes. It can be clear. As a result, the user who viewed the image can easily feel the distance between the near view and the distant view, and the display device 300 can display an image that allows the user to feel the depth.

  In the present embodiment, the display device 300 including the depth information calculation unit 301 has been described. However, when image data having information such as a stereoscopic video is input, the parallax is calculated from the stereoscopic video, and the parallax is calculated. The depth information D may be calculated based on the image information, and the image processing intensity S may be determined based on the calculated depth information D. Even in that case, the display device 300 can obtain the same effect as described above. Further, when the depth information D is input to the display device 300 together with the image data, the display device 300 may input the depth information D directly to the image processing intensity calculation unit 104.

Note that the display device 300 may include a depth information analysis unit 105 and an image processing intensity calculation unit 104a, and the image processing method described in the second embodiment can be applied. As a result, when a moving image is input as image data, it is possible to reduce the variation between the frames of each parameter due to noise included in the depth information D, and to reduce the flicker of the image indicated by the output image data Go. . Furthermore, by performing image processing with image processing intensity corresponding to the difference in depth information D from the main subject, it is possible to emphasize the main subject and make it easier for the user to feel the distance between the main subject and the background. An image can be displayed.
In addition, the display device 300 can apply the image processing method described in the third embodiment. As a result, it is possible to display an image with a presentation effect according to the depth information and the imaging device setting information.

In each of the embodiments described above, the image processing intensity determination units 101 and 101a calculate an intensity correction value from the imaging apparatus setting information P when determining the image processing intensity S, and the intensity correction value and the depth information D are calculated. However, the image processing strength S may be determined directly from the imaging device setting information P and the depth information D. For example, the image processing intensity S is directly determined from the imaging apparatus setting information P and the depth information D using an LUT in which the image processing intensity S is associated with a combination of the imaging apparatus setting information P and the depth information value. May be.
Further, in each of the above-described embodiments, the image processing intensity determination units 101 and 101a determine the image processing intensity S for each pixel, but for each block of a predetermined size or for each subject surrounded by an outline. For example, the image processing intensity S may be determined.

In addition, a system including a plurality of devices processes each processing of the image processing device (100, 100a, 100b, or 100c), the imaging device 200, or the display device 300 according to each embodiment in a distributed manner. May be.
Further, a program for executing each process of the image processing apparatus (100, 100a, 100b, or 100c), the imaging apparatus 200, or the display apparatus 300 of each embodiment is recorded on a computer-readable recording medium, and the recording medium The above-described various processes relating to the image processing apparatus (100, 100a, 100b, or 100c), the imaging apparatus 200, or the display apparatus 300 may be performed by causing the computer system to read and execute the program recorded on the computer.

  Here, the “computer system” may include an OS and hardware such as peripheral devices. Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW (World Wide Web) system is used. The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

  Further, the “computer-readable recording medium” refers to a volatile memory (for example, DRAM (Dynamic) in a computer system serving as a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Random Access Memory)) that holds a program for a certain period of time is also included. The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  As mentioned above, although embodiment of this invention was explained in full detail with reference to drawings, the concrete structure is not restricted to this embodiment, The design etc. of the range which does not deviate from the summary of this invention are included.

(1) The present invention has been made to solve the above-described problems, and one aspect of the present invention refers to setting information of an imaging apparatus when an image is captured and depth information corresponding to the image. An image processing intensity determining unit (101, 101a) that determines an image processing intensity for each region into which the image is divided, and an image processing unit (102 that performs image processing on the image data of the image according to the image processing intensity) An image processing apparatus (100, 100a, 100b, 100c).

  Thereby, even when the setting of the imaging device that captured the input image is changed, an image that allows the user to feel the depth can be generated.

(2) According to another aspect of the present invention, there is provided the image processing apparatus according to (1), wherein the setting information is information indicating a focal length, and the image processing intensity determination unit includes the focal length. The value of the image processing intensity is determined so as to increase as the value of the information indicating the value increases.

  Thereby, even if the focal length of the imaging device that captured the input image changes, an image that allows the user to feel the depth can be generated.

(3) According to another aspect of the present invention, in the image processing apparatus according to (1), the setting information is information indicating an F value, and the image processing intensity determination unit The value of the image processing intensity is determined so as to decrease as the value of the information indicating the value increases.

  Thereby, even if the F value of the imaging device that captured the input image changes, an image that allows the user to feel the depth can be generated.

(4) According to another aspect of the present invention, in the image processing apparatus according to (1), the setting information is information indicating sensitivity, and the image processing intensity determination unit indicates the sensitivity. The value of the image processing intensity is determined to increase as the value of information increases.

  Thereby, even if the sensitivity of the imaging device that captured the input image changes, an image that allows the user to feel the depth can be generated.

(5) According to another aspect of the present invention, there is provided the image processing apparatus according to (1), wherein the setting information is information indicating an exposure time, and the image processing intensity determination unit includes the exposure time. The value of the image processing intensity is determined so as to decrease as the value of the information indicating the value increases.

  Thereby, even if the exposure time of the imaging device that captured the input image changes, an image that allows the user to feel the depth can be generated.

(6) According to another aspect of the present invention, there is provided the image processing apparatus according to (1), wherein the setting information is information indicating a focal position in the image, and the image processing intensity determination unit includes: , Determining a correction value according to the difference between the depth represented by the depth information for the in-focus position and the depth represented by the depth information for a region divided in the image, and refer to the correction value and the depth information Then, the image processing intensity is determined.

  Thereby, even if the position of the focal point of the imaging device that captured the input image changes, it is possible to generate an image that allows the user to feel the depth.

(7) According to another aspect of the present invention, an imaging unit that images a subject and the image processing apparatus according to any one of (1) to (6) are provided, and the image includes the imaging The image pickup apparatus is an image picked up by the image pickup unit, and the setting information is setting information when the image pickup unit picks up the image.

(8) According to another aspect of the present invention, there is provided an image processing device according to any one of (1) to (6), and an image display unit that displays an image processed by the image processing device. It is provided with the following.

(9) According to another aspect of the present invention, image processing is performed for each region obtained by dividing the image with reference to setting information of the imaging device when the image is captured and depth information corresponding to the image. An image processing method comprising: a first step of determining an intensity; and a second step of performing image processing on image data of the image according to the image processing intensity.

(10) According to another aspect of the present invention, the computer refers to the setting information of the imaging device when the image is captured and the depth information corresponding to the image for each area into which the image is divided. An image processing program for functioning as an image processing intensity determining unit that determines image processing intensity and an image processing unit that performs image processing on image data of the image according to the image processing intensity.

  DESCRIPTION OF SYMBOLS 100, 100a, 100b, 100c ... Image processing apparatus 101, 101a ... Image processing intensity determination part 102 ... Image processing part 103 ... Intensity correction value determination part 104, 104a ... Image processing intensity calculation part 105 ... Depth information analysis part 111 ... Frequency Distribution calculation unit 112 ... Depth information extraction unit 200 ... Imaging device 201 ... Imaging unit control unit 202, 203 ... Imaging unit 204 ... Parallax calculation unit 205 ... Image display unit 206 ... Image storage unit 300 ... Display device 301 ... Depth information calculation unit

Claims (5)

The image processing intensity is determined for each unit area of the divided image with reference to setting information indicating the setting of the imaging device used when imaging the subject and depth information corresponding to the captured image. An image processing intensity determination unit to
An image processing unit that performs image processing on the image data of the image so as to make the distant view of the image clear according to the image processing intensity,
The image constitutes each frame of a moving image composed of a plurality of continuous frames,
The image processing apparatus, wherein the setting information has a value for each predetermined number of frames. An imaging unit for imaging the subject;
And an image processing apparatus according to claim 1,
The image is an image captured by the imaging unit,
The imaging apparatus, wherein the setting information is setting information representing settings of the imaging unit when the imaging unit images the image. An image processing apparatus according to claim 1;
An image display unit that displays an image processed by the image processing device. The image processing intensity is determined for each unit area of the divided image with reference to setting information indicating the setting of the imaging device used when imaging the subject and depth information corresponding to the captured image. A first process to
A second step of performing image processing on the image data of the image so as to make the distant view of the image clear according to the image processing intensity,
The image constitutes each frame of a moving image composed of a plurality of continuous frames,
An image processing method, wherein the setting information has a value for each predetermined number of frames. Computer
The image processing intensity is determined for each unit area of the divided image with reference to setting information indicating the setting of the imaging device used when imaging the subject and depth information corresponding to the captured image. An image processing intensity determining unit,
An image processing program for causing an image processing unit to perform image processing on image data of the image so as to make a distant view of the image clear according to the image processing intensity,
The image constitutes each frame of a moving image composed of a plurality of continuous frames,
An image processing program characterized in that the setting information has a value for each predetermined number of frames.

Images