JP6325656B2 - Apparatus, program and method for assisting focus evaluation - Google Patents

Description

  The present invention relates to a technique for assisting in confirming whether a human is in focus (hereinafter referred to as “focus evaluation”) when shooting a moving image or a still image with a camera or the like.

  Conventionally, focus evaluation when a moving image or a still image is captured by a camera or the like has been generally evaluated by focusing on pixels that represent the edge of an image (for example, pixels that represent the contour of an object). In order to assist such focus evaluation, there is a technique for enhancing a pixel representing an edge using an edge enhancement process.

  For example, Japanese Patent Laid-Open No. 2010-114556 normalizes luminance values constituting image data, and performs a filter operation using HPF on the normalized image data to obtain a filter operation value for each pixel. A technique for coloring a pixel having a filter calculation value larger than the edge threshold with a specific color is described (see paragraphs [0080] to [0092] and FIGS. 6 to 11 of the document). Here, the pixel representing the edge generally has a large filter calculation value after the filter calculation using HPF.

  In addition, there is a technique for enhancing an output image by generating a frequency component that is not included in an input image signal using nonlinear processing regardless of focus evaluation (see Japanese Patent Nos. 5320538 and 5396626). ).

  Furthermore, in JP 2010-16783 A, a predetermined limit value is provided for the edge enhancement level, and when the edge amount increases due to gain multiplication and the edge enhancement level reaches the limit value, the edge enhancement level is determined in advance. Techniques for clipping to values and saturating are described.

JP 2010-114556 A Japanese Patent No. 5320538 Japanese Patent No. 5396626 JP 2010-16783 A

  In the edge enhancement process, pixels representing edges are emphasized. Therefore, the contour enhancement processing does not sufficiently assist the focus evaluation for an image having few edges and having few pixels whose luminance changes rapidly from neighboring pixels, that is, an image having a small high frequency component as a spatial frequency component. .

  In order to deal with such a problem, it is conceivable to increase the HPF gain or widen the HPF band when performing edge enhancement processing on an image having a small number of high-frequency components. However, simply increasing the gain increases the noise of the image. Also, when the band is widened, it is rather difficult to discriminate between pixels that are edges and pixels that are not edges. This is because, for example, according to the technique described in Japanese Patent Application Laid-Open No. 2010-114556, the difference between the filter calculation value of a pixel that is a high-frequency component and the filter calculation value of a pixel that is not a high-frequency component is small. This is because it becomes difficult to set.

  In addition to the case where the image does not contain sufficient high-frequency components, even when the image is displayed in a reduced size depending on the shooting environment, the contour enhancement processing may not provide sufficient assistance for focus evaluation. For example, according to the technique described in Japanese Patent Application Laid-Open No. 2010-114556, one pixel to which a specific color is assigned may be crushed due to reduction and may not be displayed. In recent years, an image is sometimes captured at 4K (4096 pixels × 2160 pixels), and when such an image is confirmed on a small, low-resolution monitor equipped with a camera, the image has to be reduced and displayed. However, the conventional contour enhancement processing does not provide sufficient assistance for focus evaluation. For this reason, when shooting a video at 4K, it is actually the case that a large 4K monitor is brought in and the focus of the shot image is evaluated.

  In the first place, the applicant has noticed that the pixel representing the edge may be one index for evaluating the focus, but the focus evaluation is not performed by itself. Information useful for focus evaluation is also included in pixels that are not pixels that represent edges. For example, the atmosphere of the edge of the entire screen can be confirmed from a pixel that is not a pixel representing the edge. However, this information has not been utilized in assisting focus evaluation in the conventional contour enhancement processing.

  The present invention has been made in view of the above problems.

  One embodiment of the present invention is a focus evaluation assisting device. The apparatus is means for generating an edge component from a first image, wherein the first image includes a plurality of pixels, the edge component includes a plurality of elements, and the value of each element of the edge component is a first value. Edge component generating means corresponding to the value of each pixel of the first image after removing at least the direct current component from the spatial frequency component in the image of the image, and means for generating the second image from the edge component, The second image includes a plurality of pixels, and the means includes image generation means including means for determining the color of each pixel of the second image as a function of the value of each element of the edge component. The apparatus is configured to output an image based on the newly generated second image to the display unit in response to a new input of the first image to the edge component generation unit. .

  The edge component generation means may include means for performing nonlinear processing on the value of each element of the edge component before applying the limiter. The value of the element of the edge component after the nonlinear processing is the output of the first nonlinear function that receives the value of the element of the edge component before the nonlinear processing.

  Note that the edge component generation means may include means for performing nonlinear processing including limiter application on the value of each element of the edge component. The value of the element of the edge component after the nonlinear processing is an output of the second nonlinear function that receives the value of the element of the edge component before the nonlinear processing, and the second nonlinear function includes at least the first nonlinear function and at least It is a composite function with a function for limiter application.

  The first nonlinear function may be a function defined by the first quadrant and the third quadrant and passing through the origin.

  The apparatus may further include means for switching the first nonlinear function between a plurality of nonlinear functions when the first image is sequentially input.

  The edge component generation unit may include a unit that applies a two-dimensional high-pass filter to the first image and removes at least a direct current component from the spatial frequency component in the first image. Alternatively, the edge component generation means includes a first direction one-dimensional high-pass filter means for the first image, a second direction one-dimensional high-pass filter means for the first image, and an output of the first direction one-dimensional high-pass filter means. And a means for synthesizing the output of the two-way one-dimensional high-pass filter means.

  The edge component generation means is an amplification means for changing the value of the element to a value obtained by multiplying the value of the element of the edge component by a predetermined coefficient, and the value of the element of the edge component is larger or smaller than a predetermined threshold value. One or both of limiter applying means for changing the value to the threshold value may be provided.

  The edge component generation means replaces the value of an element having an absolute value larger than the absolute value of the surrounding element in the edge component with a value obtained from the value of the surrounding element; The edge component may include one or both of spreading filter means for replacing an element value having an absolute value smaller than the absolute value of the surrounding element value with a value obtained from the surrounding element value. Good. Here, the isolated point removal filter means specifies a continuous region of 3 elements × 3 elements in the edge component, and the product of the maximum value that the element can take and the first ratio in the elements excluding the central element of the region. Is used as a threshold value, the number of elements having an absolute value less than or equal to the threshold value is obtained, and when the obtained number of elements is equal to or greater than the first number, The value can be configured to replace the value of the center element of the region. The spreading filter means specifies a continuous area of 3 elements × 3 elements in the edge component, and in elements other than the element at the center of the area, the product of the maximum value that the element can take and the second ratio is used as an absolute value. The number of elements having a value that is greater than or equal to the threshold value is obtained, and when the number of obtained elements is the second number or more, the median value of the element values excluding the central element of the area, It can be configured to replace the value of the central element.

In the apparatus, the first image is generated from the original image, the original image includes a plurality of pixels, and the value of each pixel of the first image is derived from the color of the corresponding pixel of the original image. The apparatus replaces at least a part of the original image with the second image and / or superimposes the original image and the second image, thereby outputting an image output to the display unit. Further image generation means for generating the image may be further provided. Note that the image generation means performs the superposition of the original image and the second image,
C _out = αC _in1 + βC _in2
(Α is a real number satisfying 0 ≦ α ≦ 1, β is a real number satisfying 0 ≦ β ≦ 1)
Thus, it is possible to obtain a color of a certain pixel in the image after superposition. Here, C _out is a value that specifies the color of a certain pixel in the superimposed image, and C _in1 and C _in2 are values that specify the color of the corresponding pixel in the original image and the second image, respectively. is there. Further, the further image generating means may include means for adjusting the color of the original image.

  Another embodiment of the present invention is a program that causes a computer to function as a focus evaluation auxiliary device.

  Another embodiment of the present invention is a focus evaluation assisting method. This method is a step of generating an edge component from a first image by an edge component generation means, wherein the first image includes a plurality of pixels, the edge component includes a plurality of elements, and each element of the edge component Is obtained from the edge component by the edge component generation step corresponding to the value of each pixel of the first image after removing at least the direct current component from the spatial frequency component in the first image. Generating a second image, wherein the second image includes a plurality of pixels, the step including determining a color of each pixel of the second image as a function of a value of each element of the edge component. And an image generation step and an image output step of outputting an image based on the second image to the display means by the image output means. In this method, the edge component generation step, the image generation step, and the image output step are repeated in response to a new input of the first image.

  The edge component generation step may include a step of performing nonlinear processing on the value of each element of the edge component before applying the limiter. The value of the element of the edge component after the nonlinear processing is the output of the first nonlinear function that receives the value of the element of the edge component before the nonlinear processing.

  The edge component generation step may include a step of performing non-linear processing including limiter application on the value of each element of the edge component. The value of the element of the edge component after the nonlinear processing is an output of the second nonlinear function that receives the value of the element of the edge component before the nonlinear processing, and the second nonlinear function includes at least the first nonlinear function and at least It is a composite function with a function for limiter application.

  As described above, the first nonlinear function may be a function defined by the first quadrant and the third quadrant and passing through the origin.

  The method may further include switching the first non-linear function between a plurality of non-linear functions when the first image is sequentially input.

  The edge component generation step may include a two-dimensional high-pass filter application step for the first image. Alternatively, the edge component generation step includes a first direction one-dimensional high-pass filter application step for the first image, a second direction one-dimensional high-pass filter application step for the first image, a first direction one-dimensional high-pass filter application step, And a step of synthesizing the output obtained by the second direction one-dimensional high-pass filter applying step.

  In the edge component generation step, an amplification step for changing the value of the element to the value obtained by multiplying the value of the element of the edge component by a predetermined coefficient, and the value of the element of the edge component is larger or smaller than a predetermined threshold One or both of the limiter application steps for changing the value to the threshold value may be included.

  The edge component generation step replaces the value of an element having an absolute value greater than the absolute value of the surrounding element in the edge component with a value obtained from the value of the surrounding element; One or both of a broadening filter application step of replacing an element value having an absolute value smaller than the absolute value of the surrounding element value with a value obtained from the surrounding element value in the edge component. May be. Here, the isolated point removal filter applying step includes a step of specifying a continuous area of 3 elements × 3 elements in the edge component, and a maximum value and a first ratio that the element can take in elements excluding the central element of the area And a step of obtaining the number of elements having a value whose absolute value is less than or equal to the threshold value, and the element at the center of the region is excluded when the number of obtained elements is equal to or greater than the first number Replacing the value of the element at the center of the region with the median of the values of the elements. The spreading filter applying step includes a step of specifying a continuous region of 3 elements × 3 elements in the edge component, and a product of the maximum value that the element can take and the second ratio in the elements other than the central element of the region. And determining the number of elements having a value whose absolute value is equal to or greater than the threshold, and the value of the element excluding the central element in the region when the number of determined elements is equal to or greater than the second number And replacing the value of the element at the center of the region with the median value of.

In the method, the first image is generated from the original image, the original image includes a plurality of pixels, and the value of each pixel of the first image is derived from the color of the corresponding pixel of the original image. In this method, the display is performed by replacing at least a part of the original image with the second image and / or superimposing the second image on the original image by the further image generation means. It may further comprise a further image generation step for generating an image to be output to the means. In the further image generation step, when the original image and the second image are superimposed, the additional image generation means
C _out = αC _in1 + βC _in2
(Α is a real number satisfying 0 ≦ α ≦ 1, β is a real number satisfying 0 ≦ β ≦ 1)
The step of obtaining the color of a certain pixel in the superimposed image can be included. Here, C _out is a value that specifies the color of a certain pixel in the superimposed image, and C _in1 and C _in2 are values that specify the color of the corresponding pixel in the original image and the second image, respectively. is there. Further, the further image generation step may further include a step of adjusting the color of the original image.

  According to the present invention, a luminance change between pixels in an image is displayed in a continuous manner using a color corresponding to the luminance change. Therefore, even on a small and low-resolution screen, focus evaluation is relatively easy from the edge atmosphere grasped from the entire image. Note that the edge atmosphere means a distribution of luminance change between pixels over the entire image.

  In addition, according to an embodiment, desired enhancement can be performed on the edge component by generating a frequency that does not exist in the spatial frequency of the edge component before the nonlinear processing by the nonlinear function.

  According to one embodiment, the removal of at least a direct current component from the frequency component in the first image can be realized by a one-dimensional high-pass filter or a two-dimensional high-pass filter. Since the operation speed of the one-dimensional high-pass filter is higher, when the one-dimensional high-pass filter is used, the removal of at least the direct current component from the frequency component in the first image can be realized at a higher speed.

  According to one embodiment, the value of the edge component can be converted into a manageable range by amplifying or limiting the value of the edge component.

  According to one embodiment, noise can be reduced by smoothing edge components.

  Furthermore, according to one embodiment, by displaying the second image in association with the original image, it is possible to simultaneously grasp the features of the original image that cannot be recognized from the second image. At that time, the original image or the second image can be relatively emphasized by adjusting the color of the original image.

3 shows a flowchart of a focus evaluation assisting method according to an embodiment of the present invention. An alternative configuration of the edge component generation step 120 in FIG. 1A is shown. A first image is shown. A two-dimensional high-pass filter is shown. Edge components are shown. It shows the effect of non-linear processing when using E ³ as a non-linear function f (E). A one-dimensional high-pass filter is shown. Non-linear processing is described. An example of a nonlinear function, The effect by the nonlinear function shown by FIG. 4A is shown. A generalized function for amplification / limiter application is shown. Fig. 4 shows an example of a function for amplification / limiter application. Fig. 5 shows another example of a function for amplification / limiter application. The function for assigning the color of the pixel of the 2nd picture is shown. The figure corresponding to the edge component before execution of the smoothing step 131 in FIG. 1A is shown. The figure corresponding to the edge component after execution of the smoothing step 131 in FIG. 1A is shown. The graph showing the isolated point 411 and the insect eater 412 in FIG. 7A is shown. The flowchart of the isolated point removal filter application step 132 in FIG. 1A is shown. The flowchart of the spreading filter application step 133 in FIG. 1A is shown. The region specified in the edge component is shown. An original image (first image) is shown. A second image is shown. A display image generated by superimposing the original image and the second image is shown. A display image generated by partially overlapping the original image and the second image is shown. The display image produced | generated by superimposing the original image and the 2nd image imaged using the function different from FIG. 8B is shown. The display image produced | generated by partially superimposing the original image and the 2nd image imaged using the function different from FIG. 8B is shown. Another original image is shown. The other display image produced | generated by superimposition is shown. 1 is a block diagram of a focus evaluation assisting apparatus that is an embodiment of the present invention. FIG. 9B shows an alternative configuration of the edge component generation means 620 in FIG. 9A.

  First, a method for assisting focus evaluation, which is an embodiment of the present invention, will be described.

  FIG. 1A is a flowchart of a method for assisting focus evaluation. Note that the configuration shown in this flowchart is merely an example, and some steps are not essential as described below.

  The method begins at step 110 where a first image is input. The first image is, for example, 1 of a moving image (consisting of a plurality of frames. In addition, a moving image displayed on a camera finder or monitor for capturing a still image) is currently captured by the camera. It may be derived from a frame, one frame of a moving image already recorded by a camera (hereinafter referred to as “original image”), or the like.

  The first image includes a plurality of pixels. The value of each pixel of the first image is derived from the color of the corresponding pixel of the original image. In the following description, it is assumed that the value is the luminance value of the pixel of the original image. Note that the value of each pixel of the first image is an arbitrary value that can be derived from the color of the original image pixel, such as the R, G, or B value of the pixel color of the original image, or the chroma (saturation) value. It may be. Further, when the luminance value is used as the pixel value of the first image, since the first image can be regarded as an image in which the color is expressed in grayscale, when the color is expressed in grayscale in the original image, The derivation of the value of the first pixel may be simply using the color (value) of the pixel of the original image.

  The method then continues to step 120 where an edge component is generated from the first image. Here, the edge component is a component obtained by removing at least a direct current component from a spatial frequency component in the first image. The edge component includes a plurality of elements, and the value of each element of the edge component corresponds to the value of each pixel of the first image after removing at least the direct current component from the spatial frequency component in the first image. In the present embodiment, the edge component is represented by an array including the same number of elements as the number of pixels of the first image.

  The method for removing at least the DC component from the spatial frequency component in the first image is arbitrary, but in the present embodiment, it is realized by executing step 121 of applying a two-dimensional high-pass filter (HPF) to the first image. doing. Alternatively, it can be realized by two one-dimensional high-pass filter applying steps 121a and 121b, as described below with respect to FIG. 1B. Further, at least a direct current component can be removed from the first image by subtracting the first image obtained by applying the low-pass filter from the first image instead of applying the high-pass filter. Hereinafter, the two-dimensional high-pass filter will be described with reference to FIGS.

FIG. 2A is an example of the first image. Here, one square represents one pixel, and the value I _xy in the square represents the value of the pixel, and in the following description, represents the luminance value of the corresponding pixel of the original image. Note that x and y are integers where 0 ≦ x <w and 0 ≦ y <h, respectively, and w and h are the numbers of pixels in the horizontal and vertical directions of the first image, respectively.

FIG. 2B is an example of a two-dimensional high-pass filter. In this example, a filter having a size of 3 elements × 3 elements is used, but the size may be as desired. The coefficients f _ij (i and j are integers where 0 ≦ i <3 and 0 ≦ j <3, respectively) of the example high-pass filter can take the following values.

  Needless to say, the value of the coefficient is an example, and an arbitrary coefficient that functions as a high-pass filter can be used.

  FIG. 2C is an example of an edge component. This edge component is obtained by applying the two-dimensional high-pass filter illustrated in FIG. 2B to the first image, and the value of each element is calculated by the following equation.

When using the above equation, I _xy (x <0, y <0, w ≦ x, h ≦ y) of the region outside the image may be calculated as an arbitrary value, for example, 0. Note that the value E _xy of the element of the edge component calculated by the above equation can be both positive and negative.

The edge component generation step 120 may further include a non-linear processing step 122. It should be noted that the non-linear processing performed in this step is not a simple clip processing as in the prior art, but is also performed before the limiter application described later. In this process, the following calculation is performed.
E2 _xy = f (E1 _xy ) (3)

Here, E1 _xy and E2 _xy are the values of the elements of the edge component before and after the nonlinear processing, respectively, and f (E) is an arbitrary nonlinear function. Preferably, the following conditions are imposed: It is a nonlinear function.
1 Defined in the first and third quadrants.
2 Pass through the origin.

  If the possible value of E exceeds the domain of f (E) to be used, it was obtained by multiplying E by a rational number, adding a rational number offset to E, or taking the absolute value of E. , F (E) can be calculated with E being a value within the domain of f (E).

Examples of non-linear functions include, but are not limited to, non-linear functions of E ^p , sin ^p (E) and log ^p (E) (p is a rational number) (that is, excluding E ¹ ), and these functions are rational multiples And a function obtained by one or more of adding a rational number offset to these functions and combining these functions. Specifically, f (E) can be a function as follows.

The non-linear function can also be realized using a table. For example, all values that can be taken by E _xy and predetermined outputs corresponding to the respective values are stored in a table, and when nonlinear processing is performed, the output is calculated using a function as this table. be able to.

  When nonlinear processing is performed on an image, frequency components that are not included in the spatial frequency component of the original image are generated. FIG. 3 shows the frequency components included in the input image and the output image when the output image is obtained by applying a nonlinear function to the luminance value of each pixel of the input image using the circuit shown. Note that HPF is a high-pass filter, NLF is a nonlinear function, LMT is a limiter, ADD is an adder, and MSB is a bit representing a most significant bit, that is, a sign.

  Since the generated frequency component differs depending on the nonlinear function, the nonlinear function in the nonlinear processing described above can be appropriately selected according to the desired effect. For example, FIG. 2D is a graph showing the effect of nonlinear processing when using Equation (4) as f (E), and this graph is connected to a plot showing the value of the element of the edge component before nonlinear processing. A curve 211 and a curve 212 connected with plots representing the values of the elements of the edge components after nonlinear processing are included. Here, the horizontal axis of the graph represents a continuous element having an edge component, and the vertical axis represents the value of the element. Equation (4) is a nonlinear function in which the output / input ratio increases as the input increases. Therefore, the curve 212 after the nonlinear process has a sharper peak 213 than the curve 211 before the nonlinear process. This means that the edge component element having a larger value becomes larger and the difference from the element having a smaller value is emphasized as compared with a case where the gain is simply increased. This effect is generally brought about when a nonlinear function that becomes a convex function when an absolute value is taken is used in the nonlinear processing described above.

  In addition, when a nonlinear function (for example, equations (6) and (7)) that becomes a concave function when an absolute value is obtained using a logarithmic function or a square root as a nonlinear function in the above nonlinear processing is used. In general, noise increases in the element of the edge component, but an effect that a low level signal can be confirmed is obtained.

  Furthermore, when a function with a step in the middle or a function with a negative slope is used as the nonlinear function in the nonlinear processing described above, the effect of reducing the sensitivity of a certain level of signal can be obtained. become. An example of such a nonlinear function is shown in FIG. 4A.

  The function shown in FIG. 4A also includes a so-called dead band, and as shown in FIG. 4B, noise can be removed or at least reduced to zero a signal whose absolute value is below a certain level. . Such a function can also be regarded as a combination with a function for applying a limiter described later.

  Note that the nonlinear function may be asymmetric with respect to the origin. Also, in order to obtain different effects depending on the range of input edge component values, functions having different properties may be combined to form a nonlinear function in the above-described nonlinear processing. Such a nonlinear function is partially linear. Some of them have constant values, some have discontinuous parts, some have quadratic functions, and others have cubic functions.

  When focus evaluation is performed on an output image of a video camera, the image from the video camera changes every moment, and an appropriate nonlinear function may differ depending on the image. For example, while using a video camera, it is convenient if the nonlinear function used in the nonlinear processing described above can be switched between a plurality of nonlinear functions when images are successively input to process the image in this method. . Such switching can be performed by any means.

  The edge component generation step 120 may further include an amplification / limiter application step 123. In this step, the value of the element of the edge component after the nonlinear processing step 122 (after the two-dimensional high-pass filter applying step 121 if the nonlinear processing step 122 is not included) is multiplied by a predetermined coefficient to the value. One or both of changing the value of the element and changing the value to the threshold when the value of the element is greater than or less than a predetermined threshold is performed.

The amplification / limiter application step 123 or the application of the limiter among the steps may be realized by a function as generally shown in FIG. 5A. In this figure, the horizontal axis and the vertical axis represent input and output, respectively, T _HN , T _LN , T _LP and T _HP are thresholds related to input, and C _HN , C _LN , C _LP and C _HP are these This is a value related to the output corresponding to each of the threshold values. The absolute values of T _LN and T _LP, the absolute values of T _HN and T _HP , the absolute values of C _LP and C _LN , and the absolute values of C _HP and C _HN may be equal or different. Good. According to this function, the value of an edge component greater than (or greater than) T _LN and less than (or less than) T _LP is zero, and the value of an edge component less than (or less than) T _HN is C The value of the edge component that is _HN and is greater than (or greater than) T _HP is C _HP . In FIG. 5A, the function is depicted on the assumption that the slope of the function of the section from T _HN to T _LN and the function of the section from T _LP to T _HP is 1, but the slope of at least one function of these sections is It may not be 1. By making the slope greater than 1, this function allows the limiter to be applied and amplified simultaneously.

In the function shown in FIG. 5A, C _LP = T _LP and C _LN = T _LN may be used, and the function in this case is as shown in FIG. 5B. Further, in the function shown in FIG. 5A, C _LP = C _LN = 0, and the function in this case is as shown in FIG. 5C.

  Needless to say, the above-described function may be realized using a table instead of numerical calculation.

  The nonlinear processing step 122 and the amplification / limiter applying step 123 may be executed as the same step by using a function obtained by synthesizing the functions used in the former and the latter.

As shown in FIG. 1B, the edge component generation step 120 applies a one-dimensional high-pass filter in a direction perpendicular to the line (vertical direction of the first image) instead of the step 121 of applying a two-dimensional high-pass filter. Step 121a and step 121b of applying a one-dimensional high-pass filter in the sample direction (the same lateral direction) parallel to the line can be included. In this case, the final edge component E _xy can be obtained by step 124 of combining the first edge component E _Lxy and the second edge component E _Sxy obtained by the two one-dimensional high-pass filters, respectively.

  FIG. 2E is an example of a one-dimensional high-pass filter. In this example, a filter having a size of 3 elements is used, but the size may be as desired. Moreover, you may use the filter of a magnitude | size different in the direction perpendicular | vertical to a line, and the sample direction parallel to a line. The coefficients of the example high-pass filter can take the following values.

  The value of the coefficient is an example, and an arbitrary coefficient functioning as a high-pass filter can be used. It goes without saying that the coefficient may be different in the direction perpendicular to the line and the sample direction parallel to the line.

  Application of the high-pass filter in the direction perpendicular to the line is expressed by the following equation.

  The application of a high-pass filter in the sample direction parallel to the line is represented by the following equation:

When using the above equation, I _xy (x <0, y <0, w ≦ x, h ≦ y) of the region outside the image may be calculated as an arbitrary value, for example, 0.

Although the synthesis method in the edge component synthesis step 124 is arbitrary, for example, it can be synthesized by simply calculating E _xy = E _Lxy + E _Sxy .

As shown in FIG. 1B, at least one of non-linear processing steps 122a and 122b and amplification / limiter application steps 123a and 123b is applied to the first edge component E _Lxy and the second edge component E _Sxy , respectively. These steps are similar to the non-linear processing step 122 and the amplification / limiter application step 123 described above.

  The method then continues to step 130 where a second image is generated. This step includes step 134 of imaging the edge component by assigning a color according to the value of the element of the edge component.

Hereinafter, a method of generating the second image will be described with reference to FIG. FIG. 6 shows three graphs having the edge component value E _xy as the horizontal axis and the R, G and B component values J _Rxy , J _Gxy and J _Bxy of the pixels of the second image as the vertical axis, respectively. Is shown.

The second image can be generated by assigning the pixel color of the second image from the values of the elements of the edge components in the relationship shown in the plots of these graphs. That is, if the plots shown in FIG. 6 are respectively given functions r (E _xy ), g (E _xy ), and b (E _xy ) in order from the upper graph, the RGB values (J _Rxy ) of the pixels of the second image are _set. , J _Gxy , J _Bxy ) can be expressed as follows:
J _Rxy = r (E _xy )
J _Gxy = g (E _xy ) (10)
J _Bxy = b (E _xy )

  Needless to say, the range in which the function should be defined varies depending on the possible values of the edge component elements.

As an alternative technique, the second image can be generated by assigning the value of the second pixel by the following function.
J _Rxy = E _xy + offset
J _Gxy = E _xy + offset (11)
J _Bxy = E _xy + offset

  Here, offset is an arbitrary offset value. According to this method, the second image is represented by shading from white to black.

  The above approach is merely exemplary. The second image can be generated by any technique that determines the color of each pixel of the second image as a function of the value of each element of the edge component.

  The step 130 of generating the second image may include a smoothing step 131 before the imaging step 134. The smoothing step 131 includes one or both of an isolated point removing filter applying step 132 and a spreading filter applying step 133. Hereinafter, the isolated point removal filter application step 132 and the spread filter application step 133 will be described with reference to FIGS.

  FIG. 7A shows an image 410 including an isolated point 411 and a worm-eaten 412 for the sake of explanation, in which the edge component is imaged by binarizing the absolute value of each element value by an appropriate threshold value. In FIG. 7A (and FIG. 7B described later), a white pixel indicates that the absolute value of the corresponding element value is greater than or equal to a certain threshold value, and a black pixel is less than a certain threshold value. Indicates. Originally, the isolated point 411 is an element having an edge component that has a value that is larger than the absolute value of the value of the surrounding element, preferably a value that is significantly larger than the absolute value of the value of the surrounding element. The worm-eaten 412 is an element having a value that is smaller than the absolute value of the value of the surrounding element, preferably an absolute value that is significantly smaller than the absolute value of the value of the surrounding element. Note that isolated points 411 and worm-eaten 412 are also shown as different expressions in FIG. 7C using a graph in which the horizontal axis is a continuous element having an edge component and the vertical axis is the value of the element. Such isolated points 411 and worm eaters 412 may occur due to the nature of the first image. The application of the isolated point removal filter removes the isolated point 411, and the application of the spread filter removes the worm-eating 412. For example, the purpose is to replace the median value, average value, maximum value, or minimum value of the values of surrounding elements. An example of an isolated point removal filter and a spreading filter will be described below. However, any filter, for example, a median filter capable of removing a protruding value, can be used as long as it can achieve the above-described purpose. Needless to say.

FIG. 7D is a flowchart of an exemplary isolated point removal filter. The filter first executes step 431 for specifying a continuous region 450 (see FIG. 7F) of 3 elements × 3 elements in the edge component. Here, assuming that the value of the central element 451 is E _XY , the value of the peripheral element 452 is E _xy , and the maximum value that the element can take is E _max , then the peripheral elements excluding the central element 451 of the specified region 450 In step 452, step 432 is performed to determine the number of elements having a value E _xy whose absolute value is equal to or less than the threshold value, using the product of E _max and the first ratio as a threshold value. Here, the first ratio may be 10%. Further, step 433 is executed to determine whether or not the obtained number of elements is greater than or equal to the first number. Here, the first number may be three. If the determination is true, step 434 is executed to replace the value E _XY of the central element 451 with the median value of the values E _xy of the surrounding elements 452.

For example, the value E _XY of the central element 451 is 100, the value E _xy of the eight surrounding elements 452 is {33, 10, 15, -20, −5, 5, −42, 12}, and the maximum value E Assume that _max = 255. Here, the threshold value is 255 × 10% = 25.5, the absolute value of E _xy = {33, 10, 15, 20, 5, 5, 42, 12}, and E is equal to or less than the threshold value 25.5. _Since the absolute value of _xy is {10, 15, 20, 5, 5, 12}, the number of elements whose absolute value of value _xy is less than or equal to the threshold value is 6. Therefore, since the obtained number of elements 6 is equal to or greater than the first number 3, the central element 451 is determined to be an isolated point, and its value E _XY = 100 is 7.5 (the median value of the surrounding pixels 452). Here, since the number of surrounding pixels 452 is 8, the median is obtained as an arithmetic average of two values E _xy = {5, 10} close to the center).

FIG. 7E is a flowchart of an exemplary spreading filter. The filter first executes step 441 of specifying a continuous region 450 (see FIG. 7F) of 3 elements × 3 elements in the edge component. Here, assuming that the value of the central element 451 is E _XY , the value of the peripheral element 452 is E _xy , and the maximum value that the element can take is E _max , then the peripheral elements excluding the central element 451 of the specified region 450 In step 452, step 442 is performed in which the product of E _max and the second ratio is used as a threshold value to determine the number of elements having a value E _xy whose absolute value is equal to or greater than the threshold value. Here, the second ratio may be 10%. Further, step 443 is performed for determining whether the number of elements obtained is equal to or greater than the second number. Here, the second number may be five. If the determination is true, step 444 is executed in which the value E _XY of the central element 451 is replaced with the median value of the values E _xy of the surrounding elements 452. Since a specific example for the spreading filter is similar to that for the isolated point removal filter, a description thereof will be omitted.

  If the smoothing step 131 includes both the isolated point removal filter application step 132 and the spread filter application step 133, either of these steps may be performed first, but empirically, the isolated point removal filter application step 132 is performed. It seems that a better result is obtained if it is executed first. Note that the first ratio and the second ratio described above may be different, and the first number and the second number may be the same.

  The method can include a display image generation step 140. The display image generation step 140 includes an image composition step 142. 8A and 8B show an original image (substantially equivalent to the first image because it is expressed in gray scale) and a second image, respectively, for the purpose of comparison with the synthesized image. . FIG. 8C shows a display image generated by combining the original image and the second image so as to overlap each other.

  FIGS. 8G and 8H show another example of the display image in which the original image and the original image and the second image are superimposed. The lower image in FIG. 8H is a display image obtained for the original image in focus shown in FIG. 8G. A display image as shown in FIG. 8H is obtained from an unfocused original image (not shown).

The color of a certain pixel in the superimposed image is a value that identifies the color (for example, the R, G, and B values of the color, or the R, G, and B values are each a 24-bit color value or 32 If it is stored as one value such as a bit color value, it can be obtained by calculating the value by the following equation.
C _out = αC _in1 + βC _in2
(Α is a real number satisfying 0 ≦ α ≦ 1, β is a real number satisfying 0 ≦ β ≦ 1) (12)

Here, C _out is a value that specifies the color of a certain pixel in the superimposed image, and C _in1 and C _in2 are values that specify the color of the corresponding pixel in the original image and the second image, respectively. is there.

  The weights α and β can be arbitrarily designated by the user. For example, when α = 0.5 and β = 1, the second image appears more emphasized than the original image in the superimposed image. In general, the value specifying the color of each pixel in the original image is assumed to be a positive value, but the value specifying the color of each pixel in the second image is the second value. Depending on the above-described function used when generating an image, both positive and negative values may be taken. In such a case, for example, since a picture having a brightness higher than the maximum brightness that can be expressed is taken, the value specifying the color of a certain pixel When the original image and the second image in which the maximum possible value is generated are overlapped with α = 1 and β = 1, the value specifying the color of the overexposed pixel in the original image is the first value. In some cases, the pixel representing the edge in the image after superimposition can be confirmed by subtracting by the value specifying the color of the corresponding pixel which is a negative value in the second image. In the case where the relation of β = 1−α is satisfied, Expression (12) represents a general alpha blending expression.

  The display image can also be generated by combining so that a part of the original image is replaced with the second image. Here, the position and size of the portion to be replaced can be arbitrarily designated by the user. It is also possible to combine images so that partial replacement of images and superposition are performed simultaneously. FIG. 8D shows a display image 510 generated by performing partial replacement and superposition of images at the same time and superimposing a part 511 of the original image and the second image.

  By changing the function used for imaging the second image, the expression of the combined image can be adjusted. FIG. 8E shows a display image generated by combining the original image shown in FIG. 8A and the second image so as to overlap each other. The second image in FIG. 8A is formed from the original image shown in FIG. 8A using a function different from the function that generated the second image (second image shown in FIG. 8B). The function used for generating the second image in FIG. 8E is configured to further add an offset to the values of each RGB color with respect to the function used for generating the second image in FIG. 8C. A brighter second image can be generated. FIG. 8F corresponds to FIG. 8E in the case where the partial replacement of the image is further performed.

  The display image generation step 140 may include a step 141 for adjusting the color of the original image. By this step, the color adjustment of the original image can be performed before the composition. For example, by reducing the contrast of the original image by the color adjustment, a display image in which pixels representing edges are more conspicuous may be generated.

  The method then proceeds to step 150 where an image based on the second image is output to a display means, such as a camera finder or monitor. Here, the image based on the second image is a display image generated when the present method includes the display image generation step 140, and is the second image itself when not including the display image generation step 140.

  In the above, the method for assisting focus evaluation has been described. Focus is generally evaluated while moving a focus ring such as a camera during shooting. When the focus ring is moved, the camera focus changes and the original image also changes. Therefore, since the input first image changes, the second image used for display must be updated. Accordingly, the steps involved in the method are repeated in response to a new input of the first image.

  Next, an apparatus for assisting focus evaluation, which is an embodiment of the present invention, will be described.

  FIG. 9A is a block diagram of an apparatus for assisting focus evaluation. Note that the configuration shown in this block diagram is merely an example, and some means described below are not essential. In this figure, the graph shown in relation to the first image, the edge component, the second image, the original image before color adjustment, and the original image before color adjustment is shown for each means for explanation. It is a graph which shows what kind of output. In these graphs, the horizontal axis and the vertical axis indicate a continuous pixel and a luminance value obtained from the pixel for the first image, the second image, the original image before color adjustment, and the original image after color adjustment, as edge components. Represents a certain continuous element and element value.

  This apparatus includes first image generation means 610. This means is a means for generating the first image from the original image.

  This apparatus includes an edge component generation unit 620. This means executes the edge component generation step 120. Therefore, the edge generation unit 620 is only required to remove at least a direct current component from the spatial frequency component included in the first image. However, in the present embodiment, the two-dimensional high-pass filter unit 621 that executes the two-dimensional high-pass filter application step. It is realized by having. Alternatively, it can be realized by providing two one-dimensional high-pass filter means 624a and 624b, as will be described later with reference to FIG. 9B. It can also be realized by providing means for subtracting a low-pass filter applied to the first image from the first image. The edge component generation unit 620 may further include one or both of a nonlinear processing unit 622 and an amplification / limiter unit 623 that execute the nonlinear processing step 122 and the amplification / limiter application step 123, respectively.

  The edge generation unit 620 includes a filter buffer unit 624. The filter buffer unit 624 is at least data used at a time by the two-dimensional high-pass filter unit 621, for example, 3 pixels × 3 pixels of the first image (corresponding to the size of the filter in the two-dimensional high-pass filter unit 621). Is a means for temporarily storing data representing and outputting the data to the two-dimensional high-pass filter means 621. If the number of pixels in the horizontal direction of the first image is set to w, the filter buffer means 624 considers the processing efficiency, and the continuous w pixels × 3 pixels (two-dimensional high-pass filter) of the first image at a time. Data corresponding to the vertical size of the means 621) may be temporarily stored.

As shown in FIG. 9B, instead of the two-dimensional high-pass filter unit 621, the edge component generation unit 620 includes a one-dimensional high-pass filter unit 621a in the direction perpendicular to the line and a one-dimensional high-pass filter unit in the sample direction parallel to the line. 621b can be provided. The one-dimensional high-pass filter means 621a and 621b execute the one-dimensional high-pass filter application steps 121a and 121b, respectively. As shown in this figure, the edge component generation means 620 includes at least one of nonlinear processing means 622a and 622b and amplification / limiter application means 623a and 623b in a direction perpendicular to the line and in a sample direction parallel to the line. Can be provided. These means _execute the non-linear processing step 122 and the amplification / limiter application step 123 for the first edge component E _Lxy and the second edge component E _Sxy (perform steps 122a and 122b and steps 123a and 123b, respectively). Is.

When two one-dimensional high-pass filter means are used, the edge generation means 620 includes a line buffer means 624a, a sample buffer means 624b, and an edge synthesis means 626. The line buffer means 624a has data corresponding to the size of the filter in the 1 line × 1D high-pass filter means 621a of the first image, for example, the number of pixels in the horizontal direction of the first image is w, and the size of the filter When the length is 3, data representing continuous w × 3 pixels is temporarily stored. From the stored data, data representing continuous pixels in the direction perpendicular to the line is sequentially output to the one-dimensional high-pass filter 621a, and data used at one time by the one-dimensional high-pass filter 621b is output to the sample buffer unit 624b. For example, three consecutive pixels in the sample direction parallel to the line (corresponding to the size of the filter in the one-dimensional high-pass filter means 621b) are sequentially output. The sample buffer unit 624b temporarily stores data used at one time by the one-dimensional high-pass filter unit 621b and outputs the data to the one-dimensional high-pass filter unit 621b. The edge component synthesis means 626 temporarily stores the first edge component E _Lxy and the second edge component E _Sxy and executes the edge component synthesis step 124.

  The apparatus includes second image generation means 630 that executes the second image generation step 130. The second image generating unit 630 includes an imaging unit 634 that executes the imaging step 134. The second image generation unit 630 may further include a smoothing unit 631 that executes the smoothing step 131. The smoothing means 631 includes one or both of an isolated point removal filter means 632 and a spread filter means 633 that execute the isolated point removal filter application step 132 and the spread filter application step 133, respectively.

  The apparatus can include display image generation means 640 that executes the display image generation step 140. The display image generation means 640 includes image composition means 642 that executes the image composition step 142. The display image generating unit 640 may further include a color adjusting unit 641 that executes Step 141 for adjusting the color of the original image.

  The display image generation means 640 receives the original image and the second image. Therefore, the display image generation means 640 delays the timing of passing the original image to the image composition means 642 (and the color adjustment means 641 if present) when the input timing of the original image and the second image is significantly different. The delay adjusting means 643 is included.

  The apparatus is configured to output an image based on the second image to a display unit (not shown). Here, the image based on the second image is a display image generated when the present apparatus includes the display image generating means 640, and is the second image itself when not including the display image generating means 640.

  The apparatus for assisting focus evaluation has been described above. As described above with respect to the method for assisting focus evaluation, during the focus evaluation, the input first image changes, so the second image used for display must be updated. Accordingly, the apparatus is configured to output at least an image based on the newly generated second image to the display unit in response to a new input of the first image to the edge component generation unit 620. Is done.

  The present invention can also be implemented as a program for assisting focus evaluation. Such a program causes a computer to function as an apparatus for evaluating the focus described above.

  Here, the “computer” refers to a system including at least one of an arithmetic / control device, which is a hardware resource, a storage device, an input device, and an output device. The arithmetic / control device includes a CPU and an MPU. The storage device includes a memory, a hard disk, an SSD, and the like. The input device includes a chip pin, a mouse, a keyboard, a touch panel, a network interface, and the like. Output devices include chip pins, network interfaces, displays, printers, speakers, and the like. Two or more of the arithmetic / control device, the storage device, the input device, and the output device can be physically integrated by using an FPGA or a microcomputer. Those skilled in the art will be able to implement the present invention in cooperation with one or more of these hardware / resources of arithmetic / control devices, storage devices, input devices and output devices, and programs that are software. Will be clear.

  In the above, a plurality of embodiments of the present invention have been described. However, it should be noted that these embodiments are merely illustrative of the invention. The present invention is specified only by the scope of the claims, and includes embodiments in which various changes, modifications, deletions and substitutions have been made to the above-described embodiments.

Claims (33)

A device,
Means for generating an edge component from a first image, wherein the first image includes a plurality of pixels, the edge component includes a plurality of elements, and a value of each element of the edge component is a space in the first image Edge component generation means corresponding to the value of each pixel of the first image after removing at least the DC component from the frequency component;
Means for generating a second image from an edge component, wherein the second image includes a plurality of pixels, the means comprising: a color of each pixel of the second image as a function of the value of each element of the edge component; An image generation means including a means for determining, in response to a new input of the first image, an image based on the newly generated second image is output to the display means, The edge component generation means includes means for performing nonlinear processing on the value of each element of the edge component before applying the limiter, and the value of the element of the edge component after the nonlinear processing is the edge component before the nonlinear processing. Is an output of a first non-linear function that takes as an input the value of an element of the input, and in the first non-linear function, a certain value between a minimum value that can be taken by an input on the negative side and a maximum value that can be taken by an input on the positive side Where x is the function input and y is the function output. When represented on the original xy plane, a section between the minimum value that can be taken by the input on the negative side and the certain value, or a section between the certain value and the maximum value that can be taken by the input on the positive side Is a device in which the certain value is a curve. A device,
Means for generating an edge component from a first image, wherein the first image includes a plurality of pixels, the edge component includes a plurality of elements, and a value of each element of the edge component is a space in the first image Edge component generation means corresponding to the value of each pixel of the first image after removing at least the DC component from the frequency component;
Means for generating a second image from an edge component, wherein the second image includes a plurality of pixels, the means comprising: a color of each pixel of the second image as a function of the value of each element of the edge component; An image generation means including a means for determining, in response to a new input of the first image, an image based on the newly generated second image is output to the display means, The edge component generation means includes means for performing nonlinear processing on the value of each element of the edge component. The value of the element of the edge component after the nonlinear processing is the value of the element of the edge component before the nonlinear processing. It is an output of a second nonlinear function as an input, and the second nonlinear function is a composite function of the first nonlinear function and at least a function for applying a limiter. In the first nonlinear function, an input on the negative side is The minimum possible value and the input on the positive side The minimum value and the certain value that can be taken by the input on the negative side when the function input is represented on the two-dimensional xy plane with x representing the function input and y representing the function output. Or the interval between the certain value and the maximum value that can be taken by the input on the positive side, the certain value that is a curve exists. A device,
Means for generating an edge component from a first image, wherein the first image includes a plurality of pixels, the edge component includes a plurality of elements, and a value of each element of the edge component is a space in the first image Edge component generation means corresponding to the value of each pixel of the first image after removing at least the DC component from the frequency component;
Means for generating a second image from an edge component, wherein the second image includes a plurality of pixels, the means comprising: a color of each pixel of the second image as a function of the value of each element of the edge component; An image generation means including a means for determining, in response to a new input of the first image, an image based on the newly generated second image is output to the display means, The edge component generation means includes means for performing nonlinear processing on the value of each element of the edge component, and the value of the element of the edge component before the nonlinear processing is input as the value of the element of the edge component after the nonlinear processing. Output of the first non-linear function, which is a value between the minimum value that the negative input can take and the maximum value that the positive input can take, Expressed on the two-dimensional xy plane with x as the input and y as the function output The interval between the minimum value that can be taken by the input on the negative side and the certain value or the interval between the certain value and the maximum value that can be taken by the input on the positive side is a curve. A device whose value exists. The apparatus according to any one of claims 1 to 3 , wherein the first nonlinear function is a curve when the function input is represented on a two-dimensional xy plane with x representing the function input and y representing the function output. . The apparatus according to any one of claims 1 to 4 , wherein the section is a curve when the function input in the first nonlinear function is represented on the two-dimensional xy plane as x and the function output is represented on the two-dimensional xy plane, or The first nonlinear function is
(A) a function that multiplies the input to a rational power;
(B) a function that is a rational power of a sine function;
(C) a function that is a rational power of a logarithmic function;
(D) One or both of multiplying a function included in (a), (b) or (c) by a rational number and adding an offset of a rational number to the function included in (a), (b) or (c) The function obtained by
(E) a function obtained by combining at least two of the functions included in (a), (b), (c) or (d), or
(F) A function included in (a), (b), (c), (d) or (e), wherein the input is rationally multiplied before the application of the function, and the rational number offset in the input And a function that performs one or more of taking an absolute value of an input, wherein the rational power is a function that includes the first power and excludes a linear function. 6. The apparatus according to any one of claims 1 to 5 , further comprising means for switching the first nonlinear function between a plurality of nonlinear functions when the first image is input one after another. Equipment. A device,
Means for generating an edge component from a first image, wherein the first image includes a plurality of pixels, the edge component includes a plurality of elements, and a value of each element of the edge component is a space in the first image Edge component generation means corresponding to the value of each pixel of the first image after removing at least the DC component from the frequency component;
Means for generating a second image from an edge component, wherein the second image includes a plurality of pixels, the means comprising: a color of each pixel of the second image as a function of the value of each element of the edge component; An image generation means including a means for determining, in response to a new input of the first image, an image based on the newly generated second image is output to the display means, The edge component generation means includes means for performing nonlinear processing on the value of each element of the edge component, and the value of the element of the edge component before the nonlinear processing is input as the value of the element of the edge component after the nonlinear processing. The apparatus further comprises means for switching the first nonlinear function between a plurality of nonlinear functions when the first image is input one after another. The apparatus according to any one of claims 1 to 7 , wherein the first nonlinear function is a function defined by the first quadrant and the third quadrant and passing through the origin. 9. The apparatus according to claim 1, wherein the edge component generation unit applies a two-dimensional high-pass filter to the first image to at least a direct current component from a spatial frequency component in the first image. An apparatus comprising means for removing 9. The apparatus according to claim 1, wherein the edge component generation means includes a first direction one-dimensional high-pass filter means for the first image and a second direction one-dimensional high pass for the first image. An apparatus comprising: filter means; means for combining the output of the first direction one-dimensional high-pass filter means and the output of the second direction one-dimensional high-pass filter means. The apparatus according to any one of claims 1 to 10 , wherein the edge component generation unit changes the value of the element to a value obtained by multiplying the value of the element of the edge component by a predetermined coefficient, And an apparatus comprising one or both of limiter applying means for changing the value of the element of the edge component to a threshold value when the value of the element of the edge component is larger or smaller than a predetermined threshold value. The apparatus according to any one of claims 1 to 11 , wherein the image generation means calculates an element value having an absolute value greater than an absolute value of a surrounding element value in the edge component. The isolated point removal filter means for replacing with a value obtained from the value of the surrounding element, and the value of the element having an absolute value smaller than the absolute value of the value of the surrounding element in the edge component from the value of the surrounding element An apparatus comprising one or both of a spreading filter means for replacing with a desired value. The apparatus according to claim 12 , wherein the isolated point removal filter means comprises:
Specify a continuous region of 3 elements x 3 elements in the edge component,
In the elements other than the element at the center of the region, the product of the maximum value that the element can take and the first ratio is used as a threshold value, and the number of elements having an absolute value equal to or less than the threshold value is obtained.
The spread filter means configured to replace the value of the element at the center of the region with the median value of the elements excluding the element at the center of the region when the number of elements obtained is equal to or greater than the first number, Is
Specify a continuous region of 3 elements x 3 elements in the edge component,
In the elements other than the element at the center of the region, the product of the maximum value that can be taken by the element and the second ratio is used as a threshold, and the number of elements having an absolute value equal to or larger than the threshold is obtained.
An apparatus configured to replace a value of a central element of a region with a median value of elements excluding the central element of the region when the determined number of elements is equal to or greater than a second number. 14. The apparatus according to any one of claims 1 to 13 , wherein the first image is generated from an original image, the original image includes a plurality of pixels, and the value of each pixel of the first image is an original value. Derived from the color of the corresponding pixel of the image,
Another method for generating an image to be output to the display means by replacing at least a part of the original image with the second image and / or superimposing the original image and the second image. An apparatus further comprising the image generating means. 15. The apparatus according to claim 14 , wherein another image generation unit performs superposition of the original image and the second image,
C _out = αC _in1 + βC _in2
(Α is a real number satisfying 0 ≦ α ≦ 1, β is a real number satisfying 0 ≦ β ≦ 1)
Is used to obtain the color of a certain pixel in the image after superimposition, where C _out is a value specifying the color of a certain pixel in the image after superimposition, and C _in1 and C _in2 are respectively the original values. An apparatus that is a value that identifies the color of the corresponding pixel in the image and the second image. 16. The apparatus according to claim 14 , wherein the other image generation means includes means for color-adjusting the original image. A program that causes a computer to function as the device according to claim 1. A method,
The step of generating an edge component from the first image by the edge component generation means, wherein the first image includes a plurality of pixels, the edge component includes a plurality of elements, and the value of each element of the edge component is: An edge component generation step corresponding to the value of each pixel of the first image after removing at least the direct current component from the spatial frequency component in the first image;
Generating a second image from the edge component by the image generating means, the second image including a plurality of pixels, wherein the step includes the second image as a function of the value of each element of the edge component; An image generation step including the step of determining the color of each pixel of
An image output step of outputting an image based on the second image to the display means by the image output means, and in response to a new input of the first image, an edge component generation step, an image generation step, and an image output The steps are repeated, and the edge component generation step includes a step of performing nonlinear processing on the value of each element of the edge component before applying the limiter, and the value of the element of the edge component after the nonlinear processing is The output of the first nonlinear function that takes the value of the element of the previous edge component as an input. In the first nonlinear function, the minimum value that the input on the negative side can take and the maximum value that the input on the positive side can take An interval between the minimum value and the certain value that can be taken by the input on the negative side when the function input is represented on the two-dimensional xy plane with x representing the function input and y representing the function output. Or said That values and intervals between the maximum value input can take on the positive side there is the certain value is a curve, method. A method,
The step of generating an edge component from the first image by the edge component generation means, wherein the first image includes a plurality of pixels, the edge component includes a plurality of elements, and the value of each element of the edge component is: An edge component generation step corresponding to the value of each pixel of the first image after removing at least the DC component from the spatial frequency component in the first image;
Generating a second image from the edge component by the image generating means, the second image including a plurality of pixels, wherein the step includes the second image as a function of the value of each element of the edge component; An image generation step including the step of determining the color of each pixel of
An image output step of outputting an image based on the second image to the display means by the image output means, and in response to a new input of the first image, an edge component generation step, an image generation step, and an image output steps are repeated, the edge component generating step includes a step of performing nonlinear processing on the value of each element of the edge components, the values of the elements of the edge component after the non-linear processing, before the non-linear processing edge component The second nonlinear function is a composite function of the first nonlinear function and at least a function for applying the limiter, and is negative in the first nonlinear function. It is a certain value between the minimum value that can be taken by the input on the side and the maximum value that can be taken by the input on the positive side, where x is the function input and y is the function output. Further, the interval between the minimum value and the certain value that can be taken by the input on the negative side or the interval between the certain value and the maximum value that can be taken by the input on the positive side is the certain value that is a curve. There is a way. A method,
The step of generating an edge component from the first image by the edge component generation means, wherein the first image includes a plurality of pixels, the edge component includes a plurality of elements, and the value of each element of the edge component is: An edge component generation step corresponding to the value of each pixel of the first image after removing at least the direct current component from the spatial frequency component in the first image;
Generating a second image from the edge component by the image generating means, the second image including a plurality of pixels, wherein the step includes the second image as a function of the value of each element of the edge component; An image generation step including the step of determining the color of each pixel of
An image output step of outputting an image based on the second image to the display means by the image output means, and in response to a new input of the first image, an edge component generation step, an image generation step, and an image output The steps are repeated, and the edge component generation step includes a step of performing nonlinear processing on the value of each element of the edge component, and the value of the element of the edge component after the nonlinear processing is the value of the edge component before the nonlinear processing. An output of a first nonlinear function having an element value as an input, and in the first nonlinear function, a value between a minimum value that can be taken by an input on the negative side and a maximum value that can be taken by an input on the positive side. When the function input is represented by x and the function output is represented by y on the two-dimensional xy plane, the interval between the minimum value and the certain value that can be taken by the input on the negative side or the certain value On the positive side Interval between the maximum value force can take is there is the certain value is a curve, method. 21. The method according to any one of claims 18 to 20 , wherein the first non-linear function is a curve when the input of the function is represented on a two-dimensional xy plane with x representing the function input and y representing the function output. . The method according to any one of claims 18 to 21 , wherein the function input in the first nonlinear function is x and the output of the function is y, and the section is a curve when represented on a two-dimensional xy plane, or The first nonlinear function is
(A) a function that multiplies the input to a rational power;
(B) a function that is a rational power of a sine function;
(C) a function that is a rational power of a logarithmic function;
(D) One or both of multiplying a function included in (a), (b) or (c) by a rational number and adding an offset of a rational number to the function included in (a), (b) or (c) The function obtained by
(E) a function obtained by combining at least two of the functions included in (a), (b), (c) or (d), or
(F) A function included in (a), (b), (c), (d) or (e), wherein the input is rationally multiplied before the application of the function, and the rational number offset in the input And a function that performs one or more of taking an absolute value of an input, wherein the rational power is a function that includes the first power and excludes a linear function. 23. The method according to any one of claims 18 to 22 , further comprising the step of switching the first nonlinear function between a plurality of nonlinear functions when the first image is input one after another. A method,
The step of generating an edge component from the first image by the edge component generation means, wherein the first image includes a plurality of pixels, the edge component includes a plurality of elements, and the value of each element of the edge component is: An edge component generation step corresponding to the value of each pixel of the first image after removing at least the direct current component from the spatial frequency component in the first image;
Generating a second image from the edge component by the image generating means, the second image including a plurality of pixels, wherein the step includes the second image as a function of the value of each element of the edge component; An image generation step including the step of determining the color of each pixel of
An image output step of outputting an image based on the second image to the display means by the image output means, and in response to a new input of the first image, an edge component generation step, an image generation step, and an image output The steps are repeated, and the edge component generation step includes a step of performing nonlinear processing on the value of each element of the edge component, and the value of the element of the edge component after the nonlinear processing is the value of the edge component before the nonlinear processing. A method further comprising the step of switching the first non-linear function between a plurality of non-linear functions when the first image is input one after another, the output of the first non-linear function having an element value as an input. 25. The method according to any one of claims 18 to 24 , wherein the first nonlinear function is a function defined in the first and third quadrants and passing through the origin. 26. The method according to any one of claims 18 to 25 , wherein the edge component generation step includes a two-dimensional high-pass filter application step on the first image. 26. The method according to any one of claims 18 to 25 , wherein the edge component generation step includes a first direction one-dimensional high-pass filter applying step for the first image and a second direction one-dimensional for the first image. A method comprising: applying a high-pass filter; and synthesizing outputs obtained by the first-direction one-dimensional high-pass filter application step and the second-direction one-dimensional high-pass filter application step. The method according to any one of claims 18 to 27 , wherein the edge component generation step changes the value of the element to a value obtained by multiplying the value of the element of the edge component by a predetermined coefficient, And a limiter application step of changing the value of the element of the edge component to a threshold when the value of the element of the edge component is larger or smaller than a predetermined threshold. 29. The method according to any one of claims 18 to 28 , wherein the image generation step includes a value of an element having an absolute value greater than an absolute value of a value of a surrounding element in the edge component. An isolated point removal filter applying step that replaces with a value obtained from the value of the surrounding element, and the value of the element having an absolute value smaller than the absolute value of the value of the surrounding element in the edge component A method comprising one or both of applying a broadening filter to replace with a value obtained from. 30. The method of claim 29 , wherein the isolated point removal filter applying step comprises:
Identifying a continuous region of 3 elements × 3 elements in the edge component;
Obtaining a number of elements having an absolute value equal to or less than the threshold, with a product of the maximum value that can be taken by the element and the first ratio as a threshold in the elements other than the central element of the region;
Applying a spread filter including replacing the value of the central element of the region with the median of the values of the elements excluding the central element of the region when the number of elements obtained is equal to or greater than the first number The steps are
Identifying a continuous region of 3 elements × 3 elements in the edge component;
Obtaining a number of elements having an absolute value greater than or equal to a threshold value, with a product of a value obtained by subtracting the maximum value that can be taken by the element and the second ratio as the threshold value in elements other than the central element of the region;
Replacing the value of the central element of the region with the median of the values of the elements excluding the central element of the region when the determined number of elements is greater than or equal to the second number. 31. The method according to any one of claims 18 to 30 , wherein the first image is generated from the original image, the original image includes a plurality of pixels, and the value of each pixel of the first image is the original value. Derived from the color of the corresponding pixel of the image,
Output to the display means by replacing at least a part of the original image with the second image and / or superimposing the original image and the second image by another image generating means A method further comprising another image generation step of generating a processed image. 32. The method according to claim 31 , wherein in another image generation step, when the original image and the second image are superimposed, another image generation means
C _out = αC _in1 + βC _in2
(Α is a real number satisfying 0 ≦ α ≦ 1, β is a real number satisfying 0 ≦ β ≦ 1)
To calculate the color of a certain pixel in the image after superposition, where C _out is a value specifying the color of a certain pixel in the image after superposition, and C _in1 and C _in2 are the original values, respectively. A method that is a value that identifies the color of the corresponding pixel in the image and the second image. 33. A method according to claim 31 or 32 , wherein the further image generating step further comprises the step of color adjusting the original image.