JP6232712B2 - Image processing apparatus, imaging apparatus, and image processing program - Google Patents

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program.

  For example, a plurality of feature components such as a color component, a contrast component, and an edge component are acquired from an image, and a weighted addition is performed on the acquired map (feature map) for each of the plurality of feature components and a ratio set for each feature component. Thus, a technique for generating a saliency map indicating a distribution of saliency has been proposed (see Patent Document 1). This saliency map has a property that the saliency with respect to the region of interest of the image is higher than the saliency of other regions.

JP 2011-34311 A

  Here, the ratio set for each of the plurality of feature components often uses a preset fixed value. When this ratio is a fixed value, depending on the environment at the time of image acquisition, the ratio for each feature map may not be appropriate, and a saliency map may not be generated appropriately.

  It is an object of the present invention to provide an image processing apparatus, an imaging apparatus, and an image processing program that can appropriately generate a saliency map.

One aspect of the image processing apparatus of the present invention shows the saliency in the image based on a feature map for each of a plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components. A plurality of features based on the information acquired by the map generating means for generating the saliency map, the information acquiring means for acquiring information indicating the focal length of the imaging optical system at the time of image acquisition, and the information acquired by the information acquiring means Adjusting means for adjusting the contribution rate associated with at least two or more feature components among the components, wherein one of the plurality of feature components is a directional edge component, and the adjusting means Adjusts the contribution rate associated with the directional edge component to a value smaller than the initial value when the focal length of the imaging optical system is shortened.
Further, a map generation unit that generates a saliency map indicating a saliency in the image based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components. And a plurality of characteristics based on the information acquired by the information acquisition means, the information acquisition means for acquiring information indicating whether the image is an image representing the intensity of light by color shading. Adjusting means for adjusting the contribution rate associated with at least two or more feature components among the components, one of the plurality of feature components is a color component, and the adjusting means includes: When the image is an image that expresses the intensity of light with color shading, the contribution rate associated with the color component is adjusted to zero.
According to another aspect of the image processing apparatus of the present invention, the degree of saliency in the image is based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components. Based on the information acquired by the map acquisition means for generating the saliency map, the information acquisition means for acquiring information indicating the brightness of the object scene when the image is acquired, and the information acquired by the information acquisition means Adjusting means for adjusting the contribution ratio associated with at least two or more feature components among the plurality of feature components, wherein the plurality of feature components include a color component, a luminance contrast component, and a directional edge. And the adjustment means associates with at least one of the color component and the luminance contrast component when the brightness of the object scene when the image is acquired is dark The The contribution ratio is adjusted to a value smaller than the initial value.
According to another aspect of the image processing apparatus of the present invention, the degree of saliency in the image is based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components. Based on the information acquired by the information acquisition means, the information acquisition means for acquiring defocus amount information in the image, and the information acquisition means, Adjusting means for adjusting the contribution rate associated with at least two or more feature components, wherein one of the plurality of feature components is a directional edge component, As the defocus amount in the image is larger, the contribution rate associated with the directional edge component is adjusted to a value smaller than the initial value.
According to another aspect of the image processing apparatus of the present invention, the degree of saliency in the image is based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components. Based on the information acquired by the information acquisition means, the information acquisition means for acquiring the contrast information in the image, and the information acquisition means, at least of the plurality of characteristic components Adjusting means for adjusting the contribution rate associated with two or more feature components, wherein the plurality of feature components are a color component, a luminance contrast component, and a directional edge component, As the contrast value obtained from the image is smaller, the contribution rate associated with the color component is adjusted to a value larger than the initial value.
According to another aspect of the image processing apparatus of the present invention, the degree of saliency in the image is based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components. A map generation means for generating a saliency map indicating information, an information acquisition means for acquiring information on whether or not the image is an image obtained during a focusing operation, and the information acquired by the information acquisition means. And adjusting means for adjusting the contribution ratio associated with at least two or more feature components among the plurality of feature components, wherein one of the plurality of feature components is a directional edge. The adjustment unit adjusts the contribution rate associated with the directional edge component to a value smaller than an initial value when the image is an image obtained during a focusing operation.
According to another aspect of the image processing apparatus of the present invention, the degree of saliency in the image is based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components. Based on the information acquired by the information acquisition means for acquiring information related to the focus operation, the information acquisition means for generating information related to the focus operation, at least two of the plurality of characteristic components Adjusting means for adjusting the contribution rate associated with the above feature components, wherein one of the plurality of feature components is a directional edge component, and the adjusting means is configured to perform the focusing operation. In the single AF operation, the contribution rate associated with the directional edge component used when generating the saliency map obtained from the image based on the operation of the release button is Than contribution associated with the directional edge components to be used in generating the saliency map obtained from the image based on non-operation of the release button to adjust to a large value.

  The imaging apparatus of the present invention includes the above-described image processing apparatus, an imaging optical system capable of performing a focusing operation, and an imaging element that receives subject light captured by the imaging optical system and outputs an image signal. And.

According to another aspect of the image processing program of the present invention, the degree of saliency in the image is based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components. Based on the map generation step of generating a saliency map indicating information, the information acquisition step of acquiring information indicating the focal length of the imaging optical system at the time of image acquisition, and the information acquired by the information acquisition step An adjustment step of adjusting the contribution rate associated with at least two or more feature components among the feature components, wherein one of the plurality of feature components is a directional edge component, The adjustment step adjusts the contribution rate associated with the directional edge component to a value smaller than the initial value when the focal length of the imaging optical system is shortened.
According to another aspect of the image processing program of the present invention, the degree of saliency in the image is based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components. Acquired by the map generation step of generating a saliency map indicating information, an information acquisition step of acquiring information indicating whether or not the image is an image in which the intensity of light is represented by color shading, and the information acquisition step An adjustment step of adjusting the contribution rate associated with at least two or more feature components among the plurality of feature components based on the information, wherein one of the plurality of feature components is The color component, and the adjusting step adjusts the contribution rate associated with the color component to 0 when the image is an image representing the intensity of light in the shade of color.
According to another aspect of the image processing program of the present invention, the degree of saliency in the image is based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components. A map generation step for generating a saliency map, an information acquisition step for acquiring information indicating the brightness of the object scene when the image is acquired, and the information acquired by the information acquisition step Adjusting the contribution ratio associated with at least two or more feature components among the plurality of feature components, wherein the plurality of feature components include a color component, a luminance contrast component, and a directional edge. And the adjusting step applies to at least one of the color component and the luminance contrast component when the brightness of the object scene when the image is acquired is dark. The Tagged contribution adjusted to a value smaller than the initial value.
According to another aspect of the image processing program of the present invention, the degree of saliency in the image is based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components. A map generation step for generating a saliency map, an information acquisition step for acquiring information on a defocus amount in the image, and the information acquired by the information acquisition step. Adjusting the contribution rate associated with at least two or more feature components, wherein one of the plurality of feature components is a directional edge component, and the adjustment step includes: As the defocus amount in the image is larger, the contribution rate associated with the directional edge component is adjusted to a value smaller than the initial value.
According to another aspect of the image processing program of the present invention, the degree of saliency in the image is based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components. A map generation step for generating a saliency map, an information acquisition step for acquiring contrast information in the image, and at least one of the plurality of feature components based on the information acquired by the information acquisition step. An adjustment step of adjusting the contribution rate associated with two or more feature components, wherein the plurality of feature components are a color component, a luminance contrast component, and a directional edge component, and the adjustment step includes: As the contrast value obtained from the image is smaller, the contribution rate associated with the color component is adjusted to a value larger than the initial value.
According to another aspect of the image processing program of the present invention, the degree of saliency in the image is based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components. A map generation step of generating a saliency map indicating information, an information acquisition step of acquiring information on whether or not the image is an image obtained during a focusing operation, and the information acquired by the information acquisition step An adjustment step of adjusting the contribution rate associated with at least two or more feature components among the plurality of feature components, wherein one of the plurality of feature components is a directional edge. The adjustment step adjusts a contribution rate associated with the directional edge component to a value smaller than an initial value when the image is an image obtained during a focusing operation.
According to another aspect of the image processing program of the present invention, the degree of saliency in the image is based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components. A map generation step for generating a saliency map indicating information, an information acquisition step for acquiring information related to a focus operation, and at least two of the plurality of feature components based on the information acquired by the information acquisition step An adjustment step of adjusting the contribution rate associated with the above feature components, wherein one of the plurality of feature components is a directional edge component, and the adjustment step is performed when the focus operation is performed Contribution rate associated with a directional edge component used when generating a saliency map obtained from an image based on an operation of a release button in a single AF operation , Adjusted to a value greater than the contribution factor associated with the directional edge components to be used in generating the saliency map obtained from the non-operation based on the image of the release button.

  According to the present invention, a saliency map can be appropriately generated.

It is a figure which shows one structure of the imaging device of this invention. It is a figure which shows the structure of a map production | generation part. It is a flowchart which shows the flow of the process at the time of imaging | photography.

  Hereinafter, an embodiment of the present invention will be described with reference to FIG. As shown in FIG. 1, the imaging apparatus 10 includes an imaging optical system 15, an imaging element 16, an operation member 17, a lens driving circuit 18, a photometric circuit 19, a CPU 20, an imaging element driving circuit 21, a signal processing circuit 22, and a data processing circuit. 23, a liquid crystal monitor 24, a compression / decompression processing circuit 25, a display output circuit 26, and a storage medium 27.

  A subject image (not shown) is formed on the image pickup surface of the image pickup device 16 by the image pickup optical system 15. The imaging optical system 15 includes a plurality of lenses, and is configured to be able to adjust focus, zoom, and the like via the lens driving circuit 18 based on the operation of the operation member 17 and the like.

  The image sensor 16 is an image sensor that can capture a still image as well as continuously capture a still image and a moving image, and is composed of, for example, a CCD image sensor or a CMOS image sensor. The image pickup device 16 is driven by the image pickup device drive circuit 21 under the control of the CPU 20 based on the subject photometry data obtained by the photometry circuit 19. The image signal read by the image sensor 16 is input to the signal processing circuit 22.

  The signal processing circuit 22 performs signal processing such as direct current reproduction processing, A / D conversion processing, white balance processing, and γ conversion processing on the image signal read out by the image pickup device 16 to obtain data processing circuit as image data. To 23.

  The data processing circuit 23 performs resolution (number of pixels) conversion processing for displaying on the input image data on the liquid crystal monitor 24, outputs the converted image data to the display control circuit 26, and performs processing by the data processing circuit 23. The processed image data is output to the compression / decompression processing circuit 25. The display output circuit 26 performs predetermined signal processing on the image data input from the data processing circuit 23 and outputs the processed image data to the liquid crystal monitor 24. The display output circuit 26 performs processing for superimposing overlay image data such as a shooting menu and a cursor on the image data output from the data processing circuit 23 under the control of the CPU 20. Thus, the overlay image is superimposed on the subject image and displayed on the liquid crystal monitor 24.

  The compression / decompression processing circuit 25 performs compression processing on the input image data and stores it in the storage medium 27. The liquid crystal monitor 24 displays an image corresponding to the image data compressed and stored in the storage medium 27.

  The liquid crystal monitor 24 can also display an image corresponding to the image data stored in the storage medium 27 as a reproduced image. The CPU 20 reads out the image data stored in the storage medium 27, performs a decoding process by the compression / decompression processing circuit 25, and supplies it to the liquid crystal monitor 24 via the data processing circuit 23 and the display output circuit 26.

  The CPU 20 calculates a focus evaluation value in a preset area (AF area) on the imaging screen based on the operation of a release button that constitutes a part of the operation member 17, and performs imaging based on the calculation result. An operation of adjusting the focus state of the subject image on the imaging surface of the element 16 is executed. For example, the focus evaluation value of each AF area is calculated based on the output of focus state detection pixels (phase difference AF pixels) arranged on the imaging surface of the image sensor 16.

  In the present embodiment, the AF operation using the focus state detection pixels (phase difference AF pixels) arranged on the imaging surface of the image sensor 16 is described as an example. However, the present invention is not limited to this. It is also possible to use a known contrast AF operation (TV-AF). Also in this case, the shooting distance information of each region can be obtained by the automatic focusing operation.

  The CPU 20 drives the imaging optical system 15 via the lens driving circuit 18 based on the operation of the zoom operation member that constitutes a part of the operation member 17, and the subject image formed on the imaging surface of the image sensor 16. An optical zoom operation for enlarging or reducing the image is performed. Further, the CPU 20 converts the image data obtained by the imaging device 16 or the image data stored in the storage medium 27 based on the operation of the zoom operation member constituting a part of the operation member 17 into the resolution by the data processing circuit 23. (Number of pixels) An electric zoom operation for enlarging or reducing by the conversion process is controlled.

  Next, the configuration of the data processing circuit 23 will be described. As shown in FIG. 1 or 2, the data processing circuit 23 has functions of an information acquisition unit 31, a ratio adjustment unit 32, and a map generation unit 33. The information acquisition unit 31 acquires from the CPU 20 information on the operation mode (single AF mode, continuous AF mode) and shooting mode (including scene mode) of the automatic focusing operation set at the time of shooting, and shooting condition information. To do. Examples of information on the imaging conditions include information on the brightness of the object scene and information on the type of light source in addition to information on the focal length and defocus amount in the imaging optical system 15. In addition to this, the information acquisition unit 31 acquires luminance contrast information in the image.

  The ratio adjustment unit 32 adjusts the ratio (contribution rate) affected by each feature map in the saliency map obtained from the plurality of feature maps based on the information obtained by the information acquisition unit 31. Hereinafter, a case where a color component, a luminance contrast component, and a directional edge component are used as the plurality of characteristic components will be described as an example.

  As a contribution rate in each of the plurality of feature maps (or feature components), a preset ratio (initial ratio) is assigned. The ratio adjustment unit 32 adjusts the ratio of at least two feature components based on the information obtained by the information acquisition unit 31 with reference to the initial ratio assigned to each feature component. Then, the ratio adjustment unit 32 sets the ratio for each feature component after adjustment as a contribution ratio used when generating the saliency map. Note that the sum of the initial ratios assigned to each feature component and the sum of the ratios after adjustment are 1.

  Here, the initial ratio with respect to each feature component may be a fixed value regardless of the shooting mode (including the scene mode), or may be a different value for each shooting mode (including the scene mode). For example, when different initial ratios are set for each of a plurality of shooting modes (including scene modes), the ratio adjustment unit 32 sets each feature component based on shooting mode information obtained by the information acquisition unit 31. The initial ratio corresponding to the shooting mode is selected from the plurality of initial ratios. Then, the ratio adjustment unit 32 may adjust the ratio of at least two feature components based on the information obtained by the information acquisition unit 31 with reference to the initial ratio of each selected feature component.

  The map generation unit 33 generates, for each feature component, a map (feature map) related to a plurality of feature components obtained from the image data, using the image processed image data applied by the data processing circuit 23. Here, examples of the feature map include a luminance information map, a color information map, and an edge information map. The map generation unit 33 generates a saliency map indicating the distribution of the saliency of the image using the feature map generated for each feature component. When the saliency map is generated, the contribution rate set by the ratio adjustment unit 32 is used.

  The map generation unit 33 includes a luminance information extraction unit 41, a color information extraction unit 42, an edge information extraction unit 43, and a map synthesis unit 44. The luminance information extraction unit 41 generates luminance image data in which the pixel value of each pixel is a luminance value based on the input image data. The luminance information extraction unit 41 generates a plurality of luminance image data with different resolutions using the generated luminance image data. The luminance information extraction unit 41 generates luminance image data of eight resolutions from level 1 to level 8, for example, by sequentially performing Gaussian filter processing on the luminance image data. Note that the luminance image data at level 1 is set to have the same resolution as the input image data. The resolution is set lower in the order of level 2 luminance image data, level 3 luminance image data,..., Level 8 luminance image data.

  After generating the luminance image data having different resolutions, the luminance information extraction unit 41 selects two adjacent luminance images such as level 1 luminance image data and level 2 luminance image data from among the generated plurality of luminance image data. A level of luminance image data is selected, and a difference between the two selected luminance image data (hereinafter, difference data) is obtained. In addition, when the number of pixels of each luminance image data is different, when obtaining the difference data, the luminance information extraction unit 41 determines that the number of pixels of the luminance image data with low resolution among the two luminance image data is the resolution. The number of pixels of low-resolution luminance image data is converted so that the number of pixels of high-luminance image data is high.

  The luminance information extraction unit 41 generates a predetermined number of difference data, and then normalizes the generated difference data. Based on the normalized difference data, a luminance information extraction unit 41 luminance information map is generated. The generated luminance information map is output to the map composition unit 44.

  The color information extraction unit 42 generates RG difference image data including a difference between a red (R) color component value and a green (G) color component value using the input image data. And the color information extraction part 42 produces | generates several RG difference image data from which resolution differs using the produced | generated RG difference image data. Thereby, RG difference image data of eight resolutions from level 1 to level 8, for example, is generated in the same manner as the luminance image data.

  The color information extraction unit 42 selects two adjacent levels of RG difference image data, such as level 1 RG difference image data and level 2 RG difference image data, from among the plurality of generated RG difference image data. The difference between the two selected RG difference image data (hereinafter referred to as difference data) is obtained. Also in this case, when the number of pixels of the plurality of RG difference image data is different from each other, the color information extraction unit 42 determines that the number of pixels of the RG difference image data with low resolution among the two RG difference image data is the resolution. The number of pixels of the RG difference image data having a low resolution is converted so that the number of pixels of the RG difference image data is high. The color information extraction unit 42 normalizes the difference data obtained from the RG difference image data, and then generates a color information map based on the difference between the red (R) color component value and the green (G) color component value.

  In addition, the color information extraction unit 42 uses the input image data to calculate the difference between the blue (B) color component value and the yellow (Y) color component value by the same method as the RG difference image data generation described above. BY difference image data is generated. Then, the color information extraction unit 42 performs the same processing as that of the RG difference image data, and generates a color information map based on the difference between the blue (B) color component value and the yellow (Y) color component value. Each color information map generated by the color information extraction unit 42 is output to the map synthesis unit 44.

  The edge information extraction unit 43 performs a filtering process using a Gabor filter on the input image data. By this filtering process, for example, edge image data having edge strengths in directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees as pixel values on the screen is generated. The edge information extraction unit 43 further generates a plurality of edge image data having different resolutions using the generated edge image data. Also in this case, for example, edge image data of eight resolutions from level 1 to level 8 are generated by sequentially performing the obtained cow draft filter processing. The edge information extraction unit 43 selects two adjacent edge image data such as level 1 edge image data and level 2 edge image data from among the generated plurality of edge image data, and selects the selected 2 A difference between the two edge image data (hereinafter, difference data) is obtained. Also in this case, when the number of pixels of the plurality of edge image data is different from each other, the edge information extraction unit 43 sets the edge number of the edge image data having the low resolution among the two edge image data to the edge having the high resolution. The number of pixels of edge image data having a low resolution is converted so as to be the number of pixels of the image data. The edge information extraction unit 43 normalizes the difference data obtained from the edge image data, and then generates an edge information map based on the normalized difference data. Note that the edge information extraction unit 43 performs the above-described processing for each edge image data in each direction. These generated edge information maps are output to the map composition unit 44.

  The map synthesis unit 44 generates a saliency map by weighted addition using the input luminance information map, color information map, edge information map, and contribution rate set for each feature component. This saliency map is a map showing the distribution of saliency at each pixel in the image.

  Next, the flow of processing during imaging will be described using the flowchart shown in FIG. In this flowchart, processing related to the automatic exposure operation is omitted.

  Step S101 is processing for acquiring a through image. The CPU 20 drives the image sensor 16 via the image sensor drive circuit 21. This is a process for acquiring a through image. When the shooting mode is started, the CPU 20 controls the image sensor 16 via the image sensor drive circuit 21. Thereby, the image sensor 16 is driven with a predetermined time interval. Thereby, the image signal which becomes the origin of a through image is acquired. This image signal is output to the signal processing circuit 22. The signal processing circuit 22 performs signal processing such as direct current reproduction processing, A / D conversion processing, white balance processing, and γ conversion processing on the input image signal, and outputs it to the data processing circuit 23 as image data.

  If a through image is to be displayed, the image data is output to the display output circuit 26 via the data processing circuit 23. In response to this, the display output circuit 26 displays an image based on the input image data on the liquid crystal monitor 24. As a result, a through image is displayed on the liquid crystal monitor 24.

  Step S102 is a process for acquiring information. The data processing circuit 23 receives from the CPU 20 information on the operation mode (single AF mode, continuous AF mode) and shooting mode (including scene mode) of automatic focusing operation set at the time of shooting, and information on shooting conditions. get.

  Step S103 is processing for setting a contribution rate. The data processing circuit 23 sets a contribution rate when generating the saliency map based on the information acquired in step S102.

  Here, a specific example of setting the contribution rate will be described. The ratio adjustment unit 32 reads the initial ratio allocated to each feature component. Hereinafter, a case where the initial ratio of each characteristic component of the color component, the luminance contrast component, and the directional edge component is set to 0.33 will be described as an example.

After reading the initial ratio, the ratio adjusting unit 32 based on the information the information acquisition unit 31 has acquired, the initial ratio of each feature component as a reference, adjusting the ratio of at least 2 or more feature components.

[Example 1]
A case where the ratio to each feature component is adjusted based on information on the focal length f of the imaging optical system 15 will be described. The CPU 20 calculates the focal length f from the position information of each lens of the imaging optical system 15. The ratio adjusting unit 32 adjusts the ratio for each feature component based on the information on the focal length f.

  For example, when the focal length f is 28 mm or less (when f ≦ 28 mm), the ratio adjusting unit 32 increases the ratio with respect to the color component and the luminance contrast component (0.33 → 0.5), and with respect to the directional edge component. Set the ratio to zero.

  Further, when the focal length f exceeds 28 mm and is 50 mm or less (when 28 mm <f ≦ 50 mm), the ratio adjusting unit 32 increases the ratio with respect to the color component and the luminance contrast component (0.33 → 0.45). ), The ratio to the directional edge component is lowered (0.33 → 0.1).

  When the focal length f exceeds 50 mm and is 120 mm or less (when 50 mm <f ≦ 120 mm), the ratio adjusting unit 32 increases the ratio with respect to the color component and the luminance contrast component (0.33 → 0.4). ), The ratio to the directional edge component is lowered (0.33 → 0.2).

  Furthermore, when the focal length f exceeds 120 mm (when 120 mm <f), the ratio adjusting unit 32 does not adjust the ratio for each color component. In this case, the ratio with respect to each feature component remains the initial ratio (0.33).

  Thus, the shorter the focal length, the wider the angle of view, and the difference in the edge of the subject in the image is less likely to occur. For this reason, the ratio with respect to each characteristic component is adjusted so that the ratio with respect to the directional edge is lowered and the ratio with respect to the color component and the luminance contrast component is increased as the focal length is shorter.

[Example 2]
A case where the ratio to each feature component is adjusted based on information on the shooting mode will be described. The shooting mode is set based on the operation of the operation member 17 by the user at the time of shooting.

  When the monochrome shooting mode is set as the shooting mode, the ratio adjusting unit 32 increases the ratio to the luminance contrast component and the directional edge component (0.33 → 0.5), and sets the ratio to the color component to zero. On the other hand, when another shooting mode is set as the shooting mode instead of the monochrome shooting mode, the ratio adjusting unit 32 does not adjust the ratio with respect to each feature component. In this case, the ratio with respect to each feature component remains the initial ratio (0.33).

  In the case of the monochrome photography mode described above, in the image, the contribution of the color component is low, and the contribution of the luminance contrast component and the directional edge component is high. Therefore, when the monochrome photography mode is set, the color component The ratio to each feature component is adjusted so that the ratio to the luminance contrast component and the directional edge component is increased.

  Note that the monochrome shooting mode includes a shooting mode for acquiring an image in which the intensity of light is expressed in shades of color, such as a sepia tone image, in addition to a shooting mode for acquiring a monochrome image.

[Example 3]
Next, the case where the ratio with respect to each feature component is adjusted based on information on the brightness of the object scene will be described. Here, the case where the brightness information obtained from the acquired through image is used as the brightness information of the object scene will be described. Note that, as the adjustment of the ratio to each feature component based on the brightness information of the object scene, for example, it is conceivable to lower at least one of the ratio to the color component and the ratio to the luminance contrast component. [Table 3] shows a case where the ratio to the color component is lowered.

The ratio adjustment unit 32 normalizes the brightness in the through image obtained from the information on the brightness of the scene so that the maximum value that the brightness of the scene can take is 1. After this normalization, the ratio adjusting unit 32 adjusts the ratio for each feature component according to the value that the maximum brightness value B _max in the through image can take. For example, when the maximum brightness value B _max exceeds 0.3 and is 1 or less (when 0.3 <B _max ≦ 1), the ratio adjustment unit 32 adjusts the ratio for each feature component. do not do. In this case, the ratio with respect to each feature component remains the initial ratio (0.33).

On the other hand, when the maximum brightness value B _max is 0.3 or less (when B _max ≦ 0.3), the ratio adjusting unit 32 reduces the ratio to the color component (0.33 → 0.1), The ratio to the luminance contrast component and the directional edge component is increased (0.33 → 0.45).

  Next, the case where the ratio to the luminance contrast component is lowered is shown in [Table 4].

For example, when the maximum brightness value B _max exceeds 0.3 and is 1 or less (when 0.3 <B _max ≦ 1), the ratio adjusting unit 32 does not adjust the ratio for each feature component. . In this case, the ratio with respect to each feature component remains the initial ratio (0.33). On the other hand, when the maximum brightness value B _max is 0.3 or less (B _max ≦ 0.3), the ratio adjusting unit 32 decreases the ratio to the luminance contrast component (0.33 → 0.1), and the color The ratio to the component and the directional edge component is increased (0.33 → 0.45).

  Next, [Table 5] shows a case where the ratio to the color component and the luminance contrast component is lowered.

For example, when the maximum brightness value B _max exceeds 0.3 and is 1 or less (when 0.3 <B _max ≦ 1), the ratio adjusting unit 32 does not adjust the ratio for each feature component. . In this case, the ratio with respect to each feature component remains the initial ratio (0.33). On the other hand, when the maximum brightness value B _max is 0.3 or less (B _max ≦ 0.3), the ratio adjusting unit 32 reduces the ratio to the color component and the ratio to the luminance contrast component (0.33 → 0.1) Increase the ratio to the directional edge component (0.33 → 0.8).

  In this example 3, when the object scene becomes dark, the color component and the luminance contrast value included in the image generally decrease. Therefore, the ratio to the color component and the luminance contrast component is lowered, and the ratio to the directional edge component. Raise.

  In addition, when adjusting the ratio to each feature component based on the brightness information of the object scene, the maximum value of the brightness in the image is used, but it is not necessary to be limited to this, and the brightness in the image It is also possible to adjust the ratio for each feature component using the average value of.

[Example 4]
Next, a case where the ratio for each feature component is adjusted based on the defocus amount information will be described. In the imaging apparatus 10, an automatic focusing operation is performed using the set AF area. That is, the defocus amount is acquired by performing the automatic focusing operation. The ratio adjusting unit 32 uses the acquired defocus amount information to adjust the ratio for each feature component. Specifically, the ratio to each feature component is adjusted based on the value that the minimum defocus amount DF _min can take. An example of this case is shown in [Table 6].

  Here, α and β are threshold values, and α <β.

For example, when the minimum value DF _{min of the} defocus amount is equal to or greater than the threshold value β (when β ≦ DF _min ), the ratio adjusting unit 32 increases the ratio with respect to the color component and the ratio with respect to the luminance contrast component (0.33 → 0). .5), the ratio to the directional edge component is set to zero.

In addition, when the minimum defocus amount DF _min is less than the threshold β and is equal to or greater than the threshold α (when α ≦ DF _min <β), the ratio adjusting unit 32 increases the ratio to the color component and the luminance contrast component ( 0.33 → 0.4), and lower the ratio to the directional edge component (0.33 → 0.2).

When the minimum defocus amount DF _min is equal to or less than the threshold value α (when DF _min ≦ α), the ratio adjusting unit 32 does not adjust the ratio for each feature component. In this case, the ratio with respect to each feature component remains the initial ratio (0.33).

  In [Example 4], the larger the defocus amount value on the screen, the worse the focus state in the obtained image. Therefore, the ratio to the directional edge component is decreased and the ratio to the color component and the luminance contrast component is increased. The ratio for each feature component is adjusted.

  In [Example 4], the minimum value of the defocus amount is used. However, the present invention is not limited to this, and an average value of the defocus amount may be used.

  The defocus amount can be obtained as a focus evaluation value from each AF pixel, for example.

[Example 5]
Next, the case where the ratio with respect to each feature component is adjusted based on information relating to white balance will be described. In the imaging apparatus 10, the white balance at the time of shooting is set by the operation of the operation member 17 by the user. Examples of the white balance setting include a light source type setting and a color temperature setting. As the setting of the light source type, for example, any one of auto, light bulb, fluorescent lamp, clear sky, flash, cloudy sky, and clear sky shade is set. On the other hand, as the setting of the color temperature, the user may input by operating the operation member 17 within a range of 2500 to 10000K.

  The ratio adjustment unit 32 determines whether it is outdoor or sunny from the information of the light source type setting and the color temperature, and when it is determined that it is outdoor or sunny, the ratio to the color component and the luminance contrast component is determined. Increase and decrease ratio to directional edge component. In addition, the ratio adjustment unit 32 reduces the ratio to the color component and the luminance contrast component when it is indoors or outdoors, but it is cloudy based on the information on the light source type setting and the color temperature. Increase the ratio to the edge component.

  In [Example 5], in the setting of the light source type and the color temperature, as the brightness becomes brighter, the entire image to be obtained becomes brighter and the contrast becomes clearer. Therefore, the ratio to the color component and the brightness contrast component is increased, The ratio with respect to each feature component is adjusted so as to reduce the ratio with respect to the directional edge component.

  When adjusting the ratio for each feature component based on the information related to white balance, for example, whether the brightness at the time of shooting is equal to or higher than a predetermined brightness, and whether there is a blue area or not, whether it is outdoor or clear sky It is also possible to determine whether or not.

[Example 6]
Next, a case where the ratio for each feature component is adjusted based on the image contrast information will be described. The CPU 20 acquires information on the luminance contrast YC in the through image. The ratio adjusting unit 32 adjusts the ratio for each feature component based on the relationship between the maximum value YC _{max of the} luminance contrast and the threshold value. [Table 7] shows the relationship between the luminance contrast maximum value YC _max and the threshold value, and the ratio to each feature component.

  Here, C and D are threshold values, and C <D.

For example, when the maximum value YC _{max of the} luminance contrast is equal to or greater than the threshold value D (when D ≦ YC _max ), the ratio adjusting unit 32 does not adjust the ratio for each feature component. In this case, the ratio with respect to each feature component remains the initial ratio (0.33).

When the maximum value YC _max of the luminance contrast is less than the threshold value D and equal to or greater than the threshold value C (when C ≦ YC _max <D), the ratio adjusting unit 32 decreases the ratio to the luminance contrast component (0.33 → 0.2) Increase the ratio to the color component and the directional edge component (0.33 → 0.4).

Furthermore, when the maximum value YC _max of the luminance contrast is less than the threshold C (when YC _max <C), the ratio adjusting unit 32 sets the ratio to the luminance contrast component to 0, and sets the ratio to the color component and the directional edge component. Raise (0.33 → 0.5).

  In other words, the lower the maximum value of the luminance contrast, the smaller the luminance contrast on the screen. Therefore, the ratio to the luminance contrast component is lowered, and the ratio of the color component and the directional edge is increased.

  In addition to this, it is also possible to adjust the ratio to the feature component based on the distance information obtained when the image is acquired. In this case, the difference between the maximum value and the minimum value of the distance information in the image is obtained, and the smaller the difference, the lower the ratio to the directional edge component and the higher the ratio to the color component and the luminance contrast component.

  In this way, the contribution rate set for each feature component is appropriately adjusted according to the acquisition situation when the image is acquired.

  In [Example 1] to [Example 6], an example is given in which the ratio for each feature component is adjusted based on any one piece of information acquired by the ratio adjustment unit 32. The ratio to each feature component can be adjusted based on a plurality of pieces of information.

[Example 7]
[Example 7] shows an example in which the contribution rate is adjusted by combining a plurality of elements of the photographing conditions. In [Example 7], an example will be described in which the contribution ratio is adjusted by combining the focal length information shown in [Example 1] and the information related to white balance shown in [Example 5] as shooting conditions.

  For example, when it is estimated from the information related to white balance that the shooting environment is outdoors and clear sky, the ratio adjustment unit 32 is based on the information on the focal length f of the imaging optical system 15 [Table 8 below]. ] To adjust the ratio for each of the color component, the luminance contrast, and the directional edge.

  In addition, when the shooting environment is estimated to be indoors or cloudy from information related to white balance, the ratio adjustment unit 32 is based on the information on the focal length f of the imaging optical system 15 according to the following [Table 9], the ratio for each of the color component, luminance contrast, and directional edge is adjusted.

  In this way, by combining a plurality of shooting conditions, it is possible to appropriately adjust the ratio relating to the color component, the luminance contrast, and the directional edge according to the shooting environment.

  Step S104 is processing to generate a saliency map. The data processing circuit 23 generates a color information map, a luminance information map, and an edge information map using the through image data. The data processing circuit 23 generates a saliency map using the generated map and the contribution rate for each feature component set in step S103. The generated saliency map is stored in a memory (not shown).

  Step S105 is processing for setting an AF area based on the saliency map. The CPU 20 reads the saliency map stored in the memory. The CPU 20 extracts a region having the highest saliency from the saliency in the read saliency map. Then, the CPU 20 sets the extracted area as the AF area. Information regarding the set AF area is output to the display output circuit 26. The display output circuit 26 displays the through image and a frame indicating the AF area on the liquid crystal monitor 24 in a superimposed manner.

  Step S106 is a process for determining whether or not the release button is half-pressed. Although not described in detail, the release button is provided with a switch that is turned on when the half-pressing operation and the full-pressing operation are performed. When these switches are turned on, signals to be turned on (hereinafter, operation signals) are input to the CPU 20 respectively. The CPU 20 determines whether or not the release button has been half-pressed depending on whether or not an operation signal at the time of half-pressing the release button has been input. For example, when the operation signal at the time of half-pressing the release button is input to the CPU 20, the CPU 20 determines Yes in step S106. In this case, the process proceeds to step S107. On the other hand, when the operation signal at the time of half-pressing the release button is not input to the CPU 20, the CPU 20 determines No in step S106. In this case, the process returns to step S101, and the processing from step S101 to step S105 is executed again.

  Step S107 is an AF process. An AF area is set in step S105. The CPU 20 moves the focus lens of the imaging optical system 15 in the optical axis direction at a predetermined pitch via the lens driving circuit 18 to obtain a focus evaluation value in the set AF area, and focus based on this focus evaluation value. The lens is moved to execute AF processing.

  Step S108 is a process for determining whether or not the release button is fully pressed. As described above, the release button is provided with a switch that is turned on when the half-pressing operation and the full-pressing operation are performed. The CPU 20 determines whether or not the release button has been fully pressed depending on whether or not an operation signal at the time of fully pressing the release button has been input. For example, when an operation signal when the release button is fully pressed is input to the CPU 20, the CPU 20 determines Yes in step S108. In this case, the process proceeds to step S109. On the other hand, if the operation signal when the release button is fully pressed is not input to the CPU 20, the CPU 20 determines No in step S108. In this case, the process proceeds to step S113.

  Step S109 is an imaging process. The CPU 20 performs AF processing similarly to the AF processing in step S107. In this case, the pitch for moving the focus lens in the optical axis direction is finer than that in step S107.

  After this processing, the CPU 20 outputs the set aperture value to the lens driving circuit. In response to this, the lens driving circuit 18 controls the aperture (not shown) so that the input aperture value is obtained. Further, the CPU 20 outputs a drive signal based on the shutter speed set by the AE process to the image sensor drive circuit 21. In response to this, the image sensor 16 is driven and controlled. Thereby, an imaging process is performed. An image signal obtained by this imaging process is output to the signal processing circuit 22. The signal processing circuit 22 performs signal processing such as direct current reproduction processing, A / D conversion processing, and γ conversion processing on the input image signal. Thereby, image data (main image data) obtained by imaging is generated.

  Step S110 is processing to display an image. The image data generated in step S109 is input to the data processing circuit 23. The data processing circuit 23 subjects the input image data to a resolution (number of pixels) conversion process for displaying on the liquid crystal monitor 24 as necessary, and outputs it to the display output circuit 26. The display output circuit 26 performs predetermined signal processing on the image data input from the data processing circuit 23 and outputs the processed image data to the liquid crystal monitor 24.

  Step S111 is processing to determine whether or not to store image data. The CPU 20 instructs the display output circuit 26 to perform display for allowing the user to select whether or not to store the displayed image data. In response to this, the display output circuit 26 superimposes a comment such as “Do you store image data?” On the displayed image. Based on the operation signal from the operation member 17 executed in response to this display, the CPU 20 determines whether or not to store image data. If the operation signal from the operation member 17 is a signal indicating that image data is stored, the CPU 20 determines Yes in step S111 and proceeds to step S112. On the other hand, if the operation signal from the operation member 17 is a signal indicating that image data is not stored, the CPU 20 determines No in step S111. In this case, the process of the flowchart of FIG. 3 ends.

  Step S112 is processing for storing image data. The data processing circuit 23 outputs the image data to the compression / decompression processing circuit 25. In response to this, the compression / decompression processing circuit 25 performs compression processing on the input image data and stores it in the storage medium 27. Here, the image data is stored in the storage medium 27 in a state in which the saliency map generated by the process in either step S104 or step S117 is attached as auxiliary information.

  If the release button is not fully pressed in step S108 described above, the determination in step S108 is No and the process proceeds to step S113.

  Step S113 is a process for determining whether or not the operation mode of the automatic focusing operation is the continuous AF mode. If the operation mode of the automatic focusing operation is the continuous AF mode, the CPU 20 determines Yes in step S113, and proceeds to step S114.

  On the other hand, if the operation mode of the automatic focusing operation is the single AF mode, the CPU 20 sets the determination process in step S113 to No. In this case, the process returns to step S108. When the operation mode of the automatic focusing operation is the single AF mode, the position of the focus lens set by the AF process executed in step S107 is fixed.

  Step S114 is processing to acquire a through image. The process of step S114 is the same process as step S101.

  Step S115 is a process for acquiring information. The process of step S115 is the same process as step S102.

  Step S116 is processing for setting a contribution rate. The process in step S116 is the same as that in step S103.

  Step S117 is processing for generating a saliency map. The process of step S117 is the same process as step S104.

  Step S118 is processing for setting an AF area based on the saliency map. The process of step S118 is the same process as step S105.

  Step S119 is an AF process. The process in step S119 is the same as that in step S107.

  Here, when the continuous AF mode is set as the operation mode of the automatic focusing operation and the release button is continuously pressed halfway, the processing from step S114 to step S119 is repeatedly executed. That is, each time a through image is acquired, a saliency map is generated, and AF processing based on the saliency map is executed.

  Thus, based on the information obtained by the information acquisition part 31, the ratio with respect to each feature component is adjusted, and the contribution rate with respect to each feature map is set. Thereby, it is possible to set a contribution rate according to the image acquisition environment at the time of shooting, and it is possible to generate an appropriate saliency map.

  In addition, since the AF area is set from the generated saliency map, it is possible to appropriately perform the automatic focusing operation.

  In the flowchart of FIG. 3, when adjusting the ratio to the feature component, the operation mode of the automatic focusing operation is not taken into account, but for example, in the setting of the contribution rate in step S103, the operation mode of the automatic focusing operation If the single AF mode is set, the result of determining the in-focus state of the imaging optical system can be taken into account. In this case, when the imaging optical system is in focus, the ratio to each feature component is adjusted so that the ratio to the directional edge component is increased and the ratio to the color component and the luminance contrast component is decreased. Further, when the AF operation is in progress, the ratio to each feature component may be adjusted so that the ratio to the directional edge component is lowered and the ratio to the remaining feature components is increased. Further, during the execution of the AE operation, the ratio to each feature component may be adjusted so that the ratio to the luminance contrast component is decreased and the ratio to the remaining feature components is increased. As described above, the captured image obtained during various adjustments of the imaging apparatus may be configured to adjust the ratio of the characteristic components related to the adjustment to be reduced, for example. Therefore, it is possible to determine an appropriate saliency according to the shooting situation.

  In the process of the flowchart of FIG. 3, the imaging process is executed when it is determined that the release button has been fully pressed (when Step S <b> 108 is Yes), but the present invention is not limited to this. When it is determined that the release button has been fully pressed, a saliency map is generated, and after performing AF processing based on the generated saliency map, imaging processing can be executed. In this case, in setting the contribution ratio of each feature component used when generating the saliency map based on the through image, the contribution ratio of the directional edge component is lowered and the contribution ratio of the color component and the luminance contrast component is increased. Then, when setting the contribution ratio of each feature component used when generating the saliency map due to the release button being fully pressed, the contribution ratio of the directional edge component is increased and the contribution ratio of the color component and the luminance contrast component is increased. Lower. That is, the saliency map generated based on the through image is used to set an area (AF area) to be used when the automatic focusing operation is performed, and the saliency map resulting from the full pressing operation of the release button. Can be used to determine whether or not the saliency in the AF area is higher than the saliency with respect to other areas, and to determine whether or not an image preferable for the user can be obtained.

  In the present embodiment, the case of performing the automatic focusing operation using the saliency map is described. However, the present invention is not limited to this, and a region with high saliency is generated from the generated saliency map. Each lens of the imaging optical system may be driven and controlled to change the shooting magnification so that the estimated main subject area has an appropriate composition.

  In addition to this, a rectangular region including a region with high saliency is extracted from the saliency map as a region that is a candidate for trimming (crop) processing, and the extracted region is superimposed on the through image. Is also possible.

  In this embodiment, the saliency map is generated based on the through image. However, the present invention is not limited to this, and the saliency map can be generated based on the image obtained at the time of shooting. When a saliency map is generated based on an image obtained at the time of shooting, it can be used, for example, for setting a candidate area for trimming (crop) processing. In the case of an image obtained by continuous shooting, an optimum image can be selected from the plurality of acquired images using a saliency map obtained from the plurality of images.

  In the present embodiment, the imaging apparatus is used. Specifically, a digital camera, a portable terminal represented by a portable phone having a camera function, and the like can be given. In addition to the image pickup apparatus, the present invention can be applied to an image processing apparatus having the functions of the respective units of the data processing circuit 23 shown in FIGS. Furthermore, in addition to the imaging device and the image processing device, the invention can be applied to an imaging system including an imaging device such as a digital camera and a PC.

  In addition, an image processing program for executing the function of each unit of the data processing circuit 23 shown in FIGS. 1 and 2 may be used. This image processing program is preferably stored in a computer-readable storage medium such as a memory card, magnetic disk, or optical disk.

  DESCRIPTION OF SYMBOLS 10 ... Imaging device, 20 ... CPU, 22 ... Signal processing circuit, 23 ... Data processing circuit, 24 ... Liquid crystal monitor, 27 ... Storage medium, 31 ... Information acquisition part, 32 ... Ratio adjustment part, 33 ... Map generation part, 41 ... Luminance information extraction part, 42 ... Color information extraction part, 43 ... Edge information extraction part, 44 ... Map composition part

Claims (17)

Map generating means for generating a saliency map indicating saliency in the image based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components;
Information acquisition means for acquiring information indicating the focal length of the imaging optical system at the time of image acquisition ;
Adjusting means for adjusting the contribution rate associated with at least two or more feature components among the plurality of feature components based on the information acquired by the information acquisition unit;
Equipped with a,
One of the plurality of feature components is a directional edge component;
The adjustment unit is an image processing apparatus that adjusts a contribution rate associated with the directional edge component to a value smaller than an initial value when a focal length of the imaging optical system is shortened . Map generating means for generating a saliency map indicating saliency in the image based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components;
Information acquisition means for acquiring information indicating whether or not the image is an image in which the intensity of light is represented by color shading;
Adjusting means for adjusting the contribution rate associated with at least two or more feature components among the plurality of feature components based on the information acquired by the information acquisition unit;
With
One of the plurality of characteristic components is a color component,
The image processing apparatus that adjusts a contribution rate associated with the color component to 0 when the image is an image in which the intensity of light is expressed by color shading . Map generating means for generating a saliency map indicating saliency in the image based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components;
Information acquisition means for acquiring information indicating brightness of the object scene when the image is acquired;
Adjusting means for adjusting the contribution rate associated with at least two or more feature components among the plurality of feature components based on the information acquired by the information acquisition unit;
With
The plurality of characteristic components are a color component, a luminance contrast component, and a directional edge component,
The adjustment unit has a contribution rate associated with at least one of the color component and the luminance contrast component smaller than an initial value when the brightness of the object scene when the image is acquired is dark An image processing device that adjusts values . Map generating means for generating a saliency map indicating saliency in the image based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components;
Information acquisition means for acquiring information of a defocus amount in the image;
Adjusting means for adjusting the contribution rate associated with at least two or more feature components among the plurality of feature components based on the information acquired by the information acquisition unit;
With
One of the plurality of feature components is a directional edge component;
The adjustment unit is an image processing apparatus that adjusts a contribution rate associated with the directional edge component to a value smaller than an initial value as a defocus amount in the image is larger . Map generating means for generating a saliency map indicating saliency in the image based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components;
Information acquisition means for acquiring information of contrast in the image;
Adjusting means for adjusting the contribution rate associated with at least two or more feature components among the plurality of feature components based on the information acquired by the information acquisition unit;
With
The plurality of characteristic components are a color component, a luminance contrast component, and a directional edge component,
The image processing apparatus , wherein the adjustment unit adjusts a contribution rate associated with the color component to a value larger than an initial value as a contrast value obtained from the image is smaller . Map generating means for generating a saliency map indicating saliency in the image based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components;
Information acquisition means for acquiring information as to whether or not the image is an image obtained during a focusing operation;
Adjusting means for adjusting the contribution rate associated with at least two or more feature components among the plurality of feature components based on the information acquired by the information acquisition unit;
With
One of the plurality of feature components is a directional edge component;
The image processing apparatus that adjusts a contribution rate associated with the directional edge component to a value smaller than an initial value when the image is an image obtained during a focusing operation . Map generating means for generating a saliency map indicating saliency in the image based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components;
Information acquisition means for acquiring information related to the focus operation;
Adjusting means for adjusting the contribution rate associated with at least two or more feature components among the plurality of feature components based on the information acquired by the information acquisition unit;
With
One of the plurality of feature components is a directional edge component;
When the focus operation is a single AF operation, the adjustment unit calculates a contribution rate associated with a directional edge component used when generating a saliency map obtained from an image based on an operation of a release button. An image processing apparatus that adjusts a value larger than a contribution rate associated with a directional edge component used when generating a saliency map obtained from an image based on non-operation of a release button . The image processing apparatus according to any one of claims 1 to 7 ,
The contribution rate associated with each feature component has a preset initial value,
The image processing apparatus, wherein the adjusting unit adjusts the contribution rate associated with each feature component with reference to an initial value of the contribution rate associated with each feature component . The image processing apparatus according to claim 8 .
A plurality of initial values of the contribution ratio associated with each feature component are set for each of a plurality of shooting modes,
The image processing apparatus that adjusts a contribution rate associated with each feature component, with the adjustment unit being based on an initial value of the contribution rate associated with a shooting mode . The image processing apparatus according to any one of claims 1 to 9,
An imaging optical system capable of performing a focusing operation;
An image sensor that receives subject light captured by the imaging optical system and acquires an image;
An imaging apparatus comprising: A map generation step of generating a saliency map indicating a saliency in the image based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components;
  An information acquisition step of acquiring information indicating the focal length of the imaging optical system at the time of image acquisition;
  An adjustment step of adjusting the contribution rate associated with at least two or more feature components among the plurality of feature components based on the information acquired by the information acquisition step;
  With
  One of the plurality of feature components is a directional edge component;
  The adjustment process is an image processing program that adjusts a contribution rate associated with the directional edge component to a value smaller than an initial value when a focal length of the imaging optical system is shortened. A map generation step of generating a saliency map indicating a saliency in the image based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components;
  An information acquisition step of acquiring information indicating whether or not the image is an image in which the intensity of light is represented by color shading;
  An adjustment step of adjusting the contribution rate associated with at least two or more feature components among the plurality of feature components based on the information acquired by the information acquisition step;
  With
  One of the plurality of characteristic components is a color component,
  The adjustment process is an image processing program that adjusts a contribution rate associated with the color component to 0 when the image is an image that expresses the intensity of light by color shading. A map generation step of generating a saliency map indicating a saliency in the image based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components;
  An information acquisition step of acquiring information indicating the brightness of the object scene when the image is acquired;
  An adjustment step of adjusting the contribution rate associated with at least two or more feature components among the plurality of feature components based on the information acquired by the information acquisition step;
  With
  The plurality of characteristic components are a color component, a luminance contrast component, and a directional edge component,
  In the adjustment step, when the brightness of the object scene when the image is acquired is dark, a contribution rate associated with at least one of the color component and the luminance contrast component is smaller than an initial value. An image processing program that adjusts values. A map generation step of generating a saliency map indicating a saliency in the image based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components;
An information acquisition step of acquiring information on a defocus amount in the image ;
An adjustment step of adjusting the contribution rate associated with at least two or more feature components among the plurality of feature components based on the information acquired by the information acquisition step;
Equipped with a,
One of the plurality of feature components is a directional edge component;
The adjustment process is an image processing program for adjusting a contribution rate associated with the directional edge component to a value smaller than an initial value as a defocus amount in the image is larger. A map generation step of generating a saliency map indicating a saliency in the image based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components;
  An information acquisition step of acquiring contrast information in the image;
  An adjustment step of adjusting the contribution rate associated with at least two or more feature components among the plurality of feature components based on the information acquired by the information acquisition step;
  With
  The plurality of characteristic components are a color component, a luminance contrast component, and a directional edge component,
  The adjustment process is an image processing program for adjusting a contribution rate associated with the color component to a value larger than an initial value as a contrast value obtained from the image is smaller. A map generation step of generating a saliency map indicating a saliency in the image based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components;
  An information acquisition step of acquiring information as to whether or not the image is an image obtained during a focusing operation;
  An adjustment step of adjusting the contribution rate associated with at least two or more feature components among the plurality of feature components based on the information acquired by the information acquisition step;
  With
  One of the plurality of feature components is a directional edge component;
  The adjustment process is an image processing program for adjusting a contribution rate associated with the directional edge component to a value smaller than an initial value when the image is an image obtained during a focusing operation. A map generation step of generating a saliency map indicating a saliency in the image based on a feature map for each of the plurality of feature components obtained from the image and a contribution rate associated with each of the plurality of feature components;
  An information acquisition process for acquiring information related to the focus operation;
  An adjustment step of adjusting the contribution rate associated with at least two or more feature components among the plurality of feature components based on the information acquired by the information acquisition step;
  With
  One of the plurality of feature components is a directional edge component;
  In the adjustment step, when the focus operation is a single AF operation, a contribution rate associated with a directional edge component used when generating a saliency map obtained from an image based on an operation of a release button is An image processing program for adjusting to a value larger than a contribution rate associated with a directional edge component used when generating a saliency map obtained from an image based on non-operation of a release button.

Images