JP2014022826A - Image processing apparatus, imaging apparatus, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain an image in which a main subject is in a satisfactory state.SOLUTION: An image processing apparatus comprises: area detection means for extracting a first area included in an inputted image; map generation means for generating a saliency map for representing the distribution of saliency based on plural feature elements using the inputted image; and image correction means for, on the basis of the first area extracted by the area detection means and the saliency map generated by the map generation means, correcting the image for at least one of the first area and a second area that is the area other than the first area.

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program.

  Various imaging apparatuses that perform AF processing and AE processing using the results of face detection processing and face recognition processing have been proposed. In an imaging apparatus equipped with these processes, the face detection process and face recognition process take time due to restrictions on the processing capability and processing time of the apparatus itself, and a problem occurs that the shutter timing intended by the user is lost. Sometimes. In order to suppress the occurrence of such a problem, for example, it has been proposed to use a saliency map that maps the result of feature extraction simulating human vision (see Patent Document 1). By using this saliency map, it is possible to extract a prominent region that is easily visible to human eyes in an image as a subject without performing the face detection process or face recognition process described above.

JP 2011-34311 A

  In Patent Document 1, it is assumed that the degree of saliency (saliency) in a main subject and a subject to be photographed shows a high value. However, in consideration of the shooting environment when acquiring the image, the image obtained by shooting may have a lower saliency in the main subject or shooting target, and an image in which the main subject is in good condition is always obtained. It's hard to say.

  An object of the present invention is to provide an image processing apparatus, an imaging apparatus, and an image processing program that can acquire an image in which a main subject is in a good state.

  In order to solve the above-described problem, an image processing apparatus according to the present invention uses a region detection unit that extracts a first region included in an input image and a distribution of saliency based on a plurality of feature elements using the input image. Based on the map generation means for generating the saliency map, the first area extracted by the area detection means, and the saliency map generated by the map generation means. Image correction means for performing image correction on at least one of the second areas excluding the first area.

  Further, the image correction unit performs image correction on the first area when the saliency of the first area is lower than the saliency of the second area, and sets the saliency of the first area to the first area. This is higher than the saliency of the two regions.

  In this case, it is preferable that the image correction unit performs image correction on the first region with respect to the feature element having the lowest saliency in the second region among the plurality of feature elements. In addition to this, it is preferable that the image correction unit performs image correction on a feature element having the lowest saliency in the first region among the plurality of feature elements.

  Further, the image correction means performs image correction on the second area when the saliency of the second area is higher than the saliency of the first area, and sets the saliency of the second area to the first level. This is lower than the saliency of one region.

  In this case, it is preferable that the image correction unit performs image correction on the second region with respect to the feature element having the highest saliency in the second region among the plurality of feature elements.

  The second region is preferably a region within a predetermined range from the first region. The second region is preferably a region having a higher saliency than the saliency of the first region.

The first area is an area of a main subject included in the image,
The area detecting unit extracts the area of the main subject as the first area based on an area used when focus adjustment is performed when the image is acquired.

  In addition, display means for displaying the image corrected by the image correction means is provided.

  The image processing apparatus further includes storage means for storing the saliency map generated by the map generation means and the image corrected by the image correction means in association with each other.

  In addition, a setting unit that sets a priority order for image correction in the image correction unit is set for each of the plurality of feature elements, and the image correction unit is set by the setting unit among the plurality of feature elements. The image correction is executed in order from the feature elements set with a high priority.

  A determination unit configured to determine a scene of the image; and when the determination unit determines that the scene is a group photo of a plurality of persons from the image, The first area is detected, and the image correction unit performs image correction on each of the plurality of person areas so that the saliency of each first area is included in a predetermined range. It is.

  In this case, the area detection unit detects a person area excluding the plurality of persons in addition to the plurality of person areas, and the image correction unit determines that the saliency of the plurality of person areas is the plurality of person areas. It is preferable to perform image correction on the plurality of person areas or the person areas excluding the plurality of persons so as to be higher than the saliency of the person area excluding the person.

  In addition, the area detection unit can detect a plurality of person areas included in the image, and the image correction unit can detect a person area that is a main subject among the plurality of person areas. Image correction is performed on one of the area of the person as the main subject and the area of the other person so that the degree of saliency is higher than the degree of saliency in the area of the other person.

  According to another aspect of the present invention, there is provided an imaging apparatus including the above-described image processing apparatus and an imaging element that acquires the input image.

In addition, the image processing program of the present invention generates a saliency map indicating a distribution of saliency based on a plurality of feature elements by using a region detection step of extracting a first region included in an input image and the input image. A first region extracted from the map generation step, the first region extracted by the region detection step, and the second region excluding the first region based on the saliency map generated by the map generation step An image correction step of performing image correction on at least one of
Can be executed by a computer.

  According to the present invention, it is possible to acquire an image in which the main subject is in a good state.

It is a functional block diagram showing an example of an imaging device using the present invention. It is a functional block diagram which shows the structure of a data processing circuit. It is a functional block diagram which shows the structure of a map production | generation part. It is a flowchart which shows the flow of the process at the time of imaging. (A) is a diagram showing the arrangement state of the AF area on the photographing surface, (B) is a diagram showing the AF area used when imaging a person against the background of the lighthouse, and (C) is a diagram of the face of the person It is a figure which shows the case where the area | region and the area | region of the lighthouse used as object when comparing an saliency degree adjoin. It is a figure which shows the case where the area | region of a person's face and the area | region of the lighthouse used as object when comparing a saliency are in the position away. It is a figure which shows an example of a group photograph.

  Embodiments of the present invention will be described below. FIG. 1 is a functional block diagram illustrating an example of an imaging apparatus according to the present embodiment. The imaging device 10 includes an imaging lens 15, an imaging device 16, an operation member 17, a lens driving circuit 18, a photometric circuit 19, a CPU 20, an imaging device driving circuit 21, a signal processing circuit 22, a data processing circuit 23, a liquid crystal monitor 24, and a compression / expansion. A processing circuit 25, a display output circuit 26, and a storage medium 27 are provided.

  A subject image (not shown) is formed on the image pickup surface of the image pickup device 16 by the image pickup lens 15. The imaging lens 15 is composed of a plurality of lenses, and is configured such that adjustments such as focus and zoom can be performed via the lens driving circuit 18 based on the operation of the operation member 17 or the like.

  The image sensor 16 is an image sensor that can capture a still image as well as continuously capture a still image and a moving image, and is composed of, for example, a CCD image sensor or a CMOS image sensor. The image pickup device 16 is driven by the image pickup device drive circuit 21 under the control of the CPU 20 based on the subject photometry data obtained by the photometry circuit 19. The image signal read by the image sensor 16 is input to the signal processing circuit 22.

  The signal processing circuit 22 performs signal processing such as direct current reproduction processing, A / D conversion processing, white balance processing, and γ conversion processing on the image signal read out by the image pickup device 16 to obtain data processing circuit as image data. To 23.

  The data processing circuit 23 performs a resolution (number of pixels) conversion process for displaying the input image data on the liquid crystal monitor 24 as necessary, and outputs the converted image data to the display output circuit 26. The image data is output to the compression / decompression processing circuit 25. The display output circuit 26 performs predetermined signal processing on the image data input from the data processing circuit 23 and outputs the processed image data to the liquid crystal monitor 24. The display output circuit 26 further performs a process of superimposing overlay image data such as a shooting menu and a cursor on the image data output from the data processing circuit 23 as necessary under the control of the CPU 20. Thus, the overlay image is superimposed on the subject image and displayed on the liquid crystal monitor 24.

  The compression / decompression processing circuit 25 performs compression processing on the input image data and stores it in the storage medium 27. The liquid crystal monitor 24 displays an image corresponding to the image data compressed and stored in the storage medium 27.

  The liquid crystal monitor 24 can also display an image corresponding to the image data stored in the storage medium 27 as a reproduced image. The CPU 20 reads out the image data stored in the storage medium 27, performs a decoding process by the compression / decompression processing circuit 25, and supplies it to the liquid crystal monitor 24 via the data processing circuit 23 and the display output circuit 26.

  The CPU 20 extracts image data of an area (AF area) set on the imaging screen based on an operation of a release button that constitutes a part of the operation member 17, and the contrast value of the area (or the area) A high spatial frequency component amount) is calculated, and a so-called contrast AF operation for adjusting the focus state of the subject image on the imaging surface of the image sensor 16 is performed based on the calculation result.

  In addition, the CPU 20 drives the imaging lens 15 and analyzes sequentially obtained image signals for each subject in the image, and shoots for each subject based on the lens position when the contrast value in the region becomes maximum. Get distance information.

  Although the contrast AF operation is exemplified as the automatic focusing operation, the present invention is not limited to this, and a well-known pupil division type phase difference AF operation can also be used. Also in this case, the shooting distance information of each region can be obtained by the automatic focusing operation.

  The CPU 20 drives the imaging lens 15 via the lens driving circuit 18 based on the operation of the zoom operation member that constitutes a part of the operation member 17, and displays the subject image formed on the imaging surface of the image sensor 16. Perform an optical zoom operation to enlarge or reduce. Further, the CPU 20 converts the image data obtained by the imaging device 16 or the image data stored in the storage medium 27 based on the operation of the zoom operation member constituting a part of the operation member 17 into the resolution by the data processing circuit 23. (Number of pixels) An electric zoom operation for enlarging or reducing by the conversion process is controlled.

  Next, the configuration of the data processing circuit 23 will be described. As illustrated in FIG. 2, the data processing circuit 23 includes a map generation unit 31, a subject extraction unit 32, and an image correction unit 33. As described above, image data that has been subjected to signal processing by the signal processing circuit 22 is input to the data processing circuit 23.

  Based on the input image data, the map generation unit 31 generates a feature amount map in which feature amounts such as luminance information, color information, and edge information are collected for each information. And the map production | generation part 31 produces | generates a saliency map using the feature-value map of each produced | generated information. Note that each piece of information used when generating the saliency map is not limited to the above-described luminance information, color information, and edge information, and may use other information such as subject movement information in the captured image. Details of the map generation unit 31 will be described later.

  The subject extraction unit 32 extracts a main subject region (hereinafter referred to as a main subject region) included in the image using the position information of the AF area used during the AF operation. For example, when imaging, it is common to perform an AF operation so that the main subject is in focus. The subject extraction unit 32 extracts a region of color component values of similar colors (color component values included in a predetermined range) in the AF area used in the AF operation and in the vicinity of the AF area as a main subject region. . In addition to extracting a region having color component values of similar colors as the main subject region, it is also possible to extract the main subject region using a technique such as a level set method or a snake method. Information relating to the main subject area extracted by the subject extraction unit 32 is output to the image correction unit 33. The subject extraction unit 32 may be configured to extract a main subject region by subject recognition processing using image data or the like.

  The image correction unit 33 executes image correction processing on the input image data based on the saliency map generated by the map generation unit 31 and the main subject region extracted by the subject extraction unit 32. Here, the image correction processing executed on the image data includes correction processing for information used when generating the saliency map, that is, correction processing for the luminance component, correction processing for the color component, or directional edge. There is at least one correction process of the correction process for the component.

  Examples of the correction processing for the luminance component include increasing the luminance of each pixel included in the main subject region in the image. Specifically, the image data is amplified so that the luminance of each pixel included in the main subject region is increased. Alternatively, the contrast in the main subject region is enhanced by performing high-frequency emphasis processing on the pixel values of the pixels included in the main subject region. Note that, instead of performing high-frequency emphasis processing on the pixel values of the pixels included in the main subject area, it is also possible to perform gradation conversion processing by changing the γ characteristics for the pixel values of the pixels included in the main subject area. is there. Furthermore, it is possible to use both the high-frequency emphasis processing for the pixel values of the pixels included in the main subject area and the gradation conversion processing for changing the γ characteristics for the pixel values of the pixels included in the main subject area. By performing such processing, the luminance difference value between the original image and a low-resolution image obtained by performing Gaussian filter processing on the image as described later increases. As a result, the image data is corrected so that the saliency of the luminance component in the main subject region in the image is higher than the saliency of the luminance component in other regions.

  As the correction processing for the color component, the difference between the red (R) color component value and the green (G) color component value, or the blue (B) color component value for each pixel included in the main subject region in the image. And increasing at least one of the differences between the yellow (Y) color component values. That is, the saturation of a specific color component is increased and the saturation of other color components is decreased. Thereby, the image data is corrected so that the saliency of the color component in the main subject area in the image is higher than the saliency of the color component in the other areas.

  Further, the correction processing for the directional edge component includes increasing the directional edge component included in the main subject region in the image. Specifically, high-pass filter processing is performed on pixel values of pixels included in the main subject area. As described above, the edge information extraction unit 43 generates edge image data in which the edge intensity with respect to directions of, for example, 0 degrees, 45 degrees, 90 degrees, and 135 degrees is a pixel value. In the high-pass filter process, for example, the high-pass filter process is executed in at least one of 0 degree, 45 degrees, 90 degrees, and 135 degrees. As a result, the image data is corrected so that the saliency of the directional edge component in the main subject region in the image is higher than the saliency of the directional edge component in other regions.

  Next, the configuration of the map generation unit 31 will be described. As illustrated in FIG. 3, the map generation unit 31 includes a luminance information extraction unit 41, a color information extraction unit 42, an edge information extraction unit 43, and a map synthesis unit 44.

  As described above, the image data that has been subjected to signal processing by the signal processing circuit 22 is input to the map generation unit 31. Note that image data is input to each of the luminance information extraction unit 41, the color information extraction unit 42, and the edge information extraction unit 43.

  The luminance information extraction unit 41 generates luminance image data in which the pixel value of each pixel is a luminance value based on the input image data. The luminance information extraction unit 41 generates a plurality of luminance image data with different resolutions using the generated luminance image data. The luminance information extraction unit 41 generates luminance image data of eight resolutions from level 1 to level 8, for example, by sequentially performing Gaussian filter processing on the luminance image data. Note that the luminance image data at level 1 is set to have the same resolution as the input image data. The resolution is set lower in the order of level 2 luminance image data, level 3 luminance image data,..., Level 8 luminance image data.

  After generating the luminance image data having different resolutions, the luminance information extraction unit 41 selects two adjacent luminance images such as level 1 luminance image data and level 2 luminance image data from among the generated plurality of luminance image data. A level of luminance image data is selected, and a difference between the two selected luminance image data (hereinafter, difference data) is obtained. When the number of pixels of each luminance image data is different, the luminance information extraction unit 41 has the number of pixels of luminance image data with low resolution out of the two luminance image data. The number of pixels of the low-resolution luminance image data is converted so as to obtain the difference, and difference data is obtained.

  The luminance information extraction unit 41 generates a predetermined number of difference data, and then normalizes the generated difference data. A luminance information map is generated based on the normalized difference data. The generated luminance information map is output to the map composition unit 44.

  The color information extraction unit 42 generates RG difference image data including a difference between a red (R) color component value and a green (G) color component value using the input image data. And the color information extraction part 42 produces | generates several RG difference image data from which resolution differs using the produced | generated RG difference image data. Thereby, RG difference image data of eight resolutions from level 1 to level 8, for example, is generated in the same manner as the luminance image data.

  The color information extraction unit 42 selects two adjacent levels of RG difference image data, such as level 1 RG difference image data and level 2 RG difference image data, from among the plurality of generated RG difference image data. The difference between the two selected RG difference image data (hereinafter referred to as difference data) is obtained. Also in this case, when the number of pixels of the plurality of RG difference image data is different from each other, the color information extraction unit 42 determines that the number of pixels of the RG difference image data with low resolution among the two RG difference image data is the resolution. The RG difference image data field having a low resolution converts the prime number so that the number of pixels of the RG difference image data having a higher value is obtained. The color information extraction unit 42 normalizes the difference data obtained from the RG difference image data, and then generates a color information map based on the difference between the red (R) color component value and the green (G) color component value.

  In addition, the color information extraction unit 42 uses the input image data, and the difference between the blue (B) color component value and the yellow (Y) color component value by the same method as the generation of the RG difference image data described above. BY difference image data consisting of Then, the color information extraction unit 42 performs the same processing as that of the RG difference image data, and generates a color information map based on the difference between the blue (B) color component value and the yellow (Y) color component value. Each color information map generated by the color information extraction unit 42 is output to the map synthesis unit 44.

  The edge information extraction unit 43 performs a filtering process using a Gabor filter on the input image data. By this filtering process, for example, edge image data having edge strengths in the directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees on the screen as pixel values is generated. The edge information extraction unit 43 further generates a plurality of edge image data having different resolutions using the generated edge image data. Thus, edge image data of eight resolutions from level 1 to level 8 are generated by sequentially performing Gaussian filter processing. The edge information extraction unit 43 selects two adjacent edge image data such as level 1 edge image data and level 2 edge image data from among the generated plurality of edge image data, and selects the selected 2 A difference between the two edge image data (hereinafter, difference data) is obtained. When the number of pixels of the plurality of edge image data is different from each other, the edge information extraction unit 43 determines that the number of pixels of the edge image data with low resolution is the number of pixels of the edge image data with high resolution among the two edge image data. The number of pixels of the low resolution edge image data is converted so that The edge information extraction unit 43 normalizes the difference data obtained from the edge image data, and then generates an edge information map based on the normalized difference data. Note that the edge information extraction unit 43 performs the above-described processing for each edge image data in each direction. These generated edge information maps are output to the map composition unit 44.

  The map combining unit 44 generates a saliency map by combining the input luminance information map, color information map, and edge information map. This saliency map is a map showing the distribution of saliency at each pixel in the image.

  Next, the flow of image correction processing will be described based on the flowchart of FIG. Hereinafter, the main subject area will be described as a main subject area.

  Step S101 is processing for acquiring a through image. When the shooting mode is started, the CPU 20 controls the image sensor 16 via the image sensor drive circuit 21. Thereby, the image sensor 16 is driven with a predetermined time interval. Thereby, the image signal which becomes the origin of a through image is acquired. This image signal is output to the signal processing circuit 22. The signal processing circuit 22 executes various signal processes on the input image signal. Thereby, the through image data is generated.

  Step S102 is processing for executing AWB processing and AE processing. The CPU 20 performs known exposure calculation and white balance adjustment processing based on the subject photometry data obtained by the photometry circuit 19.

  Step S103 is an AF process. The CPU 20 moves the focusing lens constituting the imaging lens 15 via the lens driving circuit 18. A position that is the maximum value of the focus evaluation value obtained when the focusing lens is moved is obtained. This focus evaluation value is calculated using any of the AF areas provided in the shooting screen. The AF area used when calculating the focus evaluation value may be selected by operating the operation member 17 in advance by the user, or the AF area corresponding to the subject area obtained by a method such as subject recognition may be selected. Good.

  Step S104 is processing to generate a saliency map. The through image data that has been subjected to various types of signal processing by the signal processing circuit 22 is input to the data processing circuit 23. The data processing circuit 23 generates a saliency map using the input through image data.

  Step S105 is processing to extract a subject area. The data processing circuit 23 extracts a subject area by using the input through image data and the AF area information used during the AF operation.

  Step S106 is processing for determining whether or not to perform image correction. The CPU 20 refers to the saliency map generated in step S104 and the position information of the subject area extracted in step S105, and determines whether to perform image correction on the through image data.

  Hereinafter, whether or not to perform image correction on through image data is determined by comparing the saliency in the main subject region in the image with the saliency in other regions in the image. To do. As the saliency in the main subject area and the saliency in other areas in the image, for example, an average value of saliency of each pixel included in each area may be obtained.

  Here, examples of the other areas in the image include the entire area (background area) excluding the main subject or an area having a high degree of saliency among the areas excluding the main subject area. Here, the region having a high degree of saliency among the regions excluding the main subject region may be a region located within a predetermined range with respect to the subject region, may be a region adjacent to the main subject region, or may be a main subject region. It may be an area at a position away from.

  For example, when the saliency in the main subject area is lower than the saliency in other areas, the CPU 20 determines that image correction is necessary. That is, the CPU 20 sets the determination process in step S106 to Yes. In this case, the process proceeds to step S107.

  On the other hand, if the saliency in the main subject area is higher than the saliency in the other areas, the CPU 20 determines that it is not necessary to perform image correction. That is, the CPU 20 sets the determination process in step S106 to No. In this case, the process proceeds to step S108.

  Step S107 is processing for executing image correction. The data processing circuit 23 performs image correction on the through image data based on the saliency map generated in step S104 and the main subject region extracted in step S105.

  Step S108 is processing to display a through image. Image data that does not require image correction or image data that has undergone image correction processing is output to the display output circuit 26. In response to this, the display output circuit 26 displays an image based on the input image data on the liquid crystal monitor 24. As a result, a through image is displayed on the liquid crystal monitor 24.

  Step S109 is processing to determine whether or not there is a half-press operation of the release button. Although not described in detail, the release button is provided with a switch that is turned on when the half-pressing operation and the full-pressing operation are performed. When these switches are turned on, signals to be turned on (hereinafter, operation signals) are input to the CPU 20 respectively. The CPU 20 determines whether or not the release button has been half-pressed depending on whether or not an operation signal at the time of half-pressing the release button has been input. For example, when an operation signal for half-pressing the release button is input to the CPU 20, the CPU 20 determines Yes in step S109. In this case, the process proceeds to step S110. On the other hand, when the operation signal at the time of half-pressing the release button is not input to the CPU 20, the CPU 20 determines No in step S109. In this case, the process proceeds to step S101, and the processing from step S101 to step S108 is executed again.

  Step S110 is a process for determining whether or not the release button is fully pressed. As described above, the release button is provided with a switch that is turned on when the half-pressing operation and the full-pressing operation are performed. The CPU 20 determines whether or not the release button has been fully pressed depending on whether or not an operation signal at the time of fully pressing the release button has been input. For example, when the operation signal at the time of fully pressing the release button is input to the CPU 20, the CPU 20 determines Yes in step S110. In this case, the process proceeds to step S112. On the other hand, when the operation signal when the release button is fully pressed is not input to the CPU 20, the CPU 20 determines No in step S110. In this case, the process proceeds to step S111.

  Step S111 is processing to determine whether or not the operation of the release button has been released. The CPU 20 determines whether or not the operation of the release button is released depending on whether or not an operation signal at the time of half-pressing the release button is input. For example, if the operation signal at the time of half-pressing the release button is not input to the CPU 20, the CPU 20 determines Yes in step S111. In this case, the process of the flowchart in FIG. 4 ends. On the other hand, when the operation signal at the time of half-pressing the release button is input to the CPU 20, the CPU 20 determines No in step S111. In this case, the process returns to step S110.

  Step S112 is an imaging process of the main shooting following the shooting of the through image. The CPU 20 drives the image sensor 16 via the image sensor drive circuit 21 using the imaging conditions obtained by the exposure calculation in step S108. Thereby, an imaging process is performed. An image signal obtained by this imaging process is output to the signal processing circuit 22. The signal processing circuit 22 performs signal processing such as direct current reproduction processing, A / D conversion processing, and γ conversion processing on the input image signal. Thereby, image data (main captured image data) obtained by imaging is generated.

  Step S113 is processing for generating a saliency map. The data processing circuit 23 generates a saliency map using the actual captured image data output from the signal processing circuit 22.

  Step S114 is processing to extract a subject area. The data processing circuit 23 extracts a subject area using the actual captured image data output from the signal processing circuit 22. The process of step S112 is the same process as step S103.

  Step S115 is processing for determining whether or not to perform image correction. The CPU 20 refers to the saliency map generated in step S113 and the main subject area extracted in step S114, and determines whether or not to perform image correction on the through image data. The determination process in step S115 includes the same process as the determination process in step S104. For example, when it is necessary to perform image correction, the CPU 20 determines Yes in step S115. In this case, the process proceeds to step S116. On the other hand, when it is not necessary to perform image correction, the CPU 20 determines No in step S115. In this case, the process proceeds to step S117. In step S104, it is determined whether or not it is necessary to perform image correction on the through image data. Therefore, the determination process in step S115 uses the result of the determination process in step S104. Also good.

  Step S116 is an image correction process. The process of step S113 is the same process as step S104. By executing the process of step S116, an image correction process for the actual captured image data is executed.

  Step S117 is a process of displaying the actual captured image with the image corrected. The captured image data that has been subjected to the image correction processing in step S116 is output to the display output circuit 26. The display output circuit 26 displays an actual captured image based on the input actual captured image data on the liquid crystal monitor 24.

  Step S118 is processing for determining whether or not to store the main image data that has undergone image correction. The user confirms the actual captured image displayed on the liquid crystal monitor 24 and operates the operation member 17 to select whether to store the actual captured image data to be displayed. As described above, the operation signal based on the operation of the operation member 17 is input to the CPU 20. If the operation signal based on the operation of the operation member 17 is a signal indicating that the actual captured image data displayed on the liquid crystal monitor 24 is stored, the CPU 20 determines Yes in step S118. In this case, the process proceeds to step S119. On the other hand, if the operation signal based on the operation of the operation member 17 is a signal indicating that the actual captured image data displayed on the liquid crystal monitor 24 is not stored, the CPU 20 determines No in step S118. In this case, the process of the flowchart in FIG. 4 ends.

  Step S119 is processing for storing an image. The data processing circuit 23 outputs the captured image data to the compression / decompression processing circuit 25. In response to this, the compression / decompression processing circuit 25 performs a compression process on the inputted main photographed image data and stores it in the storage medium 27. Here, the actual captured image data is stored in the storage medium 27 in a state where the saliency map generated in step S113 is attached as additional information. For the image data that has been subjected to the image correction processing in step S115, the saliency map generated in step S113 and the information on the image correction processing are appended to the actual captured image data as supplementary information. Or after generating the saliency map again based on the image data that has been subjected to the image correction processing, the newly generated saliency map is attached to the captured image data as additional information and stored. It is also possible.

  Hereinafter, a case where a plurality of AF areas are provided on the imaging surface will be considered. FIG. 5A shows a case where a total of 15 AF areas 51 to 65, that is 5 horizontal and 3 vertical, are set on the imaging screen 50. As shown in FIG. 5B, for example, when a person is imaged with a lighthouse in the background, the AF operation is performed using the AF area 58 among the AF areas 51 to 65 at the time of imaging. The subject area extraction process is executed on the AF area 58 and the area in the vicinity of the AF area 58 in the image I obtained by imaging. By the process of extracting the subject area, for example, a rectangular area 68 indicated by a two-dot chain line is extracted as the main subject area (see FIG. 5C).

  Thereafter, using the generated saliency map, a region having high saliency is detected from other regions excluding the extracted main subject region. In FIG. 5, since a person is imaged with the lighthouse in the background, a lighthouse area 69 is determined as a highly prominent subject area excluding the main subject area. FIG. 5C shows a case where a lighthouse region 69 exists adjacent to the human face region 68.

  In such a case, the data processing circuit 23 obtains the saliency in the human face area 68 and the saliency in the lighthouse area 69 based on the saliency map.

For example, in the case where the luminance component saliency, the color component saliency, and the directional edge component saliency in the human face area 68 are 1.0, 0.3, 0.3, and 0.4. explain. Further, a case will be described in which the luminance component saliency, the color component saliency, and the directional edge component saliency in the lighthouse region 69 are 1.1, which is 0.5, 0.5, and 0.1. . Here, when the total value (1.0) of the saliency of the person's face area 68 is compared with the total value (1.1) of the saliency of the lighthouse area 69, the saliency of the person's face area 68 is compared. The total value of the degrees is lower than the total value of the saliency of the lighthouse area 69.
In such a case, image correction processing is performed on the person's face area 68 so that the total value of the saliency in the person's face area 68 is higher than the total value of the saliency in the lighthouse area 69. . It should be noted that in the image correction processing for the human face area 68, each component is changed so that the saliency values of the luminance component, the saliency of the color component, and the saliency of the directional edge component are changed. Image correction processing for (feature elements) may be executed, or image correction processing may be executed so that one of the above-described saliency values is increased.

  For example, when executing the image correction process for changing the saliency value of any one of the saliency of the luminance component, the saliency of the color component, and the saliency of the directional edge component, the following first to third processes Any one of the processes can be considered.

  As the first processing, the saliency value that is the lowest value among the saliency of the luminance component, the saliency of the color component, or the saliency of the directional edge component in the human face area 68 is the original value. The image correction processing is executed on the human face area 68 so that the height is higher than the upper limit. In the image I shown in FIG. 5C, the saliency of the luminance component, the saliency of the color component, and the saliency of the directional edge component in the human face area 68 are 0.3, 0.3, 0.4. It becomes. In such a case, the image correction process is executed on the human face area 68 so that one of the saliency of the luminance component and the saliency of the color component is higher than the original value.

  It should be noted that the image correction processing is performed on the human face area 68 so that one of the saliency of the luminance component and the saliency of the color component is higher than the original value. The image correction processing may be executed on the human face area 68 so that both the saliency and the saliency of the color component are higher than the original value.

  As the second processing, the saliency value that is the highest value among the saliency of the luminance component, the saliency of the color component, and the saliency of the directional edge component in the human face area 68 is higher than the original value. The image correction process is executed on the human face area 68 so that the height of the face is also higher. That is, in the image I shown in FIG. 5C, the saliency value of the directional edge component is the highest saliency value. In such a case, the image correction process is executed on the human face area 68 so that the saliency of the directional edge component is higher than the original value. As described above, examples of the directional edge component include edge components in four directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees. For this reason, when the saliency of the directional edge component is changed by the image correction process, it is only necessary to change the saliency of the edge component in at least one of the four directional components.

  Further, as the third processing, image correction for the component having the lowest saliency among the saliency of the luminance component, the saliency of the color component, and the saliency of the directional edge component in the lighthouse region 69 is performed. It is applied to the area 68 of the person's face. As described above, in the image I in FIG. 5C, the luminance component saliency, the color component saliency, and the directional edge component saliency in the lighthouse region 69 are 0.5, 0.5, 0.1. In such a case, image correction for the directional edge component having the lowest saliency is performed on the human face area 68.

  By performing any of the above-described image correction processes on such a main subject region, the saliency of the main subject region becomes higher than the saliency of other regions. Note that the saliency is an index that indicates how easily human eyes can see, and thus it is possible to direct the user's point of interest to the subject area by performing such image correction. In other words, an image in which the main subject is in a good state can be obtained.

  In the present embodiment, a process of generating a saliency map, a process of extracting a subject area, and a process of correcting an image are executed for each of the through image data and the actual captured image data. However, these processes do not have to be performed on both the through image data and the actual captured image data, and can be performed on one image data of the through image data or the actual captured image data.

  When the above-described processing is performed on the through image data, information on the image correction processing performed on the through image data is stored, and the image data obtained at the time of actual imaging is stored. It is also possible to perform data processing in consideration of image correction processing information. Further, when the above-described processing is performed on the through image data, it is not necessary to perform these processing on each of the through image data, and the through image obtained immediately before the half-press operation of the release button is performed. It is good also as a structure performed only with respect to data. Furthermore, it is good also as a structure which performs the process performed with respect to the through image mentioned above to the moving image acquired by this imaging | photography.

  In the present embodiment, the actual captured image data subjected to the image correction process is stored. However, the present captured image data is not limited to this, and the captured image data before the image correction process is performed; It is also possible to store information related to the processing content of the image correction process in association with each other. Further, signal processing (saliency map) performed in the signal processing circuit 22 and the data processing circuit 23 on the actual captured image data (so-called RAW data) that has not been subjected to partial signal processing in the signal processing circuit 22. It is also possible to store information relating to the process of generating the image, the process of extracting the subject area, and the process of correcting the image as additional information.

  In this embodiment, the total value (integrated value) of each saliency in the main subject area is compared with the total value of each saliency in the other subject areas, but the present invention is not limited to this. It is also possible to compare the average value of each saliency in the subject area with the average value of other subject areas.

  In the present embodiment, when the saliency in the main subject region is lower than the total value of the other regions, the image for the main subject region is set so that the saliency in the main subject region is higher than the saliency in the other regions. Although correction is performed, it is not necessary to be limited to this, and it is also possible to perform image correction on other areas so that the saliency of the other areas is lower than the total saliency of the main subject area. It is. In this case, image correction may be performed so that the saliency of other regions (for example, the saliency of the luminance component, the saliency of the color component, and the saliency of the directional edge component) is lower than the original value, Image correction may be performed so that at least one of the saliency of the luminance component, the saliency of the color component, and the saliency of the directional edge component is lower than the saliency of the main subject region.

  For example, when the saliency of the luminance component is lowered, low-pass filter processing is performed on the pixel values of the pixels included in the other subject areas, thereby reducing the contrast in the other subject areas. In addition, instead of performing low-pass filter processing on the pixel values of pixels included in other subject areas, it is also possible to perform gradation conversion processing that changes the γ characteristics for the pixel values of pixels included in other subject areas. It is. Furthermore, it is also possible to use a low-pass filter process for pixel values of pixels included in other subject areas and a gradation conversion process in which the γ characteristics for the pixel values of pixels included in other subject areas are changed.

  Further, when the saliency of the color component is lowered, the difference between the red (R) color component value and the green (G) color component value, or the blue (B) color component value and the yellow (Y) color component value. Image correction is performed so as to reduce one of the differences.

  Further, when the saliency of the directional edge component is lowered, it can be dealt with by performing low-pass filter processing on other subject areas. In order to reduce the saliency of the directional edge component, the saliency of the edge component in at least one of the four direction edge components of 0 degree, 45 degrees, 90 degrees, and 135 degrees is changed. The image correction may be executed.

  In addition, it is also possible to perform both image correction for increasing the saliency of the main subject area and image correction for decreasing the saliency of the other areas in parallel.

  In this embodiment, the average value for each saliency is obtained for each region, and the saliency comparison is performed using a value obtained by summing the obtained average values for each saliency, but it is necessary to be limited to this. For example, the maximum value of the total value of each saliency can be obtained for each region, and the saliency can be compared using the maximum value of the calculated total value of each saliency.

  In the present embodiment, the saliency comparison is performed using the main subject region and a region having a high saliency among the regions included in the predetermined range from the main subject region. However, the present invention is not limited to this. Alternatively, it may be a subject area with a high degree of saliency among the areas excluding the main subject area. As shown in FIG. 6, when a person is imaged with a lighthouse in the background, an image I ′ in which the person's face area 70 and the lighthouse area 71 are separated from each other is obtained. In such a case, the human face area 70 is extracted as the main subject area, and the lighthouse area 71 is extracted as the area for comparing the saliency. Then, after the saliency is compared using these areas 70 and 71, image correction processing is executed as necessary.

  In this embodiment, when the saliency in the main subject area is lower than the saliency in the other areas, image correction processing is performed on the main subject area, and the saliency in the main subject area It is higher than saliency. However, depending on the degree of image correction processing performed on the actually obtained image data, there is a problem that the sameness as the original image before the image correction processing cannot be maintained. For this reason, you may restrict | limit the correction amount of the saliency corrected by image correction processing. In addition, in order to maintain the sameness as the original image, a priority order in executing image correction processing is set for each of the luminance component, the color component, and the directional edge component.

  For example, consider the case where the saliency of the luminance component, the saliency of the color component, and the saliency of the directional edge component in the main subject region are 0.2, 0.1, and 0.5. In this case, the upper limit value of the saliency correction amount is, for example, twice the original saliency value. In the image correction process, the priority order of the image correction process with respect to the color component is set to the first order, the priority order of the image correction process with respect to the luminance component is the second order, and the priority order of the image correction process with respect to the directional edge component is the third order. Set each to the ranking. In this case, the total value of the saliency of the main subject is 0.8. For example, when the total value of the saliency of the subject areas that are the targets in the comparison of the saliency is 1.0, the image correction process is executed on the main subject area. In this case, since the priority order of the image correction process for the color component is the first order, the image correction process for the color component is executed first. By performing the image correction process for this color component, the saliency of the color component is doubled (0.2). As a result, the saliency of the luminance component, the saliency of the color component, and the saliency of the directional edge component in the main subject region are 0.2, 0.2, and 0.5. That is, the total saliency value of the main subject area is 0.9.

  On the other hand, the total value of the saliency of the other areas is 1.0, which is higher than the total value (0.9) of the saliency of the main subject area, and thus image correction processing is required again. Since the priority order of the image correction process for the luminance component is set to the second order, the image correction process for the luminance component is executed next. As a result, the saliency of the luminance component, the saliency of the color component, and the saliency of the directional edge component in the main subject region are 0.4, 0.2, and 0.5. That is, the total saliency value of the main subject area is 1.1. By performing the image correction process on the luminance component, the total value of the saliency of the main subject area exceeds the total value of the saliency of the other areas, and thus the image correction process on the main subject area ends. If the total value of the saliency of the subject area after performing the image correction process on the luminance component does not exceed the total value of the saliency of other areas, the image correction process on the directional edge component is performed. The

  In this case as well, the image correction process is performed on the main subject region and the total value of the saliency in the main subject region is changed to a high value. However, the present invention is not limited to this. It is also possible to execute image correction processing on subject areas other than the main subject area, and set the total saliency value in the subject area lower than the original total value.

  In the above example, when limit values are individually set as image correction limit values for changing the saliency of the luminance component, color component, and directional edge component, For example, the correction of increasing the saliency of the color component in the first rank is performed up to the limit value of the color component correction, and then the total value of the saliency of each component is calculated. In addition, when the total value of the saliency of the main subject is lower than the saliency of the other area, the same correction may be performed on, for example, the luminance component in the second rank.

  In this embodiment, the scene at the time of imaging is not taken into consideration, but the image correction process can be executed in consideration of the scene at the time of imaging. For example, when acquiring a group photo, a plurality of persons are included in the imaging range (image). In this case, the focus evaluation values of the face areas of a plurality of persons are obtained, and an AF operation based on the obtained focus evaluation values is performed. That is, each of a plurality of human faces becomes an AF area. Also in this case, similarly to the present embodiment, regions of similar color component values (color component values included in a predetermined range) included in each AF area are extracted as main subject regions.

  As shown in FIG. 7, when a plurality of human face areas (areas indicated by reference numerals 81 to 90) are extracted from the image I ″ as the main subject area, the CPU 20 determines that the captured scene is a group photograph. Also in this case, as in the present embodiment, the data processing circuit 23 generates a saliency map using the captured image data, and the data processing circuit 23 is extracted based on the generated saliency map. The data processing circuit 23 determines whether the saliency for each face area of the plurality of persons is within a predetermined range with respect to the average value. The data processing circuit 23 does not perform image correction on the image data when the saliency for each face area of a plurality of persons is included in a predetermined range with respect to the average value. On the other hand, If the saliency of the person's face area deviates from the predetermined range with respect to the average value, image correction processing is performed on the face area of the person whose saliency deviates from the predetermined range, So that the saliency at is included within a predetermined range.

  By the way, when taking a group photo, in addition to a plurality of persons as main subjects, persons who are not main subjects such as passers-by may be included in the imaging range and acquired images. In such a case, in addition to the subject extraction process described above, the data processing circuit 23 executes a process of identifying a person who is a main subject and other persons. The process for identifying the person who is the main subject and the other person is whether or not there is an area having a greatly different size (area) from among the detected face areas of the persons, or Appropriate methods such as whether there is a region where the feature amount of each region is significantly different, whether there is a region where the distribution of frequency components is different, or whether there is a person whose motion is different from other people, etc. It is done.

  The data processing circuit 23 classifies the extracted subject area into a main subject area and other subject areas after performing a process of identifying a person who is a main subject and other persons. Then, the data processing circuit 23 compares the saliency of the main subject area with the saliency of the other subject areas. Finally, when the saliency of the main subject area is lower than the saliency levels of the other subject areas, the data processing circuit 23 executes image correction processing on the main subject area, and sets the saliency of the main subject area to the other saliency levels. It should be higher than the saliency of the subject area. The data processing circuit 23 may perform image correction processing on other subject areas so that the saliency of the other subject areas is lower than the saliency of the main subject area.

  Note that the group photo is taken as an example, but the present invention is not limited to this, and a plurality of persons may be included in the imaging range even in normal imaging. Even in such a case, the saliency of the main subject region is determined by identifying the main subject region and any one of the other subject regions and performing image correction processing on any of the regions. It can be made higher than the saliency of other subject areas.

  Here, when determining whether or not it is a main subject area from a plurality of extracted subject areas, a face image that is a main subject is registered in advance, and the face image and the feature amount are registered in the plurality of extracted subject areas. A subject area that approximates is a main subject area. Then, the data processing circuit uses the saliency map to compare the saliency of the main subject region with the saliency of other subject regions. Based on this comparison, the data processing circuit performs image correction.

  In the present embodiment, the imaging apparatus is used. Specifically, a digital camera, a portable terminal represented by a portable phone having a camera function, and the like can be given. In addition, the present invention can also be applied to an image processing apparatus having the functions of the respective units of the data processing circuit 23 shown in FIGS. 1 to 3 and the function of the flowchart of FIG. . Furthermore, in addition to the imaging device and the image processing device, the invention can be applied to an imaging system including an imaging device such as a digital camera and a PC.

  For example, in the case of an image processing apparatus, a process of generating a saliency map, a process of extracting a subject area, and a process of correcting an image may be executed on input image data. Note that. The image data input to the image processing apparatus may be image data that has undergone signal processing, or image data that has not been subjected to signal processing.

  In the present embodiment, the imaging apparatus is taken as an example, but the functions of the respective units of the data processing circuit 23 shown in FIGS. 1 to 3 and the functions of the flowchart of FIG. 4 can be executed by a computer such as a PC. An image correction program may be used. This image correction program is preferably stored in a computer-readable storage medium such as a memory card, magnetic disk, or optical disk.

  DESCRIPTION OF SYMBOLS 10 ... Imaging device, 20 ... CPU, 22 ... Signal processing circuit, 23 ... Data processing circuit, 24 ... Liquid crystal monitor, 27 ... Storage medium, 31 ... Map generation part, 32 ... Subject extraction part, 33 ... Image correction part, 41 ... Luminance information extraction part, 42 ... Color information extraction part, 43 ... Edge information extraction part, 44 ... Map composition part

Claims (17)

Area detecting means for extracting a first area included in the input image;
Map generating means for generating a saliency map indicating a distribution of saliency based on a plurality of feature elements using the input image;
Based on the first region extracted by the region detection unit and the saliency map generated by the map generation unit, at least one of the first region or the second region excluding the first region Image correction means for performing image correction;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
The image correction means performs image correction on the first area when the saliency of the first area is lower than the saliency of the second area, and sets the saliency of the first area to the second area. An image processing apparatus characterized in that the image processing apparatus has a higher degree of saliency. The image processing apparatus according to claim 2,
The image processing apparatus, wherein the image correction unit performs image correction on a feature element having the lowest saliency in the second area among the plurality of feature elements. The image processing apparatus according to claim 2,
The image processing device according to claim 1, wherein the image correction unit performs image correction on a feature element having the lowest saliency in the first region among the plurality of feature elements. The image processing apparatus according to claim 1.
The image correction unit performs image correction on the second area when the saliency of the second area is higher than the saliency of the first area, and sets the saliency of the second area to the first area. An image processing apparatus characterized by lowering the saliency of the image. The image processing apparatus according to claim 5.
The image processing apparatus, wherein the image correction unit performs image correction on a feature element having the highest saliency in the second area among the plurality of feature elements. The image processing apparatus according to any one of claims 1 to 6,
The image processing apparatus according to claim 1, wherein the second area is an area within a predetermined range from the first area. The image processing apparatus according to any one of claims 1 to 6,
The image processing apparatus according to claim 2, wherein the second area is an area having a higher saliency than the saliency of the first area. The image processing apparatus according to any one of claims 1 to 8,
The first area is an area of a main subject included in the image,
The image processing apparatus according to claim 1, wherein the area detection unit extracts the area of the main subject as the first area based on an area used when focus adjustment is performed when the image is acquired. The image processing apparatus according to any one of claims 1 to 9,
An image processing apparatus comprising display means for displaying an image image-corrected by the image correction means. The image processing apparatus according to any one of claims 1 to 10,
An image processing apparatus, further comprising a storage unit that stores the saliency map generated by the map generation unit in association with the image corrected by the image correction unit. The image processing apparatus according to any one of claims 1 to 6,
For each of the plurality of feature elements, comprising setting means for setting a priority order for image correction in the image correction means,
The image correction unit, wherein the image correction unit executes the image correction in order from a feature element set with a high priority set by the setting unit among the plurality of feature elements. The image processing apparatus according to claim 1.
A determination means for determining a scene of the image;
The area detection means detects a plurality of person areas as the first area when the determination means determines that the scene is a group photo of a plurality of persons from the image, respectively.
The image processing apparatus according to claim 1, wherein the image correction unit performs image correction on each of the plurality of person areas so that the saliency for each of the first areas is included in a predetermined range. The image processing apparatus according to claim 13.
The area detecting means detects a person area excluding the plurality of persons in addition to the plurality of person areas,
The image correction means may be configured such that the areas of the plurality of persons or the persons excluding the plurality of persons are such that the saliency of the areas of the plurality of persons is higher than the saliency of the areas of the persons excluding the plurality of persons. An image processing apparatus characterized in that image correction is performed on the area. The image processing apparatus according to claim 1.
The area detecting means can detect areas of a plurality of persons included in the image,
The image correction unit is configured such that, among the plurality of person areas, the saliency in the person area as the main subject is higher than the saliency in the other person areas. Or an image processing apparatus that performs image correction on one of the areas of the other person. The image processing apparatus according to any one of claims 1 to 15,
An image pickup apparatus comprising: an image pickup device that acquires the input image. A region detection step of extracting a first region included in the input image;
A map generation step of generating a saliency map indicating a distribution of saliency based on a plurality of feature elements using the input image;
Based on the first region extracted by the region detection step and the saliency map generated by the map generation step, at least one of the first region or the second region excluding the first region An image correction process for performing image correction;
An image processing program capable of causing a computer to execute.

Images