JP2019102847A - Image processing device - Google Patents

Abstract

To reproduce the actual color tone that makes it difficult for a user who has confirmed an image to feel uncomfortable while accurately determining whether a shooting scene is a natural scene.SOLUTION: A CPU(32) divides image data of a natural image including only plants or plants and persons as subjects into a plurality of unit images, and obtains the feature amount of each of the unit images to calculate the plant area size or the person area size as a specific area size on the basis of the feature amounts of each of the unit images. Next, the CPU (32) determines whether to perform adjustment processing for the exposure amount of the image data according to the specific area size, and when it is determined to perform the adjustment processing of the exposure amount, the CPU adjusts the exposure amount of the image data according to the specific area size.SELECTED DRAWING: Figure 10

Description

  The present invention relates to an image processing apparatus.

  Some electronic cameras are equipped with an image processing apparatus that discriminates a photographed scene such as a natural scene and applies hue correction to the image according to the discriminated photographed scene. As a technique related to such an image processing apparatus, Patent Document 1 is disclosed.

  Specifically, Patent Document 1 discloses a technique in which a shooting scene is determined by determining the type of subject (for example, natural or building), and saturation correction is performed on an image according to the determination result. It is done.

Patent No. 5826001

  In order to reproduce the actual color tone in an image, it is first required to accurately determine the shooting scene.

  However, even if the shooting scene can be accurately determined, the color of the subject becomes vivid only by the saturation correction as shown in the above-mentioned Patent Document 1, and the color tone of the subject becomes unnatural and the user may feel discomfort. There is.

  This phenomenon is noticeable in natural scenes such as plants. The plant color is often "green", which is the color of leaves and the like, but if only the saturation of green is raised, the green in the image is simply emphasized.

  Therefore, the present invention is to accurately determine whether a shooting scene is a natural scene and reproduce an actual color tone that makes it difficult for a user who has checked an image to feel discomfort.

  A first aspect of the present invention is a feature amount calculation unit for obtaining feature amounts of unit images when image data of a natural image including only plants or plants and persons as subjects is divided into a plurality of unit images, and the unit According to the specific area size, an area amount calculation unit that calculates, as a specific area size, a plant area size where a plant appears in the image data or a person area size where a person appears in the image data. A determination unit that determines whether to perform adjustment processing of the exposure amount of the image data, and the image data according to the specific area size when it is determined to perform the adjustment processing of the exposure amount of the image data And an exposure control unit that adjusts the amount of exposure of the image processing apparatus.

  Here, the image data of the natural image is subdivided into a plurality of unit images, and the feature amount is obtained for each unit image. Whether the subject is a plant or a person is determined from the feature amount of each unit image, and the size of the plant area or the person area is calculated. Such processing allows the image processing apparatus to relatively accurately recognize whether or not it is a natural scene. Then, according to the specific area size, which is the plant area size or the person area size, whether or not the adjustment process of the exposure amount of the image data should be performed, and when the adjustment process of the exposure amount is performed, the exposure adjustment control is executed. Ru. That is, processing is performed to increase and decrease the overall brightness for each natural image that has been recognized relatively accurately. As a result, the vividness of a specific color such as green is not emphasized, and an image in which an actual color tone is reproduced, which makes it difficult for the user to feel discomfort can be obtained.

  In a second aspect based on the first aspect, when the plant area size of the specific area size is equal to or larger than a first predetermined size, the determination unit performs the adjustment process of the exposure amount of the image data. The image processing apparatus is characterized in that the exposure control unit determines the amount of exposure of the image data to be reduced.

  In many cases, the plants are green. Here, when the plant area size is relatively large, by reducing the exposure amount of the image data, it is possible to obtain an image in which the overall brightness is suppressed while leaving the green color of the plant. As a result, an image of natural color in which the green color of the plant is settled can be obtained.

  A third aspect of the invention is the image processing device according to the second aspect, wherein the exposure control unit changes the degree of reducing the exposure amount in accordance with the size of the plant area.

  As a result, fine exposure control is performed according to the actual subject, so that an image of natural color in which the green color of the plant is settled can be obtained.

  In a fourth aspect based on the third aspect, the exposure control unit is configured to adjust the exposure amount according to the size of the plant area such that the exposure amount of the image data decreases as the plant area size increases. The image processing apparatus is characterized in that the degree of lowering is reduced.

  As a result, fine exposure control is performed according to the actual subject, so that an image of natural color in which the green color of the plant is settled can be obtained.

  In a fifth invention according to any one of the second invention to the fourth invention, when the plant area size is smaller than the first predetermined size, the determination unit adjusts the exposure amount of the image data. It is an image processing apparatus characterized by determining not to perform processing.

  Here, exposure control is not performed because the plant area size is relatively low. If exposure is performed, the entire image will be dark and the actual green color of the plant will be lost.

  In a sixth invention according to any one of the second invention to the fourth invention, when the plant area size of the specific area size is equal to or larger than a second predetermined size, the green color of the plant in the natural image The image processing apparatus is further provided with a first color attribute adjustment unit that corrects the saturation of the natural image so that the saturation of the component is increased.

  Here, when the plant area size is relatively large, not only the exposure control but also the correction for increasing the saturation is performed. As a result, while the brightness is generally suppressed by the control for reducing the exposure, an image of a natural color tone in which the green is not too dark due to the control for increasing the saturation is obtained.

  A seventh invention according to the sixth invention is characterized in that the first color attribute adjustment unit changes the degree of increasing the saturation of the green component of the plant according to the size of the plant area. It is an image processing device.

  Thereby, it can suppress that the green color of the plant which concerns on an image is emphasized too much.

  In an eighth aspect based on the first aspect, when the person area size of the specific area size is equal to or larger than a third predetermined size, the determination unit does not perform the adjustment process of the exposure amount of the image data. And an image processing apparatus characterized by

  Here, in a state in which a plant and a person are included in the natural image, when the size of the person area is relatively large, the process of adjusting the exposure amount is not executed with emphasis on the person rather than the plant in one image. If the amount of exposure is lowered, the entire image becomes dark, and it is likely that the image may be affected by the fact that the person is difficult to see.

  In a ninth aspect based on the eighth aspect, when the person area size of the specific area size is smaller than the third predetermined size, the determination unit adjusts the exposure amount of the image data. And the exposure control unit is configured to reduce the exposure amount of the image data.

  Here, in a state where a plant and a person are included in the natural image, when the size of the person area is relatively small, control is performed to reduce the exposure amount by placing more importance on the plant than the person in one image. As a result, the overall brightness is suppressed while leaving the green color of the plant, and a natural color image in which the green color of the plant is calm is obtained.

  A tenth invention is the image processing apparatus according to the ninth invention, wherein the exposure control unit changes the degree to which the exposure amount is reduced according to the size of the person area.

  Here, with emphasis on plants rather than the person in one screen, fine exposure control is performed according to the area size of the plant in the actual subject, so an image of natural color in which the green color of the plant is settled is obtained. Be

  In an eleventh aspect based on the tenth aspect, the exposure control unit is configured to set the exposure amount according to the size of the person area such that the exposure amount of the image data decreases as the person area size decreases. The image processing apparatus is characterized in that the degree of lowering is reduced.

  Here, with emphasis on plants rather than the person in one screen, fine exposure control is performed according to the area size of the plant in the actual subject, so an image of natural color in which the green color of the plant is settled is obtained. Be

  In a twelfth invention according to any one of the second invention to the eleventh invention, when the plant in the natural image includes a part of a flower, the saturation of the color component of the flower is increased. The image processing apparatus may further include a second color attribute adjustment unit that corrects the saturation of the natural image.

  Flowers often have brighter colors (eg red) than green. Here, when the plant contains a flower, correction is performed to increase the saturation of the color component of the flower. As a result, the color of the flower part included in the image becomes bright, and an image conforming to the actual color tone is obtained.

  In a thirteenth invention according to any one of the second invention to the twelfth invention, when the luminance of the subject is equal to or higher than a predetermined luminance and the subject distance of the object is equal to or longer than a predetermined distance, the image data is A scene determination unit that determines that the image is a landscape scene, and a sharpness process that is performed on the image data so that the sharpness of the natural image is increased when the scene determination unit determines that the scene is a scene According to another aspect of the present invention, there is provided an image processing apparatus further comprising:

  Here, in the case of a landscape scene, sharpness processing is performed such that the contour of the subject included in the natural image becomes clearer. As a result, it is possible to obtain an image that is easy to clearly grasp what is shown in the natural image.

  In a fourteenth invention according to any one of the second invention to the fourth invention, the natural image further includes a sky, and the area amount calculation unit reflects the sky in the image data. Auto-focus control unit performs pan-focus control or auto-focus control of distant priority when the empty area size is further calculated and the empty area size is the fourth predetermined size or more, and the empty area size is the fourth predetermined size or more The image processing apparatus may further include a second sharpness adjustment unit that performs sharpness processing on the image data so that the sharpness of the natural image is increased.

  When the sky area size is relatively large, the shooting scene is likely to be an outdoor landscape scene. Therefore, the image processing apparatus performs pan focus control or auto focus control with distant priority. Thereby, an image captured without blurring the natural image can be obtained. Furthermore, the image processing apparatus performs sharpness processing so that the contour of the subject included in the natural image is clearer. As a result, it is possible to obtain an image that is easy to clearly grasp what is shown in the natural image.

  According to the present invention, it is relatively accurately judged whether or not it is a natural scene, and the vividness of a specific color is not emphasized, and the actual color tone is difficult for the user to feel uncomfortable. A reproduced image is obtained.

FIG. 1 is a block diagram schematically showing a configuration of an imaging device provided with an image processing apparatus. FIG. 2 is a block diagram showing a configuration example of a signal processing circuit. FIG. 3 is an illustrative view showing one example of a block formed in an image. FIG. 4 is an illustrative view showing one example of a mapping state of the SDRAM. FIG. 5 is an explanatory view of the concept of dictionary data. FIG. 6 is a diagram showing a flow of main operations after activation of the image processing apparatus. FIG. 7 is a diagram showing in detail the flow of operation for determining the possibility of a natural scene. FIG. 8 is a diagram for explaining the calculation of the possibility of a natural scene and the calculation of the specific area size when the main subject is only a plant. FIG. 9 is a diagram for describing calculation of the possibility of a natural scene and calculation of a specific region size when main subjects are a person and a plant. FIG. 10 is a diagram showing the flow of the exposure control operation and the color tone adjustment operation in the natural scene in the first embodiment. FIG. 11 is a graph showing the relationship between the plant area size and the reduction in exposure. FIG. 12 is a graph showing the relationship between the plant area size and the amount of increase in saturation. FIG. 13 is a graph showing the relationship between the person area size and the amount of decrease in exposure. FIG. 14 is a diagram showing the flow of the operation of the image processing apparatus when the landscape scene is determined in the second embodiment. FIG. 15 is a diagram showing only the part of the flow of the exposure control operation and the color tone adjustment operation in the natural scene, which is different from the first embodiment, in the third embodiment.

  Hereinafter, embodiments of the present invention will be described in detail based on the drawings. The following embodiments are essentially preferred examples, and are not intended to limit the scope of the present invention, its applications, or its applications.

First Embodiment
<Overview>
The image processing apparatus (100) according to the present embodiment is included in an imaging apparatus (10) capable of capturing images in various shooting modes, and performs image processing on the captured image. As an imaging device (10), electronic cameras, such as a digital camera, are mentioned.

  The image processing apparatus (100) has a determination function of a shooting scene and an image processing function of providing a user with an optimal image according to the shooting scene.

  In particular, in the present embodiment, when the imaging device (10) images nature including only a plant or a person and a plant as a subject, the image processing device (100) automatically determines that the photographed scene is a natural scene. . Then, the image processing apparatus (100) causes the image to have a natural color tone such that the color tone of the natural image captured under the natural scene is such that the user does not feel so much discomfort according to the content of the natural image. It performs exposure control and hue correction at the time of imaging.

-Imaging device-
The imaging device (10) includes an imaging unit (11) and a monitor (30) in addition to the image processing device (100).

  The imaging unit (11) acquires image data, and includes drivers (18a, 18b, 18c), a focus lens (12), an aperture unit (14), and an image sensor (20).

  The drivers (18a, 18b, 18c) are connected correspondingly to the focus lens (12), the aperture unit (14) and the image sensor (20) to drive the connected functional units. The focus lens (12) is a lens for focusing on an object, and is movably provided in a housing (not shown) of the imaging device (10). The aperture unit (14) is for adjusting the exposure amount. The image sensor (20) is an imaging device, and an optical image of an object is incident on the light receiving surface of the image sensor (20) through the focus lens (12). The image sensor (20) photoelectrically converts this optical image to generate a raw image signal corresponding to the optical image. Furthermore, in the image sensor (20), noise removal and level adjustment are performed on the read raw image signal of each frame.

  An imaging apparatus (10) including such an imaging unit (11) has an autofocus function that automatically focuses on a main subject, and an automatic exposure that automatically controls exposure (in other words, the amount of aperture opening and exposure period) It has a function. Each driver (18 a to 18 c) of the imaging unit (11) is controlled by a CPU (32) of an image processing apparatus (100) described later in order to realize each of these functions.

  Although not shown, the monitor (30) is provided on the surface of the housing of the imaging device (10) at a position where the user can visually recognize. The monitor (30) displays an image captured by the imaging device (10), a menu screen used for various settings of the imaging device (10), and the like.

<Configuration of Image Processing Device>
The configuration of the image processing apparatus (100) according to the present embodiment will be specifically described along with the operation at the time of shooting.

  As shown in FIG. 1, the image processing apparatus (100) includes a signal processing circuit (26), a memory control circuit (24), an SDRAM (38), a video encoder (28), an I / F circuit (34), and a memory card. (36) A JPEG codec (27), a storage unit (42), a CPU (32) and a shutter button (40) are provided.

  Among these, the signal processing circuit (26) has functions as a first color attribute adjustment unit, a second color attribute adjustment unit, a first sharpness adjustment unit, and a second sharpness adjustment unit. The CPU (32) has functions as a feature amount calculation unit, an area amount calculation unit, a determination unit, an exposure control unit, a scene determination unit, and an autofocus control unit.

  The signal processing circuit (26) has the configuration shown in FIG. The signal processing circuit (26) performs color separation processing, sharpness adjustment processing, color adjustment processing, YUV conversion processing, and the like on the raw image signal (digital signal) of each frame output from the image sensor (20). Generate data.

  Specifically, since each pixel that generates the raw image signal has only one color information of Cy, Ye, Mg, and G, the color separation circuit (26a) first complements the color information that each pixel lacks Be done. In the sharpness adjustment circuit (26b), sharpness adjustment processing for improving sharpness is performed on the image data after the color information is complemented by the color separation circuit (26a). The RGB conversion circuit (26c) performs RGB conversion on the image signal output from the sharpness adjustment circuit (26b). The white balance adjustment circuit (26d) has an amplifier (not shown) and performs white balance adjustment by amplifying the image signal output from the RGB conversion circuit (26c). The image signal subjected to the white balance adjustment is input to the LCH adjustment circuit (26e) and the integration circuit (26g).

  The LCH adjustment circuit (26e) combines the R component, the B component, and the G component of the image signal adjusted by the white balance adjustment circuit (26d) with the L component as the lightness component and the C component as the saturation component. When converted into an H component which is a hue component, each of the converted components is subjected to each correction of lightness, saturation and hue as necessary. The YUV conversion circuit (26f) converts the image signal output from the LCH adjustment circuit (26e) into a YUV signal. The generated YUV signal includes a Y component as luminance data, a U component as a color difference component, and a V component, and the ratio of Y: U: V is “4: 2: 2”.

  The image signal output from the white balance adjustment circuit (26d) is also input to the integration circuit (26g). The Y component relating to the YUV signal output from the YUV conversion circuit (26f) is input to the integration circuit (26h). As shown in FIG. 3, an object field (i.e., a screen) is divided into 16 in each of the vertical direction and the horizontal direction, and 256 blocks are formed on the screen. Each block is assigned a vertical position signal i (i is an integer of 0 to 15) and a horizontal position signal j (j is an integer of 0 to 15).

  The integration circuit (26g) integrates each of the R component, G component and B component of the input image signal for each block, and the integration circuit (26h) integrates the Y component of the input image signal for each block. Integrate. As a result, 256 integral values r (i, j) for R component, 256 integral values g (i, j) for G component, and 256 integral values b (i, j) for B component are integrated. The circuit (26g) outputs one frame period at a time, and 256 integral values y (i, j) of the Y signal are outputted from an integrating circuit (26h) at one frame period.

  The integral values r (i, j), g (i, j) and b (i, j) can be used as AE evaluation values at the time of exposure control. The integral value y (i, j) can be used to determine a shooting scene.

  The integration circuit (26g) may further calculate an integrated value of high frequency components of the R component, the G component, and the B component. This is because it can be used as an AF evaluation value at the time of autofocus control.

  Returning to FIG. 1, the YUV signal output from the signal processing circuit (26) is input to the memory control circuit (24). The input YUV signal is written as image data to the display image area 38a of the SDRAM 38 according to FIG. 4 or read out from the display image area 38a by the memory control circuit 24. .

  When the image data of each frame stored in the display image area (38a) of the SDRAM (38) is read by the memory control circuit (24), the video encoder (28) converts the image data into a composite image signal of NTSC format. Encode The video encoder (28) outputs this composite image signal to the monitor (30). Thereby, a real time moving image (through image) of the subject is displayed on the monitor (30).

  The 256 integral values y (i, j) output from the integration circuit (26h) shown in FIG. 2 are taken in by the CPU (32) and set in the register (rgst1). Since the integrated value y (i, j) is generated every one frame period, the set value of the register (rgst1) is updated every one frame period.

  When the shutter button (40) is pressed halfway, a status signal indicating that the shutter button is pressed halfway is given to the CPU (32). The CPU (32) sets 256 integral values y (i, j) output from the integration circuit (26h) after half depression of the shutter button (40) in the register (rgst2). As a result, the integral values y (i, j) of two consecutive frames are set in the register (rgst1, rgst2). The CPU (32) performs scene determination of an object scene using these integral values y (i, j), and performs focus adjustment according to the scene.

  For example, the CPU (32) determines whether the object scene is a sports scene based on the integral value y (i, j) set in the register (rgst1, rgst2). When the possibility judgment of the sport scene is completed, the CPU (32) performs focus adjustment using, for example, an AF evaluation value. At this time, the focus lens (12) is moved in the optical axis direction by the driver (18a) and set at the in-focus position. When focus adjustment is completed, the CPU (32) sets each integral value r (i, j), g (i, j), b (i, j), y (i, j) in the register (rgst3). Do. When loading of integral values r (i, j), g (i, j), b (i, j), y (i, j) for one frame is completed, the CPU (32) enables portrait scene Determine the sex, the possibility of a sunset scene, the possibility of a night scene, the possibility of a natural scene. The determination of each scene will be described later in the "flow of the entire operation" and "natural scene determination operation".

  When the possibility of each scene is calculated, the most probable scene is determined as the object scene. The camera setting, that is, the shooting mode is changed according to the decided scene. A message corresponding to the determined scene is displayed on the monitor (30), and exposure adjustment according to the scene is performed.

  When the photographer fully presses the shutter button (40) after the exposure adjustment is completed, the photographing process and the recording process are performed. First, the CPU (32) causes the image sensor (20) to execute main exposure, and causes the image sensor (20) to read one frame of raw image signal obtained by the main exposure. This raw image signal is supplied to the memory control circuit (24), and is written to the raw image area (38b) of the SDRAM (38) according to FIG. 4 by the memory control circuit (24).

  After writing to the raw image area (38b) is completed, the memory control circuit (24) reads the raw image signal. The raw image signal read by the memory control circuit (24) is given to the signal processing circuit (26). The signal processing circuit (26) executes a series of processing such as color separation processing, sharpness adjustment processing, RGB conversion processing, white balance adjustment processing, LCH adjustment processing, YUV conversion processing described with reference to FIG. A signal (main YUV signal) is generated.

  The main YUV signal output from the signal processing circuit (26) is applied to the memory control circuit (24) and is written to the main image area (38c) of the SDRAM (38) according to FIG. 4 by the memory control circuit (24). . When writing is complete, the CPU (32) generates a reduced YUV signal based on the main YUV signal. Specifically, the CPU (32) accesses the SDRAM (38) via the memory control circuit (24) and generates a reduced YUV signal by software processing. The generated reduced YUV signal is written to the reduced image area 38d of the SDRAM 38 according to FIG.

  The memory control circuit (24) reads the main YUV signal and the reduced YUV signal from the SDRAM (38) and provides each YUV signal to the JPEG codec (27). The JPEG codec (27) compresses the provided main YUV signal and the reduced YUV signal according to the JPEG format to generate a compressed main YUV signal and a compressed reduced YUV signal. The generated compressed primary YUV signal and the compressed reduced YUV signal are written by the memory control circuit (24) to the compressed main image area (38e) and the compressed image area (38f) of the SDRAM (38) according to FIG. .

  Thus, when the photographing process is completed, the CPU (32) executes a recording process. Specifically, the CPU (32) accesses the SDRAM (38) through the memory control circuit (24) to compress the compressed main YUV signal and the compressed reduced YUV signal, the compressed main image area (38e) and the compressed reduced image. It reads from each area (38f). Further, the CPU (32) records the read compressed main YUV signal and the compressed / reduced YUV signal in a file format on the memory card (36) via the I / F circuit (34). The memory card (36) is detachably provided in a housing (not shown) of the imaging device (10).

  Further, a storage unit (42) configured by a flash memory or the like is electrically connected to the CPU (32). The storage unit (42) stores dictionary data (DB) used when recognizing the type of subject. As an example, in the dictionary data (DB) of FIG. 5, data (G1) of feature amounts of leaves of plants, data (G2) of feature amounts of flowers, and data (G3) of feature amounts of people are registered. There is. The feature amount includes brightness, lightness, saturation, hue, color component, edge component, direction, or a combination thereof. Although the type of the feature amount is not particularly limited, in the present embodiment, a case where at least an edge component and a color component are included in the feature amount is exemplified.

  When the subject is a person, the shapes of the organs (eyes, mouth, etc.) of the face are almost the same, and the color of the face is skin color, so the above data (G3) pertaining to the person is an averaged value of edge components And the averaged value of skin color can be registered. On the other hand, plants are more complicated in shape and more complex, and have all four edge components including vertical, horizontal, diagonal right and diagonal left. With regard to the color of the plant, green for leaves and various colors other than green for flowers (red, yellow, purple, etc.) can be mentioned. Therefore, it is preferable that information indicating that edge components in all directions exist in the same unit image, and further, an averaged value of green be registered in the data (G1) relating to the leaves of the plant. The above-mentioned data (G2) relating to the flower of a plant may contain edge components of various flower simple substances or values obtained by averaging these edge components, or all directions in the same unit image may be registered. In addition to the information that the edge component is present, various color components or averaged values for each color component may be registered in combination.

  In the present embodiment, when it is determined that the object scene is a natural scene, the exposure adjustment and the color tone adjustment processing according to the contents of the object scene are performed. The exposure control operation and the color tone adjustment operation in FIG.

<Operation>
Next, the operation performed by the image processing apparatus (100) will be described in detail.

-Flow of overall operation-
FIG. 6 is a diagram showing the flow of the main operation after activation of the image processing apparatus (100).

  First, when the power of the imaging apparatus (10) is turned on, the CPU (32) instructs the drivers (18a to 18c), the signal processing circuit (26) and the video encoder (28) to perform processing related to through image processing. give. Thereby, through image processing is performed (step S1).

  The through image processing is as described in “<Configuration of image processing apparatus>”. That is, the signal processing circuit (26) generates image data based on the raw image signal (digital signal) output from the image sensor (20), and the memory control circuit (24) Store in the image area of). Thereafter, the memory control circuit (24) reads the image data of each frame from the image area of the SDRAM (38), and the video encoder (28) encodes the image data into a composite image signal and displays it on the monitor (30). Display.

  Thereafter, when the shutter button (40) is pressed halfway (Yes in step S2), the CPU (32) calculates the possibility that the object scene is a sports scene (step S3), and the possibility of being a portrait scene (Step S4), calculation of the possibility of being a sunset scene (step S5), and calculation of the possibility of being a night scene (step S6). In addition, the application of a prior art is possible for these specific calculation methods, and the detailed description is abbreviate | omitted here.

  The CPU (32) calculates the possibility that the object scene is a natural scene by the method according to the present embodiment (step S7).

  Next, the CPU (32) specifies the possibility that the percentage is the highest among the possibilities calculated in steps S3 to S7, and determines the photographing scene corresponding to the specified possibility as the photographing scene corresponding to the object scene Then, the photographing mode corresponding to the decided photographing scene is set (step S8). The setting of the shooting mode includes various settings such as a setting related to exposure adjustment according to a shooting scene and a setting related to color tone adjustment.

  When it is determined that the object scene is a sports scene, the CPU (32) corrects the program diagram so that the subject with motion is clearly photographed (sports mode setting). When the scene is determined to be a portrait scene, the CPU (32) corrects the program diagram so that the background is blurred, and is used in white balance adjustment so that the change in the skin color of the person can be suppressed Correct the gain (portrait mode setting). The CPU (32) corrects the program diagram so that the distant view is captured clearly when the object scene is finalized in the evening scene, and is used in white balance adjustment so that the change in the color of the sunset can be suppressed. Correct the gain (evening mode setting). When it is determined that the object scene is a night scene, the CPU (32) corrects the program diagram so that the illumination is prominent (night scene mode setting). When it is determined that the object scene is a natural scene, the CPU (32) corrects the program diagram for realizing the exposure control operation shown in FIG. 10 and realizes the color tone adjustment operation shown in FIG. The correction of the target value used in the LCH adjustment processing is performed using the AE evaluation value or the like.

  After that, the CPU (32) sets the optimum aperture and the optimum exposure based on the integrated value y (i, j) set in the register (rgst2) and the program diagram corrected by the setting of the photographing mode described above. The time (shutter speed) is specified, and the specified aperture amount and exposure time are set in the aperture unit (14) via the driver (18b).

  Thereafter, after the shutter button (40) is full-pressed (Yes in step S9), the CPU (32) detects this. While the shutter button (40) is full-pressed, shooting processing and recording processing are performed (step S10). In the photographing process, the CPU (32) sets a shutter mechanism (not shown) disposed between the aperture unit (14) and the image sensor (20) at a time when an optimal exposure time has elapsed from the start of the main exposure. Drive by. An object scene image is stored in the memory card (36) by such photographing processing and storage processing.

  When the shutter button (40) is not full-pressed (No in step S9) and after the photographing process and the storage process according to step S10 are completed, the operations after step S2 are repeated.

-Natural scene judgment operation-
The flow of the operation of determining the possibility of the natural scene according to step S7 in FIG. 6 will be described below using FIGS.

  First, the CPU (32) divides the image data into a plurality of unit images (br1, br2,...) (Step S71). The size of the unit image (br1, br2, ...) is not particularly limited, but may be, for example, the same size as the block shown in FIG.

  The CPU (32), as shown in FIGS. 8 and 9, searches the search frame (sf) of the same size as the unit image (br1, br2,...) And sets the first unit image (br1) on the image data. When the search frame (sf) is moved to the adjacent unit image (br2,...), The search frame (sf) is moved (step S72). Along with the moving process of the search frame (sf), the CPU (32) extracts the feature amount for the unit images (br1, br2, ...) on which the search frame (sf) is superimposed (step S73) . The CPU (32) collates the extracted feature amount with dictionary data (DB) in the storage unit (42) (step S74) to recognize what kind of subject is shown in the unit image Do.

  In the present embodiment, for convenience of explanation, the case where the main subject to be imaged is only a plant (FIG. 8) or a combination of a plant and a person (FIG. 9) will be taken as an example. For example, if the subject shown in the unit image is a plant containing only leaves, the CPU (32) extracts the color component "green" and the edge component "all directions" of the plant as feature quantities, and extracts them as a dictionary It is recognized as a unit image of “plant leaf” by collating with data (DB). When the subject shown in the unit image is a plant flower, the CPU (32) extracts only the combination of the color component and the edge component of the flower or the color component of the flower as the feature amount, and these are extracted as dictionary data ( It matches with DB) and recognizes it as a unit image of "flower". When the subject shown in the unit image is a person, the CPU (32) extracts the color component "skin color" and the edge component of the person as feature amounts, collates them with dictionary data (DB), and It recognizes as a unit image of "person".

  The processes after step S72 are repeated until the collation process of step S74 is executed for all unit images (br1, br2,...) Included in the image data (No in step S75). When the collation process of step S74 is executed for all unit images (br1, br2, ...) included in the image data (Yes in step S75), the CPU (32) recognizes all unit images Based on the result, the possibility that the subject is a natural scene is calculated according to the number of unit images whose recognition result is "plant" (step S76).

  Furthermore, based on the recognition results of all unit images, the CPU (32) determines the size of the area in which the plant appears in the image data (hereinafter referred to as the plant area size) or the area in which the person appears in the image data. The size (hereinafter referred to as the person area size) of is calculated as the specific area size (step S77). For example, in FIG. 8, the main subject is only a plant. In this case, a range including all unit images enclosed by thick lines in FIG. 8 corresponds to the plant area size (rg1). In FIG. 9, the main subjects are plants and people. In this case, a range including all unit images enclosed by thick lines of the person in FIG. 9 is the person area size (rg2), and the units enclosed by thick lines of the plants located on the left and right sides of the person respectively. The area including the entire image is the plant area size (rg1).

  After completion of the process of step S77 according to FIG. 7, the operation of the image processing apparatus (100) shifts to step S8 of the upper layer operation flow (FIG. 6).

-Exposure control operation and color tone adjustment operation in a natural scene-
Hereinafter, the flow of the exposure control operation and the color tone adjustment operation when the shooting scene is a natural scene in step S8 of FIG. 6 will be described with reference to FIGS. That is, it is premised here that as a result of the possibility judgment of each scene being made in steps S3 to S7 in FIG. 6, it is determined in step S8 that the photographed scene is a natural scene.

  When the CPU (32) determines whether or not to adjust the exposure amount of the image data according to the specific area size calculated in step S77 of FIG. 7, and determines to perform the adjustment processing of the exposure amount Performs a process of adjusting the exposure amount of the image data in accordance with the specific area size.

  Specifically, as shown in FIG. 10, when the main subject is only a plant without a person and the plant area occupies most in the image data (Yes in step S101), the CPU (32) The plant area size in the specific area size is compared with the first predetermined size (step S102).

  If the plant area size is equal to or larger than the first predetermined size (Yes in step S102), the CPU (32) determines that the adjustment processing of the exposure amount of the image data is to be performed, and performs the processing of reducing the exposure amount of the image data. It does (step S103). In the processing, the degree to which the exposure amount is reduced changes according to the size of the plant area.

  FIG. 11 is a graph showing the relationship between the plant area size and the exposure reduction amount in the processing according to step S103, where the horizontal axis is the plant area size and the vertical axis is the degree of reduction of the exposure amount (exposure reduction amount). Is represented. As shown in FIG. 11, in the present embodiment, as the plant area size is larger, the degree to which the exposure is reduced (the amount of reduction of the exposure) is higher. That is, the larger the plant area size, the lower the exposure of the image data. By applying the calculated plant area size to the graph of FIG. 11, it is determined how much the exposure amount is reduced.

  As shown in FIG. 11, it is preferable that the relationship between the plant area size and the reduction in exposure be defined such that the reduction in exposure increases linearly as the plant area size increases. It is not necessary to be limited to

  In accordance with the exposure amount determined in step S103, the CPU (32) sets the optimum aperture amount and the optimum exposure time (shutter speed) in the aperture unit (14) by the driver (18b). Thus, when the shutter button (40) is full-pressed, the photographing process is performed with the set exposure time and aperture amount.

  Further, since the plant area size is equal to or larger than the first predetermined size in step S102 (Yes in step S102), the signal processing circuit (26) (specifically, the LCH adjustment circuit (26e)) A process of correcting the saturation of the natural image is performed so that the saturation of the component is increased (step S104). In the processing, the degree of increasing the saturation of the green component of the plant changes according to the size of the plant area.

  In FIG. 12, with the horizontal axis representing the plant area size and the vertical axis representing the rate of increasing the saturation (rise amount of saturation), the relationship between the plant area size and the rise amount of saturation in the processing according to step S104 is shown. It is represented by a graph. As shown in FIG. 12, in the present embodiment, as the plant area size is larger, the degree of increase in saturation (the amount of increase in saturation) is higher. That is, as the plant area size is larger, the saturation of the image data is further increased. By applying the calculated plant area size to the graph of FIG. 12, it is determined how much the saturation is increased.

  As shown in FIG. 12, it is preferable that the relationship between the plant area size and the increase amount of saturation be defined such that the increase amount of saturation increases linearly as the plant area size increases. It is not necessary to be limited to FIG.

  In FIG. 10, the “first predetermined size” used as a threshold in the determination of whether to adjust the exposure amount is also used in common in the determination of whether to increase the saturation. (Step S102). However. In addition to the threshold "first predetermined size" used in the determination of whether to adjust the exposure amount, a threshold "second predetermined size" used in the determination of whether to increase the saturation may be set.

  Although not shown in FIG. 10, in the possibility judgment of the natural scene in FIG. 7, when the main subject such as a person or a flower is not recognized from the image data and only the plant of the green component is recognized The image is likely to be a landscape painting. In this case, in step S104, the signal processing circuit (26) (specifically, the sharpness adjustment circuit (26b)) further performs the sharpness adjustment so that the sharpness of the green component of the plant in the natural image is further increased. Good.

  In step S102, when the plant area size is smaller than the first predetermined size (No in step S102), the CPU (32) determines that the adjustment processing of the exposure amount of the image data is not performed (step S105). In this case, when the shutter button (40) is full-pressed, photographing processing is performed with the exposure time and the aperture amount set initially.

  Further, when the main subject includes a person and a plant, and both the plant area and the person area occupy most of the image data (No in step S101, Yes in step S106), the CPU (32) The person area size in the specific area size is compared with the third predetermined size (step S107).

  When the person area size is equal to or larger than the third predetermined size (Yes in step S107), the CPU (32) determines that the adjustment process of the exposure amount of the image data is not performed (step S108). This is in order to emphasize the color tone for a person whose area is larger than that of a plant among subjects. In this case, when the shutter button (40) is full-pressed, photographing processing is performed with the exposure time and the aperture amount set initially.

  When the person area size is smaller than the third predetermined size (No in step S107), the CPU (32) determines to perform the adjustment processing of the exposure amount of the image data as in step S103, and the exposure amount of the image data Is performed (step S109). This is in order to emphasize the color tone to a plant whose area is larger than that of a person among subjects. In the processing, the degree to which the exposure amount is reduced changes according to the size of the person area.

  In FIG. 13, the abscissa represents the person area size, and the ordinate represents the degree to which the exposure amount is reduced (exposure reduction amount), and the relationship between the person area size and the exposure reduction amount in the process according to step S109 is graphed. Is represented. As shown in FIG. 13, in the present embodiment, the smaller the person area size is, the higher the degree to which the exposure is reduced (the reduction amount of the exposure). That is, the smaller the person area size, the lower the exposure amount of the image data. By applying the calculated person area size to the graph of FIG. 13, it is determined how much the exposure amount is reduced.

  As shown in FIG. 13, it is preferable that the relationship between the person area size and the exposure decrease amount is defined such that the decrease amount of exposure increases linearly as the person area size decreases. It is not necessary to be limited to

  In accordance with the exposure amount determined in step S109, the CPU (32) sets the optimum aperture amount and the optimum exposure time (shutter speed) in the aperture unit (14) by the driver (18b). Thus, when the shutter button (40) is full-pressed, the photographing process is performed with the set exposure time and aperture amount.

  In step S106, when the image data does not occupy most of the plant area and the person area (No in step S106), the CPU (32) decides not to adjust the exposure amount of the image data. (Step S110). In this case, when the shutter button (40) is full-pressed, photographing processing is performed with the exposure time and the aperture amount set initially.

  When the plant in the natural image includes a flower part, that is, when there is a unit image in which the subject is recognized as "flower" among the plants in Fig. 7 (Yes in step S111), the signal processing circuit (26) (Specifically, the LCH adjustment circuit (26e)) performs a process of correcting the saturation of the natural image so that the saturation of the color component of the flower increases (step S112). As a result, an image in which the color component of the flower is vivid is obtained.

<Effect>
The image processing apparatus (100) of the present embodiment divides image data of a natural image into a plurality of unit images (br1, br2, ...), and features of each unit image (br1, br2, ...) Determine the quantity. The image processing apparatus (100) determines whether the subject is a plant or a person from the feature amount of each unit image (br1, br2, ...), and calculates the size of the plant area or the person area. By such processing, the image processing apparatus (100) can relatively accurately recognize whether or not it is a natural scene. Then, the image processing apparatus (100) determines whether the exposure amount adjustment processing of the image data should be performed according to the specific region size which is the plant region size or the person region size, and when performing the exposure amount adjustment processing Control the amount of exposure. That is, in the present embodiment, the process of raising and lowering the overall brightness is performed for each of the natural images recognized relatively accurately. Therefore, the vividness of a specific color such as green is not emphasized, and an image in which an actual color tone is reproduced, which makes it difficult for the user to feel discomfort can be obtained.

  In many cases, the plants are green. Here, when the plant area size is relatively large, by reducing the exposure amount of the image data, it is possible to obtain an image in which the overall brightness is suppressed while leaving the green color of the plant. As a result, an image of natural color in which the green color of the plant is settled can be obtained.

  The degree to which the amount of exposure is reduced varies depending on the size of the plant area. Specifically, the degree of decreasing the amount of exposure changes in accordance with the size of the plant area so that the amount of exposure of the image data decreases as the plant area size increases. As a result, fine exposure control is performed according to the actual subject, so that an image of natural color in which the green color of the plant is settled can be obtained.

  On the other hand, when the plant area size is smaller than the first predetermined size, the adjustment processing of the image data exposure amount is not performed. If exposure is performed, the entire image will be dark and the actual green color of the plant will be lost.

  Further, in the present embodiment, when the plant area size is relatively large, not only the exposure control but also the correction for increasing the saturation is further performed. As a result, while the brightness is generally suppressed by the control for reducing the exposure, an image of a natural color tone in which the green is not too dark due to the control for increasing the saturation is obtained.

  The degree to which the saturation of the green component of the plant is increased varies depending on the size of the plant area. Thereby, it can suppress that the green color of the plant which concerns on an image is emphasized too much.

  Further, in the present embodiment, when the size of the person area is relatively large in a state in which the plant and the person are included in the natural image, the person is considered more important than the plant in one image, and the exposure adjustment processing is performed. Will not run. If the amount of exposure is lowered, the entire image becomes dark, and it is likely that the image may be affected by the fact that the person is difficult to see.

  On the other hand, in the present embodiment, when the person area size is relatively small, control is performed to lower the exposure amount by placing more importance on plants than the person in one image. As a result, the overall brightness is suppressed while leaving the green color of the plant, and a natural color image in which the green color of the plant is calm is obtained.

  The degree to which the amount of exposure is reduced varies depending on the size of the person area size. In particular, the degree of decreasing the exposure amount changes according to the size of the person area so that the exposure amount of the image data decreases as the person area size decreases. That is, in this case, the plant is emphasized rather than the person in one screen, and since the fine exposure control according to the area size of the plant in the actual subject is performed, the image of natural color in which the green color of the plant is settled Is obtained.

  In addition to green leaves, plants often contain colorful flowers. In the present embodiment, when the plant contains a flower, correction is performed to increase the saturation of the color component of the flower. As a result, the color of the flower part included in the image becomes bright, and an image conforming to the actual color tone is obtained.

Second Embodiment
In the present embodiment, the operation shown in FIG. 14 is further added to the “natural scene determination operation (FIG. 7)” and the “exposure control operation and color tone adjustment operation in a natural scene (FIG. 10)” described in the first embodiment. Case will be described.

  The configuration of each of the imaging device (10) and the image processing device (100) and the flow of the operation in FIG. 6 are the same as those in the first embodiment.

  First, the image processing apparatus (100) obtains subject brightness of a subject based on an evaluation value of lightness of a specific area or the like, and also obtains a subject distance of a subject as an AF result. Conventional techniques can be applied to these calculation methods, and the calculation may be performed by either the CPU (32) or the signal processing circuit (26). For example, integrated values r (i, j), g (i, j), b (i, j) y (i, j) etc. are used to calculate subject brightness, and to calculate subject distance, The integral value of the high frequency components of the R component, the G component, and the B component of the image signal for each of the 256 blocks of 3 is used.

  The CPU (32) performs the natural scene determination operation according to FIG. 7 from step S71 to step S77.

  Next, as shown in FIG. 14, the CPU (32) compares the calculated subject brightness with a predetermined brightness set in advance (step S201), and calculates the calculated subject distance with the preset predetermined distance. A comparison is made (step S202).

  When the subject brightness is equal to or higher than the predetermined brightness (Yes in step S201) and the subject distance is equal to or longer than the predetermined distance (Yes in step S202), a series of operations in FIG. "Color adjustment operation" is performed from step S101 to step S112.

  Next, the CPU (32) determines that the image data is captured as a natural scene, particularly as a landscape scene, since the subject brightness is a predetermined brightness or more and the subject distance is a predetermined distance or more. . In this case, the signal processing circuit (26) (specifically, the sharpness adjustment circuit (26b)) performs sharpness adjustment processing on the image data so as to increase the sharpness of the natural image (step S203).

  That is, here, in the case of a landscape scene, sharpness adjustment processing is performed so as to make the contour of the subject included in the natural image clearer. As a result, it is possible to obtain an image that is easy to clearly grasp what is shown in the natural image.

Third Embodiment
In this embodiment, the case where the operation according to FIG. 15 is added to the “exposure control operation and color tone adjustment operation in a natural scene (FIG. 10)” described in the first embodiment will be described.

  The configuration of each of the imaging device (10) and the image processing device (100) is the same as that in the first embodiment. The operations in FIGS. 6 and 7 are also the same as in the first embodiment.

  In this embodiment, assuming that the main subject is a plant and the sky, the CPU (32) further calculates the size of the region where the sky is reflected in the image data (hereinafter referred to as empty region size). I assume. Any method may be used as a method of detecting an area in which the sky is captured from image data. For example, the unit image having the feature value that the edge component of the unit image (br1, br2,...) Does not exist, the color component is blue, and the unit image having such a feature is located at the top of the natural image It is possible to adopt a method in which the CPU (32) recognizes that the unit image (br1, br2...) Is "empty" based on the above.

  First, when it is No in step S106 of FIG. 10 and the plant area and the sky area occupy most in the image data (Yes in step S301), the CPU (32) calculates the plant area in the calculated specific area size. The size is compared with the first predetermined size (step S302). When the plant area size is equal to or larger than the first predetermined size (Yes in step S302), the CPU (32) determines to perform the adjustment process of the exposure amount of the image data, and reduces the exposure amount of the image data (Step S303). The details of the process for reducing the exposure amount are the same as step S103 in FIG.

  Furthermore, when the sky area size is equal to or larger than the fourth predetermined size (Yes in step S304), the natural image is highly likely to be an outdoor natural scenery scene in which, for example, a mountain or a forest is photographed. Therefore, the CPU (32) sets the focal point to a fixed focus at a predetermined distance and fixes the aperture value at a predetermined aperture value or pan focus control, or moves the focus lens (12) by the driver (18a) The distance-focused autofocus control is performed to bring the camera into the distance (step 305). Further, the signal processing circuit (26) (specifically, the sharpness adjustment circuit (26b)) performs sharpness adjustment processing on the image data so as to increase the sharpness of the natural image (step S306). That is, sharpness adjustment processing is performed to make the contour of the subject included in the natural image clearer.

  As a result, it is possible to obtain an image that is easy to clearly grasp what is shown in the natural image.

  In addition, in FIG. 15, when the main subject is not a combination of a plant and the sky (No in step S301), the operations after step S101 in FIG. 10 are performed.

  In FIG. 15, when the plant area size is smaller than the first predetermined size (No in step S302), the operations in and after step S105 in FIG. 10 are performed.

  In FIG. 15, when the empty area size is smaller than the fourth predetermined size (No in step S304), and after step S306, operations in step S111 and subsequent steps in FIG. 10 are performed.

«Other Embodiments»
From the feature quantities of each unit image (br1, br2,...), The CPU (32) finds out that various color components are distributed in the green component in the possibility judgment of the natural scene in FIG. If so, it can be recognized that the colorful colors distributed are flowers. In this case, the signal processing circuit (26) (specifically, the LCH adjustment circuit (26e) etc.) performs correction to increase the saturation of the flower area in order to make the flower color vivid. good.

  In the third embodiment, in addition to the focus related control (step S305) and the correction to increase the sharpness (step S306), the signal processing circuit (26) further performs a correction to the natural image to increase the saturation. It is good.

  In recognition of the type of subject, a method not using dictionary data (DB) may be adopted.

  Although it has been described in step S103 in FIG. 10 that the degree to which the exposure is reduced changes depending on the size of the plant area, this is not essential. Similarly, FIG. 11 shows the case where the exposure amount decreases linearly as the plant area size increases, but the invention is not limited to this.

  Although step S104 in FIG. 10 is described as performing correction to increase the saturation, this is not essential. Similarly, FIG. 12 shows the case where the saturation of the green component of the plant linearly increases according to the size of the plant area, but the invention is not limited to this.

  Although it has been described in step S109 in FIG. 10 that the degree to which the exposure is reduced changes according to the size of the person area, this is not essential. Similarly, FIG. 13 shows the case where the exposure amount decreases linearly as the person area size increases, but the invention is not limited to this.

  Although step S111 and step S112 in FIG. 10 have been described as performing correction to increase the saturation of the flower part, this is not essential.

  The present invention is an image processing apparatus capable of appropriately determining that a photographed scene is a natural scene, and reproducing an actual natural color which does not cause much discomfort to an image captured under the natural scene. Useful as.

10 imaging device 11 imaging unit 26 signal processing circuit (first color attribute adjustment unit, second color attribute adjustment unit, first sharpness adjustment unit, second sharpness adjustment unit)
32 CPU (feature amount calculation unit, area amount calculation unit, determination unit, exposure control unit, scene determination unit, autofocus control unit)
100 image processing device

Claims (14)

A feature amount calculation unit for obtaining feature amounts of unit images when image data of a natural image including only plants or plants and persons as subjects is divided into a plurality of unit images;
An area amount calculation unit which calculates, as a specific area size, a plant area size where a plant appears in the image data or a person area size where a person appears in the image data based on the feature amounts of the unit images;
A determination unit that determines whether to perform adjustment processing of the exposure amount of the image data according to the specific area size;
An exposure control unit configured to adjust the exposure amount of the image data according to the specific area size when it is determined that the adjustment processing of the exposure amount of the image data is to be performed. In claim 1,
When the plant area size of the specific area size is equal to or larger than a first predetermined size, the determination unit determines to perform the adjustment process of the exposure amount of the image data, and the exposure control unit determines the image data And reducing the exposure amount of the image processing apparatus. In claim 2,
The image processing apparatus, wherein the exposure control unit changes a degree of reducing the exposure amount in accordance with the size of the plant area. In claim 3,
The exposure control unit may change the degree to which the exposure amount is reduced according to the size of the plant area so that the exposure amount of the image data decreases as the plant area size increases. Image processing device. In any one of claims 2 to 4,
The image processing apparatus, wherein the determination unit determines not to perform the adjustment process of the exposure amount of the image data when the plant area size is smaller than the first predetermined size. In any one of claims 2 to 4,
The first color for correcting the saturation of the natural image so that the saturation of the green component of the plant in the natural image is increased when the plant area size of the specific area size is equal to or larger than the second predetermined size Attribute Adjustment Department,
An image processing apparatus further comprising: In claim 6,
The image processing apparatus according to claim 1, wherein the first color attribute adjustment unit changes the degree of increasing the saturation of the green component of the plant according to the size of the plant area. In claim 1,
The image processing apparatus, wherein the determination unit determines not to perform the adjustment process of the exposure amount of the image data when the person area size of the specific area size is equal to or larger than a third predetermined size. In claim 8,
When the person area size of the specific area size is smaller than the third predetermined size, the determination unit determines to perform the adjustment process of the exposure amount of the image data, and the exposure control unit An image processing apparatus characterized by reducing the amount of exposure of data. In claim 9,
The image processing apparatus, wherein the exposure control unit changes the degree to which the exposure amount is reduced according to the size of the person area. In claim 10,
The exposure control unit may change the degree to which the exposure amount is reduced according to the size of the person area so that the exposure amount of the image data decreases as the person area size decreases. Image processing device. In any one of claims 2 to 11,
A second color attribute adjustment unit that corrects the saturation of the natural image so that the saturation of the color component of the flower increases when the plant in the natural image includes a portion of the flower;
An image processing apparatus further comprising: In any one of claims 1 to 12,
A scene determination unit that determines that the image data is a landscape scene when the brightness of the subject is equal to or greater than a predetermined brightness and the subject distance of the subject is equal to or greater than a predetermined distance;
A first sharpness adjustment unit that performs sharpness processing on the image data such that the sharpness of the natural image is increased when the scene determination unit determines that the scene is the landscape scene;
An image processing apparatus further comprising: In any one of claims 2 to 4,
The natural image further includes the sky,
The area amount calculation unit further calculates an empty area size in which the sky appears in the image data;
An auto-focus control unit that performs pan-focus control or auto-focus control of distant priority when the empty area size is equal to or larger than a fourth predetermined size;
A second sharpness adjustment unit that performs sharpness processing on the image data such that the sharpness of the natural image is increased when the empty area size is equal to or greater than a fourth predetermined size;
An image processing apparatus further comprising: