JP2010147660A - Image processor, electronic camera and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor capable of appropriately removing noise to be generated in an image irrespective of attributes in the image, and to provide an image processing program and an electronic camera. <P>SOLUTION: The image processor includes: an area division part which discriminates each object in the image based on either of an image to be input or photography information obtained in acquisition of the image to divide the image into a plurality of object areas including each object; a setting part which sets at least one of strength and types of noise removal processing to the object areas based on the objects in the object areas for every object area divided by the area division part; and a noise removal processing part which performs the noise removal processing for every plurality of object areas according to setting of the setting part. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that performs image processing, an electronic camera, and an image processing program.

2. Description of the Related Art Conventionally, image processing apparatuses that perform noise removal processing according to image characteristics are known. For example, the image processing apparatus of Patent Literature 1 divides an image into a plurality of rectangular areas, and determines an attribute for each rectangular area. And according to the attribute of a rectangular area, a noise removal filter is applied separately with respect to each component of brightness, hue, and saturation.
JP 2002-101422 A

  However, since the image processing apparatus described above divides an image into a plurality of rectangular areas and determines the attributes of each rectangular area, the divided rectangular areas include, for example, a part of the sky and a building. A rectangular area mixed with a part is also included. In the case of such a rectangular region in which a part of the sky and a part of the building are mixed, when the attribute is determined, the attribute is either the sky or the building. If it is determined that the attribute of the rectangular area is empty, strong noise removal processing suitable for the sky is executed, and the details of the building disappear. On the other hand, when it is determined that the attribute of the rectangular area is a building, weak noise removal processing is executed to leave the details of the building, and noise generated in the empty portion cannot be removed.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus, an image processing program, and an electronic camera that can appropriately remove noise generated in an image regardless of attributes in the image.

  The image processing apparatus of the present invention discriminates each subject in the image based on either the input image or the shooting information obtained at the time of obtaining the image, and assigns to a plurality of subject areas each including the subject. An area dividing unit that divides the image, and at least one of the strength and type of noise removal processing for the subject area is set based on the subject in the subject area for each of the subject areas divided by the area dividing unit. A setting unit; and a noise removal processing unit that performs noise removal processing for each of the plurality of subject areas in accordance with the setting of the setting unit.

  The setting unit may set at least one of the strength and type of noise removal processing for the subject area based on the spatial frequency obtained from the subject area.

  Further, the setting unit sets the strength of noise removal processing for a subject region including a lot of low frequency components among the plurality of subject regions to be stronger than the strength of noise removal processing for a subject region including many high frequency components. Also good.

  In addition, the setting unit may set the noise removal processing strength for a subject region determined as a predetermined subject among the plurality of subject regions divided by the region dividing portion, and the noise removal processing strength for another subject region. May be set to different intensities.

  Further, the predetermined subject is empty, and the setting unit sets the noise removal processing strength for the subject region determined to be empty as compared to the noise removal processing strength for the other subject regions. It may be set strongly.

  In addition, the predetermined subject is skin, and the setting unit sets the noise removal processing strength for the subject region determined to be the skin to be higher than the noise removal processing strength for the other subject regions. It may be set strongly.

  The electronic camera of the present invention includes the image processing apparatus of the present invention and an imaging unit that outputs an image signal that is a source of the image by photoelectrically converting the captured subject light beam.

  The image processing program of the present invention determines a subject area based on either an input image or photographing information obtained at the time of acquiring the image, and an area dividing step of dividing the image into the subject areas A setting step for setting at least one of the strength and type of noise removal processing for the subject region based on the subject in the subject region for each subject region divided by the region dividing unit; and It is possible to cause the computer to execute a noise removal processing step of performing noise removal processing for each of the plurality of subject areas in accordance with the setting.

  According to the image processing apparatus, the image processing program, and the electronic camera of the present invention, the noise removal process is executed for each subject included in the image, so that noise generated in the image can be appropriately removed.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following embodiments, an electronic camera is used as an example of the image processing apparatus of the present invention. FIG. 1 is a block diagram showing a configuration of an electronic camera 1 according to an embodiment of the present invention. As shown in FIG. 1, the electronic camera 1 includes an imaging lens 2, an imaging element 3, an A / D conversion unit 4, a buffer memory 5, an image processing unit 6, a control unit 7, a display unit 8, an operation unit 9, and a recording I. / F unit 10, recording medium 11, and bus 12.

  The imaging lens 2 forms a subject image on the imaging surface of the imaging element 3. In addition to the imaging lens 2, an imaging optical system including a plurality of lenses such as a zoom lens and a focus lens may be used. The imaging element 3 photoelectrically converts the subject light that has passed through the imaging lens 2 and outputs analog image signals corresponding to R, G, and B colors. In the shooting mode, in the shooting standby state in which the release button 25 is not operated, the image sensor 3 performs thinning-out reading at predetermined intervals and outputs an image signal that is a source of a through image. Thereafter, when the release button 25 is fully pressed, the imaging device 3 releases the shooting standby state and outputs an image signal that is the basis of an image obtained at the time of shooting (hereinafter referred to as a shot image).

  An image signal output from the image sensor 3 is input to the A / D converter 4. The A / D conversion unit 4 performs A / D conversion on the analog image signal output from the image sensor 3 and converts the analog image signal into a digital image signal. The digital image signals are collected into one frame and recorded in the buffer memory 5 as image data. The buffer memory 5 temporarily stores image data in the pre-process and post-process of image processing by the image processing unit 6.

  The image processing unit 6 performs image processing on the image data stored in the buffer memory 5. As this image processing, in addition to known white balance adjustment, color interpolation, gradation conversion processing, contour enhancement processing, etc., noise removal processing described later can be cited. Note that this noise removal processing is executed after color interpolation processing, for example. The image processing unit 6 also performs processing for compressing the image data subjected to the above-described image processing in a JPEG (Joint Photographic Experts Group) format or the like, and processing for decompressing and restoring the compressed data. To do.

  The image processing unit 6 has the function of the noise removal processing unit 21. The noise removal processing unit 21 performs the noise removal process described above. The noise removal processing unit 21 performs noise removal processing according to the subject area on each subject region of the captured image divided by the region division unit 23 described later. The noise removal process is performed by a known low-pass filter process.

  The control unit 7 performs overall control of the electronic camera 1 according to a predetermined sequence program, and performs AE processing, AF processing, and the like. In addition, the control unit 7 has functions such as a determination unit 22, a region division unit 23, and a setting unit 24.

  The determination unit 22 determines a subject included in the captured image. Examples of subjects include landscapes such as the sky and mountains, in addition to animals and plants and buildings including people. The determination unit 22 determines a subject included in the captured image by using a combination of subject recognition, histogram analysis, and texture analysis.

  Subject recognition is a technique for recognizing a subject in an image by well-known pattern matching or feature amount extraction. The determination unit 22 determines whether or not a person who is a subject is included in the captured image by subject recognition. Note that when pattern matching is used for subject recognition, it is possible to determine whether a photographed image includes a person's face or not, as well as whether a building or a living thing is included. On the other hand, when feature amount extraction is used for subject recognition, it is described in, for example, Japanese Patent Application Laid-Open No. 2001-16573. However, eyebrow, eye, nose, lip end points, face contour points, head apexes and jaws It is determined whether or not a person is included in the captured image by obtaining a feature point such as a lower end point of the image as a feature amount. For example, when a face is detected from a photographed image, the determination unit 22 also detects hair, neck, clothes, and the like located near the face.

  Histogram analysis is a technique for analyzing a color tendency using a hue histogram created from the hue of each pixel included in a captured image. When the created hue histogram has a high-frequency distribution with respect to blue, the determination unit 22 specifies in which position in the captured image the pixel region that is blue appears. For example, when the area of the pixel that becomes blue is located in the upper part of the photographed image, it is determined that the subject indicated by the area of the pixel that becomes blue is empty. When creating this hue histogram, the color information of the photographed image may be color information made up of YCbCr components or color information made up of R, G, B color components.

  Texture analysis is a technique for recognizing a subject by analyzing a texture feature amount obtained from a captured image. For example, the luminance distribution of the luminance value of each pixel included in the captured image, the spatial frequency, and the like are used as the texture feature amount. The determination unit 22 can recognize a pattern of clothes, a meadow, a tree, and the like by performing texture analysis on the captured image.

  In addition, as an example of a method for determining a subject, an example in which subject recognition, histogram analysis, and texture analysis are combined has been described. However, it is also possible to determine a subject based only on a spatial frequency analysis result obtained from a captured image. That is, the determination unit 22 may determine a region having the same spatial frequency as the subject from the spatial frequency obtained from the captured image.

  The area dividing unit 23 divides the captured image into a plurality of subject areas for each subject determined by the determining unit 22. As shown in FIG. 2A, when the subjects included in the captured image are a person, a building, the sky, and the ground, the subjects are discriminated by the discriminating unit 22. Further, in the case of a person, hair, face, neck, clothes, etc. are determined as subjects. In this case, the area dividing unit 23 divides the captured image into a “hair” area 31, a “face” area 32, a “neck” area 33, a “clothing” area 34, a “ground” area 35, and a “building” ”Area 36 and“ empty ”area 37 (see FIG. 2B).

  The setting unit 24 sets the strength (strength) of noise removal processing for each subject region divided by the region dividing unit 23. The intensity of the noise removal process is set based on the subject included in the subject area. The intensity of the noise removal process set for each subject area is output to the noise removal processing unit 21. As the strength of the noise removal processing, for example, there are two types of “strong” and “weak”. For example, when noise removal processing is performed on a subject area corresponding to a subject having a clear outline (hereinafter referred to as a subject area having a clear outline) such as a hair or a building, the outline may be blurred. Therefore, “weak” is set as the strength of the noise removal processing for such a subject area.

  On the other hand, in the case of a subject area that corresponds to a subject that does not include a contour, such as the sky or the sea (hereinafter referred to as a subject region that does not include a contour), there is no possibility that the contour will be blurred even if strong noise removal processing is performed. . Further, since noise is conspicuous in such a subject region, “strong” is set as the strength of the noise removal processing for such a subject region. In addition, since it is generally preferable to display a skin-colored region such as a face or a neck in an image, “strong” is set as the strength of the noise removal processing for the skin-colored subject region.

  As described above, the low-pass filter process is an example of the noise removal process executed by the noise removal processing unit 21. For example, when the strength of the noise removal process is set to “strong”, the noise removal processing unit 21 executes a low-pass filter process using a total of 25 pixels of 5 × 5. Further, when the strength of the noise removal processing is set to “weak”, the noise removal processing unit 21 executes low-pass filter processing using a total of 9 pixels of 3 × 3.

  The display unit 8 displays various images under the control of the control unit 7. Various images displayed on the display unit 8 include a through image, a captured image, an image recorded on the recording medium 11, a many image, and the like. The operation unit 9 includes a release button 25, a cross key 26, and the like. The release button 25 is operated by the user at the time of shooting. The cross key 26 is operated on the above menu image or the like. The state of the release button 25 and the cross key 26 is detected by the control unit 7, and a sequence based on the detected button state is executed. The recording I / F unit 10 includes a connector for connecting the recording medium 11. When the recording I / F unit 10 and the recording medium 11 are connected, data writing / reading is executed on the recording medium 11. The bus 12 connects the buffer memory 5, the image processing unit 6, the control unit 7, the display unit 8, and the recording I / F unit 10, thereby executing input / output of data and signals.

  FIG. 3 is a flowchart showing a flow at the time of imaging in the electronic camera 1. The flowchart shown in FIG. 3 is executed when the electronic camera 1 is in a shooting standby state.

  Step S101 is processing for starting the shooting mode. When this shooting mode is started, the control unit 7 performs thinning control of the image sensor 3 and acquires a through image at predetermined time intervals. The acquired through images are sequentially displayed on the display unit 8.

  Step S102 is processing for determining whether or not the release button 25 has been operated. When the release button 25 is operated, the operation signal is input to the control unit 7. The control unit 7 determines whether or not the release button 25 is operated based on whether or not this operation signal is input. For example, when it is determined that the release button 25 has been operated by the control unit 7 (when the determination in step S102 is YES), the process proceeds to step S103. On the other hand, when it is determined that the release button 25 has not been operated (when the determination in step S102 is NO), a standby state is maintained until the release button 25 is operated.

  Step S103 is processing for acquiring a captured image. The control unit 7 controls the image sensor 3 and acquires a captured image.

  Step S104 is processing to determine the subject. The determination unit 22 uses the captured image acquired in step S103 to determine a subject included in the captured image by subject recognition, histogram analysis, texture analysis, and the like described above. For example, the determination unit 22 determines hair, face, neck, clothes, ground, building, and sky as subjects from the captured image of FIG.

  Step S105 is processing for dividing the captured image into a plurality of subject areas for each subject. The region dividing unit 23 divides the captured image into a plurality of subject regions for each subject determined by the determining unit 22 in step S104. FIG. 2B is a diagram illustrating an example of a state in which the captured image is divided for each subject in FIG.

  Step S106 is a process of setting the strength of the noise removal process for each divided subject area. The setting unit 24 sets the strength of the noise removal process for each subject region divided by the region dividing unit 23 in step S105. For example, the setting unit 24 sets “weak” as the strength of the noise removal processing for the subject region having a clear outline, that is, the region 31, the region 34, and the region 36 illustrated in FIG. The setting unit 24 sets “strong” as the strength of the noise removal processing for the subject region that does not include the outline, that is, the region 32, the region 33, the region 35, and the region 37 illustrated in FIG. . The set noise removal processing intensity is output to the noise removal processing unit 21 after being associated with each subject area.

  Step S107 is image processing on the captured image. The image processing unit 6 performs various image processes on the captured image. During this image processing, noise removal processing by the noise removal processing unit 21 is executed. The noise removal processing unit 21 performs noise removal processing using the intensity of noise removal processing set by the setting unit 24 on the captured image that has been subjected to color interpolation processing. That is, for each divided subject area, a noise removal process is executed according to the subject included in the subject area. For example, a strong noise removal process is performed on a subject area that does not include a contour, and a weak noise removal process is performed on a subject area that includes a contour. Note that step S108 is a process of recording the captured image on which the image processing has been performed on the recording medium 11.

  As described above, the electronic camera 1 according to the present embodiment divides a captured image into a plurality of subject areas for each determined subject, and then performs noise removal processing using intensity according to the subject included in the subject region. Execute. Therefore, according to the electronic camera 1 of the present embodiment, it is possible to execute an appropriate noise removal process corresponding to the subject included in the captured image.

  In addition, the electronic camera 1 according to the present embodiment does not need to generate areas each including a part of different subjects that are generated when areas are set in advance. Therefore, according to the electronic camera 1 of the present embodiment, problems such as blurring of the contour of the subject or inability to effectively remove noise occur when noise removal processing is performed on such an area. Therefore, it is possible to execute an appropriate noise removal process.

  Furthermore, when a region is set in advance, a noise removal filter larger than the region cannot be used. For this reason, noise removal processing cannot be performed uniformly over a wide area that does not contain contours, and noise may be conspicuous after the noise removal processing, or noise removal processing may not work effectively. There is sex. However, according to the electronic camera 1 of the present embodiment, since such a problem does not occur, an image with a large effect of noise removal processing can be obtained.

(Supplement of embodiment)
(1) In the present embodiment, an example of an electronic camera is shown, but the present invention is not limited to this. For example, the procedure for discriminating the subject in the present embodiment (step S104 in FIG. 3), the procedure for dividing the captured image into a plurality of subject areas for each subject (step S105 in FIG. 3), It is also possible to provide the functions shown in the procedure for setting the intensity of the noise removal processing for the subject area (step S106 in FIG. 3) and the procedure for executing the noise removal processing (step S107 in FIG. 3). In addition to the image processing apparatus, an image processing program may be used. In this case, it is possible to store the image processing program in a recording medium in advance and execute or install the image processing program after connecting to the computer.

  (2) In this embodiment, the number of pixels used in the low-pass filter processing is changed in accordance with the strength of the noise removal processing. However, in addition to this, the number of pixels used when performing the low-pass filter processing can be changed. In addition, it is possible to change the processing content (type) of the low-pass filter processing, such as changing the intensity of the noise removal processing by changing the weighting for each pixel, or changing the number of times the low-pass filter processing is executed. .

  (3) In the present embodiment, the setting unit 24 changes the intensity of the noise removal processing based on the subject included in the subject region, but in addition to this, the setting unit 24 determines the spatial frequency obtained from the photographed image by the determination unit 22. It is also possible to set the strength of noise removal processing based on the analysis result or the spatial frequency obtained for each subject area. In other words, the setting unit 24 sets “weak” as the strength of the noise removal process for a subject area with many high frequency components, and the setting unit 24 sets the noise removal process for a subject area with many low frequency components. Set “Strong” as the strength of.

  (4) In the present embodiment, two examples of “strong” and “weak” are given as the strength of the noise removal processing, but three or more types may be used.

  In addition, in a memory (not shown), data in which a representative subject is associated with a parameter relating to low-pass filter processing applied to the subject is recorded in advance, and each subject can be referred to by referring to this data. It is also possible to determine the strength of the noise removal process for. In this case, the setting unit 24 determines whether or not the corresponding parameter is recorded in the memory for each subject recognized in step S104. If the setting unit 24 determines that the parameter is recorded, the setting unit 24 reads the parameter and outputs the parameter to the noise removal processing unit 21. Next, the noise removal processing unit 21 performs noise removal processing for each subject region divided by the region division unit 23 in step S105 using the read parameters. Thereby, more appropriate noise removal processing corresponding to the subject can be performed.

  (5) In the present embodiment, an example in which a captured image subjected to image processing is recorded has been described, but the present invention is not limited to this. For example, after step S107, an image obtained by synthesizing the captured image after the noise removal process and the captured image before the noise removal process is generated. In addition, when combining, the intensity of the noise removal process can be changed by adjusting the ratio of the captured image after the noise removal process and the ratio of the captured image before the noise removal process.

  (6) If a face is detected in step S104, the determination unit 22 can also estimate whether the subject is male or female. For example, when the color near the lips is red, the determination unit 22 estimates that the subject is a woman. Then, when the subject is estimated to be female, the setting unit 24 sets stronger noise removal processing for the skin-colored subject region as compared to the case where the subject is estimated to be male. As a result, when the subject is a woman, it is possible to obtain a preferable image in which wrinkles and rough skin are not noticeable.

  (7) In step S104, the determination unit 22 may not be able to determine all subjects depending on the shooting conditions. In this case, the setting unit 24 collects subject areas where the subject cannot be identified, and sets the noise removal processing intensity for the collected subject areas.

  (8) In the present embodiment, the region dividing unit 23 divides the captured image into a plurality of subject regions for each subject determined by the determining unit 22, but the present invention is not limited to this. For example, as shown in Japanese Patent Application Laid-Open No. 2005-12307, the region dividing unit 23 creates a distance image based on the values of a plurality of distance measuring points obtained by an external light passive distance measuring device. It is also possible to divide a photographed image into subject areas included in the same subject distance using an image. In this case, the setting unit 24 obtains a spatial frequency for each of the divided subject areas, and sets the strength of noise removal processing corresponding to the obtained spatial frequency.

  In addition, the determination unit 22 is made to learn a subject in advance using a plurality of learning samples, and the region dividing unit 23 may divide the captured image into a plurality of subject regions for each subject determined based on the learning result. good. The area dividing unit 23 may use the luminance information obtained by the AE process, or may divide the captured image into a plurality of object areas for each subject by using these methods in combination.

  (9) In this embodiment, noise removal processing is performed on a captured image acquired at the time of photographing. In addition to this, after noise removal processing is performed on an image recorded on the recording medium 11 at the time of reproduction, It is also possible to display the image after the noise removal processing on the display unit 8.

  (10) In the present embodiment, the determination unit 22 determines the subject from the captured image acquired in step S103, but it is also possible to determine the subject from the through image acquired in step S101. .

  (11) In the present embodiment, an example in which the noise removal processing unit 21 automatically performs noise removal processing on the captured image acquired in step S103 has been described, but the present invention is not limited thereto. For example, the control unit 7 sets in advance whether or not to perform noise removal processing, or displays a captured image on the display unit 8 after step S103 and performs noise removal processing from the user via the cross key 26. It is also possible to accept selection of whether or not.

It is a block diagram which shows the structure of the electronic camera 1 in embodiment of this invention. It is a figure which shows the example of the state which divided | segmented the picked-up image for every picked-up image and to-be-photographed object. 3 is a flowchart showing a flow at the time of imaging in the electronic camera 1.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Electronic camera, 7 ... Control part, 21 ... Noise removal process part, 22 ... Discrimination part, 23 ... Area division part, 24 ... Setting part

Claims (8)

An area dividing unit that discriminates each subject in the image based on either the input image or the shooting information obtained at the time of acquiring the image, and divides the image into a plurality of subject areas each including the subject When,
A setting unit that sets at least one of the strength and type of noise removal processing for the subject region based on the subject in the subject region for each subject region divided by the region dividing unit;
An image processing apparatus comprising: a noise removal processing unit that performs noise removal processing for each of the plurality of subject areas according to the setting of the setting unit. The image processing apparatus according to claim 1.
The image processing apparatus, wherein the setting unit sets at least one of a strength and a type of a noise removal process for the subject region based on a spatial frequency obtained from the subject region. The image processing apparatus according to claim 2,
The setting unit sets the strength of noise removal processing for a subject region including a large amount of low-frequency components among the plurality of subject regions more strongly than the strength of noise removal processing for a subject region including a large amount of high-frequency components. An image processing apparatus. The image processing apparatus according to claim 1, wherein
The setting unit differs in noise removal processing strength for a subject region determined as a predetermined subject from a plurality of subject regions divided by the region dividing unit from noise removal processing strength for other subject regions. An image processing apparatus characterized in that the intensity is set. The image processing apparatus according to claim 4.
The predetermined subject is empty;
The image processing apparatus characterized in that the setting unit sets the strength of noise removal processing for a subject area determined to be empty as the subject is stronger than the strength of noise removal processing for the other subject regions. The image processing apparatus according to claim 4.
The predetermined subject is skin,
The image processing apparatus characterized in that the setting unit sets the strength of noise removal processing for a subject region in which the subject is determined to be skin to be stronger than the strength of noise removal processing for the other subject region. The image processing apparatus according to any one of claims 1 to 6,
An imaging unit that outputs an image signal that is a source of the image by photoelectrically converting the captured subject luminous flux;
An electronic camera characterized by comprising: A region dividing step of determining a subject region based on either an input image or photographing information obtained at the time of acquiring the image, and dividing the image into the subject regions;
A setting step for setting at least one of the strength and type of noise removal processing for the subject region based on the subject in the subject region for each subject region divided by the region dividing unit;
An image processing program capable of causing a computer to execute a noise removal processing step of performing noise removal processing for each of the plurality of subject areas according to the setting of the setting step.

Images