JP2015159357A - Image processing system, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To exactly reproduce background blurring by suppressing a circuit scale and a processing period even when the number of subjects is large when varying the degree of blurring depending on a subject distance.SOLUTION: When the number of subjects is large and a high luminance area is not included, a simple filter is used to perform filtering. At this time, when the degree of blurring is small, a simple filter composed of a single filter is used; when the degree of blurring is intermediate, a simple filter composed of five same filters are used; and when the degree of blurring is large, a simple filter composed of 13 same filters are used.

Description

  In particular, the present invention relates to an image processing apparatus, an image processing method, and a program suitable for use in performing blurring processing.

  Conventionally, taking a picture with a compact digital camera focuses on a relatively wide distance range compared to taking a picture with a single-lens reflex camera. I can't. For this reason, in a conventional compact digital camera, in order to obtain a large blurring effect, a blur equivalent to a single-lens reflex camera is added to the captured image by digital filter processing.

  For example, Patent Document 1 discloses a technique in which a reduced image is subjected to filter processing of a small size and then enlarged to the original size, and large blur is added to the image by filter processing of a small size. For example, Patent Document 2 discloses a technique for detecting a main subject region and a background region from two images having different focus conditions and forming an image with an effect of blurring the background according to the result. Yes.

JP 2004-102904 A Japanese Patent Laid-Open No. 10-233919

  However, in the conventional techniques disclosed in Patent Documents 1 and 2 described above, since the filter size is small, it is impossible to faithfully reproduce background blur as in a single-lens reflex camera. For example, with a single-lens reflex camera, it is possible to capture a background point light source with a large rounded blur, but the small size filter processing disclosed in Patent Documents 1 and 2 provides sufficient details of the shape of the round blur. I can't express it. In addition, when the degree of blur is changed depending on the subject distance, there is a problem that the overall processing time increases as the types of subject distances increase.

  The present invention has been made in view of the above-described problems, and it is an object of the present invention to faithfully reproduce background blur while suppressing the circuit scale and processing time even when the number of subjects is large when changing the degree of blur according to the subject distance. Yes.

  The image processing apparatus according to the present invention includes an input unit that inputs a captured image including at least two subjects having different distances to the focal point, a distance to each subject in the captured image, and a distance to the in-focus position. Acquisition means for acquiring information, and first processing means for performing different filter processing on each subject based on the distance to each subject and the distance to the in-focus position acquired by the acquisition means. The first processing means performs filter processing on each subject by the number of times based on the distance to each subject and the distance to the in-focus position, and synthesizes an image generated for each filter processing. It is characterized by doing.

  According to the present invention, when changing the degree of blur according to the subject distance, it is possible to faithfully reproduce the background blur while suppressing the circuit scale and processing time even when the number of subjects is large.

It is a block diagram which shows the structural example of the digital camera which concerns on embodiment. It is a figure for demonstrating a part of structure contained in an image process part. It is a figure for demonstrating the filter coefficient of the two-dimensional filter process applied to a picked-up image. It is a figure for demonstrating a mode that the two-dimensional filter of FIG. 3 is applied to a picked-up image. It is the figure which showed the positional relationship of a digital camera and the to-be-photographed object. It is a figure which shows the picked-up image image | photographed with the example of the positional relationship shown in FIG. It is a figure which shows the example of the distance map of the to-be-photographed object in a picked-up image. It is a flowchart which shows an example of a series of processing procedures which perform the blurring process in embodiment. It is a figure which shows an example of a brightness | luminance histogram. It is a flowchart which shows an example of the detailed procedure of the blurring process in embodiment. It is a figure which shows an example of the selection conditions of the filter for every blurring mode. It is a figure which shows an example of the conditions which determine the blurring degree. It is a figure which shows an example of the filter applied with the kind of filter, and the blurring degree. It is a figure which shows an example of the shape of a filter. It is a figure for demonstrating the detailed structure of a simple filter. It is a flowchart which shows an example of the detailed procedure of the filter process by a simple filter.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this embodiment, an example applied to a digital camera as an image processing apparatus will be described.

FIG. 1 is a block diagram illustrating a configuration example of a digital camera 100 according to the present embodiment.
In FIG. 1, a photographing lens 103 is a lens including a focus lens, and a shutter 101 has an aperture function. The imaging unit 22 is configured by a CCD, a CMOS element, or the like that converts an optical image into an electrical signal. The A / D converter 23 is a circuit for converting an analog signal into a digital signal, and is used for converting the analog signal output from the imaging unit 22 into a digital signal. The barrier 102 covers the imaging unit including the imaging lens 103 of the digital camera 100, thereby preventing the imaging system including the imaging lens 103, the shutter 101, and the imaging unit 22 from becoming dirty or damaged.

  The image processing unit 24 performs resize processing and color conversion processing such as predetermined pixel interpolation and reduction on the data output from the A / D converter 23 or the data output from the memory control unit 15. The image processing unit 24 performs predetermined calculation processing using image data obtained by imaging, and the system control unit 50 performs exposure control and distance measurement control based on the obtained calculation result. Thereby, AF (autofocus) processing, AE (automatic exposure) processing, and EF (flash pre-emission) processing of the TTL (through-the-lens) method are performed. In the AF processing by the system control unit 50, focus control for the main subject is performed, and the focus distance is obtained as the main subject distance. The image processing unit 24 further performs predetermined calculation processing using image data obtained by imaging, and also performs TTL AWB (auto white balance) processing based on the obtained calculation result. Further, the image processing unit 24 forms an image with a desired blur effect applied to image data obtained by imaging.

  The distance map generation unit 104 generates a distance map of the subject from, for example, two images with different focus conditions. For example, an edge component of two images, that is, a high frequency component and a medium frequency component are extracted, and a subject distance is extracted from the difference value by taking a difference value of these absolute value components. By increasing the types of focus conditions and extracting subject distance information from a plurality of photographed images, the resolution and accuracy of the subject distance can be increased. Thereby, a distance map for the captured image is generated.

  Output data output from the A / D converter 23 is directly written into the memory 32 via the image processing unit 24 and the memory control unit 15 or via the memory control unit 15. The memory 32 stores image data obtained by the imaging unit 22 and converted into digital data by the A / D converter 23 and image data to be displayed on the display unit 28. The memory 32 has a storage capacity sufficient to store a predetermined number of still images, a moving image for a predetermined time, and audio data. The memory 32 also serves as an image display memory (video memory).

  The D / A converter 13 converts the image display data stored in the memory 32 into an analog signal and supplies the analog signal to the display unit 28. Thus, the display image data written in the memory 32 is displayed on the display unit 28 via the D / A converter 13. The display unit 28 performs display according to the analog signal output from the D / A converter 13 on a display such as an LCD.

  The nonvolatile memory 56 is an electrically erasable / recordable memory, and for example, an EEPROM or the like is used. The nonvolatile memory 56 stores constants and programs for operating the system control unit 50. The program stored in the nonvolatile memory 56 is a program for executing various flowcharts described later in the present embodiment.

  The system control unit 50 controls the entire digital camera 100. By executing the program stored in the nonvolatile memory 56 described above, each process of the present embodiment to be described later is realized. The system control unit 50 also performs display control by controlling the memory 32, the D / A converter 13, the display unit 28, and the like. For example, a RAM is used as the system memory 52, and constants and variables for operation of the system control unit 50, a program read from the nonvolatile memory 56, and the like are expanded in the system memory 52.

  The mode switch 60, the shutter button 61, the first shutter switch 62, the second shutter switch 64, and the operation unit 70 are operation means for inputting various operation instructions to the system control unit 50. The mode switch 60 switches the operation mode of the system control unit 50 to any one of a still image recording mode, a moving image recording mode, a reproduction mode, and the like.

  The shutter button 61 is an operation means for issuing a shooting instruction, and includes a first shutter switch 62 and a second shutter switch. The first shutter switch 62 is turned on when the shutter button 61 provided in the digital camera 100 is being operated, so-called half-press (shooting preparation instruction), and generates a first shutter switch signal SW1. When the first shutter switch signal SW1 is input to the system control unit 50, operations such as AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and EF (flash pre-flash) processing are performed. Be started.

  The second shutter switch 64 is turned ON when the operation of the shutter button 61 is completed, that is, when it is fully pressed (shooting instruction), and generates a second shutter switch signal SW2. When the second shutter switch signal SW <b> 2 is input to the system control unit 50, a series of shooting processing operations from reading a signal from the imaging unit 22 to writing image data on the recording medium 200 is started.

  Each operation member of the operation unit 70 is appropriately assigned a function for each scene by selecting and operating various function icons displayed on the display unit 28, and functions as various function buttons. Examples of the function buttons include an end button, a return button, an image advance button, a jump button, a narrowing button, and an attribute change button. For example, when a menu button is pressed, various setting menu screens are displayed on the display unit 28. The user can make various settings intuitively using the menu screen displayed on the display unit 28, the four-way button, and the SET button.

  The power control unit 80 includes a battery detection circuit, a DC-DC converter, a switch circuit that switches a block to be energized, and the like, and detects whether or not a battery is installed, the type of battery, and the remaining battery level. Further, the power control unit 80 controls the DC-DC converter based on the detection result and an instruction from the system control unit 50, and supplies a necessary voltage to each unit including the recording medium 200 for a necessary period.

  The power supply unit 30 includes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like. The interface 18 is an interface for connecting to a recording medium 200 such as a memory card or a hard disk. The recording medium 200 is a recording medium such as a memory card, and includes a semiconductor memory or a magnetic disk. The power switch 72 is a switch for switching between power-on and power-off.

  Next, a detailed configuration and operation of the image processing unit 24 for performing highly reproducible filter processing will be described with reference to FIGS.

  FIG. 3 is a diagram for explaining the filter coefficient of the two-dimensional filter process applied to the captured image. As shown in FIG. 3, the shape of the filter is a cylindrical filter shape, and by applying this filter processing to the captured image, it is possible to express a circular blur like a single-lens reflex camera. The size of the filter is (47 taps) × (47 taps) in the cylindrical portion. .., f36 are two-dimensional filter coefficients having a size of (9 taps) × (9 taps), and the maximum (54 taps) × (54 taps) is 2 in 6 vertical blocks × 6 horizontal blocks. Dimensional filter coefficients can be expressed.

  FIG. 4 is a diagram for explaining a state in which the two-dimensional filter of FIG. 3 is applied to a captured image. In FIG. 4, an area 401 is an imaged input image area. A region 402 is a region of an output image after applying a (54 tap) × (54 tap) two-dimensional filter.

FIG. 2 is a diagram for explaining a part of the configuration included in the image processing unit 24.
A two-dimensional filter circuit 201 shown in FIG. 2A is a circuit that performs a two-dimensional filter process of (9 taps) × (9 taps). A signal 203 is a signal of a two-dimensional filter coefficient input to the two-dimensional filter circuit 201, and a signal 204 is an input image input to the two-dimensional filter circuit 201. A signal 205 output from the two-dimensional filter circuit 201 is an image signal to which a two-dimensional filter is applied.

  The two-dimensional filter circuit 201 reads a desired region of the input image and performs a two-dimensional convolution process. First, the two-dimensional filter circuit 201 applies the two-dimensional filter coefficient f1 shown in FIG. 3 to the input image. At that time, as shown in FIG. 4, a first intermediate image is generated by partially reading out a region 403 in the input image and applying a filter process. Similarly, a second intermediate image is generated by filtering the region 404 shown in FIG. 4 with the two-dimensional filter coefficient f2 shown in FIG.

  FIG. 2B shows the image composition circuit 206. In this embodiment, an example in which an image is synthesized by addition processing will be described. First, the image composition circuit 206 inputs the first intermediate image and the second intermediate image as the input signals 207 and 208, generates a first composite image, and outputs it as an output signal 209. Next, in the same way as the processing of the two-dimensional filter coefficient f2 shown in FIG. 3, in the two-dimensional filter circuit 201, the region of the input image is shifted and the filtering process using the two-dimensional filter coefficient f3 shown in FIG. An intermediate image is generated. Then, the image synthesis circuit 206 generates a second synthesized image by adding and synthesizing the first synthesized image and the third intermediate image. Thereafter, the same processing is performed on the two-dimensional filter coefficients f4 to f36 shown in FIG. 3 to finally generate a 35th composite image.

  In the present embodiment, the two-dimensional filter coefficients f1 to f36 shown in FIG. 3 are stored in advance in the nonvolatile memory 56, developed on the memory 32, and input to the two-dimensional filter circuit 201. However, the present invention is not limited to this, and the system control unit 50 generates (54 taps) × (54 taps) two-dimensional filter coefficients shown in FIG. You may divide | segment into the two-dimensional filter coefficient f1-f36. Thereby, the data of the two-dimensional filter coefficient stored in the nonvolatile memory 56 may be reduced.

  FIG. 2C shows an image division circuit 210. The image division circuit 210 receives the 35th composite image and the normalization coefficient as the input signals 211 and 213, divides the 35th composite image by the normalization coefficient, and finally outputs the final blur as the processed output signal 212. Output an image. Since the sum of the coefficient values of the two-dimensional filter coefficients shown in FIG. 3 is 1653, “1653” is input as the normalization coefficient.

  Next, a distance map generated by the distance map generation unit 104 will be described with reference to FIGS.

  FIG. 5 is a diagram showing the positional relationship between the digital camera according to the present embodiment and the subject to be photographed. FIG. 5A shows the positional relationship between the first subject 504 and the background 506, and shows the case where there are two types of subject distances (hereinafter, subject distance D) imaged by the digital camera 501. FIG. Yes. In FIG. 5A, angle of view 502 and 503 indicate a range captured by the digital camera 501, and is focused at a distance 505 of the first subject 504. Hereinafter, the focus distance is DF, and in the example shown in FIG. 5A, DF = 200.

  FIG. 5B shows the positional relationship among the first subject 514, the second subject 515, the third subject 516, and the background 520, and the distance of the subject imaged by the digital camera 511 is as follows. Four cases are shown. In FIG. 5B, the angle of view 512, 513 indicates a range imaged by the digital camera 511. Further, the first subject 514, the second subject 515, the third subject 516, and the background 520 are located at distances 517, 518, 519, and 520, respectively, and the distances are 200, 100, 300, and 1000, respectively. It is. In the example shown in FIG. 5B, the first subject 514 is in focus, and the focus distance is DF = 200.

  FIG. 6 is a diagram illustrating a captured image captured in the example of the positional relationship illustrated in FIG. FIG. 6A shows a photographed image in the positional relationship shown in FIG. 5A, and the photographed image 601 photographed by the digital camera 501 includes the first subject 602 and the background 603. Yes. FIG. 6B shows a photographed image in the positional relationship shown in FIG. 5B. In the photographed image 611 photographed by the digital camera 511, the first subject 612, the second subject 613, A third subject 614 and a background 615 are included.

  FIG. 7 is a diagram illustrating an example of a distance map of a subject in the captured image illustrated in FIG. The distance map generation unit 104 generates the distance map shown in FIG. 7 from the captured image as shown in FIG. FIG. 7A shows a distance map 701 in the photographed image shown in FIG. 6A, and shows that the distance at the first subject 702 is 200 and the distance at the background 703 is 1000. ing.

  On the other hand, FIG. 7B shows a distance map 711 in the captured image shown in FIG. In FIG. 7B, the distance at the first subject 712 is 200, the distance at the second subject 713 is 100, the distance at the third subject 714 is 300, and the background 715 The distance is 1000.

FIG. 8 is a flowchart illustrating an example of a series of processing procedures for performing the blurring process in the present embodiment.
First, in step S801, the distance map generation unit 104 acquires a distance map as shown in FIG. 7, for example. In step S802, the system control unit 50 acquires the number ND of distance types. For example, in the example shown in FIG. 7A, ND = 2, and in the example shown in FIG. 7B, ND = 4.

  Next, in step S803, the system control unit 50 acquires the focusing distance DF. In the case of the example shown in FIGS. 7A and 7B, DF = 200. In step S804, the system control unit 50 acquires high-luminance distribution information. For example, the captured images 601 and 611 are divided into M M × N block areas in the horizontal direction and N M × N block areas in the vertical direction, and a luminance histogram as shown in FIG. 9 is acquired for each block area. In the case of the example shown in FIG. 9A, the luminance value of the peak of the luminance histogram is Yp1, and in the example of FIG. 9B, the luminance value of the peak of the luminance histogram is Yp2. Thus, in the process of step S804, the peak luminance value YpMN of the M × N block region is acquired.

  The next step S805 is a blurring process, and the image processing unit 24 uses the distance map, the distance type number ND, the focusing distance DF, and the high-intensity distribution information acquired up to step S804 to determine each subject distance. Appropriate blurring is performed.

FIG. 10 is a flowchart illustrating an example of a detailed processing procedure of the blurring process in step S805 performed by the image processing unit 24. Hereinafter, the procedure of the blurring process will be described with reference to FIG.
In step S1001, the following loop processing of steps S1002 to S1012 is started for all subject distances D in the captured image. First, in step S1002, the number ND of distance types acquired in step S802 of FIG. 8 is compared with the threshold value ND0. As a result of the comparison, if the distance type number ND is larger than the threshold value ND0, the process proceeds to step S1003. If the distance type number ND is less than or equal to the threshold value ND0 (predetermined value), the process proceeds to step 1009.

  In step S1003, it is determined whether or not the blur filter satisfies the simple filter condition from the relationship between the focusing distance DF and the subject distance D acquired in step S803 in FIG. Whether or not to use a simple filter as a blurring filter is determined with reference to a table shown in FIG. 11, for example. As a result of the determination, if a simple filter with low reproducibility is used, the process proceeds to step 1004. If a filter with high reproducibility is used, the process proceeds to step S1009.

  Here, in the table shown in FIG. 11, the filter to be selected differs depending on whether the in-focus distance DF is larger than the subject distance D for each blur mode. That is, the blur filter is defined as a filter with high reproducibility or a simple filter with low reproducibility based on the context of the subject. In FIG. 11, “◯” indicates that the filter A (high quality filter) with high reproducibility is used, and “X” indicates that the filter B (simple filter) with low reproducibility is used.

  In the case of the example shown in FIG. 11, in the blur mode 1, the filter A having high reproducibility is used regardless of the subject distance D. In the blur mode 2, the filter A is used when the target subject is in front of the subject located at the focusing distance DF, and the filter B is used when the subject is behind. On the other hand, in the blur mode 3, the filter B is used when the target subject is in front of the subject located at the focusing distance DF, and the filter A is used when it is behind. In the blur mode 4, the filter B having low reproducibility is used regardless of the subject distance D. Note that these blur modes 1 to 4 may be selected by an operator from a menu screen displayed on the digital camera 100, or whether or not continuous shooting is performed, whether or not a shooting mode with a combination of a plurality of images is taken. It may be selected depending on the shooting conditions.

  Next, in step S1004, the high brightness of the area of the subject distance D from the area of the subject distance D obtained from the distance map obtained in step S801 of FIG. 8 and the high brightness distribution information obtained in step S804 of FIG. Get information. In step S1005, it is determined whether or not there is a high brightness area in the area of subject distance D from the high brightness information of the area of subject distance D acquired in step S1004. That is, it is determined whether or not the luminance value in the area of the subject distance D is greater than or equal to a predetermined value among the luminance values acquired in step S804. If the result of this determination is that there is no high brightness area, processing proceeds to step S1006, and if there is a high brightness area, processing proceeds to step S1009.

In the processing from step S1006 to S1008 described below, filter processing using a simple filter is performed.
First, in step S1006, the blurring degree is determined from the focusing distance DF and the subject distance D. Hereinafter, the method for determining the degree of blur will be described using the table shown in FIG. 12 as an example. First, the relative position DD from the in-focus position is calculated by the following equation (1).
(Relative position DD from in-focus position) = (Subject distance D) − (In-focus distance DF) Equation (1)

  Next, it is determined in the table shown in FIG. 12 which condition the relative position DD corresponds to, and the degree of blur is determined. Here, D1, D2, D3, and D4 are threshold values necessary for determining the degree of blurring with respect to the relative position DD, and are determined by a virtual aperture value. Further, the front depth of field and the rear depth of field are also determined by a virtual aperture value, optical lens characteristics, and the like. Note that the virtual aperture value is designated by the operator of the digital camera 100 using a dial, a button, a touch panel, or the like.

  In the example shown in FIG. 12, when the relative position DD is smaller than the threshold value D1, the degree of blurring is “large”. When the relative position DD is equal to or greater than the threshold value D1 and smaller than the threshold value D2, the blurring degree is “medium”. When the relative position DD is equal to or greater than the threshold value D2 and is closer to the front depth of field, the degree of blurring is “small”. When the relative position DD is between the front depth of field and the rear depth of field, the blurring degree is “none” at the in-focus position. When the relative position DD is behind the rear depth of field and is equal to or less than the threshold value D3, the degree of blurring is “small”. When the relative position DD is greater than the threshold value D3 and less than or equal to the threshold value D4, the blurring degree is “medium”. When the relative position DD is larger than the threshold value D4, the blurring degree is “large”.

  In the present embodiment, the case of four stages including “None” is described as an example of the blurring degree. However, by providing finer the blurring degree and threshold value, more stages of blurring degree can be obtained with respect to the relative position DD. It may be provided.

  Next, in step S1007, a simple filter for subject distance D is acquired. Here, the types of filters are managed in a table as shown in FIG. According to the blurring degree determined in step S1006, if the blurring degree is “small”, the filter 1B is acquired, if it is “medium”, the filter 2B is acquired, and if it is “large”, the filter 3B is acquired. Also, simplified filters 1404 to 1406 shown in FIG. 14B represent the shapes of the filter 1B, the filter 2B, and the filter 3B, respectively.

  In step S1008, filter processing is performed using the simple filter acquired in step S1007. Details of this processing will be described later.

  On the other hand, in the processing from step S1009 to S1011, filter processing using a high-quality filter with high reproducibility of a blurred image is performed. First, in step S1009, the blurring degree is determined from the focusing distance DF and the subject distance D. The method for determining the degree of blur is the same as in step S1006.

  Next, in step S1010, a high-grade filter for subject distance D is acquired. The type of filter is managed in a table as shown in FIG. According to the blurring degree determined in step S1009, if the blurring degree is “small”, the filter 1A is acquired, if it is “medium”, the filter 2A is acquired, and if it is “large”, the filter 3A is acquired. Further, the high-level filters 1401 to 1403 shown in FIG. 14A represent the shapes of the filter 1A, the filter 2A, and the filter 3A, respectively. is there.

  Next, in step S1011, filter processing is performed using the high-level filter acquired in step 1010. Filter processing by the high-level filter is performed by the same processing as the procedure described with reference to FIGS.

  Finally, in step S1012, the result of the filtering process and the intermediate image subjected to the previous blurring process are combined to generate a new intermediate image. As described above, when the loop processing is performed for all the subject distances D, the processing ends.

  Next, filter configurations of the simple filter 1B, filter 2B, and filter 3B will be described. FIG. 15 is a diagram for describing a configuration example of the filter 1B, the filter 2B, and the filter 3B. As shown in FIG. 15, the filter 1B is expressed by a configuration of a filter B0 (0, 0), and a single two-dimensional filter B0 is configured. A simplified filter 1407 shown in FIG. 14C represents the shape of the filter B0, is a (9 tap) × (9 tap) two-dimensional filter, and is a filter with all coefficients of 1.

  On the other hand, the filter 2B includes five two-dimensional filters B0 as shown in FIG. 15, and five filters B0 are connected in a cross like a simple filter 1405 shown in FIG. 14B. It has a two-dimensional shape. In the filter processing by the filter 2B, the filter processing result of the filter B0 is horizontally (9 × x) pixels and vertically (9 × y) according to the coordinates (x, y) with (9 taps) × (9 taps) as a reference. ) Pixel shift and addition synthesis.

  Further, as shown in FIG. 15, the filter 3 </ b> B includes 13 two-dimensional filters B <b> 0, and has a shape connected in an arrangement like a simple filter 1406 shown in FIG. 14B. ing. In the filter processing of the filter 3B, similarly to the filter 2B, the processing result of the filter B0 is shifted by (9 × x) pixels horizontally and (9 × y) pixels vertically according to the coordinates (x, y) and added and synthesized.

FIG. 16 is a flowchart illustrating an example of a detailed processing procedure of the filter processing by the simple filter in step S1008 of FIG.
In step S1601, loop processing is started for all the constituent filters (filter B0) constituting the simple filter. For example, in the case of the filter 2B shown in FIG. 15, since it is configured by five filters B0, the loop processing is performed five times. Hereinafter, a description will be given with reference to an example of performing filter processing using the filter 2B.

  First, in step S1602, it is determined whether image data processed by the filter to be processed already exists. As a result of this determination, if it already exists, the process proceeds to step S1605, and if it does not exist, the process proceeds to step S1603.

  In step S1603, coefficient information of a filter to be processed is acquired. For example, information on the two-dimensional filter coefficient f1 as shown in FIG. Next, in step S1604, a filter process using the configured filter B0 is performed. In step S1605, the output image F that is the filter processing result is temporarily stored and managed.

  On the other hand, in step S1606, since there is a filter processing result, an output image F that is an output filter processing result is acquired.

  In step S1607, the shift amount (x, y) of the filter to be processed is acquired. This shift amount corresponds to (x, y) shown in each filter B0 in FIG.

  Next, in step S1608, the output image F acquired in step S1605 or S1606 is shifted according to the shift amount (x, y) acquired in step S1607, and a synthesis process is performed on the intermediate image that is the processing result so far. To generate a new intermediate image. When the configured filter is (9 taps) × (9 taps), the shift amount (x, y) is (9 × x) pixels horizontally and (9 × y) pixels vertically, and the combined result is can get. For example, in the case of the filter B0 (1, 2), the processing result of the filter B0 is shifted horizontally (9 × 1) pixels and vertically shifted (9 × 2) pixels, and the combined result is output as an intermediate image. As described above, when the processing for the number of filters is completed for all the constituent filters, the loop processing is ended.

  Note that when the filter 1B is acquired in step S1007 in FIG. 10, since only one filter B0 is provided, the result of the filter processing in step S1605 is output as it is. Therefore, in this case, the processes in steps S1607 and S1608 are not performed.

  Further, if there is a filter processing result that has already been output as a result of the determination in step S1602, it is not necessary to perform the actual filter processing. Therefore, it is only necessary to shift the output image F and perform synthesis processing with the intermediate image. As a result, for example, the processing time of the filter 2B that is a simple filter with respect to the filter 2A that is a high-grade filter and the filter 3B that is a simple filter with respect to the filter 3A that is a high-quality filter is greatly reduced. it can.

  In the flowchart of FIG. 15, the example in which the filter B0 having (9 taps) × (9 taps) and all coefficients of 1 shown in FIG. 14C is used as the constituent filter has been described. However, the present invention is not limited to this. Moreover, although the example shown in FIG. 15 was demonstrated also about the combination of a structure filter, it is not limited to this. For example, by preparing several basic configuration filters and combining a plurality of types of configuration filters to configure a simple filter, the degree of freedom of the shape of the simple filter can be expanded.

  As described above, according to the present embodiment, it is possible to give a captured image a large blurring process that can be obtained by processing using a two-dimensional filter with a small size such as 9 taps. it can. That is, a large background blur like a single-lens reflex camera can be reproduced while suppressing the circuit scale of the filter processing.

  Furthermore, according to the present embodiment, when performing background blurring processing according to subject distance, focus position, and virtual aperture information, the entire processing time remains while maintaining the effect of blurring degree even when there are many types of subject distances. Can be suppressed.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

24 image processing unit 50 system control unit 104 distance map generation unit

Claims (8)

Input means for inputting a photographed image including at least two subjects having different distances to the focal point;
Acquisition means for acquiring information on the distance to each subject in the captured image and the distance to the in-focus position;
First processing means for performing different filter processing on each subject based on the distance to each subject and the distance to the in-focus position obtained by the obtaining means;
The first processing means performs filter processing on each subject by the number of times based on the distance to each subject and the distance to the in-focus position, and synthesizes an image generated for each filter processing. An image processing apparatus. Selecting means for selecting whether or not to apply the filter processing by the first processing means;
When the selection unit selects not to apply the filter processing by the first processing unit, the processed image is generated by performing filter processing with higher reproducibility than the filter processing by the first processing unit. The image processing apparatus according to claim 1, further comprising: a second processing unit that performs processing.   The image processing apparatus according to claim 2, wherein the selection unit selects not to apply the filter processing by the first processing unit when the number of subjects is equal to or less than a predetermined value.   The selection unit selects whether or not to apply the filter processing by the first processing unit according to the relationship between the distance to the subject included in the captured image and the distance to the in-focus position. The image processing apparatus according to claim 2 or 3.   5. The method according to claim 1, wherein the selection unit selects whether or not to apply the filter processing by the first processing unit according to a luminance value of a subject included in the captured image. The image processing apparatus according to item.   The image processing apparatus according to claim 2, wherein the second processing unit performs filter processing using a cylindrical filter. An input step of inputting a photographed image including at least two subjects having different distances to the focal point;
An acquisition step of acquiring information on the distance to each subject in the captured image and the distance to the in-focus position;
A processing step of performing different filter processing on each subject based on the distance to each subject and the distance to the in-focus position acquired in the acquisition step,
In the processing step, each of the subjects is subjected to filter processing by the number of times based on the distance to each subject and the distance to the in-focus position, and an image generated for each filter processing is synthesized. A featured image processing method. An input step of inputting a photographed image including at least two subjects having different distances to the focal point;
An acquisition step of acquiring information on the distance to each subject in the captured image and the distance to the in-focus position;
Causing the computer to execute a processing step of performing different filter processing on each subject based on the distance to each subject and the distance to the in-focus position obtained in the obtaining step,
In the processing step, each of the subjects is subjected to filter processing by the number of times based on the distance to each subject and the distance to the in-focus position, and an image generated for each filter processing is synthesized. A featured program.

Images