JP2003087545A - Image pickup device, image processor and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To produce an image giving such an effect obtained by photographing with a large diameter lens by using results obtained by analyzing an image and the distance or focus information of a limited part. SOLUTION: The image and the focus information of an object at a plurality of places of the image are inputted (S101 and S103), the inputted image is divided (S104 and S105), image processing is applied to each of divided images on the basis of the focus information corresponding to each of the divided images (S106), and the divided images that have been subjected to the image processing are combined into one image (S107).

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image pickup apparatus, an image processing apparatus, and an image processing method, and more specifically, an image pickup apparatus, an image processing apparatus, and an image processing apparatus for processing a picked-up image based on information obtained by distance measurement. The present invention relates to an image processing method.

[0002]

2. Description of the Related Art Conventionally, digital electronic cameras and the like, which measure the distance to a subject and perform autofocus, have been widely used. Recently, an autofocus camera having a plurality of distance measuring areas has become widespread. In a camera having a plurality of distance measuring areas, the photographer is allowed to select one of the distance measuring areas.
When shooting, the distance measurement information of one distance measurement area is used for one captured image.

In Japanese Patent Laid-Open No. 2000-259823,
An image dividing means for arbitrarily dividing the input image and a distance measuring information adding means for adding distance measuring information measured by a distance measuring device to each of the divided images are provided. An image processing apparatus capable of blurring the background by performing desired processing on divided images,
Image processing methods and storage media have been proposed.

Further, Japanese Patent Application Laid-Open No. 10-074258 proposes an improvement of a processing method of performing digital low-pass filter processing on image data having perspective information to generate a blurred image according to the perspective information. There is.

[0005]

However, conventionally, when it is premised that there is distance information for each divided image, or when an image originally having perspective information in all parts of the screen, a background is set. This is to perform a blurring process.

However, it is difficult to add the distance information and the defocus information to all the divided images obtained by dividing the image taken by the camera. In practice, several to several tens of points are displayed on the screen. Only distance information or defocus information can be obtained.

The present invention has been made in view of the above problems, and an image captured by a dark lens and an image having a deep depth due to the small size of the image sensor are limited to the result of image analysis. The objective is to create an image having the effect as if it were taken with a large-diameter lens by using the distance information or the focus information.

[0008]

In order to achieve the above object, an image pickup apparatus of the present invention is provided with an image pickup means for converting an optical image of an object into an electric signal and picking up an image, and an image pickup screen of the image pickup means. At a plurality of corresponding locations, focus detection means for detecting focus information or distance information of the image, dividing means for dividing the image obtained by the image pickup means, and for each divided image divided by the dividing means, It has an image processing means for performing image processing based on the focus information or distance information corresponding to each divided image, and a combining means for combining the divided images image-processed by the image processing means into one image.

The image processing apparatus of the present invention further comprises an image input means for inputting an image and focus information or distance information of a subject at a plurality of positions of the image, and a dividing means for dividing the image input by the image input means. An image processing unit that performs image processing on each of the divided images divided by the dividing unit based on the focus information or distance information corresponding to each divided image, and one divided image that is image-processed by the image processing unit. And a synthesizing means for synthesizing into one image.

Further, the image processing method of the present invention comprises an image input step of inputting the image and focus information or distance information of the subject at a plurality of positions of the image, and a dividing step of dividing the image input by the image input step. , An image processing step of performing image processing on each divided image divided by the dividing step based on the focus information or distance information corresponding to each divided image, and a divided image image-processed by the image processing step. And a synthesizing step of synthesizing into one image.

[0011]

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

<First Embodiment> FIG. 1 is a block diagram showing the main configuration of a single-lens reflex type digital camera which is a photographing apparatus according to the present embodiment.

In FIG. 1, reference numerals 1 and 3 are photographing lenses. Although it is shown with two lenses for convenience of explanation, it is actually composed of many lenses. Reference numeral 2 denotes an aperture, which adjusts the amount of light incident through the taking lenses 1 and 3. Reference numeral 4 denotes a main mirror, which is attached so as to be tilted in the photographing optical path so that a subject image can be observed by the optical finder system, but is retracted from the photographing optical path when a photographing instruction is given. A sub-mirror 5 reflects the light flux passing through the main mirror 4 toward the focus detection optical system 11 and the area sensor 12 located below the camera body. The main mirror 4
Similarly to the above, when shooting is instructed, it is retracted from the shooting optical path.

Reference numeral 6 is a shutter, 7 is an image pickup device such as a CCD or MOS type, 8 is a focusing plate arranged on the planned image forming planes of the taking lenses 1 and 3, 9 is a pentaprism for changing the finder optical path, and 10 is a connection. An image lens, 11 is a focus detection optical system, and 12 is a focus detection area sensor.

Reference numeral 13 is a focus adjustment circuit for driving the taking lenses 1 and 3 to adjust the focus, 14 is a diaphragm driving circuit for driving the diaphragm 2, and 15 is a main mirror 4 and a sub-mirror 5 according to the start and end of shooting. A mirror drive circuit for retracting / inserting from the optical path, 16 for an area sensor drive circuit for driving the area sensor 12, 17 for a shutter drive circuit for driving the shutter 6, 18 for an image sensor drive circuit for driving the image sensor 7, and 19 for An A / D conversion circuit that converts an analog image pickup signal output from the image pickup device 7 into a digital signal, and a signal processing circuit 20 that performs signal processing on the image pickup signal digitally converted by the A / D conversion circuit 19.

Reference numeral 21 is a RAM (random access memory) for temporarily storing digital image signals and the like, 22 is a ROM (read only memory) in which a control program for controlling the digital camera is stored, and 23 is a digital memory. A recording medium such as a compact flash (registered trademark) (CF) for finally recording an image signal, 24 is a central processing unit (CPU) of a microcomputer, 25
Is a monitor such as an LCD for displaying the photographed digital image signal.

Reference numeral 26 is a switch input circuit for detecting a switch switch such as a release switch 28, a narrowing switch, and a mode dial for switching a photographing mode. 28 is a release switch (release SW),
Consisting of a switch with at least two contacts,
For example, it has a structure in which the state is switched in two stages according to the amount of depression, and when the depression is made halfway, the first contact (hereinafter referred to as SW1) is connected (ON),
When pressed down to the end, the second contact (hereinafter referred to as SW2) is also connected (ON). When SW1 is turned on, shooting preparation operations such as AF (automatic focus detection) and AE (auto exposure) are performed, and when SW2 is turned on, shooting operation and digital image data creation operation and recording operation are performed. .

In the digital camera having the above structure, when the photographer turns on SW1, the AF operation and the AE operation are performed. Next, the photographer turns on SW2 from that state.
Then, the diaphragm 2 is adjusted and the main mirror 4 and the sub mirror 5 are adjusted.
Is retracted from the optical path, the light flux is introduced in the direction of the image sensor 7, and the shutter 6 is opened. As a result, the subject optical image is formed on the image pickup element 7, and the image pickup element 7 generates electric charges according to the light amount of the subject optical image. Then, when the shutter 6 is closed after a certain period of time, the electric charge accumulated in the image sensor 7 is output until the shutter 6 is closed. The photographed signal as the output signal is A / D converted by the A / D conversion circuit 19, further subjected to signal processing by the signal processing circuit 20, and temporarily stored in the RAM 21 as a digital image signal. The RAM
The digital image signal temporarily stored in 21 is compressed and stored in the CF 23 as a file to complete the shooting.

In parallel with the processing after the shutter 6 is closed, the main mirror 4 and the sub mirror are returned to the optical path, and the diaphragm 2 is opened.

FIG. 2 is a diagram for explaining the display in the finder. FIG. 10A is a diagram showing a case where all the distance measuring areas are displayed. In reality, only the AAR is constantly displayed, and each distance measuring area is displayed only when necessary. During the shooting operation, the distance measurement area is not displayed until the focus state is in focus.

In the figure, AAR represents the entire distance measurement area, and the distance measurement is performed inside this area. FIG. 3B is a diagram for explaining each distance measuring area. In the figure (b), the distance measurement area is p1, p2, p3, ..., p7 from the upper left, and the left of the second row from the top starts from p8 and the rightmost is labeled p17. Describe each. In this notation, for example, the distance measuring area in the center is p23, and the fourth row from the top is p29, p3 from the left.
0 ...., p38, and the rightmost AF area in the bottom row is p
Expressed as 45.

Next, the focus detection process for obtaining a plurality of defocus information, which is one of the image information of the photographing screen in the first embodiment, will be described.

FIG. 3 is a diagram showing an optical system for performing focus detection processing at a large number of positions on the photographic screen. Each distance measuring area is configured to correspond to the distance measuring area shown in FIG. .

In FIG. 3, reference numeral 53 is a field lens arranged near the planned image forming plane of the objective lens. The field mask 54 is provided in a position near the planned image forming planes of the photographing lenses 1 and 3, and the area limited by the single wide opening 54a is the focus detectable range. The subject image that has passed through the opening 54a is projected onto the pupils of the taking lenses 1 and 3 by the field lens 53. As a practical matter, the main subject is rarely located at the edge of the shooting screen,
The area limited by the opening 54a is 8 vertically and horizontally on the shooting screen.
It may be about 0%.

A diaphragm plate 55 is placed in front of the secondary imaging lens 56 consisting of a pair of positive lenses, and a diaphragm hole 55 is provided.
Each of a and 55b regulates the luminous flux incident on each positive lens. The position of the diaphragm plate 55 is placed in a position of the exit pupils of the photographing lenses 1 and 3 in a substantially image-forming relationship by the power of the field lens 53. The secondary imaging lens 56 has an opening 54a.
Area sensor pairs 12a and 12 in which photoelectric conversion elements are regularly arranged in a two-dimensional manner in the subject image determined in
Reimage on b. The optical subject image on the area sensor pair 12a and 12b is converted into an electric signal according to the light amount and read out, and focus detection calculation is executed in the processing device.

Here, in the mode using the area sensor as shown in FIG. 3, it has to be electrically performed because there is no means for optically limiting the signal reading area. Here, focus information is detected at a large number of positions on the photographing screen by sequentially and selectively performing arithmetic processing on information of a specific area in the subject image information read in a wide range.

FIG. 4 shows the area sensor pair 12a shown in FIG.
It is a figure explaining in detail the ranging area of 12b.
As shown in the figure, the area sensor pair 12a and 12b
Corresponds to the distance measurement area shown in FIG.
1. The blocks are logically divided and arranged in five columns in the vertical direction. Each small block (which is actually formed of an area sensor or a plurality of line sensors) includes a photoelectric conversion element of 25 to 40 pixels. The area sensor drive circuit 16 is connected to the area sensor pairs 12a and 12b as described above, and area designation and storage control are performed. The area sensor drive circuit 16 is connected to a microcomputer which is a processing device. The microcomputer has a CPU 24, a ROM 22, and a RAM 21,
The focus detection process is executed according to the program stored in the ROM 22.

An outline of focus detection processing according to the subject distance detection processing program stored in the ROM 22 will be described with reference to FIG.

When the focus detection processing is called in step S200, the area sensor pairs 12a and 12b are stored in step S201, and the values obtained by A / D converting the output signals of the respective pixels are stored in the RAM 21 in step S202. . Next, in step S203, the area sensor pair 1
A pair of corresponding object image pixel signals of small blocks on 2a and 12b are fetched from the RAM, and image signal processing is performed in step S204. Then, in step S205, the correlation calculation between the image signals of the captured small block pair is performed. In this process, the phase difference detection type one detects the phase difference in the primary direction. That is, the image signals in a pair of small blocks (for example, p1a and p1b) that are part of the area sensor pair 12a and 12b shown in FIG. 4 are compared and processed in a region of these small blocks by a known method. Focus detection of the corresponding photographing area is performed. When focus detection is done,
The defocus amount and its direction are obtained as the output, and the lens extension amount of the taking lenses 1 and 3 can be determined. Therefore, in step S206, the subject distance in the distance measurement area corresponding to one small block which is a part of the area sensor pair 12a and 12b can be obtained by performing a predetermined defocus calculation.

Thus, the area sensor pair 12a and 1
2b is 45 corresponding to the distance measurement area from p1 to p45 in FIG.
Can be divided into small blocks. Step S207
Then, the subject defocus detection processing for all of these 45 small blocks is repeatedly performed, and finally, the processing shown in FIG.
As shown in, it is possible to obtain the subject defocus over a wide range of 45 points on the shooting screen.

FIG. 6 shows an example of an original image obtained by photographing a subject, which is the object of the image processing of the present invention.

A portrait with a woman in front,
On the right side, you can see the pillars lined up from the front to the back. On the left is the side of the building and behind the woman is the garden plant. This image processing will be described as an example with reference to the flowchart shown in FIG.

First, in step S101, as shown in FIG.
As described above with reference to FIG. 5, focus detection is performed for each distance measurement area shown in FIG. FIG. 7 shows the distance measurement areas superimposed on the image of FIG. At the position of the distance measuring area, distance information or focus information at the time of shooting can be known. As can be seen from this figure, only a small part of the screen has information on the distance or the focus state.

Next, in step S102, the focus state (defocus state) is divided into groups. FIG. 8 is a defocus map showing a defocus state in each distance measurement area when an image is captured. Here, the distance-measuring regions having a small difference in defocus amount are grouped together.
It should be noted that in the present embodiment, the classification is performed by the defocus amount, but when the absolute distance to the object in each ranging area is obtained, the classification may be performed for each distance.

FIG. 9 shows a grouping of the defocus map in which the distance measuring areas are associated with the symbols.

Group (g) includes p1, p8, p9, p10, p18, p
19, p20, p29, p30, p31, p39 are included, and the defocus amount is
It is -1.06 to -1.01.

The group (h) includes p2, p3, p11 and p21, and the defocus amount is -1.17 to -1.13.

Group (i) includes p6, p7, p14, p15, p1
6, p26, p27, p37 are included, and the defocus amount is -1.17 ~-
It is 1.12.

Group (j) includes p4, p5, p12, p13, p22,
p23, p24, p25, p32, p33, p34, p35, p36, p40, p41, p42, p43, p4
4 is included, and the defocus amount is -0.03 to 0.03. It can be seen that this group in which the defocus amount is 0, that is, the focus is included is the main subject. In this way, the main subject can be discriminated from the individual defocus amount, but the focused distance measuring area is recorded separately from the individual defocus amount as the image information, and the group including the focused distance measuring area is recorded. May be used as the main subject. It is also possible for the photographer to designate and process the main subject.

The group (k) includes only p45 and does not include other adjacent groups. The defocus amount is -1.05.

Group (l) includes p17, p28, and p38, and the defocus amount is -0.82 to -0.44.

Next, in step S103, an image is taken.
The input image is temporarily stored in the RAM 21.

In step S104, the image is divided.

FIG. 10 is a diagram in which regions are divided using only images. The image division and group division are performed using the image brightness information and color information. If only the brightness component is used for division, different groups will be created for the shaded area, the area illuminated by the light of the same object and the area not illuminated by the same object. Is. When division processing is performed on the original image shown in FIG.
It can be divided into 6 small images of (f).

The small image (a) is the building part and the shaded floor part on the left side of the screen, and the small image (b) is the plant part behind the person, mainly the green part, and the small image (c). ) Is the light brown pillars lined up from the front to the back on the right side of the screen, the small image (d) is the part of the clothes worn by the person (for example, blue clothes), and the small image (e) is the skin of the person's face or skin. The part (skin color) small image (f) is the part of the floor.

In step S105, the divided small images are re-divided and combined in consideration of the defocus amount of the divided small images.

FIG. 11 shows a distance measuring area overlaid on a small image divided by image information. The distance measurement areas included in each area are as follows.

In the small image (a), the distance measuring areas p1, p8, p9, p1
0, p18, p19, p20, p29, p30, p31, p39 are included. this is,
It completely matches the divided area group (g) according to the defocus information.

In the small image (b), distance measuring areas p2, p3, p6, p7,
p11, p14, p15, p16, p21, p26, p27, p37 are included. This is completely the same as the combination of the divided area group (h) and the group (i) based on the defocus information. The divided area group (h) and the group (i) in the defocus information are classified as different groups because their positions are apart from each other.

Comparing the defocus amounts of the divided group (h) and the group (i) based on the defocus information, the defocus amounts are in substantially the same range. Therefore, since the ranging area does not exist in the entire screen, the group (h) and the group (h It is considered that i) has been split. Therefore, it is not necessary to subdivide the small image (b) based on the image information.

On the contrary, when one small image divided by the screen information includes different defocus groups, another subject having the same brightness and the same color is recognized as one area. It is necessary to subdivide and combine the small images by adding defocus information.

The small image (c) includes distance measuring areas p17, p28, p38. This is exactly the same as the division group (l) according to the defocus information.

The small image (d) is a distance measuring area p41, p42, p43, p4.
Includes 4. This is a part of the division group (j) according to the defocus information.

The small image (e) is a distance measuring area p4, p5, p12, p13,
p22, p23, p24, p25, p32, p33, p34, p35, p36, p40 are included. This is a part of the division group (j) according to the defocus information.

The small image (f) includes the distance measuring area p45. This perfectly matches the division group (k) according to the defocus information.

Both the small image (d) and the small image (e) are part of the defocused divided group (j). This means that different color parts (or different brightness parts) of the same subject are divided.

Therefore, as shown in FIG. 12, the small image (d) and the small image (e) are integrated to create a small image (de) which is a new area. This perfectly matches the division group (j) by defocus.

As described above, the screen is (a),
The images are subdivided and combined into five small images (b), (c), (de), and (f).

In step S106, image processing is performed on each of the divided areas.

Since the small image (a) matches the divided area group (g) based on the defocus information, the image blurring process is performed based on the defocus amount -1.06 to -1.01.
A small image (a2) is obtained by performing blurring processing on the small image (a). The larger the focus amount (as an absolute value), the larger the blur amount.

The small image (b) corresponds to a combination of the divided area group (h) and the group (i) according to the defocus information, and the defocus amount is -1.17 to -1.13. Image blurring processing is performed based on this defocus amount. A small image (b2) is obtained by subjecting the small image (b) to blur processing. In this example, this area has the largest blur amount.

The small image (c) matches the divided area group (l) according to the defocus information, and the defocus amount is-.
It is 0.82 to -0.44. Image blurring processing is performed based on this defocus amount. A small image (c2) is obtained by performing blurring processing on the small image (c).

The small image (de) matches the divided area group (j) according to the defocus information, and the defocus amount is −
It is 0.03 to 0.03. Since this area is the main subject, the image may be used as it is without performing the blurring process, or conversely, sharpening or contour enhancement may be applied. The image area is a small image (de2) even if the processing is performed or is not performed.

The small image (f) matches the divided area group (k) according to the defocus information. The defocus amount is -1.05, and the image blurring process is performed based on this defocus amount. The area that has been blurred is (f2)
And

FIG. 13 shows images obtained by performing image processing on each of the integrated small images in this way.

In step S107, the small images (a2), (b2), (c2), (de2), which have undergone image processing,
(F2) is recombined into one image. Recomposition is performed so that the area far from the defocus amount (or distance information) is at the bottom. FIG. 14 shows an example of the combined image.

As described above, the blur amount of each of the divided areas is determined from the original image divided by the luminance and color information and the defocus information (or distance information) obtained by the focus detection, and the different blur amounts are determined. The background is blurred by recombining the divided areas that have been blurred with the amount of blur.
It is possible to obtain an image that has the effect of taking a picture of a main subject in a floating manner and opening the diaphragm with a large-diameter lens.

<Second Embodiment> Next, a second embodiment of the present invention will be described. In the second embodiment, a case where the number of distance measuring areas is smaller than that in the first embodiment (when there are few areas with defocus information or distance information) will be described.

FIG. 15 shows an example in which the distance information is detected in the seven distance measuring areas, and the distance measuring areas are superimposed on the original image shown in FIG. 6 and displayed. Note that the focus detection method is the same as the concept described with reference to FIGS. 3 to 5 except for the number of areas, and therefore description thereof will be omitted here. At the position of the distance measurement area, distance information or focus information (defocus information) can be known.
In the first embodiment, the case where the defocus amount and the direction thereof are obtained has been described, but in the second embodiment, the subject distance is used, so a brief description will be given. As is well known when focus detection is performed, the defocus amount and its direction are obtained as the output, and the lens extension amount of the photographing lens is determined. Therefore, if the subject distance in the distance measuring area is D, the moving amount of the photographing lens and the focal length are P and f, respectively, the subject distance can be approximately calculated from the relationship of D≈f2 / P. As can be seen from this figure, only a small part of the screen has distance information or focus information.

The upper distance measuring point is pp1, the middle five distance measuring points are pp2, pp3, pp4, pp5, pp6 from the left, and the lower distance measuring points are pp7.

The processing in the second embodiment of the present invention will be described below with reference to FIG. Note that FIG.
In FIG. 17, the same processes as those in FIG.

First, steps S101 to S10 described above.
The image is divided by the processing of 4.

FIG. 16 is a diagram in which seven distance measuring areas are overlapped and displayed on a diagram obtained by dividing an area using only images. The image division is performed using the image brightness information, color information, and the like, as in the first embodiment. If the original image is divided, it can be divided into six small images (a) to (f).

(A) The building portion and the shaded floor portion on the left side of the screen are divided into one small image, and the distance measuring point pp2 is included. The distance measurement result at the distance measurement point pp2 is about 8 m. (B) The part of the plant behind the person, mainly the green part, is divided into one small image, and two distance measuring points pp3 and pp6 are included. The distance to the subject is 15 m or more. (C) The light brown columns that are lined up from the front to the back on the right side of the screen are divided into one small image, and no distance measuring points are included, and there is no distance information. (D) The part of the clothes worn by the person (blue clothes) is divided into one small image, and the distance measuring point pp7 is included. The distance is about 2m. (E) A skin portion (skin color) such as a person's face is divided into one small image and includes three focus detection points pp1, pp4, pp5. The distance is about 2m. (F) Since the floor portion is divided into small images and there is no distance measurement area, there is no distance information. The small image including the focus detection points is subjected to blurring processing based on the distance of the main subject and the distance information of the divided areas. On the other hand, for a region whose distance is unknown, the blurring amount is determined by the following method.

Subsequently, in step S105, the small images are subdivided and combined. Here, the small images (d) and (e) are combined.

Next, in step S301, it is determined whether all the divided small images have defocus information or distance information. If not, the process proceeds to step S302, and the blurring amount is determined by the following method, for example.

1) Spatial frequency distribution in the area According to the distribution of the spatial frequency in the area, if the high frequency component is originally large, the blurring amount is small, and if it is small, the blurring amount is large. Further, the blurring amount is set to be the same as the region including the equivalent high frequency component as compared with the region where the defocus amount (distance information) is known. Even if the image has a small blur amount as a whole, it does not mean that the image is not blurred at all, and the slight blur amount of the original image is emphasized. For example, since the spatial frequency distribution of the small image (f) is about the same as that of the small image (a), the same image processing as that of the small image (a) is performed.

2) Position of area If the area is near the screen, the amount of blurring is increased. This is because the defocus amount is essentially the same as that of the main subject, and even an unblurred subject in the peripheral portion of the screen will not be strange as the entire image even if the blur amount is large. Small image (c)
Is around the screen, so the amount of blur is increased.

3) Inferred from the information of the contact area The blur amount is the same as the blur amount of the contacting area.

In this way, if there is defocus information or defocus information or distance information that has been determined in this way, blur processing is also performed for areas where the subject distance is unknown (step S106). .

Finally, each image subjected to the blurring process is recombined to create one image (step S107).

Even if the defocus amount (distance information) is not known for all the regions divided as described above, the blur amount of the region can be determined.

After the blurring amount is determined, each region is subjected to blurring processing and the image is reconstructed in the same manner as in the first embodiment, whereby it is possible to obtain an image in which the background is blurred and the main subject is highlighted.

[0084]

Other Embodiments Even when the present invention is applied to a system including a plurality of devices (for example, a host computer, an interface device, a camera head, etc.), an apparatus including one device (for example, a digital still camera) ,
Digital video camera).

Further, an object of the present invention is to supply a storage medium (or recording medium) recording a program code of software for realizing the functions of the above-described embodiments to a system or apparatus, and to supply a computer of the system or apparatus ( Alternatively, by the CPU or MPU) reading and executing the program code stored in the storage medium,
It goes without saying that it will be achieved. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention.
Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also based on the instruction of the program code,
An operating system (OS) running on the computer does some or all of the actual processing,
It goes without saying that the processing includes the case where the functions of the above-described embodiments are realized. Here, as the storage medium for storing the program code, for example, a floppy (registered trademark) disk, a hard disk, a ROM,
RAM, magnetic tape, non-volatile memory card, CD-
ROM, CD-R, DVD, optical disk, magneto-optical disk, MO, etc. are considered.

Further, after the program code read from the storage medium is written in the memory provided in the function expansion card inserted into the computer or the function expansion unit connected to the computer, based on the instruction of the program code, , The CPU provided in the function expansion card or the function expansion unit performs some or all of the actual processing,
It goes without saying that the processing includes the case where the functions of the above-described embodiments are realized.

When the present invention is applied to the above storage medium, the storage medium stores the program code corresponding to the flowcharts shown in FIGS. 5 and 17 described above.

[0088]

As described above, an image taken with a dark lens or an image with a large depth due to the small size of the image pickup device is used with a large aperture by using only the result of analyzing the image and the limited distance or defocus information. It is possible to create a pseudo-blurred image as if taken with a lens.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration of an imaging device according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating a distance measuring area according to the first embodiment of the present invention.

FIG. 3 is a perspective view of an optical system showing the principle of focus detection in the distance measurement area shown in FIG.

FIG. 4 is a diagram showing an area sensor for focus detection according to the first embodiment of the present invention.

FIG. 5 is a flowchart of focus detection processing according to the first embodiment of the present invention.

FIG. 6 is a diagram showing an example of an original image that is a target of image processing according to the first embodiment of the present invention.

FIG. 7 is a diagram showing a range-finding area superimposed on the original image shown in FIG. 6;

FIG. 8 is a diagram showing a defocus map according to the first embodiment of the present invention.

FIG. 9 is a diagram showing a result of group division based on defocus information according to the first embodiment of the present invention.

FIG. 10 is a diagram in which an image according to the first embodiment of the present invention is divided into small images.

FIG. 11 is a diagram showing a relationship between small image division and group division according to the first embodiment of the present invention.

FIG. 12 is a diagram showing integration of small images by defocus information according to the first embodiment of the present invention.

FIG. 13 is a diagram showing an example in which image processing is performed on a small image based on focus information according to the first embodiment of the present invention.

14 is a diagram showing an image obtained by recombining the small images after the image processing shown in FIG.

FIG. 15 shows 7 in the original image in the second embodiment of the present invention.
It is the figure which piled up point distance measurement.

FIG. 16 is a diagram showing a relationship between a small image division and a focus detection point according to the second embodiment of the present invention.

FIG. 17 is a flowchart illustrating an operation of processing according to the first embodiment of the present invention.

FIG. 18 is a flowchart illustrating an operation of processing according to the second embodiment of the present invention.

[Explanation of symbols]

1, 3 Photographing lens 2 Aperture 4 Main mirror 5 Sub-mirror 6 Shutter 7 CCD or MOS type image sensor 8 LCD (transmissive liquid crystal display) display 9 Penta prism 10 Imaging lens 11 Focus detection optical system 12 Focus Detection area sensor 13 Focus adjustment circuit 14 Aperture drive circuit 15 Mirror drive circuit 16 Area sensor drive circuit 17 Shutter drive circuit 18 Image sensor drive circuit 19 A / D conversion circuit 20 Signal processing circuit 21 RAM (random access memory) 22 ROM ( Read-only memory 23 Recording medium such as compact flash (CF) 24 Central processing unit (CP) of microcomputer
U) 25 Monitor (display means) such as LCD 26 Switch input circuit 28 Release switch (release SW) 53 Field lens 54 Field mask 55 Aperture plate 56 Secondary imaging lens

Claims (25)

[Claims] 1. An image pickup unit for converting an optical image of an object into an electric signal to pick up an image, and a focus detection unit for detecting focus information of the image at a plurality of positions corresponding to a shooting screen of the image pickup unit. Dividing means for dividing the image obtained by the imaging means, image processing means for performing image processing on each divided image divided by the dividing means based on the focus information corresponding to each divided image, An image pickup apparatus comprising: a combining unit that combines the divided images that have been image-processed by the image processing unit into one image. 2. An image input unit for inputting an image and focus information of a subject at a plurality of positions of the image, a dividing unit for dividing the image input by the image input unit, and each divided image divided by the dividing unit. On the other hand, the image processing device further includes an image processing unit that performs image processing based on the focus information corresponding to each divided image, and a combining unit that combines the divided images processed by the image processing unit into one image. Image processing device. 3. The image dividing means divides the image data based on at least one of luminance information and color information, and the divided images are reproduced based on the focus information. Device according to claim 1 or 2, characterized in that it comprises subdividing means for dividing and combining. 4. The image processing parameter determining means for determining an image processing parameter when focus information corresponding to the divided image is not detected, wherein the image processing means has a focus corresponding to each divided image. When information is detected, image processing is performed based on the focus information, and when focus information is not detected, image processing is performed using the image processing parameter determined by the image processing parameter determination means. The device according to any one of claims 1 to 3, characterized in that. 5. An image pickup unit for converting an optical image of an object into an electric signal to pick up an image, and distance measurement to the object of the image at a plurality of locations corresponding to a shooting screen of the image pickup unit to obtain distance information. Distance measuring means for acquiring, a dividing means for dividing the image obtained by the image pickup means, and image processing for each divided image divided by the dividing means based on the distance information corresponding to each divided image. An image pickup apparatus comprising: an image processing unit for performing the above; and a combining unit for combining the divided images image-processed by the image processing unit into one image. 6. An image input unit for inputting an image and distance information of a subject at a plurality of positions of the image, a dividing unit for dividing the image input by the image input unit, and each divided image divided by the dividing unit. On the other hand, it is characterized by comprising image processing means for performing image processing based on the distance information corresponding to each divided image, and combining means for combining the divided images image-processed by the image processing means into one image. Image processing device. 7. The dividing means divides the image data based on at least one of luminance information and color information, and divides each divided image based on the distance information. 7. Device according to claim 5 or 6, characterized in that it comprises subdividing means for dividing and combining. 8. The image processing parameter determining means for determining an image processing parameter when there is no distance information corresponding to the divided image, wherein the image processing means has distance information corresponding to each divided image. In this case, the image processing is performed based on the distance information, and when there is no distance information, the image processing is performed using the image processing parameter determined by the image processing parameter determining means. The device according to any one of 1. 9. The apparatus according to claim 1, wherein the image processing means performs low-pass filter processing on each divided image. 10. The apparatus according to claim 4, wherein the parameter determining unit determines the parameter based on the amount of high frequency components in the divided areas. 11. The apparatus according to claim 4, wherein the parameter determining unit determines the parameter based on the position of the divided area in the image. 12. The apparatus according to claim 4, wherein the parameter determining unit determines the parameter based on a parameter of another divided area with which the divided area is in contact. 13. An image inputting step of inputting an image and focus information of a subject at a plurality of positions of the image, a dividing step of dividing the image input by the image inputting step, and each divided image divided by the dividing step. On the other hand, an image processing step of performing image processing based on the focus information corresponding to each divided image, and a combining step of combining the divided images image-processed by the image processing step into one image, Image processing method. 14. The dividing step comprises: an image dividing step of dividing the image data based on at least one of luminance information and color information; and dividing each of the divided images based on the focus information. The image processing method according to claim 13, further comprising a subdivision step of dividing and combining. 15. The method further comprises an image processing parameter determining step of determining an image processing parameter when focus information corresponding to the divided image is not detected, wherein the image processing step includes focus corresponding to each divided image. When information is detected, image processing is performed based on the focus information, and when focus information is not detected, image processing is performed using the image processing parameter determined by the image processing parameter determination step. 15. The image processing method according to claim 13 or 14. 16. An image input step of inputting an image and distance information of a subject at a plurality of positions of the image, a dividing step of dividing the image input by the image input step, and each divided image divided by the dividing step. On the other hand, an image processing step of performing image processing based on the distance information corresponding to each divided image, and a combining step of combining the divided images image-processed by the image processing step into one image are characterized. Image processing method. 17. The dividing step comprises: an image dividing step of dividing the image data based on at least one of luminance information and color information; and dividing each of the divided images based on the distance information. The image processing method according to claim 16, further comprising a subdivision step of dividing and combining. 18. The image processing parameter determining step of determining an image processing parameter when the distance information corresponding to the divided image is not detected, wherein the image processing step includes a distance corresponding to each divided image. When the information is detected, the image processing is performed based on the distance information, and when the distance information is not detected, the image processing is performed using the image processing parameter determined by the image processing parameter determining step. The image processing method according to claim 16 or 17, characterized in that. 19. The image processing method according to claim 13, wherein in the image processing step, low-pass filter processing is performed on each divided image. 20. The image processing method according to claim 15, wherein in the parameter determining step, the parameter is determined based on the amount of high frequency components in the divided area. 21. The image processing method according to claim 15, wherein in the parameter determining step, the parameter is determined based on the position of the divided area in the image. 22. The image processing method according to claim 15, wherein in the parameter determining step, the parameter is determined based on a parameter of another divided area with which the divided area is in contact. 23. A program executable by an information processing device, the information processing device executing the program comprising:
A program causing the image processing apparatus according to any one of claims 2 to 4 and 6 to 12 to function. 24. A program executable by an information processing device, having a program code for realizing the image processing method according to claim 13. 25. A storage medium storing the program according to claim 23 or 24.