JP2010177860A - Image composition apparatus, imaging apparatus, and image composition method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To speedily set an image plane as an object of image composition when an image of the image plane focused on a subject at an arbitrary distance is composed of photographic data after photography. <P>SOLUTION: An image composition apparatus includes a microlens array 7 in which a plurality of microlenses are arrayed, a light receiving element array 6 having a plurality of light receiving elements for the plurality of microlenses, and receiving luminous flux from an optical system 1 through the microlenses to output a plurality of received light signals, a detecting means 11 of detecting a deviation amount of an image plane by the optical system 1 when the received light signals are received, and an image generating means 17 of selecting some of the plurality of received light signals based upon the deviation amount detected by the detecting means 11 and generating an image signal based upon the selected some of the received light signals. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image composition device, an imaging device, and an image composition method.

  The applicant of the present application has applied for an image composition method for compositing an image of a focused image plane onto a subject at an arbitrary distance after photographing from data obtained by one photographing (see Patent Document 1).

JP 2007-4471 A

  In the image composition method described above, no proposal has been made as to how to set the image plane to be subjected to image composition.

An image composition device according to the present invention has a microlens array in which a plurality of microlenses are arranged, and a plurality of light receiving elements for the plurality of microlenses, and receives a plurality of light beams from the optical system via the microlenses. A light receiving element array that outputs a received light signal, a detecting means that detects a deviation amount of an image plane by an optical system when the received light signal is obtained, and one of a plurality of received light signals based on the deviation amount detected by the detecting means. And an image generation means for generating an image signal based on the selected part of the received light signals.
An imaging device according to the present invention includes the above-described image composition device and an optical system.
In the image composition method according to the present invention, a light beam from an optical system is received by a plurality of light receiving elements arranged with respect to a plurality of microlenses to obtain a plurality of light reception signals, and an optical when the light reception signal is obtained. The amount of deviation of the image plane by the system is detected, a part of the plurality of light reception signals is selected based on the detected amount of deviation of the image plane, and an image signal is generated based on the part of the light reception signals.

  According to the present invention, when an image of an image plane focused on a subject at an arbitrary distance after shooting is combined from shooting data, an image plane that is an object of image combining can be quickly set.

It is a block diagram which shows the principal part structure of the digital camera by one Embodiment of this invention. It is a front view of an image sensor and a focus detection optical system. It is the figure which expanded a part of FIG. It is a figure which shows the example which selected 2 pixel columns as a pixel corresponding to a focus detection area. It is a figure which shows the example which selected the pixel row of 1 row as a pixel corresponding to a focus detection area. It is a figure for demonstrating the synthetic | combination method of a captured image. It is a figure for demonstrating the setting method of a focus detection area. It is a flowchart of an imaging process. It is a flowchart of an image composition process. It is a figure which shows the example which uses a personal computer as an image composition apparatus.

  A single-lens reflex digital camera (hereinafter simply referred to as a camera), which is an embodiment of an image composition device and an imaging device according to the present invention, will be described below. FIG. 1 is a block diagram showing a main configuration of the camera. The camera has a camera body and a lens barrel, and the lens barrel is replaceably attached to the camera body. In the lens barrel, a lens optical system 1 and a lens driving motor 14 are provided. The lens optical system 1 is an optical system for forming a subject image on an imaging surface, and includes a plurality of lenses including a focus adjustment lens. The focus adjustment lens is movable in the optical axis direction by the operation of the lens driving motor 14.

  Inside the camera body are a quick return mirror 2, a viewfinder screen 3, a pentaprism 4, an eyepiece lens 5, an image sensor 6, a focus detection optical system 7, a sensor control unit 8, a subject recognition unit 9, and a focus detection region setting unit 10. A focus detection calculation unit 11, a lens drive amount calculation unit 12, a lens drive control unit 13, a photometric lens 15, a photometric sensor 16, an image generation unit 17, and a control device 18 are provided. A display unit 19 is provided on the back of the camera body. The sensor control unit 8, the subject recognition unit 9, the focus detection area setting unit 10, the focus detection calculation unit 11, the lens drive amount calculation unit 12, the lens drive control unit 13, and the image generation unit 17 are controlled by the function of the control device 18. It is realized.

  The imaging element 6 is a light receiving element array having a plurality of light receiving elements (photoelectric conversion elements). In the image sensor 6, red (R), green (G), and blue (B) light receiving elements are arranged in a predetermined arrangement pattern. The image sensor 6 receives the light beam that has passed through the lens optical system 1 at each light receiving element, thereby capturing an image of the image plane formed by the lens optical system 1, and color information and luminance information of each light receiving element. The light reception signal corresponding to the is output. The light reception signal output from the image sensor 6 is converted into image data under the control of the control device 18. Thereby, a captured image is acquired in the camera. For example, a CCD or a CMOS is used for the image sensor 6. An infrared cut filter for cutting infrared light, an optical low-pass filter for preventing image aliasing noise, and the like are disposed in front of the imaging surface of the image sensor 6.

  The focus detection optical system 7 is a microlens array configured by two-dimensionally arranging a plurality of microlenses for forming an image of a light beam that has passed through the lens optical system 1 on each pixel of the image sensor 6. It is arranged in front of the image sensor 6. FIG. 2 is a front view of the image sensor 6 and the focus detection optical system 7. As shown in FIG. 2, the plurality of microlenses 71 of the focus detection optical system 7 are two-dimensionally arranged with respect to the effective pixel range of the image sensor 6. FIG. 3 is an enlarged view of a part of FIG. As shown in FIG. 3, a plurality of light receiving elements 61 are provided for one microlens 71. A plurality of light receiving elements 61 corresponding to the one microlens 71 constitute one pixel in the image sensor 6. With such a configuration, the imaging device 6 can receive the light flux from the lens optical system 1 via the microlens 71 and output a plurality of light reception signals corresponding to each light receiving device 61. Based on this received light signal, captured image information is acquired, and a defocus amount calculation as described later is performed. In FIG. 3, a total of 25 light receiving elements 61 (5 in the vertical direction and 5 in the horizontal direction) form one pixel, but the number of the light receiving elements 61 constituting one pixel is not limited to this.

  A quick return mirror 2 is disposed between the lens optical system 1 and the focus detection optical system 7 to reflect the incident light beam from the subject that has passed through the lens optical system 1 to the viewfinder optical system. Part of the incident light beam passes through the semi-transmissive region of the quick return mirror 2 and is guided to the image sensor 6 via the focus detection optical system 7. The incident light beam that has not passed through the semi-transmissive region of the quick return mirror 2 is reflected by the quick return mirror 2.

  The incident light beam reflected by the quick return mirror 2 forms a subject image on the finder screen 3 provided at a position optically equivalent to the image sensor 6. The subject image formed on the finder screen 3 is guided to the photographer's eyes through the pentaprism 4 and the eyepiece 5 and is visually recognized by the photographer.

  The subject image formed on the finder screen 3 is also guided to the photometric sensor 16 through the pentaprism 4 and the photometric lens 15. As with the image sensor 6, the photometric sensor 16 has red (R), green (G), and blue (B) light receiving elements arranged in a predetermined arrangement pattern, and is imaged by the lens optical system 1. Image information on the imaging plane is captured. Then, a photometric signal corresponding to the color information and luminance information corresponding to each pixel is output to the control device 18. The control device 18 can detect the brightness of the image plane based on the photometric signal from the photometric sensor 16.

  When the camera performs shooting, the quick return mirror 2 is moved from the optical path to the outside of the optical path, and a subject image is formed on the image sensor 6 by the incident light beam through the lens optical system 1. Captured image information is acquired by capturing the subject image with the image sensor 6. The acquired captured image information is recorded in a memory card (not shown). The operation of the quick return mirror 2 is controlled by the control circuit 18.

  In the camera of the present embodiment, color information and luminance information respectively corresponding to a total of 25 light receiving elements 61 (refer to FIG. 3) of 5 vertical and 5 horizontal constituting one pixel in the image sensor 6 are obtained. Recorded as captured image information for one pixel. That is, the captured image information acquired by the camera of the present embodiment includes information on a plurality of light receiving elements per pixel. Based on this captured image information, an image composition process, which will be described later, is performed, so that a captured image in focus on an image plane at an arbitrary distance can be synthesized.

  The sensor control unit 8 controls the gain amount and accumulation time of the image sensor 6 in accordance with the operation status of the camera, and reads out light reception signals output from the light receiving elements of the image sensor 6. The received light signal read by the sensor control unit 8 is converted into a digital signal by analog-digital conversion processing. The received light signal after digital conversion is output to the subject recognition unit 9 and the focus detection calculation unit 11 before photographing, and is subjected to predetermined signal processing at the time of photographing and then recorded on the memory card as captured image information.

  The subject recognizing unit 9 recognizes and recognizes a specific image among the images by the lens optical system 1 as a subject based on the light reception signal from the imaging element 6 before photographing converted into a digital signal by the sensor control unit 8. The position, number, size, etc. of the selected subject are specified. Further, one of the recognized subjects is identified as the main subject. When the camera shoots, these subject recognition results are recorded on the memory card in association with the captured image information described above as part of subject information representing the state of the subject at the time of shooting. For the recognition of the subject, a technique such as template matching that calculates a similarity with a template image registered in advance and detects an image region having a similarity equal to or higher than a predetermined value as the position of the subject can be used.

  The focus detection area setting unit 10 sets, in the image plane, a focus detection area (focus detection position) that is a detection target of the defocus amount by the focus detection calculation unit 11 based on the recognition result of the subject by the subject recognition unit 9. . This focus detection area can be set in an arbitrary size and an arbitrary number at an arbitrary position in the image plane. A specific method for setting the focus detection area by the focus detection area setting unit 10 will be described later.

  The focus detection calculation unit 11 outputs light received from light receiving elements belonging to pixels corresponding to the focus detection area set by the focus detection area setting unit 10 among the light reception signals output from the image sensor 6 via the sensor control unit 8. Based on the received light signal, a defocus amount indicating a focus adjustment state of the lens optical system 1 (an image plane shift amount by the lens optical system 1) is detected. The detection of the defocus amount is performed as described below by a known pupil division type phase difference detection method.

  A method of detecting the defocus amount by the focus detection calculation unit 11 will be described with reference to FIGS. FIG. 4 is an enlarged view of pixels corresponding to the focus detection area set by the focus detection area setting unit 10. The length in the longitudinal direction of the focus detection area, which is the image shift detection direction, is appropriately determined. FIG. 4 shows an example in which two pixel columns are selected as pixels corresponding to the focus detection region, but one pixel column may be selected as pixels corresponding to the focus detection region as shown in FIG. Alternatively, three or more pixel columns may be selected as pixels corresponding to the focus detection area. Based on the light reception signals from the plurality of light receiving elements corresponding to the microlenses in the focus detection region, focus detection signals indicating the amount of image shift due to the pair of light beams that have passed through different pupil regions of the lens optical system 1, that is, the pair of signals A focus detection signal sequence is generated.

4 and 5, the focus detection calculation unit 11 outputs the output of the black light receiving element below the microlens as a first signal sequence {a (i)} and a second signal sequence as a pair of focus detection signals. Extracted as a signal sequence {b (i)} (i = 1, 2, 3,...).
First signal sequence {a (i)} = a (1), a (2), a (3),.
Second signal sequence {b (i)} = b (1), b (2), b (3), ... (1)
In the example shown in FIG. 4, the output of the three light receiving elements under each microlens is added, and the signal output is generated by adding the upper and lower two pixel outputs of the two focus detection pixel columns. Further, when the focus detection area is set in a 45-degree direction, a focus detection pixel is determined and a signal sequence is generated as shown in FIG.

The focus detection calculation unit 11 calculates a defocus amount by performing a known image shift calculation using the first signal sequence {a (i)} and the second signal sequence {b (i)}. First, a correlation amount C (N) of a pair of images (signal sequences) is obtained from the first signal sequence {a (i)} and the second signal sequence {b (i)} by the following equation.
C (N) = Σ | a (i) −b (j) | (2)
In the equation (2), j−i = N (the number of shifts), Σ represents a summation operation where the upper base is qL and the lower base is pL.

The shift amount is obtained from the discrete correlation amount C (N) obtained by the equation (2). Let C be the correlation amount that gives the minimum value when the shift amount is N in C (N), Cr be the correlation amount at the shift amount (N−1), and Cf be the correlation amount at the shift amount (N + 1). A precise shift amount La is obtained from the arrangement of these three correlation amounts Cr, Co, Cf by the following equation.
DL = 0.5 × (Cr−Cf),
E = max {Cf-Co, Cr-Co},
Na = N + DL / E (3)
A correction amount (constant const) corresponding to the position of the focus detection surface is added to this to calculate an image shift amount Δn on the focus detection surface.
Δn = Na + const (4)

Next, the focus detection calculation unit 11 calculates the defocus amount Df from the image shift amount Δn by the following equation using the constant Kf that depends on the detection opening angle.
Df = Kf × Δn (5)
By calculating the defocus amount Df in this manner, the focus detection calculation unit 11 detects the defocus amount. When the camera takes a picture, the defocus amount detected for each focus detection area and the position of the corresponding focus detection area are recorded on the memory card as a part of the subject information.

  The lens drive amount calculation unit 12 calculates the drive amount of the focus adjustment lens included in the lens optical system 1 based on the defocus amount calculated by the defocus calculation unit 10. Here, the lens drive amount is calculated by calculating the lens target position which is the target of the drive position of the focus adjustment lens. The lens target position corresponds to the position of the focus adjustment lens where the defocus amount is zero.

  The lens drive control unit 13 outputs a drive control signal to the lens drive motor 14 based on the lens drive amount calculated by the lens drive amount calculation unit 12, that is, the lens target position with respect to the focus adjustment lens. In response to the drive control signal, the lens driving motor 14 drives the focus adjustment lens and moves it to the lens target position, thereby adjusting the focus of the lens optical system 1.

  The image generation unit 17 synthesizes a captured image focused on an image plane at an arbitrary distance based on captured image information acquired by the image sensor 6 and recorded on a memory card (not shown). The captured image synthesized by the image generation unit 17 is displayed on the display unit 19. The display unit 19 is configured by a display device such as a liquid crystal display, and displays various images and videos including the composite captured image according to display control performed by the control device 18.

  A method for synthesizing captured images by the image generation unit 17 will be described with reference to FIG. FIG. 6A is a diagram for explaining a method of synthesizing a captured image focused on an image plane pa whose distance (image plane deviation amount) from the imaging plane by the lens optical system 1 is h. In FIG. 6A, incident light beams 21 to 25 from the lens optical system 1 corresponding to the image position indicated by reference numeral 20 on the image plane pa pass through the focus detection optical system 7 (microlens 71), and then the pixel 61a. Are received by the light receiving elements a1, b2, c3, d4, and e5. Therefore, by selecting the light receiving signals of the light receiving elements a1, b2, c3, d4 and e5 from the respective light receiving signals represented by the captured image information recorded on the memory card, and synthesizing these light receiving signals, the image plane pa is obtained. An image signal of the pixel 61c corresponding to the image position 20 can be generated. By synthesizing the received light signals for other image positions in the same manner and generating image signals for the corresponding pixels, it is possible to synthesize a captured image in focus on the image plane pa.

  FIG. 6B synthesizes a captured image focused on the image plane pb whose distance from the imaging plane (image plane deviation) by the lens optical system 1 is 0, that is, a captured image of the imaging plane. It is a figure explaining a method. In FIG. 6B, incident light beams 31 to 35 from the lens optical system 1 corresponding to the image position indicated by reference numeral 30 on the image plane pb pass through the focus detection optical system 7 (microlens 71), and then the pixel 61c. Of the light receiving elements c1, c2, c3, c4, and c5. Therefore, by selecting the light receiving signals of the light receiving elements c1, c2, c3, c4 and c5 from the respective light receiving signals represented by the captured image information recorded on the memory card, and synthesizing these light receiving signals, the image plane pb An image signal of the pixel 61c corresponding to the image position 30 can be generated. Similarly, the received light signals are synthesized for other image positions, and an image signal of the corresponding pixel is generated, so that a captured image in focus on the image plane pb can be synthesized.

  As described above, the light reception signals are appropriately selected and synthesized in accordance with the distance from the image plane (image plane deviation amount), and the image signal of each pixel is generated. A focused image that is in focus is synthesized.

  Furthermore, by changing the number of light receiving elements to be selected as light receiving signals, a plurality of types of captured images having the same image plane position and different aperture sizes can be combined. For example, in the examples of FIGS. 6A and 6B, if only the light reception signal of one light receiving element (in the pixel 61c, the light receiving element c3) at the center of each pixel is selected, the image is picked up when the aperture value is minimum. Images can be combined. The captured image combined at this time is a pan focus image in which all subjects are in focus. From this point, as the range of light receiving elements to be selected for the received light signal is expanded and the number thereof is increased, the captured image with a larger aperture value is synthesized.

  The distance from the image plane (image plane deviation amount) corresponds to the defocus amount detected by the focus detection calculation unit 11 with respect to the focus detection area of the subject area in focus on the image plane. Yes. That is, as described above, the defocus amount detected by the focus detection calculation unit 11 at the time of shooting and recorded as part of the subject information is the lens optical when the sensor control unit 8 obtains the light reception signal from the image sensor 6. The amount of image plane displacement due to the system 1 is shown.

  In FIGS. 6A and 6B, the method for synthesizing the captured image when the position of the image plane to be image-combined is on the front side (position close to the lens optical system 1) from the imaging plane has been described. However, even when the position of the image plane that is the object of image synthesis is set behind the imaging plane (position far from the lens optical system 1), the captured image can be synthesized in the same manner.

  The control device 18 is configured by a microcomputer, a memory, and the like, and includes the sensor control unit 8, the subject recognition unit 9, the focus detection area setting unit 10, the focus detection calculation unit 11, the lens drive amount calculation unit 12, and the lens drive control. Processing for realizing the functions of the unit 13 and the image generation unit 17 is executed, and camera operation control is performed. Further, information necessary for these processes and controls is temporarily stored.

  8 and 9 show flowcharts of processing executed by the control device 18 in the camera of the present embodiment. FIG. 8 is a flowchart of the photographing process executed at the time of photographing, and FIG. 9 is a flowchart of the image synthesizing process executed when the captured image focused on the image plane at an arbitrary distance is synthesized and displayed. is there.

  A flowchart of the photographing process in FIG. 8 will be described. In step S10, the control device 18 determines whether or not a release button (not shown) provided on the camera has been half-pressed by the user. When the release button is half-pressed, the control device 18 proceeds to the next step S20.

  In step S20, the control device 18 sets camera shooting conditions. Here, the ISO sensitivity, shutter speed, and aperture value at the time of shooting are determined based on the brightness of the imaging surface detected based on the photometric signal output from the photometric sensor 16 and the type of shooting mode set in advance. By setting, shooting conditions are set.

  In step S <b> 30, the control device 18 recognizes the subject by the subject recognition unit 9. Here, as described above, using a method such as template matching, a specific image is recognized as a subject among the images of the lens optical system 1 represented by the light reception signal from the image sensor 6, and the position, number, and size of the subject are recognized. The main subject is identified.

  In step S <b> 40, the control device 18 causes the focus detection area setting unit 10 to set a focus detection area in the image plane by the lens optical system 1 based on the recognition result of the subject in step S <b> 30. Here, the focus detection area is set as described below.

  FIG. 7 is a diagram for explaining a method of setting the focus detection area in step S40. FIG. 7A shows an example of an image represented by a light reception signal from the image sensor 6. By subject recognition performed by the subject recognition unit 9 in step S30 on such an image, for example, the image plane is divided into a plurality of subject areas indicated by reference numerals 41 to 45 in FIG. The subject area 41 is an area corresponding to a vehicle located near the center of the image plane recognized as the main subject in the image of FIG. Note that which subject is the main subject can be determined based on the positional relationship and size of each subject. The subject area 42 is in the background, the subject area 43 is in the other vehicle running on the right side of the main subject vehicle, the subject area 44 is in the road surface around the main subject vehicle, and the subject area 45 is in front of the main subject vehicle. Correspond to each road surface.

  In step S40, the focus detection area setting unit 10 sets the number of focus detection areas corresponding to the sizes of the subject areas 41 to 45 as shown in FIG. 7C. That is, one focus detection area 41a, 43a, and 44a is set for each of the relatively small subject areas 41, 43, and 44. In addition, two focus detection areas 42a and 42b are set for a subject area 42 larger than these subject areas, and three focus detection areas 45a, 45b and 45c are set for a larger subject area 45. It should be noted that the setting positions of these focus detection areas are preferably as balanced as possible within the subject area. For example, the focus detection areas 41a, 43a, and 44a are set to the gravity center positions of the subject areas 41, 43, and 44, respectively. Further, the focus detection areas 42a and 42b are set at the barycentric positions of the respective areas obtained by dividing the subject area 42 into two, and the focus detection areas 45a, 45b and 45c are set as the barycenters of the respective areas obtained by dividing the subject area 45 into three equal parts. Set to each position.

  As described above, the focus detection area setting unit 10 changes the number of focus detection areas set in each subject area according to the size of each subject area divided by the subject recognition unit 9. Note that whether or not to change the set number of focus detection areas can be determined according to the ratio of the size of the subject area to the image plane. For example, if the size of the subject area is less than 15% with respect to the image plane, the number of focus detection areas set is 1. Further, if the ratio is 15% or more and less than 30%, the setting number of focus detection areas is 2, and if it is 30% or more, the setting number of focus detection areas is 3. The relationship between the ratio of the size of the subject area and the number of focus detection areas set here is merely an example, and the present invention is not limited to this.

  Alternatively, the set position and number of focus detection areas may be determined based on the contrast of the image corresponding to each subject area. For example, a focus detection area is set for each position of an image having a contrast of a predetermined value or more in each subject area. At this time, regardless of the size of the subject area, the focus detection area may be set for all positions where the contrast is equal to or greater than a predetermined value, or a focus detection area that can be set according to the size of the subject area. You may decide the upper limit number. In this way, it is possible to detect a highly reliable defocus amount for the set focus detection region.

  Here, in the above description, the example in which the focus detection area is set for all of the subject areas 41 to 45 divided by the subject recognition unit 9 has been described, but the focus detection area is set for a subject area that does not satisfy the predetermined setting condition. May not be set. For example, a subject area whose image contrast is less than the predetermined value or a subject area whose size is less than a predetermined value can be excluded from the focus detection area setting targets. In this way, since the focus detection area is not set for the subject area where the correct defocus amount cannot be detected, it is possible to prevent the defocus amount detection accuracy from being lowered.

  In step S50, the control device 18 detects the defocus amount for each focus detection area set in step S40 by the focus detection calculation unit 11. The detection of the defocus amount is performed based on the light reception signal from the light reception element belonging to the pixel corresponding to the set focus detection area among the light reception signals from the imaging element 6 as described above. The defocus amount detected here indicates the focus adjustment state of the lens optical system 1 with respect to the subject corresponding to the focus detection area. That is, it represents the amount of deviation of the image plane focused on the subject corresponding to the focus detection area with respect to the image plane formed by the lens optical system 1.

  In step S60, the control device 18 adjusts the focus of the lens optical system 1 by the lens driving amount calculation unit 12 based on the defocus amount detected in step S50. Here, based on the defocus amount of the focus detection area corresponding to the main subject among the detected defocus amounts, the lens drive amount is calculated so that the main subject is in focus. Then, a driving control signal is output from the lens driving amount calculation unit 12 to the lens driving motor 14, and the lens driving motor 14 drives the focus adjusting lens, thereby adjusting the focus of the lens optical system 1.

  In step S70, the control device 18 determines whether or not a release button (not shown) provided on the camera has been fully pressed by the user. If the release button has not been fully pressed, the process returns to step S10 to repeat the processing as described above. When the release button is fully pressed, the control device 18 proceeds to the next step S80.

  In step S80, the control device 18 performs shooting. At this time, as described above, the quick return mirror 2 is moved from the optical path to the outside of the optical path, and the subject image formed by the incident light beam through the lens optical system 1 is imaged by the imaging element 6, and per pixel. Captured image information having information of a plurality of light receiving elements is acquired.

  In step S90, the control device 18 acquires subject information. Here, the subject information as described above is acquired based on the execution results of steps S30 to S50 immediately before the release button full-press operation is detected in step S70. This subject information includes information on a subject recognition result at the time of shooting, a focus detection area, and a defocus amount.

  In step S100, the control device 18 records the captured image information acquired in step S80 and the subject information acquired in step S90 on a memory card (not shown) in association with each other. If step S100 is performed, the control apparatus 18 will complete | finish the flowchart of FIG. The photographing process is performed as described above.

  Next, a flowchart of the image composition process in FIG. 9 will be described. The image composition process is performed when the user instructs the camera to compose an image by operating an operation button (not shown) provided on the camera. In step S110, the control device 18 reads the captured image information recorded in the memory card in step S100 of FIG. 8, and reproduces and displays an initial image based on the captured image information on the display unit 19. Here, the information of the received light signal by the light receiving element at the center of each pixel of the image sensor 6 is selected from the read captured image information, and the image represented by the received light signal information is displayed as the initial image. As described above, this initial image is a pan focus image in which all subjects are in focus. Alternatively, a light reception signal may be selected as described with reference to FIG. 6B, and an image represented by the light reception signal information may be displayed as an initial image. The initial image in this case is a captured image corresponding to the imaging surface by the lens optical system 1 that has been adjusted so that the main subject is focused in step S60 of FIG. 8, that is, a captured image in which the image plane deviation amount is zero. It is.

  In step S120, the control device 18 sets the focus position for combining the captured images in the initial image displayed in step S110. Here, for example, when the user designates an arbitrary position on the initial image by operating an operation button (not shown) provided on the camera, the position is set as a focus position. Alternatively, any position on the initial image may be automatically selected by the camera and set as the focus position.

  In step S130, the control device 18 extracts subject information corresponding to the focus position set in step S120 from the subject information recorded in the memory card in association with the captured image information read in step S110. . That is, information such as the position, number, and size of the corresponding subject area is extracted with respect to the set focus position. Further, information on the focus detection area set for the subject area and the defocus amount detected for the focus detection area is extracted.

  In step S140, the control device 18 determines the position of the image plane where the captured images are combined based on the defocus amount indicated by the subject information extracted in step S130. When multiple defocus amount information is extracted as subject information, the average value, minimum value, maximum value, etc. are calculated as representative defocus amounts, and the position of the image plane where the captured images are combined is determined. May be.

  In step S150, the control device 18 sets an aperture for combining captured images. The aperture setting is performed, for example, when the user operates an operation button (not shown) provided on the camera. Alternatively, the camera may automatically set the aperture.

  In step S160, the control device 18 causes the image generation unit 17 to generate an image signal for displaying a captured image corresponding to the image plane position determined in step S140. Here, based on the defocus amount indicated by the subject information extracted in step S130 corresponding to the image plane shift amount at the determined image plane position, each of the captured image information represented by the method described in FIG. 6 is used. A part of the light reception signal by the light receiving element of the pixel is selected. At this time, the number of light receiving elements to be selected for the received light signal is determined according to the diaphragm set in step S150. By generating an image signal based on the received light signal thus selected, a captured image focused on the subject set at the focus position is synthesized.

  In step S170, the control device 18 reproduces and displays the captured image based on the image signal generated in step S160 on the display unit 19. If step S170 is performed, the control apparatus 18 will complete | finish the flowchart of FIG. The image composition process is performed as described above.

  According to the embodiment described above, the following operational effects can be achieved.

(1) In the photographing process of FIG. 8, the control device 18 detects a defocus amount indicating the amount of deviation of the image plane by the lens optical system 1 when the received light signal is obtained from the image sensor 6 (step S50). Further, in the image composition process of FIG. 9, subject information corresponding to the focus position is extracted (step S130), and the image sensor recorded on the memory card as captured image information based on the defocus amount in the extracted subject information. A part of the plurality of light receiving signals from each of the six light receiving elements is selected as shown in FIG. 6, and an image signal is generated based on the selected part of the light receiving signals (step S160). Since the captured image is synthesized by generating the image signal in this way, when compositing the image of the image plane in focus on the subject at an arbitrary distance after photographing from the photographing data, The image plane to be performed can be set quickly.

(2) The control device 18 sets a focus detection area in the image plane by the lens optical system 1, thereby specifying a position in the image plane that is a target for detecting the defocus amount in step S50 (step S40). . In step S160, a part of the plurality of received light signals is selected based on the defocus amount with respect to the position in the image plane thus specified as the focus detection area. Since it did in this way, a defocus amount can be detected with respect to the appropriate position in an image surface, and a captured image can be synthesize | combined.

(3) The control device 18 recognizes a specific image among the images obtained by the lens optical system 1 as a subject (step S30). In step S50, the defocus amount of the focus detection area set in step S40 with respect to the position of the subject thus recognized is detected. Therefore, it is possible to automatically detect the defocus amount with respect to the subject in the image plane.

(4) In step S30, the control device 18 divides the image plane by the lens optical system 1 into a plurality of subject areas based on the recognized subject. In step S50, the defocus amount of the focus detection area set in step S40 for each of the subject areas thus divided is detected. Since it did in this way, when a several subject exists in an image surface, the defocus amount with respect to each subject can be detected reliably.

(5) When the size of the subject area divided in step S30 is equal to or larger than a predetermined ratio of the image plane, the control device 18 respectively sets a plurality of focus detection areas set in step S40 for the subject area. A plurality of corresponding defocus amounts are detected in step S50. Therefore, an appropriate number of defocus amounts can be detected in accordance with the ratio of the area in the image plane.

(6) The control device 18 also sets a focus detection area in step S40 for the position of an image having a contrast equal to or higher than a predetermined value among the images of the lens optical system 1 based on the light reception signal from the image sensor 6; In step S50, the defocus amount may be detected. In this way, a highly reliable defocus amount can be detected.

(7) The control device 18 adjusts the focus of the lens optical system 1 based on the defocus amount detected in step S50 (step S60). In step S160, an image signal is generated based on the received light signal output by the image sensor 6 that has received the light beam from the lens optical system 1 that has been subjected to focus adjustment in this manner and recorded as captured image information. Since it did in this way, the focus adjustment of the lens optical system 1 can be performed appropriately at the time of imaging | photography.

  In the above embodiment, the subject is recognized based on the light reception signal output from the image sensor 6 in step S30. However, the subject may be recognized based on the photometry signal output from the photometry sensor 16. . Alternatively, an image sensor for subject recognition may be provided in the camera separately from the image sensor 6 and the photometric sensor 16, and the subject may be recognized based on image information captured using the image sensor.

  Further, the image composition process shown in the flowchart of FIG. 9 may be performed by an image composition apparatus other than the camera described in the above embodiment. In this case, a program for executing the image composition process can be provided to the image composition apparatus through a recording medium such as a CD-ROM or an electric communication line such as the Internet. FIG. 10 shows an example in which a personal computer is used as an image composition device. The personal computer 100 is provided with a program via the CD-ROM 101. Alternatively, the personal computer 100 may have a connection function with the communication line 104 and be provided with the above program from the server 103. The communication line 104 is a communication line such as the Internet or personal computer communication, or a dedicated communication line. The server 103 transmits the program to the personal computer 100 via the communication line 104. That is, the program is converted into a data signal on a carrier wave and transmitted via the communication line 101. Thus, the program can be supplied as a computer-readable computer program product in various forms such as a recording medium and a carrier wave.

  The above-described personal computer 100 is loaded with a recording medium 102 on which captured image information and subject information are recorded by the camera control device 18 executing the shooting process of FIG. The personal computer 100 executes the program provided from the CD-ROM 101 or the server 103 to read the captured image information and the subject information from the recording medium 102, and based on the read captured image information and the subject information, FIG. By executing this image composition processing, a captured image focused on an arbitrary image plane position is synthesized and displayed on the display.

  The embodiment and various modifications described above are merely examples of the embodiment of the present invention. Therefore, when interpreting the invention, the correspondence between the above described items and the items described in the claims is not limited or restricted. In addition, the present invention is not limited to the above description unless the features of the invention are impaired.

DESCRIPTION OF SYMBOLS 1 Lens optical system 6 Image pick-up element 7 Focus detection optical system 9 Subject recognition part 10 Focus detection area setting part 11 Focus detection calculating part 12 Lens drive amount calculating part 13 Lens drive control part 17 Image generation part 18 Control apparatus

Claims (11)

A microlens array in which a plurality of microlenses are arranged; and
A light receiving element array having a plurality of light receiving elements for the plurality of micro lenses, receiving a light beam from an optical system via the micro lens, and outputting a plurality of light receiving signals;
Detecting means for detecting a shift amount of an image plane by the optical system when the light reception signal is obtained;
Image generating means for selecting a part of the plurality of received light signals based on the deviation detected by the detecting means and generating an image signal based on the selected received light signals. An image composition device. The image composition apparatus according to claim 1,
A specifying unit that specifies a position within the image plane of the optical system that is a target for detecting a shift amount by the detecting unit;
The image synthesizing apparatus, wherein the image generating unit selects a part of the plurality of received light signals based on a deviation amount of the image plane with respect to the position specified by the specifying unit. The image composition device according to claim 2,
The specifying means includes a recognition means for recognizing a specific image among the images by the optical system,
The image synthesizing apparatus according to claim 1, wherein the detecting unit detects a shift amount of the image plane with respect to the position of the image recognized by the recognizing unit. The image composition device according to claim 3.
The recognizing unit divides the image plane by the optical system into a plurality of regions based on the recognized specific image,
The image synthesizing apparatus according to claim 1, wherein the detection unit detects a shift amount of the image plane with respect to the region divided by the recognition unit. The image composition device according to claim 4, wherein
The detection means detects a plurality of shift amounts of the image plane with respect to the area when the size of the area divided by the recognition means is equal to or larger than a predetermined ratio of the image plane. An image composition device. In the image composition device according to any one of claims 1 to 5,
The image synthesizing apparatus according to claim 1, wherein the detection unit detects a shift amount of the image plane with respect to a position of an image having a contrast equal to or higher than a predetermined value among images of the optical system based on the light reception signal. The image composition device according to any one of claims 1 to 6,
An imaging apparatus comprising the optical system. The imaging apparatus according to claim 7,
A focus adjusting means for adjusting the focus of the optical system based on the shift amount of the image plane detected by the detecting means;
The image generation unit generates the image signal based on the light reception signal output by the light receiving element array that receives a light beam from the optical system whose focus is adjusted by the focus adjustment unit. An imaging device. A plurality of light receiving signals are obtained by receiving a light beam from the optical system by a plurality of light receiving elements arranged for a plurality of microlenses,
Detecting the amount of image plane displacement by the optical system when the received light signal is obtained,
An image synthesis method for selecting a part of the plurality of light reception signals based on the detected shift amount of the image plane and generating an image signal based on the part of the light reception signals. The image composition method according to claim 9.
Specify the position in the image plane by the optical system as a target for detecting the amount of deviation of the image plane,
A method of synthesizing an image, wherein a part of the plurality of light reception signals is selected based on a deviation amount of the image plane with respect to the specified position. The image composition method according to claim 10,
By recognizing a specific image among the images by the optical system, the position to be detected is determined as a target for detecting the shift amount of the image plane,
An image composition method, comprising: detecting an amount of deviation of the image plane with respect to the recognized position of the specific image.

Images