JP5526733B2 - Image composition device, image reproduction device, and imaging device - Google Patents

Description

  The present invention relates to an image composition device, an image reproduction device, and an imaging device.

  2. Description of the Related Art Conventionally, an imaging apparatus that outputs a subject image formed on an imaging surface of an imaging element as an imaging signal is known. In such an imaging apparatus, a technique for synthesizing an image focused on an arbitrary image plane of an optical system based on an imaging signal obtained by one imaging is disclosed (Patent Document 1).

JP 2007-4471 A

  However, in the prior art, no proposal has been made on how to determine an image plane for acquiring a focused image for a specific subject.

  The problem to be solved by the present invention is to provide an image synthesizing apparatus that can appropriately obtain a focused image of a specific subject in a captured image.

  The present invention solves the above problems by the following means. In addition, although the code | symbol corresponding to drawing which shows embodiment of this invention is attached | subjected and demonstrated, this code | symbol is only for making an understanding of invention easy, and is not the meaning which limits invention.

[1] image synthesizing apparatus according to the present invention includes an input means for inputting a light receiving signal obtained by receiving the light beam in a single exposure operation (101), a partial area from the image based on the received light signal an area selecting means for selecting (104), on the basis of the light reception signal corresponding to the partial region, a first image signal in a first image plane position of the partial region, said first image plane position different, the generates the second image signal in the second image plane position of the partial region, an evaluation value relating to the contrast of the first image signal, on the basis of the evaluation value about the contrast of the second image signal And image generation means (104) for generating an image signal for generating an image signal corresponding to an image plane position where the evaluation value is a value in the vicinity of the maximum value .

[2] In the invention relating to the image composition device, the image generation means (104) generates an image signal corresponding to an image plane position where the evaluation value is maximum based on evaluation values of at least three image signals . Can be configured to.

[ 3 ] In the invention relating to the image composition device, the area selection means (104) includes a recognition means (104) for recognizing an image scene based on the light reception signal, and the image generation means (104) The partial area may be determined according to the scene recognized by the recognition means.

[ 4 ] An image reproduction device according to the present invention is characterized by including the above-described image composition device and a display unit (102) for displaying an image based on the image signal.

[ 5 ] An imaging device according to the present invention is provided for the above-described image composition device, a microlens array (212) in which a plurality of microlenses (212a) are two-dimensionally arranged, and the plurality of microlenses. An image sensor (211) having a plurality of photoelectric conversion elements (213a) and inputting the received light signal obtained by receiving a light beam from an optical system via the microlens array to the input means (101). And.

  According to the present invention, it is possible to appropriately obtain an image focused on a specific subject in a captured image.

FIG. 1 is a block diagram showing a configuration of a personal computer 100 according to the present embodiment. FIG. 2 is a block diagram illustrating a configuration of the camera 200 according to the present embodiment. FIG. 3 is an example of an image displayed on the display of the display unit 102. FIG. 4 is a partial enlarged view of the imaging surface of the imaging device 211. FIG. 5 is a diagram for explaining the configuration of the image sensor 211. FIG. 6 is a flowchart illustrating an operation example of the image composition system according to the present embodiment. 7, through the micro lens 212a, among the plurality of photoelectric conversion elements 213a constituting the photoelectric conversion element array 213 is a diagram showing an example of a light beam incident on a particular photoelectric conversion element c _1. FIG. 8 is a diagram for explaining an example of an image composition method when the image plane corresponding to the position where the subject exists is a specific plane (Z = 0). FIG. 9 is a diagram for explaining an example of an image composition method when the image plane corresponding to the position where the subject exists is a specific plane (Z = h ₁ ). FIG. 10 is a diagram for explaining an example of an image composition method when the image plane corresponding to the position where the subject exists is a specific plane (Z = h ₂ ).

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The image composition system according to this embodiment includes a personal computer (personal computer) 100 shown in FIG. 1 and a camera 200 shown in FIG. In addition, the image composition system of the present embodiment captures image data of a captured image captured by the camera 200 into the personal computer 100, and the personal computer 100 performs image composition processing based on the captured image data of the captured image. It is.

  FIG. 1 is a block diagram showing a configuration of a personal computer 100 according to the present embodiment. As shown in FIG. 1, the personal computer 100 includes an input unit 101, a display unit 102, an operation unit 103, and a control unit 104.

  The input unit 101 acquires image data of a captured image captured by the camera 200, captures the image data in the personal computer 100, and transmits the captured image data of the captured image to the control unit 104. Here, as the input unit 101, for example, a USB interface for connecting an external device via a USB cable can be used. In this case, a memory card reader for inputting / outputting data to / from a memory card such as an SD card is connected to the input unit 101, and a memory card storing image data of a photographed image taken by the camera 200 is connected to the memory. By inserting the card in the card reader, the input unit 101 can acquire image data of a photographed image photographed by the camera 200. Note that the input unit 101 is not limited to the USB interface described above, and may be, for example, an interface into which a memory card can be directly inserted, or from the camera 200 via a wired or wireless communication line (not shown). An interface that can receive image data of a captured image may be used, or an interface that can be directly connected to the camera 200 via a predetermined cable.

  The display unit 102 displays an image based on image data of a captured image captured by the camera 200 on a display included in the display unit 102. FIG. 3 shows an example of an image displayed by the display unit 102.

  The operation unit 103 includes various operation devices such as a keyboard, a mouse, and a touch panel used by the user in order to perform various operations such as an operation of selecting a subject to be focused by the user. For example, when a subject to be focused is selected from subjects in a photographed image displayed on the display of the display unit 102 by the user via the operation unit 103, photographing corresponding to the selected subject is performed. Information on the area in the image is transmitted to the control unit 104 and set as a subject area described later.

  The control unit 104 includes a memory, a CPU, and other peripheral components, acquires image data of a captured image captured by the camera 200 from the input unit 101, calculates a focus evaluation value related to contrast based on the acquired image data, and Evaluate. Further, the control unit 104 synthesizes an image corresponding to the image plane position where the focus evaluation value is maximized based on the evaluation result of the focus evaluation value, and obtains a composite image.

  In addition, when the user selects the subject to be focused from among the subjects in the captured image displayed on the display of the display unit 102 via the operation unit 103, the control unit 104 selects the subject. An area in the captured image corresponding to the selected subject is set as the subject area. For example, in the example shown in FIG. 3, when a streetlight (lamp) displayed on the display of the display unit 102 is selected as a subject to be focused by the user, the control unit 104, as shown in FIG. An area corresponding to a street lamp (lamp) in the captured image is set as a subject area.

  Next, a description will be given of the camera 200 that captures a captured image that is used to synthesize a composite image in the personal computer 100. FIG. 2 is a block diagram illustrating the configuration of the camera 200. As shown in FIG. 2, the camera 200 includes a camera body 210 and a lens barrel 220.

  The lens barrel 220 incorporates a photographing optical system including lenses 221, 222, 223 and a diaphragm 224.

  The focus lens 222 is provided so as to be movable along the optical axis L1 of the lens barrel 220, and extends from an end on the camera body 210 side (also referred to as the closest end) to an end on the subject side (also referred to as an infinite end). It is possible to move in the direction of the optical axis L1. Incidentally, the movement of the focus lens 222 is controlled based on a command from the lens control unit 227, and its position is adjusted by the lens driving motor 226.

  The aperture 224 has an aperture diameter centered on the optical axis L1 in order to limit the amount of light flux that passes through the imaging optical system and reaches the image sensor 211 provided in the camera body 210 and adjusts the amount of blur. It is configured to be adjustable. Adjustment of the aperture diameter by the aperture 224 is performed by transmitting an appropriate aperture diameter calculated in the automatic exposure mode from the camera control unit 215 to the aperture drive unit 225 via the lens control unit 227, for example.

  On the other hand, in the camera body 210, an image sensor 211 that receives a light beam from a subject is provided on the planned focal plane of the photographing optical system on the optical axis L1. The image sensor 211 is composed of a device such as a two-dimensional CCD image sensor, a MOS sensor, or a CID, and converts a received optical signal into a received light signal. Here, FIG. 4 is a partially enlarged view of the imaging surface of the imaging device 211. As shown in FIG. 4, the imaging device 211 includes a microlens array 212 in which a plurality of microlenses 212a are densely arranged in a two-dimensional manner. In addition, the imaging element 211 has each photoelectric conversion element array 213 configured with a plurality of photoelectric conversion elements 213a with respect to each microlens 212a. In FIG. 4, the number (pixel density) of the photoelectric conversion elements 213a constituting the photoelectric conversion element array 213 is five in both the vertical direction and the horizontal direction, but these numbers are particularly limited. is not.

  Further, FIG. 5 is a diagram for explaining the configuration of the image sensor 211, and is an enlarged perspective view of the image pickup surface of the image sensor 211 as in FIG. As shown in FIG. 5, the photoelectric conversion element array 213 is arranged behind the microlens array 212, and corresponds to the focal length of the microlens 212 a between the microlens array 212 and the photoelectric conversion element array 213. An interval is provided. A light beam (optical axis L1) from the subject is first incident on the microlens 212a, passes through the microlens 212a, and is received by the photoelectric conversion element 213a. Each photoelectric conversion element 213a outputs a light reception signal corresponding to the intensity of the received light, and the output light reception signal is transmitted to the camera control unit 215.

  The camera operation unit 214 includes, for example, a shutter release button, a mode setting switch for setting various operation modes of the camera 1, and the camera operation unit 214 can switch between an autofocus mode and a manual focus mode. It has become.

  The camera control unit 215 performs various processes such as A / D conversion on the light reception signal obtained from the image sensor 211 and stores it in the memory 216 as image data. In addition, the camera control unit 215 performs focusing based on the light reception signal obtained from the image sensor 211. By calculating the driving amount of the lens 212 and transmitting the calculated driving amount of the focus lens 212 to the lens control unit 227, the operation of the entire camera 1 is controlled such as driving control of the focus lens 212.

  Next, an operation example of the image composition system according to the present embodiment will be described. FIG. 6 is a flowchart illustrating an operation example of the image composition system according to the present embodiment. In the following description, it is assumed that image data of a photographed image photographed by the camera 200 is captured in the personal computer 100. The processing described below is performed by the personal computer 100 constituting the image composition system.

  First, in step S <b> 101, a photographed image photographed by the camera 200 is displayed on the display unit 102. For example, when a user selects one of a plurality of captured images captured in the personal computer 100 via the operation unit 103, the selected captured image is displayed on the display of the display unit 102. Is displayed. Note that, as described above, the image data used here is a light reception signal obtained by receiving the light beam that has passed through the microlens 212a of the imaging element 211 of the camera 200 shown in FIG. 5 by each photoelectric conversion element 213a. It is based on.

  Subsequently, in step S <b> 102, the control unit 104 determines whether or not a subject to be focused is selected via the operation unit 103. When the subject is selected by the user, the process proceeds to step S103, and an area in the captured image corresponding to the selected subject is set as the subject area. By setting the subject area in this way, the subject area corresponding to the subject selected by the user, that is, the image data corresponding to a part of the photographed image, not the entire area in the photographed image. Based on this, an image composition process to be described later is performed. For example, in the example shown in FIG. 3, when the user selects a streetlight (lamp) in the captured image as a subject to be focused, the process proceeds to step S103, and as shown in FIG. 3, the streetlight in the captured image is displayed. An area corresponding to (lamp) is set as a subject area. On the other hand, if the subject has not been selected by the user, the process returns to step S101 and waits until the subject is selected by the user.

  In step S104, based on the image data of the photographed image, the control unit 104 determines image planes corresponding to a plurality of image plane positions within an image synthesis range (a range in which a synthesized image signal can be synthesized; details will be described later). A plurality of different image signals are combined as combined image signals.

  Here, the plurality of synthesized image signals synthesized in the present embodiment are used for calculation of a focus evaluation value in step S105 described later and detection of a peak in the focus evaluation value in step S106. In the present embodiment, the number of synthesized image signals to be synthesized within the image synthesis range is not particularly limited, but may be any number that can detect the peak of the focus evaluation value, and is usually three or more.

  In the present embodiment, the composite image signal is not the entire region in the captured image, but the subject region set in step S103 among all the regions in the captured image, that is, a partial region in the captured image. Are synthesized based on the image data corresponding to. For example, in the example shown in FIG. 3, when an area in the captured image corresponding to the street lamp (lamp) shown in FIG. 3 is set as the subject area in step S103, all areas in the captured image are set in step S104. The synthesized image signal is synthesized based on the data corresponding to the subject region shown in FIG. 3 in the image data, not the corresponding image data.

Furthermore, in this embodiment, a plurality of synthesized image signals corresponding to a plurality of different image plane positions (focused on the image plane positions) are synthesized based on the image data of the captured image. The range of the image plane in which the signals are combined is an image combining range described below. FIG. 7 is a diagram for explaining an image composition range, which is a range in which a composite image signal can be synthesized, and a specific photoelectric conversion element c ₁ among a plurality of photoelectric conversion elements 213 a constituting the photoelectric conversion element array 213. It is a figure which shows the light beam which injects with respect to through the micro lens 212a. In FIG. 7, the photoelectric conversion elements 213 a constituting the photoelectric conversion element array 213 are indicated by a ₁ , b ₁ , c ₁ , d ₁ , and e ₁ . The resolution of an image obtained by the imaging apparatus according to the present embodiment corresponds to one microlens that is a unit of pixel. Therefore, as shown in FIG. 7, the range of the image plane in which the image can be synthesized while maintaining the resolution is such that the size of the back-projected image by the microlens 212a of the photoelectric conversion element 213a is substantially equal to the effective diameter D of the microlens 212a. The distance L from the micro lens 212a can be the same. That is, if light from a range having the same size as the effective diameter D (array pitch P> D) of the microlens 212a passes through the microlens 212a and enters _one photoelectric conversion element c1, it is a unit of a pixel. A resolution corresponding to the size of the microlens can be obtained. Therefore, this distance L becomes an image composition range, and if it is an image plane position within this image composition range, it is possible to synthesize a composite image signal for a plurality of different image plane positions. Therefore, in step S104, a plurality of synthesized image signals having different image planes corresponding to a plurality of image plane positions are synthesized by the control unit 104 in accordance with the image synthesizing method described below within the image synthesis range described above. .

8 to 10 are diagrams for explaining an example of the image composition method according to the present embodiment. In the example of the scene shown in FIG. 8, when the height of the image plane from the microlens array 212 (distance from the microlens array 212) is Z, the image synthesis target is positioned at a position where the height of the image plane Z = 0. This is a case where there is a subject to be. In FIG. 8, for each photoelectric conversion element array 213 of the image sensor 211, each light beam (microlens array 212 is incident on five photoelectric conversion elements 213 a among the photoelectric conversion elements 213 a constituting each photoelectric conversion element array 213. Only the chief ray passing through the center of the microlens 212a to be configured) is shown. Further, in the figure 8, in order to identify the respective photoelectric conversion elements 213a, each of the photoelectric conversion element _{213a, a 1 ~e 1, a} 2 ~e 2, a 3 ~e 3, a 4 ~e 4 , A _{5 to} e ₅ and out of the coordinates X ₁ , X ₂ , X ₃ , X ₄ , X ₅ at the height Z = 0 of the image plane, the emitted light beam from X ₃ (light rays r ₁ , r ₂ , r ₃ , r ₄ , r ₅ ) are indicated by solid lines, and other light beams emitted from X ₁ , X ₂ , X ₄ , X ₅ are indicated by dotted lines (hereinafter the same applies to FIGS. 9 and 10). .)

As shown in FIG. 8, the emitted light beams (rays r ₁ , r ₂ , r ₃ , r ₄ , r ₅ ) from the coordinate X ₃ at the height Z = 0 of the image plane are respectively converted into photoelectric conversion elements a ₃ , b ₃ , c ₃ , d ₃ , and e ₃ , respectively. Therefore, the pixel value L (Z = 0, X ₃ ) at the coordinate X ₃ at the height Z = 0 of the image plane combines the outputs at these photoelectric conversion elements a ₃ , b ₃ , c ₃ , d ₃ , e ₃ . (See the following formula (1)).
L (Z = 0, X ₃ ) = Out (a ₃ ) + Out (b ₃ ) + Out (c ₃ ) + Out (d ₃ ) + Out (e ₃ ) (1)

Similarly, the pixel value L (Z = 0, X ₄ ) at the coordinate X ₄ adjacent to the coordinate X ₃ can be obtained according to the following equation (2).
L (Z = 0, X ₄ ) = Out (a ₄ ) + Out (b ₄ ) + Out (c ₄ ) + Out (d ₄ ) + Out (e ₄ ) (2)

Therefore, the pixel value L (Z = 0, X _i ) at the coordinate X _i can be obtained according to the following equation (3).
L (Z = 0, X _i ) = Out (a _i ) + Out (b _i ) + Out (c _i ) + Out (d _i ) + Out (e _i ) (3)

In addition, the said Formula (3) is a formula employ | adopted when the designated aperture value by a user is open | released (opening size maximum). Therefore, if the aperture value designated by the user is the maximum (minimum aperture size), the light beam composed of the light beams r ₁ , r ₂ , r ₃ , r ₄ , r _{5 is changed} to the light beam composed only of the light beam r _3. Therefore, the following formula (4) may be adopted instead of the above formula (3) (the same applies to the cases shown in FIGS. 9 and 10 described later).
L (Z = 0, X _i ) = Out (c _i ) (4)

When the aperture value is an intermediate value (intermediate aperture size), a light beam composed of the light beams r ₁ , r ₂ , r ₃ , r ₄ , and r ₅ is converted into a light beam composed of only the light beams r ₂ , r ₃ , and r _4. Therefore, the following equation (5) may be adopted instead of the above equation (3) (the same applies to the cases shown in FIGS. 9 and 10 described later).
L (Z = 0, X _i ) = Out (b _i ) + Out (c _i ) + Out (d _i ) (5)

In the above description, only the five photoelectric conversion elements a ₃ , b ₃ , c ₃ , d ₃ , and e ₃ arranged in a certain direction are focused on, and the sum of the output values of these five photoelectric conversion elements is taken. However, actually, it is necessary to take the sum of the output values of 25 photoelectric conversion elements arranged in two directions (the same applies to the cases shown in FIGS. 9 and 10 described later).

Next, as shown in FIG. 9, a case will be described in which a subject to be image-combined exists at a position where the image plane height Z = h ₁ . As shown in FIG. 9, the emitted light beam (rays r ₁ , r ₂ , r ₃ , r ₄ , r ₅ ) from the coordinate X ₃ at the image plane height Z = h ₁ is different from the case of FIG. The light enters the photoelectric conversion elements a ₁ , b ₂ , c ₃ , d ₄ , and e ₅ . Therefore, the pixel value L (Z = h ₁ , X ₃ ) at the coordinate X ₃ at the height Z = h ₁ of the image plane is the output from the photoelectric conversion elements a ₁ , b ₂ , c ₃ , d ₄ , e ₅ . Can be obtained by synthesizing (see the following formula (6)).
L (Z = h ₁ , X ₃ ) = Out (a ₁ ) + Out (b ₂ ) + Out (c ₃ ) + Out (d ₄ ) + Out (e ₅ ) (6)

Further, as shown in FIG. 10, a case where a subject to be synthesized exists at a position where the height Z of the image plane is Z = h ₂ will be described. As shown in FIG. 10, the emitted light beam (rays r ₁ , r ₂ , r ₃ , r ₄ , r ₅ ) from the coordinate X ₃ at the height Z = h ₂ of the image plane is the case of FIGS. Unlike the case, the light enters across a plurality of photoelectric conversion elements. Specifically, as shown in FIG. 10, the light beam r ₁ is converted into photoelectric conversion elements a ₁ and b ₁ , the light beam r ₂ is converted into photoelectric conversion elements b ₂ and c ₂ , and the light beam r ₄ is converted into a photoelectric conversion element c ₄ , The light beam r ₅ enters the photoelectric conversion elements d ₅ and e ₅ across d ₄ and d ₄ , respectively. The light beam r ₃ is incident only on the photoelectric conversion element c ₃ as shown in FIG. Therefore, when focusing on the light _{r 1,} the light quantity of the light rays _{r 1} is determined by the weighted sum of the output value of the photoelectric conversion elements _{a 1} Out _{(a 1),} the output value of the photoelectric conversion elements _{b 1} Out _{(b 1)} (See the following formula (7)). Here, in the following formula (7), w ₁₁ and w ₁₂ are weighting coefficients, which are determined according to the height Z of the image plane from the microlens array 212.
Out (a ₁ ) × w ₁₁ + Out (b ₁ ) × w ₁₂ (7)

Since the light amounts of the light beam r ₂ , the light beam r ₄ , and the light beam r ₅ can be similarly obtained by weighted sum, the pixel value L (Z = h _{2) at the} coordinate X ₃ at the height Z = h ₂ of the image plane. , X ₃ ) can be determined according to the following formula (8). In the following equation (8), W ₂₁ , W ₂₂ , W ₄₁ , W ₄₂ , W ₅₁ , W ₅₂ are weighting factors, and are determined according to the height Z of the image plane from the microlens array 212. It is.
L (Z = h ₂ , X ₃ ) = [Out (a ₁ ) × w ₁₁ + Out (b ₁ ) × w ₁₂ ] + [Out (b ₂ ) × W ₂₁ + Out (c ₂ ) × w ₂₂ ] + Out ( c ₃ ) + [Out (c ₄ ) × W ₄₁ + Out (d ₄ ) × W ₄₂ ] + [Out (d ₅ ) × W ₅₁ + Out (e ₅ ) × W ₅₂ ] (8)

  As described above, the photoelectric conversion element 213a on which the light flux from the subject is incident and the value of the weighting coefficient necessary for the image composition are determined according to the image plane position Z where the subject to be image-synthesized exists. Become. Note that the photoelectric conversion element 112a on which the light beam from the subject is incident and the value of the weighting coefficient necessary for image composition corresponding to each image plane position Z are stored in advance in, for example, a memory provided in the control unit 104. The configuration may be such that this is used.

  As described above, the control unit 104 can synthesize the composite image signal corresponding to the predetermined image plane position Z using the image data based on the light reception signals obtained from the plurality of photoelectric conversion elements 112a. Therefore, in step S104, the control unit 104 synthesizes a plurality of synthesized image signals having different image planes corresponding to the plurality of image planes in the image synthesis range according to the above-described image synthesis method.

  Next, in step S105, the control unit 104 calculates a focus evaluation value related to the contrast of the combined image signal for each of the plurality of combined image signals combined in step S104. The focus evaluation value is obtained, for example, by extracting a high-frequency component from the spatial frequency of the synthesized composite image signal using a high-frequency transmission filter and integrating the absolute values of the extracted high-frequency component. Further, the focus evaluation value may be obtained by extracting high-frequency components using two high-frequency transmission filters having different cutoff frequencies and integrating each of the absolute values of the extracted high-frequency components.

  In step S106, the control unit 104 detects the peak of the focus evaluation value within the image composition range, and determines whether or not the peak of the focus evaluation value is detected within the image composition range. Specifically, in step S106, the control unit 104 detects a peak of the focus evaluation value by a three-point interpolation method based on at least three focus evaluation values calculated in step S105. If the focus evaluation value peak can be detected within the image composition range, it is determined that the image plane position corresponding to the subject exists within the image composition range, and the process proceeds to step S107. If it cannot be detected, it is determined that the image plane position corresponding to the subject does not exist within the image composition range, and the process proceeds to step S110.

  In step S107, the control unit 104 detects the image plane position where the focus evaluation value is maximized based on the peak of the focus evaluation value. Specifically, first, the control unit 104 synthesizes a plurality of synthesized image signals (synthesized image signals corresponding to the subject area) based on the image data of the captured image. Here, in step S107, in order to obtain an image plane position where the focus evaluation value is maximized, usually, a composite image signal corresponding to more image plane positions than the composite image signal combined in step S104 is combined. Then, based on the focus evaluation value of the synthesized composite image signal, the image plane position where the focus evaluation value is maximized is detected. In step S107, in order to obtain an image focused on the subject selected by the user, the image plane position where the focus evaluation value is maximum is detected for the subject region corresponding to the subject selected by the user. Done.

  In step S108, an image corresponding to the image plane position with the maximum focus evaluation value detected in step S107 is combined as a combined image. In subsequent step S109, the combined image combined in step S108 is displayed on the display unit. 102 is displayed on the display. As a result, the user can obtain a focused image of the selected subject.

  On the other hand, if the focus evaluation value peak cannot be detected in step S106, the process proceeds to step S110. In step S110, the control unit 104 determines whether or not to end the search for the peak of the focus evaluation value. For example, if the focus evaluation value peak cannot be detected in the entire range of the image synthesis range, it is determined that the search for the focus evaluation value peak is terminated (step S110 = YES), and the process proceeds to step S112. Is displayed, and an error message indicating that the subject selected by the user cannot be focused is displayed on the display of the display unit 102. On the other hand, when the focus evaluation value peak is not detected in the entire range of the image synthesis range, it is determined that the search for the focus evaluation value peak is continued (step S110 = NO), and the process proceeds to step S111. move on. For example, when the focus evaluation value peak cannot be detected in a part of the image composition range, and there is a range in which the focus detection peak is not detected in the image composition range. Therefore, it is determined that the search for the peak of the focus evaluation value is continued in order to detect the peak of the focus evaluation value for the remaining range in which the detection of the peak of the focus evaluation value is not performed in the image synthesis range (step S110 = NO), the process proceeds to step S111.

  In step S111, the synthesized image signal is synthesized for a new image plane position that is different from the image plane position of the synthesized image signal that has already been synthesized in the image synthesis range. Then, a focus evaluation value is calculated for the composite image signal corresponding to the new image plane position (step S105), and the focus evaluation value peak of the focus evaluation value is calculated using the focus evaluation value of the composite image signal corresponding to the new image plane position. Detection is performed (step S106). As described above, in this embodiment, until a focus evaluation value peak is detected, synthesis of synthesized image signals corresponding to different image plane positions in the image synthesis range (step S111) and detection of the focus evaluation value peak are performed. (Step S106) is repeated. If the peak of the focus evaluation value is detected (step S106 = YES), the image plane position where the focus evaluation value is maximized is detected based on the peak of the focus evaluation value (step S107). The composite image corresponding to the image plane position where the maximum is is synthesized and displayed (steps S108 and S109). On the other hand, when the focus evaluation value peak is not detected in the entire range of the image synthesis range (step S110 = YES), an error to that effect is displayed on the display of the display unit 102 (step S112).

  As described above, in the present embodiment, image data of a captured image captured by the camera 200 is captured in the personal computer 100, and the personal computer 100 corresponds to a plurality of image plane positions based on the captured image data of the captured image. A plurality of composite image signals having different image planes are combined, and based on the focus evaluation value of the combined composite image signal, the image plane position where the focus evaluation value is maximum within the image combination range is detected, and the focus evaluation value is A synthesized image corresponding to the maximum image plane position is synthesized. As described above, in the present embodiment, based on the image data of the captured image, a plurality of synthesized image signals having different image planes corresponding to the plurality of image plane positions are synthesized, and focus detection based on the focus evaluation value of the synthesized image signal is performed. By performing the above, it is possible to detect the position of the image plane focused on the subject in the captured image, and it is possible to appropriately synthesize the image focused on the subject in the captured image.

  In particular, in the present embodiment, an area in the captured image corresponding to the subject selected by the user is set as the subject area, and a composite image signal is synthesized based on the image data corresponding to the subject area. Then, an image plane position at which the focus evaluation value of the composite image signal is maximum in this subject area is detected, and a composite image corresponding to the image plane position at which the focus evaluation value is maximum in the subject area is synthesized. Thereby, in this embodiment, the focused image can be obtained about the specific subject selected by the user. Furthermore, by combining the composite image signal for the subject area among the areas in the captured image, the processing burden is reduced compared with the case of combining the composite image signal for all the areas in the captured image, and the user selects It is possible to quickly synthesize an image focused on a specific subject.

  The embodiment described above is described for facilitating the understanding of the present invention, and is not described for limiting the present invention. Therefore, each element disclosed in the above embodiment is intended to include all design changes and equivalents belonging to the technical scope of the present invention.

  For example, in the present embodiment, a predetermined area in the image selected by the user is set as the subject area via the operation unit 103, but the method for setting the subject area is not particularly limited. The control unit 104 may recognize a scene based on image data of a captured image, and set a predetermined area in the image corresponding to the recognized scene as a subject area. Specifically, when the control unit 104 recognizes a scene capturing a person based on the image data of the captured image, the controller 104 recognizes the face part of the person by template matching or the like, and determines the recognized face part. It can be set as a subject area. If the user can select a scene via the operation unit 103, the subject area may be set according to the scene selected by the user. When a scene is recognized, a pan-focus image signal with a deep focal depth may be synthesized based on the image data of the captured image, and scene recognition may be performed based on the pan-focus image signal. Thereby, a scene can be recognized more appropriately.

  Further, in the present embodiment, an example in which the image composition processing shown in FIG. 6 is performed using a single image data is shown, but the operation of the camera 1 is not limited to this, and for example, the same composition is used. When image data of a plurality of captured images obtained by continuously capturing images at different focus lens positions are captured in the personal computer 100 in a state of being associated with each other, an image is generated using the plurality of image data. It is good also as composition which performs composition processing. In this case, it is determined that the search for the peak of the focus evaluation value is finished even if the peak of the focus evaluation value cannot be detected within the image composition range based on the image data obtained at a certain focus lens position (step S110). (= YES), the peak of the focus evaluation value can be detected using other associated image data (other image data obtained at different focus lens positions). Thereby, the user can more appropriately obtain an image focused on the subject in the captured image.

  Furthermore, in step S105 to step S111 of the above-described embodiment, a known process performed in a focus detection process using a focus evaluation value such as a process for preventing false focusing or a filter process may be appropriately performed. .

  In the present embodiment, the personal computer 100 synthesizes an image focused on a predetermined subject based on the photographed image. However, the apparatus that performs such processing is not limited to the personal computer 100, for example, In this way, a camera or a printer (image output device) that can synthesize an image focused on a predetermined subject may be used.

  Also, the camera 200 of the present embodiment is not particularly limited. For example, the present invention is applied to other optical devices such as a single-lens reflex digital camera, a lens-integrated digital camera, a digital video camera, and a mobile phone camera. Also good.

100 ... Personal computer (PC)
DESCRIPTION OF SYMBOLS 101 ... Input part 102 ... Display part 103 ... Operation part 104 ... Control part 200 ... Camera 210 ... Camera body 211 ... Image pick-up element 212 ... Micro lens array 212a ... Micro lens 213 ... Photoelectric conversion element array
213a ... Photoelectric conversion element 214 ... Camera operation unit 215 ... Camera control unit 216 ... Memory 220 ... Tube 222 ... Focus lens 224 ... Aperture 225 ... Aperture drive unit 226 ... Lens drive motor 227 ... Lens control unit

Claims (5)

Input means for inputting a light receiving signal obtained by receiving the light beam in one exposure operation,
A region selecting means for selecting a partial region from within the image based on the received light signal;
Based on the light receiving signal corresponding to the partial region, a first image signal in a first image plane position of the partial region, different from said first image plane position, the second image plane of the partial region to generate a second image signal at the position,
An image for generating an image signal corresponding to an image plane position where the evaluation value is a value near the maximum value based on the evaluation value related to the contrast of the first image signal and the evaluation value related to the contrast of the second image signal. And an image synthesizing apparatus. The image composition apparatus according to claim 1,
The image generating unit on the basis of the evaluation values of at least three image signals, an image synthesizing apparatus, wherein the evaluation value to generate an image signal corresponding to the image plane position with the maximum. The image composition device according to claim 1 or 2 ,
The region selection means includes a recognition means for recognizing an image scene based on the light reception signal,
The image synthesizing apparatus according to claim 1, wherein the image generating unit determines the partial area according to a scene recognized by the recognizing unit. The image composition device according to any one of claims 1 to 3 ,
And a display unit that displays an image based on the image signal. The image composition device according to any one of claims 1 to 3 ,
A microlens array in which a plurality of microlenses are arranged two-dimensionally, and a plurality of photoelectric conversion elements provided for the plurality of microlenses, and receives a light beam from an optical system via the microlens array An image pickup device comprising: an image pickup device that inputs the received light signal obtained as described above to the input means.

Images