JP2014066959A - Imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make the exposure of each input image to be composited with a swing panorama proper.SOLUTION: A plurality of input images are sequentially photographed while a camera is swung. A segmentation area (CR) is set in each input image and the images in the segmentation areas of the respective input images are joined to each other in a swing direction, to produce a panorama composite image. When the swing direction is a right direction, the segmentation area is set on the left side from the center of the image in each input image (I"i"). In each input image, an area located in the swing direction from the segmentation area is included in an AE evaluation area. The luminance of an object in the AE evaluation area is evaluated by photometric processing, so that the luminance of the object in the segmentation area of a future input image is derived (predicted) and an exposure value is changed or fixed based on the result.

Description

  The present invention relates to an imaging apparatus such as a digital camera.

  Digital cameras often have a panorama shooting function called a swing panorama. Swing panorama is a panoramic composite image with a wider angle of view than each input image by sequentially shooting multiple input images while shaking the camera horizontally and vertically, and combining the input images by image mosaicing method. Is generated. In FIGS. 29A and 29B, reference numeral 900 represents a whole view to be acquired as a panoramic composite image, and rectangular areas 901 and 902 are taken when the i-th and (i + 1) -th input images are taken, respectively. The reference numeral 905 represents the sun as one of the subjects (i is an integer). In the examples of FIGS. 29A and 29B, the camera is swung in the right direction of the drawing during panorama shooting. In general, for example, a cutout region CR is set at the center of each input image, an image (image signal) in the cutout region CR is cut out from each input image, and the cut out images are connected along the swing direction. A panorama composite image 910 (see FIG. 30) is obtained.

  On the other hand, a digital camera is usually provided with an exposure control function for optimizing exposure of captured images. Patent Document 1 also proposes the following method. At the time of shooting the first still image (input image), the ISO sensitivity is set to the maximum, and at the subsequent shooting of the still image, the ISO sensitivity is set to the maximum and the first still image that has been shot is taken. The shutter speed is set so as to match the exposure (see claim 10 of Patent Document 1).

  In addition, when moving image shooting is performed using digital zoom or the like, as shown in FIGS. 31A and 31B, cutout regions 921 and 922 are set in shooting regions 901 and 902 corresponding to effective pixel regions of the image sensor. The moving image is generated by extracting the image signal of the clip region at each time including the clip regions 921 and 922. In the example of FIGS. 31A and 31B, the camera is swung in the right direction of the drawing during the moving image shooting (so-called pan operation is performed).

JP 2008-289095 A

  As an exposure control method during panoramic photography, a first control method is also considered in which exposure conditions (aperture value, shutter speed, photographing sensitivity, etc.) are fixed during panoramic photography. However, if the exposure conditions are fixed, the exposure near the end of shooting may become abnormal. For example, if you take a panoramic shot of a scene where the brightness of the shooting area gradually darkens along the camera swing direction with the exposure conditions fixed, the area near the start of shooting is shot with appropriate exposure. In the vicinity of the end of shooting, the image is shot with underexposure, and as a result, the obtained panoramic composite image includes a properly exposed image region and an underexposed image region.

  A method of applying continuous AE generally used as exposure control for moving image shooting to exposure control during panoramic shooting (hereinafter referred to as a second control method) is also considered. In the second control method, the photometric process is sequentially performed during panoramic photography, and the exposure condition is updated based on the photometric process result at each time (the method of Patent Document 1 is close to the second control method). In typical continuous AE, the entire shooting area of the current frame is set as the AE evaluation area, and the exposure condition of the next frame is determined according to the luminance in the AE evaluation area (that is, the overall luminance of the shooting area). When such a second control method is simply applied to the above panoramic shooting, the exposure of the next frame is subject luminance in a region that is not related to the cutout region CR (for example, the region CR in FIG. 29A). It may become abnormal under the strong influence of the subject brightness of the sun 905 located on the left side. In order to eliminate such an influence, when the left area of the cutout area CR in FIGS. 29A and 29B is simply excluded from the AE evaluation area, the area where the subject luminance can be evaluated becomes narrower and exposure control is performed. Since the amount of luminance information that can be used is reduced, there is a concern that the accuracy of exposure control may be reduced.

  Even when a moving image signal is obtained by cutting out a part of the shooting area, there is a situation similar to panoramic shooting. That is, for example, in the case where the current frame and next frame regions that form a moving image are the cutout regions 921 and 922 (FIGS. 31A and 31B), only the subject luminance in the cutout region 921 of the current frame is determined. If the exposure of the next frame is determined, the exposure of the next frame may be abnormally influenced strongly by the subject brightness in an area unrelated to the cut-out area 922. The same situation exists in focus adjustment and white balance adjustment in moving image shooting.

  Therefore, the present invention provides an imaging device that contributes to optimizing exposure of a plurality of input images to be combined, and an imaging device that contributes to optimizing moving image exposure control, focus adjustment, or white balance adjustment. Objective.

  A first imaging apparatus according to the present invention is an imaging apparatus having an imaging element that acquires an image of a subject by shooting, and a plurality of input images that are sequentially acquired by the imaging element with movement of the imaging apparatus. A synthesis processing unit that generates an output image having a wider angle of view than each input image by synthesizing the image signals in the cutout region set for each of the above, and a motion detection unit that detects the motion direction of the imaging device, A cutout region setting unit that sets the cutout region in each input image according to the movement direction; and a luminance evaluation unit that evaluates subject luminance in a region located in the movement direction from the cutout region using photometry processing And an exposure control unit that performs exposure control on the plurality of input images based on an evaluation result of the luminance evaluation unit.

  In the region located in the movement direction from the current input image cutout region, there is a subject in the input image cutout region for the next time or the next time or later. Therefore, it is possible to realize exposure control more suitable for the next or next input image by evaluating subject luminance in an area located in the movement direction from the cutout area and performing exposure control based on the evaluation result. Become. At this time, by adopting a configuration in which the clip region is set according to the movement direction, it is possible to ensure the information amount of the subject luminance to be evaluated, and the high accuracy of control and the like is promoted, and each input image is promoted. The exposure is optimized.

  A second imaging device according to the present invention is an imaging device having an imaging device that acquires an image of a subject by shooting and outputs an image signal of the acquired image, and a plurality of inputs sequentially acquired by the imaging device. Based on a moving image processing unit that generates a moving image signal from an image signal in a cutout region set for each of the images, a motion detection unit that detects the movement of the imaging device, and a detection result of the motion detection unit A cut-out area setting unit that sets the cut-out area in each input image, and performs exposure control in shooting by evaluating the subject luminance in the first evaluation area set in each input image using photometry processing. A first control unit to perform, a second control unit to perform focus adjustment in shooting based on an image signal in the second evaluation region set in each input image, and an image in the third evaluation region set in each input image A target evaluation area that includes at least one control section among the third control sections that perform white balance adjustment in photographing based on the signal, and is the first, second, or third evaluation area according to the cutout area. The target evaluation area setting unit sets an area located in the moving direction of the imaging apparatus from the cutout area when the imaging apparatus is moving in a certain direction. It is included in the target evaluation area.

  In the region located in the movement direction from the current input image cutout region, there is a subject in the input image cutout region for the next time or the next time or later. Therefore, by performing exposure control or the like by including the region positioned in the movement direction from the cutout region in the target evaluation region, it is possible to realize exposure control or the like that is more suitable for the next or subsequent input image. At this time, it is possible to secure the amount of information obtained from the target evaluation area by adopting a configuration in which the cutout area is set according to the movement of the imaging device. It is possible to obtain a moving image by various exposures.

  According to the present invention, it is possible to provide an imaging device that contributes to optimization of exposure of a plurality of input images to be combined, and an imaging device that contributes to optimization of moving image exposure control, focus adjustment, or white balance adjustment. Is possible.

1 is a schematic overall block diagram of a camera according to a first embodiment of the present invention. It is an internal block diagram of the imaging part which concerns on 1st Embodiment of this invention. It is the figure which showed the general external appearance of the camera with the subject. It is a figure for demonstrating a swing direction. It is a figure which defines each direction on an image. It is a figure which shows the mode of the input image sequence according to a swing direction. It is a block diagram of the site | part which concerns on 1st Embodiment of this invention and is related to exposure control. It is a figure which shows a mode that an AE evaluation area | region is set to an input image. It is a figure which shows the some input image arranged in a time series. It is a figure which shows the relationship between a full view and an imaging | photography area | region. It is a figure which shows each area | region which forms a full view. It is a figure which shows the imaging | photography area | region and cutting-out area | region of each time when a swing direction is a right direction. It is a figure which shows the imaging | photography area | region and cut-out area | region of each time when a swing direction is a left direction. It is a figure which shows the example of a panoramic composite image. It is a figure which shows the example of a setting of the cut-out area | region according to a swing direction. It is a figure which shows the imaging | photography area | region (input image) and cut-out area | region, etc. of a certain time. It is a figure which shows the 1st example of distribution of a subject brightness | luminance and corresponding brightness distribution. It is a figure which shows the 2nd example of distribution of object brightness | luminance and corresponding brightness distribution. It is a figure which shows the 3rd example of distribution of object brightness | luminance and corresponding brightness distribution. It is a partial block diagram of the camera concerning a 2nd embodiment of the present invention. It is a figure which shows the cut-out area | region which concerns on 2nd Embodiment of this invention. It is a figure which shows the example of a setting of the cutting-out area | region according to the motion of a camera. It is a figure which shows the example of a setting of the object evaluation area | region according to the motion of a camera. It is a figure which shows the imaging | photography area | region (input image) and cut-out area | region, etc. of a certain time. It is a figure which concerns on 2nd Embodiment of this invention and shows the 1st example of distribution of evaluation value. It is a figure which concerns on 2nd Embodiment of this invention and shows the 2nd example of distribution of evaluation value. It is a figure which concerns on 2nd Embodiment of this invention and shows the 3rd example of distribution of evaluation value. It is a figure which shows the example of a setting of the cut-out area | region and object evaluation area | region when a swing direction is a diagonally upward right direction. It is a figure which shows the relationship between the panoramic view, imaging | photography area | region, and cutting-out area | region in each time concerning the conventional panoramic imaging | photography. It is a figure which shows the panorama synthetic | combination image corresponding to Fig.29 (a) and (b). It is a figure which concerns on a prior art and is a figure which shows the imaging | photography area | region and cut-out area | region in each time at the time of video recording.

  Hereinafter, an example of an embodiment of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle. In this specification, for simplification of description, a symbol or reference that refers to information, signal, physical quantity, state quantity, member, or the like is written to indicate information, signal, physical quantity, state quantity or Names of members and the like may be omitted or abbreviated.

<< First Embodiment >>
A first embodiment of the present invention will be described. FIG. 1 is a schematic overall block diagram of a camera 1 according to the first embodiment of the present invention. The camera 1 that is an imaging device is a digital video camera that can capture and record still images and moving images, or a digital still camera that can capture and record only still images. The camera 1 may be mounted on a mobile terminal such as a mobile phone. The camera 1 is provided with each part referred by the codes | symbols 11-16. The display unit 13 may be a display device provided outside the camera 1.

  The imaging unit 11 captures a subject using an imaging element. FIG. 2 is an internal configuration diagram of the imaging unit 11. The imaging unit 11 includes an optical system 35, a diaphragm 32, an imaging device (solid-state imaging device) 33 including a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and a driver 34. ing. The optical system 35 includes a zoom lens 30 for adjusting the angle of view of the imaging unit 11 and a focus lens 31 for focusing, which are movable in the optical axis direction of the imaging unit 11. The driver 34 controls the positions of the lenses 30 and 31 and the opening of the diaphragm 32 (that is, the diaphragm value) based on the control signal from the main control unit 12. The image sensor 33 is formed by arranging a plurality of light receiving pixels in the horizontal and vertical directions, and each light receiving pixel photoelectrically converts a subject image (an optical image of the subject) incident through the optical system 35 and the diaphragm 32. As a result, the signal of the subject image is output as the output signal of the imaging unit 11.

  The main control unit 12 is formed of a signal processing circuit, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and performs predetermined signal processing on the output signal of the imaging unit 11. As a result, an image signal of a captured image of the subject by the imaging unit 11 is generated. The display unit 13 is a display device having a display screen such as a liquid crystal display panel, and displays an arbitrary video under the control of the main control unit 12. The recording medium 14 is a non-volatile memory such as a card-like semiconductor memory or a magnetic disk, and records an image signal or the like of a captured image under the control of the main control unit 12. The operation unit 15 receives various operations from the user. The operation unit 15 can be formed using a touch panel. The camera motion detection unit 16 detects the motion of the camera 1.

FIG. 3 is a diagram showing a schematic appearance of the camera 1 together with the subject SUB of the camera 1. In FIG. 3, one person is shown as an example of the subject SUB, but the subject SUB is composed of one or more arbitrary subjects. An alternate long and _short dash line AX _OPT represents the optical axis of the imaging unit 11 (the optical system 35). In this embodiment, for convenience of explanation, it is assumed that each subject including the subject SUB is stationary in real space. The captured image of the camera 1 is displayed as a moving image on the display unit 13, and the photographer can perform various shooting operations while confirming the display content of the display unit 13. Hereinafter, a still image photographed by the photographing unit 11 (that is, a still image based on an output signal of the image sensor 33) is referred to as an input image.

The motion detection targets of the camera motion detection unit 16 include the rotational motion of the camera 1 in the yaw direction, the rotational motion of the camera 1 in the pitch direction, and the translational motion of the camera 1. The rotational movements in the yaw direction and the pitch direction are movements that rotate the camera 1 and the optical axis AX _OPT in the horizontal direction and the vertical direction, respectively, with the optical center of the camera 1 fixed. The translational motion of the camera 1 is a motion that translates the camera 1 and the optical axis AX _OPT horizontally or vertically. The horizontal direction corresponds to the left-right direction, and the vertical direction corresponds to the up-down direction.

  Hereinafter, the movement direction of the camera 1 is referred to as a swing direction, and the swing direction is defined as follows. As shown in FIGS. 4A and 4B, the swing direction by the yaw-direction rotational motion for rotating the camera 1 facing the subject SUB in the right direction, the translational motion for translating to the right direction, or a combination motion thereof. Is in the right direction. As shown in FIGS. 4C and 4D, the swing direction by the yaw-direction rotational motion for rotating the camera 1 facing the subject SUB to the left, the translational motion for translating to the left, or a combination motion thereof. Is the left direction. The same applies to the upper and lower directions (see FIGS. 4E to 4H). In the following, the movement of the camera 1 with the swing directions being right, left, up and down will be expressed as the camera 1 moving in the right, left, up and down directions, respectively.

  The camera motion detector 16 can determine at least whether the swing direction is right, left, up or down (or only whether the swing direction is right or left, or Only determine whether the swing direction is up or down). The motion detection unit 16 may be able to determine the swing direction in five steps or more or continuously, or may be able to detect the speed (magnitude) of the movement of the camera 1. Of course, the motion detection unit 16 can also detect whether or not the camera 1 is stationary. The motion detection unit 16 may detect a swing direction or the like using a motion sensor (an angular acceleration sensor, an acceleration sensor, or the like) that outputs a signal according to the motion of the camera 1. Alternatively, the motion detection unit 16 may detect the swing direction or the like based on the output signal of the image sensor 33. Since a motion detection method based on the output signal of the image sensor 33 is known, a detailed description thereof is omitted. For example, the detection can be performed using an optical flow between frames (between temporally adjacent input images) derived using the representative point matching method.

An image 300 in FIG. 5 is an arbitrary input image (still image) based on an image formed in the effective pixel region of the image sensor 33. The X axis and the Y axis are axes parallel to the horizontal and vertical directions of the input image 300, respectively, and are orthogonal to each other at the center O of the input image 300. Also in the input image 300, right, left, up and down directions are defined. When the subject SUB is positioned on the right, left, upper, and lower sides of the optical axis AX _{OPT as} viewed from the camera 1, the subject SUB is located on the input image 300 obtained by photographing the subject SUB from the center O. They are located on the right, left, top, and bottom, respectively.

  On the input image sequence acquired during the period in which the camera 1 is moving in the right direction, as shown in FIG. 6A, the subject SUB moves in the left direction as time passes, and the camera 1 is moved in the left direction. On the input image sequence acquired during the moving period, as shown in FIG. 6B, the subject SUB moves in the right direction as time passes. The same applies to the top and bottom. An input image sequence refers to a collection of a plurality of input images arranged in time series. The direction in which the subject SUB moves on the input image sequence is called the image flow direction. The image flow direction is opposite to the swing direction.

  FIG. 7 is a partial block diagram of the camera 1, and each part referred to by reference numerals 21 to 26 is provided in the main control unit 12. The signal processing unit 21 performs predetermined signal processing (digitization, signal amplification, demosaicing processing, noise reduction processing, etc.) on the output signal of the image sensor 33 representing the input image.

  The exposure condition of the input image is designated by an exposure control value, and the exposure control value is composed of an aperture control value, a shutter speed control value, and a sensitivity control value. The aperture value (F value), shutter speed, and sensitivity (hereinafter also referred to as imaging sensitivity) at the time of capturing an input image are in accordance with the aperture control value, shutter speed control value, and sensitivity control value, respectively. That is, the exposure value defined by the aperture value, shutter speed, and shooting sensitivity follows the exposure control value. The brightness (luminance) of the input image can be adjusted by adjusting the aperture value, shutter speed, or shooting sensitivity. The shutter speed indicates the reciprocal of the exposure time of the image sensor 33 for obtaining the image signal of the input image. The photographing sensitivity is a sensitivity according to ISO sensitivity (sensitivity defined by International Organization for Standardization) or ISO sensitivity, and an amplification factor of an image signal including a luminance signal in an input image is adjusted by adjusting the photographing sensitivity. . In other words, the signal processing unit 21 can amplify the image signal based on the output signal itself of the image sensor 33 or the output signal of the image sensor 33 during the signal processing. The size also increases, and the brightness of the input image after the signal processing increases.

  The exposure information acquisition unit 22 acquires exposure information including the current exposure conditions (that is, the current aperture value, shutter speed, and shooting sensitivity) and predetermined target luminance.

  The luminance evaluation unit 23 evaluates subject luminance in an arbitrary region by photometry processing, and outputs evaluation luminance information including the evaluation result. The subject brightness represents the brightness of the subject in real space. As shown in FIG. 8, the luminance evaluation unit 23 sets an arbitrary area in the input image as the AE evaluation area, evaluates the subject brightness in the AE evaluation area by the photometric process, and calculates a value corresponding to the evaluated subject brightness. The AE evaluation value possessed is included in the evaluation luminance information. Arbitrary information derived by the luminance evaluation unit 23 can be included in the evaluation luminance information. In FIG. 8, the AE evaluation area is a single rectangular area, but the AE evaluation area may include a plurality of areas separated from each other, and the shape of the AE evaluation area is not limited to a rectangle. The luminance evaluation unit 23 implements photometric processing using an exposure meter that outputs a signal corresponding to the subject luminance. The luminance evaluation unit 23 may realize photometric processing based on the output signal of the image sensor 33 using the image sensor 33 as an exposure meter, or an exposure meter (not shown) provided in the camera 1 separately from the image sensor 33. ) May be used to implement photometric processing.

The exposure control unit 24 sets the exposure control value based on the exposure information and the evaluation luminance information supplied from the exposure information acquisition unit 22 and the luminance evaluation unit 23, and thereby controls the exposure control (the input image of the input image). To achieve proper exposure). For example, the brightness level of the input image output from the signal processing unit 21 is matched with the target brightness by exposure control. Since the exposure value (exposure amount) follows the exposure control value as described above, the setting of the exposure control value for a certain input image may be considered synonymous with the setting or control of the exposure value of the input image. May be considered to correspond to control of the exposure value. Referring to FIG. 9, the exposure control unit 24 can set an exposure control value at the time of shooting the input image I [i + n] based on the exposure information and evaluation luminance information regarding the input image I [i] (i is Integer, n is a natural number). For an arbitrary integer i, the input image I [i] is an input image taken and acquired at time t _i . Therefore, the input image I [i + n] is an input image taken and acquired at time t _{i + n} . Time t _{i + 1} is a time later than time t _i .

  The panorama synthesizing unit (synthesizing processing unit) 25 uses a known image mosaicing method for a plurality of input images photographed with the movement of the camera 1 in any direction during the panorama photographing period (compositing target period). Are combined to generate a panoramic composite image (output image) having a wider angle of view than the angle of view of each input image. However, the synthesizing unit 25 generates a panoramic synthesized image using the image signal of the input image after the signal processing by the signal processing unit 21 is performed. The user can specify the start and end time of the panorama shooting period via the operation unit 15. Actually, the panorama synthesizing unit 25 generates and records a panorama composite image by synthesizing and recording the image signals in the cutout region (composition target region) set for each input image during the panorama shooting period. Recording refers to recording on the recording medium 14 in FIG. 1 unless otherwise specified.

  The cutout region setting unit 26 sets a cutout region for each input image during the panorama shooting period according to the swing direction detected by the camera motion detection unit 16.

  With reference to FIGS. 10 to 14, a method for setting a cutout region according to the swing direction and the like will be described. In FIG. 10 and the like, the inside of the solid-line square frame to which reference numeral 400 is attached represents the entire view in real space that should be included in the panorama composite image, and the area within the solid-line square frame to which reference numeral SR is attached is the panorama. The imaging region of the imaging unit 11 at a certain time during the imaging period is shown. As shown in FIG. 11, the entire view 400 is formed including regions H [1] to H [13] aligned in the horizontal direction. When viewed from the camera 1, the region H [i + 1] is located to the right of the region H [i]. For simplification of description, unless otherwise specified, in the following description of the first embodiment, it is assumed that the swing direction is the right or left direction and the camera 1 moves at a constant speed during the panorama shooting period. It is assumed that the input image in the following description of the embodiment is an input image shot during the panorama shooting period.

The imaging region SR at the time t _i is particularly represented by the symbol SR [i]. The cutout area is referred to by the symbol CR. FIG. 12 shows the shooting areas SR [1], SR [7], and SR [13] at times t ₁ , t _7, and t ₁₃ superimposed on the entire view 400 when the swing direction is the right direction. When the swing direction is the right direction, the center of the imaging region SR (that is, SR [i]) at the time t _i is in the region H [i]. FIG. 13 shows the shooting areas SR [1], SR [7], and SR [13] at times t ₁ , t _7, and t ₁₃ superimposed on the entire view 400 when the swing direction is the left direction. When the swing direction is the left direction, the center of the imaging region SR (that is, SR [i]) at the time t _i is in the region H [14-i]. A captured image in the imaging region SR [i] is acquired as an input image I [i]. The imaging region SR at each time includes a plurality of regions among the regions H [1] to H [13].

  When the swing direction is right or left, the cutout region CR is set on the left or right side in the imaging region SR at each time. Specifically, when the swing direction is the right or left direction, the setting unit 26 is configured so that the position of the cutout region CR is located in the opposite direction of the swing direction when viewed from the center of the input image in each input image. The position of the cutout region CR in each input image is set.

  Regarding the attention area which is an arbitrary area, the position of the attention area refers to the center position or the gravity center position of the attention area. The attention area is, for example, an arbitrary evaluation area including a cutout area, an AE evaluation area, or an arbitrary area forming the evaluation area.

Here, for the sake of concrete explanation, when the swing direction is the right direction, it is assumed that a cutout region CR corresponding to the region H [i-4] is set in the input image I [i] (provided that “5 ≦ i ”. In the case of“ 5> i ”, the cutout region CR is set in accordance with the same principle). For example, when the swing direction is the right direction, (Fig input image I [7] region H [3] with respect to (= H [7-4]) corresponding to the cutout region CR is set at time t ₇ 11 and FIG. 12). When the swing direction is the left direction, the cutout region CR corresponding to the region H [18-i] is set in the input image I [i] (provided that “5 ≦ i” is satisfied). In the case of “5> i”, the cutout region CR is set according to the same principle). For example, when the swing direction is the left direction, (Fig input image I [7] region H [11] with respect to (= H [18-7]) corresponding to the cutout region CR is set at time t ₇ 11 and FIG. 13).

  The panorama composition unit 25 can generate an image obtained by joining the image signals in the cutout region CR of the input images I [1] to I [13] along the swing direction as a panorama composite image. However, when the swing direction is the right direction, the panorama synthesis unit 25 includes the image signal of the right side of the cutout region CR of the input image I [13] at the right end of the panorama composite image, and the swing direction is the left direction. In this case, the panorama composition unit 25 includes the image signal of the left region of the cutout region CR of the input image I [13] at the left end of the panorama composite image. Thereby, the panorama composite image 410 shown in FIG. 14 can be obtained both when the swing direction is the right direction and when the swing direction is the left direction.

  In FIGS. 15A and 15B, an area within a broken-line rectangle denoted by reference numeral 420 represents an AE evaluation area. FIGS. 15A and 15B show the relationship between the cutout region CR and the AE evaluation region when the swing direction is right and left, respectively.

  Regardless of whether the swing direction is right or left, in each input image, the AE evaluation region includes a region located in the swing direction from the cutout region CR. That is, when the swing direction is the right direction, the luminance evaluation unit 23 includes at least an area located on the right side of the cutout area CR in the AE evaluation area (see FIG. 15A), and the swing direction is the left direction. In addition, at least a region located on the left side of the cutout region CR is included in the AE evaluation region (see FIG. 15B).

  Further, regardless of whether the swing direction is right or left, in each input image, the AE evaluation area includes all or part of the cutout area CR, but the AE evaluation area does not include the cutout area CR at all. It is also possible to set the AE evaluation area so that it is not included (so that the AE evaluation area and the cutout area CR do not partially overlap).

  The luminance evaluation unit 23 may variably set the position and size of the AE evaluation area in each input image according to the swing direction. For example, in each input image, the position (center or centroid position) of the AE evaluation area when the swing direction is the right direction is greater than the position (center or centroid position) of the AE evaluation area when the swing direction is the left direction. May be arranged on the right side. However, the position and size of the AE evaluation area in each input image may not depend on the swing direction (in this case, the AE evaluation area in FIG. 15A and the AE evaluation area in FIG. Will be exactly the same).

[Example of exposure control]
A specific example of exposure control will be described with the assumption that the swing direction is the right direction. Here, shown in FIG. 16, considered relative to the imaging region SR [5] and the input image I [5] at time t _5, the photometry in the light measurement process (i.e. the time t ₅ the time of acquisition of the input image I [5] Process) is referred to as a reference photometric process, and the input image I [5] is also referred to as a reference input image. Under the above assumption for realizing the description (the assumption described with reference to FIG. 12), the cutout region CR of the input image I [5] corresponds to the region H [1]. By the reference photometry process, the subject luminance in the AE evaluation area of the input image I [5] is evaluated. The AE evaluation area of the input image I [5] includes areas H [2] to H [9] positioned in the swing direction from the cutout area CR of the input image I [5], and the cutout area of the input image I [5]. It may further include CR (ie, region H [1]).

  The exposure control unit 24 uses the reference input image based on the evaluation result of the subject brightness in the regions H [2] to H [9] or the subject brightness in the regions H [1] to H [9] based on the reference photometry process. It is possible to perform exposure control value setting and exposure control for an input image (all or part of I [6] to I [13]) taken after (I [5]).

  Specifically, in the reference photometry process, the subject luminances of the regions H [1] to H [9] corresponding to the cutout region CR of the input images I [5] to I [13] are derived individually. The subject brightness in the regions H [1] to H [9] derived by the reference photometry process is represented by symbols BV [1] to BV [9], respectively. When the AE evaluation area of the input image I [5] does not include the area H [1], the subject brightness BV [1] of the area H [1] is not derived by the reference photometry process.

  The exposure control unit 24 performs a reference based on the change state of the subject brightness along the swing direction in the region including the regions H [2] to H [9] or the region including the regions H [1] to H [9]. An exposure control value (and therefore an actual exposure value) between the input image (I [5]) and an input image (all or a part of I [6] to I [13]) taken after the reference input image. ) To decide whether to change. Determining that the change is to be made is called change determination, and deciding not to make the change is called non-change determination. When the exposure control unit 24 makes a change determination, for example, when the exposure control value (and hence the actual exposure value) of the input image I [6] is different from that of the input image I [5], and the non-change determination is made. , At least the exposure control value (and hence the actual exposure value) of the input image I [6] is matched with that of the input image I [5].

  In panoramic photography, when there is a global brightness change, it is desirable to follow the exposure value with the brightness change. However, if the exposure value is made to follow the local brightness change, the panorama composite image In some cases, the continuity of luminance is impaired and an unnatural panoramic composite image is obtained. Considering this, it is preferable to perform the following change determination / non-change determination.

  The exposure control unit 24 regards the subject brightness BV [j] as a function of the variable j (where j is an integer from 2 to 9, or an integer from 1 to 9). Then, as shown in FIG. 17A, in the range satisfying “2 ≦ j ≦ 9” (or the range satisfying “1 ≦ j ≦ 9”), the subject brightness BV [j] is increased with respect to the increase of the variable j. When increasing or decreasing, non-change determination is performed. In other words, in the function, when there are both a section in which the subject brightness BV [j] increases and a section in which the subject brightness BV [j] decreases with respect to the increase in the variable j, the non-change determination is performed. In such a case, as shown in FIG. 17 (b), it is considered that the subject brightness varies locally along the swing direction.

  When the exposure value is caused to follow a local variation in subject luminance, for example, an image of a region H [1] with a high exposure value (a cutout region CR of the input image I [5]) and a region H [2 with a high exposure value. ] A panorama composite image including the image of the cutout area CR of the input image I [6] and the image of the low exposure value area H [3] (the cutout area CR of the input image I [7]) adjacent to each other. In some cases, the brightness of the panorama composite image changes unnaturally at the boundary between these regions. For this reason, in the cases as shown in FIGS. 17A and 17B, the non-change determination is performed to fix the exposure value.

  On the other hand, as shown in FIGS. 18A and 18B, the exposure control unit 24 performs subject brightness BV [in a range satisfying “2 ≦ j ≦ 9” (or a range satisfying “1 ≦ j ≦ 9”). When j] decreases monotonously with respect to the increase in variable j, a change determination is made, and the exposure value is changed from the input image I [5] toward the input image I [13] toward the high exposure side. Conversely, when the subject brightness BV [j] monotonously increases with respect to the increase of the variable j, the change determination is performed, and the exposure value is reduced from the input image I [5] to the input image I [13]. Change to the side. In the cases shown in FIGS. 18A and 18B, the subject brightness decreases globally toward the right side of the entire view 400. Therefore, changing the exposure value in accordance with the subject brightness changes the entire panorama composite image. This is because it is suitable for proper exposure. The same applies when a monotonous increase in subject brightness is observed. The change of the exposure value to the high exposure side is realized by decreasing the aperture value, decreasing the shutter speed (that is, increasing the exposure time), or increasing the photographing sensitivity. The change of the exposure value to the low exposure side is realized by increasing the aperture value, increasing the shutter speed (that is, decreasing the exposure time), or decreasing the photographing sensitivity.

  Further, as in the case shown in FIGS. 19A and 19B, the exposure control unit 24 has a predetermined difference between the subject brightness BV [1] and the average value of the subject brightness BV [2] to BV [9]. Change determination is performed when the variance of the subject brightness BV [2] to BV [9] is equal to or less than a predetermined value. If the subject brightness BV [1] is higher than the average value of the subject brightness BV [2] to BV [9], the input image I [6] is made higher than the exposure value of the input image I [5], and the subject If the luminance BV [1] is lower than the average value of the subject luminance BV [2] to BV [9], the input image I [6] is made lower than the exposure value of the input image I [5]. In the cases shown in FIGS. 19A and 19B, since a global decrease in subject luminance is observed between the regions H [1] and H [2], the exposure value is changed following that. This is because it is suitable for optimizing the overall exposure of the panorama composite image. The same applies when a global increase in subject brightness is observed.

  Although not particularly shown, when the subject brightness BV [1] and the subject brightness BV [2] to BV [9] match, the non-change determination is made. When the difference between the subject brightness BV [1] and the subject brightness BV [j] is less than or equal to a predetermined value (j is an integer between 2 and 9), the subject brightness BV [1] and the subject brightness BV [j] match. You may consider that

  Since the cutout region CR and the AE evaluation region are set as described above, the luminance evaluation unit 23 according to the first embodiment evaluates the subject luminance in the region located in the swing direction from the cutout region CR using photometry processing. be able to. Regarding the input image I [i], a subject in the cutout area CR of the input image taken after the input image I [i] exists in the area located in the swing direction from the cutout area CR. For this reason, the luminance evaluation unit 23 can predict the subject luminance in the cutout region CR of the input image taken after the input image I [i] at the stage when the input image I [i] is acquired. The exposure control unit 24 can optimize exposure control for an input image captured after the input image I [i] using the prediction result (from BV [2] to BV [9]). Acquisition is equivalent to prediction). At this time, by setting the cutout region CR according to the swing direction (specifically, as shown in FIGS. 12, 13, 15A and 15B, the reverse of the swing direction). By arranging the cutout region CR in a biased direction), the predictable region is expanded. By expanding the predictable area, the amount of information obtained from the predictable area increases, and it becomes possible to analyze in detail what exposure control is optimal for subsequent input images. . As a result, the exposure control of the input image to be synthesized can be made more appropriate.

  Specifically, by expanding the predictable area, it is possible to confirm in detail whether or not there is a global luminance change in which the exposure value should be changed (the cutout area CR is fixed at the center of the input image). In this case, the predictable area becomes narrow and it becomes difficult to detect a global luminance change along the swing direction). Also, the existing luminance change is a local luminance change whose exposure value should be fixed. It is possible to accurately determine whether there is any. The exposure control of each input image can be optimized according to the confirmation and determination results.

<< Second Embodiment >>
A second embodiment of the present invention will be described. The second embodiment is an embodiment based on the first embodiment. With respect to matters not specifically described in the second embodiment, the description of the first embodiment is the second embodiment unless otherwise specified and there is no contradiction. Also applies to form.

  In the second embodiment, characteristic operations of moving image shooting and recording in the camera 1 will be described. FIG. 20 is a partial block diagram of the camera 1 particularly related to the characteristic operation of the second embodiment. Each part referred by the codes | symbols 51-56 can be provided in the main control part 12 of FIG. However, any one or two of the control units 53, 54, and 55 can be omitted. As shown in FIG. 21, the effective pixel area of the image sensor 33 is considered to correspond to the imaging area SR and to the entire area of the input image. In the second embodiment, the image in the cutout region CR set in the input image (effective pixel region) corresponds to a frame forming the moving image 500 (see FIG. 20).

  The imaging unit 11 sequentially captures the subject at a predetermined frame period, and accordingly, an image at each time is formed in the cutout region CR. The moving image processing unit 51 generates a moving image 500 having an image at each time in the cutout region CR as a frame at each time. That is, the processing unit 51 generates an image signal of the moving image 500 from the image signal in the cutout region CR set for each of the plurality of input images sequentially acquired by the image sensor 33. The processing unit 51 can record the image signal of the moving image 500 on the recording medium 14. The input image in the second embodiment refers to an input image acquired during generation or recording of the moving image 500.

  Although the cutout area CR can match the entire area of the input image, in the second embodiment, the cutout area CR is smaller than the entire area of the input image. The camera 1 can perform so-called digital zoom, and in particular, for example, when performing digital zoom, the cutout region CR can be made smaller than the entire region of the input image. In the following, for convenience of explanation, it is assumed that the size of the cutout region CR is smaller than the entire region of the input image and is unchanged.

  The cutout region setting unit 52 sets a cutout region CR for each input image based on the detection result of the camera motion detection unit 16. The motion detection unit 16 uses a motion sensor (an angular acceleration sensor, an acceleration sensor, or the like) that outputs a signal corresponding to the motion of the camera 1, or based on the output signal of the image sensor 33, the camera 1 moves in a certain direction. It is possible to detect whether the camera 1 is moving in a certain direction, and it is possible to detect the direction of the movement (that is, the swing direction described above). A known pan / tilt determination method for determining the presence or absence of a pan or tilt operation can be used to detect whether or not the camera 1 is moving in a certain direction.

The detection that the camera 1 is not moving in a certain direction is called non-swing detection. The detection that the camera 1 is moving in the right direction (that is, the swing direction is the right direction) is called right swing detection, and the camera 1 is moving in the left direction (that is, the swing direction is the left direction). ) Is called left swing detection. Regions CR _O , CR _A , and CR _B in FIGS. 22A, 22B, and 22C are set in each input image when a non-swing is detected, when a right swing is detected, and when a left swing is detected, respectively. A cutout region CR is shown. The center of the cutout region CR _O coincides with the center of the input image, for example. Each of the clip region CR cutout region viewed from _O CR _A and CR _B is displaced in the direction opposite to the swing direction. That is, the setting unit 52 shifts the cutout region (shift cutout region CR _A ) at the time of detecting the right swing to the left as viewed from the cutout region at the time of non-swing detection (reference cutout region CR _O ), and detects the left swing. The cutout area (shift cutout area CR _B ) is not shifted to the right.

  The AE control unit 53 corresponds to a part obtained by combining the exposure information acquisition unit 22, the luminance evaluation unit 23, and the exposure control unit 24 of FIG. The AE control unit 53 evaluates the subject luminance within the AE evaluation area set for each input image using photometry processing, and sets an exposure control value for photographing each input image using the evaluation result (that is, exposure). Control). When the setting of the exposure control value includes the setting of the photographing sensitivity (see FIG. 7), the signal amplification process corresponding to the set photographing sensitivity is performed in the image signal generation process of the moving image 500. Is executed.

  The AF evaluation unit 54 performs focus adjustment in photographing each input image based on the image signal in the AF evaluation area set for each input image. The focus adjustment is realized by adjusting the positional relationship between the image sensor 33 and the focus lens 31 so that an image of a subject is formed on the image sensor 33 (see FIG. 2). The adjustment of the positional relationship is realized by adjusting the position of the focus lens 31 disposed between the subject and the image sensor 33. However, the imaging unit 11 is formed so that the imaging device 33 can move in the optical axis direction of the imaging unit 11, and the positional relationship is adjusted by adjusting the position of the focus lens 31 or the imaging device 33 or the focus lens 31. Alternatively, it may be realized by adjusting the position of the image sensor 33.

  The AWB control unit 55 performs white balance adjustment in shooting (that is, white balance adjustment of each input image) based on the image signal in the AWB evaluation area set for each input image. In the white balance adjustment, a predetermined color correction process according to the color temperature of the subject in the AWB evaluation area based on the image signal in the AWB evaluation area is performed in the generation process of the image signal of the moving image 500. Is executed. As a result, each frame of the moving image 500 becomes an image after white balance adjustment.

  The evaluation region setting unit (target evaluation region setting unit) 56 sets AE, AF, and AWB evaluation regions according to the cutout region CR. Of the AE, AF, and AWB evaluation areas, an evaluation area that depends on the cutout area CR is referred to as a target evaluation area. Each of the AE, AF, and AWB evaluation areas may be the target evaluation area, and any one or two of the AE, AF, and AWB evaluation areas may be the target evaluation area.

In FIGS. 23A, 23B, and 23C, regions TER _O , TER _A , and TER _B are set for each input image when non-swing is detected, when right swing is detected, and when left swing is detected, respectively. Represents the target evaluation area.

The target evaluation area TER _O set at the time of non-swing detection consists of all or part of the cut-out area CR _O. The region outside of the clip region CR _O is may also be further included in the target evaluation region TER _O.

Regardless of whether the right swing is detected or the left swing is detected, in each input image, the target evaluation region includes a region located in the swing direction from the cutout region CR. That is, the setting unit 56, when the right swing detection, including at least the target evaluation area a region located to the right of the cut-out region CR _A (see FIG. 23 (b)), at the left swing detection, on the left side of the cutout region CR _B The located area is included in at least the target evaluation area (see FIG. 23C).

  Moreover, in either of the right swing detection and the left swing detection, in each input image, the target evaluation area includes all or part of the cutout area CR, but the target evaluation area includes the cutout area CR. It is also possible to set the target evaluation area so that it is not included at all (so that the target evaluation area and the cutout area CR do not partially overlap).

The setting unit 56 may variably set the position and size of the target evaluation area in each input image according to the swing direction. For example, in each input image, the position of the target evaluation area (center or barycentric position) when the swing direction is the right direction is greater than the position of the target evaluation area (center or barycentric position) when the swing direction is the left direction. May be arranged on the right side. However, the position and size of the target evaluation area in each input image may not depend on the swing direction (in this case, the target evaluation areas TER _A and TER _B in FIGS. 23A and 23B are exactly the same). To become a thing). In the present embodiment, it is assumed that the target evaluation area is a rectangular area, but the shape is not limited to a rectangle.

[Example of exposure control]
A specific example of exposure control will be described with the assumption that the swing direction is the right direction. Here, the imaging region SR [1] and the input image I [1] at time t ₁ shown in FIG. 24 are considered as a reference. Therefore, the reference input image according to the second embodiment is the input image I [1].

The shooting area SR [1] and the input image I [1] include areas L [1] to L [9] arranged in the horizontal direction. For any integer j, region L [j + 1] is adjacent to the right side of region L [j]. The cutout region CR _A set for the input image I [1] includes the region L [1], but does not include the regions L [2] to L [9]. The setting unit 56 (or the control unit 53, 54, or 55) assumes a region L in the cutout region CR _A of the input image I [i] at time t _i under the assumption that the speed of movement of the camera 1 is kept constant. The horizontal size of the region L [i] is defined based on the speed of movement of the camera 1 so that the subject in [i] is included (here, as a result of this definition, the shooting region SR [1] and Consider the case where the input image I [1] includes regions L [1] to L [9]). The target evaluation region of the input image I [1] includes regions L [2] to L [9] positioned in the swing direction from the cutout region CR of the input image I [1], and may further include a region L [1]. .

--- AE change judgment / non-change judgment by AE control section ---
The reference photometry processing in the second embodiment is photometry processing at the time of acquisition of the input image I [1] (that is, photometry processing at time t ₁ ). The subject luminance in the AE evaluation area of the input image I [1] is evaluated by the reference photometry process. The AE control unit 53 uses the reference input image based on the evaluation result of the subject brightness in the regions H [2] to H [9] or the subject brightness in the regions H [1] to H [9] based on the reference metering process. An exposure control value is set and exposure control is performed for an input image (all or part of I [2] to I [9]) taken after (I [1]).

The AE control unit 53 includes subjects in the areas L [1] to L [9] that are or are expected to be included in the cutout area CR _A of the input images I [1] to I [9] in the reference photometry process. The brightness can be obtained individually, and the values of the subject brightness in the areas L [1] to L [9] obtained by the reference photometry processing are respectively obtained as evaluation values VAL _AE [1] to VAL _AE [9]. Represent. When the AE evaluation area of the input image I [1] does not include the area L [1], the evaluation value VAL _AE [1] is not derived by the reference photometry process.

The AE control unit 53 subjects brightness (VAL _AE [j]) along the swing direction in the region including the regions L [2] to L [9] or the region including the regions L [1] to L [9]. Exposure control between a reference input image (I [1]) and an input image (all or part of I [2] to I [9]) taken after the reference input image based on the change state of Determine whether to change the value (and hence the actual exposure value). Determining that the change is made is called AE change determination, and deciding not to make the change is called AE non-change determination. When making the AE change determination, for example, the AE control unit 53 makes the AE non-change determination by making the exposure control value (and hence the actual exposure value) of the input image I [2] different from that of the input image I [1]. When this is done, at least the exposure control value (and hence the actual exposure value) of the input image I [2] is matched with that of the input image I [1].

--- AF change judgment / non-change judgment by AF control section ---
The AF control unit 54 is in focus in the AF evaluation area of the input image I [1] by the AF evaluation process based on the image signal of the AF evaluation area of the input image I [1] (whether or not it is in focus). To evaluate. The AF control unit 54 performs processing for obtaining an evaluation value VAL _AF [j] that is an AF evaluation value of the region L [j] based on the image signal in the region L [j] of the input image I [1]. ] To L [9] to obtain evaluation values VAL _AF [1] to VAL _AF [9]. The evaluation value VAL _AF [j] has a value proportional to the contrast of the image in the area L [j] of the input image I [1]. Among the spatial frequency components in the region L [j] of the input image I [1], spatial frequency components having a predetermined frequency or higher are extracted using a high-pass filter or the like, and the extracted components are integrated in the region L [j]. As a result, the evaluation value VAL _AF [j] can be obtained. The evaluation value VAL _AF [j] represents the in-focus state in the area L [j] of the input image I [1], and the evaluation value VAL _AF is increased as the subject in the area L [j] is in focus. [J] increases.

Input AF control unit 54, which based on the evaluation value _{VAL AF [2] ~VAL AF [} 9] or _{VAL AF [1] ~VAL AF [} 9], the reference input image (I [1]) is taken after the It is possible to determine the position of the focus lens 31 at the time of shooting an image (all or part of I [2] to I [9]) (that is, focus adjustment can be performed).

At this time, the AF control unit 54 changes the focus state along the swing direction in the region including the regions L [2] to L [9] or the region including the regions L [1] to L [9]. Based on (VAL _AF [j]), between the reference input image (I [1]) and an input image (all or part of I [2] to I [9]) taken after the reference input image Thus, it is determined whether or not to change the position of the focus lens 31. Determining that the change is made is called AF change determination, and deciding not to make the change is called AF non-change determination. When the AF control unit 54 makes an AF change determination, for example, when the position of the focus lens 31 at the time of shooting the input image I [2] is made different from that of the input image I [1], an AF non-change determination is made. At least the position of the focus lens 31 at the time of photographing the input image I [2] is made to coincide with that of the input image I [1].

Alternatively, the AF control unit 54 performs the process of obtaining the subject distance of the subject in the region L [j] of the input image I [1] on each of the regions L [1] to L [9], thereby evaluating the evaluation value. VAL _AF [1] may also be determined ~VAL _AF [9], the evaluation value VAL _AF [j] in the case, the input image I [1] region L [j] of the subject distance of the subject present in the Has a value. The subject distance of a certain subject refers to the distance in real space between the subject and the camera 1. There is parallax between a plurality of input images obtained while moving the camera 1. Therefore, the AF control unit 54 uses the image signals of a plurality of input images including the image signals of the subjects in the regions L [1] to L [9], and evaluates each subject distance using the principle of triangulation. can determine the value _{VAL AF [1] ~VAL AF [} 9]. When the evaluation value VAL _AF [j] indicating the subject distance is obtained, the AF control unit 54 includes the area including the areas L [2] to L [9] or the area including the areas L [1] to L [9]. The AF change determination or the AF non-change determination may be selectively made based on the change state of the subject distance (VAL _AF [j]) along the swing direction.

--- AWB change judgment / non-change judgment by AWB control part ---
The AWB control unit 55 performs processing for _{obtaining the} evaluation value VAL _AWB [j], which is the AWB evaluation value of the region L [j], based on the image signal in the region L [j] of the input image I [1]. ] To L [9] to obtain evaluation values VAL _AWB [1] to VAL _AWB [9]. The evaluation value VAL _AWB [j] indicates the color temperature in the region L [j] of the input image I [1].

AWB control unit 55, based on the evaluation value VAL _AWB based on the image signal of the AWB evaluation area [2] ~VAL _AWB [9] or _{VAL AWB [1] ~VAL AWB [} 9] of the input image I [1], the reference White balance adjustment can be performed on an input image (all or part of I [2] to I [9]) taken after the input image (I [1]) (that is, the content of the color correction process). Can be determined).

The AWB control unit 55 changes the color temperature along the swing direction (VAL _AWB [VAL] in the region including the regions L [2] to L [9] or the region including the regions L [1] to L [9]. j]) and applied between the reference input image (I [1]) and the input image (all or part of I [2] to I [9]) taken after the reference input image. It is determined whether or not to change the color correction process. Determining that the change is made is called AWB change determination, and deciding not to make the change is called AWB non-change determination. When the AWB change determination is made, for example, the AWB control unit 55 makes the color correction processing applied to the image signal of the cutout region CR of the input image I [2] different from that of the input image I [1], and the AWB non- When the change determination is made, at least the color correction processing applied to the image signal in the cutout region CR of the input image I [2] is made to match that of the input image I [1].

--- Specific distinction between change judgment / non-change judgment ---
When a global luminance change, in-focus state change (or subject distance change), or color temperature change is observed during shooting and recording of the moving image 500, the exposure value and the position of the focus lens 31 are changed according to each change. It is desirable to change the color correction process. However, if such a change is also performed for a local change, the continuity of luminance is lost on the moving image 500 and the moving image 500 flickers, or a moving image is displayed. There is a possibility that the subject distance in focus on the image 500 varies excessively and an unnatural moving image 500 is obtained, or the color of an object in the moving image 500 varies unnaturally. Considering this, the following change determination / non-change determination can be performed. Since the following change determination / non-change determination can be applied to each of the control units 53, 54, and 55, the control unit that performs the following change determination / non-change determination is referred to as the target control unit in order to explain them collectively. Call it. The target control unit is the control unit 53, 54, or 55. In the following description, evaluation value VAL [j], change determination, and non-change determination indicate evaluation value VAL _AE [j], AE change determination, and AE non-change determination when the target control unit is AE control unit 53. When the target control unit is the AF control unit 54, the evaluation value VAL _AF [j], AF change determination, AF non-change determination is indicated, and when the target control unit is the AWB control unit 55, the evaluation value VAL _AWB [J], AWB change determination, AWB non-change determination.

  The target control unit regards the evaluation value VAL [j] as a function of the variable j (where j is an integer of 2 to 9 or an integer of 1 to 9). Then, as shown in FIG. 25, in the range satisfying “2 ≦ j ≦ 9” (or the range satisfying “1 ≦ j ≦ 9”), the evaluation value VAL [j] increases as the variable j increases. If it decreases, a non-change determination is performed. That is, in the function, when there are both a section in which the evaluation value VAL [j] increases and a section in which the evaluation value VAL [j] decreases with respect to the increase in the variable j, the non-change determination is performed. This is because the change in the evaluation value in such a case is considered to correspond to a local change.

  On the other hand, as shown in FIG. 26, the target control unit increases the variable j in the evaluation value VAL [j] in a range satisfying “2 ≦ j ≦ 9” (or a range satisfying “1 ≦ j ≦ 9”). On the other hand, when it decreases monotonously, a change determination is performed. Conversely, a change determination is also made when the evaluation value VAL [j] monotonically increases with respect to the increase of the variable j. This is because the change in the evaluation value in such a case is considered to correspond to a global change.

  In addition, as in the case shown in FIG. 27, the target control unit has a difference between the evaluation value VAL [1] and the average value of the evaluation values VAL [2] to VAL [9] equal to or greater than a predetermined difference threshold value. When the variance of the evaluation values VAL [2] to VAL [9] is equal to or less than a predetermined value, a change determination is performed. This is because the change in the evaluation value in such a case is considered to correspond to a global change.

  Although not particularly shown, when the evaluation value VAL [1] and the evaluation values VAL [2] to VAL [9] match, the non-change determination is made. When the difference between the evaluation value VAL [1] and the evaluation value VAL [j] is less than or equal to a predetermined value (j is an integer of 2 to 9), the evaluation value VAL [1] and the evaluation value VAL [j] match. You may consider that

  By setting the cutout region CR and the target evaluation region as in this embodiment, the control units 53, 54, and 55 allow the subject brightness, focus state or subject distance, white in the region located in the swing direction from the cutout region CR. The balance state (color temperature) can be evaluated. Regarding the input image I [i], a subject in the cutout area CR of the input image taken after the input image I [i] exists in the area located in the swing direction from the cutout area CR. For this reason, the control units 53, 54, and 55, when the input image I [i] is acquired, the subject brightness and focus state in the cutout region CR of the input image taken after the input image I [i]. Alternatively, the subject distance and the white balance state (color temperature) can be predicted, and the prediction result is used to perform exposure control, focus adjustment, and white balance adjustment for an input image taken after the input image I [i]. It becomes possible to optimize (acquisition of evaluation values VAL [2] to VAL [9] corresponds to prediction). At this time, by setting the cutout region CR according to the swing direction (specifically, as shown in FIGS. 23B and 23C, the cutout region CR is set in the direction opposite to the swing direction. The predictable area is expanded (by being biased). By expanding the predictable area, the amount of information obtained from the predictable area increases, and details on what exposure control, focus adjustment, and white balance adjustment are optimal for subsequent input images Can be analyzed and, as a result, they can be made more appropriate.

  More specifically, the presence / absence of a global luminance change for which the exposure value should be changed and the position change of the focus lens 31 (and / or the image sensor 33) should be changed by expanding the predictable region. It is possible to confirm in detail whether or not there is a change in the in-focus state or subject distance, and whether or not there is a global color temperature change that should be changed in the color correction processing for white balance (the cutout region CR is input image) If they are fixed in the center, the predictable areas become narrower and it is difficult to detect their global changes). Also, whether the existing luminance change corresponds to a local luminance change whose exposure value should be fixed, or the existing focus state or subject distance change is the focus lens 31 (and / or the image sensor 33). ) Is equivalent to a local focus state change or subject distance change to be fixed, or whether the existing color temperature change is a local color temperature change to which the color correction processing should be fixed It becomes possible to judge accurately. The exposure control, focus adjustment, and white balance adjustment of each input image can be optimized according to the confirmation and determination results.

<< Deformation, etc. >>
The embodiment of the present invention can be appropriately modified in various ways within the scope of the technical idea shown in the claims. The above embodiment is merely an example of the embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the above embodiment. The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. As annotations applicable to the above-described embodiment, annotation 1 and annotation 2 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
In each of the above-described embodiments, it is mainly assumed that the swing direction is right or left. However, when the swing direction is upward or downward, or when the swing direction is any direction other than up / down / left / right. In addition, the above-described processes including the setting of the cut-out area and the arbitrary evaluation area are performed according to the same principle as described above. For example, if the swing direction is upward, “right” and “left” in the above explanations assuming that the swing direction is rightward may be read as “up” and “down”, respectively. .

The same applies when the swing direction is oblique. For example, when the swing direction in the second embodiment is a upper right direction, it is possible to set the cut-out region CR _C and the target evaluation region TER _C in Figure 28. Cutout region CR _C, when viewed from the cutout region CR _O at the time of non-swing detection (see FIG. 23 (a)), are offset in the direction opposite to the swing direction. In the input image, the target evaluation region TER _C is the enclosing region located from the cutout region CR _C in the swing direction may be further contained all or part of the region CR _C cut.

[Note 2]
The target device that is the main control unit 12 or the camera 1 can be configured by hardware or a combination of hardware and software. Arbitrary specific functions that are all or part of the functions realized in the target device are described as a program, the program is stored in a flash memory that can be mounted on the target device, and the program is executed by the program execution device. The specific function may be realized by executing on a microcomputer (for example, a microcomputer that can be mounted on the target device). The program can be stored and fixed in an arbitrary recording medium (not shown). A recording medium (not shown) for storing and fixing the program may be mounted or connected to a device (such as a server device) different from the target device.

DESCRIPTION OF SYMBOLS 1 Camera 11 Imaging part 16 Camera motion detection part 23 Luminance evaluation part 24 Exposure control part 25 Panorama synthetic | combination part 26 Cutout area setting part 33 Image pick-up element 51 Moving image process part 52 Clipping area setting part 53 AE control part 54 AF control part 55 AWB Control unit 56 Evaluation area setting unit

Claims (9)

An imaging device having an imaging element that acquires an image of a subject by shooting,
An output image having a wider angle of view than each input image by synthesizing the image signals in the cut-out area set for each of the plurality of input images sequentially acquired by the imaging device with the movement of the imaging device A synthesis processing unit for generating
A motion detector that detects the direction of motion of the imaging device;
A cutout region setting unit that sets the cutout region in each input image according to the movement direction;
A luminance evaluation unit that evaluates subject luminance in a region located in the movement direction from the cutout region using photometry processing;
An image pickup apparatus comprising: an exposure control unit that performs exposure control on the plurality of input images based on an evaluation result of the luminance evaluation unit. The cutout region setting unit sets the position of the cutout region in each input image so that the center or center of gravity of the cutout region is located in the opposite direction of the movement direction when viewed from the image center in each input image. The imaging apparatus according to claim 1. The luminance evaluation unit evaluates subject luminance in an area located in the movement direction from a cutout area of the reference input image by reference photometry processing when a reference input image that is one of the plurality of input images is acquired. And
The imaging apparatus according to claim 1, wherein the exposure control unit performs exposure control on an input image acquired after the reference input image based on the evaluation result. The exposure control unit is based on a change state of subject luminance along the movement direction in an area located in the movement direction from the cut-out area of the reference input image, or the cut-out area of the reference input image and the reference Between the reference input image and the input image acquired after the reference input image, based on the change state of the subject brightness along the movement direction in the region located in the movement direction from the cut-out region of the input image. The imaging apparatus according to claim 3, wherein it is determined whether or not to change an exposure value. An imaging apparatus having an imaging element that acquires an image of a subject by shooting and outputs an image signal of the acquired image,
A moving image processing unit that generates an image signal of a moving image from an image signal in a cutout region set for each of a plurality of input images sequentially acquired by the imaging device;
A motion detector for detecting the motion of the imaging device;
A cutout region setting unit that sets the cutout region in each input image based on the detection result of the motion detection unit, and
A first control unit that performs exposure control in shooting by evaluating subject luminance within a first evaluation region set for each input image using photometry processing;
A second control unit that performs focus adjustment in shooting based on an image signal in the second evaluation region set for each input image; and
At least one control unit is included in a third control unit that performs white balance adjustment in shooting based on an image signal in the third evaluation region set for each input image, and
A target evaluation area setting unit that sets a target evaluation area that is the first, second, or third evaluation area in each input image according to the cutout area,
The target evaluation area setting unit includes, in the target evaluation area, an area located in the movement direction of the imaging apparatus from the cutout area when the imaging apparatus is moving in a certain direction. The motion detection unit detects whether the imaging device is moving in a certain direction, and detects the movement direction when the imaging device is moving in a certain direction,
The cutout region setting unit detects that the imaging device is moving in a fixed direction when viewed from the position of a reference cutout region that is the cutout region when it is detected that the imaging device is not moving in a fixed direction. Shift the position of the shift cutout area that is the cutout area in the case of the
When it is detected that the imaging device does not move in a certain direction, the target evaluation region setting unit sets a part or all of the reference cutout region as the target evaluation region, and the imaging device moves in a certain direction. 6. The imaging apparatus according to claim 5, wherein a region located in the movement direction when viewed from the shift cut-out region is included in the target evaluation region. The imaging apparatus includes the first control unit, the target evaluation area is the first evaluation area,
The plurality of input images include a reference input image acquired during a period in which the imaging device is moving in a certain direction,
During the period, the first control unit evaluates subject luminance in a region located in the movement direction from the shift cutout region of the reference input image by reference photometry processing when the reference input image is acquired, The imaging apparatus according to claim 6, wherein exposure control is performed on an input image acquired after the reference input image based on the evaluation result. The imaging apparatus includes the second control unit, the target evaluation area is the second evaluation area,
The plurality of input images include a reference input image acquired during a period in which the imaging device is moving in a certain direction,
During the period, the second control unit performs focus adjustment on an input image acquired after the reference input image based on an image signal in an area located in the movement direction from a shift cut-out area of the reference input image. The imaging device according to claim 6, wherein the imaging device is performed. The imaging apparatus includes the third control unit, the target evaluation area is the third evaluation area,
The plurality of input images include a reference input image acquired during a period in which the imaging device is moving in a certain direction,
During the period, the third control unit adjusts a white balance for an input image acquired after the reference input image based on an image signal in an area located in the movement direction from the shift cut-out area of the reference input image. The imaging apparatus according to claim 6, wherein: