JP2014068080A - Image pick-up device and image pick-up method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow for image pick-up that covers a wide dynamic range by a simple operation, while preventing occurrence of a difference in the level of brightness at the joint of a panoramic image.SOLUTION: An image pick-up device comprises: image element which outputs an image signal obtained by exposing a plurality of areas that divides a light-receiving part in a predetermined direction, under different exposure conditions; control means for controlling to perform image capturing in a direction perpendicular to a predetermined direction every time when a predetermined angle or the field of view changes; panorama synthesis means for connecting the image signals in the areas of different field of view under the same exposure conditions, out of the image signals in a plurality of fields of view, and generating a plurality of panoramic images of expanded field of view for each different exposure conditions; and dynamic range expansion synthesis means for generating one panoramic image of wide apparent dynamic range by synthesizing a plurality of panoramic images. The predetermined angle is an angle of overlapping the parts of field of view before and after change of each area.

Description

  The present invention relates to an imaging apparatus and an imaging method having a panoramic imaging function capable of imaging in a range exceeding the angle of view of a lens in a vertical or horizontal direction.

  Conventionally, when taking a panoramic image of a wide view that exceeds the angle of view of a wide-angle lens (for example, when a focal length 24 mm lens is used in a 35 mm size 24 × 36 mm format), a panoramic image is obtained as follows. Was getting. First, I took multiple photos while changing the camera angle in the horizontal direction, and cut and pasted the printed photos to obtain a landscape panorama photo. Also, there has been a method of shooting using a panoramic shooting dedicated camera in which a rotary lens is placed at the center of an arc on a film loaded in an arc shape and exposure is performed as the lens rotates.

  In recent years, as a function of digital still cameras, a function called panoramic stitch that obtains a panoramic image by smoothly connecting the left and right overlapping parts by image processing of multiple images taken while changing the camera angle in the horizontal direction There is a camera equipped with. In addition, a camera equipped with a function called swing panorama that enables panoramic photography by simply swinging in the horizontal direction while pressing the shutter button has been released, making panoramic photography more familiar.

  On the other hand, since panorama shooting has a wide shooting range, there are many cases where a very bright portion and a dark portion are simultaneously reflected in the same composition. FIG. 18 is a diagram illustrating an example of a panoramic photograph in which a bright part and a dark part are reflected at the same time. This is an example of a sunrise view taken on a sandy beach. In part B, the sun as the light source is the subject and the sky is bright, so in ISO 400 sensitivity settings, the gradation of the sky near the sun is crushed and white unless the shutter speed is set to 1/1000 or F8. I will jump. In addition, although the portions A and C are on the mountain side, the sky is still dark before dawn, so if the shutter speed is not set to 1/30 or about F2, the gradation of the subject such as the gradation of the mountain surface will be crushed and black It will be crushed. Under such subject conditions, the dynamic range of the entire subject exceeds the dynamic range of the image sensor, so the dynamic range of the entire subject cannot be covered with the same exposure setting (combined shutter speed and aperture value). .

  For the problem described above, for example, a countermeasure as described in Patent Document 1 has been proposed. In this proposal, for each predetermined area, a plurality of images are captured with different exposure conditions and images with no overexposure detected and images with no blackout detected. Then, these are combined to obtain an image free from overexposure and blackout, and the same area is continuously taken and combined to obtain a panoramic image by connecting these combined images.

JP 2004-180308 PR

  However, since the camera described above is configured to shoot a plurality of images at the same angle, it is necessary to fix the camera body with a tripod or the like when shooting a plurality of images. Furthermore, it is necessary to repeat the shooting of a plurality of images and the adjustment of the camera angle, which requires a complicated procedure.

  In addition, it is configured to generate images with a wide dynamic range for each angle and to adjust the brightness of the images when connecting them, but by connecting images with different exposure conditions, There may be a difference in brightness.

  The present invention has been made in view of the above problems, and enables photographing that covers a wide dynamic range with a simple operation and prevents a brightness step from occurring at the joint of panoramic images. Objective.

  In order to achieve the above object, an imaging device of the present invention includes an imaging device that outputs an image signal obtained by exposing a plurality of areas obtained by dividing a light receiving unit in a predetermined direction under different exposure conditions, and Control means for controlling to take a picture with the imaging device every time a predetermined angle and field of view change in a direction perpendicular to a predetermined direction, and a plurality of values obtained for each predetermined angle Panorama synthesizing means for connecting a plurality of image signals of the different fields of view exposed under the same exposure conditions, and generating a plurality of panorama images for each of the different exposure conditions having an expanded field of view, Dynamic range expansion and synthesis means for synthesizing the plurality of panoramic images generated by the panorama synthesis means to generate one panoramic image having a wide dynamic range; It has the predetermined angle, characterized in that part of the front and rear of the field of view of the change of each area of the plurality of areas is an angle overlapping.

  According to the present invention, it is possible to perform photographing that covers a wide dynamic range with a simple operation, and it is possible to prevent a brightness step from occurring at the joint of panoramic images.

1 is a block diagram illustrating a configuration example of a digital camera according to a first embodiment. The figure which shows the concept of the panoramic imaging by the digital camera concerning 1st and 2nd embodiment. The figure which shows operation | movement of the angular velocity sensor of the digital camera concerning 1st and 2nd embodiment. 9 is a flowchart showing an operation in a panoramic shooting mode by the digital camera according to the first and second embodiments. The conceptual diagram of the determination method of the exposure conditions of the digital camera concerning 1st and 2nd embodiment. FIG. 3 is a diagram illustrating a configuration example of an image sensor of a digital camera according to first and second embodiments. FIG. 6 is a diagram illustrating an example of a control signal of an image sensor of a digital camera according to the first and second embodiments. FIG. 6 is a conceptual diagram of panoramic synthesis and dynamic range expansion synthesis of the digital camera according to the first and second embodiments. The figure which shows another structural example of the image pick-up element of the digital camera concerning the modification of 1st Embodiment. The figure which shows the concept of the panoramic photography by the digital camera concerning the modification of 1st Embodiment. The figure which shows the structural example of the image pick-up element part of the digital camera concerning the modification of 1st Embodiment. The block diagram which shows the structural example of the digital camera in 2nd Embodiment. The block diagram which shows the structural example of the display image processing block concerning 2nd Embodiment. The figure which shows the relationship between the area of the image pick-up element of the digital camera concerning 2nd Embodiment, and the image output from an image pick-up element. The figure which shows the image processing method by the display image processing block of the digital camera of 2nd Embodiment. The figure which shows the image processing method by the display image processing block of the digital camera in the modification 1 of 2nd Embodiment. The figure which shows the image processing method by the display image processing block of the digital camera in the modification 2 of 2nd Embodiment. The figure which shows the example of the photograph which requires a wide dynamic range.

  The best mode for carrying out the present invention will be described below in detail with reference to the accompanying drawings. However, the dimensions, shapes, relative arrangements, and the like of the components exemplified in the present embodiment should be changed as appropriate according to the configuration of the apparatus to which the present invention is applied and various conditions. It is not limited to.

<First Embodiment>
A first embodiment in which the present invention is implemented as a digital camera will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration example of a digital camera according to the first embodiment. The taking lens 101 captures a subject image and forms the subject image on the image sensor 103. In the digital camera of the first embodiment, the zoom lens is fixed to a focal length corresponding to 28 mm in terms of 35 mm film equivalent in a panoramic shooting mode described later. The subject image formed on the image sensor 103 is photoelectrically converted and an image signal is output. This image signal is digitized by the A / D converter 104, subjected to gamma correction, white balance correction, noise reduction processing, and the like by the image processing unit 105, and then output to the image data bus 106 as uncompressed image data.

  The photometric sensor 111 detects the luminance distribution in the range of the shooting screen and outputs EV value data to the control bus 116. In the first embodiment, the photometric sensor 111 is an external light type and is installed on the upper surface of the camera, has a luminance detection range equivalent to the field angle range for panoramic photography, and has a maximum luminance and a minimum luminance within the measurement range. Has a measurable configuration. The angular velocity sensor 112 detects the angular velocity generated when the user changes the angle of the digital camera body for each of the X (yaw) and Y (pitch) directions orthogonal to the camera optical axis, and supports the angular velocity data in the X and Y directions. Output to the control bus 116 for each of the two channels.

  The SDRAM (1) 109 accumulates uncompressed image data for a plurality of frames and provides a memory space that is a work space for performing panoramic synthesis and wide dynamic range (HDR) synthesis, which will be described later. The JPEG / RAW codec 110 compresses and encodes the uncompressed image data that has been subjected to the synthesis process in the SDRAM (1) 109 as a still image, and generates JPEG still image data or RAW still image data. JPEG is a general-purpose 8-bit gradation recording device that is compatible with reproduction. RAW is a 14-bit gradation data that is not subjected to image processing by the image processing unit 105 and is encoded using a lossless compression method. Thus, this recording method is compatible with reproduction only for specific devices. Note that the data format of still images in the present invention is not limited to JPEG and RAW, and may be still image data in a known data format. In that case, a codec that generates still image data in a predetermined data format is provided instead of JPEG and / or RAW.

  The SDRAM (2) 113 provides a buffer memory space for writing and reading the generated still image data in accordance with the writing status of the recording medium 114, and arbitrating with the recording speed of the recording medium 114. The SDRAM (2) 113 also serves as an image display memory (video memory). The liquid crystal panel 107 is a display unit that displays images and various information. The liquid crystal driver 108 converts display image data stored in the SDRAM (2) 113 into a liquid crystal display signal and supplies the liquid crystal display signal to the liquid crystal panel 107. To do. Thus, the display image data written in the SDRAM (2) 113 is displayed on the liquid crystal panel 107 via the liquid crystal driver 108. In addition, a digital signal that has been A / D converted by the A / D converter 104 and accumulated in the SDRAM (2) 113 is converted into a liquid crystal display signal by the liquid crystal driver 108, and sequentially transferred to the liquid crystal panel 107 for display. Functions as an electronic viewfinder. Thereby, a through image display can be performed.

  The recording medium 114 is a memory card configured with, for example, a NAND flash memory that can be attached to and detached from the camera body. On the recording medium 114, JPEG still image data or RAW still image data is formed in accordance with a PC compatible FAT (File Allocation Table) file system that is recognized as a disk drive when connected to a PC. Is recorded. The operation keys 115 are various switches that accept various operations from the user, such as a shutter button for performing still image shooting operation, a mode switch for selecting whether to perform panoramic shooting or normal still image shooting, and the like. Including.

  The control bus 116 is a signal path for communicating control signals between the blocks. The control bus 116 transmits a control signal from the main microcomputer 118 to each component, and transmits a response signal from each component and a data signal from each sensor. Tell the main microcomputer 118. The flash ROM 119 is an electrically erasable / recordable nonvolatile memory, and stores constants, programs, and the like for the operation of the main microcomputer 118. Here, the program is a program for executing various flowcharts described later in the present embodiment. The main microcomputer 118 controls the entire digital camera. By executing a program recorded in a flash ROM 119 described later, each process of the present embodiment described later is realized. Reference numeral 117 denotes a system memory, which uses a RAM. In the RAM 117, constants and variables for operation of the main microcomputer 118, a program read from the flash ROM 119, and the like are expanded. The main microcomputer 118 also performs display control by controlling the liquid crystal driver 108 and the like.

  FIG. 2 is a diagram showing a use image when a horizontally long panoramic photograph is taken with the digital camera according to the first embodiment, and FIG. 2A is a view of FIG. 2B when viewed from the photographer side. ) Shows the case of looking at the photographer. A panoramic photograph can be taken by continuously changing the camera angle so that the photographer rotates from left to right as viewed from the photographer while pressing the shutter button.

  Next, the operation of the digital camera of the first embodiment having the above configuration will be described. FIG. 3 is a diagram showing the state of the digital camera according to the first embodiment (FIG. 3A) and the output of the angular velocity sensor 112 corresponding thereto. The rotation speed in the yaw direction of the camera is output as X (FIG. 3 (b)), the rotation speed in the pitch direction of the camera is output as Y output (FIG. 3 (c)), and rotation is obtained by taking the integral value of each output. The amount can be calculated.

  FIG. 4 is a flowchart showing the operation in the panoramic shooting mode in the digital camera of the present invention. When the power of the digital camera is turned on in the panoramic shooting mode in S401, it is detected in S402 whether the shutter button is half-pressed. If it is half-pressed, the photometric sensor 111 detects the subject in S403. Luminance is measured. In S404, the maximum luminance and the minimum luminance of the subject are detected from the measured luminance of the subject.

  In S405, based on the luminance difference obtained from the detected maximum luminance and minimum luminance, the required number N of standard dynamic range images necessary for synthesizing and obtaining a wide dynamic range image having a wide apparent dynamic range is obtained. calculate. In step S406, the shift amount of the exposure condition for obtaining the calculated required number N of images is calculated.

FIG. 5 is a conceptual diagram showing a method for determining the calculated required number N and the shift amount of the exposure condition in S405 and S406. In FIG. 5, EV is an abbreviation of Exposure Value (exposure index), which is the amount of light that can sensitize a film with ISO 100 sensitivity to 18% gray (to obtain proper exposure). When the aperture value is F1.0 and the shutter speed is 1 second, EV = 0, and the EV value increases by 1 each time the aperture value is doubled or the shutter speed (exposure time) is halved. There is no single combination of aperture value and shutter speed for the same EV. As shown in the table of FIG. 5, for example, to obtain a proper exposure for a light quantity of 7 EV, F4.0 · 1/8 second. , F2.8 · 1/15 seconds, and F2.0 · 1/30 seconds. The width of the dynamic range is also expressed by an EV value. For example, ¹⁰ EV indicates a dynamic range that can cover a luminance difference of 2 ¹⁰ = 1024 times. The required number N and the exposure condition shift amount are calculated by the following equations (1) and (2), respectively. In the following equation (1), the fractional part is rounded down. Equation (2) is calculated when N is 2 or more.
N = ((maximum luminance−minimum luminance) / D range of the image sensor) +1 (1)
Exposure shift amount = ((maximum luminance−minimum luminance) −D range of image sensor) / (N−1)
... (2)

  If the standard dynamic range by the image sensor 103 of the first embodiment is 10 EV, and it is detected in S404 that the maximum luminance is 22 EV and the minimum luminance is 2 EV, N = 3 is calculated. In addition, the exposure condition shift amount determined in S406 is 5 EV. In other words, the exposure amount is 7 EV, which is 5 EV shifted from the minimum luminance 2 EV, which is half of the standard dynamic range 10 EV, 12 EV, which is further shifted by 5 EV of the exposure condition determined in S406, and 17 EV, which is further shifted by 5 EV. Become the center. Further, when 5 EV, which is half of the standard dynamic range 10 EV, is shifted, the maximum luminance 22 EV is obtained. Thus, the EV value for determining the exposure conditions for the three areas is determined, the standard exposure amount is 12 EV, the under side exposure amount is 7 EV, and the over side exposure amount is 17 EV.

  Next, an aperture value and a shutter speed are determined in S407 and S408. Here, since the change of the exposure condition performed when obtaining a plurality of standard dynamic range images is performed by changing the shutter speed, N shutter speeds are determined for one aperture value. In this case, first, conditions for obtaining the highest shutter speed are set within the limits of the aperture value and shutter speed that can be set on the camera side. For example, when 1/32000 seconds is set as the maximum shutter speed, in the first embodiment, the aperture value is set to F2.0 from FIG. When the aperture value is F2.0, 1/30 second is determined as the shutter speed for the under-side exposure amount 7 EV, and the standard exposure amount is 1/1000 second as the shutter speed for 12 EV. Note that the shutter speed for the over-side exposure amount 17 EV is 1/32000 seconds as described above. Here, the shutter speed is controlled by controlling the charge accumulation time of the image sensor 103, and shooting is performed with different settings for each area of the image sensor 103. This will be described later.

  When it is detected in S409 that the user has pressed the shutter button, exposure of the subject is started in each area of the image sensor 103 in S410, and after a predetermined charge accumulation time, the exposure ends and the image signal Is transferred. The transferred imaging signal is A / D converted by the A / D converter 104, subjected to image processing such as gamma correction, white balance correction, and noise reduction processing in the image processing unit 105, and then the SDRAM (1). 109 is temporarily stored. Thereafter, in S411, it is detected whether the user has shaken the shooting angle (field of view) horizontally (here, 10 degrees) by one area which is a composition unit for panorama composition described later. Here, the synthesis unit is an angle at which a part of the field of view of each area before and after the shooting angle changes overlaps.

  If it has been shaken, it is detected in S412 whether the user has released the shutter button. If it is detected that the shutter button has not been released and the shutter button has been continuously pressed, the shooting angle swing amount by the user after the start of shooting is the maximum angle of view that can be shot in the panoramic shooting mode in S413. For example, it is detected whether it exceeds 270 degrees. If not, the process returns to S410 again. If the user releases the shutter button in S412, or if it is detected in S413 that the shooting angle swing amount by the user after the start of shooting reaches 270 degrees, the imaging operation ends.

  Then, panorama synthesis is performed in S414, dynamic range expansion synthesis is performed in S415, and a panorama image that is finally recorded on the recording medium 114 is generated. Thereafter, in S416, the panoramic image generated by the JPEG / RAW codec 110 is JPEG-encoded or RAW-encoded, and is recorded on the recording medium 114 in S417, and a series of operations ends in S418.

  Next, the exposure process of the image sensor 103 performed in S410 with the shutter speed determined in S408 will be described in detail. FIG. 6 is a diagram showing an outline of the image sensor 103. In the first embodiment, the image sensor 103 is 2592 × 1944 pixels (about 5 million pixels), and a line is formed in the vertical direction.

  FIG. 6B is a conceptual diagram showing the area of the light receiving unit of the image sensor 103. When N = 3, the area is divided as follows. Lines 1 to 431 are exposed but are invalid areas that are not used in panoramic synthesis, which will be described later, and lines 432 to 1007 are areas A in which exposure is performed with a charge accumulation time of 1/32000 seconds. A line 1008 to a line 1583 are an area B where exposure is performed with a charge accumulation time of 1/1000 second, and a line 1584 to a line 2159 is an area C where exposure is performed with a charge accumulation time of 1/30 second. Lines 2160 to 2592 are invalid areas that are exposed but are not used in panoramic synthesis described later.

  FIG. 6A is a diagram illustrating the configuration of the image sensor 103. The light receiving element 603 forms one pixel, and the signal reading unit 604 reads the amount of charge accumulated by the light receiving element 603 as an image signal. The driver 605 outputs, for each line, an RST signal that indicates the timing for discharging (resetting) the electric charge accumulated by the light receiving element 603. Further, pixel charge control is performed by obtaining the accumulated charge amount information at the time of assertion of the RST signal and outputting a TRS signal for instructing the timing of transfer to the signal reading unit 604 for each line. The time difference from when the RST signal is asserted to when the TRS signal is asserted is the charge accumulation time, which corresponds to the shutter speed.

  The line selection unit 606 performs line selection for pixel line control performed by the driver 605, and the scanning control unit 607 controls the signal readout unit 604 and the line selection unit 606 to perform scanning for one screen.

  FIG. 7 shows waveforms of the RST signal and the TRS signal output by the driver 605. In FIG. 7, lines 1 to 1007 have signal waveforms such that the time difference between the rising edges of the RST signal and the TRS signal is 1/32000 seconds. Lines 1008 to 1583 have signal waveforms such that the time difference between the rises of the RST signal and the TRS signal is 1/1000 second, and lines 1584 to 2592 have a signal waveform such that the time difference between the rises of the RST signal and the TRS signal is 1/30 seconds. It has become. Thereby, the shutter speed is controlled to be different in each area.

  Next, the panorama synthesis performed in S414 and the wide dynamic range (HDR) expansion synthesis performed in S415 will be described. FIG. 8 is a conceptual diagram showing panoramic synthesis and HDR expansion synthesis. Reference numeral 801 denotes an image taken in the panoramic shooting mode obtained by the image sensor 103. With the above-described configuration, shooting is performed every time the horizontal angle changes by 10 degrees, and a total of 29 images are shot. It is shown that. Each area of each image is 1944 x 576 pixels in the vertical direction, and as described above, the focal length of the taking lens 101 is equivalent to 28 mm in terms of 35 mm film, so the angle of view in the horizontal direction is about 15 Degree. Therefore, the images captured by the respective areas are photographed with an overlap of 5 degrees on both ends every 15 degrees. In the panorama composition, three sets of panorama composite images are generated while performing blurring processing and the like so that the joints are not conspicuous using the overlap portion. 802 is a panoramic image obtained by synthesizing an image captured at the under exposure amount in area A, 803 is a panoramic image obtained by synthesizing an image captured at the standard exposure amount in area B, and 804 is an over side in area C. It is the panoramic image which combined the image imaged with the exposure amount of. Each panoramic image has a gradation of 10 bits.

  In the dynamic range expansion / combination performed thereafter, first, the brightness of the panorama image 802 obtained by combining the images captured in the area A is increased by an exposure shift amount of 5 EV. Further, the brightness of the panoramic image 804 obtained by combining the images picked up in the area C is reduced by 5 EV which is the exposure shift amount. Then, a panoramic image 803 obtained by synthesizing the images picked up in area B is connected to a portion without whiteout or blackout to generate an image with a wide dynamic range having a gradation wider than 10 bits. Thereafter, contrast compression is performed. In the recording by JPEG encoding, the rounding process is performed to the gradation of 8 bits. In the recording by the RAW encoding, the rounding process is performed to the gradation of 14 bits. Encoding is performed, and the encoded data is recorded on the recording medium 114.

  With the above configuration, it is possible to perform shooting that covers a wide dynamic range by the same operation as a general panoramic shooting operation, and it is possible to perform panoramic shooting that does not cause a brightness difference at the joint of panoramic images.

<Modification of First Embodiment>
In the digital camera according to the first embodiment described above, the image sensor 103 having the line array in the vertical direction shown in FIG. 6 is used, but the present invention is not limited to this. For example, when performing panoramic shooting in the horizontal direction using an image sensor having a line array in the horizontal direction as shown in FIG. 9, the angle is set with the camera held in the vertical position as shown in FIG. It is good also as a structure by operation which shakes in a horizontal direction. In FIG. 9, the same reference numerals are assigned to components having the same functions as those in FIG. In this configuration, an image sensor 103 having a general horizontal line arrangement can be used, and the completed panoramic image has a wide vertical angle of view.

  Furthermore, in the digital video camera of the first embodiment, different exposure conditions for each area of the image sensor 103 are set according to the difference in charge accumulation time. However, the present invention is not limited to this. As shown in FIG. 11, movable neutral density filters 1102 to 1104 may be installed in front of the image sensor 103. In this case, exposure conditions can be made different for each area by putting the neutral density filters 1102 to 1104 in and out of the optical path. Alternatively, a neutral density filter having a different density for each area may be used instead of the number of neutral density filters.

  In this configuration, since the same shutter speed can be selected for the image sensor 103, it is difficult to be restricted by the aperture value and shutter speed that can be set on the above-described camera side, and the user can widen the selection range of the shutter speed. it can.

<Second Embodiment>
Next, a second embodiment in which the present invention is implemented with a digital camera will be described with reference to the drawings. FIG. 12 is a block diagram illustrating a configuration example of the digital camera according to the second embodiment. In FIG. 12, the same components as those in FIG. The difference between the configuration shown in FIG. 12 and the configuration shown in FIG. 1 is that a display image processing unit 1220 is added. The display image processing unit 1220 is located between the image data bus 106 and the liquid crystal driver 108, and performs predetermined image processing on the image data once A / D converted by the A / D converter 104 and stored in the SDRAM (2) 113. Apply. Thereafter, the digital signal subjected to the image processing is converted into a liquid crystal display signal by the liquid crystal driver 108 and sequentially transferred and displayed on the liquid crystal panel 107, thereby functioning as an electronic viewfinder and displaying a through image.

  In the digital camera of the second embodiment, similarly to the operation in the panoramic shooting mode of the digital camera of the first embodiment described above, first, a plurality of exposures obtained with different exposure times for each area in the image sensor 103. Acquire a captured image of the condition. Then, a panoramic image with a wide dynamic range is generated from the acquired photographed image and recorded on the recording medium 114. At this time, the image output from the image sensor 103 is an image signal having different brightness for each area as shown in FIG. As described above, this image signal is used for generating a panoramic image with a wide dynamic range and also used as an electronic viewfinder display on the liquid crystal panel 107. In the display image processing unit 1220, the following image processing is performed. Is done.

  FIG. 13 is a block diagram showing details of the display image processing unit 1220. In FIG. 13, an image reduction block 1200 reduces the image data input from the image data bus 106 in accordance with the number of pixels of the liquid crystal panel 107 to create an electronic viewfinder display image. The area division block 1201 divides the electronic viewfinder display image into areas A, B, and C shown in FIG. The image processing (1) block 1202 performs image processing on the imaging signal from the area A in FIG. 14, and the image processing (2) block 1203 performs image processing on the imaging signal from the area B in FIG. Apply. Then, the image processing (3) block 1204 performs image processing on the imaging signal from area C in FIG. The SDRAM (3) 1205 provides a work area for the image reduction block 1200 and the image processing (1) to (3) blocks 1202 to 1204. The area synthesis block 1206 connects the images from area A, area B, and area C and combines them into an image for one frame. The control bus interface (I / F) 1207 communicates control signals such as parameter settings of the image processing (1) to (3) blocks 1202 to 1204 with the control bus 116.

  FIG. 15A is a diagram showing the processing contents by the display image processing unit 1220. In the figure, reference numeral 1401 denotes an image output from the image sensor 103. The input image 1401 is divided into video for each area by the area division block 1201, and the image of the area A is subjected to the image processing (1) block 1202 and the difference in exposure amount from the area B imaged under the standard exposure conditions. It is corrected brightly for 5 EV. The image in area B is transferred from the image processing (2) block 1203 to the SDRAM (3) 1205. Then, the image processing (2) block 1203 is returned to the image processing (1) block 1202 and the image processing block (3) 1204 so as to match the image processing timing without being corrected. Further, the image of area C is darkened by image processing (3) block 1204 by 5 EV, which is the exposure amount difference from area B imaged under the standard exposure conditions.

  Image processing (1) to (3) The images of the areas corrected in the blocks 1202 to 1204 are combined by the area combining block 1206 and displayed on the liquid crystal panel 107. Thereby, as shown in FIG. 15B, an electronic viewfinder display in which each area has a uniform brightness is obtained. With this configuration, in order to obtain a panoramic image with a wide dynamic range, even if images are captured with different exposure amounts for each area of the image sensor 103, a display with uniform brightness can be obtained with the electronic viewfinder. Therefore, it is possible to provide a camera having a panoramic shooting mode with higher visibility for the photographer.

  Note that the second embodiment can also be applied to the case where the image sensor 103 has the configuration described with reference to FIG. 11 in the modification of the first embodiment. Even when the image sensor 103 has the configuration described with reference to FIGS. 9 and 10, the second embodiment can be applied with the same configuration only by changing the vertical direction to the horizontal direction. is there.

<Modification Example 1 of Second Embodiment>
The display image processing unit 1220 may perform image processing shown in FIG. In the figure, reference numeral 1501 denotes an image output from the image sensor 103. After the input image 1501 is divided into video for each area by the area division block 1201, the image of the area A is subjected to the image processing (1) block 1202 (first image processing means) and the area B and the front of the previous photographed image. It is replaced with the image of area B of the captured image. The pre-photographed image and the pre-photographed image are temporarily stored in the SDRAM (3) 1205 in the process of processing the image of area B described later. The image of area B is stored in the SDRAM (3) 1205 from the image processing (2) block 1203. Then, the image processing (2) block 1203 is returned to the image processing (1) block 1202 and the image processing (3) block 1204 so as to be synchronized with the image processing timing without being corrected. As described above, the image data stored in the SDRAM (3) 1205 by this processing is used for image processing for the image of the next area A. Further, the image of area C is darkened by image processing (3) block 1204 (second image processing means) by 5 EV, which is the difference in exposure amount from area B imaged under standard exposure conditions. The

  Image processing (1) to (3) The images of the areas corrected in the blocks 1202 to 1204 are combined in the area combining block 1206 and displayed on the liquid crystal panel 107. Thereby, as shown in FIG. 16B, an electronic viewfinder display in which each area has uniform brightness can be obtained. With this configuration, in order to obtain a panoramic image with a wide dynamic range, the electronic viewfinder can obtain a display with uniform brightness even though images are captured with different exposure amounts for each area of the image sensor. In addition, in the area A, the portion where information is lost due to black crushing caused by shooting under the exposure condition of −5 EV is corrected to be brighter by +5 EV, so that the image is displayed brightly with information missing. Degradation can be avoided. Thereby, there is an effect that it is possible to provide a camera having a panoramic shooting mode with higher visibility for a photographer.

  In this modification, as shown in FIG. 9, panoramic shooting is performed by swinging from left to right. However, a block for detecting the direction of the swing is added, and the swing direction of panoramic shooting is used. May be detected. When panoramic shooting is performed by a swing from right to left, the processing of area C in FIG. 15 is performed by the image processing (1) block 1202, and the processing of area A in FIG. 15 is performed by image processing (3). This is done by block 1204. With this configuration, it is possible to provide a camera having the effect obtained by the configuration of FIG. 15 regardless of whether the swing direction for panoramic shooting of the user is from left to right or from right to left.

<Modification 2 of the second embodiment>
The display image processing unit 1220 may perform image processing shown in FIG. In FIG. 17A, reference numeral 1601 denotes an image output from the image sensor 103. This image 1601 is displayed on the liquid crystal panel 107 after black masks are applied to the areas A and C by the display image processing unit 1220. As shown in FIG. 17B, an electronic viewfinder display in which an image of only area B imaged under standard exposure conditions is displayed. With this configuration, it is possible to provide a camera having a panoramic shooting mode with an electronic viewfinder display that does not make the photographer feel unnatural with a simpler configuration.

<Other embodiments>
Note that the present invention can be applied to a system composed of a plurality of devices (for example, a host computer, an interface device, a camera head, etc.) or an apparatus composed of a single device (for example, a digital camera). Good.

Claims (8)

An image sensor that outputs an image signal obtained by exposing a plurality of areas obtained by dividing a light receiving unit in a predetermined direction under different exposure conditions;
Control means for controlling the imaging device to perform imaging every time a predetermined angle and field of view change in a direction perpendicular to the predetermined direction;
Among the image signals of a plurality of fields of view obtained for each predetermined angle, the image signals of the areas of different fields of view exposed under the same exposure conditions are connected, and a plurality of images for the different exposure conditions whose fields of view are expanded Panorama synthesizing means for generating a panoramic image of
Dynamic range expansion / synthesizing means for synthesizing the plurality of panoramic images generated by the panorama synthesizing means to generate a panoramic image with a wide apparent dynamic range and a wide dynamic range;
The imaging apparatus according to claim 1, wherein the predetermined angle is an angle at which a part of the visual field before and after the change of each of the plurality of areas overlaps. Photometric means;
The difference between the maximum luminance and the minimum luminance detected by the photometric means is detected, and the number of the plurality of areas is determined according to a value obtained by dividing the detected luminance difference by the dynamic range of the image sensor. The imaging apparatus according to claim 1, further comprising: a determination unit that performs the determination.   The imaging device according to claim 1, wherein the imaging device performs exposure under the different exposure conditions by making charge accumulation times of the plurality of areas different from each other.   3. The imaging apparatus according to claim 1, wherein the imaging element performs exposure under the different exposure conditions by covering the plurality of areas with different number of neutral density filters or different density filters. 4.   Image processing means for correcting an image signal for each field of view output from the image sensor and correcting the difference with an exposure amount according to a predetermined exposure condition for each area, and outputting the image signal to a display device. The imaging device according to any one of claims 1 to 4, wherein Among the different exposure conditions, an image signal output from an area exposed under an exposure condition that is different from a predetermined exposure condition is obtained in the same field of view as obtained in the previous photographing under the control of the control means. First image processing means for replacing with an image signal of an area exposed under the predetermined exposure condition;
The predetermined exposure when an area exposed under the predetermined exposure condition having the same field of view as the area exposed under the exposure condition different from the predetermined exposure condition is not obtained by the previous photographing. Second image processing means for correcting a difference between an exposure amount according to the predetermined exposure condition with respect to an image signal output from the area exposed under an exposure condition different from the condition;
5. The imaging apparatus according to claim 1, further comprising display control means for outputting an image signal processed by the first and second image processing means to a display device.   5. The display control unit according to claim 1, further comprising a display control unit that outputs an image signal obtained from an area exposed under a predetermined exposure condition among the plurality of areas to a display device. The imaging device according to item. An imaging process in which an imaging element outputs an image signal obtained by exposing a plurality of areas obtained by dividing the light receiving unit in a predetermined direction under different exposure conditions;
A control step in which the control means controls to perform photographing with the imaging element every time a predetermined angle and field of view change in a direction perpendicular to the predetermined direction;
The panorama composition unit connects the image signals of the different fields of view exposed under the same exposure condition among the image signals of the plurality of fields of view obtained at the predetermined angles, and the different fields whose fields of view are enlarged A panorama composition step for generating a plurality of panorama images for each exposure condition;
A dynamic range expansion / synthesizing step, wherein a dynamic range expansion / synthesizing unit combines the plurality of panoramic images generated in the panorama combining step to generate a panoramic image having a wide dynamic range with a single apparent dynamic range; Have
The imaging method according to claim 1, wherein the predetermined angle is an angle at which a part of the visual field before and after the change of each of the plurality of areas overlaps.