JP6242244B2 - Imaging apparatus, imaging method, and program - Google Patents

Description

  The present invention relates to an imaging apparatus, an imaging method, and a program capable of synthesizing a plurality of image data and generating a synthesized image.

  An imaging device such as a digital camera is commercially available that has a function of realizing a picture quality that cannot be represented by a single image by taking a plurality of images and combining the plurality of images.

  As a technique for synthesizing a plurality of images, for example, a plurality of images are photographed while moving a focal position, a plurality of image data acquired by the photographing is used, a subject is aligned, and the images are synthesized. In addition, it has been proposed to generate an omnifocal image or an image in which blur is controlled (see Patent Document 1). In addition, by photographing a plurality of images having different gradations and combining a plurality of image data acquired by the photographing, an image having an HDR (High Dynamic Range) effect having gradation characteristics that cannot be realized by a single image. Producing is proposed (see Patent Document 2).

JP 2008-271240 A JP 2011-004353 A

  As described above, various techniques for synthesizing a plurality of image data and generating the synthesized image data have been proposed. However, when the images are synthesized, generally, as the number of images increases, a better result can be obtained. For this reason, it can be said that it is preferable to increase the number of shots as much as possible from the viewpoint of image quality. On the other hand, a memory such as an SDRAM mounted on an imaging apparatus such as a digital camera has a limited capacity, and therefore the number of image data that can be stored simultaneously is also limited.

  The present invention has been made in view of such circumstances, and an imaging device, an imaging method, and an imaging device that can secure as many shots as possible and generate a composite image with sufficient image quality in a limited capacity. And to provide a program.

In order to achieve the above object, an image pickup apparatus according to a first aspect of the present invention focuses an image pickup unit that picks up a subject with an image pickup element and outputs image data, and focuses the subject on the image pickup element of the image pickup unit. A focusing unit that adjusts a focal position; a reference image generation unit that generates reference image data from the image data; and a data amount of the image data is reduced, and composition image data having a data amount smaller than the reference image data is reduced. and generating image data amount reduction unit which performs the synthesis of image data by using the reference image data and the image data for the above synthesis, an image combining unit for generating a composite image data, the possess, the image data amount reduction unit Extracts the gamma component of the image data, reduces the amount of data based on the compression characteristics according to the gamma component, and at each in-focus position of a plurality of different in-focus positions by the in-focus unit, Obtaining a plurality of image data output by the serial imaging unit by the image data quantity reduction unit reduces the respective amount of data of the plurality of image data to generate synthesized image data by the image combining unit.

In the imaging device according to a second aspect of the present invention, in the first aspect, the image data amount reduction unit reduces the bit accuracy of the image data.
According to a third aspect of the present invention, in the image pickup apparatus according to the first aspect , the image data amount reduction unit includes a gradation conversion unit using a table or a broken line, and performs gradation conversion on the image data.

In the imaging device according to a fourth aspect based on the first aspect, the image data amount reduction unit includes a compression unit that encodes and compresses the image data. In the fourth invention, the compression unit performs lossy compression.
Imaging device according to a sixth invention, in the above Symbol first invention, the image data amount reduction unit to reduce the amount of data of the color difference signal.

The image pickup apparatus according to a seventh invention is the image pickup apparatus according to the first invention, wherein the reference image data is the first image data picked up from a plurality of pieces of image data to be combined in the image combiner, or in focus. and that is the image data.

An imaging apparatus according to an eighth aspect of the present invention is the imaging apparatus according to the first aspect, wherein the imaging apparatus has an image recording unit that records the composite image data, and the image data for synthesis whose data amount is reduced by the image data amount reduction unit is used. Te, when generating the composite image data, on the accompanying reduction information to the composite image data is recorded in the image Zoki recording unit.
In the imaging device according to a ninth aspect based on the first aspect, the image data amount reduction unit reduces the data amount by an irreversible method.

Imaging method according to the tenth invention generates image data of the reference image by imaging a subject by the imaging device, the subject, and adjust the focus position to focus on SL IMAGING element, subject Image data of the image for synthesis is generated, the data amount of the image data of the image for synthesis is reduced so that the data amount is smaller than the image data of the reference image, and the image data of the reference image and the image data of the reference image The image data of the image for synthesis is used to synthesize the image data, generate the synthesized image data, further extract the gamma component of the image data, and reduce the amount of data based on the compression characteristics according to the gamma component Acquiring a plurality of image data output by the imaging device at each of a plurality of different in-focus positions, reducing a data amount of each of the plurality of image data, and combining image data; To generate the data.

Program according to the eleventh invention generates image data of the reference image by imaging a subject by the imaging device, by imaging a subject to generate image data of the composite image, the object, the upper Symbol IMAGING element The focus position is adjusted so that the image is in focus, the data amount of the image for synthesis is reduced so that the amount of data is less than the image data of the reference image, and the image data of the reference image is combined with the image data of the reference image Using the image data of the image for the image, the image data is synthesized, and the synthesized image data is generated. Further, the gamma component of the image data is extracted, the data amount is reduced based on the compression characteristic according to the gamma component, A plurality of image data output by the imaging element is obtained at each of the plurality of different in-focus positions, the amount of each of the plurality of image data is reduced, and the composite image Generating a chromatography data, it causes the computer to execute the.

  According to the present invention, it is possible to provide an imaging device, an imaging method, and a program that can secure as many shots as possible with a limited capacity and can generate a composite image with sufficient image quality.

It is a block diagram which shows mainly an electrical structure of the camera which concerns on 1st Embodiment of this invention. It is a flowchart which shows the main operation | movement of the camera which concerns on 1st Embodiment of this invention. It is a flowchart which shows the main operation | movement of the camera which concerns on 1st Embodiment of this invention. It is a flowchart which shows operation | movement of imaging | photography and image processing of the camera which concerns on 1st Embodiment of this invention. It is a flowchart which shows operation | movement of imaging | photography and image processing of the camera which concerns on 1st Embodiment of this invention. It is a graph which shows an example of the RAW compression characteristic in the camera which concerns on 1st Embodiment of this invention, (a) shows an example of the RAW compression characteristic at the time of HDR synthetic | combination, (b) is the RAW compression characteristic at the time of depth synthetic | combination. An example is shown. 5 is a graph illustrating an example of a RAW expansion characteristic in the camera according to the first embodiment of the present invention, where (a) illustrates an example of a RAW expansion characteristic during HDR composition, and (b) illustrates a RAW expansion characteristic during depth composition. An example is shown. It is a block diagram which mainly shows the electrical structure of the camera which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the main operation | movement of the camera which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the main operation | movement of the camera which concerns on 2nd Embodiment of this invention. It is a block diagram which mainly shows the electrical structure of the camera which concerns on 3rd Embodiment of this invention. It is a flowchart which shows the main operation | movement of the camera which concerns on 3rd Embodiment of this invention. It is a flowchart which shows the main operation | movement of the camera which concerns on 3rd Embodiment of this invention.

  Hereinafter, an example applied to a digital camera as an embodiment of the present invention will be described. This digital camera has an imaging unit, and the imaging unit converts the subject image into image data, and based on the converted image data, the subject image is displayed in live view on a display unit disposed on the back of the main body. . The photographer determines the composition and the photo opportunity by observing the live view display. During the release operation, image data is recorded on the recording medium. The image data recorded on the recording medium can be reproduced and displayed on the display unit when the reproduction mode is selected.

  The camera according to the first embodiment, when combining a plurality of image data, reduces the size of the image data to be combined from the reference image data and then stores it in an internal memory such as an SDRAM. Therefore, a large amount of captured image data can be stored in an internal memory having a limited capacity, and a higher-quality composite image can be generated as compared with the case of an internal memory having the same capacity.

  FIG. 1 is a block diagram mainly showing an electrical configuration of the camera according to the first embodiment of the present invention. This camera includes a camera body 100 and an interchangeable lens 200 that can be attached to and detached from the camera body 100. In the present embodiment, the photographing lens is an interchangeable lens type, but the present invention is not limited to this, and a digital camera of a type in which the photographing lens is fixed to the camera body may of course be used.

  The interchangeable lens 200 includes a photographing lens 201, a diaphragm 203, a driver 205, a microcomputer 207, and a flash memory 209, and has an interface (hereinafter referred to as I / F) 199 with a camera body 100 described later.

  The taking lens 201 is composed of a plurality of optical lenses for forming a subject image, and is a single focus lens or a zoom lens. A diaphragm 203 is disposed behind the optical axis of the photographing lens 201. The diaphragm 203 has a variable aperture diameter, and restricts the amount of light of the subject light beam that has passed through the photographing lens 201. The photographing lens 201 can be moved in the optical axis direction by a driver 205, and the focus position of the photographing lens 201 is controlled based on a control signal from the microcomputer 207. In the case of a zoom lens, the focal length is also controlled. Is done. The driver 205 also controls the aperture diameter of the diaphragm 203.

  The microcomputer 207 connected to the driver 205 is connected to the I / F 199 and the flash memory 209. The microcomputer 207 operates according to a program stored in the flash memory 209, communicates with a microcomputer 121 in the camera body 100 described later, and controls the interchangeable lens 200 based on a control signal from the microcomputer 121. Do.

  The flash memory 209 stores various information such as optical characteristics and adjustment values of the interchangeable lens 200 in addition to the above-described program. The I / F 199 is an interface for performing communication between the microcomputer 207 in the interchangeable lens 200 and the microcomputer 121 in the camera body 100.

  A mechanical shutter 101 is disposed in the camera body 100 on the optical axis of the taking lens 201. The mechanical shutter 101 controls the passage time of the subject luminous flux and employs a known focal plane shutter or the like. An image sensor 103 is disposed behind the mechanical shutter 101 and at a position where a subject image is formed by the photographing lens 201.

  The imaging element 103 functions as an imaging unit, and photodiodes constituting each pixel are two-dimensionally arranged in a matrix, and each photodiode generates a photoelectric conversion current corresponding to the amount of received light. The photoelectric conversion current is accumulated by a capacitor connected to each photodiode. A Bayer array RGB filter is arranged in front of each pixel. The image sensor 103 has an electronic shutter. The electronic shutter controls the exposure time by controlling the time from charge accumulation to charge readout of the image sensor 103. Note that the image sensor 103 is not limited to the Bayer array, and may of course be a stacked form such as Foveon (registered trademark). The imaging element 103 functions as an imaging unit that images a subject and outputs image data.

  The image sensor 103 is connected to an analog processing unit 105. The analog processing unit 105 performs waveform shaping on the photoelectric conversion signal (analog image signal) read from the image sensor 103 while reducing reset noise and the like. Further, the gain is increased so as to obtain a more appropriate luminance.

  The analog processing unit 105 is connected to an A / D conversion unit 107. The A / D conversion unit 107 performs analog-digital conversion on the analog image signal, and converts the digital image signal (hereinafter referred to as image data) into the bus 110 and The data is output to the RAW data compression unit 108. In this specification, raw image data before image processing in the image processing unit 109 is referred to as RAW data.

  The RAW data compression unit 108 compresses the RAW data output from the A / D conversion unit 107 and uses it when capturing RAW data other than the reference image. The reference image is an image used as a reference when combining a plurality of pieces of image data. As the reference image, any one of a plurality of pieces of image data to be combined in an image combining unit 109c, which will be described later, is first captured image data, focused image data, or image data captured with appropriate exposure. It is.

  The RAW data compression unit 108 converts the A / D converted RAW data into data having a data amount smaller than that of the input data using a polygonal line or a table. A simple conversion method such as truncation of lower bits or multiplication by a coefficient less than 1 may be used. However, in consideration of gamma conversion in the image processing unit 109, conversion using a polygonal line or a table may reduce image quality degradation. it can. The RAW data compression unit 108 functions as an image data amount reduction unit that reduces the amount of image data and generates image data for synthesis that has a smaller data amount than the reference image data. The image data amount reduction unit reduces the data amount by an irreversible method when reducing the image data amount. The compression characteristics in the RAW data compression unit 108 will be described later with reference to FIG.

  The bus 110 is a transfer path for transferring various data read or generated in the camera body 100 to the camera body 100. The bus 110 includes an image processing unit 109, an AE (Auto Exposure) processing unit 111, an AF (Auto Focus) processing unit 113, an image compression unit 115, in addition to the A / D conversion unit 107 and the RAW data compression unit 108 described above. An image expansion unit 117, a microcomputer 121, an SDRAM 127, a memory interface (hereinafter referred to as a memory I / F) 129, and a display driver 133 are connected.

  The image processing unit 109 includes a RAW data expansion unit 109a that expands compressed RAW data, a basic image processing unit 109b that performs normal image processing, and an image composition unit 109c that performs image composition. When combining a plurality of images, the RAW data expansion unit 109a, the basic image processing unit 109b, and the image combining unit 109c are used.

  Image data other than the reference image is compressed by the RAW data compression unit 108, and the RAW data decompression unit 109a decompresses the compressed RAW data. The RAW data decompression unit 109a applies the reverse characteristics of the RAW data compression unit 108, and converts the data into the same data immediately after A / D conversion. If the compression is reversible, the data is the same. If the compression is irreversible, the data is converted to the same data. In general, since the data after A / D conversion is linear with respect to the exposure amount, the compressed RAW data stored in the SDRAM is converted into linear RAW data. The expansion characteristics in the RAW data expansion unit 109a will be described later with reference to FIG.

  The basic image processing unit 109b performs optical black (OB) subtraction processing, white balance (WB) correction, synchronization processing performed in the case of Bayer data, color reproduction processing, gamma correction processing, color matrix calculation, noise on RAW data. Reduction (NR) processing, edge enhancement processing, and the like are performed. If a single image is shot and no special effect or the like is set, the image processing is completed only by the processing by the basic image processing unit 109a.

  The image composition unit 109c performs various image composition according to the set composition mode and the like. The image synthesizing unit 109c functions as an image synthesizing unit that synthesizes image data using the reference image data and the image data for synthesis, and generates synthesized image data. In the present embodiment, as will be described later, three types of composition modes can be set: HDR composition that expands the dynamic range, depth composition that increases the depth of the object scene, and super-resolution composition that increases the resolution. When the HDR synthesizing mode is set, the image synthesizing unit 109c performs alignment and synthesizing with respect to the reference image with respect to the plurality of image data photographed with the plurality of exposure amounts, so that the single image can be obtained. Generate a wide dynamic range image.

  When the depth compositing mode is set, the image compositing unit 109c performs alignment and compositing with respect to the reference image for a plurality of image data photographed at a plurality of focus positions, so that a single image is captured. To generate an image with a different depth of field. When the super-resolution composition mode is set, the image composition unit 109c can perform alignment and composition with respect to the reference image with respect to a plurality of image data, so that a sense of resolution can be improved as compared with single shooting. Generate an image.

  The AE processing unit 111 measures subject brightness based on image data input via the bus 110, and outputs the subject brightness information to the microcomputer 121 via the bus 110. Although a dedicated photometric sensor may be provided for measuring the subject brightness, in this embodiment, the subject brightness is calculated based on the image data.

  The AF processing unit 113 extracts a high-frequency component signal from the image data, acquires a focus evaluation value by integration processing, and outputs the focus evaluation value to the microcomputer 121 via the bus 110. In the present embodiment, the photographing lens 201 is focused by a so-called contrast method. In this embodiment, the AF control based on the contrast method has been described as an example. However, the subject luminous flux is divided and a phase difference sensor is provided on the optical path, or a phase difference sensor is provided on the image sensor, and AF control based on the phase difference AF is performed. You may also focus.

  The image compression unit 115 compresses the image data read from the SDRAM 127 when recording the image data on the recording medium 131 according to various compression methods such as a JPEG compression method for a still image and MPEG for a moving image. To do.

  The image decompression unit 117 also decompresses JPEG image data and MPEG image data for image reproduction display. In decompression, the file recorded on the recording medium 131 is read out, subjected to decompression processing in the image decompression unit 115, and the decompressed image data is temporarily stored in the SDRAM 127. In the present embodiment, a JPEG compression method or an MPEG compression method is adopted as an image compression method, but the compression method is not limited to this, and TIFF, H.264, or the like. Of course, other compression methods such as H.264 may be used. The compression method may be lossless compression or lossy compression.

  The microcomputer 121 functions as a control unit for the entire camera, and comprehensively controls various sequences of the camera according to a program stored in the flash memory 125. In addition to the I / F 199 described above, an operation unit 123 and a flash memory 125 are connected to the microcomputer 121.

  The operation unit 123 includes operation members such as various input buttons and various input keys such as a power button, a release button, a moving image button, a playback button, a menu button, a cross key, and an OK button, and detects operation states of these operation members. The detection result is output to the microcomputer 121. The microcomputer 121 executes various sequences according to the user's operation based on the detection result of the operation member from the operation unit 123. The power button is an operation member for instructing to turn on / off the power of the digital camera. When the power button is pressed, the power of the digital camera is turned on. When the power button is pressed again, the power of the digital camera is turned off.

  The release button includes a first release switch that is turned on when the button is half-pressed and a second release switch that is turned on when the button is further depressed after being half-pressed and then fully pressed. When the first release switch is turned on, the microcomputer 121 executes a shooting preparation sequence such as an AE operation and an AF operation. When the second release switch is turned on, the mechanical shutter 101 and the like are controlled, image data based on the subject image is acquired from the image sensor 103 and the like, and a series of shooting sequences for recording the image data on the recording medium 131 are executed. Take a picture.

  The moving image button is an operation button for instructing start and end of moving image shooting. When the moving image button is first operated, moving image shooting is started, and when operated again, moving image shooting is ended. The playback button is an operation button for setting and canceling the playback mode. When the playback mode is set, the image data of the captured image is read from the recording medium 131 and the captured image is reproduced and displayed on the display panel 135.

  The menu button is an operation button for displaying a menu screen on the display panel 135. Various camera settings can be made on the menu screen. Examples of camera settings include synthesis modes such as HDR synthesis, depth synthesis, and super-resolution synthesis.

  The flash memory 125 stores a program for executing various sequences of the microcomputer 121. The microcomputer 121 controls the entire camera based on this program.

  The SDRAM 127 is an electrically rewritable volatile memory for temporary storage of image data and the like. The SDRAM 127 temporarily stores image data output from the A / D conversion unit 107 and image data processed by the image processing unit 109, the image compression unit 115, the image expansion unit 117, and the like.

  The memory I / F 129 is connected to the recording medium 131, and controls writing and reading of image data and data such as a header attached to the image data on the recording medium 131. The recording medium 131 is, for example, a recording medium such as a memory card that is detachably attached to the camera body 100, but is not limited thereto, and may be a hard disk or the like built in the camera body 100. The recording medium 131 functions as an image recording unit that records the composite image data.

  The display driver 133 is connected to the display panel 135, and displays an image on the display panel 135 based on the image data read from the SDRAM 127 or the recording medium 131 and expanded by the image expansion unit 117. The display panel 135 is disposed on the back surface of the camera body 100 and performs image display. The display panel 135 is a display unit that is easily affected by external light because the display surface is disposed on the exterior of the camera body such as the back surface, but a large display panel can be set. In addition, as a display part, various display panels, such as a liquid crystal display panel (LCD, TFT) and organic EL, are employable.

  As the image display on the display panel 135, a rec view display that displays recorded image data for a short time immediately after shooting, a playback display of a still image or moving image file recorded on the recording medium 131, a live view display, or the like. Video display.

  Next, the main processing of the camera in this embodiment will be described using the flowcharts shown in FIGS. 2 and 3, and the flowcharts shown in FIGS. 4, 5, 9, 10, 12, and 13, which will be described later, are controlled by the microcomputer 121 in accordance with a program stored in the flash memory 125. And run.

  When the power button in the operation unit 123 is operated and the power is turned on, the main flow shown in FIG. When the operation is started, first, initialization is executed (S1). As initialization, electrical initialization such as mechanical initialization and initialization of various flags is performed. As one of the various flags, a recording flag indicating whether or not a moving image is being recorded is reset to OFF (see steps S13, S15, S31, etc.).

  Once initialization has been carried out, it is next determined whether or not the playback button has been pressed (S3). Here, the operation state of the playback button in the operation unit 123 is detected and determined. If the result of this determination is that the playback button has been pressed, playback / editing is executed (S5). Here, image data is read from the recording medium 131 and a list of still images and moving images is displayed on the LCD 135. The user operates the cross key to select an image from the list, and confirms the image with the OK button. In addition, the selected image can be edited.

  When playback / editing in step S5 is executed, or if the result of determination in step S3 is that the playback button has not been pressed, it is determined whether or not camera settings are to be made (S7). When a menu button in the operation unit 123 is operated, camera settings are performed on the menu screen. Therefore, in this step, determination is made based on whether or not this camera setting has been performed.

  If the result of determination in step S7 is camera setting, camera setting is carried out (S9). As described above, various camera settings can be performed on the menu screen. As described above, as camera settings, for example, normal shooting, HDR synthesis, depth synthesis, super-resolution synthesis, and the like can be set as the shooting mode. As the still image recording mode, modes such as JPEG recording, TIFF recording, JPEG-RAW recording, and RAW recording can be set. Movie recording modes include Motion JPEG recording, H.264 recording, It is possible to set a mode such as H.264 recording. As the image quality mode, a mode such as Fine or Normal can be set.

  If camera setting is performed in step S9, or if the result of determination in step S7 is not camera setting, it is next determined whether or not the movie button has been pressed (S11). Here, the microcomputer 121 inputs the operation state of the moving image button from the operation unit 123 and determines.

  If the result of determination in step S11 is that a movie button has been pressed, the recording flag is reversed (S13). The recording flag is set to on (1) during moving image shooting, and is reset to off (0) when no moving image is being shot. In this step, the flag is inverted, that is, when ON (1) is set, it is inverted to OFF (0), and when OFF (0) is set, it is inverted to ON (1). Let

  If the recording flag is reversed in step S13, it is next determined whether or not a moving image is being recorded (S15). Here, the determination is made based on whether the recording flag inverted in step S13 is set to ON or OFF.

  If the result of determination in step S15 is that movie recording is in progress, a movie file is generated (S19). In step S61, which will be described later, a moving image is recorded. In this step, a moving image file for moving image recording is generated, and preparation is made so that moving image data can be recorded.

  On the other hand, if the result of determination is that a moving image is not being recorded, the moving image file is closed (S17). Since the moving image button is operated and the moving image shooting is completed, the moving image file is closed in this step. When the moving image file is closed, the number of frames is recorded in the header of the moving image file so that the moving image file can be reproduced, and the file writing is ended.

  If the moving image file is closed in step S17, or if the moving image file is generated in step S19, or if the result of determination in step S11 is that the moving image button has not been pressed, it is next determined whether or not moving image recording is in progress. Perform (S31). In this step, as in step S15, the determination is made based on whether the recording flag is on or off.

  If the result of determination in step S31 is that moving image recording is not in progress, it is determined whether or not the release button has been half-pressed, in other words, whether or not the first release switch has been turned on from off (S33). This determination is made based on the detection result obtained by detecting the state of the first release switch linked to the release button by the operation unit 123. As a result of the detection, when the first release switch is changed from OFF to ON, the determination result is Yes. On the other hand, when the ON state or the OFF state is maintained, the determination result is No.

  If the result of determination in step S33 is that the release button has been half-pressed and transitioned from OFF to first release, AE / AF operation is executed (S35). Here, the AE processing unit 111 detects subject brightness based on the image data acquired by the image sensor 103, and calculates a shutter speed, an aperture value, and the like for appropriate exposure based on the subject brightness.

  In step S33, an AF operation is performed. Here, the driver 205 moves the focus position of the photographing lens 201 via the microcomputer 207 in the interchangeable lens 200 so that the focus evaluation value acquired by the AF processing unit 113 becomes a peak value. Accordingly, when the movie shooting is not being performed and the release button is pressed halfway, the photographing lens 201 is focused at that time. Thereafter, the process proceeds to step S35.

  If the result of determination in step S31 is that the release button has not transitioned from off to first release, it is next determined whether or not the release button has been fully pressed and the second release switch has been turned on (S41). . In this step, the state of the second release switch linked to the release button is detected by the operation unit 123, and a determination is made based on the detection result.

  If the result of determination in step S41 is that the release button has been fully pressed and the second release switch has been turned on, shooting and image processing are carried out (S43). Here, the aperture 203 is controlled by the aperture value calculated in step S33, and the shutter speed of the mechanical shutter 101 is controlled by the calculated shutter speed. When the exposure time corresponding to the shutter speed elapses, an image signal is read from the image sensor 103, and RAW data processed by the analog processing unit 105 and the A / D conversion unit 107 is output to the bus 110.

  In step S43, image processing is performed after shooting. RAW data acquired by the image sensor 103 is read, and image processing is performed by the image processing unit 109. If the image composition mode is set, in this step, shooting is performed a plurality of times in accordance with the image composition mode, and image data used for image composition is temporarily stored in the recording medium 133 or the like. evacuate. Detailed operations of the photographing / image processing will be described later with reference to FIGS.

  Once shooting / image processing has been performed, still image recording is performed (S45). Here, the image data of the still image subjected to the image processing is recorded on the recording medium 131. When recording a still image, recording is performed in a set format. If JPEG is set, the image processed data is JPEG compressed and recorded by the image compression unit 115. In the case of the TIFF format, it is converted into RGB data and recorded in the RGB format. In addition, when RAW recording is set, when combining with RAW data acquired by photographing, combined RAW data is also recorded. Here, when RAW data acquired by photographing is combined, a plurality of RAW data before combining is recorded, and in step S5, the user can select a plurality of RAW data as one set, and the user instructs editing Alternatively, only the image processing portion in step S43 (see FIG. 5) may be processed. The recording destination of the image data may be the recording medium 131 in the camera body, or may be recorded in an external device via a communication unit (not shown).

  If the result of determination in step S41 is not the release button 2nd, or if the result of determination in step S31 is that moving image recording is in progress, then AE is performed (S51). If the determination in Step S41 is No, it means that no operation is performed on the release button. In this case, live view display is performed in Step S57 described later. If the determination in step S31 is Yes, moving image recording is in progress. In this step, the shutter speed and ISO sensitivity of the electronic shutter of the image sensor 103 for performing live view display or moving image shooting with appropriate exposure are calculated.

  Once AE is performed, next, photographing with an electronic shutter is performed (S53). Here, the subject image is converted into image data. That is, charge accumulation is performed for an exposure time determined by the electronic shutter of the image sensor 103, and image data is acquired by reading the accumulated charge when the exposure time has elapsed.

  Once photographing with the electronic shutter is performed, next, image processing is performed on the acquired image data (S55). In this step, the basic image processing unit 109b performs basic image processing such as WB correction, color matrix calculation, gamma conversion, edge enhancement, and noise reduction.

  Once the basic image processing is performed, next, live view display is performed (S57). In this step, live view display is performed on the display panel 135 using the image data that has undergone the basic image processing in step S55. That is, since image data is acquired and image processing is performed in step S53, the live view display is updated using the processed image. The photographer can determine the composition and shutter timing by observing the live view display.

  If live view display is performed in step S57, it is next determined whether or not moving image recording is in progress (S59). Here, it is determined whether or not the recording flag is ON. If the result of this determination is that video recording is in progress, video recording is performed (S61). Here, the image data read from the image sensor 103 is subjected to image processing on moving image data and recorded in a moving image file.

  If moving image recording is performed in step S61, or if the result of determination in step S59 is that moving image recording is not being performed, still image recording is performed in step S45, or AE / AF is performed in step S35, It is determined whether or not it is off (S37). In this step, it is determined whether or not the power button of the operation unit 123 has been pressed again. If the result of this determination is that power is not off, processing returns to step S3. On the other hand, if the result of determination is that the power is off, the main flow is terminated after the main flow is completed.

  As described above, in the main flow in the first embodiment of the present invention, the image composition mode or the like can be set (S9). When the image composition mode is set, the shooting / image processing is performed according to the image composition mode. Then, photographing is performed a plurality of times (S43), and the synthesized image data is recorded (S45).

  Note that when image data is recorded on the recording medium 131 in step S45 or S61, the image data is compressed in steps S115, S127, and S138, which will be described later, and the image composition is performed in steps S157, S167, and S177. The reduction information may be attached to the composite image data and recorded on the recording medium 131. In this case, the reduction information may be, for example, a reduction rate in addition to the presence or absence of data amount reduction.

  In steps S115, S127, and S137, when the image data is reduced, the RAW data compression unit responds according to the data amount of the image data and according to the condition of the transmission destination when transmitting data to the outside of the camera. The image data reduction amount (reduction rate) may be changed.

  Next, detailed operations of the photographing / image processing in step S43 will be described using the flowcharts shown in FIGS. When the shooting / image processing flow is entered, first, proper exposure shooting is performed (S101). Here, when the release button is half-pressed, shooting is performed under the exposure condition calculated in step S35. In addition, since focus adjustment is performed in step S35, shooting is performed in a focused state. Depending on the shooting mode and the like, shooting at this step may not be performed with proper exposure, or may be in an out-of-focus state.

  Once proper exposure shooting has been performed, RAW capture is performed (S103). The shooting in step S101 is a reference image. In order to prioritize the image quality, the RAW image data is not compressed by the RAW data compression unit 108, and the RAW image data is captured and stored in the SDRAM 127. Here, the image sensor 103 functions as a reference image generation unit that generates a reference image from image data.

  Once RAW capture has been carried out, it is next determined whether or not HDR synthesis is performed (S105). In the camera setting in step S9, it is determined whether the HDR synthesis mode is set as the shooting mode. As described above, the HDR synthesizing mode is a mode for generating an image having a dynamic range wider than that of a single shot by using a plurality of image data acquired by sequentially photographing with different exposure amounts.

  If the result of determination in step S105 is HDR composition, exposure amount change is performed (S111). For example, the exposure amount is sequentially changed to −1 step and +1 step with respect to the appropriate exposure condition determined in step S35. When the exposure amount is changed, a composite image is taken (S113). Here, photographing is performed according to the exposure amount changed in step S111.

  When shooting is performed, RAW compression capture is performed (S115). Since an image photographed by changing the exposure amount is not a reference image, some deterioration in image quality is allowed. Therefore, in order to reduce the capacity of the image data, the read image data is compressed by the RAW data compression unit 108, and the compressed RAW image data is temporarily stored in the SDRAM 127 as the composition image RAW data.

  Once RAW compression capture has been performed, it is next determined whether or not the number of composite images to be captured is a predetermined number (S117). Here, it is determined whether or not the number of sheets set for performing the HDR composition, for example, the number of images for compositing images has reached two. As a result of this determination, if the predetermined number has not been reached, the process returns to step S111 to continue shooting for HDR composition.

  If the result of determination in step S105 is not HDR combining, it is next determined whether or not it is depth combining (S121). In the camera setting in step S9, it is determined whether or not the depth synthesis mode is set as the shooting mode. As described above, the depth synthesis mode is a mode for generating an image with a depth of field different from that of single shooting by using a plurality of image data acquired by sequentially capturing images at different focus positions.

  If the result of determination in step S121 is depth synthesis, focus movement is performed (S123). Here, the microcomputer 207 in the interchangeable lens 200 is instructed to sequentially move the photographing lens 201 to a predetermined focus position. The predetermined position may be, for example, a position changed by 1/8 step from the closest side to infinity.

  Once the focus is moved, proper exposure shooting is performed (S125). Here, a composite image is taken according to the appropriate exposure condition determined in step S35. Once proper exposure photography has been performed, RAW compression capture is then performed (S127), as in step S115. Here, the read image data is compressed by the RAW data compression unit 108, and the compressed RAW image data is temporarily stored in the SDRAM 127 as the composition image RAW data.

  Once RAW compression and capture have been performed, it is next determined whether or not the number of composite images to be captured is a predetermined number (S129). Here, when the number of images set for performing the depth composition, for example, when the number of steps at the focus position is set to 1/8, whether or not the number of images to be combined has reached the predetermined number = 8 is determined. Determine whether. If the result of this determination is that the predetermined number has not been reached, the process returns to step S123 to continue shooting for depth synthesis.

  If the result of determination in step S121 is not depth combining, it is next determined whether or not super-resolution combining (S131). In the camera setting in step S9, it is determined whether or not the super-resolution composition mode is set as the photographing mode. As described above, the super-resolution synthesis mode is a mode in which a plurality of image data is used to generate an image having a sense of resolution rather than a single shot. Specifically, in the case of sensor camera shake correction, a plurality of images are sequentially taken while shifting the sensor (image sensor 103), and in the case of lens camera shake correction, the correction lens is shifted by, for example, one pixel of the sensor. In this mode, when a plurality of image data is acquired, an image having a high resolution different from that of single shooting is generated using these image data.

  If the result of determination in step S131 is super-resolution synthesis, optical axis fine movement is performed (S133). Here, in the case of sensor camera shake, the position of the sensor is slightly moved in a plane orthogonal to the optical axis so that the image sensor 103 is moved, for example, by one pixel.

  When the optical axis is finely moved, appropriate exposure shooting is performed as in step S125 (S135). Here, a composite image is taken according to the appropriate exposure condition determined in step S35. Once proper exposure photography has been carried out, RAW compression and capture are carried out in the same manner as in step S115 (S137). Here, the read image data is compressed by the RAW data compression unit 108, and the compressed RAW image data is temporarily stored in the SDRAM 127 as the composition image RAW data.

  Once RAW compression capture has been performed, it is next determined whether or not the number of composite images to be captured is a predetermined number (S139). Here, a predetermined number of images set for super-resolution composition, for example, 3 pixels shifted from the first shooting position, 1 pixel left, 1 pixel down, 1 pixel down, left It is determined whether or not. If the result of this determination is that the number of images for compositing has not reached the predetermined number, processing returns to step S133 and photographing for super-resolution composition is continued.

  If the result of determination in steps S117, S129, and S139 is that the predetermined number is reached, then image synthesis is performed in step S151 and subsequent steps. First, as in step S105, it is determined whether or not HDR synthesis is performed (S151). If the result of this determination is HDR synthesis, then RAW acquisition and decompression are performed (S153). The RAW data of the reference image and the RAW data of the composition image are acquired from the SDRAM 127. In this case, the RAW data of the reference image is uncompressed, but the RAW data of the composition image is compressed. Therefore, the RAW data decompression unit 109a performs decompression processing on the RAW data of the composition image.

  Once RAW acquisition and decompression have been performed, alignment is performed (S155). Here, alignment is performed on the image RAW data of the reference image acquired in step S153 and the image RAW data of the compositing image.

  Once alignment has been performed, HDR synthesis is then performed (S157). Here, the RAW data for combination sequentially captured with different exposure amounts is sequentially read out from the RAW data of the reference image, and the read out RAW data for combination has a wider dynamic range than that of the single shot. Generate an image.

  Once the HDR composition has been carried out, it is next determined whether or not the number of combined images for composition has reached a predetermined number, similar to step S117 (S159). Here, it is determined whether or not HDR composition has been performed for the number of images taken for HDR composition. If it is determined that the predetermined number has not been reached, the process returns to step S153 to continue the HDR synthesis.

  If the result of determination in step S151 is not HDR combining, it is next determined whether or not it is depth combining as in step S121 (S161). As a result of this determination, in the case of depth synthesis, RAW acquisition and decompression are performed as in step S153 (S163). Subsequently, alignment is performed as in step S155 (S165).

  Once alignment has been performed, next, depth synthesis is performed (S167). Here, with respect to the RAW data of the reference image, the composition RAW data sequentially photographed at different focus positions is sequentially read out, and the read out RAW data for composition is used to change the object field different from the single-shot. Generate depth images.

  Once the depth synthesis has been performed, it is then determined whether or not the number of synthesized images for synthesis has reached a predetermined number, similar to step S129 (S169). Here, it is determined whether or not depth composition has been performed for the number of images photographed for depth composition. If it is determined that the predetermined number has not been reached, the process returns to step S163 to continue the depth synthesis.

  If the result of determination in step S161 is not depth synthesis, it is next determined whether or not super-resolution synthesis is performed, as in step S131 (S171). As a result of this determination, in the case of super-resolution synthesis, RAW acquisition and expansion are performed in the same manner as in step S153 (S173). Subsequently, alignment is performed as in step S155 (S175).

  Once alignment has been performed, super-resolution synthesis is then performed (S177). Here, with respect to the RAW data of the reference image, the RAW data for synthesis sequentially photographed at different optical axis positions is sequentially read out, and the resolution different from that of single shooting is used by using the readout RAW data for synthesis. Generate a feeling image.

  Once super-resolution composition has been carried out, it is next determined whether or not the number of synthesized images for synthesis has reached a predetermined number, as in step S139 (S179). Here, it is determined whether super-resolution composition has been performed for the number of images taken for super-resolution composition. As a result of this determination, if the predetermined number has not been reached, the process returns to step S173 and the super-resolution composition is continued.

  If the result of determination in steps S159, S169, and S179 is a predetermined number, then basic image processing is performed (S181). Here, the basic image processing unit 109b performs basic image processing, that is, images such as WB correction, color matrix calculation, gamma conversion, edge enhancement, noise reduction, and the like on the RAW data synthesized in steps S157, S167, and S177. . When the basic image processing is performed, the shooting / image processing flow ends, and the flow returns to the original flow.

  As described above, in the shooting / image processing flow in the present embodiment, the reference image is first shot with appropriate exposure, the raw data of the reference image is taken in uncompressed (S103), and then the set shooting mode ( When shooting is performed according to the synthesis mode (S111, S113, S123, S125, S133, S135), the acquired RAW data is compressed and captured (S115, S127, S135). In this way, the image data for image composition is compressed and temporarily stored, so that even if the capacity of the memory for temporary storage is small, a large number of images can be obtained for image composition. Image quality can be improved.

  Next, compression characteristics in the RAW data compression unit 108 will be described with reference to FIG. FIG. 6A shows compression characteristics when the HDR synthesis mode is set as the photographing mode, and is compression characteristics used in RAW compression in step S115 (FIG. 4). In FIG. 6A and FIG. 6B, the horizontal axis represents input luminance, and the vertical axis represents output. By sampling the data compressed with the compression characteristics shown here with a smaller number of bits than the input luminance, it is possible to compress the RAW data while minimizing image quality degradation.

  In the example shown in FIG. 6A, in order to perform the HDR composition, in addition to the reference image taken with the proper exposure, the image with the +1 stage overexposure and the image with the -1 stage underexposure from the proper exposure are displayed. get. That is, in HDR composition, an image that has been over-photographed (+1 level shooting) has a dark portion gradation, and therefore a dark portion is used. To do.

  The RAW data compression unit 108 performs RAW compression along the compression characteristic line LC (+1) for the RAW data acquired by +1 step overexposure so that the dark portion of the subject can be easily reproduced. That is, when over-shooting is performed, the gradation of the dark part is maintained and the gradation of the bright part is compressed. On the other hand, RAW compression along the compression characteristic line LC (-1) is performed on the RAW data acquired by -1 stage underexposure so that the bright portion of the subject can be easily reproduced. That is, when under-shooting is performed, the gradation of the bright part is kept as much as possible, and the gradation of the dark part is compressed.

  In the drawing shown in FIG. 6A, the compression characteristic lines LC (+1) and LC (−1) are broken lines, but the characteristic may be a smooth curve. In the case of a polygonal characteristic line, the scale of hardware can be reduced. In particular, when the slope is a power of 2 (2k power: k is an integer), the conversion between the input value and the output value can be performed using bit shift without multiplication.

  FIG. 6B shows an example of the compression characteristic line LC when depth synthesis or super-resolution synthesis is set as the imaging mode, and shows the compression characteristics at the time of RAW compression in steps S127 and S137 (FIG. 4). is there. In the figure, the horizontal axis represents input luminance and the vertical axis represents output.

  In the depth synthesis, a high contrast portion is synthesized with a reference image. In super-resolution synthesis, data based on pixels of an image captured by shifting the position is interpolated between pixels in a single shot. Therefore, the composite portion is not changed according to the luminance. Therefore, after combining using RAW data, the data is compressed with characteristics similar to the gamma characteristics used in image processing. As a result, image quality degradation such as tone jump can be minimized when image processing is performed after synthesis. In FIG. 6B, the compression characteristic line LC draws a smooth curve, but it may have a polygonal line shape as shown in FIG.

  Next, the expansion characteristics in the RAW data expansion unit 109a will be described with reference to FIG. FIG. 7A shows the expansion characteristics when the HDR synthesis mode is set as the photographing mode, and is the expansion characteristics used in the RAW expansion in step S153 (FIG. 5). In FIG. 7A, the horizontal axis represents input luminance, and the vertical axis represents output.

  In the case of RAW expansion, the compression characteristic lines at the time of compression (shown by thin lines in the figure) LC (+1), LC (-1) are opposite to the expansion characteristic lines LE (+1), LE (-1 ) To return to the same data as the original RAW data. That is, in the case of over shooting, the change rate of the dark part of the subject is increased and the change rate of the bright part is reduced by the compression characteristic line LC (+1). As can be seen from the above, by decreasing the change rate of the dark part of the subject and increasing the change rate of the bright part, it is possible to reproduce the gradation of the dark part of the subject as compared with the case of single shooting. When under shooting is performed, the compression characteristic line LC (-1) and the expansion characteristic line LE (-1) tend to be opposite to those when over shooting is performed. Tones can be reproduced. The characteristic line L indicates the expansion characteristic in the case of non-compression.

  FIG. 7B shows the expansion characteristics when the depth synthesis / super-resolution synthesis mode is set as the imaging mode. The expansion characteristics used in the RAW expansion in steps S167 and S177 (FIG. 5). is there. Also in FIG. 7B, the horizontal axis represents input luminance, and the vertical axis represents output. Also in this case, in the case of RAW expansion, the compression characteristic line (denoted by a thin line in the figure) LC at the time of compression is converted with the reverse characteristic expansion characteristic line LE, so that it is the same as the original RAW data. Return to data.

  Thus, in the first embodiment of the present invention, the reference image data is temporarily stored without being compressed (S103 in FIG. 4), but the composition image data is temporarily RAW-compressed with the compression characteristics as shown in FIG. (S115, S127, S137 in FIG. 4). At the time of image composition, the RAW compressed image data is RAW decompressed with the decompression characteristics as shown in FIG. 7, and various kinds of data such as HDR composition are used by using the reference image data and the decompressed image data for composition. Image composition is performed. For this reason, it becomes possible to store a large number of image data for synthesis, and the quality of the synthesized image can be improved.

  Next, a second embodiment of the present invention will be described with reference to FIGS. In the first embodiment, image data of a composition image is temporarily stored after being RAW compressed, and image compression is performed after the compressed image data is RAW expanded. On the other hand, in the present embodiment, the image data of the image for synthesis is temporarily stored after down-sampling processing, and image synthesis is performed after the down-sampled image data is up-sampled. .

  Compared with the block diagram of FIG. 1 according to the first embodiment, the configuration of the present embodiment replaces the RAW data compression unit 108 and the RAW data decompression unit 109a in the first embodiment with a downsample processing unit 109d and an upsample. The difference is that the processing unit 109a is arranged and the connection relationship of the basic image processing unit 109b and the like is changed, and the others are the same. Therefore, the configuration of the present embodiment will be described with reference to FIG.

  In FIG. 8, the output of the A / D conversion unit 107 is output to the bus 110 and the basic image processing unit 109b. Similar to the first embodiment, the basic image processing unit 109b in the image processing unit 109 performs synchronization for RAW data in the case of optical black (OB) subtraction processing, white balance (WB) correction, and Bayer data. Processing, color reproduction processing, gamma correction processing, color matrix calculation, noise reduction (NR) processing, edge enhancement processing, and the like are performed. The image data processed by the basic image processing unit 109b is output to the bus 110 and the downsample processing unit 109d.

  The down-sample processing unit 109d in the image processing unit 109 performs processing to down-sample image data and reduce the data amount of the color difference component after the basic image processing by the basic image processing unit 109b. For example, YC422 data that leaves only one set of color difference components CbCr in units of two horizontal pixels or YC420 data that leaves only one set of CbCr in two horizontal and vertical pixels (2 × 2 region). The image data processed by the downsample processing unit 109 is output to the bus 110. It should be noted that how to generate a set of CbCr in downsampling is a method defined in the JEIDA standard digital file camera format (Exif) Version 2.1 (JEIDA-49-1998) and the like. Can be used.

  The down-sample processing unit 109d functions as an image data amount reduction unit that reduces the data amount of image data and generates image data for synthesis that has a smaller data amount than the reference image data. This image data amount reducing unit reduces the number of color difference signals (data amount) and the number of luminance signals (data amount) to be smaller. As a result, although the color resolution is lowered, the human perception special product is less likely to perceive the color change than the luminance change, so that the data amount can be reduced while suppressing the deterioration of the image quality.

  The basic image processing unit 109b processes both image data of the reference image and the composition image. The image data of the reference image is temporarily stored in the SDRAM 127 via the bus 110 after being processed by the basic image processing unit 109b. The On the other hand, the image data of the composition image is temporarily stored in the SDRAM 127 via the bus 110 after being processed by the down-sample processing unit 109d. Therefore, the image data of the reference image is not downsampled, and data (for example, YC444 data) having information on each pixel YCbCr is temporarily stored in the SDRAM 127, whereas the image data of the composition image is downsampled. Then, the data amount is reduced and temporarily stored in the SDRAM 127.

  An upsample processing unit 109e in the image processing unit 109 upsamples image data before the image composition by the image composition unit 109c. That is, when the color difference component CbCr in the image data is smaller than the luminance data (Y) (like the above-described YC422 and YS420 data), the color difference component CbCr is up-sampled and there is data for each pixel YCbCr. Convert as follows. In the up-sampling method, the color difference component CbCr may be simply copied, or it may be obtained by linear interpolation using neighboring CbCr in YC420 or YC422 format data. Note that the image data of the reference image (YC 444 format) already has CbCr for each pixel, so there is no need for upsampling processing.

  Next, the operation of the present embodiment will be described using the flowcharts shown in FIGS. The main operation in the present embodiment is the same as the flowcharts shown in FIGS. 2 and 3, and detailed description thereof is omitted. In the present embodiment, the flowcharts shown in FIGS. 4 and 5 according to the first embodiment may be replaced with the flowcharts shown in FIGS. 9 and 10, and in these flowcharts, steps S115, S127, and S137 in FIG. 9 is replaced with S115A, S127A, and S137A in FIG. 9, and steps S153, S163, and S173 in FIG. 5 are replaced with S153A, S163A, and S173A in FIG. Therefore, steps that perform the same processing as in FIG. 5 and FIG. 6 according to the first embodiment will be given the same step numbers and will not be described in detail, and differences will be mainly described.

  When the shooting flow shown in FIG. 9 is entered, shooting is performed with appropriate exposure (S101). In this step, exposure is performed with the exposure condition and the in-focus position determined in the first release of the release button (S35 in FIG. 3). Subsequently, basic image processing is performed (S103). In this step, the basic image processing unit 109b performs basic image processing on the image data acquired by the exposure in step S101, and temporarily stores the image data in the SDRAM 127 without down-sampling.

  Once the basic image processing is performed, next, when HDR composition is set, as in the first embodiment, shooting is performed while changing the exposure amount (S105 Yes, S111, S113). When shooting is performed, basic image processing and downsampling are performed (S115A). Here, the basic image processing unit 109b performs basic image processing on the acquired image data, the down-sampling processing unit 109d performs down-sampling processing on the image data, and the down-sampled image data is temporarily stored in the SDRAM 127. . When downsampling is performed, it is determined whether or not a predetermined number has been reached (S117). If not, the process returns to step S111.

  If depth composition is set, as in the first embodiment, shooting is performed while moving the focus (S121 Yes, S123, S125). When shooting is performed, basic image processing and downsampling are performed (S127A). Here, as in step S115A, the basic image processing unit 109b performs basic image processing on the acquired image data, and the down-sampling processing unit 109d performs down-sampling processing on the image data. Is temporarily stored in the SDRAM 127. When downsampling is performed, it is determined whether or not a predetermined number has been reached (S129). If not, the process returns to step S123.

  If super-resolution composition is set, as in the first embodiment, shooting is performed while slightly moving the optical axis (S131 Yes, S133, S135). When shooting is performed, basic image processing and downsampling are performed (S137A). Here, as in step S115A, the basic image processing unit 109b performs basic image processing on the acquired image data, and the down-sampling processing unit 109d performs down-sampling processing on the image data. Is temporarily stored in the SDRAM 127. When downsampling is performed, it is determined whether or not a predetermined number has been reached (S139). If not, the process returns to step S133.

  If the result of determination in steps S117, S129, and S139 is that the predetermined number is reached, then image synthesis is performed in step S151 and subsequent steps. First, when HDR synthesis is set as the photographing mode (S151 Yes), up-sampling is first performed (S153A). Here, the reference image temporarily stored in the SDRAM 127 and the image data of the composition image are read, and up-sampling is performed on the image data of the composition image. Since the image data of the composition image is stored as YC data in which the data amount is reduced by down-sampling as described above, the up-sampling processing unit 109e performs up-sampling processing at this step.

  When the up-sampling process is performed, alignment is performed (S155), HDR synthesis is performed (S157), it is determined whether the predetermined number has been reached (S159), and if not, the process returns to step S153A.

  Further, when the depth synthesis is set as the photographing mode (S161 Yes), up-sampling is first performed as in step S153A (S163A). Here, the reference image temporarily stored in the SDRAM 127 and the image data of the composition image are read, and the upsample processing unit 109e performs upsampling on the image data of the composition image. When the up-sampling process is performed, alignment is performed (S165), depth composition is performed (S167), it is determined whether the predetermined number has been reached (S169), and if not, the process returns to step S163A.

  If super-resolution composition is set as the photographing mode (S171 Yes), up-sampling is first performed as in step S153A (S173A). Here, the reference image temporarily stored in the SDRAM 127 and the image data of the composition image are read, and the upsample processing unit 109e performs upsampling on the image data of the composition image. When the up-sampling process is performed, alignment is performed (S175), super-resolution synthesis is performed (S177), it is determined whether the predetermined number has been reached (S179), and if not, step S173A is performed. Return.

  As described above, in the second embodiment of the present invention, the reference image data is temporarily stored without being downsampled (S103 in FIG. 9), but the image data for synthesis that has undergone the basic image processing is downsampled. (S115A, S127A, S137A in FIG. 9). When image synthesis is performed, the downsampled image data is upsampled and then subjected to various types of image synthesis such as HDR synthesis. For this reason, it becomes possible to store a large number of image data for synthesis, and the quality of the synthesized image can be improved.

  In this embodiment, the image data of the reference image is not downsampled. However, it is of course possible to perform downsampling so that the data amount is not reduced more than the image data of the image for synthesis. Absent. In addition, as an example of downsampling, the luminance signal Y and the color difference signal CbCr are used. Space) and down-sampling may be performed so that a signal representing a color difference component is reduced from a signal representing a luminance component.

  Next, a third embodiment of the present invention will be described with reference to FIGS. In the first embodiment, the image data of the compositing image is temporarily stored after RAW compression, and the compressed image data is subjected to RAW decompression and then image synthesis is performed. In the second embodiment, downsampling is performed. The image data is processed and temporarily stored, and the downsampled image data is upsampled and then the image is synthesized. On the other hand, in this embodiment, the image data of the image for synthesis is temporarily stored after image compression such as JPEG, and the image compression is performed after image compression of the image compressed image. ing.

  Compared with the block diagram of FIG. 1 according to the first embodiment, the configuration of the present embodiment is an image compression unit 109f and an image expansion unit instead of the RAW data compression unit 108 and the RAW data expansion unit 109a in the first embodiment. 109e is arranged, the connection relationship of the basic image processing unit 109b and the like is changed, and the image compression unit 115 and the image expansion unit 117 are overlapped, so that they are omitted, and the others are the same. Therefore, the configuration of the present embodiment will be described with reference to FIG.

  In FIG. 8, the output of the A / D conversion unit 107 is output to the bus 110 and the basic image processing unit 109b. Similar to the first embodiment, the basic image processing unit 109b in the image processing unit 109 performs synchronization for RAW data in the case of optical black (OB) subtraction processing, white balance (WB) correction, and Bayer data. Processing, color reproduction processing, gamma correction processing, color matrix calculation, noise reduction (NR) processing, edge enhancement processing, and the like are performed. The image data processed by the basic image processing unit 109b is output to the bus 110 and the image compression unit 109f.

  Similar to the image compression unit 115 in the first embodiment, the image compression unit 109f in the image processing unit 109 converts the image data into various types such as a JPEG compression method for a still image and MPEG for a moving image. Compress according to the compression method. The image compression unit 109f functions as an image data amount reduction unit that reduces the data amount of image data and generates image data for synthesis that has a smaller data amount than the reference image data. The image data amount reduction unit includes a compression unit that encodes and compresses image data. This compression unit performs lossy compression such as JPEG.

  The basic image processing unit 109b processes both image data of the reference image and the composition image. The image data of the reference image is temporarily stored in the SDRAM 127 via the bus 110 after being processed by the basic image processing unit 109b. The On the other hand, the image data of the composition image is temporarily stored in the SDRAM 127 via the bus 110 after being subjected to image compression processing by the image compression unit 109f. Therefore, the image data of the reference image is temporarily stored in the SDRAM 127 without image compression, whereas the image data of the image for synthesis is image-compressed and temporarily stored in the SDRAM 127 after reducing the data amount. The

  The image decompression unit 109g in the image processing unit 109 decompresses compressed image data such as JPEG image data and MPEG image data, as in the first embodiment. Since the image data of the reference image is uncompressed data, there is no need for image expansion processing.

  Next, the operation of this embodiment will be described using the flowcharts shown in FIGS. The main operation in the present embodiment is the same as the flowcharts shown in FIGS. 2 and 3, and detailed description thereof is omitted. Further, in the present embodiment, the flowcharts shown in FIGS. 4 and 5 according to the first embodiment may be replaced with the flowcharts shown in FIGS. 12 and 13, and in these flowcharts, steps S103 and S115 in FIG. , S127, and S137 are replaced with S103B, S115B, S127B, and S137B in FIG. 12, and steps S153, S163, and S173 in FIG. 5 are replaced with S153B, S163B, and S173B in FIG. Therefore, steps that perform the same processing as in FIG. 5 and FIG. 6 according to the first embodiment will be given the same step numbers and will not be described in detail, and differences will be mainly described.

  When the shooting flow shown in FIG. 12 is entered, shooting is performed with appropriate exposure (S101). In this step, exposure is performed with the exposure condition and the in-focus position determined in the first release of the release button (S35 in FIG. 3). Subsequently, basic image processing and low compression are performed (S103B). In this step, the basic image processing unit 109b performs basic image processing on the image data acquired by the exposure in step S101, and after the image compression unit 109f performs low compression, the image data is temporarily stored in the SDRAM 127. Here, the image sensor 103 and the image compression unit 109f that compresses image data with low compression function as a reference image generation unit that generates a reference image from the image data.

  When basic image processing and low compression are performed, next, when HDR composition is set, shooting is performed while changing the exposure amount as in the first embodiment (S105 Yes, S111, S113). When shooting is performed, basic image processing and high compression are performed (S115B). Here, the basic image processing unit 109b performs basic image processing on the acquired image data, the image compression unit 109f performs high-compression image compression on the image data, and the high-compressed image data is temporarily stored in the SDRAM 127. Let It is determined whether or not the predetermined number has been reached when performing high compression (S117). If not, the process returns to step S111.

  If depth composition is set, as in the first embodiment, shooting is performed while moving the focus (S121 Yes, S123, S125). When shooting is performed, basic image processing and high compression are performed (S127B). Here, as in step S115B, the basic image processing unit 109b performs basic image processing on the acquired image data, and the image compression unit 109f performs high-compression image compression processing and image compression on the acquired image data. Image data is temporarily stored in the SDRAM 127. It is determined whether or not the predetermined number has been reached when performing high compression (S129). If not, the process returns to step S123.

  If super-resolution composition is set, as in the first embodiment, shooting is performed while slightly moving the optical axis (S131 Yes, S133, S135). When shooting is performed, basic image processing and high compression are performed (S137B). Here, as in step S115B, the basic image processing unit 109b performs basic image processing on the acquired image data, and the image compression unit 109f performs high-compression image compression processing and image compression on the acquired image data. Image data is temporarily stored in the SDRAM 127. It is determined whether or not the predetermined number has been reached when performing high compression (S139). If not, the process returns to step S133.

  If the result of determination in steps S117, S129, and S139 is that the predetermined number is reached, then image synthesis is performed in step S151 and subsequent steps. First, when HDR composition is set as a shooting mode (S151 Yes), first, image expansion is performed (S153B). Here, the image data of the reference image and the composition image temporarily stored in the SDRAM 127 is read, the image data of the composition image is decompressed at a high expansion ratio, and the image data of the reference image is decompressed at a low expansion ratio. Perform image decompression. Since the image data of the image for synthesis is stored by reducing the data amount by high image compression as described above, the image expansion unit 109g performs image expansion processing at a high expansion rate in this step. Since the image data of the reference image is compressed by low image compression, image expansion is performed at a low expansion rate in this step.

  When image expansion is performed, alignment is performed (S155), HDR composition is performed (S157), it is determined whether or not the predetermined number has been reached (S159), and if not, the process returns to step S153B.

  If depth synthesis is set as the photographing mode (S161 Yes), first, image expansion is performed as in step S153B (S163B). Here, the image data of the reference image and the composition image temporarily stored in the SDRAM 127 is read, and the image expansion unit 109g performs image expansion at a low expansion rate or a high expansion rate. When the image is expanded, alignment is performed (S165), depth composition is performed (S167), and it is determined whether or not the predetermined number has been reached (S169). If not, the process returns to step S163B.

  When super-resolution composition is set as the photographing mode (S171 Yes), first, image decompression is performed as in step S153B (S173B). Here, the image data of the reference image and the composition image temporarily stored in the SDRAM 127 is read, and the image expansion unit 109g performs image expansion at a low expansion rate or a high expansion rate. When the image is expanded, alignment is performed (S175), super-resolution composition is performed (S177), and it is determined whether or not the predetermined number has been reached (S179). If not, the process returns to step S173B. .

  As described above, in the third embodiment of the present invention, the reference image data is subjected to image compression at a low compression rate and temporarily stored (S103B in FIG. 12), but the composition image data subjected to basic image processing is highly compressed. The image is compressed at a rate and temporarily stored (S115B, S127B, S137B in FIG. 13). When the images are combined, the compressed image data is subjected to various types of image combining such as HDR combining after image expansion. For this reason, it becomes possible to store a large number of image data for synthesis, and the quality of the synthesized image can be improved.

  In the present embodiment, the reference image data is image-compressed at a low compression rate, but may be temporarily stored in the SDRAM 127 without performing image compression. In this case, there is no need for image expansion.

  As described above, in each embodiment of the present invention, the subject is imaged to generate the image data of the reference image (for example, S101 and S103 in FIG. 4), and the subject is imaged to generate the image data of the composition image. (For example, S113, S125, and S135 in FIG. 4), the data amount of the image for synthesis is reduced so that the data amount is smaller than the image data of the reference image (for example, S115, S127, and FIG. 4). S137, FIG. 6), image data is synthesized by using the image data of the reference image and the image data of the synthesis image to generate synthesized image data (for example, S157, S167, S177 in FIG. 5). As described above, when a plurality of images are combined, the size of the image data to be combined is made smaller than the reference image data and temporarily stored in a memory (for example, SDRAM 127 in FIG. 1). As a result, a large number of captured images can be stored in a memory having a limited capacity, and a higher-quality composite image can be realized when compared with a memory having the same capacity.

  In each embodiment of the present invention, image data such as RAW compressed data is temporarily stored in the SDRAM 127, but may be temporarily stored in the storage medium 137 or the flash memory 123. There is an embodiment in which basic image processing is performed after composition processing is completed with RAW data, and conversely, there is an embodiment in which composition processing is performed after basic image processing. You may select suitably whether it performs a synthetic | combination process by data.

  In the first and second embodiments of the present invention, the image data of the reference image is not reduced in data amount, but the reduction rate (irreversible reduction method) is lower than the data amount of the image for synthesis. In this case, the data amount of the image data of the reference image may be reduced if the image quality is higher than that of the composition image.

  In each embodiment of the present invention, a digital camera has been described as an apparatus for photographing. However, the camera may be a digital single lens reflex camera or a compact digital camera, such as a video camera or a movie camera. It may be a camera for moving images, and may be a camera incorporated in a mobile phone, a smartphone, a personal digital assistant (PDA), a personal computer (PC), a tablet computer, a game machine, or the like. In any case, the present invention can be applied to any device for photographing that combines a plurality of image data. Further, the present invention can be applied to any image synthesizing apparatus that synthesizes a plurality of image data photographed by a photographing device.

  Of the techniques described in this specification, the control mainly described in the flowchart is often settable by a program and may be stored in a recording medium or a recording unit. The recording method for the recording medium and the recording unit may be recorded at the time of product shipment, may be a distributed recording medium, or may be downloaded via the Internet.

  In addition, regarding the operation flow in the claims, the specification, and the drawings, even if it is described using words expressing the order such as “first”, “next”, etc. It does not mean that it is essential to implement in this order.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, you may delete some components of all the components shown by embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 100 ... Camera body, 101 ... Mechanical shutter, 103 ... Imaging element, 105 ... Analog processing part, 107 ... A / D conversion part, 108 ... Raw data compression part, 109 ... Image processing unit 109a RAW data decompression unit 109b Basic image processing unit 109c Image synthesis unit 109d Down-sample processing unit 109e Up-sample processing unit 109g ... Image decompression unit, 110 ... Bus, 111 ... AE processing unit, 113 ... AF processing unit, 115 ... Image compression unit, 117 ... Image decompression unit, 121 ... Micro Computer, 123, operation unit, 125, flash memory, 127, SDRAM, 129, memory I / F, 131, recording medium, 133, display driver 1 DESCRIPTION OF SYMBOLS 5 ... Display panel, 199 ... I / F, 200 ... Interchangeable lens, 201 ... Shooting lens, 203 ... Aperture, 205 ... Driver, 207 ... Microcomputer, 209 ... Flash memory

Claims (11)

An imaging unit for imaging a subject with an imaging device and outputting image data;
A focusing unit that adjusts a focusing position so that the subject is focused on the image sensor of the imaging unit;
A reference image generation unit that generates reference image data from the image data;
An image data amount reduction unit that reduces the data amount of the image data and generates image data for synthesis that has a smaller data amount than the reference image data;
An image composition unit that synthesizes image data using the reference image data and the image data for composition, and generates composite image data;
I have a,
The image data amount reduction unit extracts a gamma component of the image data, reduces the data amount based on compression characteristics according to the gamma component,
The focusing unit acquires a plurality of image data output by the imaging unit at each of a plurality of different focusing positions, and the image data amount reduction unit acquires each of the plurality of image data. An image pickup apparatus that reduces the amount of data and generates composite image data by the image composition unit .   The image pickup apparatus according to claim 1, wherein the image data amount reduction unit reduces the bit accuracy of the image data. The imaging apparatus according to claim 1, wherein the image data amount reduction unit includes a gradation conversion unit using a table or a broken line, and performs gradation conversion on the image data. The imaging apparatus according to claim 1, wherein the image data amount reduction unit includes a compression unit that encodes and compresses the image data.   The imaging apparatus according to claim 4, wherein the compression unit performs lossy compression. The imaging apparatus according to claim 1, wherein the image data amount reduction unit reduces a data amount of the color difference signal.   2. The reference image data according to claim 1, wherein the reference image data is image data first captured or image data in focus among a plurality of image data to be combined in the image combining unit. Imaging device. An image recording unit for recording the composite image data;
Using the synthetic image data in which the data amount is reduced by the image data quantity reduction unit, when generating the composite image data, on the accompanying reduction information to the composite image data, the image Zoki recording unit The image pickup apparatus according to claim 1, wherein the image pickup apparatus is recorded on the image pickup apparatus.   The imaging apparatus according to claim 1, wherein the image data amount reduction unit reduces the data amount by an irreversible method. Image a subject with an image sensor to generate image data of a reference image,
The subject, and adjust the focus position to focus on SL IMAGING element,
Capture the subject and generate image data for the composition image,
Reduce the amount of data so that the image data of the image for synthesis is less than the amount of data of the image of the reference image,
Using the image data of the reference image and the image data of the image for synthesis, the image data is synthesized to generate synthesized image data,
Furthermore, the gamma component of the image data is extracted, and the data amount is reduced based on the compression characteristics corresponding to the gamma component.
Obtaining a plurality of image data output by the imaging element at each of a plurality of different in-focus positions, reducing a data amount of each of the plurality of image data, and generating composite image data;
An imaging method characterized by the above. Image a subject with an image sensor to generate image data of a reference image,
Capture the subject and generate image data for the composition image,
The subject, and adjust the focus position to focus on SL IMAGING element,
Reduce the amount of data so that the image data of the image for synthesis is less than the amount of data of the image of the reference image,
Using the image data of the reference image and the image data of the image for synthesis, the image data is synthesized to generate synthesized image data,
Furthermore, the gamma component of the image data is extracted, and the data amount is reduced based on the compression characteristics corresponding to the gamma component.
Obtaining a plurality of image data output by the imaging element at each of a plurality of different in-focus positions, reducing a data amount of each of the plurality of image data, and generating composite image data;
A program characterized by causing a computer to execute the above.