JP2016054447A - Imaging apparatus, image processing apparatus, image processing method and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus, an image processing apparatus, an image processing method and an image processing program which allow photographic images of an astronomical body to be photographed without being saturated and clipped, and subjected to image processing with filter effect equivalent to an optical filter.SOLUTION: A subject is photographed (a graph (a)) during exposure time (Tsec) shorter than standard exposure time (Ssec), image data of a plurality of frames obtained through the photographing are composed to generate composed image data (a graph (b)), filter processing is applied to the composed image data (a graph (c)), and bit lengths of the image data subjected to the filter processing are adjusted so as to be shorter than bit lengths of the composed image data (a graph (d)).SELECTED DRAWING: Figure 8

Description

  The present invention relates to an imaging apparatus, an image processing apparatus, and an image processing capable of performing filter processing equivalent to and higher performance than an optical filter frequently used for astrophotography in an imaging apparatus such as a digital still camera. The present invention relates to a method and an image processing program.

  In recent years, it has become possible to perform astronomical photography more easily than before due to higher performance and multifunctional imaging devices such as digital cameras, and more photographers have come to enjoy astronomical photography. Some stars have a brightness range of approximately -1.5 to 6 and so on, and the brightness is about 1000 times different. If you shoot a dark star that has a larger contrast than the digital camera's dynamic range (approx. 3 to 4 steps = 8 to 16 times) from the proper exposure to saturation, most of the stars will be clipped with digital saturation output. It will be.

  In the digital camera's optical system, the star image appears as a very sharp point image, so all stars with saturated digital output appear as a similar point image, and the brightness of the star is not well understood in photographs. If you can shoot without saturating the color and brightness of the stars, it will be an impressive and beautiful photo. In addition, information on stars and constellations can be easily distinguished from photographs, and the photographs are preferred for art and research purposes.

  Therefore, light diffusion is performed so that the brighter the stars are, the larger the size of the bright stars, the shape of the light stars, or the color information remains (so that the color information is not lost by saturating the output of the stars) An optical filter such as a filter (for example, a soft filter) or a cross filter is used. In addition, by performing HDR (wide dynamic range) processing as disclosed in Patent Document 1, it is possible to synthesize an image without saturating pixel outputs that would be saturated in conventional imaging.

Japanese Patent No. 4973115

  However, the use of the above-described optical filter is costly, and the image pickup apparatus becomes large, which makes installation difficult. Therefore, the image of the star can be greatly blurred by image processing that changes the shape of the light beam with the aim of producing the same effect as the optical filter without mounting the optical filter. However, as described above, even if image processing is performed on the pixel output that has been clipped because the digital output is saturated and the brightness information of the star is above saturation, the star image is not converted to a digital value. Compared with an optical filter that diffuses into the filter, the output of the pixel to be filtered is small, and an equivalent filter effect cannot be obtained. Also, because color information is lost due to saturation, color is lost from the star image.

  Further, in the HDR processing as in Patent Document 1, the pixel output can be recorded without being saturated. However, an image having the same effect as a photograph taken with the above-described optical filter attached to the camera cannot be taken only by the HDR process.

  The present invention has been made in view of such circumstances, and astronomical images in which data is clipped with full bits (hereinafter referred to as “saturated clips”) when full-bit digital output is reached in normal shooting. An image pickup apparatus capable of taking a picture without saturation clipping, performing image processing with a filter effect equivalent to that of an optical filter, and taking a more impressive astronomical photograph that cannot be realized by photography equipped with an optical filter An object is to provide an image processing apparatus, an image processing method, and an image processing program.

  In order to achieve the above object, an image pickup apparatus according to a first invention includes an image pickup unit that takes a plurality of frames of an object with an exposure time shorter than a standard exposure time, and image data of a plurality of frames obtained by shooting with the image pickup unit. An image composition unit that generates composite image data by combining the image data, a filter processing unit that performs filter processing on the image data before generating the composite image data or the composite image data synthesized by the image composition unit, The composite image data obtained by synthesizing the image data of the plurality of frames subjected to the filtering process or the bit length of the image data subjected to the filtering process on the synthesized image data synthesized by the image synthesizing unit A bit length adjustment unit that adjusts the bit length to be shorter than the data bit length.

In the imaging apparatus according to a second aspect of the present invention, in the first aspect, the imaging unit sets the exposure time such that the maximum value of the pixel data is not saturated and takes a picture.
In the imaging device according to a third aspect based on the first aspect, the imaging unit divides the image into a plurality of regions, and the average value of the image data of the regions corresponding to all the divided images is not saturated. Set the exposure time to take a picture.

In the imaging device according to a fourth aspect based on the first aspect, the bit length adjusting unit is synthesized by the synthesized image data obtained by synthesizing the image data of the plurality of frames subjected to the filtering process or by the image synthesizing unit. The image data subjected to the filtering process is clipped so that the bit length of the image data that has been subjected to the filtering process on the combined image data is shorter than the bit length of the combined image data.
In the imaging device according to a fifth aspect based on the first aspect, the bit length adjusting unit is synthesized by the synthesized image data obtained by synthesizing the image data of the plurality of frames subjected to the filtering process or by the image synthesizing unit. The gradation of the image data subjected to the filtering process is compressed so that the bit length of the image data subjected to the filtering process on the combined image data is shorter than the bit length of the combined image data.

The image pickup apparatus according to a sixth aspect of the present invention is the image pickup apparatus according to the first aspect, further comprising: an image output determination unit that determines a level of the image data before generating the combined image data or the image data combined by the image combining unit. And when performing the filtering process on the image data before generating the synthesized image data, the filtering process is performed on the synthesized image data according to the level of the image data before generating the synthesized image data. When performing the above, the filter processing is changed according to the level of the composite image data.
In the imaging device according to a seventh aspect based on the first aspect, the image composition unit performs at least one kind of image composition among addition composition, average composition, comparative light composition, comparative dark composition, and moving image composition. .

An image pickup apparatus according to an eighth aspect of the present invention further includes an image display unit that displays the image combined with the image combining unit in the first aspect of the invention, and the image display unit captures image data by the image pickup unit. The synthesized images obtained by sequentially synthesizing each time are sequentially displayed.
In the imaging device according to a ninth aspect based on the first aspect, the filtering process is a soft filtering process or a cross filtering process.

An image processing apparatus according to a tenth aspect of the invention is an image synthesizing unit that synthesizes a plurality of frames of image data obtained by photographing a subject with an exposure time shorter than a standard exposure time, and generates the synthesized image data. A filter processing unit that performs a filtering process on the image data before generating the image data or the synthesized image data synthesized by the image synthesizing unit, and a synthesis that combines the image data of the plurality of frames subjected to the filtering process A bit length adjusting unit that adjusts the bit length of the image data that has been subjected to the filtering process on the image data or the synthesized image data synthesized by the image synthesizing unit so as to be shorter than the bit length of the synthesized image data; .
An image processing method according to an eleventh aspect of the invention is an image synthesizing step for synthesizing a plurality of frames of image data obtained by photographing a subject with an exposure time shorter than a standard exposure time, and generating synthesized image data. A filter processing step for performing a filtering process on the image data before generating the image data or the synthesized image data synthesized in the image synthesizing step, and a synthesis in which the image data of the plurality of frames subjected to the filtering process are synthesized. A bit length adjusting step for adjusting the bit length of the image data or the image data that has been subjected to the filtering process on the synthesized image data synthesized in the image synthesizing step to be shorter than the bit length of the synthesized image data; .
An image processing program according to a twelfth aspect of the present invention is an image composition step for synthesizing a plurality of frames of image data obtained by photographing a subject with an exposure time shorter than a standard exposure time to generate composite image data; A filter processing step for performing a filtering process on the image data before generating the image data or the synthesized image data synthesized in the image synthesizing step, and a synthesis in which the image data of the plurality of frames subjected to the filtering process are synthesized. A bit length adjusting step for adjusting the bit length of the image data or the image data that has been subjected to the filtering process on the synthesized image data synthesized in the image synthesizing step to be shorter than the bit length of the synthesized image data; , Execute on the computer.

  According to the present invention, even an astronomical image that is saturated during normal photographing can be photographed without saturation clipping, and can be subjected to image processing with a filter effect equivalent to that of an optical filter, and further equipped with an optical filter. It is possible to provide an imaging apparatus, an image processing apparatus, an image processing method, and an image processing program capable of capturing a more impressive astronomical photograph that cannot be realized by photographing.

1 is a block diagram mainly showing an electrical configuration of a camera according to an embodiment of the present invention. It is a flowchart which shows the imaging | photography operation | movement of the camera which concerns on one Embodiment of this invention. It is a flowchart which shows the imaging | photography operation | movement of the camera which concerns on one Embodiment of this invention. It is a flowchart which shows the imaging | photography operation | movement of the camera which concerns on one Embodiment of this invention. It is a figure which shows an example of the filter coefficient of the soft filter of the camera which concerns on one Embodiment of this invention. It is a figure which shows an example of the filter coefficient of the cross filter of the camera which concerns on one Embodiment of this invention. The conventional example which provides the effect of a diffusion filter by image processing is shown. It is a figure which shows an example of the output at the time of providing the effect of a diffusion filter by image processing in the camera which concerns on one Embodiment of this invention. It is a figure which shows another example of the output at the time of providing the effect of a diffusion filter by image processing in the camera which concerns on one Embodiment of this invention.

  Hereinafter, a preferred embodiment using a camera to which the present invention is applied will be described. A camera according to a preferred embodiment of the present invention is a digital camera, and roughly displays image data based on image data read from an image sensor on a display unit and performs image processing for recording according to an operation of a release button. The recorded image data is recorded in the external memory.

  In addition, when the digital filter processing is set, the camera according to the present embodiment performs divided exposure (for example, see S15 and S17 in FIG. 3 and FIG. 8A), and the divided exposure time elapses. Image data is read out every time, and synthesized image data is generated by synthesizing a plurality of read image data (see, for example, S21, S27 in FIG. 3, and FIG. 8B), and a filter is applied to this synthesized image data Processing is performed (for example, see S33 in FIG. 4 and FIG. 8C). Then, the bit length of the image data subjected to the filter processing is adjusted so as to be shorter than the bit length of the composite image data (see, for example, S37 in FIG. 4 and FIG. 8D).

  FIG. 1 is a block diagram mainly showing an electrical configuration of a camera according to an embodiment of the present invention. The camera in the present embodiment includes an imaging unit 1, an image processing unit 10, an image output detection unit 19, a system control unit 20, and a bus 31 and each unit connected thereto. In the present embodiment, the lens 2 is configured integrally with the camera body, but may of course be an interchangeable lens.

  In the imaging unit 1, a lens 2, a mechanical shutter 3, and an image sensor 4 are provided. The imaging unit 1 functions as an imaging unit that captures a plurality of frames of a subject with a divided exposure time (for example, see S9 in FIG. 2) shorter than the standard exposure time (for example, see S7 in FIG. 2). Further, as will be described in detail later, the imaging unit sets a division exposure time so that the maximum value of the pixel data is not saturated (for example, refer to S9 in FIG. 2).

  The lens 2 forms an optical image of the subject on the image sensor 4. The lens 2 includes an aperture that changes its aperture diameter and determines an aperture value for adjusting the exposure amount. The lens 2 detects information such as the currently set focal length and outputs the detected information to the system control unit 20. Further, the mechanical shutter 3 performs exposure and shading on the image sensor 4 by opening and closing operations, and controls the exposure time.

  The image sensor 4 includes an image sensor such as a CMOS image sensor or a CCD image sensor, converts an optical image of a subject formed by the lens 2 into an electrical signal for each pixel, and converts the image data into an image output detection unit 19 and Output to bus 31. The image processing unit 10 described later inputs image data via the bus 31, performs various image processing, and outputs the image data to the bus 31. The bus 31 is a signal line for transmitting and receiving signals between the blocks. The image sensor 4 has an electronic shutter, and can read out image data every divided exposure time (Tsec) to be described later, and can simultaneously photograph the next frame.

  The image output detection unit 19 detects whether the output of the image data output from the image sensor 4 is saturated. Further, the image output detection unit 19 detects the subject luminance value based on the image data from the image sensor 4 in order to perform AE control. The detected subject luminance value is output to the system control unit 20, and in AE (Auto Exposure) control, the aperture of the lens 2, the shutter speed of the mechanical shutter 3, the electronic shutter speed of the image sensor 4, and the ISO sensitivity. Used to control exposure conditions such as

  The image processing unit 10 performs image processing on the image data output from the image sensor. The image processing unit 10 includes an image synthesis unit 11, a filter processing unit 13, a clip processing unit 15, and a development processing unit 17. The image synthesis unit 11 includes an addition synthesis unit 11a, a comparison synthesis unit 11b, and an addition average synthesis. Unit 11c and moving image generation unit 11d.

  The image synthesizing unit 11 synthesizes image data of a plurality of frames obtained by photographing with the image sensor 4 to generate synthesized image data. As will be described later, when shooting for one sheet, if the shooting time (see the standard exposure time (Ssec) of S7 in FIG. 2) is longer than a predetermined time, the divided exposure time (S9 of FIG. 2) is divided. Image data is read from the image sensor 4 at every exposure time (see Tsec) (see S13 to S17 in FIG. 3). The image composition unit 11 performs image composition of the read image data (see S21 in FIG. 3). The image synthesizing unit 11 functions as an image synthesizing unit that generates synthesized image data by synthesizing image data of a plurality of frames obtained by photographing a subject with a divided exposure time shorter than the standard exposure time.

  The addition combining unit 11a in the image combining unit 11 adds the outputs for each corresponding pixel among the plurality of image data read from the image sensor 4, and generates combined image data. The comparison / synthesis unit 11b compares outputs for each corresponding pixel among a plurality of image data read from the image sensor 4 and uses the pixel output of the larger output as the pixel data of the synthesized image, or The composite image data is generated by comparative dark composition using the pixel output with the smaller output as the pixel data of the composite image.

  The addition average combining unit 11c generates the combined image data by adding and averaging the outputs of the corresponding pixels among the plurality of image data read from the image sensor 4. The moving image generation unit 11d generates moving image data by connecting image data of a plurality of frames read continuously in time series. As described above, the image composition unit 11 performs at least one kind of image composition among addition composition, average composition, comparative light composition, comparative dark composition, and moving image composition. In the present embodiment, these four combinations can be performed, but any one or a plurality of them may be performed, and other image composition methods may be executable.

  The filter processing unit 13 performs filter processing such as soft filter processing and cross filter processing on image data that has been subjected to normal image processing by the image processing unit 10 and image data that has been subjected to image processing by the image synthesis unit 11. (See S33 in FIG. 4). An example of the filter coefficient in the filtering process will be described with reference to FIGS. The filter processing unit 13 functions as a filter processing unit that performs filter processing on the combined image data combined by the image combining unit.

  The clip processing unit 15 clips the image data filtered by the filter processing unit 13 with a predetermined bit (see, for example, S35 in FIG. 4). This clip processing will be described later with reference to FIGS. The clip processing unit 15 functions as a bit length adjustment unit that adjusts the bit length of the image data subjected to the filter processing so as to be shorter than the bit length of the composite image data.

  The development processing unit 17 performs development such as demosaicing, white balance adjustment, gamma correction, and image compression on the RAW image data generated by the image synthesis unit 11 and the RAW image data clipped by the clip processing unit 15. Process.

  In addition to the imaging unit 1, the image processing unit 10, and the image output detection unit 19, an internal memory 33, an external memory 36, a display unit 37, an input IF 38, and the system control unit 20 are connected to the bus 31. .

  The internal memory 33 temporarily stores various setting information necessary for camera operation and image data in the middle of image processing. The internal memory 33 is configured by a rewritable nonvolatile memory such as a flash memory or SDRAM.

  The external memory 36 is a non-volatile storage medium that can be loaded into the camera body or fixed inside, and is, for example, an SD card or a CF card. The external memory 36 records the image data developed by the development processing unit 13, and at the time of reproduction, the recorded image data can be read out and output to the outside of the camera.

  The display unit 37 includes a rear display unit such as a TFT (Thin Film Transistor) liquid crystal or an organic EL, and an EVF (electronic viewfinder), and displays an image developed by the development processing unit 17. The display unit 37 displays images such as live view display, playback display of recorded images, and menu screen display.

  The input IF 38 includes an operation member such as a release button, a touch panel for inputting a touch operation on the rear display unit, and the like, and instructs various shooting operations such as mode setting and release based on a user operation.

  The system control unit 20 includes a CPU (Central Processing Unit) and the like, and performs overall control by controlling each unit of the camera according to a program stored in the internal memory 33. In addition, the system control unit 20 performs overall control of the imaging unit 1, the image processing unit 10, the image output detection unit 19, and the like.

  Next, using the flowcharts shown in FIG. 2 to FIG. 4, the operation of the camera according to the present embodiment when digital filter processing is mainly selected will be described. This flowchart is executed by the system control unit 20 controlling each unit in accordance with a program stored in the internal memory 33.

  Although not explicitly shown in this flowchart, it is assumed that the photographer has set the exposure control mode of the camera via the input IF 38. As the exposure control mode, for example, an exposure program auto mode, a shutter speed priority mode, an aperture priority mode, an exposure manual mode, a bulb photographing mode, and the like can be set. In the present embodiment, a case where the digital filter setting for performing the filter processing in the image processing unit 10 is selected will be described.

  In the flowchart shown in FIG. 2, first, live view display is performed (S1). In this step, the image processing unit 10 performs image processing for live view display on the image data continuously read from the image sensor 4 at a predetermined frame rate, and the image processed on the display unit 37 is processed. A live view image is displayed using the data.

  When live view is displayed, the photographer shoots the composition, focus position, exposure conditions (exposure time, aperture value, ISO sensitivity, EV value, etc.) for the main shooting while checking the live view image displayed in real time. Set conditions. For example, when the exposure program auto mode is selected as the exposure condition, the image output detection unit 19 detects the brightness of the subject, and the system control unit 20 achieves proper exposure so-called APEX (Additive System of Photographic Exposure). ) Set the exposure conditions based on the calculation.

  In the shutter speed priority mode, the exposure condition is automatically set by fixing the shutter specified by the photographer and adjusting the other apertures and ISO sensitivity so as to obtain an appropriate exposure. Here, when the photographer has set exposure correction, the exposure condition is automatically set so that the exposure correction corresponding to the photographer's setting is performed with respect to the appropriate exposure amount. In the exposure manual mode, the photographer manually adjusts both the aperture and the shutter speed while checking the brightness of the live view image displayed on the display unit 37. Hereinafter, in various exposure control modes, among the exposure conditions set as described above, the exposure time is referred to as a standard exposure time (Ssec).

  In the bulb shooting mode, the aperture is a value set by the photographer, and shooting is continued while the release button is pressed, and shooting is terminated when the release is released. However, the present invention is not limited to this, and the shooting may be started by pressing the first release and the shooting may be ended by pressing the second release.

  Once live view display is performed in step S1, image output is detected next (S3). Here, the image output detection unit 19 detects the luminance level of the subject based on the image data read from the image sensor 4.

  If image output is detected in step S3, it is next determined whether or not the bulb photographing mode is set (S5). In this step, it is determined whether or not the bulb photographing mode is set by the input IF 38.

  If the result of determination in step S5 is that bulb shooting mode is not set, the standard exposure time (Ssec) is determined (S7). When exposure program auto or aperture priority mode is set as the exposure control mode, it is calculated by APEX calculation. When exposure manual mode or shutter speed priority mode is set, the standard exposure time is set by the user. (Ssec) is determined.

  When the standard exposure time (Ssec) is determined in step S7, or when the set exposure control mode is the bulb photographing mode as a result of the determination in step S5, the divided exposure time (Tsec) is determined next. (S9). Here, a division exposure time (Tsec) is determined that does not saturate even if the high-luminance output in the subject is taken with the aperture for the main shooting and ISO sensitivity.

  Here, the divided exposure time Tsec indicates an exposure time that does not saturate all the pixels of the image (in other words, does not reach the full bit or more when converted to digital output). Specifically, if there is a pixel that saturates even one pixel among all the pixels of the current live view image in the aperture setting and ISO setting for actual shooting, the electronic shutter speed (frame rate) for obtaining the live view image is shortened. Then, the exposure time during which the high-luminance pixels in the image are not saturated is determined as the divided exposure time Tsec.

  In step S9, when the divided exposure time (Tsec) is determined, it is determined whether or not the release button is pressed (S11). The photographer observes the live view display, determines the composition, and if the shutter timing is good, presses the release button to start photographing. In this step, the determination is made based on the state of the switch that is turned on in response to the depression of the release button that is one of the input IFs 38. If the release button is not depressed as a result of this determination, the process returns to step S1, the above-described live view display is performed, and the determination of the standard exposure time (Ssec) and the divided exposure time (Tsec) is repeated.

  If the release button is pressed as a result of the determination in step S11, it is next determined whether or not the bulb photographing mode is set or whether the standard exposure time (Ssec)> the divided exposure time (Tsec) (S13). If the divided exposure time (Tsec) is longer than the standard exposure time (Ssec), the high-luminance pixels in the subject are saturated. Therefore, in such a case, divided exposure is performed in steps S15 to S27 to generate a composite image. In the bulb exposure mode, the exposure time is determined by the end timing of the bulb exposure of the photographer, and the shutter speed for main photography is unknown. Therefore, an image is read out every time the divided exposure time Tsec elapses to generate a composite image. I am doing so.

  Note that when the release button is pressed down, shooting starts, but when shooting, the mechanical shutter 3 is once closed and then opened, and exposure is started according to this opening operation. When the image data is read every divided exposure time Tsec, the mechanical shutter 3 remains open. When the standard exposure time (Ssec) elapses, the mechanical shutter 3 is temporarily closed, and then the mechanical shutter 3 is used for live view display. Open 3

  If the result of determination in step S13 is not bulb shooting mode or if Ssec> Tsec, image readout is performed after the standard exposure time (Ssec) (S29). Here, since the standard exposure time (Ssec) is shorter than the divided exposure time (Tsec) and the high-luminance pixels do not saturate, exposure is performed with the determined standard exposure time, and when the exposure ends, the image sensor 4 Read image data from.

  On the other hand, as a result of the determination in step S13, if the bulb photographing mode is set or if Ssec> Tsec, it is determined whether or not the divided exposure time (Tsec) has elapsed (S15). The system control unit 20 includes a timer that measures the time from when the image sensor 4 starts exposure, and makes a determination based on the timing operation of this timer. If the result of this determination is that the divided exposure time (Tsec) has not elapsed, the system waits for this time to elapse.

  If the result of determination in step S15 is that the divided exposure time Tsec has elapsed, image reading is performed and shooting of the next frame is started (S17). The image sensor 4 has an electronic shutter, and when reading of image data is completed, the next frame can be photographed (accumulation of charges generated in the pixels is started). The read image data is temporarily stored in the internal memory 33. In addition, when shooting of the next frame is started, the timer for detecting Tsec is reset and the timing operation is restarted.

  In step S17, when image reading and shooting of the next frame are started, it is determined whether or not it is the first shot image after the start of shooting (S19). If the image read in step S17 is the first image, it is determined in this step because it is not a target for image composition.

  If the result of determination in step S19 is not the first captured image after the start of imaging, image composition is performed (S21). In the case of the second photographed image, the image composition unit 11 composes an image using the first and second image data, and stores the image data of the composite image in the internal memory 33. When the image data of the third photographed image is acquired, the composite image data stored in the internal memory 33 and the image data of the third photographed image are combined, and this image data is stored in the internal memory 33. Remember. Accordingly, when the image data of the N (an integer equal to or greater than 2) captured image is read, the image composition unit 11 combines the 1st to (N-1) th composite image data and the Nth captured image. A composite image is generated using the image data. Examples of image synthesis include addition synthesis, comparative bright synthesis or comparative dark synthesis, and addition average synthesis. The image synthesis may be set by the photographer or may be automatically set by the camera.

  If image synthesis is performed in step S21 or if the result of determination in step S19 is the first captured image after the start of imaging, image recording is performed (S23). Here, the combined image data combined by the image combining unit 11 or the image data of the first photographed image is stored in the internal memory 33. In this storage, if the bit accuracy of the digital value of one read image is, for example, 8 bits, the digital value of the pixel output of the image obtained by combining two images of 8 bits is not clipped with full bits. In addition, the bit length is recorded with a margin.

  Specifically, in the case of addition synthesis, if the number of addition synthesis is M (integer), the data capacity for storing the data of one pixel becomes 8 + log2 (M) bits or more, so that it becomes more than full bits. To prevent clipping (see FIG. 8B described later). In the case other than the bulb photographing mode, since the added composite number M is known at the time of photographing start, the data capacity (the number of bits of the composite image) can be determined in advance. On the other hand, in the bulb photographing mode, the added composite number M increases with the bulb photographing time at the start of photographing, so the data capacity (the number of bits of the composite image) may be increased with the photographing time.

  Once image recording has been performed in step S23, it is next determined whether or not shooting has ended (S25). In the case of bulb shooting, the photographer releases the release button (or starts shooting when the first release is pressed, and in the shooting mode where shooting ends when the second release is pressed, when the second release is pressed. ) Ends bulb shooting. In modes other than bulb shooting, shooting ends when the exposure time set by the photographer or the standard exposure time (Ssec) corresponding to the exposure time set automatically has elapsed. If the result of this determination is that photography has not been completed, processing returns to step S15, and image data is read out as described above every time the divided exposure time (Tsec) elapses, image synthesis is performed, and composite image data is recorded. I do.

  As a result of the determination in step S25, when shooting is completed, image reading and image composition are performed (S27). Here, the last image is read out from the image sensor 4 and combined with the previous combined image data. Here, in the case of bulb shooting, if the photographer releases the press of the release button before Tsec elapses, only the image reading is performed without performing the processing of steps S19 to S23 (the first reading is performed). Therefore, image composition is not performed).

  If image reading and image composition are performed in step S27, or if image reading is performed in step S29, image recording is then performed (S31). Here, the synthesized image data synthesized in step S27 or the image data read in step S29 is recorded in the internal memory 33.

  Once image recording has been performed in step S31, filter processing is next performed (S33). Here, filter processing is performed on the image stored in the internal memory 33. Since various filter effects such as a soft filter effect, a cross filter effect, and a color temperature conversion filter can be realized by image processing, the filter effect may be selected by the photographer.

For example, a case where the soft filter effect is selected will be described. In order to realize the soft filter effect in the image processing, for example, as shown in FIG. 5, for the output of each pixel, the coordinates (0, 0) of the corresponding pixel in the (x, y) two-dimensional coordinate system are used as filter coefficients. By applying a convolution operation to the image data with a filter such that the filter coefficient decreases toward the periphery, filter processing is performed so that the pixel output diffuses to the periphery. As a result, an image that is almost the same as that captured by the optical soft filter is generated. In FIG. 5, the filter coefficients are represented by 11 × 11 pixels and 10-bit Gaussian filter coefficients. Filter processing is performed by multiplying each pixel by the filter coefficient and dividing by 2 ¹⁰ .

  Various filters having different blurring effects are commercially available as optical filters, but in order to obtain the same effect as each filter, not only the filter pattern shown in FIG. The strength (blurring effect) of the filter can be changed by expanding or narrowing the range or increasing the number of times the filter is applied.

Further, not only the soft filter but also a cross filter, for example, a filter that diffuses the output in the cross direction as shown in FIG. 6 can be applied. By changing the filter pattern, it is possible to realize cross filter effects having different strengths in various directions other than the cross direction. In FIG. 6, the filter coefficients are represented by 11 × 11 pixels and 10-bit Gaussian filter coefficients, and the filter processing is performed by multiplying each pixel by the filter coefficient and dividing by 2 ¹⁰ .

  If filter processing is performed in step S33, saturation level clipping processing is then performed (S35). As described in step S23, in this embodiment, the number of bits of the composite image data stored in the internal memory 33 is set larger than the number of bits of the image data of the read image. Therefore, in this step, the clip processing unit 15 performs a saturation level clipping process so that the number of bits of the composite image data larger than the number of bits of the read image becomes the number of bits of the image data of the read image.

  The merit of not clipping with full bits when the composite image data is stored in the internal memory 33 will be described. The image output of the star image taken by the digital camera reaches the saturation level of the image sensor 4 and is clipped at the saturation output. The star image has a very steep output distribution, and it is considered that the star light has reached an output many times the saturation output of the image sensor. Here, for simplification, the output of the captured image of the star is shown and considered only in the X-axis direction.

  Assume that a star image having an output distribution as shown in FIG. In the conventional camera, the central portion of the star read from the image sensor reaches a saturated output. For example, in a system with a bit accuracy of 8 bits, it reaches a full bit and clips (see FIG. 7B). For example, if the output is filtered using a soft filter having a filter coefficient as shown in FIG. 7C, the output can be diffused and blurred (see FIG. 7D). ). However, since a filter coefficient is not applied to the output component above saturation shown in FIG. 7B, it does not diffuse much compared to an optical filter that diffuses light before the output sensor performs saturation clipping. In addition, if saturation clipping is performed, information on the color of the subject is lost, so that the output is below saturation after applying the filter as shown in FIG. 7D, but the star color cannot be reproduced.

  Therefore, in the present embodiment, as described above, the number of bits of the image data of the read image is stored with a larger number of bits, and the saturation level clip process is performed after the filter process.

  With reference to FIG. 8, a description will be given of processing when shooting is performed so that a stared image is not saturated and clipped. FIG. 8A shows the output of stars of an image taken with, for example, a divided exposure time Tsec. As described in S15 and S17 of FIG. 3, the star image is read at the divided exposure time Tsec so as not to reach the full bit. FIG. 8B shows the output of a star obtained by performing the divided exposure for the standard exposure time (Ssec) and combining a plurality of image data acquired by the divided exposure. In the example shown in FIG. 8B, what is saturated when the conventional bit length is 8 bit processing is recorded so as not to be clipped, for example, by extending the bit length to 12 bits.

  A filter similar to that shown in FIG. 7C is applied to the composite image data shown in FIG. 8B (see FIG. 8C). After this filtering process, the clip processing unit 15 clips the saturation level with the original 8 bits (FIG. 8D). By processing in this way, similarly to the optical filter, it is possible to perform the filtering process without saturation even with respect to the output of a star image that has conventionally been saturated and clipped. Since it is larger than the conventional filtered image (FIG. 7 (d)), the star image is blurred and the image data of the composite image does not reach saturation (see FIG. 8 (b)). It can be accurately reproduced as when using an optical filter. Note that the clip processing unit 15 can also be referred to as a bit length adjustment unit that adjusts the bit length of the image data subjected to the filter processing so as to be smaller than the bit length of the composite image data.

  If saturation level clipping processing is performed in step S35, then development processing is performed (S37). Here, the development processing unit 17 performs development processing such as demosaicing, white balance adjustment, gamma correction, and image compression on the image data on which the saturation level clipping processing has been performed in step S35.

  If development processing is performed in step S37, recording is performed in an external memory (S39). Here, the image data developed in step S37 is recorded in the external memory 36. When image data is recorded in the external memory 36, this flow ends.

  As described above, in the flow when the digital filter processing in the present embodiment is selected, the subject is photographed with a divided exposure time (Tsec) shorter than the standard exposure time (Ssec) (S15, S17 in FIG. 2, FIG. 8 (a)), and composite image data is generated by synthesizing the image data of a plurality of frames obtained by this shooting (see S21 in FIG. 3, FIG. 8 (b)), and the composite image data is subjected to a filtering process. (See S33 in FIG. 4 and FIG. 8C), and the bit length of the image data subjected to the filter processing is adjusted to be shorter than the bit length of the composite image data (FIG. 4). S35, see FIG. 8 (d)). For this reason, a captured image that is saturated with normal imaging can be obtained without saturation clipping, and image processing with a filter effect equivalent to that of an optical filter can be performed.

  In one embodiment of the present invention, as described with reference to FIG. 8D, the saturation level is clipped by the number of bits of one original read image (in this example, 8 bits). However, for example, the gradation may be compressed so that the pixel with the highest luminance of the image after the filter processing becomes a full bit. In this case, tone compression processing is performed by the development processing unit 17. By performing this processing, it is possible to perform image processing without saturating a bright celestial image having a maximum luminance portion of 8 bits or more even after the filtering processing as shown in FIG. Color information can be recorded. In this case, the processing unit (development processing unit 17) that compresses the gradation adjusts the bit length so that the bit length of the image data subjected to the filter processing is smaller than the bit length of the composite image data. It functions as a part.

  Further, in FIG. 8D, the method of clipping to the saturation level with the original bit length of one read image (in this example, 8 bits) has been described. However, for example, the gradation may be compressed and developed so that the photographer can set the desired output level to full bits. Depending on the photographer, there is a preference for how much gradation the photographer wants to take, so this method can take photographs that meet the tastes of various photographers. In this case, the RAW image data is preferably stored with the bit accuracy extended so that all pixel outputs do not undergo saturation clipping. RAW image data after shooting is subjected to filter processing and developed, so that it is possible to develop with the same filter processing as the optical filter, and the effect of the filter can be strengthened or weakened. It becomes possible to develop an image according to the photographer's preference. In this case, the development processing unit 17 compresses the gradation of the image data subjected to the filter processing so that the bit length of the image data subjected to the filter processing is shorter than the bit length of the composite image data. Functions as an adjustment unit.

  Although not shown in the flowcharts shown in FIGS. 2 to 4, the read image data is synthesized every time the divided exposure time elapses, the image data is developed, and the developed image data is processed. An image may be displayed on the display unit 37 based on the above. That is, the synthesized image obtained by sequentially synthesizing each time image data is captured from the imaging unit 1 may be displayed on the display unit 37 sequentially. As a result, the photographer can confirm the composite image as a shooting progress image, and can determine the timing for ending the shooting in the bulb shooting mode or the like.

  Further, in the storage in the external memory in step S39, the raw image data (RAW image data) may be stored without performing compression processing such as JPEG conversion in the development processing in step S37. Some photographers may re-develop the RAW image data by changing the development processing settings after photographing, and this demand can be met.

  In the embodiment of the present invention, the image output detection unit 19 has been described with respect to the method of shooting by setting the divided exposure time Tsec at which the image data is not saturated. However, the present invention is not limited to this, and the photographer may set the divided exposure time Tsec. In this case, the celestial image may be saturated and clipped at the set divided exposure time Tsec. However, the divided exposure is more effective than the case where the filter processing is performed on one piece of image data read without performing the divided exposure. It is preferable that the combined image data is recorded so that the pixel data that is greater than or equal to the full bit by the combination is recorded so as not to be saturated and clipped, and the filtering process is performed, so that the filtering process close to the optical filter can be performed.

  In the embodiment of the present invention, the sequence for performing the filtering process on the combined image data subjected to the image combining has been described. However, the present invention is not limited to this, and image processing may be performed on a plurality of image data that has been subjected to filter processing. That is, the process in step S33 in FIG. 4 may be performed before or after step S19 in FIG. In this way, the number of times of processing is increased as compared with the case where the processing once applied to the composite image data is performed, but an equivalent filter effect can be applied.

  Moreover, although the soft filter and the cross filter have been described for the filter process, the present invention is not limited to this, and can be applied to all filter processes that can be realized by other image processes.

  In the embodiment of the present invention, the case has been described in which additive synthesis is used as image synthesis. However, the same effect can be obtained by not clipping at full bits or more even in other synthesis. For example, even in the case of addition-average synthesis or comparatively bright synthesis, there is a case where one or both of the image data to be synthesized is multiplied by a gain and synthesized, and it may be possible to reach an output of full bits or more. It is.

  Further, it may be determined by providing a predetermined threshold level for the output of the composite image data, and the processing may be changed depending on the threshold level or higher. That is, an image output determination unit that determines the level of the combined image data combined by the image combining unit 11 may be further provided, and the filter processing may be changed according to the level of the combined image data. For example, before performing filter processing, for each pixel output of the composite image, the image output determination unit determines whether the output of the pixel is equal to or higher than the threshold level (TH). When a soft filter is applied and the level is lower than the threshold level, a process not applying the soft filter is performed. FIG. 9 shows changes in the image due to the star image filtering when the threshold level TH is 255 LSB (8-bit full bit). In addition, when performing a filter process with respect to the image data before producing | generating synthetic image data, a filter process is changed according to the level of the image data before producing | generating synthetic image data.

  Dark and bright stars are similarly blurred by conventional filters (FIGS. 9B and 9E). However, if the filtering process is made different depending on whether it is above or below the threshold level, the blurring effect can be made different depending on the brightness of the star. For example, if dark stars are not filtered (FIG. 9 (c)), but only bright stars are filtered (FIG. 9 (e)), the dark stars are diffused and darkened by the filtering, It is possible to prevent the image from being clearly visible. Dark stars are clear and clear, bright stars appear large, and color information remains. This makes it possible to take impressive photographs that cannot be taken with conventional optical filters.

  FIG. 9 shows processing for changing whether or not to perform filter processing with one threshold level. For example, a filter that provides a plurality of threshold levels and increases the diffusion effect of the filter as the pixel level increases. By performing the processing, it is possible to perform a filtering process such that a brighter star appears more naturally. On the contrary, by processing the darker star so that the diffusion effect is stronger, and the brighter the light is weaker and weakening the diffusion effect, the darker star becomes harder to see, and the brighter star remains brighter. In this way, photographs with various impressions can be taken.

  As for the cross filter, if the filter effect is strengthened for pixels with a larger output, the bright light caused by the cross filter can be strongly generated only in bright stars to be conspicuous, and a sharp and impressive image can be generated. .

  In addition, not only stars but also ground scenes can be taken as subjects for astronomical photography. In that case, the brightness and shape of the subject, the instruction from the photographer's input IF unit 38, etc. identify which part of the image is the ground scenery and which part is empty, and the image is applied only to the empty part. The output may be detected, and divided exposure and synthesis may be performed so as not to saturate. If a scene brighter than the celestial body is reflected in the ground scene, the divided exposure time Tsec becomes too short and the number of images to be combined increases. In addition synthesis, the noise included in the image data is added and the image quality deteriorates as the synthesis is performed. In this case, the imaging unit 1 functions as an imaging unit that divides an image into a plurality of areas and sets and shoots a divided exposure time that does not saturate the average value of image data in the area corresponding to the divided divided image. To do.

  Further, the filtering process may be performed only on empty portions. If an object brighter than the celestial object is reflected in the scenery on the ground, the filter process will cause the image to be more blurred than the celestial object. Therefore, such a process is preferable when the main object is a celestial object and only the celestial object is to be greatly blurred.

  In the embodiment of the present invention, the configuration has been described in which the digital camera sequentially synthesizes the image data read from the image sensor 4 and performs the filter processing. However, the present invention is not limited to this, and for example, a plurality of digital image data drawn by CG (Computer Graphics) may be synthesized, for example, an image that has already been shot by a camera. Processing functions when performing image composition processing inside a digital camera or on a personal computer etc. For the bit accuracy of the pixel data of each image, composition and recording are performed so as not to clip with full bits after composition, and the composition An effect equivalent to that of the present embodiment can be obtained by filtering the image. In this case, it is convenient to select and combine only the images that the photographer wants to synthesize from among a plurality of images taken after shooting. The same effect can be obtained in addition to images taken with a camera.

  In the embodiment of the present invention, the sequence for recording one image to which the filter process is applied is shown. However, the present invention is not limited to this. For example, a plurality of composite images photographed by the addition synthesis / filter processing of the present embodiment are generated, and comparison synthesis, addition average synthesis, video generation, and the like are further performed on the plurality of composite images May be. More impressive and beautiful photographs and moving images can be taken by synthesizing image data that have been subjected to filter processing equivalent to or not realized by the optical filter by the method of the present embodiment.

  In the embodiment of the present invention, the image processing unit 10, the image output detection unit 19, and the like are configured separately from the system control unit 20, but all or a part of each unit is configured by software. Of course, the program may be executed in the system control unit 20 or by a microcomputer separate from the system control unit 20.

  In the present embodiment, the digital camera is used as the photographing device. However, the camera may be a digital single-lens reflex camera or a compact digital camera, and may be used for moving images such as video cameras and movie cameras. It may be a camera, and may be a camera built in a mobile phone, a smart phone, a personal digital assistant (PDA), a personal computer (PC), a tablet computer, a game machine, or the like. In any case, the present invention can be applied to any imaging device that can perform divided exposure, or an image processing device that can process image data captured by these imaging devices.

  Of the techniques described in this specification, the control mainly described in the flowchart is often settable by a program and may be stored in a recording medium or a recording unit. The recording method for the recording medium and the recording unit may be recorded at the time of product shipment, may be a distributed recording medium, or may be downloaded via the Internet.

  In addition, regarding the operation flow in the claims, the specification, and the drawings, even if it is described using words expressing the order such as “first”, “next”, etc. It does not mean that it is essential to implement in this order.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, you may delete some components of all the components shown by embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 1 ... Imaging part, 2 ... Lens (aperture), 3 ... Mechanical shutter, 4 ... Image sensor, 10 ... Image processing part, 11 ... Image composition part, 11a ... Addition Synthesis unit, 11b ... Comparison / synthesis unit, 11c ... Addition / average synthesis unit, 11d ... Movie generation unit, 13 ... Filter processing unit, 15 ... Clip processing unit, 17 ... Development processing , 20 ... System control unit, 31 ... Bus, 33 ... Internal memory, 36 ... External memory, 37 ... Display unit, 38 ... Input IF

Claims (12)

An imaging unit that captures multiple frames of an object with an exposure time shorter than the standard exposure time;
An image composition unit that synthesizes image data of a plurality of frames obtained by photographing by the imaging unit to generate composite image data;
A filter processing unit that performs a filtering process on the image data of the plurality of frames or the synthesized image data synthesized by the image synthesizing unit;
The composite image data obtained by synthesizing the image data of the plurality of frames subjected to the filtering process or the bit length of the image data subjected to the filtering process on the synthesized image data synthesized by the image synthesizing unit A bit length adjusting unit that adjusts the bit length to be shorter than the data bit length;
An imaging apparatus comprising:   The imaging apparatus according to claim 1, wherein the imaging unit sets an exposure time so that a maximum value of pixel data is not saturated.   The image capturing unit divides an image into a plurality of regions, and sets the exposure time so that an average value of image data of regions corresponding to all the divided images is not saturated. The imaging apparatus according to 1.   The bit length adjustment unit is configured to combine composite image data obtained by synthesizing the image data of the plurality of frames subjected to the filter processing or image data obtained by performing filter processing on the composite image data synthesized by the image synthesis unit. 2. The imaging apparatus according to claim 1, wherein the image data subjected to the filtering process is clipped so that a bit length is shorter than a bit length of the composite image data.   The bit length adjustment unit is configured to combine composite image data obtained by synthesizing the image data of the plurality of frames subjected to the filter processing or image data obtained by performing filter processing on the composite image data synthesized by the image synthesis unit. The imaging apparatus according to claim 1, wherein the gradation of the image data subjected to the filtering process is compressed so that a bit length is shorter than a bit length of the composite image data. An image output determination unit for determining the level of the image data before generating the combined image data or the combined image data combined by the image combining unit;
When the filtering process is performed on the image data before generating the synthesized image data, the filtering process is performed on the synthesized image data according to the level of the image data before generating the synthesized image data. In this case, the filter processing is changed according to the level of the composite image data.
The imaging apparatus according to claim 1.   The imaging apparatus according to claim 1, wherein the image composition unit performs at least one kind of image composition among addition composition, average composition, comparatively bright composition, comparative dark composition, and moving image composition. An image display unit that displays the combined image in the image combining unit;
The image display unit sequentially displays a composite image obtained by sequentially combining each time image data is captured by the imaging unit.
The imaging apparatus according to claim 1.   The imaging measure according to claim 1, wherein the filtering process is a soft filtering process or a cross filtering process. An image composition unit that synthesizes image data of a plurality of frames obtained by photographing a subject with an exposure time shorter than the standard exposure time, and generates composite image data;
A filter processing unit that performs a filtering process on the image data before generating the combined image data or the combined image data combined by the image combining unit;
The composite image data obtained by synthesizing the image data of the plurality of frames subjected to the filtering process or the bit length of the image data subjected to the filtering process on the synthesized image data synthesized by the image synthesizing unit A bit length adjusting unit that adjusts the bit length to be shorter than the data bit length;
An image processing apparatus comprising: An image compositing step for compositing a plurality of frames of image data obtained by photographing a subject with an exposure time shorter than the standard exposure time to generate composite image data;
A filter processing step for performing filter processing on the image data before generating the combined image data or the combined image data combined in the image combining step;
The composite image data obtained by synthesizing the image data of the plurality of frames subjected to the filtering process or the bit length of the image data subjected to the filtering process for the synthesized image data synthesized in the image synthesizing step is used as the synthesized image. A bit length adjustment step for adjusting to be shorter than the bit length of the data;
An image processing method comprising: An image compositing step for generating composite image data by compositing a plurality of frames of image data obtained by photographing a subject with a divided exposure time rather than a standard exposure time;
A filter processing step for performing filter processing on the image data before generating the combined image data or the combined image data combined in the image combining step;
The composite image data obtained by synthesizing the image data of the plurality of frames subjected to the filtering process or the bit length of the image data subjected to the filtering process for the synthesized image data synthesized in the image synthesizing step is used as the synthesized image. A bit length adjustment step for adjusting to be shorter than the bit length of the data;
An image processing program for causing a computer to execute.

Images