JP5463782B2 - Electronic camera - Google Patents

Description

  The present invention relates to an electronic camera capable of photographing at a plurality of resolutions.

  In a conventional electronic camera, when a moving image shot at a predetermined interval is played back slowly, there is a problem that the motion becomes unnatural because there is no image between frames. In view of this, a technique has been considered in which images between frames are generated by interpolating temporally surrounding frames (see, for example, Patent Document 1).

JP 2008-141772 A

  However, in the case of the prior art that generates an interpolated image from the images of previous and subsequent frames, there is a problem that an unnatural interpolated image can be obtained because it cannot cope with a subject that suddenly appears between frames or a scene that changes rapidly. .

  An object of the present invention is to provide an electronic camera that can acquire a preliminary image having a different resolution within an idle time from the end of capturing the previous frame image to the start of capturing the next frame image.

The electronic camera according to the present invention includes an imaging unit that captures a subject image at a plurality of designated resolutions, a storage unit that stores an image captured by the imaging unit, and a first preset among the plurality of resolutions. The original image acquisition means for specifying the resolution to the imaging means and acquiring the original image of the first resolution from the imaging means to the storage means, and the time from the end of the acquisition of the first original image acquired by the original image acquisition means A resolution determining unit that determines a resolution of a preliminary image different from the first resolution that can be captured from the imaging unit within an idle time until the second original image capturing start time, and a resolution determined by the resolution determining unit the specify the imaging means that have a and the preliminary image acquisition means for capturing said preliminary image in the storage unit.

The resolution determining unit determines a resolution of a plurality of preliminary images different from the first resolution within the idle time, and the preliminary image acquiring unit applies the resolution determined by the resolution determining unit to the imaging unit. The plurality of preliminary images are specified and fetched into the storage means.

  Preferably, a similarity determination unit for obtaining a similarity between a plurality of images captured by the imaging unit before acquisition of the preliminary image is further provided, and the resolution determination unit is configured to resolve the resolution of the preliminary image captured from the imaging unit. When the similarity determined by the similarity determination unit is low, the resolution of the preliminary image is increased. When the similarity determined by the similarity determination unit is high, the resolution of the preliminary image is decreased. It is characterized by doing.

  Preferably, the similarity is at least one of a shape change, brightness, hue, contour component, frequency component, and motion vector of the main subject.

Preferably, in addition to the image pickup means, a second image pickup means for picking up an image having the same angle of view as the image pickup means is further provided, and the similarity determination means is a plurality of images picked up by the second image pickup means. And determining the resolution of the preliminary image captured from the imaging means.
The electronic camera according to the present invention includes an imaging unit that captures a subject image at a plurality of designated resolutions, a storage unit that stores an image captured by the imaging unit, and a preset one of the plurality of resolutions. An original image acquisition unit that designates the first resolution as the image pickup unit and takes the original image of the first resolution from the image pickup unit into the storage unit, and a time point when the acquisition of the first original image that the original image acquisition unit takes in The resolution of the preliminary image that is different from the first resolution that can be captured from the imaging means within an idle time from the start of capturing the second original image to the next is determined based on the image captured before the preliminary image A resolution determining unit that performs the resolution determination, and a preliminary image acquisition unit that designates the resolution determined by the resolution determination unit to the imaging unit and captures the preliminary image into the storage unit. And butterflies.

  The electronic camera according to the present invention can acquire a preliminary image having a different resolution within an idle time from the end of capturing the previous frame image to the start of capturing the next frame image.

It is a block diagram which shows the structure of the electronic camera 101 which concerns on this embodiment. It is explanatory drawing which shows the mode of multi-resolution. It is explanatory drawing which shows the relationship between resolution and data amount. It is explanatory drawing which shows the example of the resolution of the preliminary image for slow reproduction. It is explanatory drawing which shows the example which image | photographs several preliminary images. It is explanatory drawing which shows the example of the resolution of the preliminary image at the time of still image photography during video photography. It is explanatory drawing which shows the example of the resolution of the preliminary image at the time of still image photography. It is explanatory drawing which shows the relationship between the similarity and the resolution of a preliminary image. It is explanatory drawing which shows the example which determines the resolution of a preliminary image according to a similarity. It is a block diagram which shows the structure of the electronic camera 101a of the modification of this embodiment. It is explanatory drawing which shows the operation example of the electronic camera 101a.

  Hereinafter, embodiments of the electronic camera 101 according to the present invention will be described. FIG. 1 is a block diagram illustrating a configuration of an electronic camera 101 according to the present embodiment. In FIG. 1, the electronic camera 101 basically includes an imaging unit 102, an image buffer 103, an image processing unit 104, a storage unit 105, a control unit 106, and an operation unit 107. In FIG. 1, the units are connected to each other via a system bus 108 and can input / output control signals, image data, and the like. In addition, the display unit 109 may be provided so that the electronic camera 101 in FIG. Alternatively, the storage unit 105 may be configured by a memory card, and a captured image may be read out and reproduced by a personal computer or the like.

  In FIG. 1, the imaging unit 102 includes an optical system 201, a CCD or CMOS solid-state imaging device 202, an AFE (analog front end) 203, and a TG (timing generator) 204.

  Here, the solid-state image sensor 202 is an image sensor having a function of outputting multi-resolution data that can read a captured image at a plurality of designated resolutions. The control unit 106 designates the resolution, and the control unit 106 instructs the TG 204 of the imaging unit 102 to specify the resolution of the image read from the solid-state imaging device 202. The TG 204 outputs a row readout timing signal and a column readout timing signal to the solid-state imaging device 202 and the AFE 203 so as to obtain an image with the resolution instructed from the control unit 106, and takes in the image data with the instructed resolution into the image buffer 103. .

  If the solid-state image sensor 202 is a CMOS solid-state image sensor, the row and column positions to be read out of the pixels arranged in a two-dimensional matrix can be specified by the addressing function, so that an image with a desired resolution can be obtained directly. However, if the solid-state image sensor 202 is a CCD solid-state image sensor, the column position of the pixel to be read out cannot be specified, so it is not necessary to specify the row position and read out the desired resolution. It is necessary to delete the image data of the correct column. In the description of this embodiment, for the sake of easy understanding, the solid-state image sensor 202 is assumed to be a CMOS solid-state image sensor. In FIG. 2, when the resolution of all pixel readout is 1, it is possible to read at 1/4 resolution such as 1/4, 1/16, 1/64, 1/256, 1/1024. However, it may be read at other resolutions such as 1/8 or 1/10. Further, it is not necessary to use the resolution of all pixel readout as a reference, and for example, an image having a resolution of 1/4 may be used as a reference.

  In FIG. 1, the control unit 106 includes an original image acquisition processing unit 151, a resolution determination processing unit 152, a preliminary image acquisition processing unit 153, and a similarity determination processing unit 154. The original image acquisition processing unit 151 performs still image shooting and moving image shooting at a predetermined frame rate, as in a normal electronic camera. Here, an image captured as a still image and an image captured as a moving image at a predetermined frame interval are defined as original images, and an image captured other than the original image is defined as a preliminary image. The resolution determination processing unit 152 performs processing for determining the resolution of the preliminary image. The preliminary image acquisition processing unit 153 performs processing for capturing a preliminary image at the resolution determined by the resolution determination processing unit 152 between the capturing of the original image. The similarity determination processing unit 154 performs processing for determining the similarity between the preliminary image and the original image. The processing of each unit will be described in detail later.

  In FIG. 1, light from a subject is imaged on a light receiving surface of a solid-state image sensor 202 by an optical system 201. A photoelectric conversion unit is two-dimensionally arranged on the light receiving surface of the solid-state imaging device 202. Each photoelectric conversion unit converts the photoelectric conversion unit into an analog signal having a magnitude corresponding to the amount of received light and outputs the analog signal to the AFE 203. The AFE 203 performs noise removal and amplification of the analog signal output from the solid-state imaging device 202, A / D converts it into digital image data, and temporarily stores it in the image buffer 103. The image data temporarily stored in the image buffer 103 is subjected to predetermined image processing (for example, color interpolation processing, white balance processing, contour processing, gamma conversion processing) by the image processing unit 104. Note that the image processing unit 104 performs image compression processing conforming to, for example, the JPEG standard in the case of still image shooting, and performs image compression processing in accordance with, for example, the MPEG standard in the case of moving image shooting. The image data after the image compression processing is stored in the storage unit 105 as a captured image. Here, the control unit 106 operates according to a program stored therein in advance, and controls the entire operation of the electronic camera 101 according to the operation content of the operation unit 107. The operation unit 107 includes operation members such as a power button, a release button, a moving image shooting button, a mode switching dial, and the user uses the electronic camera 101 by operating these operation members. For example, the power button is used to turn on / off the electronic camera 101, the release button is used to take a still image, and the movie shooting button is used to start / end movie shooting. The shooting mode and the playback mode can be switched with the mode switching dial. In shooting mode, still image single shooting mode (mode to shoot a single still image), still image continuous shooting mode, movie shooting mode, movie / still image shooting mode (mode that can shoot still images during movie shooting), etc. Can be selected. Further, in the playback mode, it is possible to select display of still images that have been shot or playback of moving images, and in particular, in playback of moving images, special playback modes such as slow playback, frame advance playback, and fast forward playback can be selected.

  Here, in the description of the present embodiment, a moving image shot at a predetermined frame rate and a still image shot when the release button is pressed are defined as an original image (or a main image) as in a normal electronic camera, and an original image is defined. Apart from that, an auxiliary image is defined as a preliminary image. The electronic camera 101 according to the present embodiment has a function of capturing not only an original image but also a preliminary image. For example, the preliminary image at the time of moving image shooting is used as an interpolated image between frames of the original image when the moving image is slowly reproduced. In addition, the preliminary image at the time of still image shooting is used for, for example, processing for improving the image quality of a shot still image or increasing the resolution at the time of printing. Note that the preliminary image may be an image having the same resolution as the original image. However, in this embodiment, when a plurality of preliminary images are captured, the resolution of each preliminary image is different. It doesn't matter. In particular, the electronic camera 101 according to the present embodiment is configured to vary the resolution of the preliminary image in accordance with the similarity (difference in feature amount) between images captured before the preliminary image.

  Here, in the example of FIG. 2, the resolution of the preliminary image is 1/4, 1/16, 1/64, 1/256, 1/1024, where the resolution of the original image of the solid-state imaging device 202 is 1. Corresponds to a resolution of 1/4. Note that the resolution of the original image does not necessarily need to be a resolution corresponding to all-pixel reading of the solid-state imaging device 202, and for example, moving image shooting may be performed with a resolution of 1/4 of all the pixels of the solid-state imaging device 202. .

  Next, the relationship between the resolution of the original image and the preliminary image and the amount of data (or data acquisition time) taken into the image buffer 103 at the time of shooting will be described with reference to FIG. FIG. 3 is a diagram showing a period (time) between two original images that move back and forth in time, which corresponds to the frame interval in the moving image shooting mode, and in the still image continuous shooting mode. Corresponds to the interval. Even in the still image single shooting mode, it is possible to correspond to the time until the next shooting is possible.

  In FIG. 3, the horizontal axis indicates time t, the acquisition of the first original image is started at timing T1, and the acquisition of the next original image is started at timing T3. In the video shooting mode, a period from timing T1 to timing T3 (T_cap) is a frame acquisition period. For example, when the frame rate is 30 frames / second, the period from timing T1 to timing T3 is 1/30 seconds ( sec). However, in actual processing, the time for acquiring one original image is shorter than the period T_cap, and the substantial original image acquisition time is a period (T_base) from timing T1 to timing T2, from timing T2 to timing T3. The period is an idle period (T_idle) in which no image acquisition process is performed. For example, when it is assumed that the frame acquisition possible period T_cap = 1.25 sec and the original image acquisition time T_base = 1 sec, the idle period T_idle = 0.25 sec. That is, the idle period T_idle (0.25 sec) corresponds to a time during which an image having a resolution of 1/4 of the original image can be taken and captured in the image buffer 103. Accordingly, while an original image having a resolution of 1 is captured, an image having a resolution of 1/4 (preliminary image) can be captured.

  Depending on the circuit configuration of the electronic camera 101, there is an electronic camera that must perform not only image capture but also other processing at the time of shooting. Therefore, as described above, the entire idle period T_idle can be simply captured. However, it is possible to capture an image with a resolution smaller than ¼ resolution. In the description of the present embodiment, for the sake of easy understanding, it is assumed that the relationship of frame acquisition possible period T_cap = original image acquisition period T_base + idle period T_idle is satisfied. In addition, since there may be a case where a common processing time necessary for image acquisition that does not depend on the resolution of the image to be actually taken is included, if the resolution is 1 / k (k is a natural number), the image acquisition time is 1 / k. Although not necessarily k, in the description of this embodiment, the description will be made assuming that resolution ratio = image acquisition time ratio.

  For example, in FIG. 3, when the original image acquisition time T_base = 1 sec and the idle period T_idle = 0.25 sec, one preliminary image having a resolution of 1/4 of the original image can be taken in the idle period T_idle. In addition, if the preliminary image has a resolution of 1/16 of the original image, four images can be captured, and if the preliminary image has a resolution of 1/64 of the original image, 16 images can be captured. Similarly, 64 preliminary images can be captured if the preliminary image has a resolution of 1/256 of the original image, and 256 preliminary images if the preliminary image has a resolution of 1/1024 of the original image.

  Here, the number of preliminary images to be captured during the idle period T_idle may be such that only one preliminary image with the maximum resolution that can be captured within this period may be captured, or a plurality of preliminary images with small resolutions may be combined. You can shoot. In this case, the resolutions of the plurality of preliminary images may be the same resolution or different resolutions. For example, in the example of FIG. 3, when the idle period T_idle = 0.25 sec, a preliminary image having a resolution of 1/16 of the original image is not captured, but a single preliminary image having a resolution of 1/4 of the original image is captured. 2 images, 4 preliminary images having a resolution of 1/64 of the original image, and 16 preliminary images having a resolution of 1/256 of the original image may be taken in combination.

As described above, the electronic camera 101 according to the present embodiment can capture at least one preliminary image during the idle period T_idle while capturing the original image and take it into the image buffer 103 in accordance with a command from the control unit 106. It is like that.
[Example of shooting a preliminary image for slow playback]
Next, an example in which a preliminary image for slow playback is captured during moving image shooting will be described with reference to FIG. In this embodiment, when a captured moving image is slowly played back, a substantial frame rate is increased by using a preliminary image captured between frames of the original image as an interpolation image. Compared to the method used, smooth and natural slow reproduction can be performed.

  FIG. 4 shows a processing example of the control unit 106, and a preliminary image captured by the preliminary image acquisition processing unit 153 between the original images of a moving image captured by the original image acquisition processing unit 151 at a predetermined frame rate (predetermined frame interval). The resolution determination processing unit 152 determines the resolution of. In FIGS. 4A and 4B, the vertical axis indicates the resolution of the original image and the preliminary image, and the horizontal axis indicates the temporal flow of the frame number. Here, FIG. 4A shows a state in which a moving image original image from the frame (n−3) to the frame (n + 3) is captured at a predetermined frame interval with a resolution of 1/1. FIG. 4B is a diagram illustrating an example of the position and resolution of a preliminary image captured between the frame (n−1) and the frame (n) in FIG.

  In the example of FIG. 4B, the resolution determination processing unit 152 of the control unit 106 determines a preliminary image at a position close in time to the original image of the frame (n−1) or the frame (n) to a low resolution. The preliminary image at a position far in time from the original image of frame (n-1) or frame (n) is determined to have a high resolution. Then, the preliminary image acquisition processing unit 153 of the control unit 106 captures each preliminary image with the determined resolution. Here, Δf indicating the shooting interval of each preliminary image is not necessarily equal, but in FIG. 4B, the time interval when the frame interval is 1/6 is drawn as Δf for easy understanding. For example, the preliminary image 301 is a preliminary image (a frame number is expressed as (n−Δf)) taken at a position before the time Δf from the original image of the frame (n). The preliminary image 302 is a preliminary image (a frame number is expressed as (n−2 · Δf)) taken at a position before the time (2 · Δf) from the original image of the frame (n). Here, 2 · Δf indicates that the time is twice the time Δf. Similarly, the preliminary image 303 is a preliminary image (frame number (n−3 · Δf)) taken at a position before the time (3 · Δf) from the original image of the frame (n). n) is a preliminary image (frame number (n−4 · Δf)) taken at a position before time (4 · Δf) from the original image of n), and the preliminary image 305 is time (5) from the original image of frame (n). Δf) A preliminary image (frame number (n−5 · Δf)) taken at the previous position.

  Here, since the preliminary image 303 is located between the frame (n) and the frame (n−1), the resolution determination processing unit 152 is located farther from both frames than the other preliminary images. Is set as high as possible (1/4 resolution of the original image). Further, since the preliminary image 301 and the preliminary image 305 are close to the frame (n) or the frame (n−1), respectively, the resolution is determined to be low (1/64 resolution of the original image). Further, since the preliminary image 302 and the preliminary image 304 are positioned between the preliminary image 301 (or the preliminary image 305) and the preliminary image 303, respectively, the resolution of the preliminary image 301 (or the preliminary image 305) and the resolution of the preliminary image 303 are determined. An intermediate resolution (1/16 the resolution of the original image) is determined. Then, the preliminary image acquisition processing unit 153 of the control unit 106 captures each preliminary image with the determined resolution.

  FIG. 4B is a diagram for explaining an example in which the resolution determination processing unit 152 determines the resolution of the preliminary image. Actually, at least one preliminary image that is captured between frames for moving image capturing is illustrated. The total amount of time required for capturing the image must not exceed the idle period T_idle described with reference to FIG. For example, FIG. 4B actually captures in the idle period T_idle after the time T_base until the timing T2 when the frame (n−1) of the original image is captured in the image buffer 103, as shown in FIG. Possible preliminary images (for example, two preliminary images 301 and 302 in FIG. 4B) are taken. Alternatively, when only the preliminary image 302 is captured during the idle period T_idle, only the preliminary image 302 may be captured as shown in FIG. The preliminary image 301 may be taken.

As described above, in the moving image shooting mode, when a plurality of preliminary images can be captured within the idle period T_idle, the resolution of the preliminary image located near the original image of the preceding and following frames is set to the preliminary image located far from the original image. By making the resolution lower than the resolution of, as many preliminary images as possible can be taken within a limited idle period T_idle. This is because a preliminary image at a position close to the original image is more likely to be similar to the original image than a preliminary image at a far position, so it can be used as an interpolated image of the original image even if it is taken at a low resolution. Because it can. As a result, when a captured moving image is played back slowly, a high-quality interpolation image can be generated, and smooth and natural slow playback can be performed. The similarity will be described in detail later.
[Example of taking a still image while shooting a movie]
Next, an example of shooting a preliminary image for improving the image quality of a shot still image in a shooting mode for shooting a still image by pressing the release button of the operation unit 107 during moving image shooting will be described with reference to FIG. To do.

  FIG. 6A is a diagram corresponding to FIG. 4A described above, and the original image acquisition processing unit 151 of the control unit 106 is an original image of a moving image from frame (n−3) to frame (n + 3). A state in which (resolution 1/1) is photographed at a predetermined frame interval is shown. 6B shows that when the release button of the operation unit 107 is pressed at the timing of the frame (n) in FIG. 6A, the original image acquisition processing unit 151 of the control unit 106 selects the frame (n). A state in the case of shooting as a still image (original image) is shown. At this time, the preliminary image acquisition processing unit 153 of the control unit 106 is a diagram illustrating an example of the position and resolution of the preliminary image captured before and after the frame (n) in order to improve the quality of the original image of the frame (n). . As in FIG. 4, in FIGS. 6A and 6B, the vertical axis indicates the resolution of the original image and the preliminary image, and the horizontal axis indicates the temporal flow of the frame number.

  In FIG. 6B, the resolution determination processing unit 152 of the control unit 106 determines a preliminary image at a position close in time to the original image of the frame (n) to a low resolution, and converts it to the original image of the frame (n). A preliminary image located far in time is determined to have a high resolution. Then, the preliminary image acquisition processing unit 153 of the control unit 106 captures each preliminary image with the determined resolution. Here, Δf indicates the time interval when the frame interval is 1/6 as in FIG. 4B, but Δf does not necessarily have to be equal. For example, the preliminary image 401 is a preliminary image (frame number is expressed as (n + Δf)) taken at a position after a time Δf from the original image of the frame (n). The preliminary image 402 is a preliminary image (frame number is expressed as (n + 2 · Δf)) taken at a position after the time (2 · Δf) from the original image of the frame (n). Here, 2 · Δf indicates that the time is twice the time Δf, as in FIG. 4B. Hereinafter, similarly, the preliminary image 403 is a preliminary image (frame number (n + 3 · Δf)) taken at a position after the time (3 · Δf) from the original image of the frame (n). n) is a preliminary image (frame number (n−Δf)) taken at a position before time (Δf) from the original image of n), and the preliminary image 405 is before time (2 · Δf) from the original image of frame (n). Is a preliminary image (frame number (n−2 · Δf)), and the preliminary image 406 is a preliminary image (at a time (3 · Δf) before the frame (n). Frame number (n−3 · Δf)).

  Here, the resolution determination processing unit 152 of the control unit 106 determines the low resolution (the resolution of 1/64 of the original image) because the preliminary image 401 and the preliminary image 404 are closest to the frame (n). Since the preliminary image 403 and the preliminary image 406 are located farthest from the frame (n), they are determined to have a high resolution (1/4 resolution of the original image). Further, since the preliminary image 402 and the preliminary image 405 are positioned between the preliminary image 401 and the preliminary image 404 and the preliminary image 403 and the preliminary image 406, the intermediate resolution of these resolutions (1/16 the resolution of the original image). To decide. Then, the preliminary image acquisition processing unit 153 of the control unit 106 captures each preliminary image with the determined resolution.

  FIG. 6B is a diagram for explaining an example in which the resolution determination processing unit 152 determines the resolution of a preliminary image that is captured to increase the resolution of a still image that is performed during moving image capturing. Actually, the total time required for capturing at least one preliminary image to be captured between frames for moving image capture must not exceed the idle period T_idle described with reference to FIG. For example, when the preliminary image that can be captured during the idle period T_idle is one preliminary image having a resolution of 1/16 and one preliminary image having a resolution of 1/64, a frame (n 2), only the preliminary image 404 and the preliminary image 405 are photographed, and after the frame (n), only the preliminary image 401 and the preliminary image 402 are photographed. As a result, it is possible to capture a preliminary image for improving the image quality of the captured still image without affecting moving image shooting. For example, a technique for increasing the number of pixels to achieve a super spatial resolution can be used to improve the image quality. A still image with a resolution of 1 in frame (n) is interpolated with image data of the preliminary image 401 and the preliminary image 402. For example, the resolution can be increased to double the resolution (resolution 2).

  Here, in FIG. 6B, the preliminary image 404, the preliminary image 405, and the preliminary image 406 are actually acquired before the release button is pressed (before the original image of the frame (n) is captured). Although it is difficult, for example, a preliminary image is repeatedly taken into the ring buffer from the point when the release button is half-pressed (in general electronic cameras, focus detection and photometry are performed by half-pressing the release button). When the release button is fully pressed (still image capturing time), the preparatory image before still image capturing can be captured by terminating the capturing of the preliminary image into the ring buffer. In this case, the storage capacity of the ring buffer is sufficient if there is a data amount of a preliminary image with the maximum resolution that can be captured in the idle period T_idle.

As described above, in a shooting mode in which still image shooting is performed during moving image shooting, when a plurality of preliminary images can be shot within the idle period T_idle, the resolution of the preliminary image close to the original image (still image) is set to the original. By making the resolution lower than the resolution of the preliminary image located far from the image, as many preliminary images as possible can be taken within the limited idle period T_idle, and the still image can be improved in image quality with high accuracy. This is because, as explained in the previous embodiment, a preliminary image at a position close to the original image is more likely to be similar to the original image than a preliminary image at a far position, so This is because it can be used as an interpolation image of the original image.
[Example of shooting still images]
Next, an example of taking a preliminary image in the still image single shooting mode in which one still image is shot each time the user presses the release button will be described with reference to FIG. In the present embodiment, a preliminary image is captured in order to improve the quality of the captured still image, as in the case of still image capturing during moving image capturing described with reference to FIG.

  In the example of FIG. 7, when the user presses the release button of the operation unit 107 (full press), the original image acquisition processing unit 151 of the control unit 106 captures a still image (original image) 501. In a normal electronic camera, only the still image 501 is taken into the image buffer 103, but in this embodiment, the preliminary image acquisition processing unit 153 of the control unit 106 captures a preliminary image before and after the still image 501. For example, the preliminary image acquisition processing unit 153 captures a preliminary image 503 or a preliminary image 504 that can be captured in an idle period T_idle until the user presses the release button again to capture the next still image 502 after the still image 501 is captured. Shoot. Here, when the resolution of the still image 501 and the still image 502 is set to 1, the resolution determination processing unit 152 of the control unit 106, for example, reduces the resolution of the preliminary image 503 closest to the still image 501 to 1 / 64, and the resolution of the preliminary image 504 located far from the preliminary image 503 is determined to be 1/16, which is higher. Note that the preliminary image acquisition processing unit 153 captures only the preliminary image 503 when the preliminary image 503 is the only preliminary image that can be captured during the idle period T_idle. When the preliminary image 505 or the preliminary image 506 before the release button is pressed is captured, as described in the embodiment of FIG. 6, for example, the preliminary image is repeatedly captured in the ring buffer from the time when the release button is half-pressed. By doing so, it is possible to capture a preliminary image before taking a still image.

As described above, even in the still image single shooting mode in which one still image 501 is shot, a preliminary image is shot during the idle period T_idle until the user presses the release button again to shoot the next still image 502. be able to. In particular, when a plurality of preliminary images can be taken within the idle period T_idle, the resolution of the preliminary image located close to the original image (still image) is made lower than the resolution of the preliminary image located far from the original image. As many preliminary images as possible can be taken within the limited idle period T_idle, and still images can be improved in image quality with high accuracy. This is because, as explained in the previous embodiment, a preliminary image at a position close to the original image is more likely to be similar to the original image than a preliminary image at a far position, so This is because it can be used as an interpolation image of the original image. Next, the resolution and similarity of the preliminary image will be described in detail.
[Relationship between preliminary image resolution and similarity]
In each of the embodiments described above, when the resolution determination processing unit 152 of the control unit 106 determines the resolution of the preliminary image, the determination is made based on whether the shooting position of the preliminary image is near or far from the original image, The resolution of the preliminary image at a position close to the original image is set to be lower than that of the preliminary image at a position far from the original image so that as many preliminary images as possible can be taken within the limited idle period T_idle. . As described in each embodiment, this is because the preliminary image located closer to the original image (the difference in photographing time is smaller) has a higher similarity to the original image, so even a low-resolution preliminary image can be reproduced during slow playback. This is because it is sufficient for creating an interpolated image and supplementary data for a still image. In other words, because the preliminary image located far from the original image has a large difference in shooting time, the image feature amount (shape of main subject, brightness, hue, contour component, frequency component, motion vector, etc.) changes greatly. There is a high possibility. For this reason, since the difference between the original image and the preliminary image is large, it is desirable that the preliminary image to be captured is an image with as high a resolution as possible.

  Here, the similarity can also be defined as the reciprocal of the difference between the feature quantities. For example, when the shape of the main subject does not change, the similarity is increased because the difference in feature amount is small. Similarly, when the brightness does not change and the hue does not change, the difference in feature amount is small, so the degree of similarity increases. Alternatively, the contour component corresponds to the amount of the edge, and the contour component increases in an image with many structures. For example, when the image changes from a building image to an image such as a sky or a cloud, the difference in feature amount increases. Similarity is reduced. The same applies to the high frequency components. Further, when the feature amount is a motion vector, if the motion vector of the main subject changes, the difference in the feature amount increases and the similarity decreases. For example, even if the same brightness, the same hue, the same contour component, and the same main subject shape are used, if the main subject moves within the screen, only the motion vector may change. Even in such a case, the optimum resolution of the preliminary image can be determined by determining the similarity based on the change of the motion vector. Such similarity determination processing is performed by the similarity determination processing unit 154 of the control unit 106 of FIG.

Here, the relationship between the resolution of the preliminary image and the similarity is shown in FIG. In FIG. 8, the horizontal axis indicates the similarity and the vertical axis indicates the resolution of the preliminary image. For example, when brightness (such as V (Value) data screen average difference in HSV space) is used as a parameter as the similarity, the similarity is increased if the images have the same brightness, and the similarity decreases as the brightness is different. Become. In FIG. 8, the relationship between the resolution and the similarity of the preliminary image may be a linear inverse proportion such as the characteristic 601, or the number of preliminary images that can be photographed has a relatively large margin. As with the characteristic 602, even a slight error (difference in feature amount) may be used for shooting with high resolution. Alternatively, when there is not enough room for the number of preliminary images that can be captured, the characteristics 603 may be used to capture at the lowest possible resolution. Here, since the similarity between the preliminary image to be acquired and the original image or another preliminary image cannot be obtained in practice, the similarity determination processing unit 154 determines the previous preliminary image and the previous previous image. The similarity is obtained using an original image or the like. This point will be described in detail in the following example.
[Example of determining the resolution of the preliminary image according to the degree of similarity]
Next, an example in which the resolution of the preliminary image is determined according to the similarity obtained by the similarity determination processing unit 154 of the control unit 106 will be described with reference to FIG. FIG. 9 is a diagram corresponding to an example in which still image shooting is performed during moving image shooting described with reference to FIG. 6, and FIG. 9A is exactly the same as FIG. 6A. Accordingly, FIG. 9B shows a frame (the frame (n) captured by the original image acquisition processing unit 151 of the control unit 106 when the user presses the release button of the operation unit 107 at the timing of the frame (n) in FIG. 9A. It is the figure which showed the example of the position of the still image (original image) of n), and the position and resolution of the preliminary image which the preliminary image acquisition process part 153 image | photographs before and after that. As in FIG. 4, in FIGS. 9A and 9B, the vertical axis indicates the resolution of the original image and the preliminary image, and the horizontal axis indicates the temporal flow of the frame number.

  In FIG. 9B, as in FIG. 6B, the preliminary images are taken at intervals of Δf. However, in practice, it is only necessary to secure a time during which each preliminary image can be acquired. There is no need.

  In FIG. 9B, the resolution determination processing unit 152 of the control unit 106 includes a preliminary image 701, a preliminary image 702, a preliminary image 703, a preliminary image 704, and a preliminary image 705 captured before and after the original image 700 of the frame (n). And the resolution of the preliminary image 706 is determined. At this time, the similarity determination processing unit 154 of the control unit 106 obtains the similarity between each preliminary image and the original image (frame (n)).

  Here, in the above description, for the sake of easy understanding, the similarity determination processing unit 154 of the control unit 106 determines the resolution of the preliminary image according to the similarity between the original image and the preliminary image. Since the similarity with the preliminary image to be acquired cannot be obtained, the similarity determination processing unit 154 obtains the similarity using the previous preliminary image and the previous original image, and then acquires the similarity. The resolution of the preliminary image to be determined is determined. For example, in FIG. 9B, the similarity determination processing unit 154 obtains the similarity between the preliminary image 702 and the original image (frame (n−1)) as the resolution of the preliminary image 701 to be captured. It is determined or the similarity between the preliminary image 702 and the preliminary image 703 is obtained and determined. Alternatively, the degree of similarity between the original image (frame (n−1)) and the immediately previous original image (frame (n−2)) may be obtained and determined. In any case, it is desirable to obtain the degree of similarity with an image at a position as close as possible in time. In FIG. 9B, when the similarity determination processing unit 154 determines that the similarity between the preliminary image 701, the preliminary image 702, the preliminary image 703, and the original image (frame (n-1)) is large, the feature amount changes. Therefore, the resolution determination processing unit 152 determines a relatively low resolution (1/64 resolution of the original image), and the preliminary image acquisition processing unit 153 captures these preliminary images at the determined resolution. .

  In addition, the resolution of the preliminary image 704 captured after capturing the original still image (frame (n)) is determined by the similarity determination processing unit 154 by the similarity between the original image (frame (n)) and the preliminary image 701 or The similarity between the original image (frame (n)) and the original image (frame (n−1)) is obtained, and is determined by the resolution determination processing unit 152 according to the similarity. For example, when the similarity determination processing unit 154 determines that the similarity between the original image (frame (n)) and the preliminary image 701 is large, the resolution determination processing unit 152 determines that the resolution is relatively low (1/64 of the original image). The preliminary image acquisition processing unit 153 captures the preliminary image 704 with the determined resolution.

  Similarly, the resolution of the preliminary image 705 is determined by the resolution determination processing unit 152 according to the similarity between the original image (frame (n)) and the preliminary image 704 obtained by the similarity determination processing unit 154. The For example, when the similarity determination processing unit 154 determines that the similarity between the original image (frame (n)) and the preliminary image 704 is slightly smaller than the previous similarity, the resolution determination processing unit 152 determines the immediately preceding resolution. The resolution (resolution of 1/16 of the original image) is slightly higher than the resolution (resolution of 1/16 of the original image), and the preliminary image acquisition processing unit 153 captures the preliminary image 705 at the determined resolution.

  Further, as for the resolution of the preliminary image 706, the similarity determination processing unit 154 obtains the similarity between the preliminary image 705 and the preliminary image 704 or the similarity between the original image (frame (n)) and the preliminary image 704, and the similarity. Is determined by the resolution determination processing unit 152 according to the above. For example, when the similarity determination processing unit 154 determines that the similarity between the preliminary image 705 and the preliminary image 704 is smaller than the previous similarity, the resolution determination processing unit 152 determines the resolution determined immediately before (1 of the original image). ) (Resolution of 1/4 of the original image), and the preliminary image acquisition processing unit 153 captures the preliminary image 706 at the determined resolution.

  Here, in FIG. 9B, as in the case of FIG. 6B, the preliminary image 701, the preliminary image 702, and the preliminary image 703 are displayed before the release button is pressed (the original image of the frame (n)). Actually, it is difficult to acquire a preliminary image because it is before shooting), but for example, it can be realized by repeatedly capturing the preliminary image in the ring buffer from the time when the release button is half-pressed.

In this way, a preliminary image is captured at a low resolution when the similarity is large, and a preliminary image is captured at a high resolution when the similarity is small, so that as many preliminary images as possible within the limited idle period T_idle. Can be taken, and still images can be improved in quality with high accuracy. In the case of moving image shooting, the resolution of the preliminary image can be similarly determined using the similarity, and the same effect as in the case of a still image can be obtained.
(Modification of electronic camera 101)
Next, a modification of the electronic camera 101 will be described with reference to FIG. FIG. 10 is a block diagram showing a configuration of an electronic camera 101a that is a modification of the electronic camera 101 described in FIG. In FIG. 10, the same reference numerals as those in FIG. 10 differs from FIG. 1 in that a second imaging unit 110 is provided. The second imaging unit captures an image for similarity determination at a fixed resolution.

  In FIG. 10, the second imaging unit 110 can shoot a subject image having the same angle of view as the imaging unit 102 regardless of the imaging process of the imaging unit 102. For example, the second imaging unit 110 may also be used as a photometric sensor, for example. In addition, the second imaging unit 110 is directly connected to the control unit 106, but may be connected via the system bus 108.

  The control unit 106 images a subject image by the second imaging unit 110 separately from the process of capturing the original image and the preliminary image by the imaging unit 102. This is shown in FIG. FIG. 11 shows the similarity determination image 801 to the similarity determination image 801 in the second imaging unit 110 in correspondence with the preliminary image 706 from the original image (frame (n)) and the preliminary image 701 in FIG. It is the figure which showed the example in the case of imaging 814.

  In the example of FIG. 11, the similarity determination image 800 to the similarity determination image 807 are captured immediately before the preliminary image 706 is captured from the original image (frame (n)) and the preliminary image 701. Accordingly, for example, when the resolution determination processing unit 152 of the control unit 106 determines the resolution when the preliminary image 703 is captured, the similarity determination processing unit 154 causes the similarity determination image 800 captured by the second imaging unit 110 to be captured. And the similarity determination image 801 are used to determine the similarity. Here, the similarity determination image 800 is an image captured by the second imaging unit 110 immediately before the original image (frame (n−1)), and the similarity determination image 801 is captured immediately before the preliminary image 701. It is an image that was made. Since the second imaging unit 110 has a fixed resolution, the similarity can be determined quickly, stably and accurately. The subsequent processing is performed in the same manner as in FIG. 9, and the resolution when the resolution determination processing unit 152 captures the preliminary image 703 is determined according to the similarity obtained by the similarity determination processing unit 154, and the preliminary image is obtained. The imaging processing unit 153 captures the preliminary image 703.

  Further, when determining the resolution of the preliminary image 702, the similarity determination processing unit 154 uses the similarity determination image 801 immediately before the previous preliminary image 701 and the similarity immediately before the preliminary image 702 to be photographed next. The degree of similarity is obtained using the degree determination image 802, and the preliminary image 702 is captured at a resolution corresponding to the degree of similarity. Similarly, when determining the resolution of the preliminary image 703, the similarity determination processing unit 154 immediately precedes the similarity determination image 802 immediately before the previous preliminary image 702 and the preliminary image 703 to be photographed next. Similarity determination images 803 are used to determine the similarity, and the preliminary image 704, the preliminary image 705, and the preliminary image 706 are respectively similar to the previous similarity determination image and the previous preliminary image similarity determination image. Is used to obtain the similarity, and each preliminary image is photographed at a resolution corresponding to the similarity.

  In the example of FIG. 11, the similarity determination image is taken one by one before the original image or each preliminary image is taken, but it is not necessary to take the similarity determination image irregularly. For example, the similarity determination processing unit 154 always captures images with the second imaging unit 110 at a fixed frame rate that is short enough to capture a plurality of similarity determination images between the original image or each preliminary image. In this way, the similarity between the similarity determination image close to the previous preliminary image or the original image capturing timing and the similarity determination image immediately before the preliminary image to be captured next is obtained, and this similarity The resolution determination processing unit 152 may determine the resolution of the preliminary image to be captured next using the degree.

  Although various embodiments have been described above, the electronic camera 101 or the electronic camera 101a according to the present embodiment captures preliminary images having different resolutions during an idle period (T_idle) that is not used for capturing an original image. Therefore, it can be used as a frame interpolated image at the time of slow playback of a moving image to realize high-quality slow playback, and can also be used for applications such as increasing the resolution of still images. In particular, since the resolution of the preliminary image to be shot is determined according to the similarity of the preliminary image before shooting, an appropriate preliminary image can be acquired efficiently.

  The taken preliminary image may be stored in the storage unit 105 together with the original image in time series, or may be stored in the storage unit 105 as a file different from the original image. When the preliminary image is stored in another file, the attribute is changed with the same file name, or information indicating that the file is related to the header information is described.

  As mentioned above, although the electronic camera according to the present invention has been described by way of example in each embodiment, it can be implemented in various other forms without departing from the spirit or main features thereof. Therefore, the above-described embodiment is merely an example in all respects and should not be interpreted in a limited manner. The present invention is defined by the claims, and the present invention is not limited to the text of the specification. Further, all modifications and changes belonging to the equivalent scope of the claims are within the scope of the present invention.

DESCRIPTION OF SYMBOLS 101,101a ... Electronic camera 102 ... Imaging part 103 ... Image buffer 104 ... Image processing part 105 ... Memory | storage part 106 ... Control part 107 ... Operation part 108 ... System Bus 109 ... Display unit 110 ... Second imaging unit 201 ... Optical system 202 ... Solid-state imaging device 203 ... AFE 204 ... TG
151 ... Original image acquisition processing unit 152 ... Resolution determination processing unit 153 ... Preliminary image acquisition processing unit 154 ... Similarity determination processing unit

Claims (5)

Imaging means for imaging subject images at a plurality of designated resolutions;
Storage means for storing an image captured by the imaging means;
An original image acquisition unit that designates a first resolution set in advance among the plurality of resolutions to the imaging unit and takes an original image of the first resolution from the imaging unit into the storage unit;
A preliminary image different from the first resolution that can be captured from the imaging unit within an idle time from the end of capturing the first original image captured by the original image acquisition unit to the start of capturing the next second original image. Resolution determining means for determining the resolution of
Specifying the resolution the resolution determination means has determined the imaging means possess a preliminary image acquisition means for capturing said preliminary image in the storage means,
The resolution determining means determines resolutions of a plurality of preliminary images different from the first resolution within the idle time;
The preliminary image acquisition means specifies the resolution determined by the resolution determination means to the imaging means, and takes the plurality of preliminary images into the storage means . The electronic camera according to claim 1 ,
A similarity determination unit for obtaining a similarity between a plurality of images captured by the imaging unit before obtaining the preliminary image;
The resolution determining unit increases the resolution of the preliminary image when the similarity obtained by the similarity determination unit is low when determining the resolution of the preliminary image captured from the imaging unit, and the similarity determination unit The electronic camera is characterized in that if the degree of similarity determined by is high, the resolution of the preliminary image is lowered. The electronic camera according to claim 2 ,
The electronic camera according to claim 1, wherein the similarity is at least one of a shape change, brightness, hue, contour component, frequency component, and motion vector of a main subject. The electronic camera according to claim 2 or 3 ,
In addition to the image pickup means, a second image pickup means for picking up an image having the same angle of view as the image pickup means is further provided,
The electronic camera according to claim 1, wherein the similarity determination unit determines the resolution of the preliminary image to be captured from the imaging unit by obtaining the similarity of a plurality of images captured by the second imaging unit. Imaging means for imaging subject images at a plurality of designated resolutions;
Storage means for storing an image captured by the imaging means;
An original image acquisition unit that designates a first resolution set in advance among the plurality of resolutions to the imaging unit and takes an original image of the first resolution from the imaging unit into the storage unit;
A preliminary image different from the first resolution that can be captured from the imaging unit within an idle time from the end of capturing the first original image captured by the original image acquisition unit to the start of capturing the next second original image. Resolution determination means for determining the resolution of the image based on an image taken before the preliminary image;
A preliminary image acquisition unit that designates the resolution determined by the resolution determination unit in the imaging unit and loads the preliminary image into the storage unit;
An electronic camera comprising:

Images