JP2006211426A - Image sensing device and its image generating method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image sensing device having a new image-sensing system which has not exist in conventional devices and a method for generating an image for the image sensing device. <P>SOLUTION: When a temporary stop command is issued during a stream reproduction by operating an operation input 20 by an operator (a user) in one stream in which the images having a high resolution are mixed at every specified interval in an animation having a low resolution, an image processor 14 generates and processes the images having the high resolution and displays them on a display unit 19 when the images of one frame in this case have the high resolution, and generates and processes the images having the high resolution from the before-after images and displays them on the display unit 19 when the images of one frame in this case have the low resolution. Even when a temporary stop is conducted during the reproduction of the animation, the images having a high minuteness can be displayed. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an imaging apparatus such as a digital camera and an image generation method. More specifically, an image to be extracted as a moving image (relatively low number of pixels) and an image that can be handled as a still side (relatively high number of pixels) are on the same stream. The present invention relates to an image pickup apparatus handled by, and an image generation method thereof.

Conventionally, a digital video signal based on an image signal captured using an image sensor is compressed by discrete cosine transform (DCT), wavelet transform and variable length code, and recorded on a recording medium such as a magnetic tape, a magnetic disk, or an optical disk. Digital cameras have been proposed.
This digital camera has a moving image recording mode and a still image recording mode. In the moving image recording mode, the digital camera performs compression recording for a moving image and records it on a recording medium. In the still image recording mode, the digital camera performs compression for a still image. Record on a recording medium.

  Conventionally, various imaging devices capable of capturing high-resolution still images during moving image shooting have been proposed, as shown as first to eighth imaging devices below.

  The first imaging device captures a still image at a predetermined cycle during moving image capturing (see Patent Document 1).

  The second imaging device captures a high-resolution still image at a fixed interval by an operator's operation or automatically during normal-resolution video recording, and the still image is used as still image data. The recorded moving image is recorded as moving image data, and the presence of still image data is displayed during moving image reproduction, and the still image display is switched by the operation of the operator (see Patent Document 2).

  The third imaging device is an image recording device that captures a still image by shooting a moving image of a predetermined period by an operator's shot button operation, and corrects the captured still image using the preceding and following moving images. It is.

  The fourth imaging device is a moving image recording device such as a camcorder that captures a high-definition still image when the shutter button is operated during moving image shooting, and forcibly encodes the high-definition still image when the shutter button is operated. It is encoded as an image (I picture) (see Patent Document 3).

  The fifth imaging device is capable of still image shooting during moving image shooting, and when there is no desired still image, a high-resolution image corresponding to the desired moving image is obtained as a desired moving image and It is synthesized with related still images (see Patent Document 4).

  The sixth imaging device is a digital camera that starts recording a moving image by half-pressing the shutter button and captures a still image while recording an operation by pressing the shutter button. Is stored in association with the moving image data (see Patent Document 5).

  A seventh imaging device is an image recording device that captures a still image by operating an operator's shutter button during moving image shooting, and corrects the captured still image using moving images before and after (Patent Document). 6).

The eighth imaging device records moving image and still image data as a series of files, the moving image is recorded in MP @ ML (Main Profile at Main Level), and the still image is recorded in MP @ HL (Main Profile at High). Level) (see Patent Document 7).
JP-A-7-245722 Japanese Patent Laid-Open No. 9-51498 JP-A-7-284058 JP 2002-51252 A JP 2002-84442 A JP-A-7-143439 JP-A-11-234623

By the way, in each of the above-described imaging apparatuses, in general, when an image of a moving image level is captured, exposure and data transfer are sequentially performed by repeating a rolling shutter. In addition, when continuous shooting of still images is performed, a rolling shutter is similarly continued or a mechanical shutter is used. Here, in the image capture using only the rolling shutter, image distortion occurs at the top and bottom of the image. However, since it is a moving image to the last, it is permissible.
However, this image distortion becomes unacceptable when it becomes the capture of a still image. Therefore, a mechanical shutter is necessary, but there is a limit to mechanical shutter driving in order to capture continuous images at high speed.

Further, in a digital camera or the like, in order to obtain a desired image with high resolution, it is necessary for an operator to observe a subject to be photographed and to operate a release button at a timing considered here.
However, even if it is operated at the timing considered here, a desired image is not always obtained, and a continuous shooting function for solving this problem has been put into practical use.
However, if the image is taken at a high resolution, for example, several frames per second for 10 seconds, the recorded image data is enormous, and considering the capacity of the current memory card or the like, the practicality is poor.
Also, those with such a continuous shooting function, in the case of high resolution still image continuous shooting with millions of pixels, the upper limit of the continuous shooting speed greatly depends on the readout processing capacity of the CCD, etc., so about several frames per second If the desired image is a fast moving subject, for example, it is difficult to obtain the desired image even if continuous shooting is used.

  Also, if you play a relatively low-resolution video and view it as a video, it will be fine, but if you pause the playback operation and view it as a still image, the image will be rough and good quality when printed. That's not true. In addition, there is a problem that even if frame-by-frame playback is performed, it is impossible to observe the detailed state of the subject with a rough image.

  Therefore, even when pausing or frame-by-frame playback during video playback, it can be displayed as a high-definition still image, and the recording capacity is practically suppressed without worrying about the speed of continuous shooting. In addition, a new imaging device has been developed that allows the operator to obtain a desired high-resolution still image without worrying about the shutter timing.

  The present invention is completely new that has occurred in such development, and in particular, is not based on the above-described problem of obtaining a high-resolution generated image. An object of the present invention is to provide an image generation method.

  According to a first aspect of the present invention, an optical image of a subject is formed on an imaging device, high-resolution high-image data or low-resolution low-image data is read from the imaging device, and predetermined image processing is performed on these image data. A first compressed image obtained by intra-frame compressing the high image data, and a second image obtained by inter-frame compressing low image data in a period before and after the period for generating the first compressed image. Moving image generation means for generating a low-resolution moving image from a series of moving image streams including the compressed image, and the first compressed image or the first compressed image and at least the second compressed image of the moving image stream are high. Still image generation means for generating still images of resolution, and the playback mode is a video playback mode and at least frame advance mode, still image mode, slow playback mode, and intermittent playback mode. Having a mode switching means for switching to at least one of de, in accordance with the switched mode by said mode switching means, and switching means for switching between the moving image generation means and the still image generating means.

  Preferably, if the one-frame image when receiving a pause command during playback of the moving image stream is a low-resolution moving image generated from the first compressed image, the image is generated from the first compressed image. If the high-resolution image is a low-resolution moving image generated from the second compressed image and the first compressed image, a high-resolution image is generated from the second compressed image and the first compressed image. Image processing means for generating and displaying an image is further included.

  Preferably, after receiving a pause command during playback of the video stream, at least when receiving a command to switch to the frame advance mode, slow playback mode, or intermittent playback mode, sequentially generate high-resolution images. To display.

  The high image data is image data read out from the image sensor without being thinned out or image data read out by thinning out at an arbitrary thinning amount, and the low image data has a lower resolution than the high image data from the image sensor. This is an image read out by thinning out with an arbitrary thinning amount.

  Preferably, the shutter function includes a global shutter function and a rolling shutter function, and the image processing means generates a first compressed image from the pixel data captured by the global shutter, and the pixels captured by the rolling shutter A second compressed image is generated from the data.

  Preferably, when one screen of the second compressed image is designated, the image processing means decompresses and decodes the screen of the second compressed image with other images including the preceding and following first compressed images. Thus, high-resolution still image data representing the designated one screen is generated.

  Preferably, a stream output means for outputting a low-resolution continuous video stream using a high-resolution first compressed image and a low-resolution second compressed image; and A determination unit configured to determine whether the resolution is a resolution; and a determination signal output unit configured to output a signal indicating whether the first compressed image has a high resolution or a low resolution by the determination unit.

  Preferably, an imaging device that forms an optical image of a subject on an imaging device, reads high-resolution high-image data or low-resolution low-image data from the imaging device, and performs predetermined image processing on the image data. An image generation method comprising: inter-frame compressing the high image data to generate a first compressed image; and low image data in a period before and after the first compressed image generation period. A series of moving image streams including a step of generating a frame-compressed second compressed image, the first compressed image, and the second compressed image are formed, and a low-resolution moving image is generated from the moving image stream. A high-resolution still image is generated from the moving image generation step and the first compressed image or the first compressed image and at least the second compressed image of the moving image stream. A still image generation step, a mode switching step for switching the playback mode to at least one of the video playback mode, frame advance mode, still image mode, slow playback mode, and intermittent playback mode, and the mode switching step described above. According to the switched mode, there is a switching step for switching the processing of the moving image generation step and the still image generation step.

  According to the present invention, an unprecedented novel imaging device and an image generation method thereof can be provided.

  Embodiments of an imaging apparatus according to the present invention will be described below with reference to the accompanying drawings.

  FIG. 1 is a diagram illustrating an overall configuration of an imaging apparatus 1 in the present embodiment. The imaging apparatus 1 is roughly composed of an optical system, a signal processing system, a recording system, a display system, and a control system.

The imaging apparatus 1 according to the present embodiment has a function of generating a first compressed image by intra-frame compression (compression encoding) of high-pixel image data in a signal processing system during moving image shooting, and a first compressed image A function of generating a second compressed image obtained by inter-frame compression (compression encoding) of low-pixel image data in a period before and after the period of generating the first compressed image, and the second compressed image A moving image generating function for forming a series of moving image streams including the generated moving image stream and generating a low-resolution moving image from the formed moving image stream; and a high resolution from the first compressed image or the first compressed image and at least the second compressed image of the moving image stream Still image generation function for generating still images, and at least one of playback mode, video playback mode, frame advance mode, still image mode, slow playback mode, or intermittent playback mode A mode switching function for switching to One, according to the switched mode by the mode switching function has a switching function for switching the processing of the moving image generation function and a still image generating function, a.
In the imaging apparatus 1, when frame advance, stop, and slow playback are instructed in a video playback mode of an image stream in which a low resolution video and a high resolution still image are mixed, a low resolution frame feed, a low resolution still image, Instead of low-resolution slow playback, high-resolution still image frame advance, high-resolution still image still playback (when stopped), and high-resolution still image slow playback.

The imaging apparatus 1 also generates intra-frame compression of high-pixel image data to generate a first compressed image, and inter-frames the low-pixel image data in a period before and after the period for generating the first compressed image. When a second compressed image is generated by compression and one screen of the second compressed image is designated, the second compressed image and the other compressed image including the first compressed image before and after the second compressed image are decompressed and decoded. It has a function of generating high-resolution still image data showing a designated one screen.
The imaging apparatus 1 according to the present embodiment uses both the rolling shutter function and the global shutter function during the acquisition of the single moving image data.
In addition, the imaging apparatus 1 according to the present embodiment corrects the image level of the high-resolution first compressed image and the image level of the low-resolution second compressed image to substantially the same level, and the first compressed image and Control is performed to keep the output level of the second compressed image constant.
In addition, the imaging apparatus 1 according to the present embodiment includes three modes: a reproduction mode, a still image reproduction mode, and a stream data change mode. Playback modes include a moving image playback mode and at least a frame advance mode, a still image mode, a slow playback mode, and an intermittent playback mode.
Hereinafter, the configuration and function of each unit will be described.

The optical system includes a lens optical system 10 and an image sensor 11 such as a CMOS sensor.
The lens optical system 10 includes an optical lens 101 that faces a subject, an optical low-pass filter (not shown), and the like. In the lens optical system 10, an optical image of the subject is collected by the optical lens 101 and an image of the subject is formed on the image sensor 11.

The image sensor 11 as an image sensor is composed of, for example, a CMOS image sensor provided with a color filter, and photoelectrically converts an object image formed by the lens optical system 10.
Further, it has a shutter function including a global shutter function and a rolling shutter function.
The shutter function is controlled so that the global shutter function or the rolling shutter function is selectively used according to the control of the control system, and the image can be captured by the global shutter while the pixels are captured by the rolling shutter. The In other words, the rolling shutter function and the global shutter function are used in combination during capture of the single moving image data.

The signal processing system digitally outputs a CDS 12 that is a correlated double sampling circuit (CDS: Correlated Double Sampling) for reducing noise by sampling an electrical signal output from the image sensor 11 and an analog signal output from the CDS 12. An A / D converter 13 that converts the signal into a signal and an image processing unit 14 that performs predetermined image processing to be described later on the digital signal output from the A / D converter 13 are configured.
The image processing unit 14 basically includes a moving image generating unit, a still image generating unit, a mode switching unit, and a switching unit according to the present invention.

The image processing unit 14 according to the present embodiment basically generates the first compressed image by intra-frame compressing the pixel data captured by the global shutter, and before and after the period for generating the first compressed image. During this period, the pixel data captured by the rolling shutter is inter-frame compressed to generate a second compressed image.
As will be described in detail later, the image processing unit 14 reads and generates the first compressed image from the image sensor without thinning out the image data, and reads the second compressed image by thinning out the image data from the image sensor. To generate.
The image processing unit 14 performs intra-frame compression on high-pixel image data when generating the first compressed image, and inter-frame compresses on low-pixel image data when generating the second compressed image. In addition, when the image processing unit 14 generates the second compressed image by thinning out data from the image sensor, the image processing unit 14 performs thinning out reading by performing integration processing of pixels in the vicinity of the same color.
The image processing unit 14 has a function of correcting the image level of the first compressed image and the image level of the second compressed image to substantially the same level.

As will be described in detail later, the image processing unit 14 corrects the image level of the first compressed image based on the R level obtained by performing the integration process readout of the second compressed image, thereby correcting the first compressed image. It has a function of correcting the image level and the image level of the second compressed image to substantially the same level.
Further, the image processing unit 14 maintains the image level of the first compressed image, and corrects the image level of the second compressed image by dividing by the integration amount of the integration process, thereby correcting the first compressed image. It has a function of correcting the image level and the image level of the second compressed image to substantially the same level.
When the image processing unit 14 designates one screen of the second compressed image, the image processing unit 14 decompresses and decodes the second compressed image and the other compressed images including the first compressed image before and after the second compressed image. A function of generating high-resolution still image data representing a screen;

When the image processing unit 14 designates one screen of the first compressed image on one stream data, the image processing unit 14 reduces the resolution of the first compressed image to the same level as the second compressed image. The compressed image is replaced with the first compressed image with reduced resolution and recorded in the memory 15 again.
The image processing unit 14 collectively reduces the resolution of the plurality of first compressed images of one stream data to the same level as the second compressed image, and reduces the resolution of the first compressed image of one stream data. It is replaced with the compressed image 1 and recorded again in the memory 15.

  The image processing unit 14 having the above functions performs the following processing in the three modes described above, that is, the reproduction mode, the still image reproduction mode, and the stream data change mode.

  In the playback mode, the image processing unit 14 generates and displays a low-resolution continuous video stream using the high-resolution first compressed image and the low-resolution second compressed image.

In the still image reproduction mode, a specific image is designated by an operator while a low-resolution continuous video stream is reproduced using a high-resolution first compressed image and a low-resolution second compressed image.
In the still image reproduction mode, when the image designated by the operator is the first compressed image, the image processing unit 14 outputs and displays a high-precision still image of the first compressed image.
If the designated image is the second compressed image, a high-resolution image corresponding to the designated image is generated from the image and at least one of the first and second compressed images and the second compressed image. Output and display.
Alternatively, in the still image reproduction mode, a continuous image such as a low-resolution continuous thumbnail is displayed by the high-compression first compressed image and the low-resolution second compressed image, and the operator performs key operations, etc. The image processing unit 14 performs the same processing.

In the stream data change mode, the image processing unit 14 automatically converts all the high-resolution first compressed images in the designated one stream data into low-resolution first compressed images by the operator's key operation or the like. Perform the replacement process.
The file size can be reduced by processing in this stream data change mode. This process will be further described in detail.

Even if a simple moving image file is stopped on a single image being played back, the resolution of the moving image is low, so that even if the moving image is stored as a still image or printed out, the image quality is poor.
In order to solve this, in the imaging apparatus 1, as shown in FIG. 2, an image (for example, VGA size) of a plurality of frames (for example, 30 frames) is captured per second, a file is formed as a moving image, Among them, several frames (for example, 5 frames) per second have a function of shooting a still image (for example, SXGA size) having a higher resolution than a moving image and recording it as one stream.
This is simply by inserting several high-resolution still images into the video file, and complementing the high-resolution still images captured at equal intervals with the VGA-sized image, no matter where you stop in this stream. It is possible to store and print as a high-quality still image.
However, if this is the case, the file size increases due to the presence of a high-resolution still image in the moving image file, which may occupy the remaining capacity of the recording memory 15 with limited capacity.
Therefore, in the image processing unit 14 of the imaging apparatus 1 according to the present embodiment, when the operator determines that there is no need for printout or the like taken in the still image moving image mixed mode, as shown in FIG. By converting a high-resolution still image (SXGA size) from a data file into a low-resolution image (VGA size) and forming a simple moving image file, the file size can be reduced.
This can be executed by the operator himself by key operation or the like, and the remaining capacity of the memory 15 can be given a margin. In addition, since only the high-resolution information of the still image is cut, there is no problem that the image quality is deteriorated when it is reproduced as a moving image.

Further, as shown in FIG. 2, the image processing unit 14 operates the operation of the operation input unit 20 of an operator (user) in one stream in which high-resolution images are mixed at regular intervals in a low-resolution moving image. When a pause command is issued during stream playback, if the one-frame image at that time is a high-resolution image, the high-resolution image is higher than the previous and subsequent images if it is a low-resolution image. A resolution image is generated and displayed on the display unit 19.
As shown in FIG. 2, the image processing unit 14 performs stream processing by operating the operation input unit 20 of an operator (user) in a single stream in which high-resolution images are mixed at regular intervals in a low-resolution moving image. When a frame advance command or a slow playback command is issued after a pause command is issued during playback, high-resolution images are sequentially generated and displayed on the display unit 19.

As described above, FIG. 2 shows one stream in which high-resolution images are mixed at regular intervals in a low-resolution moving image.
For example, in the playback mode, the image processing unit 14 reproduces and displays the stream file on the display unit 19 as a low-resolution video stream even during playback of the stream file as a video.
During playback of a moving image, for example, when a pause command is issued by an operator (user) operation at a position A in FIG. 2, the image of A is displayed on the display unit 19 with high resolution.
When a pause command is issued by a user operation at the position B in FIG. 2, a high resolution image generated by decompression decoding from the previous and next high resolution images is displayed on the display unit 19.
When a frame advance command is issued by a user operation while the display image is paused, the next image after the currently displayed image is displayed with high resolution. Similarly, during slow playback, images are displayed in high resolution.
As described above, the imaging apparatus 1 according to the present embodiment can display a high-definition image even when paused during moving image reproduction, and even if the paused moving image is frame-fed, It is possible to sequentially display high-definition images as if high-resolution images were taken continuously.

Hereinafter, the operation in the above-described playback mode in the image processing unit (and the control unit) will be described more specifically.
FIG. 3 is a flowchart for explaining the operation in the playback mode in the present embodiment.

The playback mode for moving images and still images is set (ST1), and a moving image and still image mixed stream file is selected (ST2).
In this state, the operator (user) operates a button, GUI, or the like of the input operation unit 20 to determine whether or not there is a moving image reproduction instruction (ST3).
If it is determined in step ST3 that a moving image reproduction instruction has been given, reproduction of a moving image and still image mixed stream is started (ST4), and a moving image at a low resolution is reproduced (ST5).
Here, it is determined whether or not the operator (user) has operated the button or GUI of the input operation unit 20 to give an instruction to pause moving image playback (ST6).
If it is determined in step ST6 that a playback pause instruction has been issued, it is determined whether the image to be stopped is an intra-frame image (first compressed image) or an inter-frame image (second compressed image or third compressed image). A determination is made (ST7).
If it is determined in step ST7 that the image to be stopped is an intra frame image, a high-resolution still image is displayed on the display unit 19 (ST8). On the other hand, if it is determined that the image to be stopped is an inter-frame image, the decompressed and decoded high-resolution still image is displayed on the display unit 19 (ST9).
Next, it is determined whether or not an operator (user) has issued a frame advance reproduction instruction by operating a button or GUI of the input operation unit 20 (ST10).
If it is determined in step ST10 that there has been a frame advance playback instruction, the process stops at the next frame image (ST11), and the process returns to step ST7.
On the other hand, if it is determined in step ST10 that there is no frame advance reproduction instruction, it is determined whether or not the operator (user) operates the button or GUI of the input operation unit 20 to give a moving image reproduction instruction (ST12).
If it is determined in step ST12 that a moving image playback instruction has been issued, the process returns to step ST5.
On the other hand, if it is determined in step ST12 that there is no operation reproduction instruction, it is determined whether or not the operator (user) has operated the button or GUI of the input operation unit 20 to instruct reproduction mode end (ST13).
If it is determined in step ST13 that there is no instruction to end the reproduction mode, the high-resolution still image is continuously displayed and the process proceeds to step ST10 (ST14).
If it is determined in step ST13 that a reproduction mode end instruction has been issued, the image display (reproduction) is terminated.

In addition to the stream output function for outputting a video stream, the image processing unit 14 uses the first compressed image with a high resolution and the second compressed image with a low resolution in the above three modes. When outputting a continuous video stream, it is determined whether the first compressed image being simultaneously output is high resolution or low resolution, and a signal indicating whether the first compressed image is high resolution or low resolution is used. Has a function to output.
Specifically, when the first compressed image exists on the N-divided multi-display screen, a mark capable of determining high resolution or low resolution is displayed in the vicinity of the screen of the first compressed image. In the moving image display mode, a mark indicating high resolution is displayed on the screen in a superimposed manner.
As a further aspect, during transfer to a recording medium such as another device or a memory card, a flag signal indicating whether the first compressed image is high or low is added to the first compressed image and transferred.
In addition, on the file list display screen showing the video stream group, whether or not the first compressed image with high resolution exists in one stream data, how many screens exist, and on the existing stream data. Transfer location data (time information) together.

The recording system includes a memory 15 that stores a control program executed by the control unit 16 and compressed data of the image data generated by the image processing unit 14.
A series of stream data of the first compressed image and the second compressed image is stored in the memory 15 as storage means by the image processing unit 14.

  The display system includes a D / A converter 18 that converts image data processed by the image processing unit 14 and stored in a built-in image memory, and an LCD (Liquid Crystal) that functions as a finder by displaying an input image. The display unit 19 including a display) is included.

  The control system includes a timing generator 17 that controls the operation timing of the image sensor 11 to the A / D converter 13, an operation input unit 20 for inputting a shutter operation and other commands by the user, an image processing unit 14, A control program stored in the memory 15 is read out, and a CPU (Central Processing Unit) that controls the entire imaging apparatus 1 based on the read control program and a command from the user input from the operation input unit 20. ) And the like.

  The control unit 16 controls to use a rolling shutter function in the case of a moving image and to use a global shutter in the case of a still image when capturing image data having a different number of pixels on one stream.

  Here, when capturing image data with a different number of pixels on one stream, a moving shutter is used for a moving image, and a rolling shutter or a global shutter is used for a still image.

  That is, in the imaging apparatus 1 according to the present embodiment, in a system in which the control unit 16 extracts a different number of pixels from the imaging element in a series of operations, an image (relatively low number of pixels) to be extracted as a moving image main body and a still image are extracted. When handling images that can be handled (relatively high number of pixels) on the same stream, the global shutter is not used for low pixel capture, and the global shutter (or mechanical shutter) is used or not used only for high pixel capture. .

Normally, when capturing an image of a moving image level, exposure and data transfer are sequentially performed by repeating a rolling shutter. In addition, when performing continuous shooting of still images, control is similarly performed using a rolling shutter, a mechanical shutter, or a global shutter.
Here, in the image capture using only the rolling shutter, image distortion occurs at the top and bottom of the image. However, since it is a moving image to the last, it is permissible.
However, this image distortion becomes unacceptable when it becomes the capture of a still image. Therefore, a mechanical shutter and a global shutter are required. Infinitely, in order to capture continuous images at high speed, there is a limit to mechanical shutter drive, and a global shutter becomes indispensable.
Here, when capturing image data with a different number of pixels on one stream, the moving image uses the rolling shutter function, and the still image uses the rolling shutter or the global shutter.

FIG. 4 is a diagram illustrating an example of an image sensor (image sensor) when the rolling shutter is used.
Assuming that lines A to F exist in the element as shown in FIG. 4, the control of the image is as shown in FIG.
Normally, when this rolling shutter is used, image distortion occurs due to a shift in exposure time between the A to F lines.

  FIG. 6 is a diagram illustrating an example of a control waveform of an image when it is assumed that lines A to F exist in the element as in FIG. 2 when the global shutter is used.

In this case, when the global shutter is used, the exposure time is not shifted, so that the image is not distorted. However, since exposure ends simultaneously, it is necessary to start exposure for all pixels simultaneously. In addition, compared to the rolling shutter, the image transfer time is generated all at once, so a long time is required.
Therefore, the advantage of the rolling shutter is that the capturing speed is faster than the global shutter and the mechanical shutter. The disadvantage is that it causes image distortion between lines.
In contrast, the advantage of the global shutter (or mechanical shutter) is that data that does not cause image distortion can be obtained, while the capturing speed is slower than that of the rolling shutter.
If these advantages and disadvantages are predicted in advance as a camera system, depending on the camera situation (for example, user settings), image distortion is acceptable but speed priority is absolutely necessary. Even if it is an image, all images are captured by a rolling shutter. When priority is given to image quality, a global shutter is used for the necessary image data.
The same applies to a case where the same number of pixels is handled in a series of operations, not limited to a system in which different numbers of pixels are extracted from the image sensor.

In the imaging apparatus 1, an optical image of a subject is incident on an image sensor 11 via a lens optical system 10, and is photoelectrically converted by the image sensor 11 to become an electrical signal. The obtained electrical signal is removed of noise components by the CDS 12, digitized by the A / D converter 13, and then temporarily stored in an image memory built in the image processing unit 14.
In a normal state, an image signal is constantly overwritten at a constant frame rate in the image memory built in the image processing unit 14 by the control of the signal processing system by the timing generator 17. The image signal of the image memory built in the image processing unit 14 is converted into an analog signal by the D / A converter 18, and a corresponding image is displayed on the display unit 19.

  The display unit 19 also serves as a finder for the imaging apparatus 1. After the user presses (operates) the shutter button included in the operation input unit 20, the control unit 16 holds the image signal immediately after the shutter button is pressed to the timing generator 17, that is, image processing. The signal processing system is controlled so that the image signal is not overwritten in the image memory of the unit 14. Then, the image data held in the image memory of the image processing unit 14 is compressed by a predetermined method and recorded in the memory 15.

  Next, characteristic processing executed in the image processing unit 14 will be described.

An image that can be handled as a moving image (a relatively low number of pixels) and an image that can be handled as a still image in a system that takes out the number of different pixels from the image sensor in a series of gain scaling or averaging control operations when capturing images with different numbers of pixels In the case of handling (relatively high number of pixels) on the same stream, in the case of low pixel capture, the output pixel consists of integrated pixel data of at least one pixel of the same color pixel. In the case of high pixel capture, the number of pixels is less than the number of integrated pixels at the time of low pixel capture.
The output of one of the output pixels is naturally different due to the difference in the number of original integrated pixels. If it is determined that the outputs of the different one pixel and the same pixel are the same, in the image processing, images having particularly different luminances are continuously generated, resulting in a strange image.
In order to solve this problem, the image processing unit 14 is provided with a digital gain circuit (or an analog gain circuit) corresponding to the number of pixels or a circuit that averages the number of pixels at the pixel output portion. A circuit that corrects the occurrence of a difference in the image output of pixels and pixels is provided.

  FIG. 7 is a conceptual diagram of the image sensor. In FIG. 7, RGB indicates the three primary colors red, blue, and green.

In the sensor diagram as shown in FIG. 7, when capturing a still image (high pixel), pixels corresponding to almost all pixels are captured. On the other hand, in the case of a moving image (low pixels), since the number of pixels can be small, all pixels need not be captured, and the minimum necessary pixels are captured.
Normally, unnecessary pixels are eliminated by simple thinning, but when obtaining the quality of a moving image or the like, the surrounding R is integrated and extracted as shown in FIG.
At that time, the output pixel data differs between the case of the still image (high pixel) and the case of the moving image (low pixel).
In order to correct this, at the time of high pixel and low pixel, a coefficient corresponding to the ratio of the integrated pixel is applied to the output data for correction, or the integrated pixel data is corrected according to the number of integrated pixels. The output level for still images and moving images is corrected by averaging with.
FIG. 8 shows a conceptual diagram of correction processing according to the present embodiment.

S / N information control at the time of high pixels In a system in which a different number of pixels is extracted from the image sensor, an image to be extracted as a moving image (relatively low number of pixels) and an image that can be handled as a still image (relatively high pixels) In the case of low pixel capture, the output pixel consists of integrated pixel data of at least one pixel of the same color pixel, and in the case of high pixel capture, the low pixel The number of pixels is less than the number of integrated pixels at capture. In the case of low pixel capture, the single pixel data is generated from the integration of a plurality of pixels. Compared with that at the time of short pixel capture, the output is increased by the number of integrals. Compared with N, the information is excellent in S / N.
In the present embodiment, the integrated data (S / N information of information at the time of moving images is used to improve the image quality of short pixel data (still image information) that is considered to be relatively noisy.

The output of one of the output pixels is naturally different due to the difference in the number of original integrated pixels. If it is determined that the outputs of the different one pixel and the same pixel are the same, in the image processing, images having particularly different luminances are continuously generated, resulting in a strange image.
In order to solve this problem, as described above, a digital gain circuit (or an analog gain circuit) corresponding to the number of pixels or a circuit that averages the number of pixels is provided in the portion where the pixels are output. A circuit for correcting occurrence of a difference in the image output of one pixel is provided.

Referring to FIG. 7, FIG. 7 shows a diagram integrating four pixels.
That is, as compared with a single pixel, (R) output therefrom is an image output having an output about four times. This is advantageous for about two stages in terms of ISO sensitivity. This (R) information is always used during low pixel output.
In contrast, at the time of high pixel output, information of single pixel output information rl to r4 is used.
In order to adjust the level output from the image sensor 11, the single pixels rl to r4 are corrected by increasing the gain by analog or digital on the same stream. This gain increase causes a serious deterioration with respect to S / N.
If the original is corrected, the (R) information is the integration information of rl to r4, and can be inferred from the (R) information with less noise.
On the other hand, the single pixel rn finally becomes Rn whose gain has been increased.

That is, the sum (R1 + R2 + R3 + R4) of the data obtained by increasing the gain of each single pixel must be logically equal to (R), which is the sum of the original (r1 + r2 + r3 + r4). However, the following relationship is generally established due to an increase in noise such as gain increase.

r1 + r2 + r3 + r4 >> (R)

Therefore, when there is a component (R1 + R2 + R3 + R4) that matches the moving image information (R) from the address of a single pixel, the gain multiplied by each is calculated backward, and the content returned to the original component (r1 + r2 + r3 + r4) And (R), which should consist of the integral,
r1 + r2 + r3 + r4 >> (R)
When the relationship is established, it is determined that the excess is noise, and (r1−n1) + (r2−n2) + (r3−n3) + (r4−n4) + (r4−n4) (1)
Correction is performed to reduce N1 to N4 corresponding to n1 to n4.

At this time, n1 to n4 may be constant values, may be output ratios of rl to r4, or may be mixed. It depends on what kind of noise is dominant in the noise component in the camera system.
In addition, since there is a calculation error (quantization error or the like), the right side of equation (1) is not (R) fixed but (R) ± x.
At this time, x is the number of error corrections caused by errors such as rounding in calculation.

Method of generating still image quality even during playback at the time of video timing In a series of operations that take out a different number of pixels from the image sensor, an image that can be extracted as a video subject (a relatively low number of pixels) and an image that can be handled as a still image ( A relatively high number of pixels) is handled and controlled on the same stream, and even when it has only video image quality (low pixel count) at the time of playback, the image quality is determined based on information predicted from surrounding still image information. Is restored as a still image.

FIG. 9 shows a stream including a moving image (low pixel) and a still image (high pixel) mixed.
Here, the difference between the high pixel (still image) image data In and the low pixel (moving image) image data bn is that, in the output image information corresponding to one pixel, In is information of one corresponding pixel, and bn is a pixel to be integrated. There is a difference in information.
In addition, since In can be integrated digitally, information at the time of In substantially includes still image information and moving image information.
Here, from the moving image information bn including information predicted from I, object movement information is calculated by integrating the integrated information.
In addition, the content ratio information of each integrated single pixel is obtained from In.
Accordingly, for example, b5 still image generation having only information on the number of moving image pixels can be restored to the pixel component here by multiplying by the component ratio of each integrated single pixel expected from the fluctuation of I information. . An image of still image quality can be created by generating an image from the individual single pixel information. For the above-described reason, an image between b and b can also be created from the object movement information of bn and the content ratio information of In.

As described above, according to the imaging apparatus 1 of the present embodiment, the image processing unit 14 operates the operator (user) in one stream in which high-resolution images are mixed at regular intervals in a low-resolution moving image. ) When a pause command is issued during stream playback by operating the operation input unit 20, if the one-frame image at that time has a high resolution, the high-resolution image is converted to a low-resolution one. If there is, a high-resolution image is generated from the previous and subsequent images and displayed on the display unit 19, so that a high-definition image can be displayed even when paused during video playback.
In addition, the image processing unit 14 pauses during stream playback by operating the operation input unit 20 of an operator (user) in one stream in which high-resolution images are mixed at regular intervals in a low-resolution moving image. When a frame feed command or a slow playback command is issued after the command is issued, a high-resolution image is sequentially generated and displayed on the display unit 19. High-definition images can be sequentially displayed as if high-resolution images were continuously shot.

Further, according to the present embodiment, during moving image shooting in the signal processing system, the first compressed image is generated by intra-frame compression of high pixel image data, and the period before and after the period for generating the first compressed image Then, when the second compressed image is generated by inter-frame compression of the low pixel image data and one screen of the second compressed image is designated, the second compressed image and the first compressed image before and after the second compressed image are included. High-definition still image data representing one designated screen is generated by decompression decoding with another compressed image.
The imaging apparatus 1 according to the present embodiment uses the rolling shutter function and the global shutter function together during capturing of single moving image data, so that continuous image capturing is performed at high speed while suppressing the occurrence of image distortion. There are advantages that can be made.
In addition, the imaging apparatus 1 according to the present embodiment can correct the image level of the first compressed image and the image level of the second compressed image at substantially the same level, and continuously capture images having different image luminances. Therefore, there is an advantage that can be prevented from being generated.
In addition, the imaging apparatus 1 according to the present embodiment, when the operator determines that there is no need to print out what was captured in the still image moving image mixed mode, a high-resolution still image from the data file is a low-resolution image. Therefore, the file size can be reduced by forming a simple moving image file, the operator can execute it by key operation, and the remaining capacity of the memory 15 can be given a margin. In addition, since only the high-resolution information of the still image is cut, there is an advantage that there is no problem that the image quality is deteriorated as the moving image is reproduced.

It is a block diagram showing one embodiment of an imaging device concerning this embodiment. It is a figure for demonstrating the operation | movement at the time of the stream data change mode in this embodiment. 6 is a flowchart for explaining an operation in a playback mode in the present embodiment. It is a figure which shows an example of the image pick-up element (image sensor) at the time of rolling shutter use. FIG. 5 is a diagram illustrating an example of a control waveform of an image when it is assumed that lines A to F exist in the element as illustrated in FIG. 4. FIG. 5 is a diagram illustrating an example of a control waveform of an image when it is assumed that lines A to F exist in the element as in FIG. 4 when the global shutter is used. It is a conceptual diagram of an image sensor. It is a figure which shows the conceptual diagram of the correction process which concerns on this embodiment. It is a conceptual diagram on the stream of a moving image (low pixel) and still image (high pixel) mixed.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Imaging device 10 ... Lens optical system 11 ... Image sensor 12 ... CDS
DESCRIPTION OF SYMBOLS 13 ... A / D converter 14 ... Image processing part 15 ... Memory 16 ... Control part 17 ... Timing generator 18 ... D / A converter 19 ... Display part 20 ... Operation input part

Claims (8)

An imaging device that forms an optical image of a subject on an imaging device, reads high-resolution high-image data or low-resolution low-image data from the imaging device, and performs predetermined image processing on the image data,
A series of images including a first compressed image obtained by intra-frame compressing the high image data and a second compressed image obtained by interframe compressing the low image data in a period before and after the period for generating the first compressed image. A video generation means for generating a low-resolution video from the video stream;
Still image generation means for generating a high-resolution still image from the first compressed image or the first compressed image and at least the second compressed image of the moving image stream;
Mode switching means capable of switching the playback mode to at least one of a video playback mode and at least a frame advance mode, a still image mode, a slow playback mode, and an intermittent playback mode;
An imaging apparatus comprising: switching means for switching between the moving image generating means and the still image generating means in accordance with the mode switched by the mode switching means. If a one-frame image when receiving a pause command during playback of the video stream is a low-resolution video image generated from the first compressed image, a high-resolution video generated from the first compressed image If the image is a low-resolution moving image generated from the second compressed image and the first compressed image, a high-resolution image is generated from the second compressed image and the first compressed image. The imaging apparatus according to claim 1, further comprising image processing means for displaying the image. After receiving a pause command during playback of the video stream, at least when receiving a command to switch to the frame advance mode, slow playback mode, or intermittent playback mode, high-resolution images are sequentially generated and displayed. Item 3. The imaging device according to Item 2. The high image data is image data read out from the image sensor without being thinned out or image data read out by thinning out at an arbitrary thinning amount, and the low image data has a lower resolution than the high image data from the image sensor. The imaging apparatus according to any one of claims 1 to 3, wherein the image is a thinned-out and read-out image with an arbitrary thinning amount. The global shutter function and rolling shutter function are included as shutter functions.
The image processing means generates a first compressed image from pixel data captured by the global shutter, and generates a second compressed image from pixel data captured by the rolling shutter. The imaging device according to any one of the above. When the image processing means designates one screen of the second compressed image, the image of the second compressed image is decompressed and decoded by another image including the first compressed image before and after the designated image. The imaging device according to claim 1, wherein high-resolution still image data indicating one screen is generated. Stream output means for outputting a low-resolution continuous video stream using a high-resolution first compressed image and a low-resolution second compressed image;
Determining means for determining whether the first compressed image has a high resolution or a low resolution;
The imaging apparatus according to claim 1, further comprising: a discrimination signal output unit that outputs a signal indicating whether the first compressed image has a high resolution or a low resolution by the discrimination unit. An image generation method of an imaging apparatus that forms an optical image of a subject on an imaging device, reads high-resolution high-image data or low-resolution low-image data from the imaging device, and performs predetermined image processing on the image data. There,
Generating a first compressed image by intra-frame compressing the high image data;
Generating a second compressed image obtained by inter-frame compressing low image data in a period before and after the period for generating the first compressed image;
A video generation step of forming a series of video streams including the first compressed image and the second compressed image, and generating a low-resolution video from the video stream;
A still image generation step of generating a high-resolution still image from the first compressed image or the first compressed image and at least the second compressed image of the video stream;
A mode switching step for switching the playback mode to at least one of a video playback mode and at least a frame advance mode, a still image mode, a slow playback mode, and an intermittent playback mode;
An image generation method for an imaging apparatus, comprising: a switching step for switching processing of the moving image generation step and the still image generation step according to the mode switched by the mode switching step.

Images