JP2008147715A - Imaging apparatus and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To acquire a still picture of high resolution without interrupting moving picture recording when the still picture is taken during moving picture photography. <P>SOLUTION: The imaging apparatus comprises an imaging element (103) which has a plurality of pixels, converts the incident optical image of an object into an electric signal, and outputs image data, a camera shake correcting unit (126) which shifts the position of incidence of the optical image on the imaging element, a switch (110) for instructing still picture photography during the moving picture photography, control means (118 and 105) of controlling the camera shake correcting unit according to the instruction so that the incidence position of the optical image is shifted to a different position from each other by ≤1 pixel while making a plurality of frames one cycle, a superresolution processing circuit (116) which combines image data of a plurality of frames obtained in one cycle together with one image according to the instruction so that pixel density increases, and an electronic vibration-proof circuit for correcting the image data of the respective frames so that the shifting of incidence position of the optical image by the control means is canceled. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an imaging apparatus and method, and more particularly to an imaging apparatus and method capable of recording a still image during moving image recording.

  In recent years, there have been an increasing number of digital cameras having a function capable of shooting moving images. Among such digital cameras, digital cameras capable of shooting still images during moving image shooting have been commercialized. In such a camera, in general, when there is a scene that the user wants to save as a still image while shooting a movie, the release switch for shooting the still image is pressed, and the movie shooting is temporarily suspended in response to this operation. Take a still image, and after shooting the still image, return to movie shooting.

  In the case of a moving image, the frame rate is determined, and the number of pixels in one frame is, for example, 640x480 equivalent to the number of pixels in VGA, whereas in the still image, it is not restricted by the frame rate, and several million An image with the number of pixels is desired. For this reason, a method is disclosed in which all pixels of the image sensor are read when a still image is taken, and the number of pixels is reduced to 1 / N when reading a moving image (see, for example, Patent Document 1). By controlling in this way, a high-definition still image can be read out, and an image with the necessary number of pixels can be read out at a necessary frame rate during moving image shooting.

  On the other hand, when recording a moving image, the electronic image stabilization process is performed in units of one pixel or more to record the moving image. From the recorded moving image having a camera shake component of less than one pixel, the camera shake of less than one pixel is recorded. A method of creating a still image by super-resolution processing using components has been proposed. (For example, refer to Patent Document 2.) This super-resolution processing is a technique for creating high-resolution image data from low-resolution image data using a plurality of shifts of less than one pixel. Details are described in FIG.

JP 2000-78486 A Japanese Patent Application No. 2005-244852 Shin Aoki, “Super-Resolution Processing Using Multiple Digital Image Data”, Ricoh Technical Report No. 24, November 1998, p. 19-25

  However, in the digital camera described in Patent Document 1, since the number of pixels read from the image sensor is different between moving image shooting and still image shooting, it is necessary to change the driving method of the image sensor, and the drive control is complicated. there were. In addition, at the time of still image shooting, the drive speed of the image sensor is made faster than at the time of moving image shooting, or a still image is shot over a plurality of frames.

  In addition, the parameters used for image data signal processing differ between still image shooting and movie shooting, for example, autofocus values, exposure control values, and other parameters. Therefore, when switching to still image shooting during movie shooting, signal processing is performed. It is also necessary to switch the mode.

  Further, regarding the display, in the conventional imaging device described above, since it is necessary to switch the driving method and signal processing mode of reading of the imaging element when capturing a still image during moving image capturing, display of moving images has been interrupted. Alternatively, an unnatural display such as a fixed color display is performed to hide the screen during the interruption period.

  Furthermore, since still images cannot be recorded while still images are being recorded, the recorded still images or fixed color images were interrupted during the still image recording period. Measures such as embedding in the period were necessary. As described above, there is a problem that recording of moving images is interrupted and continuous moving images cannot be recorded.

  In contrast, the imaging apparatus described in Patent Document 2 can generate continuous moving images and high-resolution still images. However, a desired still image can be obtained while reproducing the recorded moving image, but a still image cannot be acquired during moving image shooting. In addition, for example, when shooting is performed with an imaging device installed on a tripod or the like, a still image cannot be generated by super-resolution processing when a moving image having less than one pixel is not included in the moving image.

  The present invention has been made in view of the above problems, and an object of the present invention is to enable acquisition of a high-resolution still image without interrupting recording of a moving image when shooting a still image during moving image shooting. To do.

  In order to achieve the above object, an imaging apparatus of the present invention has a plurality of pixels, an imaging means for converting an optical image of an incident subject into an electrical signal and outputting image data, and the imaging means for the optical image Shift means for shifting the incident position on the upper side, instruction means for instructing recording of a still image during moving image recording, and the optical at different positions by less than one pixel with a plurality of frames as one cycle in response to an instruction from the instruction means A control unit that controls the shift unit so as to shift an incident position of an image, and a plurality of frames of image data obtained in one cycle in accordance with an instruction from the instruction unit, so that the pixel density is improved. The image data of each frame is corrected so as to cancel the deviation of the incident position of the optical image by the control unit in accordance with an instruction from the combining unit that combines the image and the instruction unit. And a positive means.

  The imaging method of the present invention includes a plurality of pixels, an imaging unit that converts an optical image of an incident subject into an electrical signal and outputs image data, and an incident position of the optical image on the imaging unit. An imaging method in an imaging apparatus having a shift means for shifting, wherein an instruction step for instructing recording of a still image during moving image recording differs from each other by less than one pixel with a plurality of frames as one cycle according to an instruction by the instruction step The shift step for controlling the shift means to shift the incident position of the optical image to the position, and the image data of the plurality of frames obtained in one cycle according to the instruction by the instruction means so that the pixel density is improved. Each frame is combined so as to cancel the shift of the incident position of the optical image due to the shift step in accordance with an instruction from the instruction means. And a correction step of correcting the image data of.

  According to the present invention, when a still image is shot during moving image shooting, a high-resolution still image can be acquired without interrupting moving image recording.

  The best mode for carrying out the present invention will be described below in detail with reference to the accompanying drawings.

<First Embodiment>
System Configuration FIG. 1 is a block diagram illustrating an example of a system configuration of an imaging apparatus that has both a moving image recording function and a still image recording function according to the first embodiment and is capable of recording a still image during moving image recording. is there. In the present embodiment, the process of recording a still image at the same time as recording a moving image is called “simultaneous recording”.

  Reference numeral 100 denotes an optical system for performing imaging, which includes an imaging lens 101, a camera shake correction unit 126, a shutter 102, and an imaging element 103 such as a CCD or CMOS sensor. The camera shake correction unit 126 is a means for shifting the imaging position of the subject image on the imaging surface of the image sensor 103, such as a variable apex angle prism. Reference numeral 104 denotes an A / D conversion circuit for converting data captured by the optical system 100 into digital data. Reference numeral 109 denotes a CPU that controls this system. Reference numeral 110 denotes a switch including a switch for setting an imaging mode from the user, a switch for instructing start / stop of moving image recording, and a release switch for instructing recording of a still image. 123 is an external memory, 124 is a display device such as a liquid crystal display, and 125 is a medium for finally recording image data.

  An optical camera shake correction system 127 performs optical camera shake correction, and includes an actuator 105, an adder circuit 106, a camera shake sensor 107, and a camera shake correction control circuit 108. The actuator 105 drives the camera shake correction unit 126. The camera shake sensor 107 detects angular velocities in the pitch direction and the yaw direction. The camera shake correction control circuit 108 integrates the angular velocities in the pitch direction and the yaw direction detected by the camera shake sensor 107 to calculate an angle shift amount. Then, from the calculated angular displacement amount, that is, the shake angle of the imaging device and the focal length of the imaging lens 101, the amount of pixel movement on the image sensor 103 caused by camera shake is calculated, and the calculated amount of pixel movement is canceled. A camera shake correction amount is generated and output to the adder circuit 106.

  Note that the camera shake correction control circuit 108 can determine the moving image shooting mode and the still image shooting mode instructed by the switch 110 from the CPU 109, and controls the amount of camera shake correction during moving image shooting and still image shooting. be able to. For example, at the time of moving image shooting, there are many cases where shooting is performed by largely moving the imaging device, such as panning or tilting. For this reason, the amount of camera shake tends to include many low-frequency components. On the other hand, at the time of shooting a still image, since the camera is often fixed and shot, there is a tendency that a low frequency component is not included as compared with a moving image. From the above, the camera shake correction control circuit 108 can perform cutoff switching of frequency components controlled during moving image shooting and still image shooting with respect to the detection amount detected by the camera shake sensor 107.

  The adder circuit 106 is an adder circuit that adds a pixel shift amount generated by a pixel shift control circuit 118 of the signal processing system 111 described later to the camera shake correction amount output from the camera shake correction control circuit 108.

  Reference numeral 111 denotes a signal processing system, which is configured by an LSI (Large Scale Integration) in which digital circuits are integrated. The configuration of the first embodiment is divided into a signal processing system for processing moving images and a signal processing system for processing still images.

  First, a signal processing system for processing moving images will be described.

  Reference numeral 113 denotes an electronic image stabilization circuit. The electronic image stabilization circuit 113 is a circuit that operates at the time of simultaneous recording, and is a circuit that corrects a camera shake component of less than one pixel included in moving image data. Reference numeral 112 denotes a line memory that holds image data for one line so that the electronic image stabilization circuit 113 performs camera shake correction processing for less than one pixel. The processing of the electronic image stabilization circuit 113 will be described later in detail with reference to FIGS.

  Reference numeral 114 denotes a moving image signal processing circuit that converts image data into luminance color difference signals by performing YC separation, white balance correction, aperture correction, gamma correction, and the like on the image data output from the electronic image stabilization circuit 113. is there. YC separation refers to separation of a luminance signal and a color difference signal with high accuracy. Aperture correction refers to correcting a low-pass effect (which is the occurrence of a fine pattern blur, also referred to as an aperture effect) caused by an aperture (aperture) of the image sensor 103. By this aperture correction, the change in the signal value is emphasized. γ correction refers to non-linear image processing assuming an output device. By this γ correction, a more natural image can be obtained. As described above, the moving image signal processing circuit 114 converts the image data output from the electronic image stabilization circuit 113 into a luminance color difference signal.

  Next, a signal processing system for processing a still image will be described.

  Reference numeral 116 denotes a super-resolution processing circuit that performs super-resolution processing, and reference numeral 115 denotes a frame memory that holds a frame image for performing super-resolution processing by the super-resolution processing circuit 116. The processing performed by the super-resolution processing circuit 116 will be described in detail later with reference to FIG. Reference numeral 117 denotes a signal processing circuit that performs signal processing on still image data generated by super-resolution processing. Since the processing content is almost the same as that of the moving image signal processing circuit 114, the description is omitted here.

  Reference numeral 118 denotes a pixel shift control circuit, which switches a pixel shift amount of less than one pixel for each frame of a moving image during simultaneous recording and outputs it to the adder circuit 106, the electronic image stabilization circuit 113, and the super-resolution processing circuit 116.

  Reference numeral 119 denotes a compression / decompression circuit, which compresses still image data generated by the still image signal processing circuit 117 into, for example, the JPEG format, or generates image data by expanding the compressed data. . The compression / decompression circuit 119 compresses the moving image data generated by the moving image signal processing circuit 114 into, for example, an MPEG format, or expands the compressed data to generate image data.

  Reference numeral 123 is an external memory, and 120 is a memory controller for accessing the external memory 123. Reference numeral 124 denotes a display device, and 121 denotes a display device control circuit for controlling the display device 124. Reference numeral 125 denotes a removable or built-in medium in the image pickup apparatus, and reference numeral 122 denotes a media I / F circuit for controlling the medium 125. As the medium 125, a memory card such as a PCMCIA card or compact flash (registered trademark), a hard disk, or the like can be used. Further, it may be constituted by a micro DAT, a magneto-optical disk, an optical disk such as a CD-R or CD-WR, a phase change optical disk such as a DVD, etc., and the present invention is not particularly concerned with the form.

Moving Image Recording Next, the moving image recording process of the first embodiment in the imaging apparatus having the configuration shown in FIG. 1 will be described.

  At the time of moving image recording, the CPU 109 controls each of the optical system 100, the optical camera shake correction system 127, and the signal processing system 111 in the moving image shooting mode. At the time of moving image recording, since the super-resolution processing of the still image signal processing system described later is not performed, the pixel shift control circuit 118 outputs a pixel shift amount of 0 under the control of the CPU 109. The pixel shift amount will be described later in detail. Therefore, the camera shake correction amount detected by the camera shake correction control circuit 108 based on the angular velocity detected by the camera shake sensor 107 is directly input from the adder circuit 106 to the actuator 105, and the camera shake correction unit 126 performs camera shake correction. At this time, as described above, the CPU 109 performs control so that correction suitable for moving image shooting is performed, such as setting the cutoff frequency of the camera shake correction control circuit 108 to a value suitable for a moving image. The image sensor 103 is driven so as to read out image data of a predetermined number of pixels at a predetermined frame rate.

  The subject optical image subjected to camera shake correction by the camera shake correction unit 126 is formed on the image sensor 103, converted into an electric signal, and analog image data having a predetermined number of pixels is sequentially read out at a predetermined frame rate. The read analog image data is converted into digital image data by the A / D conversion circuit 104 and then input to the electronic image stabilization circuit 113. As described above, the electronic image stabilization circuit 113 is a circuit that operates at the time of simultaneous recording. At the time of moving image recording, the input signal is output to the moving image signal processing circuit 114 as it is and processed to generate a luminance color difference signal.

  The image data converted into the luminance color difference signal by the moving image signal processing circuit 114 is held in the external memory 123. The image data which is the luminance color difference signal held in the external memory 123 is output again to the compression / decompression circuit 119 via the memory controller 120 and converted into an appropriate format such as MPEG4. Then, it is stored again in the external memory 123 via the memory controller 120. The compressed image data recorded in the external memory 123 is further output to the media I / F circuit 122 via the memory controller 120 and finally recorded on the medium 125.

  The image data output from the moving image signal processing circuit 114 held in the external memory 123 is input to the display device control circuit 121 via the memory controller 120. Here, after resizing processing and appropriate format conversion for display on the display device 124 are performed to suit the display size, the data is output to the display device 124.

Simultaneous Recording Next, the simultaneous recording processing in the first embodiment will be described.

  During the simultaneous recording process, both the moving image signal processing system and the still image signal processing system in the signal processing system 111 operate. Hereinafter, electronic image stabilization processing performed by the electronic image stabilization circuit 113 for moving image recording during simultaneous recording and super-resolution processing performed by the super-resolution processing circuit 116 for still image recording will be described. First, the super-resolution process will be described, and then the electronic image stabilization process will be described.

<Super-resolution processing and still image recording>
A super-resolution process performed by the super-resolution processing circuit 116 for recording a still image will be described with reference to FIG.

  The super-resolution processing is processing for generating high-resolution image data from a plurality of image data having a shift of less than one pixel. Here, for example, an example will be described in which super-resolution is performed from an image shifted by ½ pixel in another direction using four pieces of image data.

  Frame 0 in FIG. 2A is reference image data. Image data shifted by 1/2 pixel in the horizontal direction with respect to the reference image is frame 1 (FIG. 2B), and image data shifted by 1/2 pixel in the vertical direction is frame 3 (FIG. 2D). Image data shifted by 1/2 pixel in each of the horizontal and vertical directions is referred to as frame 2 (FIG. 2C). First, the pixel data of (n, m) of frame 1 is embedded at the position of (n + 0.5, m) that is each pixel position where the pixel data of frame 0 does not exist. Similarly, the pixel data of (n, m) of frame 3 is placed at the position of (n, m + 0.5) of frame 0, and (n, m) of frame 2 is placed at the position of (n + 0.5, m + 0.5) of frame 0. ) Is embedded.

  That is, by embedding the pixel data of frames 1 to 3 in a grid pattern at a position shifted by 1/2 pixel of frame 0, a pixel density four times that of frame 0 as shown in FIG. It is possible to obtain still image data.

  Super-resolution processing is processing in which pixel data is increased to increase the resolution by performing synthesis processing as described above.

  The pixel shift control circuit 118 generates a pixel shift amount that deviates less than one pixel from the reference frame so as to enable the above-described super-resolution processing, and outputs the pixel shift amount to the adder circuit 106. In the example shown in FIG. 2, with 4 frames as one cycle, the pixel shift amount is (0, 0) in the first frame, (0, 0.5) in the second frame, and (0.5, 0) in the third frame. .5) The fourth frame is (0.5, 0).

  In the above example, a case where still image data having a pixel density of 4 times is generated using image data for four frames has been described, but the present invention is not limited to this. The present invention only needs to achieve a pixel density higher than the number of pixels of the image sensor 103 using image data for a plurality of frames, and pixel shift control is performed so that an image shifted by less than one pixel necessary for that purpose can be obtained. The circuit 118 controls the pixel shift amount. For example, it is possible to shift 1/3 pixel at a time to make 9 frames one cycle, or shift 1/4 to make 16 frames a cycle. However, if it takes too much time to acquire image data for the number of frames for one cycle, the subject may move (subject shift), and the quality of the synthesized still image may deteriorate. It is preferable to configure one period with the number of frames that do not.

  During simultaneous recording, the CPU 109 controls the optical system 100 in the moving image shooting mode, controls the optical camera shake correction system 127 in the still image shooting mode, and performs both signal processing for moving images and still images in the signal processing system 111. To control. As described above, during simultaneous recording, the pixel shift control circuit 118 outputs a pixel shift amount of less than one pixel that is different for each frame under the control of the CPU 109. Therefore, the addition circuit 106 adds a pixel shift amount output from the pixel shift control circuit 118 to the camera shake correction amount obtained by the camera shake correction control circuit 108 based on the angular velocity detected by the camera shake sensor 107. Is output.

  As described above, by adding the camera shake correction amount and the pixel shift amount by the adding circuit 106, the image data read from the image sensor 103 is the reference image data for super-resolution processing or 1 from which the camera shake component is removed. The image data is shifted by less than a pixel.

  The correction value output from the adder circuit 106 is input to the actuator 105, and optical camera shake correction is performed by the camera shake correction unit 126. At this time, as described above, the CPU 109 performs control so that correction suitable for moving image shooting is performed, such as setting the cutoff frequency of the camera shake correction control circuit 108 to a value suitable for moving images. The image sensor 103 is driven in a moving image shooting mode in which image data of a predetermined number of pixels is read at a predetermined frame rate.

  The image data that has undergone super-resolution processing by the super-resolution processing circuit 116 is converted into a luminance color difference signal by the still image signal processing circuit 117. The image data converted into the luminance / color difference signal is recorded in the external memory 123 via the memory controller 120. Image data, which is a luminance color difference signal recorded in the external memory 123, is output again to the compression / decompression circuit 119 via the memory controller 120 and converted into an appropriate compressed moving image format such as JPEG. Then, it is recorded again in the external memory 123 via the memory controller 120. The compressed image data recorded in the external memory 123 is further output to the media I / F circuit 122 via the memory controller 120 and finally recorded on the media 125.

<Electronic image stabilization and video recording>
Next, an electronic image stabilization process performed by the electronic image stabilization circuit 113 for recording a moving image will be described with reference to FIGS.

  As described above, the image data input to the electronic image stabilization circuit 113 at the time of simultaneous recording is image data from which a camera shake component is removed and which has a different pixel shift amount for each frame. This pixel shift amount is output from the pixel shift control circuit 118 to the adder circuit 106 and also to the electronic image stabilization circuit 113. The electronic image stabilization circuit 113 uses this pixel shift amount to generate image data. Interpolation processing of less than one pixel is performed.

  FIG. 3 is a diagram illustrating an example of a specific configuration of the electronic image stabilization circuit 113. First, pixel interpolation processing for less than one pixel in the vertical direction will be described. In FIG. 3, a multiplier 202 multiplies input image data by a coefficient Vk. The input image data is simultaneously held in the line memory 112 and delayed by one line. The multiplier 201 multiplies (1-Vk) on the image data delayed from the line read out from the line memory 112 by one line. An adder 203 adds the outputs of the multiplier 201 and the multiplier 202. That is, the output of the adder 203 is a weighted average of image data between adjacent lines with the weight of the coefficient Vk.

  Next, image interpolation processing for less than one pixel in the horizontal direction will be described. The multiplier 204 multiplies the image data output from the adder 203 by a coefficient Hk. The image data output from the adder 203 is simultaneously sent to the 1-clock delay circuit 205 and delayed by one pixel. The multiplier 206 multiplies the image data delayed by one pixel output from the one-clock delay circuit 205 by a coefficient (1-Hk). Adder 207 adds the outputs from multiplier 204 and multiplier 206. That is, the output of the adder 207 is a weighted average of pixel values adjacent in the horizontal direction with the weight of the coefficient Hk.

  Next, with reference to FIG. 4, an example in which interpolation processing is specifically performed for a shift of less than one pixel will be described. Here, a case will be described in which the image data input to the electronic image stabilization circuit 113 is deviated by 1/2 pixel in the vertical downward direction and 1/2 pixel in the horizontal right direction with respect to the reference image data.

  First, an interpolation data generation method using image data shifted by 1/2 pixel in the vertical direction will be described. FIG. 4A shows a process of generating image data shifted by 1/2 pixel in the vertically downward direction from the image data 301 of n lines and the image data 302 of n + 1 lines. In this case, the n-line image data 301 is halved to generate image data 303, the n + 1-line image data 302 is halved to generate image data 304, and each is added. Pixel data shifted by 1/2 pixel in the vertical downward direction can be obtained. This corresponds to the case where Vk is 1/2 in FIG.

  Next, an interpolation data generation method using image data shifted by 1/2 pixel in the horizontal right direction will be described. FIG. 4B shows a process of generating image data shifted by 1/2 pixel in the horizontal right direction from the image data 311 of n pixels and the image data 312 of n + 1 pixels. In this case, the image data 311 of n pixels is halved to generate image data 313, the image data 312 of n + 1 pixels is halved to generate an image 314, and each is added to obtain horizontal data. Pixel data shifted by ½ pixel in the right direction can be obtained. This corresponds to the case where Hk is 1/2 in FIG.

  As described with reference to FIG. 2, when processing a frame image with pixel shift amounts of (0, 0), (0, 0.5), (0.5, 0.5), (0.5, 0), The electronic image stabilization circuit 113 may control Vk and Hk as shown in the following table based on these pixel shift amounts.

  In this way, the camera shake component is removed by the camera shake correction unit 126, and the pixel shift amount of less than one pixel for the super-resolution processing is corrected by the above processing of the electronic image stabilization circuit 113. Obtainable.

  Image data that has undergone interpolation processing of less than one pixel by the electronic image stabilization circuit 113 is converted into a luminance / color difference signal by the moving image signal processing circuit 114 and held in the external memory 123 via the memory controller 120. The image data which is the luminance color difference signal recorded in the external memory 123 is output again to the compression / decompression circuit 119 via the memory controller 120 and converted into an appropriate format such as MPEG4. Then, it is stored again in the external memory 123 via the memory controller 120. The compressed image data recorded in the external memory 123 is further output to the media I / F circuit 122 via the memory controller 120 and finally recorded on the media 125.

Flow of Simultaneous Recording Operation Next, the flow of the simultaneous recording operation in the first embodiment will be described with reference to the flowchart of FIG. The process shown in the flowchart of FIG. 5 is started when the power of the imaging apparatus is turned on by the switch 110 of FIG.

  In step S <b> 11, the CPU 109 determines whether or not the user gives an instruction to start moving image recording by using the switch 110. When an instruction to start moving image recording is issued, the process proceeds to step S12, where the CPU 109 sets the above-described moving image shooting mode for the optical system 100, the optical camera shake correction system 127, and the signal processing system 111. In step S13, moving image recording is started. When moving image recording is started, moving image data is acquired by the above-described moving image recording process and recorded on the medium 125 as needed. In step S14, it is determined whether the end of the moving image is instructed by the user during moving image shooting or whether the moving image shooting is completed from the status of the imaging apparatus. If the result of determination is that movie shooting is to be terminated, the movie recording process is terminated.

  On the other hand, when the moving image shooting is continuing (NO in step S14), the process proceeds to step S15, and it is determined whether or not the release switch for still image shooting among the switches 110 is pressed by the user. When the release switch for still image shooting is pressed, the above-described simultaneous recording process is performed. In starting the simultaneous recording process, in step S16, the CPU 109 resets the optical system 100, the optical camera shake correction system 127, and the signal processing system 111 for simultaneous recording. Specifically, for example, the cutoff frequency for optical camera shake correction of the camera shake correction control circuit 108 of the optical camera shake correction system 127 is set in the same manner as in the case of still image shooting. In the signal processing system 111, the optical camera shake correction system 127 is controlled by the pixel shift control circuit 118 so as to perform, for example, the above-described pixel shift with the four frames as one cycle. That is, a reference frame without pixel shifting, a frame 1 shifted by 1/2 pixel in the horizontal right direction, a frame 2 shifted by 1/2 pixel in the vertical downward direction, 1/2 pixel in the horizontal right, and a vertical downward direction A pixel shift amount for obtaining frame 3 shifted by 1/2 pixel is output.

  In step S17, the CPU 109 controls the optical system 100, the optical camera shake correction system 127, and the signal processing system 111 so as to start simultaneous recording. Note that the driving method (frame rate, number of readout pixels) of the image sensor 103 is the same as that during moving image shooting, and the driving method is not switched during simultaneous recording.

  When simultaneous recording is started, optical image stabilization is performed, and image data having a shift of less than one pixel (for example, 1/2 pixel) is transferred from the image sensor 103 via the A / D conversion circuit 104. Input to the signal processing system 111. In the signal processing system, since the above-mentioned electronic stick vibration processing for moving images and the super-resolution processing for still images are performed in parallel, it is possible to simultaneously generate moving images and still images. Since the generated moving image is recorded on the medium 125 in the same manner as the moving image recording, the moving image recording can be continued even during the simultaneous recording.

  In step S18, it is determined whether or not the still image signal processing has been completed. When a still image can be generated by the super-resolution processing described above, in step S19, the moving image shooting mode is restored, and moving image recording is resumed in step S20.

  In step S21, an image is displayed on the display device 124. However, even if an image of an arbitrary frame used for the super-resolution processing is displayed or the still image generated in step S18 is displayed as a review, Or you may continue displaying a moving image as it is. For example, in the above-described still image generation method, a still image having a quadruple resolution is created using four frame images. However, if it is desired to output the still image at the time of simultaneous recording to the display device 124, 4 Any one of the images may be displayed as a review on the display device 124 as a still image. Alternatively, a review image may be created from a still image having a resolution four times that has undergone super-resolution processing.

  In step S22, the generated still image is recorded on the medium 125. When the recording of the still image ends, the process returns to step S14, and the above processing is repeated until the moving image recording ends.

  As described above, according to the first embodiment, there is no need to change the driving method of the image sensor 103 when shooting of a still image is instructed during moving image shooting. In addition, a high-resolution still image can be acquired by super-resolution processing.

  When only a still image is recorded, it may be acquired by the super-resolution processing described above, or if the resolution of the image sensor 103 is originally high, a still image may be acquired by all-pixel readout driving. Good.

<Second Embodiment>
Next, a second embodiment of the present invention will be described. FIG. 6 is a block diagram illustrating an example of a system configuration of an imaging apparatus having a moving image recording and still image recording function according to the second embodiment. In FIG. 6, the same components as those in FIG. 1 are denoted by the same reference numerals, and components different from those in FIG. 1 will be described below.

  In the first embodiment described above, the pixel shift amount is a known value generated by the pixel shift control circuit 118. In the electronic image stabilization circuit 113 and the super-resolution processing circuit 116, the pixel shift that is this known value. The amount was used for processing. In contrast, in the second embodiment, the pixel shift amount is used only in the optical camera shake correction system 127. In the second embodiment, the motion detection circuit 628 detects the amount of motion of the image data input to the signal processing system 611.

  Ideally, the motion detection circuit 628 should detect only the amount of pixel shift by the pixel shift control circuit 118, but the camera shake correction amount actually supplied to the camera shake correction unit 126 includes the shooting environment. There are errors due to disturbances. For this reason, perfect camera shake correction and pixel shift are not always performed. Accordingly, the motion detection circuit 628 detects the amount of motion between frames using a plurality of frames. Note that the motion amount detection method between frames is described in detail, for example, in Patent Document 2.

  The motion detection circuit 628 also detects a motion of one pixel or more. Therefore, the motion detection amount output from the motion detection circuit 628 may be a shift amount of one pixel or more, and the electronic image stabilization circuit 613 and the super-resolution processing circuit 616 may correct the shift of one pixel or more. It is necessary to do. Therefore, the electronic image stabilization circuit 613 and the super-resolution processing circuit 616 according to the second embodiment perform the processing based on the pixel shift amount of less than one pixel described in the first embodiment at the time of simultaneous recording. A process of correcting a shift beyond the pixel is also performed.

  As a method for correcting a pixel shift of one pixel or more, for example, a method disclosed in Patent Document 2 may be used. In this method, the shift is corrected by shifting the range in which the image is cut out from the input frame so that the pixel shift amount is canceled.

  In the above embodiment, the case where the optical image stabilization unit 126 shifts the imaging position of the subject image on the imaging surface of the image sensor 103 has been described. However, the shifting method is not limited to this. Absent. For example, the image sensor 103 may be driven and shifted.

<Other embodiments>
Note that the present invention may be applied to a system constituted by a plurality of devices (for example, a host computer, an interface device, a camera head, etc.) or an apparatus constituted by a single device.

  The object of the present invention can also be achieved as follows. First, a storage medium (or recording medium) that records a program code of software that implements the functions of the above-described embodiments is supplied to a system or apparatus. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also the following can be achieved. That is, when the operating system (OS) running on the computer performs part or all of the actual processing based on the instruction of the read program code, the functions of the above-described embodiments are realized by the processing. It is. Examples of the storage medium for storing the program code include a flexible disk, hard disk, ROM, RAM, magnetic tape, nonvolatile memory card, CD-ROM, CD-R, DVD, optical disk, magneto-optical disk, MO, and the like. Can be considered. Also, a computer network such as a LAN (Local Area Network) or a WAN (Wide Area Network) can be used to supply the program code.

1 is a block diagram showing a schematic configuration of a system according to a first embodiment of the present invention. It is a figure for demonstrating a super-resolution process. It is a figure which shows the structural example of the electronic vibration isolator in the 1st Embodiment of this invention. It is a figure for demonstrating the electronic image stabilization process in the 1st Embodiment of this invention. It is a flowchart explaining the control of the moving image recording and simultaneous recording in the 1st Embodiment of this invention. It is a block diagram which shows schematic structure of the system which concerns on the 2nd Embodiment of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 100: Optical system, 101: Imaging lens, 102: Shutter, 103: Image sensor, 104: A / D conversion circuit, 105: Actuator, 106: Adder circuit, 107: Camera shake sensor, 108: Camera shake correction control circuit, 109: CPU: 110: switch, 111: signal processing system, 112: line memory, 113: electronic image stabilization circuit, 114: moving image signal processing circuit, 115: frame memory, 116: super-resolution processing circuit, 117: signal processing circuit, 118: Pixel shift control circuit, 119: Compression / decompression circuit, 120: Memory controller, 121: Display device control circuit, 122: Media I / F circuit, 123: External memory, 124: Display device, 125: Media, 126: Camera shake Correction unit, 127: Optical image stabilization system

Claims (12)

Imaging means having a plurality of pixels, converting an optical image of an incident subject into an electrical signal, and outputting image data;
Shift means for shifting the incident position of the optical image on the imaging means;
Instruction means for instructing still image recording during moving image recording;
Control means for controlling the shift means so as to shift the incident position of the optical image to positions different from each other by less than one pixel in a cycle of a plurality of frames in response to an instruction from the instruction means;
Combining means for combining the image data of the plurality of frames obtained in one cycle into a single image so as to improve pixel density in response to an instruction by the instruction means;
An image pickup apparatus comprising: a correction unit that corrects image data of each frame so as to cancel a shift of an incident position of the optical image by the control unit in response to an instruction from the instruction unit. The control means includes
Shift amount output means for outputting a shift amount of less than one pixel;
Camera shake detection means for detecting camera shake and obtaining a camera shake correction amount;
The imaging apparatus according to claim 1, further comprising: a calculation unit that calculates a control amount for controlling the shift unit based on the shift amount and the camera shake correction amount.   The imaging apparatus according to claim 2, wherein the synthesizing unit and the correcting unit perform the synthesizing and the correction based on a shift amount output by the shift amount output unit. From the image data for a plurality of frames output from the imaging means, further comprising a detection means for detecting the amount of motion of the image,
The imaging apparatus according to claim 1, wherein the combining unit and the correcting unit perform the combining and correction based on the amount of motion obtained by the detecting unit. It further has a display means,
When shooting a still image while recording the moving image, the moving image is continuously displayed on the display unit, an image of an arbitrary frame is displayed from a plurality of frames for creating a still image, or synthesized by the synthesizing unit. The imaging apparatus according to claim 1, wherein an image is displayed. In an imaging apparatus having a plurality of pixels, an imaging unit that converts an optical image of an incident subject into an electrical signal and outputs image data, and a shift unit that shifts an incident position of the optical image on the imaging unit An imaging method,
An instruction process for instructing still image recording during video recording;
A shift step of controlling the shift means to shift the incident position of the optical image to a position different from each other by less than one pixel in a cycle of a plurality of frames according to an instruction by the instruction step;
In accordance with an instruction in the instruction step, a combining step of combining the image data of the plurality of frames obtained in one cycle into a single image so as to improve pixel density;
And a correction step of correcting the image data of each frame so as to cancel the shift of the incident position of the optical image due to the shift step in response to an instruction from the instruction means. In the shifting step,
A shift amount output step for outputting a shift amount of less than one pixel;
A camera shake detection process for detecting camera shake and obtaining a camera shake correction amount;
The imaging method according to claim 6, further comprising a calculation step of calculating a control amount for controlling the shift unit based on the shift amount and the camera shake correction amount.   The imaging method according to claim 7, wherein in the combining step and the correcting step, the combining and correction are performed based on the shift amount output in the shift amount output step. From the image data for a plurality of frames output from the imaging means, further comprising a detection step of detecting the amount of motion of the image,
The imaging method according to claim 6 or 7, wherein in the synthesis step and the correction step, the synthesis and correction are performed based on the amount of motion obtained in the detection step.   When shooting a still image while recording the moving image, the moving image is continuously displayed on the display means, an image of an arbitrary frame is displayed from a plurality of frames for creating a still image, or synthesized in the synthesis step. The imaging method according to claim 6, further comprising a display step of displaying an image.   The program for making a computer perform each process of the imaging method of any one of Claim 6 thru | or 10.   A computer-readable storage medium storing the program according to claim 11.

Images