JP2015154354A - Imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To compose a moving image in real time without a high-speed processing circuit.SOLUTION: An imaging apparatus comprises: imaging means; processing means that composes a plurality of frames from moving image signals outputted by the imaging means, and generates an image signal in one frame; recording means that records the moving image signal including the image signal generated by the processing means into recording media; and control means that controls the imaging means. The control means controls the imaging means so that the second frame with the number of pixels smaller than the first frame into the first frame in a predetermined period. The processing means detects a positional deviation amount between two images of the second frame from the moving signal outputted from the imaging means, and generates the image signal of one frame by composing a plurality of first frame images on the basis of the detected positional deviation amount.

Description

  The present invention relates to an imaging apparatus.

  In recent years, proposals have been made to generate a single image from a plurality of images to correct camera shake, expand a dynamic range, and improve S / N (see, for example, Patent Document 1). In the case of a still image, post-shooting post-processing may be processed over time. Therefore, many implementation means using post-processing using software processing have been proposed. Similarly, for a moving image, it is conceivable to generate one frame from a plurality of frames.

JP2009-111173A

  However, when a moving image is once recorded and then combined at the time of reproduction, it is necessary to record a large amount of data, and as a result, the recording capacity increases. Therefore, for example, in an imaging device such as a digital camera, it is possible to avoid an increase in the capacity of recorded data by performing image composition and recording when shooting a moving image.

  However, when performing image synthesis at the time of recording, it is necessary to process a large amount of data in real time, and a processing circuit is required. For this reason, there are problems such as an increase in the size of the apparatus due to an increase in circuit scale or an increase in power consumption.

  An object of the present invention is to solve such problems and to provide an apparatus capable of synthesizing a moving image in real time without requiring a high-speed processing circuit.

  The imaging apparatus of the present invention is generated by an imaging unit, a processing unit that synthesizes a plurality of frames of the moving image signal output from the imaging unit, and generates an image signal of one frame. A recording unit configured to record a moving image signal including an image signal on a recording medium; and a control unit configured to control the imaging unit, wherein the control unit includes the first frame at a predetermined period between the first frames. The imaging means is controlled to output a second frame having a smaller number of pixels than the two, and the processing means is an image of two of the second frames of the moving image signal output from the imaging means. And a plurality of images of the first frame are synthesized based on the detected amount of positional deviation to generate an image signal of one frame.

  According to the present invention, it is possible to synthesize a moving image in real time without requiring a high-speed processing circuit.

It is a block diagram which shows the structure of the imaging device in embodiment of this invention. It is a figure explaining an image composition process. It is a figure explaining an image composition process. It is explanatory drawing of a main frame and an auxiliary | assistant frame. It is explanatory drawing of a main frame and an auxiliary | assistant frame.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram illustrating a configuration of an imaging apparatus according to an embodiment of the present invention. In FIG. 1, a control unit 101 includes a CPU, a memory, and the like, and controls each unit of the imaging apparatus. The operation unit 107 includes various switches, and the user can give an instruction to the imaging apparatus by operating the operation unit 107. The control unit 101 controls each unit in response to an operation instruction from the operation unit 107. The imaging unit 102 includes a lens, an imaging device, a signal processing circuit, and the like. In this embodiment, a CMOS sensor is used as the image sensor. The imaging unit 102 captures a subject image and outputs a moving image signal. The imaging unit 102 can output a video signal of 240 frames / second (fps), for example, one screen (one frame) is horizontal 1920 pixels × vertical 1080 pixels.

  The moving image signal from the imaging unit 102 is stored in the memory 103 via the bus 109. The image synthesizing unit 104 synthesizes image signals of a plurality of screens stored in the memory 103 to generate one frame of image signal. Therefore, the image composition unit 104 reads a necessary number of image signals from the memory 103, and detects a positional shift between the images of each frame from the plurality of read frames. Then, based on the detected shift amount, image synthesis is performed on the target frame image, and the result is recorded in the memory 103.

  The recording / playback unit 105 reads out a moving image signal including the synthesized image signal from the memory 103, performs encoding / compression processing, and the like, and then records it on the recording medium 108. For example, a flash memory card or the like is used as the recording medium 108. The recording / playback unit 105 reads the moving image signal recorded on the recording medium 108, performs signal processing such as decoding processing, and outputs playback data. The output unit 106 outputs the image synthesized by the image synthesis unit 104 or the reproduced image to the outside.

  Next, an image composition process when recording a moving image signal will be described. FIG. 2 is a diagram for explaining each frame of a plurality of frames of moving image signals output from the imaging unit 102 and an image combining process using the moving image signals. In this embodiment, the frame rate of the moving image signal recorded on the recording medium 108 is 60 fps. In addition, the frame rate of the moving image signal output from the imaging unit 102 is 240 fps as described above. The imaging unit 102 images and outputs a synthesis source frame (main frame (first frame)) at an interval of 1/120 seconds. The main frame corresponds to main frames 1, 2, 3, and 4 in FIG. Further, the number of pixels of the main frame is 1920 horizontal pixels × 1080 vertical pixels.

  On the other hand, auxiliary frames (second frames) 1, 2, 3, and 4 in FIG. 2 are frames output from the imaging unit 102 while each of the main frames 1, 2, 3, and 4 is imaged. For example, in the case of FIG. 2, the time interval between the main frame and the auxiliary frame is 1/240 seconds. The control unit 101 controls the shutter speed of the auxiliary frame separately from the main frame shooting. Here, shutter control refers to electronic shutter control of a CMOS sensor, for example. For the auxiliary frame, the shutter speed is set relatively higher than that of the main frame in order to obtain an image with less camera shake.

  In the present embodiment, the amount of displacement for image composition is detected using this auxiliary frame. Then, based on the detected shift amount, a correction value for the position shift at the time of image composition of the main frame is determined. A known method is used as means for detecting the positional deviation of the frame image. For example, a block matching method or a template matching method is used. For example, paying attention to a certain limited area or feature point in the frame image, calculating the vector deviation amount of the feature point between frames, and correcting the displacement when synthesizing the image using the deviation amount Do. Alternatively, the misregistration information is obtained so that the correlation between the frames is high so that the misregistration error information between the frames is minimized over the entire image instead of a certain limited area. Furthermore, both may be used.

  In this way, image synthesis is performed between the main frames based on the shift amount detected using the auxiliary frame. In the case of FIG. 2, alignment of the main frames 1 and 2 ((2) alignment information) is performed based on the positional deviation information obtained by the auxiliary frames 1 and 2 ((1) alignment parameter search). Then, the composite frame 1 is generated from the main frames 1 and 2 by performing an image composition process ((3) a plurality of sheets composition process). Further, the composite frame 1 is recorded on the recording medium 108.

  The control unit 101 controls the accumulation time of the subject light so that the main frame has proper exposure. On the other hand, the auxiliary frame is not recorded on the recording medium 108 but is used only for detecting misregistration information necessary for image composition. For this reason, the control unit 101 captures images with an accumulation time suitable for detecting misregistration information, for example, with a shutter time necessary to reduce image blur due to camera shake, independently of the exposure amount of the main frame. The imaging unit 102 is controlled.

  That is, the shutter speed at the time of imaging is changed between the main frame and the auxiliary frame. As a result, the image to be recorded (main frame) is imaged with appropriate exposure, and the image (auxiliary frame) for calculating the synthesis condition (position shift information) is imaged under the condition for suppressing image blur. Further, in the alignment parameter search, the search may be performed by paying attention to a certain limited area or feature point in the frame image. Or you may search so that the error information between frames may become the minimum over the whole image. A supplementary explanation will be given regarding the case where the search area is limited to some limited areas.

  In this case, shooting is performed so that the angle of view of the auxiliary frame is ¼ the size of the main frame. The number of pixels in each auxiliary frame is 960 pixels wide × 540 pixels high. That is, the image size is such that a quarter area of the main frame is cut out. The control unit 101 controls the imaging unit 102 so as to output an image signal having such a size for the auxiliary frame.

  This is because an image having a large size corresponding to the main frame is not required when searching in a limited area. However, in this case, necessary feature points and the like need to be included in the auxiliary frame. That is, how to select the imaging area of the auxiliary frame is important.

  FIG. 3 shows an example of misalignment detection using an auxiliary frame in consideration of the above points. In FIG. 3, an area containing many feature points (singular points and edges) is detected from the image data of the main frame 2 by signal processing ((4) feature point detection area search). And based on the information, the imaging area at the time of imaging of the auxiliary frames 3 and 4 to be subsequently imaged on the time axis is controlled ((5) Control of imaging area).

  FIG. 4 shows an example of the imaging area of the auxiliary frame. In the main frame, for example, when an area including many edges is set as the imaging area of the auxiliary frame, in this case, it is determined that the area including the building (house) includes many edges, and the auxiliary frame example 1 In addition, this area is set as an imaging area of the auxiliary frame. Alternatively, for example, when an area including many feature points in the main frame is set as an imaging area of the auxiliary frame, an area including a human face is determined as an area of the auxiliary frame using a known technique such as face detection. Then, as in auxiliary frame example 2, the area is set as an imaging area of the auxiliary frame.

  This can be realized by the control device 101 reading the image signal of the main frame from the memory 103, determining the imaging area of the auxiliary frame from the read image signal, and controlling the imaging unit 102.

  Here, the case where the search area is limited to a part of the limited region has been described. Next, a supplementary description will be given regarding the case where the search is performed so that error information between frames is minimized over the entire image. FIG. 5 shows an example of a captured image of the auxiliary frame. In this case, for example, the angle of view of the auxiliary frame is equal to that of the main frame, and the number of pixels in one frame is 960 pixels × 540 pixels. That is, the image is a reduced image having a resolution of 1/4 of the main frame. Using this reduced image, the positional deviation information is obtained so that the error between the auxiliary frames is minimized, and the image of the main frame is synthesized.

  In this case, the control unit 101 controls the imaging unit 102 and outputs an auxiliary frame image signal. For example, thinning out of horizontal lines at the time of pixel readout from a CMOS sensor, synthesis by simultaneous readout of pixels, and synthesis by analog signal processing at the subsequent stage are performed. Alternatively, a reduced auxiliary frame image is generated by the signal processing unit in the imaging unit 102.

  As described above, in the present embodiment, when a plurality of frames of the captured moving image signal are combined and the combined moving image signal is recorded, a position shift is detected between the recorded moving image frames. Shoot the frame. The size of the frame for detecting displacement is set to be smaller than the recorded moving image signal. For this reason, the processing load for detecting the displacement is reduced, and image synthesis can be performed in real time without using a high-speed processing circuit.

  In this embodiment, the main frame and the auxiliary frame are alternately photographed one by one. However, the present invention is not limited to this. That is, it is not necessary to shoot alternately every frame. What is necessary is just to make it output an auxiliary | assistant frame with a predetermined period between main frames. For example, one auxiliary frame may be captured for a plurality of main frames. Alternatively, one or more auxiliary frames may be taken between different main frames.

  For example, when two auxiliary frames are captured for a plurality of main frames, the first image captures an image corresponding to a reduced image, and the second image captures an image corresponding to a cutout. Using the image corresponding to the first reduced image, the cutout area of the image corresponding to the cutout imaged later in time may be controlled.

  In addition, as an example of the auxiliary frame, an example of an image obtained by cutting out a quarter area of the main frame and an image having a resolution of 1/4 of the main frame has been described. Also good. Further, although the number of pixels of the auxiliary frame is set to ¼ of the main frame, it is not limited to this.

  Further, in this embodiment, the description has been made on the premise of moving image shooting, but in recent years also in still image shooting, there is a tendency to require high frame rate imaging such as high-speed continuous shooting. It is applicable not only to moving images but also to still image shooting.

Claims (7)

Imaging means;
Processing means for synthesizing a plurality of frames of the moving image signal output from the imaging means to generate an image signal of one frame;
Recording means for recording a moving image signal including the image signal generated by the processing means on a recording medium;
Control means for controlling the imaging means,
The control means controls the imaging means to output a second frame having a smaller number of pixels than the first frame in a predetermined cycle during the first frame,
The processing means detects a positional deviation amount between two images of the second frame in the video signal output from the imaging means, and based on the detected positional deviation amount, An image pickup apparatus that generates an image signal of one frame by synthesizing the image of the first frame.   The image pickup apparatus according to claim 1, wherein the control unit controls the shutter speed independently for the first frame and the second frame.   The imaging apparatus according to claim 2, wherein the control unit controls the shutter speed of the second frame to be higher than that of the first frame.   The control means controls the imaging means so as to output an image signal of a predetermined size obtained by cutting out a partial area of the first frame as the second frame. The imaging apparatus according to 1.   2. The control unit according to claim 1, wherein the control unit controls the imaging unit to output an image signal having a predetermined size obtained by reducing the size of the first frame as the second frame. Imaging device.   The imaging apparatus according to claim 1, wherein the control unit controls the imaging unit so that the first frame and the second frame are alternately output every other frame.   The imaging apparatus according to claim 1, wherein the control unit controls the imaging unit to output the second frame every time a plurality of the first frames are output.

Images