JP2015119231A - Image processing apparatus and control method of the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to generate an image synthetic still image without stopping moving image photographing during the moving image photographing with an inexpensive image processing apparatus.SOLUTION: When generating an image synthetic still image during moving image photographing, a coding format of a moving image frame of the timing is changed.

Description

  The present invention relates to an image processing apparatus that can shoot still images during moving image shooting and generates moving image data by moving image shooting and still image data by still image shooting, and a control method for the image processing device.

  There have been proposed an electronic camera that can capture both moving images and still images, and an electronic camera that can capture still images while capturing moving images (Patent Document 1).

JP-A-9-284705

  However, in these electronic cameras, when taking a still image while taking a moving image, it is necessary to process the moving image and the still image at the same time. Therefore, the control of the electronic camera system is complicated and the load on the system increases. . Further, when a still image or the like is synthesized to create a more preferable still image, the tendency becomes remarkable, and when the still image processing becomes complicated, there is a problem that the moving image capturing process is stopped.

  In view of the above-described problems, the present invention provides an image processing apparatus and image processing capable of generating an image synthesized still image without stopping a moving image being shot even when creating a still image obtained by combining images in a moving image. An object is to provide a method for controlling an apparatus.

In order to solve the above-described problems, an image processing apparatus according to the present invention predicts and encodes moving image data obtained by moving image shooting between a first encoding method that predicts and encodes within a frame. A video compression means for compressing the second encoding method in a video compression format periodically mixed according to a predetermined rule;
Instruction means for instructing to shoot a still image during video recording;
A determination means for determining whether to perform still image shooting by image synthesis based on a predetermined condition when shooting a still image;
Image synthesizing means for synthesizing images from the taken continuous images to create one or more still images;
Still image compression means for compressing the created still image in a still image compression format,
When a still image shooting instruction is received during movie shooting, a plurality of frame images shot before and after the frame image compressed by the first encoding method in the vicinity of the timing of receiving the instruction are combined. And a composing means for generating a still image.

  Even in the case of creating a still image obtained by combining images in a moving image, it is possible to generate an image combined still image without stopping the moving image being shot.

It is a block diagram of the image processing apparatus in a present Example. It is a conceptual diagram of moving image frame extraction and image composite still image creation. It is a figure which shows the mode of the encoding of each flame | frame. It is a figure explaining the encoding object block in a present Example. FIG. 3 is a diagram illustrating a sequence of the first embodiment. It is a figure showing the correspondence of the timing of the conventional imaging | photography instruction | indication, and an encoding system. It is a figure showing a response | compatibility with the timing of the imaging | photography instruction | indication in a present Example, and an encoding system. FIG. 10 is a diagram illustrating a sequence of the second embodiment.

  The image processing apparatus according to the present embodiment will be described using a digital camera as an example.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a diagram showing a configuration of an embodiment of the present invention.

  In FIG. 1, reference numeral 100 denotes an image processing apparatus.

  10 is a photographing lens, 12 is a mechanical shutter having an aperture function, 14 is an image sensor that converts an optical image into an electrical signal, and 16 is an A / D converter that converts an analog signal output of the image sensor 14 into a digital signal. .

  A timing generation circuit 18 supplies a clock signal and a control signal to the image sensor 14 and the A / D converter 16, and is controlled by the memory control circuit 22 and the system control circuit 50. In addition to the twelve mechanical shutters, the accumulation time can be controlled as an electronic shutter by controlling the reset timing of the eighteen image sensors, and can be used for moving image shooting and the like.

  An image processing circuit 20 performs predetermined pixel interpolation processing and color conversion processing on the data from the A / D converter 16 or the data from the memory control circuit 22. Further, an electronic zoom function is realized by performing image cutting and scaling processing by 20 image processing circuits.

  Further, the image processing circuit 20 performs predetermined calculation processing using the captured image data, and the system control circuit 50 controls the exposure control means 40 and the distance measurement control means 42 based on the obtained calculation result. TTL AF processing, AE processing, and EF processing are performed. The image processing circuit 20 also performs a high dynamic range synthesis (HDR synthesis) process, which is an image synthesis process for expanding a dynamic range by synthesizing images of a plurality of frames with different exposure conditions (exposure amounts), which will be described later.

  Further, the image processing circuit 20 performs predetermined arithmetic processing using captured image data, and also performs TTL AWB (auto white balance) processing based on the obtained arithmetic result.

  A memory control circuit 22 controls the A / D converter 16, the timing generation circuit 18, the image processing circuit 20, the memory 30, and the compression / decompression circuit 32.

  The data of the A / D converter 16 is written into the memory 30 via the image processing circuit 20 and the memory control circuit 22, or the data of the A / D converter 16 is directly written via the memory control circuit 22. Reference numeral 28 denotes an image display unit composed of a TFT LCD or the like, and display image data written in the memory 20 is displayed by the image display unit 28 via the memory control circuit 22.

  If the image data captured using the image display unit 28 is sequentially displayed, the electronic viewfinder function can be realized.

  The image display unit 28 can arbitrarily turn on / off the display according to an instruction from the system control circuit 50. When the display is turned off, the power consumption of the image processing apparatus 100 can be greatly reduced. I can do it.

  Reference numeral 30 denotes a memory for storing captured still images and moving images, and has a sufficient storage capacity to store a predetermined number of still images and a predetermined time of moving images.

  This makes it possible to write a large amount of images to the memory 30 at high speed even in continuous shooting or panoramic shooting in which a plurality of still images are continuously shot.

  The memory 30 can also be used as a work area for the system control circuit 50.

  Reference numeral 31 denotes a non-volatile memory composed of a flash ROM or the like. The program code executed by the system control circuit 50 is written in the nonvolatile memory 31, and the program code is executed while being read sequentially. In addition, an area for storing system information and an area for storing user setting information are provided in the nonvolatile memory, and various information and settings are read and restored at the next startup.

  Reference numeral 32 denotes a compression / decompression circuit that compresses and decompresses image data by adaptive discrete cosine transform (ADCT) or the like, reads an image stored in the memory 30, performs compression processing or decompression processing, and stores the processed data in the memory 30. Write to.

  Reference numeral 40 denotes an exposure control means for controlling the shutter 12 having a diaphragm function, and has a flash light control function in cooperation with the flash 48.

  Reference numeral 42 denotes distance measurement control means for controlling the focusing of the photographing lens 10, and reference numeral 44 denotes zoom control means for controlling the zooming of the photographing lens 10.

A flash 48 has an AF auxiliary light projecting function and a flash light control function.
The exposure control means 40 and the distance measurement control means 42 are controlled using the TTL method. Based on the calculation result obtained by calculating the captured image data by the image processing circuit 20, the system control circuit 50 performs the exposure control means 40 and the distance measurement. Control is performed on the control means 42.

  A system control circuit 50 controls the entire image processing apparatus 100.

  Reference numerals 60, 62, 64, 66, 70, and 72 denote operation means for inputting various operation instructions of the system control circuit 50. A single unit such as a switch, a dial, a touch panel, pointing by line-of-sight detection, a voice recognition device, or the like Consists of multiple combinations.

  Here, a specific description of these operating means will be given.

  Reference numeral 60 denotes a mode dial switch, which can be set to switch various function modes such as power off, automatic shooting mode, shooting mode, panoramic shooting mode, moving image shooting mode, playback mode, multi-screen playback / erase mode, and PC connection mode. .

  Reference numeral 62 denotes a shutter switch SW1, which is turned on during the operation of the shutter button, and instructs to start operations such as AF (autofocus) processing, AE (automatic exposure) processing, and AWB (auto white balance) processing.

  A shutter switch SW2 64 is turned on when the operation of the shutter button is completed. In response to an operation input to the shutter switch SW2 64, the system control circuit 50 instructs the start of a series of processing operations such as exposure processing, development processing, and recording processing.

  Reference numeral 66 denotes a display changeover switch, which can change the display of the image display unit 28.

  With this function, when photographing is performed using the optical viewfinder 104, it is possible to save power by cutting off the current supply to the image display unit including a TFT LCD or the like.

  An operation unit 70 includes various buttons, a touch panel, and the like, and includes a menu button, a set button, a macro button, a flash setting button, a single shooting / continuous shooting / self timer switching button, and the like. There are also a menu movement + (plus) button, menu movement-(minus) button, shooting image quality selection button, exposure correction button, date / time setting button, and the like.

  Reference numeral 72 denotes a zoom switch unit as zoom operation means for a user to give an instruction to change the magnification of a captured image. Hereinafter, the zoom switch 72 is also referred to. The zoom switch 72 includes a tele switch that changes the imaging field angle to the telephoto side and a wide switch that changes the imaging angle of view to the wide angle side. By using the zoom switch 72, the zoom control unit 44 is instructed to change the imaging field angle of the photographing lens 10 and becomes a trigger for performing an optical zoom operation. In addition, it also serves as a trigger for electronic change of the imaging angle of view by image cutting by the image processing circuit 20 or pixel interpolation processing.

  Reference numeral 86 denotes a power supply means including a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like.

  Reference numeral 90 denotes an interface with a recording medium such as a memory card or hard disk, and reference numeral 92 denotes a connector for connecting to a recording medium such as a memory card or hard disk.

  Reference numeral 102 denotes protection means that is a barrier that prevents the imaging unit from being soiled or damaged by covering the imaging unit including the lens 10 of the image processing apparatus 100.

  Reference numeral 104 denotes an optical viewfinder, which can take an image using only the optical viewfinder without using the electronic viewfinder function of the image display unit 28.

  A communication unit 110 has various communication functions such as USB and wireless communication.

  Reference numeral 112 denotes a connector for connecting the image processing apparatus 100 to another device by the communication unit 110 or an antenna in the case of wireless communication.

Reference numeral 200 denotes a recording medium such as a memory card or a hard disk.
The recording medium 200 includes a recording unit 202 composed of a semiconductor memory, a magnetic disk, or the like, an interface 204 with the image processing apparatus 100, and a connector 206 for connecting to the image processing apparatus 100.

  Reference numeral 210 denotes a recording medium such as a memory card or a hard disk.

  Next, the prior art will be described using moving image data in a moving image as an example of a control method. The image sensor 14 is generally driven at the same frame rate as the moving image frame rate to extract a moving image frame image. The extracted moving image frame image is encoded based on the MPEG standard, for example. In the MPEG technology, a predicted frame image is created based on frames at different timings, and the difference between the input frame image and the predicted frame image is encoded. This is called inter-picture predictive coding. In addition, when encoding an image without performing inter-image predictive encoding, this is called intra-screen predictive encoding. This process will be described with reference to FIG. For example, it is assumed that “frame 1”, “frame 2”, and “frame 3” are read as the frames for moving images.

  The frame of “frame 1” is encoded by dividing one image into blocks without referring to the preceding and following images (intra-frame predictive encoding method). In the intra-frame predictive coding method, data used for coding is closed in the image. Also, the encoding target block of the intra-frame prediction encoding method is generally referred to in order to predict the upper, upper left, upper right, and left blocks shown in FIG. In this respect, in the intra prediction encoding, Read / Write is generated around the encoding target block.

  On the other hand, “frame 2” “frame 3”, and inter-frame prediction is used by utilizing the pixel correlation with the previous frame (inter-frame prediction coding method). This encoding encodes an inter-frame difference between “frame 2” and “frame 3”. At this time, the encoding target frame is encoded with reference to the front frame. That is, “frame 2” encodes the difference with reference to “frame 1”, and “frame 3” encodes the difference with reference to “frame 2”. Therefore, the intra-frame predictive coding scheme requires many memory accesses with respect to the intra-frame predictive coding scheme.

  In accordance with a predetermined rule stipulated in MPEG, which is a moving image compression format, a frame is periodically mixed with a single intra-frame predictive encoded frame and a plurality of inter-frame predictive encoded frames to form one group. . This is called GOP. In general MPEG video shooting, a GOP is combined into one moving image. When a still image is shot during moving image shooting, one of the moving image frames is extracted and image processing for the still image is performed. When the image size of the still image is larger than the image resolution of the moving image, processing such as enlarging the image size by image processing is performed. When a still image is actually shot during moving image shooting, a still image may be created by combining a plurality of frames, such as when the object scene is dark or when the contrast is high. In such a case, if a still image is synthesized by combining a plurality of frames while performing inter-picture predictive coding, there is a problem that memory access increases and the system cannot be stably operated.

  Based on the above premise, an example suitable for the present invention according to the present embodiment will be described with reference to FIG. A moving image shooting mode is selected by the user, and a mode in which an image composite still image is shot during the moving image is selected (S500).

  In this embodiment, in order to create an image composite still image, particularly an image composite still image with an expanded dynamic range during moving image shooting, a plurality of moving image frames with different exposure conditions are used as a still image composite image. This will be described as an example. When the user gives an instruction to start shooting by means such as the shutter switch SW64, shooting of a moving image is started (S501).

  The moving image is captured by the imaging sensor 14 through the lens 10 and sent to the image processing block 20. In the image processing block, it is determined by a counter (not shown) whether the encoding is performed by the intra-frame prediction encoding method or the inter-frame prediction encoding, and the moving image frame is encoded according to the method.

  Thereafter, when the user gives an instruction to shoot a still image during moving image shooting (S503), the timing is close to the head of the GOP of the currently encoded moving image or the next GOP to be encoded from now on. It is determined whether it is close to the head of (S505). Next, in the case where the moving image frame for capturing the image combined still image determined in S505 is the intra prediction encoding method in the encoding method determined from the above-described counter, the image combined still image is added to the setting required for the moving image frame. Necessary settings such as sensitivity and exposure time are set in the sensor to expose and read. Similarly, with respect to a moving image frame necessary for synthesizing a still image, a setting necessary for synthesizing the image is similarly performed and an image is read from the sensor. In this way, the necessary settings are made for the number of still images to be combined, and the image is read from the sensor as a moving image frame. In the present invention, the above-described method is a method of acquiring a still image synthesizing moving image frame by using a moving image frame that encodes the beginning of a moving image frame for synthesizing an image synthesizing still image by using the intra prediction encoding method. However, the moving image frame of the intra prediction encoding method may be used as a key, and the preceding moving image frame may be used as an image for image synthesis still image. The correspondence between the timing of the shooting instruction and the encoding method will be further described with reference to FIGS. 6 and 7 separately. FIG. 6 shows a flow of a moving image frame when an image composite still image is captured in a conventional moving image. In-frame predictive coding and inter-picture predictive coding are coded according to a predetermined rule for each GOP. When the user gives an instruction to shoot a still image, the next three images are processed for image synthesis still image to create one image synthesis still image. In this case, the processing for the image synthesis still image is further performed at the timing when the memory access frequently occurs, so that the load on the image processing apparatus increases. Next, FIG. 7 is a flow of a moving image frame when an image composite still image is captured in a moving image according to the present invention. As in the conventional method shown in FIG. 6, intra-frame prediction encoding and inter-picture prediction encoding are encoded according to a predetermined rule for each GOP. When the user gives an instruction to shoot a still image, a moving image frame to be encoded using the in-screen predictive encoding method in the vicinity, and a moving image frame from two frames after the still image shooting instruction in FIG. Used as a moving image frame for an image synthesis still image. In this way, since the video frame encoded by the intra prediction encoding method is always included in the frame for creating the image composite still image, the frequency of memory access can be reduced compared to the conventional method. Can do.

  The read image is stored separately in a separate memory area for image synthesis still image while being used as a moving image frame. When shooting still images for image composition while shooting moving images in normal high-contrast scenes, movie frames are generally shot with different exposures. Therefore, different processing such as exposure correction is performed to make one frame of the actual moving image.

  If the result of S505 is close to the head of the GOP currently being encoded, a moving image frame previously stored in another memory area is selected to create an image composite still image (S506). If the current GOP is closer to the beginning of the next GOP than the currently encoded GOP, the moving image frame previously stored in another memory area is discarded and encoded using the intra-frame predictive encoding method that should come at the beginning of the next GOP. An image composite still image is created using the moving image frame (S507). In this example, an example is described in which an image synthesis still image is shot from a moving image frame when an instruction to shoot a still image is given after the previously shot moving image frame is written in the memory area. Then, a still image for image synthesis can be taken out using a moving image frame to be encoded by the next intra-frame predictive encoding method as a key frame. Furthermore, it is more preferable to dynamically change the position of the moving image frame to be encoded by the intra-frame predictive encoding method according to the shooting timing.

  By controlling the image processing apparatus in this way, it is possible to obtain a high-quality still image without interrupting the moving image while distributing the system load of the image processing apparatus.

  Still another embodiment will be described with reference to FIG. As in the previous case, the user selects a moving image shooting mode (S800), and selects a mode in which an image combined still image is shot during the moving image. Also in this embodiment, in order to create an image composite still image, particularly an image composite still image with an expanded dynamic range during moving image shooting, a plurality of moving image frames with different exposure conditions are used as a still image composite image. This will be described as an example. When the user instructs to start shooting by means such as the shutter switch SW64, shooting of a moving image is started (S801). The moving image is captured by the imaging sensor 14 through the lens 10 and sent to the image processing block 20. In step S802, it is determined whether or not the moving image shooting is completed, and the following steps are performed until the moving image shooting is completed. In the image processing block, it is determined by a counter (not shown) whether the encoding is performed by the intra prediction encoding method or the inter prediction encoding, and the moving image frame is encoded according to the method.

  After that, when the user instructs to shoot a still image during moving image shooting (S803), a plurality of moving image frames used for synthesis in the image synthesized still image from the timing of the moving image currently encoded at that timing It is determined whether there is a moving image frame to be encoded by intra prediction encoding (S804). As a result, when there is no frame for predictive encoding within the screen at the timing when the user instructs to shoot a still image, the exposure condition of the movie frame is changed and the movie frame is read from the image sensor 14. After that, the video frame is stored in a separate memory area as a frame for image synthesis still image, and it is originally the timing to encode and encode between images, but the encoding method is changed to the intra prediction encoding method Change (S805, S806). By performing this operation, the frequency of memory access can be reduced and the impact on the system can be reduced by always encoding with the intra prediction encoding method at the timing of capturing a still image for image synthesis.

  Thereafter, an image composite still image is created using the moving image frame in the memory area (S807). By controlling the image processing apparatus in this way, it is possible to obtain a high-quality still image without interrupting the moving image while distributing the system load of the image processing apparatus.

DESCRIPTION OF SYMBOLS 10 Shooting lens 12 Shutter 14 Image pick-up element 16 A / D converter 18 Timing generation circuit 20 Image processing circuit 22 Memory control circuit 28 Image display part 30 Memory 32 Image compression / expansion circuit 40 Exposure control means 42 Distance control means 44 Zoom control Means 50 System control circuit 62 Shutter switch SW1
64 Shutter switch SW2

Claims (9)

A first encoding method for predicting and encoding moving image data obtained by moving image shooting within a frame and a second encoding method for predicting and encoding between frames are periodically generated according to a predetermined rule. Video compression means for compressing in a video compression format mixed in
Instruction means for instructing to shoot a still image during video recording;
A determination means for determining whether to perform still image shooting by image synthesis based on a predetermined condition when shooting a still image;
Image synthesizing means for synthesizing images from the taken continuous images to create one or more still images;
Still image compression means for compressing the created still image in a still image compression format,
When a still image shooting instruction is received during movie shooting, a plurality of frame images shot before and after the frame image compressed by the first encoding method in the vicinity of the timing of receiving the instruction are combined. And a synthesizing unit for generating a still image. A first encoding method for predicting and encoding moving image data obtained by moving image shooting within a frame and a second encoding method for predicting and encoding between frames are periodically generated according to a predetermined rule. A video compression means for compressing in a video compression format mixed in the video, an instruction means for instructing to shoot a still image during video recording,
A determination means for determining whether to perform still image shooting by image synthesis based on a predetermined condition when shooting a still image;
Image compositing means for composing images from continuously captured images to create still images and still image compression means for compressing the created still images in still image compression format. About the frame of the moving image at the timing received
Even when it should be encoded in the second encoding format,
An image processing apparatus that shoots a moving image using the first encoding format. The image processing apparatus according to claim 1, wherein a shooting condition is changed in a frame to be encoded by the first encoding method in a shooting mode for shooting a still image during moving image shooting. When performing image composition processing in shooting mode for shooting still images during movie shooting,
The image processing apparatus according to claim 3, wherein a photographing condition is changed in a frame to be encoded by the first encoding method.   In a shooting mode in which shooting of a still image can be instructed during moving image shooting, shooting conditions from a predetermined frame before the first encoding method to a first encoding method from a frame to be encoded by the first encoding method are set. The image processing apparatus according to claim 4, wherein the image processing apparatus changes corresponding to a still image. In the shooting mode that can instruct the shooting of still images during movie shooting, when there is a possibility that still image shooting will be an image composition process,
5. The image processing apparatus according to claim 4, wherein imaging conditions from a frame to be encoded by the first encoding system to a frame after a predetermined frame are changed.   7. The image processing apparatus according to claim 5, wherein the predetermined number of frames is the same as the number of still images synthesized.   When still image shooting is image synthesis processing for expanding the dynamic range, the exposure amount of the frame to be encoded by the first encoding method is made appropriate, and the exposure amount of the other frames is made different. The image processing apparatus according to claim 5 or 6.   The image processing apparatus according to claim 8, wherein the frames made different in claim 8 do not refer to frames encoded by the second encoding method.

Images