JP2009130561A - Imaging apparatus, control method of imaging apparatus and control program of imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To repair a data omission that occurs in still image data in recording a still image during moving image recording. <P>SOLUTION: For example, during storing an imaging signal in a memory in accordance with recording instructions of a still image in recording the still image during moving image recording, if a memory band can not be secured to cause a failure in data transfer, it is considered that a failure place 101 where image data are lost occurs in a still image 100. The failure place 101 of the still image 100 is restored by referring to a corresponding moving image. With reference to a frame N of a moving image whose imaging timing corresponds to that of the still image 100, data 102 of an area corresponding to the failure place 101 in the still image 100, of the frame N, is applied in a prescribed manner to the failure place 101 of the still image 100 (data 102') to restore the still image 100. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an imaging apparatus capable of recording a moving image and a still image, a control method for the imaging apparatus, and a control program for the imaging apparatus, and more particularly to a configuration for recording a still image while recording a moving image.

  In recent years, products that record still images without stopping moving image recording during moving image recording have been developed in imaging devices that perform moving image recording. In such a recording method for recording a still image during moving image recording, RAW data using the image signal output from the image sensor as it is is compression-coded in real time, temporarily stored in the memory, and later compressed from the memory. The converted RAW data is read out, decoded and decompressed, subjected to predetermined image processing on the still image, and recorded on the recording medium.

  Processing for moving image data is performed in parallel with processing for still image data. That is, the moving image data is output for each frame timing based on the timing signal output from the timing generator, is compressed and encoded in a predetermined manner, and is temporarily stored in the memory. The compressed moving image data stored in the memory is read, for example, for each recording unit of the recording medium and recorded on the recording medium.

  By the way, in still image recording, when compression encoding processing is performed on RAW data, variable length encoding may be used to increase compression efficiency. Since still image recording is performed during moving image recording, it is necessary to perform compression encoding of RAW data in real time, and variable length encoding is performed by one-pass processing. In this case, the compression encoding process of the RAW data cannot be retried, and the compression rate greatly varies depending on the design of the target still image. On the other hand, the memory is shared with the processing of moving image data as described above. For this reason, it becomes difficult to secure a memory band according to the change in the compression rate of the still image, and the processing may fail.

  Here, in the case of an imaging device that mainly processes video recording, if it is difficult to secure memory bandwidth due to still image recording and the processing is likely to fail, priority is given to video recording processing, Even if the recording process is sacrificed, the moving image recording process is never interrupted. For example, if it is difficult to secure the memory bandwidth and the processing is likely to fail, a method of stopping still image recording and attempting to record a still image again at the next frame timing can be considered. In this case, a deviation occurs between the photographing timing intended by the photographer and the timing of the recorded still image, and a photo opportunity is missed.

  For this reason, when the process is likely to fail, it is conceivable to temporarily stop the compression encoding process of the still image and wait for the memory band to have a margin and the failure to recover. If the compression process is resumed after the still image compression process is stopped, the still image data can be recorded. However, since this data is temporarily stopped from compression encoding by variable length encoding, there is a possibility that correct data cannot be obtained when it is decoded. For example, in the data compressed and encoded using a variable length code, when the variable length code is interrupted, the data from the beginning of the variable length code to the next variable length code where the head is detected Will be missing.

As described above, as a method of relieving error data at the time of decoding encoded image data, as described in Patent Document 1, a method of outputting the immediately preceding image at the time of error to moving image data, or Patent Document As described in FIG. 2, a method of interpolating image data that has been lost at the time of decoding with surrounding normal image data can be considered.
JP 2005-229395 A JP-A-8-321944

  However, in the case of still image recording, since it is assumed that the number of image data to be recorded is one, there is a problem that the method of outputting the immediately preceding image described in Patent Document 1 cannot be applied. .

  Further, in the method described in Patent Document 2, when a region to be interpolated is wide, for example, a lack of data is continuous, a normal pixel to be used for interpolation is located near the pixel to be interpolated. It can be considered that it does not exist. In this case, there is a problem that the image after the interpolation may be remarkably deteriorated.

  Accordingly, an object of the present invention is to provide an image pickup apparatus, an image pickup apparatus control method, and an image pickup apparatus that are capable of repairing missing data generated in still image data when a still image is recorded during moving image recording. It is to provide a control program.

  In order to solve the above-described problems, the present invention converts an image of light from a subject into an electrical signal in an imaging apparatus capable of recording still image data in parallel with recording of moving image data. An imaging unit that outputs a signal at a frame timing, a moving image processing unit that continuously processes an imaging signal output from the imaging unit for each frame timing as a moving image, and an imaging signal output from the imaging unit at an arbitrary timing A still image processing unit that processes as a still image at a frame timing according to a supplied imaging instruction, and an image restoration unit that restores a missing portion of the still image using a moving image frame corresponding to the frame timing of the still image This is an imaging apparatus.

  According to another aspect of the present invention, there is provided a method of controlling an imaging apparatus capable of recording still image data in parallel with recording of moving image data. A video processing step for continuously processing the imaging signal output from the camera as a video at each frame timing, and the imaging signal output from the imaging unit at a frame timing according to an imaging instruction supplied at an arbitrary timing A method for controlling an image pickup apparatus, comprising: a step of still image processing to be processed as an image; and a step of image restoration to restore a missing portion of the still image using a moving image frame whose frame timing corresponds to the still image It is.

  According to another aspect of the present invention, there is provided an imaging control program for causing a computer to execute a control method for an imaging apparatus capable of recording still image data in parallel with recording of moving image data. Steps of moving image processing in which image signals converted and output from the imaging unit at the frame timing are continuously processed as moving images for each frame timing, and imaging signals output from the imaging unit are supplied at an arbitrary timing An imaging apparatus having a still image processing step for processing as a still image at a frame timing according to an instruction, and an image restoration step for restoring a missing portion of the still image using a moving image frame corresponding to the frame timing for the still image An imaging control program that causes a computer to execute the control method.

  As described above, according to the present invention, light from a subject is converted into an electrical signal, and an imaging signal output from the imaging unit at a frame timing is continuously processed as a moving image at each frame timing and output from the imaging unit. The captured image signal is processed as a still image at a frame timing in accordance with an imaging instruction supplied at an arbitrary timing, and a missing portion of the still image is restored using a moving image frame corresponding to the frame timing of the still image Thus, when recording a still image while recording a moving image, even if a missing portion occurs in the still image, the missing portion can be restored.

  As described above, the present invention converts the light from the subject into an electrical signal and continuously processes the imaging signal output from the imaging unit at the frame timing as a moving image for each frame timing, and outputs it from the imaging unit. The captured image signal is processed as a still image at a frame timing in accordance with an imaging instruction supplied at an arbitrary timing, and a missing portion of the still image is restored using a moving image frame corresponding to the frame timing of the still image Therefore, when a still image is recorded while a moving image is being recorded, there is an effect that the lost portion can be restored even if a missing portion occurs in the still image.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The concept of the present invention will be described with reference to FIG. For example, consider a case where recording of a still image is instructed during recording of a moving image, and the still image is captured at the timing of frame N of the moving image. When storing an imaging signal in a memory in response to a still image recording instruction, the memory bandwidth cannot be secured and data transfer fails, and image data is lost in the still image 100 as illustrated in FIG. 1A. A failure location 101 has occurred. In the present invention, the failed portion 101 of the still image 100 is restored with reference to the corresponding moving image.

  As an example, as illustrated in FIG. 1B, a frame N of a moving image corresponding to a still image 100 and an imaging timing is referred to, and data of an area corresponding to the broken portion 101 in the still image 100 of this frame N is used. Then, the still image 100 is restored. For example, information indicating a position corresponding to the failure is stored at the time of recording a still image. Then, when the still image 100 is played back, the image of the frame N of the moving image corresponding to the still image 100 and the capture timing at the time of recording is referred to, and the corresponding position of the image of the frame N is determined based on the stored failure position information. Data 102 is acquired. The acquired data is applied to the failure location 101 of the still image 100 in a predetermined manner (data 102 '), and the still image 100 is restored.

  FIG. 2 shows a configuration of an example of an imaging apparatus 1 applicable to the embodiment of the present invention. In this imaging device 1, light incident through the optical system 10 is received by the imaging element 11 and converted into an electrical signal to be an imaging signal, and this imaging signal is connected to the signal processing unit 20 by the signal processing unit 20. The data is recorded on the recording medium 37 by performing predetermined processing using the RAM 35 which is an external memory. The imaging device 1 is configured to record the imaging signal as moving image data on the recording medium 37 and also to the recording medium 37 as still image data captured at a timing according to the imaging instruction. .

  In the following, for the sake of simplicity, the effective pixel count of the image sensor 11 is assumed to be 2560 horizontal pixels and 1920 vertical pixels. The moving image has horizontal 640 pixels and vertical 480 pixels reduced to ¼ in the horizontal and vertical directions, respectively, and the still image uses the effective pixels of the image sensor 11 as they are, and the image frame is horizontal 2560 pixels and vertical. It is assumed that there are 1920 pixels.

  The optical system 10 includes a lens system, a diaphragm mechanism, a focus mechanism zoom mechanism, and the like, and controls a drive unit 13 based on a command of a CPU (Central Processing Unit) 30 to be described later, a manual operation, and the like to perform diaphragm, focus, zoom, and the like. Is to be controlled. Light from the subject enters the image sensor 11 through the optical system 10. The image pickup device 11 is composed of, for example, a CCD (Charge Coupled Device), converts incident light into an electric signal by photoelectric conversion, and outputs the signal as an image pickup signal in line order. The image sensor 11 is not limited to a CCD, and for example, a CMOS (Complementary Metal-Oxide Semiconductor) imager may be used.

  The front end (F / E) unit 12 includes, for example, a noise suppression unit, an automatic gain control unit, and an A / D conversion unit, and is a noise suppression unit for an imaging signal output as an analog signal from the imaging device 11. A correlated double sampling process is performed to suppress noise, and the automatic gain control unit controls the gain. Then, it is converted into a digital imaging signal by an A / D conversion unit and output. This digital image signal is RAW data composed of data directly corresponding to each pixel of the image sensor 11.

  Note that, based on a timing signal (clock pulse) output from a timing generator 14 (to be described later), the image pickup device 11 is controlled so that, for example, photoelectric conversion is performed at each frame timing and an image pickup signal is output, and the front end. The unit 12 is controlled to perform processing in synchronization with the frame timing.

  The signal processing unit 20 includes a camera signal processing unit 22, an extended image processing unit 23, a resolution conversion unit 24, a still image codec (CODEC) unit 25, a moving image codec unit 26, a display control unit 27, a CPU 30, an external interface (I / F). ) Unit 33, memory controller 34, and recording / playback control unit 36, and these processing blocks are connected via the bus 21.

  The camera signal processing unit 22 receives RAW data output from the front end unit 12. The camera signal processing unit 22 corrects the input RAW data to a predetermined value, and transfers the clock from the sensor driving system clock to the internal clock of the signal processing unit 20 using the buffer memory. The RAW data with the clock changed is converted into data in a format suitable for the subsequent moving image processing.

  The RAW data is compressed and encoded in a predetermined manner and sent to the bus 21 when recording a still image. The camera signal processing unit 22 decodes and decompresses the compression code of the compression-coded RAW data supplied via the bus 21, and further converts the data into a format suitable for still image processing in the subsequent stage.

  The camera signal processing unit 22 further enlarges and / or reduces the baseband moving image data and still image data, and converts them into a resolution (size) necessary for recording moving image data and output image data. In this example, for recording moving image data, data with a resolution of 2560 pixels × 1920 pixels is reduced to data with a resolution of 640 pixels × 480 pixels in accordance with the effective pixels of the image sensor 11. For still image recording, in this example, resolution conversion is not performed at this stage. The baseband moving image data and still image data subjected to resolution conversion in this way are output from the camera signal processing unit 22.

  Here, the baseband data is, for example, data obtained by converting an analog signal into a digital signal, and indicates data that has not been subjected to various types of processing such as compression coding and modulation. In this example, a digital image pickup signal obtained by converting an analog signal output from the image pickup element 11 into a digital signal by the front end unit 12 is appropriately referred to as baseband data.

  The extended image processing unit 23 performs extended image processing such as subject recognition processing and ultra-high speed continuous shooting. In the subject recognition process, for example, a part that looks like a face is extracted from an image using a previously learned pattern dictionary or the like. The ultra high-speed continuous shooting is to read out charges from the image sensor 11 and output an image pickup signal at a very high frame rate, for example, 240 fps with respect to the frame rate of 60 fps (frame per second) of normal moving image data. Since the ultra-high speed continuous shooting function requires a higher data transfer capability than normal moving image data, it is preferable to provide a separate processing unit.

  The resolution conversion unit 24 performs processing for converting the resolution of baseband moving image data or still image data to a resolution suitable for display on the display device 28 described later, or converting the resolution to a resolution corresponding to a recording mode.

  The still image codec unit 25 performs compression encoding processing on baseband still image data and decoding processing on compression-encoded still image data. The type of compression encoding method is not particularly limited, but for example, a JPEG (Joint Photographic Experts Group) method is applied. That is, the still image codec unit 25 divides the supplied still image data frame into encoded blocks of a predetermined size such as 8 × 8 pixels, and performs DCT for each encoded block. Then, the DCT coefficient obtained by DCT is quantized with a predetermined quantization scale. The quantized data is further compressed and output by variable length coding such as Huffman coding. The decoding process of the compressed still image data by the still image codec unit 25 is performed by the reverse process of the compression encoding process.

  The moving image codec unit 26 performs compression encoding processing on baseband moving image data and decoding processing on compression encoded moving image data. The type of compression coding system is not particularly limited. For example, MPEG2 (Moving Pictures Experts Group 2) system or ITU-T (International Telecommunication Union-Telecommunication Standardization Sector) recommendation H.264 or ISO (International Organization for Standarization) / IEC (International Electrotechnical Commission) International Standard 14496-10 (MPEG-4 Part 10) Advanced Video Coding (hereinafter abbreviated as H.264 | AVC) or the like can be applied. it can. In the following, it is assumed that the moving picture codec unit 26 applies the MPEG2 system as a compression encoding system.

  For example, the moving image codec unit 26 divides a frame of the supplied moving image data into encoded blocks of a predetermined size such as 8 × 8 pixels, and performs DCT for each encoded block. Then, the DCT coefficient obtained by DCT is quantized with a predetermined quantization scale. In addition, the moving image codec unit 26 also performs interframe coding on the supplied moving image data by predictive coding using motion compensation. Data subjected to intraframe coding and interframe coding is further compressed and output by variable length coding such as Huffman coding. The decoding process of the compressed moving image data by the moving image codec unit 26 is performed by the reverse process of the compression encoding process.

  The display control unit 27 converts the supplied image data into a signal in a format that can be displayed on the display device 28. The display device 28 includes, for example, an LCD (Liquid Crystal Display), is used as a viewfinder in the imaging device 1 and is used as a monitor for an image reproduced from the recording medium 37. In addition, the display control unit 27 generates a display signal for causing the display device 28 to display characters and graphics based on a command supplied from a CPU 30 described later. A display corresponding to the display signal is also displayed on the display device 28. For example, a setting screen for performing various settings for the imaging device 1, a display indicating various states of the imaging device 1, a cursor display, and the like are displayed on the display device 28 in a predetermined manner.

  A ROM (Read Only Memory) 31 and an input device 32 are connected to the CPU 30. The CPU 30 uses a RAM (Random Access Memory) (not shown) as a work memory, and controls the overall operation of the imaging apparatus 1 according to a program stored in advance in a ROM (Read Only Memory) 31.

  For example, the CPU 30 exchanges commands and data with each unit of the signal processing unit 20 via the bus 21 and controls each unit of the signal processing unit 20. Further, the CPU 30 generates a control signal for controlling the focus, aperture, zoom, and the like of the optical system 10 based on a control signal or an imaging signal according to the operation on the input device 32 described above, and supplies the control signal to the drive unit 13. To do. The drive unit 13 controls each unit of the optical system 10 according to the supplied control signal. Further, the CPU 30 issues a command to the timing generator 14 so as to output a predetermined clock pulse.

  In response to this command supplied from the CPU 30, the timing generator 14 generates a timing signal necessary for processing the image signal output from the image sensor 11, such as a frame synchronization signal, a horizontal synchronization signal, and a vertical synchronization signal. .

  The input device 32 is provided with various operators that are used to operate the imaging apparatus 1 and outputs a control signal corresponding to an operation on each operator. For example, a power key for switching ON / OFF of power supply by the power supply unit 39, a mode switching key for switching an operation mode in the imaging apparatus 1 such as a shooting mode and a recording mode, a key for moving a cursor, and the like are provided. Further, the input device 32 is used for operating a shutter button used when taking a still image with the imaging apparatus 1 and operating the focus, aperture, zoom, and the like of the optical system 10 when shooting a moving image or a still image. An operator or the like is also provided.

  The external I / F 33 controls data exchange between the imaging apparatus 1 and an external device.

  A RAM 35 is connected to the memory controller 34. The RAM 35 is, for example, an SDRAM (Synchronous Dynamic Random Access Memory) that operates in synchronization with a memory bus clock, and can input and output data by burst transfer in units of a predetermined data length (burst length). The RAM 35 is used in common with each unit connected to the bus 21. The memory controller 34 controls access to the RAM 35. In each processing block connected to the bus 21, when a transfer request to the RAM 35 occurs, the transfer request is passed from the bus 21 to the memory controller 34, and the memory controller 34 accesses the RAM 35 according to this transfer request. Is generated.

  The recording / reproducing control unit 36 performs recording control of data on the recording medium 37 and reproduction control of data recorded on the recording medium 37. As the recording medium 37, for example, a removable nonvolatile memory can be used. Also, a recordable optical disc can be used as the recording medium 37. Furthermore, a removable hard disk or a built-in hard disk in the imaging apparatus 1 can be used as the recording medium 37. Of course, as the recording medium 37, a magnetic tape conventionally used for recording moving image data can be applied.

  FIG. 3 shows an exemplary configuration of the camera signal processing unit 22. The camera signal processing unit 22 includes a RAW correction unit 50, a switch unit 51, a signal processing unit 52, and a resolution conversion unit 53, and includes a RAW compression unit 54 and a RAW expansion unit 56, and bus I / Fs 55, 57, and 58. Have.

  The bus I / Fs 55, 57, and 58 have buffer memories that can temporarily store data in the RAM 35. The bus I / Fs 55, 57, and 58 are configured to be able to read data from the buffer memory in units of burst length and send it to the bus 21 when data for the burst length is accumulated in the buffer memory. Also. The bus I / Fs 55, 57 and 58 are configured so that data read from the RAM 35 in units of burst length can be temporarily stored in a buffer and output at a predetermined timing.

  The RAW correction unit 50 includes a correction processing unit 50A and a synchronous transfer unit 50B. The correction processing unit 50A performs various correction processes such as defect correction and lens correction on the supplied RAW data, and also performs demosaic processing on the array of RAW data, and performs correction processing and color conversion processing at a later stage. Rearrange to a suitable data array.

  The synchronous transfer unit 50B performs a process of changing the sensor drive system clock to the system clock. That is, the data supplied to the camera signal processing unit 22 is an output of the image sensor 11 and is synchronized with a drive clock when reading out charges from the image sensor 11. On the other hand, the data output from the camera signal processing unit 22 needs to be synchronized with the clock inside the signal processing unit 20. The synchronous transfer section 50B has a buffer memory, writes input data to the buffer memory in accordance with the sensor drive system clock, and reads data from the buffer memory in accordance with the system clock. Data reading from the buffer memory is performed in response to a transfer request supplied to the synchronous transfer unit 50B.

  In response to the transfer request A from the signal processing unit 52, RAW data is read from the buffer memory of the synchronous transfer unit 50B and supplied to the signal processing unit 52 via one terminal 51A of the switch unit 51. The signal processing unit 52 converts the RAW data into an image signal in a format suitable for subsequent processing. For example, the signal processing unit 52 converts the RAW data into a Y / C signal composed of a luminance signal Y and a color difference signal Cb / Cr.

  The signal processing unit 52 further performs predetermined image quality correction processing such as white balance processing, gamma correction processing, and edge enhancement processing on the image signal obtained by converting the RAW data. In general, it is considered that the required image quality differs between a moving image and a still image. Therefore, for example, the signal processing unit 52 can switch parameters for performing the image quality correction processing depending on the control of the CPU 30 depending on whether the processing is performed on moving image data or the processing on still image data. Has been.

  The image signal subjected to signal processing by the signal processing unit 52 is resolution-converted to a resolution corresponding to, for example, the operation mode by the resolution conversion unit 53, and is transmitted via the bus I / F 58 in response to a transfer request C from the bus I / F 58. It is sent to the bus 21.

  The transfer request A from the signal processing unit 52 is also supplied to the bus I / F 57. The bus I / F 57 requests the memory controller 34 to read out the compressed RAW data stored in the RAM 35 via the bus 21 at a predetermined timing in response to the transfer request A, and the RAM 35 responds to this request. The compressed RAW data read from and transferred to the RAW decompression unit 56 is supplied. The RAW expansion unit 56 performs a decoding process on the compressed RAW data supplied via the bus I / F 57 by a process reverse to that of the RAW compression unit 54 described later, and expands the data. The switch 51 selects the terminal 51 </ b> B, and the output of the RAW decompression unit 56 is supplied to the signal processing unit 52 via the terminal 51 </ b> B of the switch unit 51.

  The processing of the signal processing unit 52 and the resolution conversion unit 53 is performed based on a transfer request C supplied from the bus I / F 58 and a response signal to the transfer request A output from the signal processing unit 52. That is, in the signal processing unit 52 and the resolution conversion unit 53, data transfer and signal processing are performed when both the data transfer request from the output counterpart and the data output response from the input counterpart are valid. .

  In response to the transfer request B from the bus I / F 55, RAW data is read from the buffer memory of the synchronous transfer unit 50B and supplied to the RAW compression unit 54. The RAW compression unit 54 compresses and encodes the supplied RAW data and outputs it. In the RAW compression unit 54, it is preferable to apply a compression coding method with less image quality degradation.

  For example, a conversion table that associates pre-conversion data with post-conversion data having a shorter bit length based on a conversion curve similar to a gamma curve, as described in JP-A-2005-311743, It is possible to use a method of compressing and encoding by shortening the data length by referring. In addition to the compression coding method using a conversion table based on a conversion curve similar to a gamma curve as described in Japanese Patent Application Laid-Open No. 2006-352509, compression coding by a DPCM (Defective Pulse Code Modulation) method is further performed. You may use the method which added the system. For example, Huffman code may be used for further compression coding on the data compression-coded by these methods.

  Note that the compression encoding process in the RAW compression unit 54 is performed with emphasis on real-time characteristics with respect to RAW data transfer, and is performed, for example, line-sequentially based on the readout characteristics of the image sensor 11. Not limited to this, a line memory capable of storing a plurality of lines is provided in the RAW compression unit 54, and still image data can be sequentially scanned in the horizontal direction in units of blocks of 8 horizontal pixels × 8 vertical pixels. .

  The compressed RAW data compression-encoded by the RAW compression unit 54 is sent to the bus 21 via the bus I / F 55.

  An example of the operation of the imaging apparatus 1 having such a configuration will be schematically described. In a moving image data recording mode in which moving image data is mainly recorded, light from a subject enters the image sensor 11 via the optical system 10, is converted into an electric signal by photoelectric conversion, and is output as an image signal at each frame timing. . The imaging signal is subjected to predetermined processing such as noise suppression processing and gain control by the front end unit 12, and is further A / D converted and output as RAW data at each frame timing. The RAW data output from the front end unit 12 is input to the signal processing unit 20 and supplied to the camera signal processing unit 22.

  The RAW data is corrected in the camera signal processing unit 22 by the correction processing unit 50A, and is written in the buffer memory by the synchronous transfer unit 50B with the sensor drive system clock. In response to the transfer request A output from the signal processing unit 52, RAW data is read from the buffer memory. At this time, reading of data from the buffer memory is performed according to the system clock, and clock switching is performed. The RAW data output from the synchronous transfer unit 50B is supplied to the signal processing unit 52 via the switch unit 51, converted into baseband moving image data, and the resolution conversion unit 53 performs predetermined resolution conversion. In response to the transfer request C, the moving image data is output from the resolution conversion unit 53, transferred to the bus 21 via the bus I / F 58, and written to the RAM 35.

  The moving image data written in the RAM 35 is read from the RAM 35 and supplied to the resolution conversion unit 24 and the moving image codec unit 26 via the bus 21. The resolution conversion unit 24 converts the resolution of the supplied moving image data into a resolution suitable for display on the display device 28, for example. The moving image data whose resolution is converted by the resolution conversion unit 24 is supplied to the display control unit 37 via the bus 21 and is displayed on the display device 28 as a recording monitor image.

  The moving image data written in the RAM 35 is also supplied to the moving image codec unit 26 via the bus 21. The moving image codec unit 26 performs compression encoding by, for example, the MPEG2 system. The compressed moving image data is supplied to the memory controller 34 via the bus 21 and stored in the RAM 35. The The resolution-converted moving image data is supplied to the memory controller 34 via the bus 21 and stored in the RAM 35.

  Note that the moving image data written in the RAM 35 may be supplied to the extended image processing unit 23 via the bus 21 to perform predetermined image processing, and then supplied to the resolution conversion unit 24. However, the present invention is not limited to this, and the moving image data subjected to resolution conversion by the resolution conversion unit 24 may be supplied to the extended image processing unit 23 via the bus 21 to perform image processing.

  When a predetermined amount or more of the compressed moving image data is accumulated in the RAM 35, the recording / playback control unit 36 reads the compressed moving image data from the RAM 35 for each recording unit of the recording medium 37 and records it on the recording medium 37.

  When recording a still image while recording moving image data, first, for example, during recording of a moving image, a shutter button provided as the input device 32 is pressed in order to instruct recording of a still image. A control signal corresponding to the pressing of the shutter button is supplied from the input device 32 to the CPU 30. Based on this control signal, the CPU 30 controls each unit of the imaging device 1 so as to perform still image recording.

  For example, still image data based on RAW data at the frame timing immediately after the shutter button is pressed is captured and recorded. That is, in the camera signal processing unit 22, the RAW data corresponding to the timing when the shutter button is pressed is subjected to a predetermined correction process by the correction processing unit 50A, and written to the buffer memory by the synchronous transfer unit 50B with the sensor drive system clock. The RAW data written in the buffer memory is read by the system clock in response to the transfer request B, the clock is changed, and is supplied to the RAW compression unit 54. The compressed RAW data compression-encoded by the RAW compression unit 54 is sent to the bus 21 via the bus I / F 55 and stored in the RAM 35 via the bus 21 and the memory controller 34.

  The compressed RAW data stored in the RAM 35 is read from the RAM 35 at a predetermined timing and supplied to the camera signal processing unit 22. The compressed RAW data supplied to the camera signal processing unit 22 is supplied to the RAW expansion unit 56 via the bus I / F 57 in response to the transfer request A, and the compression code is expanded. The expanded RAW data is converted into baseband still image data by the signal processing unit 52, resolution-converted by the resolution conversion unit 53, transferred to the bus 21 via the bus I / F 58, and sent to the memory controller 34. To the RAM 35.

  The RAW data written in the RAM 35 is read out in a predetermined manner, supplied to the still image codec unit 25 via the bus 21, and is compressed and encoded in accordance with, for example, the JPEG method. The compressed still image data is supplied to the recording / playback control unit 36 via the bus 21 and is recorded on the recording medium 37 at a timing when the moving image data is not recorded. The compressed still image data may be temporarily stored in the RAM 35, and the compressed still image data may be read from the RAM 35 and recorded on the recording medium 37 at a timing when moving image data is not recorded on the recording medium 37.

  Note that when still image data is recorded during recording of moving image data, the RAW data is compressed and encoded in the camera signal processing unit 22 and the compressed RAW data is written to the RAM 35 in order not to stop recording of the moving image data. It is necessary to perform in real time in synchronization with the output of the image sensor 11.

  On the other hand, reading of the compressed RAW data stored in the RAM 35, decompression processing and signal processing of the read RAW data in the camera signal processing unit 22, compression encoding processing in the still image codec unit 25, recording of compressed still image data The supply to the reproduction controller 36 is performed at a timing when the bus 21 is not congested, such as a drive signal blanking period in the image sensor 11. Since signal processing for still image data does not require real-time processing, it is possible to perform signal processing at low speed using software processing in the CPU 30 or a general-purpose DSP (Digital Signal Processor).

  Next, the processing in the camera signal processing unit 22 will be described in more detail. As described above, the synchronous transfer unit 50B temporarily stores the RAW data input in synchronization with the clock of the sensor drive system in the buffer memory, and performs reading from the buffer memory in synchronization with the system clock. , Changing clocks. It is assumed that the buffer memory has at least a capacity capable of absorbing clock transfer and instantaneous access delay in the bus 21. If writing to the buffer memory occurs without reading from the buffer memory, the buffer memory overflows, and the buffer function fails.

  In this way, as an example of buffer failure due to the difference in the reading / writing speed of RAW data to / from the buffer memory, the data transfer amount increases on the bus 21 to which the compressed RAW data compressed and encoded by the RAW compression unit 54 is sent out. The bus 21 may be congested. For example, when the bus 21 is congested due to transfer of data having a higher priority than the compressed RAW data, the bus I / F 55 waits for processing for sending the compressed RAW data to the bus 21 until the transfer of the data is completed, for example. . Alternatively, the RAW compression unit 54 is controlled to stop the RAW data compression process, and wait for the transfer of the data to end and the buffer memory to become empty.

  In addition, when the RAW compression unit 54 performs compression encoding of RAW data using variable length encoding in which the compression rate varies according to the spatial frequency component of the RAW data, if there are many high-frequency components of the RAW data to be compression encoded, In some cases, the amount of data after compression encoding becomes large. Even in such a case, a buffer failure may occur.

  If there is a possibility of buffer failure, in order to continue recording still image data, it is necessary to stop the RAW data compression encoding process and the preceding process and wait for the buffer memory to become empty. is there. The preceding process includes, for example, output of an imaging signal from the image sensor 11, output of RAW data from the front end unit 12, and correction processing in the correction processing unit 50A. However, when still image data is recorded during the recording of moving image data, it is necessary to continuously record moving image data, and therefore these preceding processes cannot be stopped.

  Therefore, the synchronous transfer unit 50B monitors the state of the buffer memory, and performs a failure process when a buffer failure is detected. FIG. 4 shows an example of the configuration of the synchronous transfer unit 50B configured to perform such processing. The synchronous transfer unit 50B includes a buffer memory 60 and a read control unit 61. The buffer memory 60 is a first-in first-out (FIFO) control memory, and writes data in response to a write enable signal supplied from the preprocessing side. The read control unit 61 issues a data read request to the buffer memory 60 based on the transfer request A and transfer request B described above and the capacity of the buffer memory 60. The capacity of the buffer memory 60 may only indicate, for example, whether the buffer memory is full (full) or empty (empty).

  When the capacity of the buffer memory 60 is not empty and both the transfer request A and the transfer request B are satisfied, the read control unit 61 reads the data from the buffer memory 60 by making the read request to the buffer memory 60 active. And output to the signal processing unit 52 and the RAW compression unit 54.

  Here, the read control unit 61 determines whether or not there is a possibility that the buffer will fail based on the capacity of the buffer memory 60 and the transfer request A and the transfer request B. That is, when the capacity of the buffer memory 60 is full and at least one of the transfer request A and the transfer request B is not established, it is determined that the buffer may fail. Whether or not the transfer request is established can be determined by determining whether or not the transfer request has arrived at the read control unit 61 at the timing of processing to be performed according to the transfer request, for example.

  A case where it is determined that only one of the transfer request A and the transfer request B is not established is called partial failure, and the cause of occurrence is held in, for example, an internal register of the read control unit 61. When it is determined that neither transfer request A nor transfer request B is established, it is called complete failure, and the cause of occurrence is held in the internal register. The CPU 30 can know whether partial failure or complete failure has occurred in the unit of frame processing by reading the internal register of the read control unit 61.

  For example, information on a transfer request that is not established can be used as the generation factor held in the internal register of the read control unit 61. As an example, when it is determined that the transfer request A is not established and a possibility of partial failure has occurred, it is conceivable that information indicating the transfer request A is held in an internal register of the read control unit 61. As another example, when it is determined that neither transfer request A nor transfer request B is established and there is a possibility of complete failure, information indicating transfer request A and transfer request B is read out from control unit 61. It can be held in an internal register.

  FIG. 5 collectively shows an example of the buffer failure determination based on the transfer request. For example, when only moving image data is recorded, only the transfer request A should be supplied to the read control unit 61. When the recording mode is the moving image data recording mode and the capacity of the buffer memory 60 is full, if the transfer request A is not supplied to the read control unit 61, it can be determined that the buffer may fail. As an example, when the bus 21 is congested and the transfer request C is not output from the bus I / F 58, the data whose resolution has been converted by the resolution converter 53 cannot be sent to the bus 21. In this case, the signal processing unit 52 does not output the transfer request A because it cannot request the next data from the synchronous transfer unit 50B.

  Further, for example, when only still image data is recorded, only the transfer request B should be supplied to the read control unit 61. When the recording mode is the still image data recording mode and the buffer memory 60 is full, if the transfer request B is not supplied to the read control unit 61, it can be determined that the buffer may fail. As an example, the bus I / F 55 can be prevented from outputting the transfer request B when the bus 21 is congested, and the RAW data compression encoding process in the RAW compression unit 54 can be stopped. If the congestion of the bus 21 is alleviated, it is conceivable that the transfer request B is output and the RAW data compression encoding process by the RAW compression unit 54 is restarted.

  Furthermore, for example, when recording moving image data and recording still image data while recording moving image data, the transfer request A and the transfer request B should be supplied to the read control unit 61, respectively. When the recording mode is the moving image and still image data recording mode and the buffer memory 60 is full, if either one of the transfer request A or the transfer request B is not supplied to the read control unit 61, the buffer is stored. If the transfer request A and the transfer request B are not supplied together, there is a possibility that the buffer will completely fail.

  In one embodiment of the present invention, when it is determined that there is a possibility that the buffer may fail in still image recording during moving image recording, the RAW data compression and encoding process by the RAW compression unit 54 is temporarily stopped. It waits for the buffer memory 60 to become empty. For example, it waits for data to be read from the buffer memory 60 in response to a transfer request A for moving image recording. At this time, the position where compression encoding on the RAW data is stopped is held in a predetermined register or memory. When the buffer memory 60 becomes empty and the buffer failure is resolved, the transfer request B is output from the bus I / F 55, the raw data sequentially stored in the buffer memory 60 is read, and the raw code is compressed by the raw compression unit 54. Turn into.

  This will be described more specifically with reference to FIG. The position of the line m in the still image data instructed to be recorded in correspondence with the data of the frame N of the moving image data (hereinafter “frame N data of the moving image data” is simply described as “frame N” as appropriate) It is assumed that the possibility of buffer failure has occurred. For example, if the read control unit 61 detects that the transfer request B is not supplied, it is determined that there is a possibility of partial buffer failure. When there is a possibility of buffer failure, the RAW compression unit 54 stops the RAW data compression encoding process. The compression code processing is stopped for a certain amount of RAW data from the position where the possibility of buffer failure has occurred. In the example of FIG. 6, the RAW data compression encoding process by the RAW compression unit 54 is resumed from the line m + α after a certain amount has elapsed from the position where the possibility of buffer failure has occurred.

  Hereinafter, the position from the position where the possibility of buffer failure occurs to the position where the buffer failure is resolved is referred to as a failure section. A position where the possibility of buffer failure has occurred is called a failure position, and a position where the buffer failure has been eliminated is called a failure recovery position. In the failed section, since the RAW data compression encoding process is stopped, still image data is lost.

  In an image in which the compression code is expanded, valid RAW data is missing in the failed section from line m to line m + α, and normal display is not performed. In order to replace the data in the failed section with valid image data when decompressing the compressed RAW data, it is necessary to retain information indicating the failed section.

  For example, as illustrated in FIG. 7, it is conceivable that a predetermined code that is not used for RAW data compression encoding is used as information indicating a failed section and is embedded in compression encoded RAW data. As an example, as shown in FIG. 7A, let us consider a case where the failure section from the failure position on the line m to the failure recovery position on the line m + α, that is, the restart position of the compression encoding process, is a failure section.

  In this case, as illustrated in FIG. 7B, the RAW data is filled with a compression encoded code until just before the failure position, and from the failure position to the failure recovery position is a predetermined code that is not used in the compression code. Buried. For example, when compression encoding of RAW data is performed by code conversion using a table, a code that is not defined in this table can be used as a code indicating a failed section. For example, when the buffer memory 60 becomes empty and is recovered from the buffer failure, the code indicating the failure interval is generated and output by the RAW compression unit 54 in a predetermined manner. Thereafter, compression coding is performed on the RAW data after the failure recovery position.

In addition, holding information indicating a failed section is not limited to a method of replacing the failed section with a predetermined code. For example, as illustrated in FIG. 8, it may be possible to hold position information indicating a failed section separately from the compressed RAW data. As the position information, pixel coordinate information can be used. When the buffer failure occurs at the coordinates (x ₁ , y ₁ ) and recovers from the failure at the coordinates (x ₂ , y ₂ ) as in the example of FIG. 8A, the failure position information is as shown in FIG. 8B. The coordinates ST (x ₁ , y ₁ ) indicating the failure position and the coordinates ED (x ₂ , y ₂ ) indicating the failure recovery position can be indicated. When the failed section is indicated using position information, the code of the failed section may be arbitrary as illustrated in FIG. 8C.

  The position information indicating the failed section is not limited to pixel coordinate information. For example, when the RAW data compression encoding process is performed line-sequentially, a line number can be used. As in the example of FIG. 8A, when the failure section is from line m to line m + α, the failure position information can be expressed as line (m, m + α). Further, when the RAW data compression encoding process is performed in units of blocks, a block number or the like for identifying a block can be used as the failure position information.

  In one embodiment of the present invention, when decompressing the compressed RAW data, the image in the failed section is restored with reference to the image data based on the corresponding moving image data. The image data referred to when restoring the failed section is stored separately from the moving image data and the still image data.

  With reference to FIG. 9, description will be given of holding image data referred to during restoration. RAW data is captured as still image data at the timing of frame N of the moving image data, and is compressed and encoded by the RAW compression unit 54. At this time, it is assumed that the section from the line m to the line m + α is a failed section. In this case, a section corresponding to the broken section of the frame N in the moving image data is separately held as restoration data.

  For example, when there is a possibility of buffer failure in still image recording during moving image recording, a frame N corresponding to still image data to be recorded before being compressed and encoded by the moving image codec unit 26 is held in the RAM 35. In addition, it is conceivable that data of an area corresponding to the failed section is extracted from the held frame N and used as restoration data. When decompressing the compressed RAW data, the stored restoration data is used after being enlarged to a predetermined size according to the ratio between the image frame size of the moving image data and the image frame size of the still image data.

  For example, when the image frame size of still image data is 2560 pixels × 1920 pixels and the image frame size of moving image data is ¼ of 640 pixels × 480 pixels, the line m / 4 to the line (m + α) / Sections up to 4 are held as restoration data. Actually, a predetermined area is added to the area corresponding to the ratio of the image frame size of the moving image data and the still image data for the expansion process of the restoration data when decompressing the compressed RAW data. Data is used as restoration data. In the example of FIG. 9, a section of frame N in which A lines are added up and down with respect to the section corresponding to the bankrupt section is used as restoration data.

  In addition, when restoring the failed section, it is conceivable to directly refer to the corresponding moving image data as illustrated in FIG. In this case, it is conceivable to directly refer to the frame N corresponding to the still image data to be recorded, which is stored in the RAM 35 for compression encoding by the moving image codec unit 26. In this case, it is necessary to perform restoration processing of the failed section in consideration of the compression encoding timing of the frame N in the moving picture codec unit 26.

  FIG. 11 is a flowchart showing control of an example of the RAW data compression encoding process according to the embodiment of the present invention. The processing according to this flowchart is controlled by the CPU 30 based on the determination by the read control unit 61, for example.

  Note that the imaging apparatus 1 is assumed to be a mode in which a recording mode can record a still image during moving image recording. Accordingly, the transfer request A and the transfer request B are supplied to the read control unit 61 of the synchronous transfer unit 50B. However, the present invention is not limited to this. When a moving image is recorded, the moving image recording mode is normally set, and only the transfer request A is supplied to the read control unit 61. You may make it change into the recording mode in which image recording is possible. In this case, the transfer request A and the transfer request B are supplied to the read control unit 61 as the recording mode is changed.

  In step S10, it is determined whether or not there is a possibility of buffer failure. For example, if only one of the transfer request A and the transfer request B is supplied to the read control unit 61 where the transfer request A and the transfer request B are to be supplied corresponding to the recording mode, a buffer is stored. It can be determined that there is a possibility of partial failure. If it is determined that both the transfer request A and the transfer request B are supplied to the read control unit 61, the read control unit 61 reads RAW data from the buffer memory 60 in a predetermined manner in step S11. The RAW compression unit 54 performs compression encoding. The compression-coded RAW data is transferred to the RAM 35 via the bus 21.

  On the other hand, if it is determined in step S10 that there is a possibility that the buffer may partially fail, the process proceeds to step S12. In step S12, information indicating the failure position is held in a predetermined manner. In the next step S13, moving image data for restoring the failed section is secured when the RAW compressed data is expanded.

  Here, the failure position is shown by replacing the failure section described with reference to FIG. 7 with a predetermined code that is not used in the RAW data compression encoding. In addition, as data for restoring the failed section, an area corresponding to the failed section is extracted from the frame N of the moving image data corresponding to the captured still image described with reference to FIG. It shall be secured by

  In the next step S14, the compression encoding process in the RAW compression unit 54 is stopped. In step S15, it is determined whether or not the possibility of buffer failure has been recovered. For example, in step S15, it is determined whether or not compression encoding for a certain amount of RAW data from the failure position has been stopped. If it is determined that there is no recovery from the possibility of buffer failure, the process returns to step S14, and the RAW data compression encoding process is stopped. On the other hand, if it is determined that recovery from the possibility of buffer failure is made, the process returns to step S10, and RAW data transfer and compression encoding are performed in the next step S11.

  Next, an example of restoration processing for a failed section will be described with reference to FIGS. 12 and 13. FIG. 12 is a flowchart illustrating an example of a restoration process of a failed section in the decoding process of compressed RAW data. Note that the position of the failed section is indicated by replacing the data of the failed section with a predetermined code that is not used in compression encoding, and data for restoring the failed section is stored in the RAM 35. It is assumed that the data corresponding to the failed section is extracted from the data before compression coding by the codec unit 26 and separately held data is used.

  In the RAW decompression unit 56, the compressed RAW data read from the RAM 35 is decoded from a position corresponding to the upper left corner of the screen, for example. At this time, a predetermined code indicating the failure position is detected (step S20), and if the code currently being decoded in the compressed RAW data is not the code of the process, the decoding process is performed in step S21 and the compressed RAW is compressed. Decompress the data. The expanded RAW data is subjected to predetermined signal processing to generate image data. When a code indicating the failure position is detected in step S20, information indicating the failure position is obtained and held based on the detection result.

  For example, while the signal processing unit 52 does not perform signal processing on moving image data, the signal processing unit 52 performs signal processing on the decompressed RAW data to generate image data. For example, the CPU 30 may perform software processing, or a DSP (Digital Signal Processor) may be separately provided for signal processing. The decompression process of the compressed RAW data does not require real-time performance like the signal process for moving image data, and can be performed by a slower process.

  On the other hand, if the predetermined code is detected in step S20, for example, the decoding process in step S21 is skipped, and the process proceeds to step S22. In step S22, it is determined whether or not the processing is completed up to the end of the screen, that is, the lower right corner of the screen. If it is determined that the process has not ended, the process returns to step S20.

  On the other hand, if it is determined in step S22 that the process up to the end of the screen has been completed, the process proceeds to step S23. In the processing from step S23 onward, based on the failure position detection result in step S20 described above, the restoration data is applied to the failure section to restore the RAW data. For example, as illustrated in FIG. 13A, it is assumed that a portion indicated by diagonal lines is detected as a broken section in still image data having an image frame size of 2560 pixels × 1920 pixels.

  In step S23, the restoration data is read from the RAM 35. In the next step S24, the restoration data read from the RAM 35 in accordance with the ratio between the image frame size of the moving image data and the image frame size of the still image data. The enlargement process is performed for. For example, as illustrated in FIG. 13B, if the image data frame size is 640 pixels × 480 pixels, and the horizontal and vertical directions are 1/4 of the still image data, the restoration data Is expanded to 4 times the size in the horizontal and vertical directions. The enlargement process can be performed, for example, by generating a new pixel value using linear interpolation or the like based on the pixel value of the pixel whose pixel distance is enlarged.

  Next, in step S25, image quality adjustment of the enlarged restoration data is performed. That is, generally, since the required characteristics are different between the moving image and the still image, the image quality such as the brightness and the hue of the image is different. As a result, it is conceivable that an unnatural image is obtained simply by replacing the moving image data corresponding to the failed section with the expanded RAW data. Therefore, the image quality of the restoration data is adjusted so that the image quality of the restoration data matches the image quality of the decompressed RAW data in the part other than the failed section.

  The image quality adjustment is performed by, for example, referring to a parameter used when the signal processing unit 52 performs image quality correction processing on moving image data and a parameter used when image quality correction processing is performed on still image data, respectively, and enlarges the restoration data. This can be done by converting the restoration data so that the characteristics of the image data matched with the characteristics of the portion other than the failed section of the decompressed RAW data.

  In the next step S26, as illustrated in FIG. 13C, a process of replacing the data in the failed section of the decompressed RAW data with the restored data that has been enlarged and adjusted for image quality is performed. Here, the restoration data is an image obtained by enlarging the moving image data having a resolution of 1/4 with respect to the still image data by four times using linear interpolation or the like. Therefore, the surrounding decompressed RAW data that is replaced by the restoration data. The image quality is coarse. For this reason, if the restoration data is simply replaced with the data of the failed section in the decompressed RAW data, the joints are not smooth at the replaced boundary portion, and the replacement becomes conspicuous.

  Therefore, as illustrated in FIG. 13D, in step S27, the boundary having been replaced by the restoration data and the data in the vicinity of the boundary are spatially subjected to low-pass filter processing, so that the boundary portion where the replacement has been performed. Smooth the joints.

  In step S28, it is determined whether or not the processing for all the failed positions in one extended RAW data has been completed. If it is determined that the processing has not been completed, the process returns to step S23, and the next failed position is reached. On the other hand, processing from step S23 to step S27 is performed.

  If it is determined in step S28 that the processing for all the broken positions in one piece of decompressed RAW data has been completed, a series of restoration processing is completed. The decompressed RAW data with the failed section restored is, for example, subjected to predetermined resolution conversion by the resolution conversion unit 53, sent to the bus 21 in response to the transfer request C from the bus I / F 58, and written to the RAM 35 via the memory controller 34. It is. Then, the data is read out to the recording / playback control unit 36 at a predetermined timing and recorded on the recording medium 37.

  As described above, according to the embodiment of the present invention, when a still image is recorded during moving image recording, even if the buffer processing of the still image RAW data may fail, the compression of the still image RAW data is performed. By temporarily stopping the encoding process, the moving image recording can be performed without stopping. In addition, the information indicating the section in which the compression encoding process of the still image is stopped is retained, and the moving image data corresponding to the still image is retained to restore the section in which the compression encoding process is stopped in the still image. Can be generated from the stored moving image data. As a result, even when there is a possibility that the RAW data buffer processing of the still image may fail, it is possible to restore the still image with the least possible degradation from the original image.

  In the above description, the failure section is restored using one frame of moving image data corresponding to one piece of still image data, but this is not limited to this example. For example, it is possible to restore a failed section using a plurality of frames of moving image data for one piece of still image data. That is, based on the motion information of images of a plurality of frames, when it can be considered that the motion of the image in the failed section is smaller than a predetermined value, a frame corresponding to the still image and a plurality of frames temporally before and after the frame The restoration data is generated using and.

As an example, as shown in FIG. 14, when a buffer failure occurs with respect to a still image captured at the timing of the frame N, the frame N of the moving image data and the frame ( It is conceivable to generate restoration data using N-1) and frame (N + 1). For example, the motion vector MV _N−1 of the frame (N−1) and the motion vector MV _{N + 1} of the frame (N + 1) with respect to the frame N are obtained based on the motion detection result in the video codec unit 26. When the motion vector MV _N-1 and / or the motion vector MV _{N + 1} is smaller than a predetermined value, restoration data is generated using the motion vector MV _N-1 and the motion vector MV _{N + 1} . In this way, by generating the restoration data from a plurality of frames, the resolution of the restoration data can be brought close to the resolution of the RAW data of the still image.

In practice, the motion vector MV is obtained in units of blocks of a predetermined size, for example, 16 pixels × 16 pixels. Here, for the sake of explanation, the motion vector for the frame N is represented by the motion vector MV _{N-1 for} the frame (N−1) and the motion vector MV _{N + 1} for the frame (N + 1).

  In addition, when one still image is recorded, there is a possibility of buffer failure many times, and the number of times the RAW data compression encoding process is stopped is large, or the compression encoding process is continuously stopped. In this case, the number of parts to be replaced with restoration data increases, and there is a risk that the image quality will be significantly deteriorated.

  Therefore, the amount of the failure section due to the possibility of the buffer failure occurring in the still image recording during the moving image recording is measured, and the RAW data is compressed and encoded and the compressed RAW data is stored in the RAM 35 even in a plurality of subsequent frames. It is conceivable that compressed RAW data with the smallest amount of failed sections is selected and decompressed and recorded later. For example, it is conceivable to use the area of the failed section as the amount of the failed section.

  This will be described more specifically with reference to FIG. In the example of FIG. 15, there is a possibility of buffer failure at the time of transfer of RAW data corresponding to the frame N, and there is a section in which compression encoding is stopped. Since compression encoding is stopped due to the possibility of buffer failure at the timing of frame N, transfer of corresponding RAW data and compression encoding are performed for the subsequent frame (N + 1) and frame (N + 2). In the example of FIG. 15, even in these frames (N + 1) and (N + 2), there is a section in which compression encoding is stopped due to the possibility of buffer failure.

  Among the compressed RAW data corresponding to the frames N, (N + 1) and (N + 2), the compressed RAW data corresponding to the frame (N + 1) with the smallest amount of the failed section is selected, and the decompression process and the predetermined signal processing are performed. Made and recorded. At the time of decompression, it is more preferable to perform a process of generating restoration data using the corresponding moving image data frame and replacing it with the data of the failed section as described above. With this processing, there is a possibility that the timing is slightly shifted with respect to the still image recording instruction, but it is possible to obtain a still image with as little image degradation as possible.

It is a basic diagram for demonstrating the concept of invention. It is a block diagram which shows the structure of an example of the imaging device applicable to one Embodiment of invention. It is a block diagram which shows the structure of an example of a camera signal processing part. It is a block diagram which shows the structure of an example of a synchronous transfer part. It is a basic diagram which shows an example of the buffer failure determination based on a transfer request. It is a basic diagram for demonstrating control of an example of the compression encoding process with respect to RAW data by one Embodiment of invention. It is a basic diagram for demonstrating using as information which shows a failure area using the predetermined code | symbol which is not used for the compression encoding of RAW data. It is a basic diagram for demonstrating holding | maintaining the positional information which shows a failure area separately from compression RAW data. It is a basic diagram for demonstrating holding | maintenance of the image data referred in the case of decompression | restoration. It is a basic diagram for demonstrating referring directly the moving image data corresponding in the case of restoration of a failure area. It is a flowchart which shows control of an example of the RAW data compression encoding process by one Embodiment of invention. It is a flowchart which shows the decompression | restoration process of an example of the failure area in the decoding process of compression RAW data. It is a basic diagram for demonstrating the decompression | restoration process of an example of the failure area in the decoding process of compression RAW data. It is an approximate line figure for explaining performing restoration of a failure section using a plurality of frames of video data to one piece of still picture data. It is an approximate line figure for explaining selecting and decompressing data with the smallest amount of a failure section among a plurality of continuous compression RAW data.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Image pick-up element 20 Signal processing part 22 Camera signal processing part 24 Resolution conversion part 25 Still image codec part 26 Movie codec part 30 CPU
34 Memory controller 35 RAM
36 Recording / Reproduction Control Unit 37 Recording Medium 50 RAW Correction Unit 50A Correction Processing Unit 50B Synchronization Transfer Unit 51 Switch Unit 52 Signal Processing Unit 53 Resolution Conversion Unit 54 RAW Compression Units 55, 57, 58 Bus I / F
56 RAW expansion unit

Claims (17)

In an imaging device capable of recording still image data in parallel with recording of moving image data,
An imaging unit that converts light from a subject into an electrical signal and outputs the signal as an imaging signal at a frame timing;
A moving image processing unit that continuously processes the imaging signal output from the imaging unit as a moving image for each frame timing;
A still image processing unit that processes the imaging signal output from the imaging unit as a still image at the frame timing according to an imaging instruction supplied at an arbitrary timing;
An imaging apparatus comprising: an image restoration unit that restores a missing portion of the still image using the moving image frame corresponding to the frame timing of the still image. The imaging apparatus according to claim 1,
The image restoration unit restores the missing portion using data of a region corresponding to the missing portion of the moving image frame corresponding to the frame timing to the still image to be restored. . The imaging device according to claim 2,
The moving image processing unit holds the frame before performing the processing, and the image restoration unit restores the missing portion using the held frame before performing the processing. An imaging device. The imaging device according to claim 2,
The imaging apparatus, wherein the image restoration unit restores the missing portion directly using the frame processed at each frame timing by the moving image processing unit. The imaging device according to claim 2,
The image restoration unit uses a plurality of frames including a frame whose frame timing matches the still image to be restored, and a predetermined number of frames temporally before and / or behind the frame whose frame timing matches. An image pickup apparatus that restores the missing portion. The imaging apparatus according to claim 5,
The imaging apparatus, wherein the image restoration unit restores the missing portion based on the motion information of the plurality of frames. The imaging device according to claim 2,
The still image processing unit is configured to process a plurality of still images at each frame timing according to the imaging instruction,
The image restoration unit selects a still image with the smallest amount of the missing portion from the plurality of still images, and uses the moving image frame corresponding to the frame timing with respect to the still image to detect the missing portion. An imaging apparatus that performs restoration. The imaging apparatus according to claim 7,
The imaging apparatus, wherein the still image processing unit performs processing on the still image at the frame timing next to the still image when the missing portion may occur in the still image. The imaging device according to claim 2,
The image restoration unit restores the missing portion by replacing the data of the missing portion of the still image with data of an area corresponding to the missing portion of the frame of the moving image. apparatus. The imaging device according to claim 9,
The image restoration unit, when performing the replacement, makes the resolution of the moving image used for the replacement match the resolution of the still image. The imaging device according to claim 9,
The image restoration unit, when performing the replacement, performs processing for matching the image quality of the moving image used for the replacement with the image quality of the still image. The imaging device according to claim 9,
The imaging apparatus according to claim 1, wherein the image restoration unit performs low-pass filter processing on the replaced boundary. The imaging apparatus according to claim 1,
The still image processing unit outputs the still image by replacing the data of the missing portion with a predetermined code,
The imaging apparatus, wherein the image restoration unit performs the restoration based on the predetermined code detected from the still image. The imaging apparatus according to claim 1,
The still image processing unit holds information indicating the position of the missing part,
The imaging apparatus, wherein the image restoration unit performs the restoration based on information indicating a position of the missing portion. The imaging apparatus according to claim 1,
The imaging apparatus, wherein the still image processing unit stops the processing for a certain amount of the imaging signal when there is a possibility that a defect occurs in the still image. In the control method of the image pickup apparatus capable of recording still image data in parallel with recording of moving image data,
A step of moving image processing in which light from a subject is converted into an electric signal and an imaging signal output from an imaging unit at a frame timing is continuously processed as a moving image at each frame timing;
A still image processing step of processing the imaging signal output from the imaging unit as a still image at the frame timing in accordance with an imaging instruction supplied at an arbitrary timing;
And a step of image restoration for restoring a missing portion of the still image using the moving image frame corresponding to the frame timing of the still image. In an imaging apparatus control program for causing a computer to execute a control method for an imaging apparatus that can record still image data in parallel with recording of moving image data,
A step of moving image processing in which light from a subject is converted into an electric signal and an imaging signal output from an imaging unit at a frame timing is continuously processed as a moving image at each frame timing;
A still image processing step of processing the imaging signal output from the imaging unit as a still image at the frame timing in accordance with an imaging instruction supplied at an arbitrary timing;
An image pickup apparatus that causes a computer to execute a control method for an image pickup apparatus including an image restoration step of restoring a missing portion of the still image using the moving image frame corresponding to the frame timing of the still image. Control program.

Images