JP2009130563A - Unit, method, and program for processing data, imaging apparatus, and method and program for controlling the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To stably transfer data compressed and coded using a variable-length code or a fixed-length code. <P>SOLUTION: An RAW compression section 54 outputs compression RAW data at a prescribed interval T<SB>int</SB>for each data unit with data length that is equal to or shorter than the burst length of an SDRAM to which the compression RAW data are written as a data unit. The compression RAW data output in the data unit from the RAW compression section 54 in this manner are supplied to a bus I/F55. When data having the burst length of the SDRAM are stored in a buffer of the bus I/F55, the data for the burst length are read from the buffer and are transferred to the SDRAM via a bus 21. The amount of transfer data transferred from the bus I/F55 to the bus 21 is smoothed, and there is no sudden occurrence of large-capacity data transfer to the bus 21 from the bus I/F55, thus relieving congestion in the bus 21. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a data processing device, a data processing method and a data processing program capable of recording a moving image and a still image, and an imaging device, a control method for the imaging device, and a control program for the imaging device. The present invention relates to a configuration in which the amount of data transfer is dispersed when a still image is recorded.

  In general, the scanning method of the image sensor of digital still cameras that mainly shoot still images and digital video cameras that mainly shoot moving images is mainly suitable for still image shooting and movie shooting. The scanning method is greatly involved in the entire system. Recent digital still cameras use an image sensor having a number of pixels of several million or more. In order to record a still image with higher resolution, the output from the image sensor is 1/30 second or more. An image signal that has been sequentially scanned over a period of time is used. Further, in recent years, there are many examples in which higher-resolution still image output is realized by using RAW data using an image pickup signal output from an image pickup device as it is. On the other hand, digital video cameras often employ image signals based on interlaced scanning every 1/60 seconds in order to increase the dynamic resolution of a subject.

  As described above, in the case where the scanning method of the image sensor is appropriately selected according to the use, it is considered that a moving image and a still image are recorded in parallel in one image capturing apparatus.

  For example, in the case of a system that prioritizes the quality of still image recording, a moving image is recorded using an image obtained by sequential scanning every 1/30 seconds or more, so that the dynamic resolution is insufficient. Conversely, in the case of a system that prioritizes the quality of moving image recording, a method of acquiring a still image from a single moving image at the moment when a still image recording instruction is issued (for example, when the shutter button is pressed) can be considered. . In this case, basic image quality processing such as color difference correction and gamma correction is mainly for moving images, and there is a possibility that processing suitable for still images is not necessarily performed.

  Furthermore, when both the moving image recording function and the still image recording function are to be realized for the purpose of mutually obtaining high quality images by using the image sensor and the processing means in common, the operation of the entire image sensor such as the scanning method can be switched. It becomes necessary. As a result, for example, when a high-resolution still image recording instruction is given during moving image recording, it is necessary to switch the control of the image sensor and switch the operation of the imaging signal processing, and during the still image acquisition period Since the image pickup signal suitable for the moving image output from the image sensor is interrupted, the moving image recording may be temporally discontinuous.

A technique for avoiding the discontinuity of the moving image recording accompanying the still image recording is described in Patent Document 1. According to the description in Patent Document 1, an image pickup signal having a resolution higher than that of a moving image is output every N (N ≧ 2) times of a moving image cycle, so that high resolution can be achieved without degrading the resolution and image quality of the moving image. It is possible to obtain a still image. On the other hand, according to the method described in Patent Document 1, outputting a high-resolution signal for a still image every N times the moving image period is, in other words, output in the remaining (N−1) frame periods. Therefore, the image pickup signal has a resolution for moving images.
JP 2002-44531 A

  In order to avoid the above-described problem, a method has been proposed in which RAW data used for still image recording is temporarily stored in a memory, and processing (development processing, etc.) is performed on the RAW data after moving image processing. In this case, the number of still images that can be recorded is limited by the storage capacity of the memory used for storing the RAW data. In order to avoid this limitation, when the RAW data is temporarily stored in the memory, the RAW data is read from the memory during recording of the moving image, for example, during the horizontal blanking period or the vertical blanking period, and the RAW data is stored in the RAW data. On the other hand, a method of performing a predetermined process such as a development process has also been proposed.

  Here, by performing compression encoding processing with reduced image quality degradation on the RAW data and transferring and storing it in the memory, it is possible to suppress an increase in the memory bandwidth of the entire system. As a result, still image recording during the moving image recording operation can be performed more reliably.

  As a compression encoding method applied to RAW data, a method of outputting a fixed-length code for a certain amount of RAW data and a method of outputting a variable-length code can be considered. A method using a variable length code is more preferable because it has higher compression efficiency than a method using a fixed length code.

  However, when RAW data is compressed and encoded using variable-length codes, the compression rate varies greatly depending on the picture pattern, quantization scale when quantizing, and the like. The distribution will fluctuate. As a result, there is a problem that it is difficult to guarantee the total amount of the bus bandwidth of the entire system. That is, when still image recording is performed during a moving image recording operation, there is a problem in that it becomes difficult to secure a memory band according to a change in the compression rate of RAW data, and processing may fail.

  In addition, even when RAW data is compressed and encoded using a fixed-length code, the process of transferring compressed RAW data to a memory is stable when still image recording is performed during a moving image recording operation. Therefore, there is a problem that a wider memory bandwidth and bus bandwidth are required to increase the device cost.

  Therefore, an object of the present invention is to provide a data processing device, a data processing method and a data processing program capable of stably transferring data compressed and encoded using a variable length code or a fixed length code, and an imaging device and an imaging device And a control program for an imaging apparatus.

  In order to solve the above-described problem, the first invention is a data transfer unit that transfers data on a bus in units of a first data, and performs predetermined data processing on the data, and the first data unit And a data processing unit that outputs data at a predetermined interval for each second data unit having a data size that is equal to or equal to the data size obtained by dividing the first data unit by an integer, and is output from the data processing unit. The data processing apparatus is characterized in that the transferred data is transferred from the data transfer unit onto the bus.

  The second aspect of the invention is a data transfer step for transferring data on the bus in units of the first data, and whether the data size is equal to the first data unit by performing predetermined data processing on the data. Or a data processing step of outputting a predetermined interval for each second data unit having a data size obtained by dividing the first data unit by an integer, and the data output by the data processing step is A data processing method is characterized in that the data is transferred onto a bus in a data transfer step.

  According to a third aspect of the present invention, there is provided a data transfer step of transferring data on the bus in a first data unit, and a predetermined data processing is performed on the data so that the data size is equal to the first data unit. Or a data processing step of outputting a predetermined interval for each second data unit having a data size obtained by dividing the first data unit by an integer, and the data output by the data processing step is A data processing program that causes a computer to execute a data processing method that transfers data onto a bus in a data transfer step.

  According to a fourth aspect of the present invention, there is provided an imaging apparatus that performs processing on moving image data based on an imaging signal output from an image sensor and performs processing on still image data based on the imaging signal in parallel with processing of moving image data. An imaging unit that converts light into an electrical signal by photoelectric conversion and outputs it as an imaging signal, an imaging signal processing unit that performs predetermined signal processing on the imaging signal output from the imaging unit and outputs the signal as image data; A moving image data processing unit that outputs one moving request, performs processing at each frame timing on the image data output from the imaging signal processing unit in response to the first transferring request, and outputs moving image data; A still image that outputs a transfer request, processes the image data output from the imaging signal processing unit in response to the second transfer request at a predetermined timing, and outputs still image data A still image data processing unit that compresses and encodes the image data output from the imaging signal processing unit, and the compressed image data output from the compression encoding unit is a first A data transfer unit that transfers the data on the bus in units of data, and the compression encoding unit has the same data size as the first data unit or data obtained by dividing the first data unit into integers. The image pickup apparatus is characterized in that a predetermined interval is output for each second data unit of size.

  According to a fifth aspect of the present invention, there is provided an imaging apparatus that performs processing on moving image data based on an imaging signal output from an image sensor and performs processing on still image data based on the imaging signal in parallel with processing of moving image data. In the control method, the imaging signal processing step for converting the light into an electrical signal by photoelectric conversion and performing predetermined signal processing on the imaging signal output from the imaging unit and outputting it as image data, and a first transfer request A video data processing step for outputting video data by performing processing at each frame timing on the image data output in the imaging signal processing step in response to the first transfer request, and a second transfer request In response to the second transfer request, the image data output in the imaging signal processing step is processed at a predetermined timing, and still image data is processed. A still image data processing step, the still image data processing step comprising: a compression encoding step for compressing and encoding the image data output by the imaging signal processing step; and a compression encoding step. A data transfer step of transferring the compressed image data output by the first data unit onto the bus, wherein the compression encoding step is performed to determine whether the compressed image data has the same data size as the first data unit, or Alternatively, the control method of the imaging apparatus is characterized in that the first data unit is output with a predetermined interval for each second data unit having a data size obtained by dividing the first data unit by an integer.

  According to a sixth aspect of the present invention, there is provided an imaging apparatus that performs processing on moving image data based on an imaging signal output from an image sensor and performs processing on still image data based on the imaging signal in parallel with processing of moving image data. In a control program for causing a computer to execute the control method, imaging signal processing is a step of performing predetermined signal processing on the imaging signal output from the imaging unit by converting light into an electrical signal by photoelectric conversion and outputting the image data as image data And a step of moving image data processing for outputting the moving image data by outputting the first transfer request, performing the processing for each frame timing on the image data output by the imaging signal processing step in response to the first transfer request And a second transfer request, and output the image data output by the imaging signal processing step in response to the second transfer request. A still image data processing step that performs processing at a predetermined timing and outputs still image data. The still image data processing step includes a compression code that compresses and encodes the image data output in the imaging signal processing step. And a data transfer step of transferring the compressed image data output by the compression encoding step onto the bus in a first data unit. The compression encoding step includes the first step of compressing the compressed image data. A method of controlling an image pickup apparatus that outputs a predetermined interval for each second data unit having a data size equal to or equal to the data size of the first data unit or a data size obtained by dividing the first data unit by an integer 6 is a program for controlling an imaging apparatus.

  As described above, the first, second, and third inventions perform predetermined data processing on data, and the data size is equal to the first data unit, or the first data unit is an integer. For each second data unit of the divided data size, the data is output with a predetermined interval and transferred on the bus in the first data unit, so that the data transferred on the bus is distributed. The

  The fourth, fifth, and sixth inventions perform processing on moving image data based on the imaging signal output from the imaging element, and perform processing on the still image data based on the imaging signal in parallel with the processing of the moving image data. In order to be able to do this, light is converted into an electrical signal by photoelectric conversion, and a predetermined signal processing is performed on the image pickup signal output from the image pickup unit and output as image data, which is output in response to the first transfer request. The image data is processed at each frame timing to output moving image data, and the image data output in response to the second transfer request is processed at a predetermined timing to output still image data. The still image data processing compresses and encodes the image data, and transfers the compressed image data to the bus in the first data unit. The compression encoding processing converts the compressed image data to the first data The data size is equal to the data unit, or the data is output at a predetermined interval for each second data unit of the data size obtained by dividing the first data unit by an integer. Even when the compressed image data is distributed and the moving image data and the compressed image data share the bus, the occurrence of instantaneous access congestion on the bus is suppressed.

  In the first, second and third inventions, as described above, predetermined data processing is performed on the data, and the first data unit is equal in data size, or the first data unit is an integer. For each second data unit of the divided data size, the data is output with a predetermined interval and transferred on the bus in the first data unit, so that the data transferred on the bus is distributed. There is an effect.

  The fourth, fifth, and sixth inventions perform processing on moving image data based on the imaging signal output from the imaging element, and perform processing on the still image data based on the imaging signal in parallel with the processing of the moving image data. In order to be able to do this, light is converted into an electrical signal by photoelectric conversion, and a predetermined signal processing is performed on the image pickup signal output from the image pickup unit and output as image data, which is output in response to the first transfer request. The image data is processed at each frame timing to output moving image data, and the image data output in response to the second transfer request is processed at a predetermined timing to output still image data. The still image data processing compresses and encodes the image data, and transfers the compressed image data to the bus in the first data unit. The compression encoding processing converts the compressed image data to the first data The data size is equal to the data unit, or the data is output at a predetermined interval for each second data unit of the data size obtained by dividing the first data unit by an integer. Even when the compressed image data to be distributed is distributed and the bus is shared between the moving image data and the compressed image data, there is an effect of suppressing occurrence of instantaneous access congestion on the bus.

  Embodiments of the present invention will be described below. First, for easy understanding, a system applicable to the present invention will be described. FIG. 1 shows a configuration of an example of an imaging apparatus 1 applicable to an embodiment of the present invention. In this imaging device 1, light incident through the optical system 10 is received by the imaging element 11 and converted into an electrical signal to be an imaging signal, and this imaging signal is connected to the signal processing unit 20 by the signal processing unit 20. The data is recorded on the recording medium 37 by performing predetermined processing using the RAM 35 which is an external memory. The imaging device 1 is configured to record the imaging signal as moving image data on the recording medium 37 and also to the recording medium 37 as still image data captured at a timing according to the imaging instruction. .

  In the following, for the sake of simplicity, the effective pixel count of the image sensor 11 is assumed to be 2560 horizontal pixels and 1920 vertical pixels. The moving image has horizontal 640 pixels and vertical 480 pixels reduced to ¼ in the horizontal and vertical directions, respectively, and the still image uses the effective pixels of the image sensor 11 as they are, and the image frame is horizontal 2560 pixels and vertical. It is assumed that there are 1920 pixels.

  The optical system 10 includes a lens system, a diaphragm mechanism, a focus mechanism zoom mechanism, and the like, and controls a drive unit 13 based on a command of a CPU (Central Processing Unit) 30 to be described later, a manual operation, and the like to perform diaphragm, focus, zoom, and the like. Is to be controlled. Light from the subject enters the image sensor 11 through the optical system 10. The image pickup device 11 is composed of, for example, a CCD (Charge Coupled Device), converts incident light into an electric signal by photoelectric conversion, and outputs the signal as an image pickup signal in line order. The image sensor 11 is not limited to a CCD, and for example, a CMOS (Complementary Metal-Oxide Semiconductor) imager may be used.

  The front end (F / E) unit 12 includes, for example, a noise suppression unit, an automatic gain control unit, and an A / D conversion unit, and is a noise suppression unit for an imaging signal output as an analog signal from the imaging device 11. A correlated double sampling process is performed to suppress noise, and the automatic gain control unit controls the gain. Then, it is converted into a digital imaging signal by an A / D conversion unit and output. This digital image signal is RAW data composed of data directly corresponding to each pixel of the image sensor 11.

  Note that, based on a timing signal (clock pulse) output from a timing generator 14 (to be described later), the image pickup device 11 is controlled so that, for example, photoelectric conversion is performed at each frame timing and an image pickup signal is output, and the front end. The unit 12 is controlled to perform processing in synchronization with the frame timing.

  The signal processing unit 20 includes a camera signal processing unit 22, an extended image processing unit 23, a resolution conversion unit 24, a still image codec (CODEC) unit 25, a moving image codec unit 26, a display control unit 27, a CPU 30, an external interface (I / F). ) Unit 33, memory controller 34, and recording / playback control unit 36, and these processing blocks are connected via the bus 21.

  The camera signal processing unit 22 receives RAW data output from the front end unit 12. The camera signal processing unit 22 corrects the input RAW data to a predetermined value, and transfers the clock from the sensor driving system clock to the internal clock of the signal processing unit 20 using the buffer memory. The RAW data with the clock changed is converted into data in a format suitable for the subsequent moving image processing.

  The RAW data is compressed and encoded in a predetermined manner and sent to the bus 21 when recording a still image. The camera signal processing unit 22 decodes and decompresses the compression code of the compression-coded RAW data supplied via the bus 21, and further converts the data into a format suitable for still image processing in the subsequent stage.

  The camera signal processing unit 22 further enlarges and / or reduces the baseband moving image data and still image data, and converts them into a resolution (size) necessary for recording moving image data and output image data. In this example, for recording moving image data, data with a resolution of 2560 pixels × 1920 pixels is reduced to data with a resolution of 640 pixels × 480 pixels in accordance with the effective pixels of the image sensor 11. For still image recording, in this example, resolution conversion is not performed at this stage. The baseband moving image data and still image data subjected to resolution conversion in this way are output from the camera signal processing unit 22.

  Here, the baseband data is, for example, data obtained by converting an analog signal into a digital signal, and indicates data that has not been subjected to various types of processing such as compression coding and modulation. In this example, a digital image pickup signal obtained by converting an analog signal output from the image pickup element 11 into a digital signal by the front end unit 12 is appropriately referred to as baseband data.

  The extended image processing unit 23 performs extended image processing such as subject recognition processing and ultra-high speed continuous shooting. In the subject recognition process, for example, a part that looks like a face is extracted from an image using a previously learned pattern dictionary or the like. The ultra high-speed continuous shooting is to read out charges from the image sensor 11 and output an image pickup signal at a very high frame rate, for example, 240 fps with respect to the frame rate of 60 fps (frame per second) of normal moving image data. Since the ultra-high speed continuous shooting function requires a higher data transfer capability than normal moving image data, it is preferable to provide a separate processing unit.

  The resolution conversion unit 24 performs processing for converting the resolution of baseband moving image data or still image data to a resolution suitable for display on the display device 28 described later, or converting the resolution to a resolution corresponding to a recording mode.

  The still image codec unit 25 performs compression encoding processing on baseband still image data and decoding processing on compression-encoded still image data. The type of compression encoding method is not particularly limited, but for example, a JPEG (Joint Photographic Experts Group) method is applied. That is, the still image codec unit 25 divides the supplied still image data frame into encoded blocks of a predetermined size such as 8 × 8 pixels, and performs DCT for each encoded block. Then, the DCT coefficient obtained by DCT is quantized with a predetermined quantization scale. The quantized data is further compressed and output by variable length coding such as Huffman coding. The decoding process of the compressed still image data by the still image codec unit 25 is performed by the reverse process of the compression encoding process.

  The moving image codec unit 26 performs compression encoding processing on baseband moving image data and decoding processing on compression encoded moving image data. The type of compression coding system is not particularly limited. For example, MPEG2 (Moving Pictures Experts Group 2) system or ITU-T (International Telecommunication Union-Telecommunication Standardization Sector) recommendation H.264 or ISO (International Organization for Standarization) / IEC (International Electrotechnical Commission) International Standard 14496-10 (MPEG-4 Part 10) Advanced Video Coding (hereinafter abbreviated as H.264 | AVC) or the like can be applied. it can. In the following, it is assumed that the moving picture codec unit 26 applies the MPEG2 system as a compression encoding system.

  For example, the moving image codec unit 26 divides a frame of the supplied moving image data into encoded blocks of a predetermined size such as 8 × 8 pixels, and performs DCT for each encoded block. Then, the DCT coefficient obtained by DCT is quantized with a predetermined quantization scale. In addition, the moving image codec unit 26 also performs interframe coding on the supplied moving image data by predictive coding using motion compensation. Data subjected to intraframe coding and interframe coding is further compressed and output by variable length coding such as Huffman coding. The decoding process of the compressed moving image data by the moving image codec unit 26 is performed by the reverse process of the compression encoding process.

  The display control unit 27 converts the supplied image data into a signal in a format that can be displayed on the display device 28. The display device 28 includes, for example, an LCD (Liquid Crystal Display), is used as a viewfinder in the imaging device 1 and is used as a monitor for an image reproduced from the recording medium 37. In addition, the display control unit 27 generates a display signal for causing the display device 28 to display characters and graphics based on a command supplied from a CPU 30 described later. A display corresponding to the display signal is also displayed on the display device 28. For example, a setting screen for performing various settings for the imaging device 1, a display indicating various states of the imaging device 1, a cursor display, and the like are displayed on the display device 28 in a predetermined manner.

  A ROM (Read Only Memory) 31 and an input device 32 are connected to the CPU 30. The CPU 30 uses a RAM (Random Access Memory) (not shown) as a work memory, and controls the overall operation of the imaging apparatus 1 according to a program stored in advance in a ROM (Read Only Memory) 31.

  For example, the CPU 30 exchanges commands and data with each unit of the signal processing unit 20 via the bus 21 and controls each unit of the signal processing unit 20. Further, the CPU 30 generates a control signal for controlling the focus, aperture, zoom, and the like of the optical system 10 based on a control signal or an imaging signal according to the operation on the input device 32 described above, and supplies the control signal to the drive unit 13. To do. The drive unit 13 controls each unit of the optical system 10 according to the supplied control signal. Further, the CPU 30 issues a command to the timing generator 14 so as to output a predetermined clock pulse.

  In response to this command supplied from the CPU 30, the timing generator 14 generates a timing signal necessary for processing the image signal output from the image sensor 11, such as a frame synchronization signal, a horizontal synchronization signal, and a vertical synchronization signal. .

  The input device 32 is provided with various operators that are used to operate the imaging apparatus 1 and outputs a control signal corresponding to an operation on each operator. For example, a power key for switching ON / OFF of power supply by the power supply unit 39, a mode switching key for switching an operation mode in the imaging apparatus 1 such as a shooting mode and a recording mode, a key for moving a cursor, and the like are provided. Further, the input device 32 is used for operating a shutter button used when taking a still image with the imaging apparatus 1 and operating the focus, aperture, zoom, and the like of the optical system 10 when shooting a moving image or a still image. An operator or the like is also provided.

  The external I / F 33 controls data exchange between the imaging apparatus 1 and an external device.

  A RAM 35 is connected to the memory controller 34. The RAM 35 is, for example, an SDRAM (Synchronous Dynamic Random Access Memory) that operates in synchronization with a memory bus clock, and can input and output data by burst transfer in units of a predetermined data length (burst length). The RAM 35 is used in common with each unit connected to the bus 21. The memory controller 34 controls access to the RAM 35. In each processing block connected to the bus 21, when a transfer request to the RAM 35 occurs, the transfer request is passed from the bus 21 to the memory controller 34, and the memory controller 34 accesses the RAM 35 according to this transfer request. Is generated.

  The recording / reproducing control unit 36 performs recording control of data on the recording medium 37 and reproduction control of data recorded on the recording medium 37. As the recording medium 37, for example, a removable nonvolatile memory can be used. Also, a recordable optical disc can be used as the recording medium 37. Furthermore, a removable hard disk or a built-in hard disk in the imaging apparatus 1 can be used as the recording medium 37. Of course, as the recording medium 37, a magnetic tape conventionally used for recording moving image data can be applied.

  FIG. 2 shows an example of the configuration of the camera signal processing unit 22. The camera signal processing unit 22 includes a RAW correction unit 50, a switch unit 51, a signal processing unit 52, and a resolution conversion unit 53, and includes a RAW compression unit 54 and a RAW expansion unit 56, and bus I / Fs 55, 57, and 58. Have.

  The bus I / Fs 55, 57, and 58 have buffer memories that can temporarily store data in the RAM 35. The bus I / Fs 55, 57 and 58 can read out the data stored in the buffer memory and send it to the bus 21. The bus I / Fs 55, 57, and 58 are configured such that data read from the RAM 35 in units of burst lengths can be temporarily stored in a buffer and output at a predetermined timing.

  The RAW correction unit 50 includes a correction processing unit 50A and a synchronous transfer unit 50B. The correction processing unit 50A performs various correction processes such as defect correction and lens correction on the supplied RAW data, and also performs demosaic processing on the array of RAW data, and performs correction processing and color conversion processing at a later stage. Rearrange to a suitable data array.

  The synchronous transfer unit 50B performs a process of changing the sensor drive system clock to the system clock. That is, the data supplied to the camera signal processing unit 22 is an output of the image sensor 11 and is synchronized with a drive clock when reading out charges from the image sensor 11. On the other hand, the data output from the camera signal processing unit 22 needs to be synchronized with the clock inside the signal processing unit 20. The synchronous transfer section 50B has a buffer memory, writes input data to the buffer memory in accordance with the sensor drive system clock, and reads data from the buffer memory in accordance with the system clock. Data reading from the buffer memory is performed in response to a transfer request supplied to the synchronous transfer unit 50B.

  In response to the transfer request A from the signal processing unit 52, RAW data is read from the buffer memory of the synchronous transfer unit 50B and supplied to the signal processing unit 52 via one terminal 51A of the switch unit 51. The signal processing unit 52 converts the RAW data into an image signal in a format suitable for subsequent processing. For example, the signal processing unit 52 converts the RAW data into a Y / C signal composed of a luminance signal Y and a color difference signal Cb / Cr.

  The signal processing unit 52 further performs predetermined image quality correction processing such as white balance processing, gamma correction processing, and edge enhancement processing on the image signal obtained by converting the RAW data. In general, it is considered that the required image quality differs between a moving image and a still image. Therefore, for example, the signal processing unit 52 can switch parameters for performing the image quality correction processing depending on the control of the CPU 30 depending on whether the processing is performed on moving image data or the processing on still image data. Has been.

  The image signal subjected to signal processing by the signal processing unit 52 is resolution-converted to a resolution corresponding to, for example, the operation mode by the resolution conversion unit 53, and is transmitted via the bus I / F 58 in response to a transfer request C from the bus I / F 58. It is sent to the bus 21.

  The transfer request A from the signal processing unit 52 is also supplied to the bus I / F 57. The bus I / F 57 requests the memory controller 34 to read out the compressed RAW data stored in the RAM 35 via the bus 21 at a predetermined timing in response to the transfer request A, and the RAM 35 responds to this request. The compressed RAW data read from and transferred to the RAW decompression unit 56 is supplied. The RAW expansion unit 56 performs a decoding process on the compressed RAW data supplied via the bus I / F 57 by a process reverse to that of the RAW compression unit 54 described later, and expands the data. The switch 51 selects the terminal 51 </ b> B, and the output of the RAW decompression unit 56 is supplied to the signal processing unit 52 via the terminal 51 </ b> B of the switch unit 51.

  The processing of the signal processing unit 52 and the resolution conversion unit 53 is performed based on a transfer request C supplied from the bus I / F 58 and a response signal to the transfer request A output from the signal processing unit 52. That is, in the signal processing unit 52 and the resolution conversion unit 53, data transfer and signal processing are performed when both the data transfer request from the output counterpart and the data output response from the input counterpart are valid. .

  In response to the transfer request B from the bus I / F 55, RAW data is read from the buffer memory of the synchronous transfer unit 50B and supplied to the RAW compression unit 54. The RAW compression unit 54 compresses and encodes the supplied RAW data and outputs it. In the RAW compression unit 54, it is preferable to apply a compression coding method with less image quality degradation.

  For example, a conversion table that associates pre-conversion data with post-conversion data having a shorter bit length based on a conversion curve similar to a gamma curve, as described in JP-A-2005-311743, It is possible to use a method of compressing and encoding by shortening the data length by referring. In addition to the compression coding method using a conversion table based on a conversion curve similar to a gamma curve as described in Japanese Patent Application Laid-Open No. 2006-352509, compression coding by a DPCM (Defective Pulse Code Modulation) method is further performed. You may use the method which added the system.

  Note that the compression encoding process in the RAW compression unit 54 is performed with emphasis on real-time characteristics with respect to RAW data transfer, and is performed, for example, line-sequentially based on the readout characteristics of the image sensor 11. Not limited to this, a line memory capable of storing a plurality of lines is provided in the RAW compression unit 54, and still image data can be sequentially scanned in the horizontal direction in units of blocks of 8 horizontal pixels × 8 vertical pixels. .

  The compressed RAW data compression-encoded by the RAW compression unit 54 is sent to the bus 21 via the bus I / F 55.

  An example of the operation of the imaging apparatus 1 having such a configuration will be schematically described. In a moving image data recording mode in which moving image data is mainly recorded, light from a subject enters the image sensor 11 via the optical system 10, is converted into an electric signal by photoelectric conversion, and is output as an image signal at each frame timing. . The imaging signal is subjected to predetermined processing such as noise suppression processing and gain control by the front end unit 12, and is further A / D converted and output as RAW data at each frame timing. The RAW data output from the front end unit 12 is input to the signal processing unit 20 and supplied to the camera signal processing unit 22.

  The RAW data is corrected in the camera signal processing unit 22 by the correction processing unit 50A, and is written in the buffer memory by the synchronous transfer unit 50B with the sensor drive system clock. In response to the transfer request A output from the signal processing unit 52, RAW data is read from the buffer memory. At this time, reading of data from the buffer memory is performed according to the system clock, and clock switching is performed. The RAW data output from the synchronous transfer unit 50B is supplied to the signal processing unit 52 via the switch unit 51, converted into baseband moving image data, and the resolution conversion unit 53 performs predetermined resolution conversion. In response to the transfer request C, the moving image data is output from the resolution conversion unit 53, transferred to the bus 21 via the bus I / F 58, and written to the RAM 35.

  The moving image data written in the RAM 35 is read from the RAM 35 and supplied to the resolution conversion unit 24 and the moving image codec unit 26 via the bus 21. The resolution conversion unit 24 converts the resolution of the supplied moving image data into a resolution suitable for display on the display device 28, for example. The moving image data whose resolution is converted by the resolution conversion unit 24 is supplied to the display control unit 37 via the bus 21 and is displayed on the display device 28 as a recording monitor image.

  The moving image data written in the RAM 35 is also supplied to the moving image codec unit 26 via the bus 21. The moving image codec unit 26 performs compression encoding by, for example, the MPEG2 system. The compressed moving image data is supplied to the memory controller 34 via the bus 21 and stored in the RAM 35. The The resolution-converted moving image data is supplied to the memory controller 34 via the bus 21 and stored in the RAM 35.

  Note that the moving image data written in the RAM 35 may be supplied to the extended image processing unit 23 via the bus 21 to perform predetermined image processing, and then supplied to the resolution conversion unit 24. However, the present invention is not limited to this, and the moving image data subjected to resolution conversion by the resolution conversion unit 24 may be supplied to the extended image processing unit 23 via the bus 21 to perform image processing.

  When a predetermined amount or more of the compressed moving image data is accumulated in the RAM 35, the recording / playback control unit 36 reads the compressed moving image data from the RAM 35 for each recording unit of the recording medium 37 and records it on the recording medium 37.

  When recording a still image while recording moving image data, first, for example, during recording of a moving image, a shutter button provided as the input device 32 is pressed in order to instruct recording of a still image. A control signal corresponding to the pressing of the shutter button is supplied from the input device 32 to the CPU 30. Based on this control signal, the CPU 30 controls each unit of the imaging device 1 so as to perform still image recording.

  For example, still image data based on RAW data at the frame timing immediately after the shutter button is pressed is captured and recorded. That is, in the camera signal processing unit 22, the RAW data corresponding to the timing when the shutter button is pressed is subjected to a predetermined correction process by the correction processing unit 50A, and written to the buffer memory by the synchronous transfer unit 50B with the sensor drive system clock. The RAW data written in the buffer memory is read by the system clock in response to the transfer request B, the clock is changed, and is supplied to the RAW compression unit 54. The compressed RAW data compression-encoded by the RAW compression unit 54 is sent to the bus 21 via the bus I / F 55 and stored in the RAM 35 via the bus 21 and the memory controller 34.

  The compressed RAW data stored in the RAM 35 is read from the RAM 35 at a predetermined timing and supplied to the camera signal processing unit 22. The compressed RAW data supplied to the camera signal processing unit 22 is supplied to the RAW expansion unit 56 via the bus I / F 57 in response to the transfer request A, and the compression code is expanded. The expanded RAW data is converted into baseband still image data by the signal processing unit 52, resolution-converted by the resolution conversion unit 53, transferred to the bus 21 via the bus I / F 58, and sent to the memory controller 34. To the RAM 35.

  The RAW data written in the RAM 35 is read out in a predetermined manner, supplied to the still image codec unit 25 via the bus 21, and is compressed and encoded in accordance with, for example, the JPEG method. The compressed still image data is supplied to the recording / playback control unit 36 via the bus 21 and is recorded on the recording medium 37 at a timing when the moving image data is not recorded. The compressed still image data may be temporarily stored in the RAM 35, and the compressed still image data may be read from the RAM 35 and recorded on the recording medium 37 at a timing when moving image data is not recorded on the recording medium 37.

  Note that when still image data is recorded during recording of moving image data, the RAW data is compressed and encoded in the camera signal processing unit 22 and the compressed RAW data is written to the RAM 35 in order not to stop recording of the moving image data. It is necessary to perform in real time in synchronization with the output of the image sensor 11.

  On the other hand, reading of the compressed RAW data stored in the RAM 35, decompression processing and signal processing of the read RAW data in the camera signal processing unit 22, compression encoding processing in the still image codec unit 25, recording of compressed still image data The supply to the reproduction controller 36 is performed at a timing when the bus 21 is not congested, such as a drive signal blanking period in the image sensor 11. Since signal processing for still image data does not require real-time processing, it is possible to perform signal processing at low speed using software processing in the CPU 30 or a general-purpose DSP (Digital Signal Processor).

  Next, processing in the camera signal processing unit 22 will be described. FIG. 3 shows an example of a data path for each operation mode of the camera signal processing unit 22. In FIG. 3, the memory controller 34 is omitted. FIG. 3A shows an example data path in the moving image data recording mode. In the moving image data recording mode, processing as moving image data for the RAW data output from the image sensor 11 and supplied to the camera signal processing unit 22 via the front end unit 12 and the moving image data are displayed on the display device 28. Monitoring process is performed. In the moving image data recording mode, these processes are performed as processes for each frame timing.

  In the moving image data recording mode, the terminal 51A is selected in the switch unit 51, and the RAW data output from the RAW correction unit 50 at each frame timing, for example, is output from the signal processing unit 52 as illustrated by a thick line in FIG. In response to the transfer request A, the signal is supplied to the signal processing unit 52 and the resolution conversion unit 53 via the switch unit 51. Data is output from the resolution converter 53 in response to a transfer request C from the bus I / F 58, and this data is transferred to the bus 21 via the bus I / F 58 and written to the RAM 35. The data written in the RAM 35 is supplied to the moving image codec unit 26 and the display control unit 27 in the subsequent stage via the bus 21.

  In this moving image data recording mode, it is necessary to perform the processing from the RAW correction unit 50 to the writing of data to the RAM 35 in real time. Note that the real time here means that processing is possible without impairing temporal continuity of data. More specifically, in the case of moving image data, real-time processing is established when data transmission and data processing in each unit are performed without loss of continuity of frames.

  3B and 3C show an example data path in the still image data recording mode. In the still image data recording mode, the data path for reading the imaging signal from the imaging element 11 is different from the data path for performing signal processing on the RAW data obtained by converting the imaging signal.

  FIG. 3B shows an example of a data path associated with image signal readout. In this case, as illustrated by a bold line in FIG. 3B, RAW data is output from the RAW correction unit 50 in response to the transfer request B from the bus I / F 55, and the RAW compression unit 54 compresses the RAW data to a predetermined compression code. And supplied to the bus I / F 55. The compressed RAW data is transferred from the bus I / F 55 to the bus 21 and written to the RAM 35.

  FIG. 3C shows an example data path when signal processing is performed on the compressed RAW data written in the RAM 35. The compressed RAM data written in the RAM 35 is read out in a predetermined manner. The compressed RAW data read from the RAM 35 is supplied to the RAW decompression unit 56 via the bus 21 and the bus I / F 57, and is supplied to the signal processing unit 52 via the switch unit 51 in which the terminal 51B is selected. The RAW data is subjected to predetermined signal processing by the signal processing unit 52 and the resolution conversion unit 53, output from the resolution conversion unit 53 in response to the transfer request C from the bus I / F 58, and via the bus I / F 58 and the bus 21. It is written again in the RAM 35. The RAW data written in the RAM 35 is supplied to the subsequent still image codec unit 25 and the like via the bus 21.

  Since processing for still image data does not require real-time processing, reading of compressed RAW data from the RAM 35 can be performed, for example, at a timing when there is a margin in the access bandwidth of the bus 21 or RAM 35.

  Next, the processing in the camera signal processing unit 22 will be described in more detail. As described above, the synchronous transfer unit 50B temporarily stores the RAW data input in synchronization with the clock of the sensor drive system in the buffer memory, and performs reading from the buffer memory in synchronization with the system clock. , Changing clocks. It is assumed that the buffer memory has at least a capacity capable of absorbing clock transfer and instantaneous access delay in the bus 21. If writing to the buffer memory occurs without reading from the buffer memory, the buffer memory overflows, and the buffer function fails.

  In this way, as an example of buffer failure due to the difference in the reading / writing speed of RAW data to / from the buffer memory, the data transfer amount increases on the bus 21 to which the compressed RAW data compressed and encoded by the RAW compression unit 54 is sent out. The bus 21 may be congested. For example, when the bus 21 is congested due to transfer of data having a higher priority than the compressed RAW data, the bus I / F 55 waits for processing for sending the compressed RAW data to the bus 21 until the transfer of the data is completed, for example. . Alternatively, the RAW compression unit 54 is controlled to stop the RAW data compression process, and wait for the transfer of the data to end and the buffer memory to become empty.

  In addition, when the RAW compression unit 54 performs compression encoding of RAW data using variable length encoding in which the compression rate varies according to the spatial frequency component of the RAW data, if there are many high-frequency components of the RAW data to be compression encoded, In some cases, the amount of data after compression encoding becomes large. Even in such a case, a buffer failure may occur.

  If there is a possibility that a buffer failure has occurred, in order to continue recording of still image data, it is necessary to stop the RAW data compression encoding process and the preceding process and wait for the buffer memory to become empty. is there. The preceding process includes, for example, output of an imaging signal from the image sensor 11, output of RAW data from the front end unit 12, and correction processing in the correction processing unit 50A. However, when still image data is recorded during the recording of moving image data, it is necessary to continuously record moving image data, and therefore these preceding processes cannot be stopped.

  Therefore, the synchronous transfer unit 50B monitors the state of the buffer memory, and performs a failure process when a buffer failure is detected. FIG. 4 shows an example of the configuration of the synchronous transfer unit 50B configured to perform such processing. The synchronous transfer unit 50B includes a buffer memory 60 and a read control unit 61. The buffer memory 60 is a first-in first-out (FIFO) control memory, and writes data in response to a write enable signal supplied from the preprocessing side. The read control unit 61 issues a data read request to the buffer memory 60 based on the transfer request A and transfer request B described above and the capacity of the buffer memory 60. The capacity of the buffer memory 60 may only indicate, for example, whether the buffer memory is full (full) or empty (empty).

  When the capacity of the buffer memory 60 is not empty and both the transfer request A and the transfer request B are satisfied, the read control unit 61 reads the data from the buffer memory 60 by making the read request to the buffer memory 60 active. And output to the signal processing unit 52 and the RAW compression unit 54.

  Here, the read control unit 61 determines whether there is a possibility that the buffer has failed based on the capacity of the buffer memory 60 and the transfer request A and the transfer request B. That is, when the capacity of the buffer memory 60 is full and at least one of the transfer request A and the transfer request B is not established, it is determined that the buffer may have failed. Whether or not the transfer request is established can be determined by determining whether or not the transfer request has arrived at the read control unit 61 at the timing of processing to be performed according to the transfer request, for example.

  FIG. 5 is a flowchart illustrating an example of processing for determining a buffer failure. First, in step S100, it is determined whether or not the buffer memory 60 is full. If it is determined that the state is not full, the process proceeds to step S101, and, for example, a normal data transfer process is performed.

  On the other hand, if it is determined in step S100 that the buffer memory 60 is full, the process proceeds to step S102, and it is determined whether transfer request A and transfer request B are established. If it is determined that both transfer request A and transfer request B are established, the process proceeds to step S101, and normal transfer processing is performed.

  If it is determined in step S102 that transfer request A or transfer request B is not established, the buffer may have failed. At this time, when it is determined that only one of the transfer request A and the transfer request B is not established, this is called partial failure (step S103), and the cause of occurrence is held in, for example, an internal register of the read control unit 61 ( Step S104). If it is determined that neither transfer request A nor transfer request B is established, it is called complete failure (step S105), and the cause is held in the internal register (step S106). The CPU 30 can know whether partial failure or complete failure has occurred in the unit of frame processing by reading the internal register of the read control unit 61.

  For example, information on a transfer request that is not established can be used as the generation factor held in the internal register of the read control unit 61. As an example, when it is determined that the transfer request A is not established and there is a possibility that the request has partially failed, it is possible to hold information indicating the transfer request A in an internal register of the read control unit 61. As another example, when it is determined that neither transfer request A nor transfer request B is established, and there is a possibility that the request has completely failed, information indicating transfer request A and transfer request B is read out from controller 61. It can be held in an internal register.

  Next, an embodiment of the present invention will be described. First, the difference in output rate depending on the type of RAW data compression encoding method will be described with reference to FIG. Hereinafter, a case where a code having a predetermined length is assigned to a compression coding unit of image data (RAW data) is referred to as fixed-length coding, and a code length assigned to the compression coding unit. Is determined is called variable length coding. The compression coding unit is, for example, a single pixel, a plurality of adjacent pixels (such as a pixel block), or a pixel constituting one or a plurality of adjacent lines.

  When the RAW data is not compressed and encoded, for example, when the RAW data is simply passed through the RAW compression unit 54 (referred to as compression processing through), the output rate is substantially equivalent to the read rate from the image sensor 11. Become. When compression encoding of RAW data is performed with fixed-length encoding, the output rate is reduced by the amount of RAW data compression compared to the case of only passing RAW data. On the other hand, when compression encoding of RAW data is performed by variable-length encoding, for example, the amount of code allocated varies depending on the picture pattern for each compression encoding unit, so the output rate is also included in this encoding process. Fluctuate accordingly.

  When the SDRAM is used as the external memory (RAM 35) for storing the compressed RAW data as in the above-described imaging apparatus 1, in the fixed length encoding, the fixed code length of the compression encoding unit is the burst length of the SDRAM. In the variable length encoding process, the transfer rate may locally become dense with a change in the code length. In such a case, it is difficult to predict the maximum transfer rate, and it becomes difficult to ensure the memory bandwidth of the entire system and to guarantee real-time performance in reading from the image sensor 11.

  Therefore, in the embodiment of the present invention, the output of the compression encoding process is performed in units of data length that is the same as or shorter than the burst length of the SDRAM used as the external memory. That is, the compressed RAW data is output from the RAW compression unit 54 with a data length equal to or shorter than the burst length of the RAM 35 to which the compressed RAW data output from the RAW compression unit 54 is written. The compressed RAW data output from the RAW compression unit 54 in this data unit is supplied to the bus I / F 55, transferred to the bus 21 in units of the burst length of the RAM 35, and written to the RAM 35.

  Thus, when outputting the compressed RAW data from the RAW compression unit 54 to the bus I / F 55 for writing to the RAM 35, the data unit of the compressed RAW data is equal to the burst length of the RAM 35, or the burst By setting the data length shorter than the length, the transfer data amount transferred from the bus I / F 55 to the bus 21 is smoothed. As a result, suddenly large-capacity data transfer from the bus I / F 55 to the bus 21 does not occur, and the congestion of the bus 21 is alleviated. Therefore, it is possible to suppress the occurrence of the above-mentioned complete failure or partial failure.

  FIG. 7 schematically shows a configuration of an example of the RAW compression unit 54 according to the embodiment of the present invention. In the example of FIG. 7, the RAW compression unit 54 includes a quantization unit 70, an encoding unit 71, a buffer control unit 72, and a code packing buffer 73. The quantization unit 70 is supplied with, for example, RAW data output from the RAW correction unit 50. The quantizing unit 70 compresses the data by quantizing the supplied RAW data to a predetermined value and converting data consisting of continuous values into data having discrete values. In the above example, for example, the quantization unit 70 performs processing for associating RAW data with a conversion curve similar to a gamma curve.

  The encoding unit 71 encodes the quantized data obtained by quantizing the RAW data by the quantizing unit 70 according to a predetermined rule. In the above-described example, for example, the conversion table is referred to based on the result of the process of associating the RAW data with the conversion curve similar to the gamma curve in the quantization unit 70, and the code corresponding to the result of the process is output. By configuring this conversion table so that a fixed-length code is output for a certain amount of RAW data, it is possible to perform compression encoding of the RAW data using a fixed-length code. On the other hand, this conversion table is configured to output a variable-length code for a certain amount of RAW data, so that the RAW data can be compressed and encoded using the variable-length code. In this case, for example, it is conceivable to use a conversion table having a code length determined based on the statistical frequency of the result of the above processing.

  The code packing buffer 73 temporarily stores the codes encoded by the encoding unit 71 with a fixed length or a variable length. The code packing buffer 73 has, for example, two memories (referred to as A-plane and B-plane, respectively) having a predetermined capacity. The two memories are written from the encoding unit 71 side and output side (bus I / O). The exclusive processing is realized by reading from the F55 side). Each surface of the code packing buffer 73 has, for example, at least a burst length capacity of the RAM 35 into which the compressed RAW data output from the RAW compression unit 54 is written. In this example, the capacity of the A side and the B side of the code packing buffer 73 is 64 bytes.

  Note that the code packing buffer 73 is not limited to a two-plane configuration, and may be configured by a FIFO-controlled memory.

  The buffer control unit 72 controls writing and reading in the code packing buffer 73. For example, the buffer control unit 72 acquires the data capacity written in the code packing buffer 73 and performs access control in the code packing buffer 73 based on this capacity. For example, the buffer control unit 72 performs control such as switching between the A side and the B side and data writing and reading timings on the A side and / or the B side based on the capacity. In addition, the operation | movement which switches the A surface and B surface of a buffer is called buffer surface flip.

  The buffer control unit 72 reads data from the code packing buffer 73 based on a transfer request from the bus I / F 55 and determines that the code packing buffer 73 is in a state where data can be written. Outputs a transfer request B. As an example, if a transfer request is not output from the bus I / F 55 to the buffer control unit 72 due to a factor such as the bus 21 being congested, data reading from the code packing buffer 73 is not performed. If the capacity of the code packing buffer 73 is full, writing to the code packing buffer 73 cannot be performed, so that the transfer request B is not output.

  Next, an example of data transfer control when compression encoding of RAW data is performed using fixed-length encoding will be described more specifically. Here, the data length of one pixel is 16 bits, and the burst length of the external memory (RAM 35) is 32 bytes. Further, as described above, the bus I / F 55 has a buffer memory that can temporarily store data in units of the burst length of the RAM 35, and outputs data read from the buffer memory to the bus 21. .

  FIG. 8 shows a timing chart of an example of compressed RAW data transfer in the case of compression processing through. FIG. 8A shows an example of the output of the RAW compression unit 54, and FIG. 8B shows an example of the output of the bus I / F 55. The output of the bus I / F 55 is transferred to the RAM 35 via the bus 21. In the case of compression processing through, the RAW compression unit 54 outputs the RAW data input for each pixel as it is (FIG. 8A). The output data is supplied to the bus I / F 55. When the pixel data is not thinned out in the image sensor 11, the output data transfer rate is equivalent to the readout rate of the image sensor 11. In the bus I / F 55, the supplied data is stored in a buffer. The data stored in the buffer is read out in a predetermined manner and output to the bus 21 (FIG. 8B). In this example in which the data length of one pixel is 16 bits, burst transfer to the RAM 35 is performed in units of 16 pixels.

  FIG. 9 shows an example timing chart when the RAW compression unit 54 outputs data by a conventional method. Here, for ease of explanation, the RAW compression unit 54 sets the compression encoding unit to 64 pixels and the compression rate to 1/2. Therefore, 64 bytes (= 16 bits × 64 pixels × 1/2) of compressed RAW data is output for each compression encoding unit.

  The RAW compression unit 54 stores the compression-encoded RAW data in the buffer. When the 64-byte compressed RAW data is stored, the RAW compression unit 54 switches between the writing surface and the reading surface and reads the compressed RAW data from the buffer. The read compressed RAW data is supplied to the bus I / F 55 (FIG. 9A). Since the bus I / F 55 outputs every time 32 bytes of compressed RAW data is supplied, as illustrated in FIG. 9B, the burst transfer in units of 32 bytes is continuously performed twice. Become.

  In this case, for example, compared with the case of the compression through shown in FIG. In addition, on the side where processing is waited, it may be necessary to provide a buffer for ensuring the memory bandwidth for processing, and in this case, the cost of the apparatus is increased.

10 and 12 are timing charts showing examples of data transfer according to the embodiment of the present invention. FIG. 10 shows an example in which the data unit output by the compression encoding process is the same as the burst length of the SDRAM used as the external memory (RAM 35). For example, the RAW compression unit 54 reads and outputs 32 bytes of compressed RAW data stored in the buffer, and when the predetermined interval T _int elapses, the remaining 32 bytes of the compression encoding unit stored in the buffer The compressed RAW data for the minute is read and output (FIG. 10A).

As described above, the output from the RAW compression unit 54 is performed with a data length corresponding to the burst length of the RAM 35 as a unit, with an interval T _int for each compression encoding unit, so that burst transfer by the bus I / F 55 is also possible. , In a distributed manner as illustrated in FIG. 10B. As a result, the time for waiting for other processes sharing the RAM 35 can be shortened compared to the transfer method shown in FIG.

  11 shows an example of the code packing buffer 73 in the RAW compression unit 54 when the data unit output by the compression encoding process shown in FIG. 10 is the same as the burst length of the SDRAM used as the external memory. It is a flowchart which shows control. In the initial state, it is assumed that the code packing buffer 73 has the A surface as a writing surface and the B surface as a reading surface. Further, as described above, in the code packing buffer 73, the A side and the B side each have a capacity of 64 bytes, and the burst length of the external memory (RAM 35) is 32 bytes.

  The compression encoding process is started using, for example, a frame synchronization signal of the image sensor 11 as a trigger. When the compression encoding process is started, the compressed RAW data output from the encoding unit 71 is written on the writing surface of the code packing buffer 73. Write control on the code packing buffer 73 will be described later. In step S10, it is determined whether or not the capacity of the data written on the writing surface has reached 64 bytes. If it is determined that the capacity has reached 64 bytes, the process proceeds to step S11. In step S11, the buffer surface of the code packing buffer 73 is flipped, and the surface that has been the writing surface so far becomes a new reading surface, and the surface that has been the reading surface becomes a new writing surface. It should be noted that the writing of the compressed RAW data is carried over to the original writing surface on the new writing surface.

In the next step S12, 32-byte compressed RAW data that is the same as the burst length of the external memory is read from the reading surface. The read compressed RAW data is output from the RAW compression unit 54 and supplied to the bus I / F 55. In step S13, awaited is processed by a distance T _int, after a lapse of the interval T _int, in step S14, the reading surface of the code packing buffer 73, the remaining 32 bytes of 32 bytes in step S12 described above is read Minutes of data are read out.

  Then, the process is returned to step S10, and it is awaited that the compressed RAW data for 64 bytes is written on the writing surface of the code packing buffer 73.

Note that the interval T _int is preferably selected so that data transfer to the bus 21 is distributed. For example, it is set to a value that is shorter than the period in which the pixel data for the compression encoding unit is supplied to the RAW compression unit 54 and that the output data for every 32 bytes is not continuously output. In consideration of dispersion of burst transfer to the RAM 35, for example, it is conceivable to set the interval T _int so as to equally divide the period during which data for the compression encoding unit is output from the RAW compression unit 54. As a more specific example, the number of clocks obtained by converting the time corresponding to the readout cycle of 32 pixels, which is ½ of the compression encoding unit, from the image sensor 11 into the clock frequency for driving the RAW compression unit 54. Set.

FIG. 12 shows an example in which the data unit output by the compression encoding process is shorter than the burst length of the SDRAM used as the external memory (RAM 35). The data unit output by the compression encoding process is, for example, a size obtained by dividing the burst length of the external memory by an integer. In the example of FIG. 12, the compressed RAW data of 64 pixels (in the case of a compression ratio of 1/2) for 64 pixels, which is a compression encoding unit, is divided into 8 parts, and the RAW compression unit according to the interval T _int every 8 bytes. 54 is output (FIG. 12A). In this example, the interval T _int is based on a time corresponding to a readout cycle of 8 pixels, which is 1/8 of the compression encoding unit, from the image sensor 11.

  The bus I / F 55 stores the compressed RAW data supplied from the RAW compression unit 54 every 4 bytes in the buffer, and compresses the compressed RAW data for 32 bytes corresponding to the burst length of the RAM 35 from the buffer. RAW data is read out and burst transferred to the RAM 35 (FIG. 12B).

  In this case, the unit of data output from the RAW compression unit 54 depends on the connection relationship with the bus I / F 55 and the buffer when the code packing buffer 73 of the RAW compression unit 54 is configured as a FIFO control memory. It can be set appropriately according to the capacity.

  13 shows an example of the code packing buffer 73 in the RAW compression unit 54 when the data unit output by the compression encoding process shown in FIG. 12 is made shorter than the burst length of the SDRAM used as the external memory. It is a flowchart which shows control. In the initial state, it is assumed that the code packing buffer 73 has the A surface as a writing surface and the B surface as a reading surface.

  Here, in the code packing buffer 73, the A and B surfaces each have a capacity of L bytes, and data is output in units of M bytes by compression encoding processing. Here, L bytes are at least the capacity of the burst length of the external memory, and M bytes are equal to the burst length or have a size obtained by dividing the burst length by an integer. Therefore, reading from one buffer surface of the code packing buffer 73 is completed by performing data transfer L / M = N times from the RAW compression unit 54 to the bus I / F 55.

Prior to the start of the flowchart, the transfer count C _t is set to zero. When the compression encoding process is started, the compressed RAW data obtained by compressing the RAW data is written to the writing surface of the code packing buffer 73. In step S20, it is determined whether or not the data capacity written on the writing surface has reached L bytes. If it is determined that the data has reached L bytes, the process proceeds to step S21, and the buffer surface is flipped and written. The surface and the readout surface are switched. The compressed RAW data is written on the newly written surface.

In the next step S22, M bytes of compressed RAW data are read from the reading surface. The read compressed RAW data is output from the RAW compression unit 54 and supplied to the bus I / F 55. In step S23, the transfer count _Ct is incremented by 1, and in step S24, processing is waited for the interval _Tint . In the next step S25, whether the number of transfers C _t has reached N times or not. If it is determined that it has not reached, the process returns to step S22, the next M bytes are transferred, the transfer count _Ct is incremented (step S23), and a wait process is performed at the interval T _int ( Step S24).

On the other hand, in step S25, the number of transfers C _t is when it is determined that reaches N times, the processing routine advances to step S26, the number of transfers Ct is zero. Then, the process returns to step S20 to wait for L bytes of compressed RAW data to be written on the writing surface of the code packing buffer 73.

  In the flowchart of FIG. 13, if the L byte is 64 bytes and the M byte is 8 bytes, the state of the timing chart of FIG. If the L byte is 64 bytes and the M byte is 32 bytes, it matches the state of the timing chart of FIG. 10 described above, and is equivalent to the process of the flowchart of FIG. As described above, an example in which the data unit output by the compression encoding process is the same as the burst length of the external memory is a special case of an example in which the data unit is shorter than the burst length of the external memory. It can be said that there is.

  FIG. 14 is a flowchart showing an example of write control for the code packing buffer 73. The processing shown in FIG. 14 is the same as the case where the data unit output by the compression encoding processing described with reference to FIGS. 10 and 11 is the same as the burst length of the SDRAM used as the external memory. This is applicable in common with the case where the data unit described with reference to 13 is shorter than the burst length of the external memory.

  In the initial state of the code packing buffer 73, it is assumed that the A surface is the writing surface and the B surface is the reading surface. Further, as described above, in the code packing buffer 73, the A side and the B side each have a capacity of 64 bytes, and the burst length of the external memory (RAM 35) is 32 bytes.

  When the compression coding process is started in the quantization unit 70 and the coding unit 71, a data request is made from the code packing buffer 73 to the coding unit 71 in step S30. For example, in the RAW compression unit 54, the buffer control unit 72 requests the encoding unit 71 to output the compressed and encoded compressed RAW data. The compressed RAW data output from the encoding unit 71 is written on the writing surface of the code packing buffer 73 (step S31). When the data capacity written on the writing surface of the code packing buffer 73 reaches a predetermined capacity, for example, 64 bytes, the upper limit of the capacity of the buffer surface (step S32), the process proceeds to step S33.

  In step S33, it is determined whether data reading from the reading surface of the code packing buffer 73 has been completed. That is, in the code packing buffer 73, when the writing to the writing surface has already reached a predetermined capacity, for example, 64 bytes, the buffer surface is flipped as will be described later, and data is written to the newly written surface. At the same time, control is performed so that data is read out from the newly designated surface.

  If it is determined in step S33 that reading from the reading surface of the code packing buffer 73 has been completed, the process proceeds to step S34, the buffer surface is flipped, and the process returns to step S30.

  On the other hand, if it is determined in step S33 that reading from the reading surface of the code packing buffer 73 has not been completed, the process proceeds to step S35, and a failure process is performed. That is, in the code packing buffer 73, if the reading from the reading surface is not completed, the buffer surface cannot be flipped. On the other hand, if data is written up to the upper limit of the buffer capacity on the writing surface of the code packing buffer 73, it is necessary to flip the buffer surface. Therefore, the buffer function is broken if the reading from the reading surface is not completed regardless of the data being written to the upper limit of the buffer capacity on the writing surface.

  In step S35, relief processing is performed when the buffer function fails in this way. More specifically, while the buffer function is broken, the empty compression process is performed, and the coordinates on the RAW data of the pixels subjected to the empty compression process are held in a predetermined manner. For example, at least the coordinates at which the empty compression process is started and the coordinates at which the empty compression process is ended are held in a register or the like. In addition, an empty compression process is performed while forcibly issuing a transfer request B to the RAW correction unit 50. Thereby, it is possible to accurately know the failure position. Further, a flag indicating that the buffer function has failed (referred to as a failure flag) is held in a predetermined register or the like. For example, the CPU 30 can read out the failure flag to execute appropriate failure relief processing.

  In the empty compression process, for example, the compressed RAW data compression-encoded for each compression encoding unit (64 pixels in this example) in the quantization unit 70 and the encoding unit 71 is not written in the code packing buffer 73. Like that.

  As an example, it is also conceivable that the quantization unit 70 and the encoding unit 71 actually perform compression encoding processing, discard the compressed RAW data generated as a result, for example, and not output from the encoding unit 71. That is, the RAW data is output from the RAW correction unit 50 and writing according to the RAW data is not generated in the code packing buffer 73 until recovery from the failure of the buffer function by the empty compression process.

  However, the RAW compression unit 54 may not perform any processing on the supplied RAW data. In this case, for example, it may be possible to discard the RAW data and output only a status indicating that compression encoding has been performed.

  When the failure processing is performed in step S35, it is determined in the next step S36 whether or not reading from the reading surface of the code packing buffer 73 is completed. If it is determined that the process has not been completed, the process returns to step S35. On the other hand, if it is determined that the reading has been completed, the process proceeds to step S37, and the failure processing is performed at the position corresponding to the pixel on which the empty compression processing has been performed on the writing surface of the code packing buffer 73. A broken code indicating that it has been broken is inserted. Then, the process returns to step S30.

  If the total transfer amount of data for one frame image is not a multiple of the capacity of each side of the code packing buffer 73 (in this example, 64 bytes), the final image of one frame to the code packing buffer 73 will be described. It is detected that the code has been written, and the compression encoding operation is terminated in a predetermined manner. That is, when the RAW compression unit 54 detects that the final RAW data of one frame image has been written in the code packing buffer 73, it waits for the completion of reading of data from the reading surface of the code packing buffer 73. The buffer plane is forcibly flipped to read all the data from the new read plane and finish the compression encoding operation.

  There are various methods for detecting the final code of an image of one frame. As an example, the RAW compression unit 54 can measure the number of RAW data supplied from the RAW correction unit 50 and determine the final code of an image of one frame based on the measured number of RAW data. For example, the measurement may be performed using a frame pulse as a reference. As another example, when the RAW correction unit 50 outputs the final RAW data of an image of one frame and supplies it to the RAW compression unit 54, the RAW compression unit 54 indicates that the RAW data is the final data of one frame. It can also be notified.

  In the above-described step S33, in the case of the first writing to the code packing buffer 73, the data to be read is not yet written on the reading surface. Therefore, at the time of the first writing, the determination process in step S33 is omitted, and the process is directly transferred from step S32 to step S34.

  Next, a modification of the embodiment of the present invention will be described. In the above-described embodiment, the RAW compression unit 54 performs compression encoding of RAW data using a fixed-length code. On the other hand, the modification of this embodiment is an example in which the RAW compression unit 54 performs compression encoding of RAW data using a variable length code.

  FIG. 15 is a timing chart showing an example of the case where the RAW compression unit 54 performs compression encoding using a variable length code and outputs data by a conventional method. Similarly to the above, the RAW compression unit 54 sets the compression encoding unit to 64 pixels and the compression rate to 1/2. Therefore, 64 bytes (= 16 bits × 64 pixels × 1/2) of compressed RAW data is output for each compression encoding unit.

  FIG. 15A shows a case where compressed RAW data compressed and encoded in a compression encoding unit (64 pixels in this example) is subjected to compression encoding using a variable length code while the data size and output timing are changed. , The state output from the RAW compression unit 54 is shown.

  Since the bus I / F 55 outputs data every time 32 bytes of compressed RAW data is supplied, as shown in an example in FIG. It is performed so as to be continued a number of times according to the size. In the example of FIG. 15B, one burst transfer is performed for 30-byte compressed RAW data, and two burst transfers are continuously performed for 40-byte compressed RAW data. Further, four burst transfers are continuously performed on the 120-byte compressed RAW data, and five burst transfers are continuously performed on the 130-byte compressed RAW data.

  As described above, when compression encoding is performed using a variable-length code, the compression rate varies in units of compression encoding according to the design and the like, and the transfer density of data to the external memory also varies. When the compression ratio is deteriorated, a difference between a period when access due to data transfer to the external memory is dense and a period when the access is sparse becomes large, and it becomes difficult to secure a memory bandwidth of the external memory.

  16 and 17 are timing charts showing an example of data output in the RAW compression unit 54 according to a modification of the embodiment of the present invention. FIG. 16 shows an example in which the data unit output by compression encoding is the same as the burst length of the SDRAM used as the external memory (RAM 35), as described with reference to FIG.

In the RAW compression unit 54, the compressed RAW data compressed and encoded using the variable length code is written to the A and B surfaces of the code packing buffer 73 every 64 bytes in accordance with the process already described with reference to the flowchart of FIG. In accordance with the processing described with reference to the flowchart of FIG. 11 or FIG. 13, data is read from the A and B surfaces of the code packing buffer 73 every 32 bytes corresponding to the burst length of the RAM 35 according to the interval T _int (FIG. 15A). . Bus I / F55 from the RAW compression unit 54 every 32 bytes, the compressed RAW data supplied at intervals T _int, burst transfers to RAM35 every 32 bytes is a burst length (Figure 15B).

Therefore, as illustrated in FIG. 15B, access by data transfer to the RAM 35 can be distributed in units of burst length. At this time, the shortest access interval to the RAM 35 can be defined by the interval T _int . Therefore, it becomes easy to guarantee the memory bandwidth of the entire system. This can also prevent concentration of memory access, so that peak current can be suppressed.

FIG. 17 shows an example in which the data unit output by the compression encoding process is made shorter than the burst length of the SDRAM used as the external memory (RAM 35), as described with reference to FIG. In the example of FIG. 17, similarly to FIG. 12, 64 bytes of compressed RAW data for 64 pixels, which are compression coding units, are divided into 8 and output from the RAW compression unit 54 at intervals of 8 bytes according to the interval T _int. I am doing so.

That is, in the RAW compression unit 54, the compressed RAW data compressed and encoded using the variable length code is applied to the A side and the B side of the code packing buffer 73 every 64 bytes according to the process already described with reference to the flowchart of FIG. In accordance with the processing described with reference to the flowchart of FIG. 13, the data is read from the A and B surfaces of the code packing buffer 73 every 8 bytes according to the interval T _int (FIG. 17A). The bus I / F 55 stores compressed RAW data supplied from the RAW compression unit 54 with an interval T _int every 8 bytes in a buffer, and 32 bytes of compressed RAW data corresponding to the burst length of the RAM 35 is buffered. Then, the compressed RAW data is read from the buffer and burst transferred to the RAM 35 (FIG. 17B).

Even in this case, as in the case where the data unit output by compression encoding described with reference to FIG. 16 is the same as the burst length of the external memory, the access by the data transfer to the RAM 35 is distributed in units of the burst length. Can be made. At this time, the shortest access interval to the RAM 35 can be defined by the interval T _int . Therefore, it becomes easy to guarantee the memory bandwidth of the entire system. This can also prevent concentration of memory access, so that peak current can be suppressed.

It is a block diagram which shows the structure of an example of the imaging device applicable to one Embodiment of invention. It is a block diagram which shows the structure of an example of a camera signal processing part. It is a basic diagram which shows the example of the data path for every operation mode of a camera signal processing part. It is a block diagram which shows the structure of an example of a synchronous transfer part. It is a flowchart which shows a process of an example of a buffer failure determination. It is a basic diagram for demonstrating the difference in the output rate by the classification of the compression encoding system of RAW data. It is a block diagram which shows roughly the structure of an example of the RAW compression part by one Embodiment of invention. 6 is a timing chart of an example of compressed RAW data transfer in the case of compression processing through. 5 is a timing chart of an example when data is output by a conventional method in a RAW compression unit. 6 is a timing chart illustrating an example of compressed RAW data transfer when the data unit output by compression encoding processing is the same as the burst length of the external memory. It is a flowchart which shows read-out control of an example of a code packing buffer when the data unit output by compression encoding processing is made the same as the burst length of the external memory. It is a timing chart of an example of compressed RAW data transfer when the data unit output by the compression encoding process is shorter than the burst length of the external memory. It is a flowchart which shows reading control of an example of a code packing buffer when the data unit output by compression encoding processing is made shorter than the burst length of an external memory. It is a flowchart which shows an example of write-in control with respect to a code packing buffer. It is a timing chart of an example of compressed RAW data transfer when compression encoding is performed using a variable-length code and data is output by a conventional method. 10 is a timing chart of an example of compressed RAW data transfer when the data unit output by compression encoding is the same as the burst length of the external memory. It is a timing chart of an example of compressed RAW data transfer when the data unit output by the compression encoding process is shorter than the burst length of the external memory.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image pick-up device 11 Image pick-up element 21 Bus 22 Camera signal processing part 30 CPU
34 Memory controller 35 RAM
50 RAW Correction Unit 50A Correction Processing Unit 50B Synchronous Transfer Unit 54 RAW Compression Units 55, 57, 58 Bus I / F
56 RAW expansion unit 60 buffer memory 61 read control unit 70 quantization unit 71 encoding unit 72 buffer control unit 73 code packing buffer

Claims (26)

A data transfer unit for transferring data on the bus in a first data unit;
Predetermined data processing is performed on the data, and the first data unit is equal in data size to the first data unit, or for each second data unit having a data size obtained by dividing the first data unit by an integer A data processing unit for outputting at intervals,
A data processing device for transferring data output from the data processing unit onto the bus from the data transfer unit. The data processing apparatus according to claim 1,
A memory connected to the bus and capable of burst transfer in units of burst length;
A data processing device, wherein data is transferred to the memory via the bus by the data transfer unit, and the data size of the first data unit is made equal to the burst length. The data processing apparatus according to claim 1,
The data processing apparatus, wherein the data processing performed by the data processing unit is compression encoding processing. The data processing apparatus according to claim 1,
The data processing device, wherein the data processing unit performs the data processing using a fixed-length code. The data processing apparatus according to claim 1,
The data processing device, wherein the data processing unit performs the data processing using a variable length code. The data processing apparatus according to claim 1,
The data processing unit
A buffer having at least a capacity corresponding to the first data unit and temporarily storing the data subjected to the data processing;
A data processing apparatus, wherein the data stored in the buffer is read for each of the first data units. The data processing apparatus according to claim 6, wherein
Based on the access status to the buffer and the transfer request from the data transfer unit, it is determined whether or not the function of the buffer has failed, and when it is determined that the function has failed, predetermined processing is performed. Characteristic data processing device. The data processing apparatus according to claim 7, wherein
The data processing apparatus according to claim 1, wherein the predetermined processing is such that the data subjected to the data processing is not written to the buffer. The data processing apparatus according to claim 8, wherein
The data is supplied to the data processor in response to a transfer request.
The predetermined processing further includes forcibly outputting the transfer request. The data processing apparatus according to claim 8, wherein
The data processing apparatus, wherein the predetermined processing further holds information indicating a position where the failure has occurred. The data processing apparatus according to claim 8, wherein
The data processing apparatus, wherein the predetermined processing further inserts a code indicating that the failure has occurred at a position where the failure has occurred in the data read from the buffer. A data transfer step of transferring data on the bus in first data units;
Predetermined data processing is performed on the data, and the first data unit is equal in data size to the first data unit, or for each second data unit having a data size obtained by dividing the first data unit by an integer. And a data processing step for outputting at intervals,
A data processing method, wherein the data output in the data processing step is transferred onto the bus in the data transfer step. A data transfer step of transferring data on the bus in first data units;
Predetermined data processing is performed on the data, and the first data unit is equal in data size to the first data unit, or for each second data unit having a data size obtained by dividing the first data unit by an integer. And a data processing step for outputting at intervals,
A data processing program for causing a computer to execute a data processing method for transferring data output in the data processing step onto the bus in the data transfer step. In an imaging apparatus that performs processing on moving image data based on an imaging signal output from an imaging element, and that can perform processing on still image data based on the imaging signal in parallel with the processing of the moving image data.
An imaging unit that converts light into an electrical signal by photoelectric conversion and outputs it as an imaging signal;
An imaging signal processing unit that performs predetermined signal processing on the imaging signal output from the imaging unit and outputs the signal as image data;
A moving image data processing unit that outputs a first transfer request, performs processing for each frame timing on the image data output from the imaging signal processing unit in response to the first transfer request, and outputs moving image data; ,
Still image data processing for outputting a second transfer request, processing the image data output from the imaging signal processing unit in response to the second transfer request at a predetermined timing, and outputting still image data And
The still image data processing unit
A compression encoding unit that compresses and encodes the image data output from the imaging signal processing unit;
A data transfer unit that transfers the compressed image data output from the compression encoding unit onto the bus in a first data unit;
The compression encoding unit has a predetermined size for each second data unit having a data size equal to the first data unit or the data size obtained by dividing the first data unit by an integer. An imaging apparatus characterized in that output is performed at intervals. The imaging device according to claim 14, wherein
A memory connected to the bus and capable of burst transfer in units of burst length;
An image pickup apparatus, wherein data is transferred to the memory by the data transfer unit via the bus, and the data size of the first data unit is made equal to the burst length. The imaging device according to claim 14, wherein
The image pickup apparatus, wherein the memory and the bus are shared by transfer processing of the moving image data and the still image data. The imaging device according to claim 14, wherein
The imaging apparatus, wherein the compression encoding unit performs the compression encoding process using a fixed-length code. The imaging device according to claim 14, wherein
The image pickup apparatus, wherein the compression encoding unit performs the compression encoding process using a variable length code. The imaging device according to claim 14, wherein
The compression encoding unit is
A buffer having at least a capacity for the first data unit and temporarily storing the compressed image data subjected to the compression encoding process;
An imaging apparatus, wherein the compressed image data stored in the buffer is read for each second data unit. The imaging device according to claim 19,
The still image data processing unit determines whether or not the function of the buffer has failed based on an access state to the buffer and a transfer request from the data transfer unit. An imaging apparatus characterized by performing processing. The imaging device according to claim 20,
The imaging apparatus according to claim 1, wherein in the predetermined processing, the compressed image data subjected to the compression encoding processing is not written to the buffer. The imaging device according to claim 21, wherein
The supply of the image data to the compression encoding unit is made in response to the second transfer request,
The predetermined processing further forcibly outputs the second transfer request. The imaging device according to claim 21, wherein
The predetermined apparatus further holds information indicating a position where the failure has occurred. The imaging device according to claim 21, wherein
The imaging apparatus, wherein the predetermined processing further inserts a code indicating that the failure has occurred at a position where the failure has occurred in the compressed image data read from the buffer. In a control method of an imaging apparatus that performs processing on moving image data based on an imaging signal output from an imaging element, and that can perform processing on still image data based on the imaging signal in parallel with the processing of the moving image data.
An imaging signal processing step of performing predetermined signal processing on the imaging signal output from the imaging unit by converting light into an electrical signal by photoelectric conversion and outputting as image data;
The first transfer request is output, and the image data output by the imaging signal processing step in response to the first transfer request is processed at each frame timing to output moving image data. Steps,
Still image data that outputs a second transfer request, processes the image data output in the imaging signal processing step in response to the second transfer request at a predetermined timing, and outputs still image data Processing steps,
The still image data processing steps are as follows:
A compression encoding step of compressing and encoding the image data output by the imaging signal processing step;
A data transfer step of transferring the compressed image data output in the compression encoding step on the bus in a first data unit;
The compression encoding step is performed for each second data unit having a data size equal to the first data unit or the data size obtained by dividing the first data unit by an integer. An image pickup apparatus control method characterized in that output is performed with an interval of. A method of controlling an imaging apparatus that performs processing on moving image data based on an imaging signal output from an image sensor and that can perform processing on still image data based on the imaging signal in parallel with the processing of the moving image data. In the control program to be executed,
An imaging signal processing step of performing predetermined signal processing on the imaging signal output from the imaging unit by converting light into an electrical signal by photoelectric conversion and outputting as image data;
The first transfer request is output, and the image data output by the imaging signal processing step in response to the first transfer request is processed at each frame timing to output moving image data. Steps,
Still image data that outputs a second transfer request, processes the image data output in the imaging signal processing step in response to the second transfer request at a predetermined timing, and outputs still image data Processing steps,
The still image data processing steps are as follows:
A compression encoding step of compressing and encoding the image data output by the imaging signal processing step;
A data transfer step of transferring the compressed image data output in the compression encoding step on the bus in a first data unit;
The compression encoding step is performed for each second data unit having a data size equal to the first data unit or the data size obtained by dividing the first data unit by an integer. An image pickup apparatus control program that causes a computer to execute a method for controlling an image pickup apparatus that outputs the image data at intervals.

Images