JP5407651B2 - Image processing apparatus and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing program.

  In recent years, in taking a still image, a digital camera capable of so-called Raw image recording, in which an image signal output from an image sensor is recorded as it is without being subjected to image processing or the like in the camera (undeveloped state) is generally used. It is becoming.

  If the photographed image is recorded as a raw image, deterioration of image quality can be minimized, and after photographing, the color tone, contrast, sharpness, etc. of the photographed image can be adjusted as desired by the user. This eliminates the need to worry about settings such as white balance and exposure during shooting, so that the user can focus on the subject and perform shooting. Thus, the merit of Raw image recording is great.

  From such merits, as described in, for example, Patent Document 1, it is desired to use Raw image recording also in moving image shooting.

JP 2009-55335 A

  However, unlike a still image, a moving image is composed of a large number of frames, so the amount of data is very large. Therefore, in the case of a moving image, it is necessary to compress the image. However, even if an existing compression encoding process / method such as MPEG (Moving Picture Experts Group) is applied, the frame of the moving image is in the RAW state. Compression is difficult in the (undeveloped state), and the original compression efficiency of MPEG cannot be obtained. This is because the Raw image has a pixel structure that depends on the color arrangement of the image sensor (such as a Bayer arrangement), and the correlation between adjacent pixels in each frame is low, and the correlation between frames is naturally low. Because it becomes.

  In addition, a method for efficiently compressing such a Raw state moving image and decoding the compressed data into the Raw state moving image has not yet been proposed.

  An object of the present invention is to provide an image processing apparatus and an image processing program capable of efficiently compressing a RAW state moving image and decoding the compressed data into a RAW state moving image.

An image processing apparatus according to a first aspect of the present invention is an image processing apparatus that compresses / decodes a plurality of Raw images acquired by a continuous series of photographing, and each image component separated from the Raw image is provided for each color component. Compression encoding / decoding means for individually compressing / encoding the data, and when compression encoding is performed with lossy compression, decoding of the compression-encoded data into the Raw image is ensured. In addition, as preprocessing for the separation, image processing for performing gain adjustment using a gain determined based on the degree of change in hue estimated from the characteristics of the compression coding processing / method for each color component of the Raw image Means .

The second invention is the first invention, the image processing means, when performing compression coding, as a pretreatment of the separation, to facilities the gradation conversion processing on the Raw image.

In a third aspect based on the first or second aspect , the compression encoding / decoding means performs compression encoding / decoding using an existing process / method.

In a fourth aspect based on any one of the first to third aspects, each color component of the Raw image is classified based on a color arrangement of an image sensor.

An image processing program according to a fifth aspect of the invention is an image processing program for compressing / encoding / decoding a plurality of raw images acquired by a continuous series of photographing, and is any one of the first to third aspects. The computer executes the processing performed by the compression encoding / decoding unit and the image processing unit of the invention.

  By using the present invention, it is possible to efficiently compress a RAW state moving image and decode the compressed data into a RAW state moving image.

1 is a block diagram illustrating an example of a configuration of a digital camera to which the present invention is applied. The flowchart which shows the operation | movement which concerns on Raw moving image recording which a digital camera performs. The figure explaining preparation of the frame data for every color component.

  Embodiments of the present invention will be described below. The present embodiment is an embodiment of a digital camera.

  FIG. 1 is a block diagram of the digital camera of this embodiment.

  The digital camera includes an imaging lens 11 and a lens driving unit 12, an imaging element 13, an analog signal processing unit 14, a timing generator (TG) 15, a buffer memory 16, an image processing unit 17, and a display control unit 18. , A display unit 19, a control unit 20, a compression / decoding unit 21, a recording interface (recording I / F) 22, a recording medium 23, an operation unit 24, and a bus 25. Here, the buffer memory 16, the image processing unit 17, the display control unit 18, the control unit 20, the compression / decoding unit 21, and the recording I / F 22 are connected via a bus 25. Further, the lens driving unit 12, the analog signal processing unit 14, the TG 15, and the operation unit 24 are each connected to the control unit 20.

  The imaging lens 11 includes a plurality of lens groups including a focus lens and a zoom lens. For simplicity, the imaging lens 11 is illustrated as a single lens in FIG.

  The lens driving unit 12 generates a lens driving signal in accordance with an instruction from the control unit 20, performs focus adjustment and zoom adjustment by moving the imaging lens 11 in the optical axis direction, and subjects the light flux that has passed through the imaging lens 11 to the subject. An image is formed on the light receiving surface of the image sensor 13.

  The image sensor 13 is an image sensor capable of capturing a moving image, and is configured by, for example, a CCD image sensor or a CMOS image sensor. Of course, the image sensor 13 can also capture still images for each frame.

  The imaging element 13 is arranged on the image space side of the imaging lens 11 and photoelectrically converts a subject image formed on the light receiving surface thereof to generate an analog image signal. The output of the image sensor 13 is connected to an analog signal processing unit 14.

  The analog signal processing unit 14 performs analog signal processing such as CDS (correlated double sampling), gain adjustment, and A / D conversion on the analog image signal output from the image sensor 13 in accordance with an instruction from the control unit 20. And the processed image signal is output. The output of the analog signal processing unit 14 is connected to the buffer memory 16.

  Further, the analog signal processing unit 14 sets an adjustment amount for gain adjustment based on an instruction from the control unit 20, and thereby adjusts photographing sensitivity corresponding to ISO sensitivity.

  The TG 15 supplies timing pulses to the image sensor 13 and the analog signal processing unit 14 based on instructions from the control unit 20. The drive timing of the image sensor 13 and the analog signal processing unit 14 is controlled by the timing pulse.

  The buffer memory 16 temporarily stores the image signal output from the analog signal processing unit 14 as image data. The buffer memory 16 temporarily stores image data read from the recording medium 23 by the control unit 20 and image data created in the course of processing by the control unit 20.

  The image processing unit 17 performs image processing such as white balance adjustment, interpolation, edge enhancement, gamma correction, and gradation conversion on the image data in the buffer memory 16 in accordance with an instruction from the control unit 20. The image processing unit 17 is configured as an ASIC or the like.

  In response to an instruction from the control unit 20, the display control unit 18 performs predetermined signal processing (for example, signal processing for converting into an NTSC composite video signal) on the image data after the image processing, and displays the image data on the display unit 19. Output. With this output, the image data is displayed on the display unit 19. The display unit 19 is an LCD monitor provided on the back surface of the digital camera housing, an electronic viewfinder having an eyepiece unit, or the like.

  The compression / decoding unit 21 performs compression processing or decoding processing on the image data in the buffer memory 16 in accordance with an instruction from the control unit 20. The compression process and the decoding process are performed, for example, in the JPEG (Joint Photographic Experts Group) format at the time of still image shooting, and are performed, for example, in the MPEG (Moving Picture Experts Group) format at the time of moving image shooting.

  In the digital camera of the present embodiment, not only moving image data such as MPEG format is created during moving image shooting, but also the moving image data obtained by compressing the captured image data of each frame in the undeveloped Raw format. (Hereinafter referred to as RAW moving image data) can be created, which will be described later.

  A connector for connecting the recording medium 23 is formed in the recording I / F 22. The recording I / F 22 accesses the recording medium 23 connected to the connector, and writes and reads various data. At the time of shooting, the control unit 20 records the image data after the compression processing of the buffer memory 16 on the recording medium 23 via the recording I / F 22. The recording medium 23 is a memory card incorporating a semiconductor memory, a small hard disk, or the like.

  The operation unit 24 includes various operation members such as a mode setting button, a release button, and a moving image shooting button, and sends an operation signal according to the content of the member operation by the user to the control unit 20.

  The control unit 20 comprehensively controls each unit of the digital camera according to the operation content of the operation member by the user.

  For example, when a moving image shooting button is pressed during operation in the shooting mode, the control unit 20 drives the lens driving unit 12, the analog signal processing unit 14, and the TG 15 to start shooting a moving image. At this time, the image sensor 13 is driven so that an image with a predetermined number of pixels is acquired at a predetermined frame rate, and image data corresponding to each frame of the moving image is transferred to the buffer memory 16 via the analog signal processing unit 14. Recorded sequentially. Thereafter, the control unit 20 drives the image processing unit 17 to perform image processing on the image data of each frame recorded in the buffer memory 16. In addition, the control unit 20 drives the display control unit 18 to display the image data after the image processing on the display unit 19. Then, the control unit 20 drives the compression / decoding unit 21 to perform compression processing in the MPEG format or the like on the image data after the image processing, and record the compressed data (moving image data) in the recording I / O. Recording is performed on the recording medium 23 via F22.

  Further, when the moving image shooting button is pressed again during moving image shooting, the control unit 20 stops driving the lens driving unit 12, the analog signal processing unit 14, and the TG 15, and ends moving image shooting.

  By the way, the digital camera of this embodiment has a function (raw moving image recording function) for creating raw moving image data and recording it on the recording medium 23 at the time of moving image shooting.

  Hereinafter, the operation performed by the digital camera of this embodiment for this function will be described with reference to the flowchart of FIG. The processing of the flowchart of FIG. 2 is executed when the user presses the moving image shooting button and instructs the digital camera to shoot a moving image when the Raw moving image recording function is set to be valid.

  Step 101 (S101): The control unit 20 drives the lens driving unit 12, the analog signal processing unit 14, and the TG 15 to start shooting a moving image. At this time, the image sensor 13 is driven so that an image with a predetermined number of pixels is acquired at a predetermined frame rate, and image data corresponding to each frame of the moving image is transferred to the buffer memory 16 via the analog signal processing unit 14. Recorded sequentially.

  Step 102: The control unit 20 creates a recording file used for recording Raw moving image data. The recording file is created on the recording medium 23 via the recording I / F 22.

  Step 103: The control unit 20 separates the image data of each frame of the moving image sequentially acquired in the buffer memory 16 for each color component, and creates frame data for each color component.

  Specifically, for example, as shown in (1) of FIG. 3, when the image data of each frame of the moving image is Raw data in a Bayer array, it is shown in (2) of FIG. As described above, the image data of each frame is separated for each color component, and four frame data of R, Gr, Gb, and B (size is 1/4 of the original) are created. The created four frame data are stored in the buffer memory 16 or the like, for example.

  Step 104: The control unit 20 compresses and encodes the R component frame data that is sequentially generated to generate R component partial moving image data. The generated partial moving image data is stored in the buffer memory 16 or the like, for example.

  By the way, the frame data of each color component including the R component can be considered as a monochrome frame by itself, and the correlation between adjacent pixels in the frame is also high. Therefore, for example, “MPEG-4 AVC / H.264”, which is an existing compression encoding process / method for moving images, can be applied to compression encoding of frame data of each color component. As described above, the present invention can be applied with the existing compression encoding process / method for moving images as it is, and has high versatility.

  Step 105: The control unit 20 compresses and encodes the sequentially generated Gr component frame data to generate Gr component partial moving image data. The generated partial moving image data is stored in the buffer memory 16 or the like, for example.

  Step 106: The control unit 20 compresses and encodes the sequentially generated Gb component frame data to generate Gb component partial moving image data. The generated partial moving image data is stored in the buffer memory 16 or the like, for example.

  Step 107: The control unit 20 compresses and encodes the B component frame data that is sequentially generated to generate B component partial moving image data. The generated partial moving image data is stored in the buffer memory 16 or the like, for example.

  Step 108: The control unit 20 records the generated partial moving image data of each color component (R, Gr, Gb, B) in a recording file on the recording medium 23 via the recording I / F 22, respectively.

  Step 109: The control unit 20 determines whether or not the end of shooting is instructed, and when instructed, the process proceeds to Step 110 (Yes side), whereas when not instructed, the process proceeds to Step 103. The above processing is repeated (No side). Note that, as described above, the instruction to end shooting is performed by the user pressing the moving image shooting button again.

  Step 110: The control unit 20 compresses and encodes the additional information and records the processed data in a recording file on the recording medium 23 via the recording I / F 22.

  The additional information is information required when decoding Raw moving image data that has been compression-encoded by the method of the present invention. For example, information indicating the color arrangement of the image sensor (the Bayer arrangement or the like) , Information indicating compression encoding processing / method (eg, MPEG-4 AVC / H.264). The information indicating the color arrangement of the image sensor is used to know what color component partial moving image data has been generated, and the information indicating the compression encoding process / method is a compression code Can be used to know if the conversion has been performed.

  For compression coding of the additional information, for example, a known compression coding process / method such as Huffman coding can be applied.

  Step 111: The control unit 20 closes the recording file on the recording medium 23 via the recording I / F 22, stops the driving of the lens driving unit 12, the analog signal processing unit 14, and the TG 15, and Stop shooting.

(Supplementary items of the embodiment)
In the above description, the process of compressing and encoding the frame data of each color component (the above steps 104 to 107) is described as being executed in a cascade manner. However, the processing may be executed in parallel.

  In step 110 described above, the additional information has been described as being compressed and encoded. However, the additional information may be encrypted. In this case, for example, a well-known encryption process / system such as AES (Advanced Encryption Standard) can be applied to the encryption. As can be easily understood from the above description, the additional information needs to be known at the time of decryption regardless of whether compression encoding or encryption is performed.

  Although not specifically described in the processing of the above flowchart, the image data of each frame of a moving image acquired by shooting may be sequentially displayed on an LCD monitor of a digital camera or the like. In this case, the image processing unit 17 performs image processing on the display image data.

  In the above description, the control unit 20 performs the process related to the creation of the raw moving image data (the above steps 103 to 107 and step 110). However, the processing may be performed by the compression / decoding unit 21 or another dedicated circuit according to an instruction from the control unit 20.

  Further, in the above, when the image data of each frame of the moving image is the Bayer array Raw data, the image data of each frame is separated for each color component, and four frames of R, Gr, Gb, and B are separated. Explained to create data. However, regarding the color components of Gr and Gb, the average of the two may be taken as G of one color component. That is, in this case, three frame data of R, G, and B are created. In that case, information indicating that the color component is G may be provided as additional information.

  In the above description, the image data of each frame of the moving image is Raw data in the Bayer array of R, G (Gr and Gb), and B. Of course, even if it is Raw data in a color array other than the Bayer, Good. Further, the color of the image data may be not only RGB color but also CMY color.

  Further, as pre-processing in step 103, gradation conversion processing may be performed on the image data of each frame of the moving image. By doing so, for example, when the original gradation of the image data is 12 bits, if the gradation conversion is performed to 10 bits, the data amount of the Raw moving image data created after compression is compared with the case where the conversion is not performed Can be reduced (reduced), and can be converted into the number of bits corresponding to the existing moving image compression encoding process / method.

  When an irreversible compression encoding process / method is used for the compression encoding in steps 104 to 107, gain adjustment processing for image color components such as white balance adjustment is performed as preprocessing in step 103. The image data may be applied to each frame of the moving image. This is because in the case of irreversible compression, there is a possibility that the hue of a moving image may change after compression (image quality degradation) due to the compression encoding process. Therefore, by performing gain adjustment using different gains for each color component (for example, for each of R, G, and B) of the image data of each frame before compression, image quality deterioration can be reduced. it can. Note that the gain used for adjustment is determined based on the degree of hue change (image quality degradation) estimated from the characteristics of the compression encoding process / method.

  Further, both the gradation conversion process and the gain adjustment process may be performed on the image data of each frame of the moving image as the pre-processing in step 103.

  In the above description, the compression encoding process has been described. However, the decoding process can be easily realized by performing a procedure reverse to the above. However, as can be easily understood from the above description, it is necessary to first decode the additional information and to decode the partial moving image data of each color component based on the decoded information.

  In the above description, an example in which a RAW-state moving image is compression-encoded has been described. However, the method of the present invention is also applied to a RAW-state still image group (continuous-shot image) that has been continuously shot. Can do. Therefore, the compression rate can be increased as compared with the case of individually compressing and encoding the still state still images that are continuously shot.

(Effect of embodiment)
As described above, in the digital camera of this embodiment, frame data for each color component is created from image data (Raw data) of each frame of a moving image acquired by shooting. Thus, the data (frame data) for each color component separated from the image data (Raw data) can be considered as a monochrome frame by itself, and the correlation between adjacent pixels in the frame is also high. It is possible to compress efficiently. Note that the compression encoding process / method for moving images can be applied as it is for compression.

  Then, the frame data for each color component is individually compressed and encoded, and partial moving image data for each color component is generated.

  The generated partial moving image data of all color components is recorded as Raw moving image data in a recording file.

  The raw moving image data recorded on the recording medium or the like in this way can be decoded by a procedure reverse to the above.

  Therefore, according to the digital camera of the present embodiment, it is possible to efficiently compress the raw state moving image and decode the compressed data into the raw state moving image.

(Other)
Note that the processing of steps 102 to 111 (excluding step 109) of the program relating to the operation of the digital camera described above may be executed by an external processing device such as a computer. In that case, a necessary program is installed in the external processing apparatus via a computer-readable storage medium such as a CD-ROM or a communication network such as the Internet.

  Further, although the embodiment of the digital camera has been described above, the present invention can be applied to other devices capable of shooting moving images and continuous shot images, such as a digital video camera and a mobile phone. Is possible.

DESCRIPTION OF SYMBOLS 11 ... Imaging lens, 12 ... Lens drive part, 13 ... Imaging device, 14 ... Analog signal processing part, 15 ... Timing generator (TG), 16 ... Buffer memory, 17 ... Image processing part, 18 ... Display control part, 19 ... Display unit, 20 ... control unit, 21 ... compression / decoding unit, 22 ... recording interface (recording I / F), 23 ... recording medium, 24 ... operation unit, 25 ... bus

Claims (5)

An image processing apparatus that compresses / decodes a plurality of Raw images acquired by a series of continuous shootings,
Compression encoding / decoding means for individually compressing / decoding data for each color component separated from the raw image ;
When performing the compression encoding by irreversible compression, for each color component of the Raw image, as the pre-processing of the separation, the compression encoded data is ensured to be decoded into the Raw image. An image processing apparatus comprising: an image processing unit configured to perform gain adjustment using a gain determined based on a hue change degree estimated from a compression / encoding process / method characteristic . The image processing apparatus according to claim 1.
Wherein the image processing means, when performing the compression encoding, as a pretreatment of the separation, to facilities the gradation conversion processing on the Raw image
An image processing apparatus. The image processing apparatus according to claim 1 or 2 ,
The image processing apparatus, wherein the compression encoding / decoding means performs the compression encoding / decoding using an existing process / method . In the image processing device according to any one of claims 1 to 3,
Each color component of the Raw image is classified based on a color arrangement of an image sensor . An image processing program that compresses / decodes a plurality of Raw images acquired by a series of continuous shootings,
An image processing program for causing a computer to execute processing performed by the compression encoding / decoding unit and the image processing unit according to any one of claims 1 to 3 .

Images