JP2015188130A - Image processing system, imaging apparatus, control method of image processing system, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique that makes it possible to quickly display a reproduction image while recording a RAW image.SOLUTION: The image processing system includes: first development means of developing a RAW image; compression means of compression coding the developed RAW image and generating a first image file; RAW compression means of compressing the RAW image and generating a RAW file; storage means of storing the first image file and the RAW file in a recording medium; second development means of performing developing with a processing load higher than that of the development processing performed by first development means against the RAW image; and reproduction means of a moving image. In a predetermined operation state of the image processing system, when the second development means performs development processing with a high processing load against a part of the RAW image and the reproduction means reproduces a moving image, in a part of a plurality of frames, a developed image is reproduced from the RAW image obtained by the second development means, and in the other part, a developed image obtained from the first image file is reproduced.

Description

  The present invention relates to an image processing apparatus, an imaging apparatus, an image processing apparatus control method, and a program.

  In a conventional imaging apparatus, raw image information (RAW image) captured by an imaging sensor is debayered (demosaiced), converted into a signal composed of luminance and color difference, noise removal, and optical distortion for each signal. So-called development processing such as correction and image optimization is performed. In general, the developed luminance signal and color difference signal are compression encoded and recorded on a storage medium.

  On the other hand, there are imaging devices that can record RAW images. RAW images require a large amount of data for recording, but are preferred by advanced users because they have the advantage that correction and deterioration of the original image can be minimized and can be edited after shooting.

  The configuration of an imaging apparatus that records a RAW image is disclosed in Patent Document 1. Patent Document 1 describes a configuration in which development parameters are recorded together with a RAW image, and the RAW image is developed and reproduced using the development parameters during reproduction.

JP 2011-244423 A

  In recent imaging apparatuses, the number of pixels per image has greatly increased due to the evolution of imaging sensors. Also, the number of images that can be taken continuously per second tends to increase. For this reason, each processing amount for development processing such as debayer processing, noise removal, and optical distortion correction for RAW images increases synergistically, and in order to perform real-time development processing in parallel with shooting, enormous circuits and consumption are required. Electricity is spent. In some cases, it may not be possible to execute high photographing performance due to the occupation of the circuit by development processing and the limitation on power consumption.

  On the other hand, if the configuration is such that the RAW image is recorded without being developed as in Patent Document 1 described above, the processing amount related to the development at the time of shooting may be reduced, but the RAW image is recorded in the undeveloped state. Therefore, it is difficult to quickly reproduce and display the image. Furthermore, the RAW image cannot be reproduced (developed) by other devices due to the uniqueness of the data specific to the RAW image, and the conventional RAW image recording method may impair the convenience of the user.

  In this way, in order to perform high shooting performance with a conventional device and to enable playback images to be output at high speed, either high-cost circuits can be installed and driven at high output, or RAW images can be recorded and processed at high speed. And it must be played back easily.

  Accordingly, an object of the present invention is to provide a technique that enables a reproduced image to be displayed promptly while recording a RAW image.

The present invention for solving the above problems is an image processing apparatus,
First developing means for developing a RAW image;
Compression means for compression-encoding the developed RAW image to generate a first image file;
RAW compression means for compressing the RAW image and generating a RAW file;
Recording means for recording the first image file and the RAW file on a storage medium;
Decompression means for decompressing the RAW file to generate a RAW image;
A second developing unit that performs a developing process with a higher processing load on the RAW image obtained by the expansion than the developing process performed by the first developing unit;
Playback means for playing back the moving image obtained from the first image file or the RAW file from the storage medium,
When the image processing apparatus is in a predetermined operation state, the expansion unit expands a part of a plurality of frames included in the RAW file, and the second developing unit extracts the RAW image obtained by the expansion. Execute development processing with high processing load,
When the reproducing unit reproduces the moving image, a part of the plurality of frames corresponding to the moving image reproduces a developed image from the RAW image obtained by the second developing unit, and the other part. Then, the developed image obtained from the first image file is reproduced.

  According to the present invention, it is possible to promptly display a reproduced image while recording a RAW image.

It is a block diagram which shows the structural example of the imaging device which concerns on embodiment of invention. It is a state transition diagram of an embodiment of the invention. It is a flowchart which concerns on the process of the still image shooting mode of embodiment of invention. It is an example of a structure of the still image file and RAW file of embodiment of invention. It is a flowchart which concerns on the process of an idle state of embodiment of invention. It is a flowchart which concerns on the process of the still image reproduction mode of embodiment of invention. It is a figure which shows the example of the display process of the still image reproduction mode of embodiment of invention. It is a flowchart which concerns on the process of the video recording mode of embodiment of invention. It is a structural example of the moving image file and RAW file of embodiment of invention. It is a flowchart which concerns on the process of the chase development mode of embodiment of invention. It is a flowchart which concerns on the process of the moving image reproduction mode of embodiment of invention. It is explanatory drawing of a pixel arrangement | sequence. It is a conceptual diagram of each moving image file when the first portion chasing development is performed according to the embodiment of the invention. It is a flowchart which concerns on the chase development process of embodiment of invention. It is a figure explaining the frame selection method which concerns on the chase development process of embodiment of invention. It is a figure for demonstrating the encoding mode in the predictive encoding of embodiment of invention, and the reference destination of a flame | frame. It is a flowchart which concerns on the process of the moving image reproduction mode in embodiment of invention. It is a figure which shows an example of the list display of the moving image file in embodiment of invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Embodiment 1>
FIG. 1 is a block diagram illustrating a configuration example of an imaging apparatus according to an embodiment of the present invention. The imaging apparatus 100 shown in FIG. 1 not only records image information obtained by imaging a subject on a storage medium, but also reproduces the image information from the storage medium, displays it after developing, and image information. Has a function to transmit / receive data to / from an external device or server (cloud). Therefore, the imaging apparatus according to the embodiment of the present invention can be expressed as an image processing apparatus, a recording apparatus, a reproducing apparatus, a recording / reproducing apparatus, a communication apparatus, and the like. In the imaging apparatus 100 of FIG. 1, each block may be configured by hardware using a dedicated logic circuit or a memory, except for physical devices such as an imaging element, a display element, an input device, and a terminal. Alternatively, the processing program stored in the memory may be configured by software by a computer such as a CPU executing the processing program.

  In FIG. 1, the control unit 161 includes a CPU and a memory that stores a control program executed by the CPU, and controls the overall processing of the imaging apparatus 100. The operation unit 162 includes input devices such as keys, buttons, and a touch panel used by the user to give an instruction to the imaging apparatus 100. An operation signal from the operation unit 162 is detected by the control unit 161, and is controlled by the control unit 161 so that an operation corresponding to the operation is executed. The display unit 123 includes a liquid crystal display (LCD) or the like for displaying an image taken or reproduced in the imaging apparatus 100, a menu screen, various information, and the like.

  When an operation unit 162 is instructed to start a shooting operation, an optical image of a subject to be imaged is input via the imaging optical unit 101 and formed on the imaging sensor unit 102. At the time of shooting, the operations of the imaging optical unit 101 and the imaging sensor unit 102 are based on evaluation value calculation results such as aperture, focus, camera shake, and the like acquired by the evaluation value calculation unit 105 and subject information extracted by the recognition unit 131. Controlled by the camera control unit 104.

  The image sensor unit 102 includes an image sensor such as a CCD or a CMOS sensor, and converts light transmitted through red, green, and blue (RGB) color filters arranged for each pixel into an electrical signal. FIG. 12 is an example of a color filter disposed in the imaging sensor unit 102 and represents a pixel arrangement of an image handled by the imaging apparatus 100. As shown in FIG. 12, red (R), green (G), and blue (B) are arranged in a mosaic pattern for each pixel, and for each 2 × 2 four pixels, one red pixel, one blue pixel, and two green pixels It has a structure in which pixels are regularly arranged in one set. Such an arrangement of pixels is generally called a Bayer array.

  The electrical signal converted by the imaging sensor unit 102 is subjected to pixel restoration processing by the sensor signal processing unit 103. The restoration process includes a process of interpolating a pixel to be restored using a peripheral pixel value or subtracting a predetermined offset value from a missing pixel value or a low-reliability pixel value in the image sensor unit 102. It is. In the present embodiment, the image information output from the sensor signal processing unit 103 is referred to as a RAW image that means a raw (undeveloped) image.

  The RAW image is developed by the developing unit 110. The development unit 110 includes a plurality of different development processing units, and includes a simple development unit 111 as a first development unit and a high-quality development unit 112 as a second development unit, and selects an output thereof. A switch unit 121 is included. Both the simple development unit 111 and the high-quality development unit 112 perform debayer processing (demosaic processing) on the RAW image, convert the signal into a signal composed of luminance and color difference, remove noise contained in each signal, A so-called development process such as correcting distortion and optimizing an image is performed.

  In particular, the high image quality development unit 112 performs each process with higher accuracy than the simple development unit 111. Since it is highly accurate, a developed image with higher image quality than that of the simple developing unit 111 can be obtained, but on the other hand, the processing load becomes high. Therefore, the high image quality developing unit 112 of the present embodiment is not specialized for real-time development in parallel with shooting, and has a configuration capable of performing distributed processing over time after shooting. As described above, the high-quality image development is not performed at the time of shooting but is performed later over time, so that an increase (peak) in circuit scale and power consumption can be suppressed to a low level. On the other hand, the simple development unit 111 has a lower image quality than the high image quality development unit 112, but is configured to have a smaller processing amount than the high image quality development so that the development process can be performed at high speed during shooting. Yes. Since the processing load of the simple developing unit 111 is small, the simple developing unit 111 is used for real-time development in parallel with the photographing operation. The switch unit 121 is switched by the control unit 161 in accordance with control according to the operation content instructed by the user from the operation unit 162 and the operation mode being executed.

  In the present embodiment, the simple developing unit 111 and the high image quality developing unit 112 exist independently in the developing unit 110. However, one developing unit switches the operation mode so that the simple developing unit and the high developing unit 112 can Even a configuration in which image quality development processing is performed exclusively is within the scope of the present invention. The image information developed by the developing unit 110 is displayed on the display unit 123 after a predetermined display process is performed by the display control unit 122. Further, the developed image information may be output to an externally connected display device through the video output terminal 124. The video output terminal 124 includes a general-purpose interface such as HDMI or SDI.

  The image information developed by the developing unit 110 is also supplied to the evaluation value calculating unit 105. The evaluation value calculation unit 105 calculates evaluation values such as a focus state and an exposure state from the image information.

  The image information developed by the developing unit 110 is also supplied to the recognition unit 131. The recognition unit 131 has a function of detecting and recognizing subject information in image information. For example, a face in the screen represented by the image information is detected. If there is a face, information indicating the position of the face is output, and a specific person is authenticated based on feature information such as the face.

  The image information developed by the developing unit 110 is also supplied to the still image compression unit 141 and the moving image compression unit 142. When compressing the image information as a still image, the still image compression unit 141 is used. When the image information is compressed as a moving image, the moving image compression unit 142 is used. The still image compressing unit 141 and the moving image compressing unit 142 respectively perform high-efficiency encoding (compression encoding) on the target image information, generate image information whose information amount is compressed, and generate an image file (still image file, Or a video file). Still image compression is JPEG, etc., and moving image compression is MPEG-2, H.264. H.264, H.C. 265 or the like can be used.

  The RAW compression unit 113 performs high-efficiency encoding on the RAW image output from the sensor signal processing unit 103 using a technique such as wavelet conversion or differential encoding, and converts the RAW image into a compressed RAW file. Section (storage medium) 115. The RAW file can remain in the buffer unit 115 and can be read again. However, after being stored in the buffer unit 115, it is moved to another storage medium and recorded (deleted from the buffer unit 115). Also good.

  The RAW file and the above-described still image file and moving image file are recorded on the storage medium 152 by the recording / playback unit 151. The storage medium 152 is a built-in large-capacity memory, a hard disk, or a removable memory card. The recording / playback unit 151 can also read a still image file, a moving image file, and a RAW file from the storage medium 152.

  The recording / playback unit 151 can write and read various files to an external storage or server via the communication unit 153. The communication unit 153 has a configuration in which the communication terminal 154 can be used to access the Internet or an external device through wireless communication or wired communication.

  When the playback operation is started, the recording / playback unit 151 acquires and plays back a desired file from the storage medium 152 or via the communication unit 153. If the file to be played is a RAW file, the recording / playback unit 151 stores the acquired RAW file in the buffer unit 115. If the file to be played back is a still image file, the recording / playback unit 151 supplies the acquired still image file to the still image expansion unit 143. If the file to be played is a moving image file, the recording / playback unit 151 supplies the acquired moving image file to the moving image decompression unit 144.

  The RAW decompression unit 114 reads the RAW file stored in the buffer unit 115, decodes the compressed RAW file, and generates a RAW image. The RAW image obtained by the RAW file expansion process by the RAW expansion unit 114 is supplied to the simple development unit 111 and the high image quality development unit 112 in the development unit 110.

  The still image expansion unit 143 decodes and expands the input still image file, and supplies the decoded still image file to the display control unit 122 as a reproduced image of the still image. The moving image extension unit 144 decodes and expands the input moving image file, and supplies it to the display control unit 122 as a moving image reproduction image (reproduced moving image).

  Next, operation modes of the imaging apparatus 100 of the present embodiment will be described in detail with reference to the drawings. FIG. 2 is a state transition diagram illustrating transition of each operation mode in the imaging apparatus 100. Such transition of the operation state is executed according to a user operation instruction from the operation unit 162 or a determination of the control unit 161, and may be manually changed or automatically changed according to the operation. . As illustrated in FIG. 2, the imaging apparatus 100 includes four modes via an idle state (200), a still image shooting mode (201), a still image playback mode (202), a moving image shooting mode (203), and a moving image playback mode. The operation is switched to (204) as appropriate.

  Next, the operation in the still image shooting mode of the imaging apparatus 100 will be described. FIG. 3 shows a flowchart relating to processing in the still image shooting mode of the present embodiment. The flowchart in FIG. 3 illustrates a processing procedure executed by controlling each processing block by the control unit 161. A program stored in a memory (ROM) of the control unit 161 is stored in the memory (RAM). This is realized by expanding and executing by the CPU.

  In FIG. 3, when the still image shooting mode process is started, the control unit 161 determines whether the processing load status of the imaging apparatus 100 is low in S <b> 301. If the processing load is low at a frequency according to the load status, the process shifts to the idle state in S320, and otherwise the process proceeds to S302. For example, during high-speed continuous shooting, the processing load is high, so the process does not transition to S320 but always proceeds to S302. In the case of normal single shooting, the process proceeds to S320 with a frequency of half, for example, between the first shooting and the second shooting.

  In step S <b> 302, the camera control unit 104 controls the operations of the imaging optical unit 101 and the imaging sensor unit 102 so that shooting is performed under suitable conditions. For example, a lens included in the imaging optical unit 101 is moved in accordance with a user's zoom or focus instruction, or a reading area of the imaging sensor unit 102 is set in accordance with an instruction on the number of pixels to be captured. In addition, based on evaluation value information and subject information supplied from an evaluation value calculation unit 105 and a recognition unit 131, which will be described later, control such as focus adjustment and tracking for a specific subject is performed.

  In step S <b> 303, the sensor signal processing unit 103 performs signal processing for pixel restoration on the electrical signal converted by the imaging sensor unit 102. Here, interpolation is performed using the peripheral pixel value or a predetermined offset value is subtracted from the value of the missing pixel or the pixel having low reliability. In the present embodiment, the image information output from the sensor signal processing unit 103 after finishing the processing of S303 is referred to as a RAW image meaning a raw (undeveloped) image.

  In S304, the simple developing unit 111 develops the RAW image. At this time, the control unit 161 switches the switch unit 121 in the developing unit 110 and selects the output of the image information developed by the simple developing unit 111. The simple developing unit 111 performs debayer processing (demosaic processing) on the RAW image, converts it into a signal composed of luminance and color difference, removes noise included in each signal, corrects optical distortion, and corrects the image. So-called development processing is performed. Here, the development processing (simple development) performed by the simple development unit 111 will be described. The simple development unit 111 limits the image size after development to 2 million pixels or less, or restricts or eliminates noise removal and optical distortion correction to a limited process, so that the development process can be performed at high speed. Processing is realized. The simple development unit 111 performs processing after reducing the image size, or restricts a part of the development processing function, so that the imaging apparatus 100 captures a performance of, for example, 60 frames per second of 2 million pixels. It can be realized with a small circuit scale and low power consumption.

  The image information developed by the simple development unit 111 is supplied to the evaluation value calculation unit 105. In S305, the evaluation value calculation unit 105 calculates an evaluation value such as a focus state or an exposure state from a luminance value or a contrast value included in the image information. Note that the evaluation value calculation unit 105 may acquire a raw image before development processing and similarly calculate an evaluation value from the raw image. Further, the image information developed by the simple development unit 111 is supplied to the recognition unit 131. In S306, the recognition unit 131 detects a subject (such as a face) from the image information, and recognizes the subject information. For example, the presence / absence of the face in the image information, the position thereof, authentication of a specific person, and the like are performed, and the result is output as information.

  The image information developed by the simple developing unit 111 is also supplied to the display control unit 122. In S307, the display control unit 122 forms a display image from the acquired image information, and outputs and displays the display image on the display unit 123 or an external display device. The display image by the display unit 123 is used for live view display (shooting through image display) for the user to appropriately frame the subject in the still image shooting mode. The display image may be displayed on another display device such as an external television from the display control unit 122 via the video output terminal 124. Further, the display control unit 122 uses the evaluation value information and the subject information supplied from the evaluation value calculation unit 105 and the recognition unit 131, for example, to display a marking in a focused area on the display image, A frame can also be displayed at the position of the recognized face.

  In S308, the control unit 161 determines an input of a shooting instruction from the user. If there is a shooting instruction, the process proceeds to S310. If there is no shooting instruction in S308, the process returns to S301 to repeat the shooting preparation operation and the live view display. In response to the shooting instruction in S <b> 308, the image information developed by the simple development unit 111 is supplied to the still image compression unit 141. In S310, the still image compression unit 141 performs high-efficiency encoding processing (still image compression) on the acquired image information to generate a still image file. Note that the still image compression unit 141 performs compression processing using a known still image compression technique such as JPEG.

  In S <b> 311, the recording / playback unit 151 records the still image file on the storage medium 152. Further, in response to the shooting instruction in S308, in S312, the RAW compression unit 113 acquires the RAW image output from the sensor signal processing unit 103 corresponding to the shot still image, and performs high-efficiency coding ( RAW compression) to convert to a RAW file. The RAW file is stored in the buffer unit 115. The high-efficiency encoding performed by the RAW compression unit 113 is processed by a known technique such as wavelet transform or differential encoding, but may be lossy encoding or lossless encoding. Alternatively, the RAW compression of the RAW compression unit 113 may be omitted, and the RAW image may be output through without being compressed. Regardless of the presence or absence of RAW compression, the present embodiment generates a RAW file that can be restored as a high-quality file without significantly losing the image information supplied from the sensor signal processing unit 103.

  In step S313, after the recording / playback unit 151 records the RAW file in the storage medium 152, the flow proceeds to step S301. In S311 and S313, the recording / playback unit 151 sends the still image file and / or the RAW file to the external storage from the communication terminal 154 via the communication unit 153, and records the file in the external storage. Also good. This completes the description of the flow relating to the still image shooting mode processing of the present embodiment.

  Here, the structure of the still image file and the structure of the RAW file according to the present embodiment will be described. FIG. 4 is a diagram illustrating a configuration example of a still image file and a RAW file. The still image file 400 shown in FIG. 4A is recorded in, for example, a predetermined recording area of the storage medium 152 by the recording / playback unit 151. The still image file 400 includes a header part 401, a metadata part 402, and a compressed data part 403. The header portion 401 includes an identification code indicating that this file is a still image file format. The compressed data section 403 includes still image compressed data that has been encoded with high efficiency.

  The metadata unit 402 includes file name information 404 of a RAW file generated at the same time as the still image file. Further, development status information 405 including a flag indicating whether the still image file has been developed with the simple developing unit 111 or developed with the high quality developing unit 112 is included. Also, evaluation values and subject information detected by the evaluation value calculation unit 105 and the recognition unit 131, and information at the time of shooting from the imaging optical unit 101 and the imaging sensor unit 102 (for example, lens type identification information, sensor type identification information) And the like. In addition, although not shown, it may further include an identification code of a storage medium in which a RAW file generated at the same time is recorded, path information of a recorded folder, and the like.

  The RAW file 410 shown in FIG. 4B is recorded in, for example, a predetermined recording area of the storage medium 152 by the recording / playback unit 151. The RAW file 410 includes a header part 411, a metadata part 412, and a compressed data part 413. The header portion 411 includes an identification code indicating that this file is in the RAW file format. The compressed data portion 413 includes RAW compressed data of still images that have been encoded with high efficiency (may be RAW image data of uncompressed still images).

  The metadata section 412 includes file name information 414 of the still image file generated simultaneously with the RAW file. Further, development status information 415 indicating that the still image file has been simply developed by the simple development unit 111 is included. Also, evaluation values and subject information detected by the evaluation value calculation unit 105 and the recognition unit 131, and information at the time of shooting from the imaging optical unit 101 and the imaging sensor unit 102 (for example, lens type identification information, sensor type identification information) And the like. In addition, although not shown, it may further include an identification code of a storage medium in which a still image file generated at the same time is recorded, path information of a recorded folder, and the like. Alternatively, the still image file itself generated at the same time may be converted into metadata and stored in the metadata unit 412. The structure of the various files according to the present embodiment described above is an example, and may be configured according to a standard such as DCF or EXIF.

  As described above, the imaging apparatus 100 according to the present embodiment performs live view display until a shooting instruction in the still image shooting mode is performed, and development processing for a still image file generated in response to the shooting instruction. This is performed by the simple developing unit 111. The simple development unit 111 restricts the image size after development to 2 million pixels or less, or restricts or eliminates noise removal and optical distortion correction. By doing so, for example, development processing with a performance of 2 million pixels and 60 frames per second can be realized with a small circuit scale and low power consumption. On the other hand, as described above, the imaging apparatus 100 according to the present embodiment generates a RAW file in response to a still image shooting instruction. The RAW file is a high-quality file that does not significantly impair the image information supplied from the sensor signal processing unit 103, but development processing is not required to generate this file. Therefore, it is possible to record a RAW file with low power consumption by a small circuit while increasing the number of pixels of an image and the speed of continuous shooting.

  Next, the idle state process in S320 of FIG. 3 will be described with reference to the flowchart of FIG. FIG. 5 shows a flowchart relating to the idle state processing of the present embodiment. The flowchart of FIG. 5 illustrates a processing procedure executed by controlling each processing block by the control unit 161. A program stored in a memory (ROM) of the control unit 161 is stored in the memory (RAM). This is realized by expanding and executing by the CPU.

  In FIG. 5, when the idle state process is started, the control unit 161 determines in S501 whether or not to perform follow-up development according to a manual operation by the user. When developing, the process proceeds to S520.

  When the follow-up development is not performed in S501, it is determined whether the control unit 161 shifts to any of the modes 201, 202, 203, and 204 shown in FIG. 2 according to the mode setting from the user (S502, S503, S504, S505). Then, the control unit 161 controls to transition to the processing flow of the selected mode (S510, S511, S512, S513).

  In the “follow-up development” of the present embodiment, after the photographing operation is completed, a development process is performed again with a high quality using a RAW file recorded in the buffer unit 115 or the storage medium 152 as a source, and a high-quality display image or a high-quality image is displayed. This means processing for generating a still image file. The RAW file that is the target of the chase development according to the present embodiment targets both a still image and a moving image. In the following, a still image will be described as an example.

  As described above, since the still image file generated at the time of shooting is developed by the simple development unit 111, the number of pixels is 2 million pixels or less, or the development process is partially omitted. Is of limited quality. Although it is effective as a rough confirmation of the photographing content, it may not be sufficient for the purpose of confirming details of an image or printing out. On the other hand, the RAW file that is generated at the same time as the still image has high quality that does not significantly impair the image information supplied from the sensor signal processing unit 103, but since it is data before development processing, it can be immediately displayed and printed out. Cannot be performed, and it takes time for RAW development. In addition, since RAW is not a widely spread file such as JPEG, the playback environment that can handle RAW files is also limited.

  Therefore, the chasing development of this embodiment is an effective function. In this embodiment, when follow-up development is executed, a RAW image is read from a RAW file that has already been recorded frame by frame, and development processing is performed with high image quality by the high image quality development unit 112, and a generated still image is generated. The file is recorded on the storage medium 152 or the like. Then, such chasing development is executed in a state where the processing load of the apparatus is relatively small, waiting for a user operation such as between shootings or in a reproduction mode or a sleep state. The follow-up development is not limited to manual, but may be designed so that the controller 161 automatically executes it.

  By configuring in this way, subsequent development processing does not occur each time even when a high-quality playback request such as confirmation display of details or printout is required, and the same as with a conventional still image file. It can be used in a general environment.

  Returning to the explanation of FIG. 5, a still image file and a RAW file are recorded as one set for each shooting instruction in the storage medium 152 or the like. When the follow-up development is performed manually or automatically, in step S520, the control unit 161 determines whether the follow-up development has been processed or not processed for each set of images. As a determination method, for example, there is a method of referring to a flag included in the development status 405 of the still image file 400 for identifying whether or not the still image file has been processed by the simple development unit 111. Alternatively, the same determination may be made with reference to the development status 415 in the RAW file 410. Alternatively, it may be determined by separately preparing a table file indicating the state of development processing for a series of still images taken.

  If the control unit 161 determines that the follow-up development has been processed, the process proceeds to S502. If there is a still image that has not been subjected to the follow-up development, the process proceeds to S521. In step S <b> 521, the control unit 161 determines whether the RAW file corresponding to the still image that has not been subjected to the follow-up development is buffered in the buffer unit 115. If buffered, the process proceeds to S523, and if not buffered, the corresponding RAW file is read from the storage medium 152 or the like in S522.

  The buffer unit 115 updates the data so that the new image captured in the still image capturing mode is preferentially retained. That is, images taken in the past are sequentially removed from the buffer. By doing so, since the image taken immediately before is always held in the buffer, S522 can be skipped and processed at high speed. Further, if the follow-up development is executed by tracing back the time taken from the immediately previous image, the processing can be completed preferentially from the image held in the buffer, so that the processing efficiency can be improved.

  In S523, the RAW decompression unit 114 decompresses the RAW file read from the buffer unit 115 or the storage medium 152, and restores the RAW image. The restored RAW image is developed to a high image quality by the high image quality development unit 112 in S524, and is output to the display control unit 122 and the still image compression unit 141 via the switch unit 121. The high image quality developing unit 112 performs debayer processing (demosaic processing) on the RAW image, converts it into a signal composed of luminance and color difference, removes noise contained in each signal, corrects optical distortion, and optimizes the image. A so-called development process is performed. The size (number of pixels) of the developed image generated by the high image quality development unit 112 is the entire size read from the image sensor unit 102 or the size set by the user, and is limited to 2 million pixels or less. The quality is much higher than that of the simple developed image. Since the high-quality developing unit 112 is more accurate in each process than the simple developing unit 111, a higher-quality developed image can be obtained, but the processing load is increased. The high image quality development unit 112 of the present embodiment is configured to suppress an increase in circuit scale and power consumption by avoiding real-time development processing in parallel with shooting and enabling development processing over time.

  The image information developed by the high image quality development unit 112 is supplied to the still image compression unit 141. In S525, the still image compression unit 141 performs high-efficiency encoding processing (still image compression) on the acquired image information. To generate a high-quality still image file. Note that the still image compression unit 141 performs compression processing using a known technique such as JPEG.

  If the recording / reproducing unit 151 records a high-quality still image file in the storage medium 152 or the like in S526, the flow proceeds to S502. The follow-up development process can be executed repeatedly for each image when there is a still image that has not undergone the follow-up development.

  The still image file recorded in S526 has the configuration of the still image file 400 shown in FIG. 4A, and includes a header portion 401, a metadata portion 402, and a compressed data portion 403. The header portion 401 includes an identification code indicating that this file is a still image file format. The compressed data section 403 includes still image compressed data that has been encoded with high efficiency.

  The metadata unit 402 includes file name information 404 of the RAW file that is the source of the still image file. Further, development status information 405 including a flag indicating whether the still image file has been developed with the simple developing unit 111 or developed with the high quality developing unit 112 is included. Development status information 405 indicating that high-quality development has been performed by the high-quality development unit 112 is included. Also, the evaluation value and subject information detected by the evaluation value calculation unit 105 and the recognition unit 131 extracted from the metadata of the original RAW file, and information at the time of shooting from the imaging optical unit 101 and the imaging sensor unit 102 The shooting metadata 406 including is included.

  The recording / playback unit 151 gives the same file name to the new still image file recorded in S526 after high-quality development as the still image file by simple development recorded at the same time as the original RAW file, Overwrite recording. That is, the still image file by simple development is deleted. Then, the recording / playback unit 151 updates the development status 415 in the metadata unit 412 with information indicating that the high-quality development has been performed (or the follow-up development has been completed) for the RAW file that is the source of the follow-up development.

  As described above, the imaging apparatus 100 according to the present embodiment performs the follow-up development when the processing load of the apparatus is relatively low, waiting for a user operation such as between shootings or in a playback mode or a sleep state. Then, the still image file by the simple development at the time of shooting is replaced with the still image file by the high quality development using the RAW file. A moving image file by simple development at the time of shooting is also replaced with a moving image file by high quality development using a RAW file. By doing this, even when a request for high-quality playback such as confirmation display of details or printout is given, extra development processing does not occur each time, and as with conventional still image files Can be used in a general environment.

  Next, the operation in the still image playback mode of the imaging apparatus 100 will be described. FIG. 6 shows a flowchart relating to processing in the still image reproduction mode of the present embodiment. The flowchart of FIG. 6 illustrates a processing procedure executed by controlling each processing block by the control unit 161. A program stored in a memory (ROM) included in the control unit 161 is stored in the memory (RAM). This is realized by expanding and executing by the CPU.

  In FIG. 6, when the still image playback mode process is started, the control unit 161 determines whether the processing load status of the imaging apparatus 100 is low in S <b> 601. If the processing load is low at a frequency according to the load status, the process shifts to the idle state in S610, and otherwise the process proceeds to S602. For example, while waiting for a user operation such as a reproduction instruction, the processing load is low, and thus the process proceeds to S610. If playback of a still image has been started in response to an operation from the user (including a state during playback), the process proceeds to S602. In step S602, the control unit 161 determines whether an enlarged display instruction has been received from the user for the still image to be played back. If there is an enlarged display, the process proceeds to S603, and if there is no enlarged display, the process proceeds to S620.

  FIG. 7 illustrates types of display forms including enlarged display. FIG. 7 is a diagram illustrating an example of display processing in the still image reproduction mode of the present embodiment. A display example 700 in FIG. 7A is an example in which six images indicated by 701 are reduced and reduced (displayed as a list) on the display unit 123. A display example 710 in FIG. 7B is an example in which an entire image 711 is displayed on the display unit 123, and this display state is a normal display. A display example 720 in FIG. 7C is an example in which an image 721 obtained by enlarging a partial area of a certain image is enlarged and displayed on the display unit 123. For example, when confirming whether or not the focus is appropriate immediately after shooting, it is generally used to enlarge and display the details of the subject image as in the display example 720.

  When the enlarged display is performed as in the display example 720, the flow in FIG. 6 transitions from S602 to S603. When the reduced display is performed as in the display example 700, the process proceeds from S602 to S620. In the case of the display example 710, since the number of display pixels of the display unit 123 is the number of pixels of the still image file by simple development, in the above example, if it is 2 million pixels or less, the display is the same size or reduced. Transition from S602 to S620.

  In S620, the recording / playback unit 151 reads a still image file to be played back from the storage medium 152 or the like. In step S621, the still image expansion unit 143 decodes and expands the still image file. In step S608, the display control unit 122 outputs the display image to the display unit 123 in the form illustrated in each drawing of FIG. .

  If the displayed image is sufficient in the number of pixels of the still image file by simple development, in the above example, 2 million pixels or less, it is sufficient even if the still image file is developed by the simple development unit 111 Can be displayed with high image quality. Needless to say, if the still image file is developed by the high image quality development unit 112, the image quality is sufficient for display.

  On the other hand, in the case of enlarged display, the displayed image may be insufficient in the number of pixels of a still image file by simple development, in the above example, 2 million pixels or less. In other words, display using a still image file by simple development causes a reduction in resolution.

  Therefore, in the case of enlarged display, in S603, the control unit 161 determines whether the still image file of the image to be displayed as a reproduction target is developed by the high image quality development unit 112. At the time of determination, for example, whether the still image file included in the development status 405 stored in the metadata unit 402 of the still image file 400 has been processed by the simple developing unit 111 or processed by the high-quality developing unit 112 Refers to a flag for identifying whether it has been done. There is a method of determining from the referenced flag. Alternatively, the same determination may be made with reference to the development status 415 in the RAW file 410. Alternatively, it may be determined by separately preparing a table file indicating the state of development processing for a series of still images taken.

  If it is determined in step S603 that the image has been developed with high image quality, it means that the image is a high-quality still image file that can be displayed with sufficient image quality even if it is enlarged, and the process advances to step S620. In S620, the recording / playback unit 151 reads the corresponding high-quality still image file from the storage medium 152 or the like, and plays back and displays it. As described above, in the case of a still image file developed by the high image quality developing unit 112, it can be displayed in high image quality by the processing from S620 onward. If it is not determined in step S603 that the image has been developed with high image quality, it means that the file is a still image file developed by the simple development unit 111. Therefore, the process proceeds to step S604, where high image quality development (following development described above) is performed. Execute. In step S <b> 604, the control unit 161 determines whether the RAW file corresponding to the still image to be reproduced is buffered in the buffer unit 115. If buffered, the process proceeds to S606. If not buffered, the recording / playback unit 151 reads the corresponding RAW file from the storage medium 152 or the like and stores it in the buffer unit 115 in S605. Note that the buffer unit 115 updates the data so that the new image captured in the still image capturing mode is preferentially retained. That is, images taken in the past are sequentially removed from the buffer. By doing so, since the image taken immediately before is always held in the buffer, S605 is skipped and the display can be performed at high speed.

  In S606, the RAW decompression unit 114 decodes and decompresses the RAW file read from the buffer unit 115 or the storage medium 152, and restores the RAW image. In step S <b> 607, the restored RAW image is developed with high image quality by the high image quality development unit 112, and is output to the display control unit 122 via the switch unit 121. In step S <b> 608, the display control unit 122 outputs an enlarged image as illustrated in FIG. 7C to the display unit 123. The high image quality developing unit 112 performs debayer processing (demosaic processing) on the RAW image, converts it into a signal composed of luminance and color difference, removes noise contained in each signal, corrects optical distortion, and optimizes the image. A so-called development process is performed. The size (number of pixels) of the developed image generated by the high image quality development unit 112 is the entire size read from the image sensor unit 102 or the size set by the user, and is limited to 2 million pixels or less. The quality is much higher than that of the simple developed image. Therefore, a still image developed by the high image quality developing unit 112 can answer the request for enlarged display with sufficient image quality. When the display in S608 is stopped, the flow returns to S601. In S601, when transitioning to the idle state S610, processing is performed according to the flowchart of FIG. 5 described above.

  As described above, high-quality development after S604 in FIG. 6 is assumed to occur at a timing at which follow-up development has not yet been performed, such as immediately after shooting. In the present embodiment, the chasing development of a still image is gradually ended when the processing load of the apparatus is relatively low, waiting for a user operation such as a shooting mode or a playback mode or a sleep state. As a result, the still image file by simple development is naturally replaced with the still image file by high-quality development. As the replacement progresses, the number of cases where high-quality image development after S604 occurs decreases, and a high-quality image can be quickly output for enlarged display, further improving the operability.

  Further, as described above, when the RAW file is held in the buffer unit 115, S605 can be skipped, so that it can be displayed quickly. Therefore, in the case of the display examples 700 and 710 in FIG. 7, the RAW file corresponding to the image 701 and the image 711 is stored in advance from the storage medium in preparation for the enlarged display so that the RAW file is held as much as possible in the buffer unit 115. It may be read and moved to the buffer unit 115. Prior to the enlargement display instruction, when the recording / playback unit 151 reads out and buffers the corresponding RAW file from the storage medium 152 or the like, when the enlargement display instruction as in the display example 720 is given, It can be displayed more quickly.

  Next, an operation in the moving image shooting mode of the imaging apparatus 100 will be described. FIG. 8 shows a flowchart relating to the processing of the moving image shooting mode of the present embodiment. The flowchart of FIG. 8 illustrates a processing procedure executed by controlling each processing block by the control unit 161. A program stored in a memory (ROM) of the control unit 161 is stored in the memory (RAM). This is realized by expanding and executing by the CPU.

  In FIG. 8, when the processing of the moving image shooting mode is started, the control unit 161 determines whether or not the processing load status of the imaging apparatus 100 is low in S801. If the processing load is low at a frequency according to the load status, the process shifts to the idle state in S820, and otherwise the process proceeds to S802. For example, when a moving image with a large number of pixels such as a horizontal resolution equivalent to 4000 pixels (4K) or a moving image with a high frame rate such as 120 frames per second (120P) is set, the processing load is high, and thus the process proceeds to S820. Does not transition and always proceeds to S802. When the setting is made to shoot a moving image in which the number of pixels is less than a predetermined value or the frame rate is lower than a predetermined speed, for example, half the frequency between the processing of the first frame and the second frame of the moving image. Then, the process proceeds to S820.

  In step S <b> 802, the camera control unit 104 controls the operations of the imaging optical unit 101 and the imaging sensor unit 102 so that moving image shooting is performed under suitable conditions. For example, a lens included in the imaging optical unit 101 is moved in accordance with a user's zoom or focus instruction, or a reading area of the imaging sensor unit 102 is set in accordance with an instruction on the number of pixels to be captured. In addition, based on evaluation value information and subject information supplied from an evaluation value calculation unit 105 and a recognition unit 131, which will be described later, control such as focus adjustment and tracking for a specific subject is performed.

  In step S <b> 803, the sensor signal processing unit 103 performs signal processing for pixel restoration on the electrical signal converted by the imaging sensor unit 102. Here, interpolation is performed using the peripheral pixel value or a predetermined offset value is subtracted from the value of the missing pixel or the pixel having low reliability. In the present embodiment, the image information output from the sensor signal processing unit 103 after finishing the processing of S803 is referred to as a RAW image meaning a raw (undeveloped) moving image. In step S804, the simple development unit 111 develops the RAW image. At this time, the control unit 161 switches the switch unit 121 in the developing unit 110 and selects the output of the image information developed by the simple developing unit 111.

  The simple development unit 111 performs debayer processing (demosaic processing) on a RAW image constituting each frame of a moving image, converts it to a signal composed of luminance and color difference, removes noise included in each signal, and optical distortion So-called development processing such as correcting the image and optimizing the image is performed. Here, a moving image developing process (simple development) performed by the simple developing unit 111 will be described. The simple development unit 111 limits the image size after development to, for example, 2 million pixels or less of HD video, or restricts noise removal and optical distortion correction to a limited process, or omits development processing. Realizes high-speed processing and simple processing. The simple developing unit 111 performs processing after reducing the image size, or restricts a part of the development processing function, so that the imaging apparatus 100 can perform high-speed shooting of, for example, HD size moving images with a small circuit scale. Can be realized with low power consumption.

  The image information developed by the simple development unit 111 is supplied to the evaluation value calculation unit 105. In step S805, the evaluation value calculation unit 105 calculates evaluation values such as a focus state and an exposure state from the luminance value and the contrast value included in the image information. Note that the evaluation value calculation unit 105 may acquire a raw image before development processing and similarly calculate an evaluation value from the raw image.

  Further, the image information developed by the simple development unit 111 is supplied to the recognition unit 131. In S806, the recognition unit 131 detects a subject (such as a face) from the image information and recognizes the subject information. For example, the presence / absence of the face in the image information, the position thereof, authentication of a specific person, and the like are performed, and the result is output as information.

  Further, the image information developed by the simple development unit 111 is supplied to the display control unit 122. The display control unit 122 forms a display image from the image information acquired in S807, and outputs and displays the display image on the display unit 123 or an external display device. The display image by the display unit 123 is used for confirmation display for the user to appropriately frame the subject in the moving image shooting mode. Specifically, as a usage mode specific to moving image shooting, a live view display for appropriately framing a subject not only during recording of a recorded moving image (during standby) but also during recording of a moving image (during REC) Used for. The display image may be displayed on another display device such as an external television from the display control unit 122 via the video output terminal 124. Further, the display control unit 122 uses the evaluation value information and the subject information supplied from the evaluation value calculation unit 105 and the recognition unit 131, for example, to display a marking in a focused area on the display image, A frame can also be displayed at the position of the recognized face.

  In step S808, the control unit 161 determines whether a moving image shot in response to a recording start instruction from the user is being recorded (during REC). If it is being recorded, the process proceeds to step S810. If REC is not in S808 (that is, during standby), the process returns to S801 to repeat the shooting operation and live view display before starting the recording of the moving image.

  In S808, the moving image to be recorded from the start of recording to the end of recording among the captured moving images is compressed by the moving image compression unit 142 in units of frames in S810. Although not described with reference to the drawings, audio information input by a microphone (not shown) is acquired at the same time as shooting of a moving image. The moving image compression unit 142 also performs compression processing on audio information corresponding to the moving image. The moving image compression unit 142 performs high-efficiency encoding processing (moving image compression) on the acquired image information and audio information of the simply developed moving image to generate a moving image file. Note that the moving image compression unit 142 is MPEG-2, H.264, or the like. H.264, H.C. The compression processing is performed using a known moving image compression technique such as H.265. In step S811, the recording / playback unit 151 records the moving image file on the storage medium 152.

  Further, a RAW image of a period corresponding to the moving image to be recorded in S808 is supplied from the sensor signal processing unit 103 to the RAW compression unit 113. The RAW compression unit 113 performs high-efficiency encoding (RAW compression) on the RAW image indicating the same scene as the moving image to be recorded in S812, and converts the RAW image into a RAW file. The RAW file is stored in the buffer unit 115. The high-efficiency encoding performed by the RAW compression unit 113 is processed by a known technique such as wavelet transform or differential encoding, but may be lossy encoding or lossless encoding. Alternatively, the RAW compression of the RAW compression unit 113 may be omitted, and the RAW image may be output through without being compressed. Regardless of the presence or absence of RAW compression, the present embodiment generates a RAW file that can be restored as a high-quality file without significantly losing the image information supplied from the sensor signal processing unit 103.

  In step S813, after the recording / playback unit 151 records the RAW file in the storage medium 152, the flow proceeds to step S801. In S811 and S813, the recording / playback unit 151 may send the moving image file and / or the RAW file to the external storage from the communication terminal 154 via the communication unit 153, and record them in the external storage. good.

  The above is the description of the flow relating to the processing in the moving image shooting mode of the present embodiment. Here, the structure of the moving image file and the structure of the RAW file according to the present embodiment will be described. FIG. 9 is a diagram illustrating a configuration example of a moving image file and a RAW file.

  The moving image file 900 shown in FIG. 9A is recorded in, for example, a predetermined recording area of the storage medium 152 by the recording / playback unit 151. The moving image file 900 includes a header portion 901, a metadata portion 902, and a compressed data portion 903. The header portion 901 includes an identification code indicating that this file is in the format of a moving image file. The compressed data portion 903 includes high-efficiency encoded moving image and audio compressed data.

  The metadata part 902 includes file name information 904 of the RAW file generated simultaneously with the moving image file. In addition, development status information 905 including a flag indicating whether the moving image file has been simply developed by the simple developing unit 111 or developed by the high image quality developing unit 112 is included. The development status information 905 may include the total number of frames included in one scene and the number of frames for which development by high-quality development has been completed (or a frame number for which development has been completed). Also, evaluation values and subject information detected by the evaluation value calculation unit 105 and the recognition unit 131, and information at the time of shooting from the imaging optical unit 101 and the imaging sensor unit 102 (for example, lens type identification information, sensor type identification information) Etc.). In addition, although not shown, it may further include an identification code of a storage medium in which a RAW file generated at the same time is recorded, path information of a recorded folder, and the like.

  The RAW file 910 shown in FIG. 9B is recorded in, for example, a predetermined recording area of the storage medium 152 by the recording / playback unit 151. The RAW file 910 includes a header part 911, a metadata part 912, and a compressed data part 913. The header portion 911 includes an identification code indicating that this file is in the RAW file format. The compressed data section 913 includes moving image RAW compressed data that has been encoded with high efficiency (may be uncompressed moving image RAW image data).

  The metadata portion 912 includes file name information 914 of a moving image file generated simultaneously with the RAW file. Further, development status information 915 indicating that the moving image file has been simply developed by the simple developing unit 111 is included. Also, evaluation values and subject information detected by the evaluation value calculation unit 105 and the recognition unit 131, and information at the time of shooting from the imaging optical unit 101 and the imaging sensor unit 102 (for example, lens type identification information, sensor type identification information) And the like. Further, although not shown in the figure, it may further include an identification code of a storage medium in which a simultaneously generated moving image file is recorded, path information of a recorded folder, and the like. Alternatively, all or part of the moving image file generated at the same time (such as the first frame) may be extracted and converted into metadata, and stored in the metadata unit 912. The structure of the various files according to the present embodiment described above is an example, and a configuration conforming to a standard such as DCF, AVCHD, or MXF may be used.

  As described above, the imaging apparatus 100 according to the present embodiment uses the simple developing unit 111 to perform a captured image display (live view display) in the moving image shooting mode and a developing process for a moving image file generated at the time of shooting. The simple development unit 111 limits the image size after development to 2 million pixels or less, or restricts noise removal and optical distortion correction to a limited process, or omits, for example, an HD size moving image. This development process can be realized with a small circuit scale and low power consumption. On the other hand, as described above, the imaging apparatus 100 according to the present embodiment generates a RAW file corresponding to the recording period of the moving image along with the moving image file. The RAW file is a high-quality file that does not significantly impair the image information supplied from the sensor signal processing unit 103, but development processing is not required to generate this file. Therefore, even if the number of image pixels is increased, such as 4K or 8K (horizontal resolution is equivalent to 8000 pixels), or the frame rate is increased, such as 120 frames per second (120P), less power is consumed by a small circuit. It is possible to record a RAW file.

  FIG. 10 shows a flowchart relating to the chase development processing of a predetermined portion in the moving image from the idle state according to the present embodiment.

  The head part chasing development process according to the present embodiment is a process utilizing the chasing development function described above. Specifically, in the interval between shootings or in a state where the processing load of the apparatus is relatively low, waiting for user operation, the RAW file recorded in the buffer unit 115 or the storage medium 152 is used as a source to improve the image quality again. Apply development processing. That is, it means processing for generating a high-quality display image or a high-quality moving image file obtained from a RAW file every time a predetermined operation state (or background) is possible. In the present embodiment, a moving image file will be described. However, the present invention can also be applied to a plurality of continuous still image files such as still image continuous shooting.

  The effectiveness of the follow-up development is as described above. On the other hand, it is not possible to track and develop all scenes (frames) for data recorded in a relatively long time such as several minutes to several tens of minutes, such as a video file. Issues such as battery consumption inside will also occur. Therefore, by chasing and developing only the head portion as part of each moving image file, it is possible to immediately reproduce a high-quality moving image file while suppressing battery consumption. Since the head part of each moving image file can be reproduced and displayed immediately, it is effective for confirming and reproducing each scene. In addition, immediately after the start of playback of a desired moving image file, it is possible to immediately play back the high-quality developed video at the beginning, and to realize high-quality development of the subsequent portions after the beginning in the background of the playback period. .

  The flowchart of FIG. 10 illustrates a processing procedure executed by the control unit 161 controlling each processing block. This is realized by developing a program stored in a memory (ROM) of the control unit 161 in the memory (RAM) and executing the CPU.

  In FIG. 10, when the process in the idle state is started, the control unit 161 determines whether or not to perform the chase development for the moving image file instructed by the user setting in S1001. When the follow-up development is not performed, the process proceeds to S1002, and when the follow-up development is performed, the process proceeds to S1020.

  When the follow-up development is not performed in S1001, it is determined whether the control unit 161 shifts to any of 201, 202, 203, and 204 shown in FIG. 2 according to the mode setting from the user (S1002, S1003, and S1004). , S1005). Then, the control unit 161 controls to transition to the processing flow of the selected mode (S1010, S1011, S1012, S1013).

  In the storage medium 152 or the like, a moving image file and a RAW file are recorded as one set for each shooting instruction (recording start instruction for one scene). When the follow-up development is performed manually or automatically, the control unit 161 determines whether the follow-up development has been processed or not processed for each set of images in S1020. At the time of determination, for example, the information included in the development status 905 stored in the metadata portion 902 of the moving image file 900 indicates whether the moving image file has been processed by the simple developing portion 111 or the high image quality developing portion 112. Refers to the flag for identifying whether it has been processed by. There is a method of determining from this referenced flag. Alternatively, the same determination may be made with reference to the development status 915 in the RAW file 910. Alternatively, it may be determined by separately preparing a table file indicating the state of development processing for a series of captured moving images.

  In S1020, when it is determined whether the follow-up development has been processed or not yet processed, it is determined whether the follow-up development processing has been completed for a predetermined time from the first frame of each of the plurality of movie files. May be determined so that the image quality is uniformly processed. Also, some steps may be provided as a determination of whether or not the processing has been completed. For example, it may be configured to determine whether or not the processing is performed for the entire shooting time of the moving image. Alternatively, it may be determined that the processing has been completed if a predetermined time (for example, 1 minute) is divided from the head portion of the moving image and the follow-up development processing is performed for a predetermined time from the head. Further, when it is determined whether the follow-up development process has been performed for a predetermined time, the time for determining whether the follow-up development process has been completed by an operation from the operation unit 162 may be set for each moving image file.

  Further, referring to the total number of frames included in the moving image file of one scene shot in response to one recording start instruction and the number of frames that have been developed by high-quality development, whether the frame is a high-quality developed frame or not. It may be determined whether or not. Instead of the number of frames for which development by high image quality development has been completed, a frame number for which development has been completed may be used. In these cases, the development status 905 of the moving image file 900 or the development status 915 of the RAW file 910 stores information on the number of frames that have been developed by high-quality development (or the number of frames that have been developed).

  If the control unit 161 determines that the follow-up development has been processed, the process proceeds to S1002. If there is a moving image that has not been subjected to the follow-up development, the flow proceeds to S1021. In step S <b> 1021, the control unit 161 determines whether a RAW file corresponding to a moving image that has not been chased and developed has been buffered in the buffer unit 115. If it is buffered, the process proceeds to S1023. If it is not buffered, the recording / playback unit 151 reads the corresponding RAW file from the storage medium 152 or the like and stores it in the buffer unit 115 in S1022. Since the buffer unit 115 updates the data so that the new image captured in the moving image capturing mode is preferentially retained, the same effect as that in the still image capturing mode described above can be obtained.

  In step S1023, the RAW decompression unit 114 decompresses a RAW image by decompressing a predetermined time (first part) from the temporal head of the RAW file read from the buffer unit 115 or the storage medium 152. The predetermined time may be a fixed value or variable (can be set by the user). Moreover, it may differ for every moving image file, and may be the same. The restored RAW image of the head portion is developed to a high image quality by the high image quality developing unit 112 in S1024, and is output to the moving image compression unit 142 via the switch unit 121. In the present embodiment, in addition to the above-described effects obtained even in the still image mode, the chasing development processing in the high image quality developing unit 112 is applied only to the head portion of the moving image, thereby reducing the power consumed in the chasing development. it can. Therefore, it is possible to lightly watch without processing wait time when the power consumption is suppressed and the user wants to view the captured moving image with high image quality. However, as a matter of course, it is needless to say that high-quality image development can be performed not only for the head portion but also for the entire shooting time of a moving image by an instruction from the operation unit 162.

  The image information developed by the high image quality developing unit 112 is supplied to the moving image compression unit 142, and the moving image compression unit 142 performs high-efficiency encoding processing (moving image compression) on the image information acquired in S1025, Generate high-quality still image files. Note that the moving image compression unit 142 is MPEG-2, H.264, or the like. H.264, H.C. The compression processing is performed by a known technique such as H.265. If the recording / playback unit 151 records a high-quality moving image file in the storage medium 152 or the like in S1026, the flow proceeds to S1002.

  The moving image file recorded in S1026 has the structure of the moving image file 900 shown in FIG. 9A, and includes a header portion 901, a metadata portion 902, and a compressed data portion 903. The header portion 901 includes an identification code indicating that this file is in the format of a moving image file. The compressed data portion 903 includes compressed data of a moving image (first portion) that has been encoded with high efficiency. The metadata part 902 includes file name information 904 of the RAW file that is the source of the moving image file. Further, a development status indicating that this moving image file has been developed with high quality by the high quality development unit 112 and time information 905 from the beginning of the developed frame as the developed range are included. Also, the evaluation value and subject information detected by the evaluation value calculation unit 105 and the recognition unit 131 extracted from the metadata of the original RAW file, and information at the time of shooting from the imaging optical unit 101 and the imaging sensor unit 102 The shooting metadata 906 including is included.

  The recording / playback unit 151 gives the same file name to the new moving image file recorded at the same time as the original RAW file to the new moving image file recorded in S1026 after high-quality development and overwrite recording. To do. That is, the moving image file by simple development is deleted. Then, the recording / playback unit 151 updates the development status 915 in the metadata unit 912 as information indicating that the high-quality development has been performed (or the follow-up development has been completed) for the RAW file that is the source of the follow-up development. If only the top part is subjected to high-quality development processing, the simple development video file recorded at the same time as the original RAW file is not deleted, and the simple development video file is retained and high-quality development is performed. The recorded moving image file is recorded on the storage medium 152 or the like.

  When a high-quality developed moving image file is recorded on the storage medium 152 or the like, the metadata portion 902 in the configuration of the high-quality developed moving image file 900 includes the file name of the related simple developed moving image file. Further, while holding the compressed data portion 913 of the simple developed moving image file, the metadata portion 912 is included so that the metadata portion 912 includes the file name of the moving image file subjected to the high quality development processing, the portion subjected to the high quality development processing, and the like. Update.

  As described above, the imaging apparatus 100 according to the present embodiment performs the follow-up development when the processing load of the apparatus is relatively low, waiting for a user operation such as between shootings or in a playback mode or a sleep state. Then, the moving image file by the simple development at the time of shooting is replaced with the moving image file by the high image quality development using the RAW file. That is, by performing the follow-up development for each predetermined operation state, the period during which the high-quality image has been developed increases according to the elapsed time, and finally, the entire period of the moving image file becomes the high-quality image developed. In this way, even when a request for high-quality playback is given before high-quality development, there is no need to wait for development processing every time, and a higher-quality image viewing environment than before. Can be provided. Finally, it is also possible to provide an image in which all have high image quality.

  Next, an operation in the moving image reproduction mode of the imaging apparatus 100 will be described.

  FIG. 11 shows a flowchart relating to processing in the moving image playback mode of the present embodiment. The flowchart of FIG. 11 illustrates a processing procedure executed by controlling each processing block by the control unit 161. A program stored in a memory (ROM) included in the control unit 161 is stored in the memory (RAM). This is realized by expanding and executing by the CPU.

  In FIG. 11, when the processing of the moving image playback mode is started, the control unit 161 determines whether or not the processing load status of the imaging apparatus 100 is low in S1101. If the processing load is low at a frequency according to the load status, the process shifts to the idle state in S1110, and otherwise the process proceeds to S1102. For example, while waiting for a user operation such as a reproduction instruction, the processing load is low, and the process proceeds to S1110. If the playback of a moving image has been started in response to an operation from the user (including a state during playback), the process proceeds to S1102.

  In step S1102, the control unit 161 determines whether there is a moving image file that has been subjected to high-quality development processing in the storage medium 152 or the like. If there is a moving image file that has undergone high-quality development processing in S1102, the process proceeds to S1103. If no moving image file exists, the process advances to step S1107. In step S1103, the control unit 161 determines whether or not the end partial playback of the moving image file that has been subjected to the high-quality image development processing has been completed. If it is determined that the reproduction of the moving image file subjected to the high-quality development processing is completed, the process proceeds to S1107. If it is determined that the reproduction of the moving image file subjected to the high image quality development processing has not been completed, the process proceeds to S1104 in order to reproduce the moving image file subjected to the high image quality development processing.

  In step S <b> 1104, the recording / playback unit 151 reads a playback target moving image file that has been subjected to high-quality development processing from the storage medium 152 or the like. In step S1105, the moving image extension unit 144 decodes and expands the read moving image file frame by frame. In step S1106, the display control unit 122 outputs the display image of the reproduced moving image to the display unit 123. Note that the display in S1005 is performed for each frame, and the flow returns to S1001 to display the next frame during playback of the moving image.

  If it is determined in S1102 that there is no moving image file that has undergone high-quality development processing, or if it is determined in S1103 that reproduction of the moving image file that has undergone high-quality development processing has been completed, the processing proceeds to S1107. In step S1107, the control unit 161 determines whether there is a moving image file that has been subjected to the simple development processing in the storage medium 152 or the like. In other words, when it is determined that there is no video file that has undergone simple development processing, it means that the high-quality development processing has been completed for the entire video file, and all of the video files have been overwritten by the simple development video file. . If it is determined that there is a moving image file that has undergone simple development processing, the process advances to step S1108.

  In step S1108, it is determined whether or not the end partial playback of the moving image file that has undergone the simple development process has been completed. If it is determined that the reproduction of the simple developed moving image file has not been completed, the process advances to S1109. In step S <b> 1109, the recording / playback unit 151 reads the playback target moving image file that has been subjected to the simple development process from the storage medium 152 or the like. In step S1105, the moving image extension unit 144 decodes and expands the read moving image file frame by frame. In step S1106, the display control unit 122 outputs the display image of the reproduced moving image to the display unit 123.

  If it is determined in S1107 that there is no moving image file that has undergone the simple development processing, or if it is determined in S1108 that the reproduction of the moving image file that has undergone the simple development processing has been completed, the flow returns to the flow of S1001. In S1001, when transitioning to the idle state S1010, processing is performed according to the flowchart of FIG. 11 described above.

  FIG. 13 is a conceptual diagram of each moving image file when the head portion chasing development is performed. As shown in the figure, a high-quality developed scene can be reproduced at the head of each moving image file (stream). Note that FIG. 13A is an example in which a certain period is set as a high-quality development target as the top part as described above, and FIG. 13B is a diagram in which the period of the top part can be individually set (variable) for each moving image file. ). In either case, it is possible to reproduce a scene that has been developed with high image quality for a predetermined period from the beginning of each stream, but for the remaining period, a scene that has been simply developed is reproduced.

  According to the embodiments described above, simple development is performed at the time of shooting, while high imaging performance is executed by performing high-quality development with a higher processing load in the idle state of the imaging apparatus, and a reproduced image can be output at high speed. It becomes possible to.

<Embodiment 2>
In the first embodiment, for the chasing process of the moving image file, the high-quality developing unit 112 develops the high-quality developing unit 112 by sequentially extending a predetermined time (first part) from the temporal head of the RAW file. On the other hand, in the present embodiment, a case will be described in which the chasing process is not performed sequentially on consecutive frames of one scene, but the frames are thinned out at a predetermined interval, and high-quality image development is performed in a plurality of times. .

  The flowchart in this embodiment is basically the same as that in FIG. 10, but the processing from S1023 to S1026 is different. FIG. 14 shows an example of the follow-up development process corresponding to this embodiment, which is performed instead of S1023 to S1026. The flowchart of FIG. 14 illustrates a processing procedure executed by controlling each processing block by the control unit 161. A program stored in a memory (ROM) included in the control unit 161 is stored in the memory (RAM). This is realized by expanding and executing by the CPU.

  First, in step S1401, the control unit 161 refers to the development status of the moving image file 900, and determines whether follow-up development has already been performed. The development status information 905 of the moving image file 900 corresponding to the present embodiment may include information regarding the number of times the follow-up development has been executed. If the follow-up development has not been performed yet, the process proceeds to S1402. If the first follow-up development has already been performed, the process proceeds to S1406.

  FIG. 15A shows all the image frames of one scene existing in the RAW file, and the frame rate is, for example, an image frame of 60 frames / second. In the first follow-up development, it is determined in S1402 that an even-numbered frame (2nth, where n is an integer) is selected so as to have an interval of one frame as shown in FIG. Next, in step S1403, the RAW decompression unit 114 decompresses the compressed data corresponding to the selected frame in the RAW file read from the storage medium 152 to restore the RAW image. In step S1404, the restored RAW image is developed with high image quality by the high image quality developing unit 112. Next, in step S1405, the moving image compression unit 142 performs high-efficiency encoding processing, generates a moving image file, and records it in the storage medium in step S1406. The moving image file generated with the frames thinned out is hereinafter referred to as “moving image file β”. By thinning out at intervals of one frame, the frame rate of the movie file β becomes 30 frames / second, but the chase development of one scene is completed in half the time compared to the processing time for chasing and developing all the frames. Can do. In addition, the moving image file β has the file structure 900 in FIG. 9 similarly to the moving image file described above, and the moving image file β can be reproduced by the moving image file β alone.

  When the first follow-up development reaches the end of the scene and is completed, the processing shifts to the first follow-up development for another RAW file (another scene) recorded on the storage medium. When the first follow-up development is completed for all the RAW files, the second follow-up development for the first RAW file is executed.

  In the second follow-up development when it is determined in S1401 that it is not the first time, as shown in FIG. 15C, the first undeveloped frame, that is, an odd number (2n + 1th, where n is an integer) High-quality image development and high-efficiency encoding processing are performed on the frame. Alternatively, the odd number may be selected at the first time and the even number may be selected at the second time. That is, in S1407, it is determined to select, for example, an even-numbered (2n + 1) th frame so as to be one frame interval as shown in FIG. Next, in step S1408, the RAW decompression unit 114 decompresses the compressed data corresponding to the selected frame in the RAW file read from the storage medium 152 to restore the RAW image. In step S1409, the restored RAW image is developed with high image quality by the high image quality developing unit 112. In step S1410, the moving image compression unit 142 performs a high-efficiency encoding process to generate a moving image file. At this time, the moving image file β is read from the storage medium, and the encoded data of the first time (moving image file β) and the encoded data of the second time are multiplexed for each frame to generate a moving image file of 60 frames / second. In step S1411, the recording / playback unit 151 records a moving image file of 60 frames / second in the storage medium 152. When the second follow-up development reaches the end of the scene and is completed, the moving image file β may be deleted from the storage medium 152.

  In the high-efficiency encoding process in the moving image compression unit 142, the moving image file β generated by the first follow-up development and the encoded data generated by the second and subsequent follow-up development can be multiplexed for each frame. It needs to be encoded. By using intra-frame coding (intra coding), coding is performed independently for each frame, and there is no dependency on the coded data between the frames. Therefore, the first and second coded data described above are used. Can be multiplexed.

  When inter-frame predictive coding (inter-coding) is used, the coding mode and the reference frame are selected so that the reference destination of the frame in the predictive coding is already processed. In this way, the first and second encoded data can be multiplexed. An example of selecting an encoding mode in this case will be described with reference to FIG.

  In the example of FIG. 16, the frame of one scene shown in FIG. First, as shown in FIG. 16B, in the first follow-up coding, frame 0 is an intra-frame coded picture (I picture), and subsequent frames 2, 4, 6,. Let it be a forward predictive coded picture (P picture). As shown in FIG. 16 (c), in the second follow-up coding, frame 1 is an intra-frame coded picture (I picture), and subsequent frames 3, 5, 7,. Let it be a predictive coded picture (P picture). Since the prediction direction is a frame that has already been processed as indicated by the arrows in the figure, the first and second encoded data can be multiplexed.

  As shown in FIG. 16D, in the first follow-up coding, frame 0 is an intra-frame coded picture (I picture), and subsequent frames 2, 4, 6,. Let it be a forward predictive coded picture (P picture). As shown in FIG. 16E, in the second follow-up coding, frames 1, 3, and 5 are set as inter-frame bi-directional predictive coded pictures (B pictures). Since the prediction direction is a frame that has already been processed as indicated by the arrows in the figure, the first and second encoded data can be multiplexed.

  As described above, in the follow-up development of a moving image, the time required for the first follow-up development is shortened by processing by thinning out the frame, so that the movie file can be reproduced by shooting with high image quality, although the frame rate is lowered. The time to become can be shortened.

As described above, the imaging apparatus 100 according to the present embodiment performs the follow-up development when the processing load of the apparatus is relatively low, waiting for a user operation such as a shooting mode or a playback mode or a sleep state. To do. Then, the moving image file by the simple development at the time of shooting is replaced with the moving image file by the high image quality development using the RAW file. A moving image file by simple development at the time of shooting is also replaced with a moving image file by high quality development using a RAW file. As a result, even when a request for high-quality reproduction is given, such as a detailed confirmation display or viewing on a large screen monitor, extra development processing does not occur each time. Further, it can be used in a general environment similar to a conventional moving image file. <Embodiment 3>
Next, as a third embodiment, an operation in the moving image reproduction mode of the imaging apparatus 100 will be described. FIG. 17 shows a flowchart relating to processing of the moving image playback mode of the present embodiment. The flowchart of FIG. 17 illustrates a processing procedure executed by controlling each processing block by the control unit 161. A program stored in a memory (ROM) included in the control unit 161 is stored in the memory (RAM). This is realized by expanding and executing by the CPU.

  In FIG. 17, when the processing of the moving image reproduction mode is started, the control unit 161 determines whether or not the processing load status of the imaging apparatus 100 is low in S1701. If the processing load is low at a frequency according to the load status, the process transits to the idle state in S1710, and otherwise the process proceeds to S1702. For example, while waiting for a user operation such as a reproduction instruction, the processing load is low, and the process proceeds to S1710. If the reproduction of the moving image has been started in accordance with the operation from the user (including the state during reproduction), the process proceeds to S1702.

  In step S <b> 1702, the control unit 161 determines whether a playback pause (pause) instruction has been received from the user for the video to be played back. If there is no pause instruction in S1702, the flow proceeds to S1703 in order to continue the video playback. The processing from S1703 to S1705 is the same as S1104 to S1106 in FIG. When the pause instruction is received in S1702, the control unit 161 causes the moving image being played back and displayed to be paused, and displays the frame at the stop position when paused as a still image, to S1720. Transition the flow. In the paused state, the image is displayed in a static state, so that it becomes easier to see the detailed image quality than when moving. Further, it is considered that an enlarged display instruction is easily received during the pause. Therefore, in order to provide a display of a higher quality image than a moving image file that is simply developed, a RAW image of a moving image RAW file corresponding to the moving image file being played back in S1720 and further corresponding to a frame that is being paused displayed. The recording / reproducing unit 151 reproduces the frames. At this time, if the RAW file to be reproduced is buffered in the buffer unit 115, the RAW file is read from the buffer unit 115. If not, the RAW file is read from the storage medium 152 or the like.

  In S1721, the RAW decompression unit 114 decodes and decompresses the RAW file read from the buffer unit 115 or the storage medium 152, and restores the RAW image. In step S1722, the restored RAW image is developed with high image quality by the high image quality developing unit 112. The imaging apparatus 100 can also capture a high-quality still image that is developed from the moving image RAW file with high image quality and that corresponds to the frame that is being paused and displayed as a new still image file. In step S <b> 1723, the control unit 161 determines whether a user instruction for capturing the display image at the still position as a still image has been received. If the still image capture instruction is not accepted in S1723, the still image developed from the moving image RAW file with high image quality is supplied to the display control unit 122. In S1705, the display control unit 122 develops the still image developed to high image quality. Are output to the display unit 123. By this processing, the image being paused and displayed by the moving image file is replaced with the display image of the still image developed from the RAW image with high image quality.

  If an instruction to capture a still image is received in S1723, the image information developed by the high image quality development unit 112 is supplied to the still image compression unit 141 in S1722. The still image compression unit 141 performs high-efficiency encoding processing (still image compression) on the image information acquired by capture in S <b> 1724 to generate a high-quality still image file. Note that the still image compression unit 141 performs compression processing using a known technique such as JPEG.

  If the recording / playback unit 151 records a high-quality still image file in the storage medium 152 or the like in S1725, the flow moves to S1705. The still image developed from the moving image RAW file with high image quality is supplied to the display control unit 122, and the display control unit 122 outputs the display image of the still image developed with high image quality to the display unit 123 in S1705. By this processing, the image being paused and displayed by the moving image file is replaced with the display image of the still image developed from the RAW image with high image quality.

  The high-quality still image file generated by the still image compression unit 141 in S1724 forms the configuration of the still image file 400 in FIG. The metadata section 402 stores the file name of the RAW file of the moving image that is the source of capture as the file name information 404 of the RAW file. Further, time information of a frame captured as a still image is stored as the shooting metadata 406, and the corresponding frame position of the moving image RAW file can be indicated. Alternatively, the corresponding frame of the moving image RAW file may be extracted as a still image, and a new RAW file 410 to be paired may be created at this time. Regarding the generation of the RAW file of the still image, the still image file and the RAW file are configured by the method described in S310 to S313 of the still image shooting mode in FIG.

  Note that the display in S1705 is performed for each frame, and the flow returns to S1701 to display the next frame during playback of the moving image. In S1701, when transitioning to the idle state S1710, processing is performed according to the flowchart of FIG. 14 described above.

  As described above, the imaging apparatus 100 according to the present embodiment can easily reproduce a moving image without delay using a moving image file recorded at the time of shooting, and in a paused state, a still image developed from a RAW file with high image quality. The image can be replaced and displayed. Furthermore, this high-quality still image can be easily captured as a still image file.

  Further, it is assumed that the high-quality development after S1720 in FIG. 17 occurs at the timing when the follow-up development is not yet executed, such as immediately after shooting. In this embodiment, the chasing development of the video is gradually terminated when the processing load of the apparatus is relatively low, waiting for a user operation such as between shootings, playback mode, sleep mode, etc. The file is naturally replaced with a video file by high-quality development. As the replacement proceeds, the number of cases where high-quality image development after S1720 occurs decreases, and a high-quality image can be output promptly and the operability is further enhanced. In the moving image reproduction mode, a list of moving image files recorded on the storage medium 152 may be displayed when starting reproduction. Reference numeral 1800 shown in FIG. 18 is a list display of moving image files, which is an example displayed on the display unit 123. In the figure, reference numeral 1801 denotes a reduced image obtained by reducing one frame (for example, the first frame) of each of the six moving image files. Reference numeral 1802 denotes a time bar representing the time when each moving image file was shot. In addition to this time bar, the development status of the above-described moving image file may be expressed.

  The development status includes “simple development state”, “the top part is in high-quality development”, “frame thinning-out high-quality development (the state of the aforementioned video file β)”, “all” "The state of the high-quality developed frame". These development status information may be displayed with different patterns and colors, or may be displayed with the time bar length changed according to the progress of development processing. By adopting such a display form, it becomes easier for the user to know which moving image file has been developed with high image quality or how long it takes to perform high image quality development, thereby further improving convenience.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (15)

An image processing apparatus,
First developing means for developing a RAW image;
Compression means for compression-encoding the developed RAW image to generate a first image file;
RAW compression means for compressing the RAW image and generating a RAW file;
Recording means for recording the first image file and the RAW file on a storage medium;
Decompression means for decompressing the RAW file to generate a RAW image;
A second developing unit that performs a developing process with a higher processing load on the RAW image obtained by the expansion than the developing process performed by the first developing unit;
Playback means for playing back the moving image obtained from the first image file or the RAW file from the storage medium,
When the image processing apparatus is in a predetermined operation state, the expansion unit expands a part of a plurality of frames included in the RAW file, and the second developing unit extracts the RAW image obtained by the expansion. Execute development processing with high processing load,
When the reproducing unit reproduces the moving image, a part of the plurality of frames corresponding to the moving image reproduces a developed image from the RAW image obtained by the second developing unit, and the other part. An image processing apparatus for reproducing a developed image obtained from the first image file.   2. The second development processing unit executes the development processing with a high processing load on a RAW image of a predetermined portion of the RAW images used for generating the first image file. An image processing apparatus according to 1.   The image processing apparatus according to claim 2, wherein the RAW image of the predetermined portion is a RAW image included in a predetermined period from the top of the RAW images used for generating the first image file.   The image processing apparatus according to claim 3, wherein the predetermined period increases in accordance with an elapsed time, and finally becomes the entire period of the first image file. Decoding means for decoding the first image file to generate a reproduced moving image;
Display control means for displaying the reproduced moving image on a display unit;
The image processing apparatus according to claim 1, further comprising: The predetermined RAW image is a 2n-th (n is an integer) RAW image among the RAW images used to generate the first image file.
The compression unit executes a compression encoding based on an encoding process based on an intra encoding or an inter encoding for the 2n-th RAW image developed by the second development processing unit, and performs a third image file. Produces
The recording means further records the third image file on the storage medium,
The image processing apparatus according to claim 2, wherein the inter coding refers to a processed intra-coded picture and a picture inter-coded for forward prediction. After the third image file is generated for the 2n-th RAW image,
The second development processing means further executes the development processing with a high processing load on the 2n + 1th RAW image (n is an integer) among the RAW images used to generate the first image file,
The compression means executes a compression process based on an encoding process based on intra coding or inter coding on the 2n + 1-th RAW image developed by the second development processing means, and performs the third image file. To generate a fourth image file,
The recording means overwrites the third image file with the fourth image file and further records the file on the storage medium;
The inter coding for the 2n + 1 th RAW image refers to a processed intra coded picture based on the 2n + 1 th RAW image and a picture inter coded for forward prediction. Item 7. The image processing apparatus according to Item 6. After the second image file is generated for the 2n-th RAW image,
The second development processing means further executes the development processing with a high processing load on the 2n + 1th RAW image (n is an integer) among the RAW images used to generate the first image file,
The compression means performs a compression process based on an encoding process by inter-coding on the 2n + 1-th RAW image developed by the second development processing means, and multiplexes with the third image file to obtain a fourth Generate an image file of
The recording means overwrites the third image file with the fourth image file and further records the file on the storage medium;
In the inter coding for the 2n + 1th RAW image, an encoded picture by bi-directional prediction that refers to an intra-coded picture and a picture that has been inter-coded for forward prediction included in the third image file. The image processing apparatus according to claim 6, wherein the image processing apparatus is generated. The first image file is a moving image file,
Decoding means for decoding the first image file to generate a reproduced moving image;
Display control means for displaying the reproduced moving image on a display unit;
Operation means for receiving an operation instruction from a user of the image processing apparatus,
When the playback moving image is displayed on the display unit, the display is paused when the operation unit receives a pause operation instruction, and the second developing unit displays the display according to the pause. Executing the development processing with a high processing load on the RAW image corresponding to the image displayed in the section,
2. The image processing according to claim 1, wherein the display control unit causes the display unit to display a RAW image developed by the second developing unit in place of the paused reproduction moving image. 3. apparatus. When the operation means receives an instruction to capture a RAW image displayed in place of the paused playback video, the compression means compresses and encodes the RAW image to generate a still image file,
The image processing apparatus according to claim 9, wherein the recording unit records the still image file on the storage medium. A display control means for displaying a list of image files recorded in the storage medium on the display unit;
11. The display of each image file in the list display includes status information related to development processing by the first developing unit and the second developing unit. An image processing apparatus according to 1. A buffer for holding a RAW file generated by the RAW compression means;
The decompressing means decompresses the RAW file held in the buffer when the RAW file to be decompressed is held in the buffer, and stores the memory when the RAW file to be decompressed is not held in the buffer. The image processing apparatus according to claim 1, wherein the RAW file read from the medium is decompressed. Imaging means for capturing a subject image and generating a RAW image;
An imaging apparatus comprising the image processing apparatus according to claim 1. First developing means for developing a RAW image;
Compression means for compression-encoding the developed RAW image to generate a first image file;
RAW compression means for compressing the RAW image and generating a RAW file;
Recording means for recording the first image file and the RAW file on a storage medium;
Decompression means for decompressing the RAW file to generate a RAW image;
A second developing unit that performs a developing process with a higher processing load on the RAW image obtained by the expansion than the developing process performed by the first developing unit;
A control method of an image processing apparatus comprising: a playback unit that plays back a moving image obtained from the first image file or the RAW file from the storage medium,
When the image processing apparatus is in a predetermined operation state,
The decompression means decompressing a part of a plurality of frames included in the RAW file;
A step in which the second developing unit executes the development processing with a high processing load on a RAW image obtained by the expansion;
When the playback means plays back the video,
In a part of the plurality of frames corresponding to the moving image, a developed image from the RAW image obtained by the second developing unit is reproduced, and in the other part, a developed image obtained from the first image file is reproduced. A method for controlling the image processing apparatus.   A program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 12.

Images