JP6192319B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method, and more particularly to a technique for handling a RAW image of a moving image or a still image.

  In a conventional imaging apparatus, raw image information (RAW image) captured by an imaging sensor is debayered (demosaiced), converted into a signal composed of luminance and color difference, noise removal, and optical distortion for each signal. So-called development processing such as correction and image optimization is performed. In general, the developed luminance signal and color difference signal are compression encoded and recorded on a recording medium.

  On the other hand, there are imaging devices that can record RAW images. RAW images require a large amount of data for recording, but are preferred by advanced users because they have the advantage that correction and deterioration of the original image can be minimized and can be edited after shooting.

  The configuration of an imaging apparatus that records a RAW image is disclosed in Patent Document 1. Patent Document 1 describes a configuration in which development parameters are recorded together with a RAW image, and the RAW image is developed and reproduced using the development parameters during reproduction.

JP 2011-244423 A

  In recent imaging apparatuses, the number of pixels per image has greatly increased due to the evolution of imaging sensors. Also, the number of images that can be taken continuously per second tends to increase. For this reason, each processing amount for development processing such as debayer processing, noise removal, and optical distortion correction for RAW images increases synergistically, and in order to perform real-time development processing in parallel with shooting, enormous circuits and consumption are required. Electricity is spent. In some cases, it may not be possible to execute high photographing performance due to the occupation of the circuit by development processing and the limitation on power consumption.

  On the other hand, if the configuration is such that the RAW image is recorded without being developed as in Patent Document 1 described above, the processing amount related to the development at the time of shooting may be reduced, but the RAW image is recorded in the undeveloped state. Therefore, it is difficult to quickly reproduce and display the image. Furthermore, the RAW image cannot be reproduced (developed) by other devices due to the uniqueness of the data specific to the RAW image, and the conventional RAW image recording method may impair the convenience of the user.

  As described above, in order to perform high shooting performance with a conventional apparatus and to allow a reproduced image to be output at high speed, a high-cost circuit can be installed so that it can be driven at high output, or a RAW image can be driven. There was a problem that had to be recorded so that it could be played back quickly and easily.

  Accordingly, an object of the present invention is to provide an image processing apparatus and an image processing method that enable high-speed display and high-quality display of necessary portions using a developed recorded image and a RAW recorded image. There is to do.

In order to achieve the above object, an image processing apparatus of the present invention provides:
An imaging unit that captures a subject image to generate a RAW image, a first developing unit that performs development processing on the RAW image, and obtains an image with higher image quality than the first developing unit for the RAW image A second developing means for performing a developing process for the image, an image obtained by performing the developing process on the RAW image generated by the imaging means in response to a photographing instruction by the first developing means, and the RAW image And a first recording control means for controlling the recording to be recorded on a recording medium, and after the photographing, the RAW image recorded on the recording medium is subjected to a developing process by the second developing process, and the second A second recording control unit that controls to record an image that has been developed by the development process on the recording medium; a display processing unit that outputs a display image based on the image recorded on the recording medium; Display image When the enlargement of the partial area of the display image is instructed by the operation means for instructing enlargement of the partial area, the development processing is performed by the second developing means corresponding to the display image When the image is recorded on the recording medium, the display is controlled so as to output an enlarged display image in which a partial area is enlarged based on the image developed by the second developing unit. The image developed by the first developing unit corresponding to the image is recorded on the recording medium, but the developing process by the second developing unit is not completed, and the second developing unit. If the image subjected to the developing process is not recorded on the recording medium, the second developing unit performs the developing process on the RAW image corresponding to the display image to generate an image, Generated Characterized by comprising control means for controlling to output the enlarged image obtained by enlarging a partial area based on the image.

  According to the present invention, it is possible to obtain a RAW image that is more convenient than conventional ones.

It is a block diagram which shows the structural example of the imaging device which concerns on embodiment of this invention. It is a state transition diagram of this embodiment. It is a flowchart which concerns on the process of the still image shooting mode of this embodiment. It is a figure which shows the structural example of the still image file and RAW file of this embodiment. It is a flowchart which concerns on the process of an idle state of this embodiment. It is a flowchart which concerns on the process of the still image reproduction mode of this embodiment. It is a figure which shows the example of the display process of the still image reproduction mode of this embodiment. It is a flowchart which concerns on the process of the moving image shooting mode of this embodiment. It is a figure which shows the structural example of the moving image file and RAW moving image file of this embodiment. It is a flowchart which concerns on the process of the moving image reproduction mode of this embodiment. It is explanatory drawing of a pixel arrangement | sequence.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram illustrating a configuration example of an imaging apparatus according to an embodiment of the present invention. The imaging apparatus 100 shown in FIG. 1 not only records image information obtained by imaging a subject on a recording medium, but also reproduces the image information from the recording medium, develops and displays the information, and image information. Has a function to transmit / receive data to / from an external device or server (cloud). Therefore, the imaging apparatus according to the embodiment of the present invention can be expressed as an image processing apparatus, a recording apparatus, a reproducing apparatus, a recording / reproducing apparatus, a communication apparatus, and the like.

  In FIG. 1, the control unit 161 includes a CPU and a memory that stores a control program executed by the CPU, and controls the overall processing of the imaging apparatus 100. The operation unit 162 includes input devices such as keys, buttons, and a touch panel used by the user to give an instruction to the imaging apparatus 100. An operation signal from the operation unit 162 is detected by the control unit 161, and is controlled by the control unit 161 so that an operation corresponding to the operation is executed. The display unit 123 includes a liquid crystal display (LCD) or the like for displaying an image taken or reproduced in the imaging apparatus 100, a menu screen, various information, and the like.

  When an operation unit 162 is instructed to start a shooting operation, an optical image of a subject to be imaged is input via the imaging optical unit 101 and formed on the imaging sensor unit 102. At the time of shooting, the operations of the imaging optical unit 101 and the imaging sensor unit 102 include the evaluation value calculation result obtained by the evaluation value calculation unit 105 such as an aperture value, focus, camera shake, and the face recognition result extracted by the recognition unit 131. Control is performed by the camera control unit 104 based on the subject information.

  The image sensor unit 102 converts light transmitted through a red, green, and blue (RGB) mosaic color filter arranged for each pixel into an electrical signal. FIG. 11 is a diagram illustrating an example of a color filter arranged in the imaging sensor unit 102, and illustrates a pixel array of an image handled by the imaging apparatus 100. As shown in FIG. 11, red (R), green (G), and blue (B) are arranged in a mosaic pattern for each pixel, and for 2 × 2 four pixels, one red pixel, one blue pixel, and two green pixels It has a structure in which pixels are regularly arranged in one set. Such an arrangement of pixels is generally called a Bayer array.

  The electrical signal converted by the imaging sensor unit 102 is subjected to pixel restoration processing by the sensor signal processing unit 103. The restoration process includes a process of interpolating a pixel to be restored using a peripheral pixel value or subtracting a predetermined offset value from a missing pixel value or a low-reliability pixel value in the image sensor unit 102. It is. In the present embodiment, the image information output from the sensor signal processing unit 103 is referred to as RAW image information (hereinafter, RAW image) meaning a raw (undeveloped) image.

  The RAW image is developed by the developing unit 110. The development unit 110 includes a plurality of different development processing units, and includes a simple development unit 111 as a first development unit and a high-quality development unit 112 as a second development unit, and selects an output thereof. A switch unit 121 is included. Both the simple developing unit 111 and the high-quality developing unit 112 perform debayer processing (demosaic processing), that is, color interpolation processing, on the RAW image, and convert it into luminance and color difference (or primary color) signals, which are included in each signal. A so-called development process is performed to remove noise, correct optical distortion, and optimize an image.

  In particular, the high image quality development unit 112 performs each process with higher accuracy than the simple development unit 111. Since it is highly accurate, a developed image with higher image quality than that of the simple developing unit 111 can be obtained, but on the other hand, the processing load increases. Therefore, the high image quality development unit 112 according to the present embodiment is not specialized for real-time development in parallel with shooting, and has a configuration capable of performing distributed processing over an idle period after shooting. Yes. As described above, the high-quality image development is not performed at the time of shooting but is performed later over time, so that an increase (peak) in circuit scale and power consumption can be suppressed to a low level. On the other hand, the simple development unit 111 has a lower image quality than the high image quality development unit 112, but is configured to have a smaller processing amount than the high image quality development so that the development process can be performed at high speed during shooting. Yes. Since the processing load of the simple developing unit 111 is small, the simple developing unit 111 is mainly used for real-time development in parallel with the photographing operation. The switch unit 121 is switched by the control unit 161 in accordance with control according to the operation content instructed by the user from the operation unit 162 and the operation mode being executed. Further, in conjunction with the switching of the switch unit 121, only one of the simple developing unit 111 and the high image quality developing unit 112 that is the target of signal output may perform the developing operation.

  In the present embodiment, the simple developing unit 111 and the high image quality developing unit 112 exist independently in the developing unit 110. However, one developing unit switches the operation mode so that the simple developing unit and the high developing unit 112 can Image quality development processing may be selectively performed.

  The image information developed by the developing unit 110 is displayed on the display unit 123 after a predetermined display process is performed by the display processing unit 122. Further, the developed image information can be output to an external display device connected to the outside through the video output terminal 124. The video output terminal 124 includes a general-purpose interface such as HDMI or SDI.

  The image information developed by the developing unit 110 is also supplied to the evaluation value calculating unit 105. The evaluation value calculation unit 105 calculates evaluation values such as a focus state and an exposure state from the image information.

  The image information developed by the developing unit 110 is also supplied to the recognition unit 131. The recognition unit 131 has a function of detecting and recognizing subject information such as a face and a person in image information. For example, a face in the screen represented by the image information is detected. If there is a face, information indicating the position of the face is output, and a specific person is authenticated based on feature information such as the face.

  Image information developed by the developing unit 110 is supplied to the still image compressing unit 141 or the moving image compressing unit 142. When compressing the image information as a still image, the still image compression unit 141 is used. When the image information is compressed as a moving image, the moving image compression unit 142 is used. The still image compressing unit 141 and the moving image compressing unit 142 respectively perform high-efficiency encoding (compression encoding) on the target image information, generate image information whose information amount is compressed, and generate an image file (still image file, Or a video file). Still image compression is JPEG, etc., and moving image compression is MPEG-2, H.264. H.264, H.C. 265 or the like can be used.

  The RAW compression unit 113 performs high-efficiency encoding on the RAW image data output from the sensor signal processing unit 103 using a technique such as wavelet conversion or differential encoding, and converts the data into a compressed RAW file. And stored in the buffer unit (storage medium) 115. The RAW file can remain in the buffer unit 115 and can be read again. However, after being stored in the buffer unit 115, it is moved to another recording medium and recorded (deleted from the buffer unit 115). Also good.

  The RAW file and the above-described still image file and moving image file are recorded on the recording medium 152 by the recording / playback unit 151. The recording medium 152 is, for example, a built-in large-capacity semiconductor memory, a hard disk, or a removable memory card. The recording / playback unit 151 can also read a still image file, a moving image file, and a RAW file from the recording medium 152.

  The recording / playback unit 151 can write and read various files to an external storage or server via the communication unit 153. The communication unit 153 has a configuration in which the communication terminal 154 can be used to access the Internet or an external device through wireless communication or wired communication.

  When the reproduction operation is started, the recording / reproducing unit 151 acquires and reproduces a desired file from the recording medium 152 or via the communication unit 153. If the file to be played is a RAW file, the recording / playback unit 151 stores the acquired RAW file in the buffer unit 115. If the file to be played back is a still image file, the recording / playback unit 151 supplies the acquired still image file to the still image expansion unit 143. If the file to be played is a moving image file, the recording / playback unit 151 supplies the acquired moving image file to the moving image decompression unit 144.

  The RAW decompression unit 114 reads the RAW file stored in the buffer unit 115, decodes and decompresses the compressed RAW file. The RAW file expanded by the RAW expansion unit 114 is supplied to the development unit 110 and input to the simple development unit 111 and the high image quality development unit 112 in the development unit 110.

  The still image decompressing unit 143 decodes and decompresses the input still image file, and supplies it to the display processing unit 122 as a playback image of the still image. The moving image decompression unit 144 decodes and decompresses the input moving image file, and supplies it to the display processing unit 122 as a reproduced image of the moving image.

  Next, operation modes of the imaging apparatus 100 of the present embodiment will be described in detail with reference to the drawings. FIG. 2 is a state transition diagram illustrating transition of each operation mode in the imaging apparatus 100. Such a mode transition is executed according to a user operation instruction from the operation unit 162 or a determination of the control unit 161, and may be manually changed according to the operation, or may be automatically changed. As illustrated in FIG. 2, the imaging apparatus 100 includes four modes via an idle state (200), a still image shooting mode (201), a still image playback mode (202), a moving image shooting mode (203), and a moving image playback mode. The operation is switched to (204) as appropriate.

  Next, the operation in the still image shooting mode of the imaging apparatus 100 will be described.

  FIG. 3 shows a flowchart relating to processing in the still image shooting mode of the present embodiment. The flowchart in FIG. 3 illustrates a processing procedure executed by controlling each processing block by the control unit 161. A program stored in a memory (ROM) of the control unit 161 is stored in the memory (RAM). This is realized by expanding and executing by the CPU.

  In FIG. 3, when the processing of the still image shooting mode is started (S300), the control unit 161 determines whether or not the processing load status of the image capturing apparatus 100 is low (S301), and the idle state with a frequency according to the load status. The process proceeds to (S320), and if not, the process proceeds to S302. For example, during high-speed continuous shooting, the processing load is high, so the process does not transition to S320 but always proceeds to S302. In the case of performing normal single shooting, the process proceeds to S320 with a half frequency between the first shooting and the second shooting, for example.

  In step S <b> 302, the camera control unit 104 controls the operations of the imaging optical unit 101 and the imaging sensor unit 102 so that shooting is performed under suitable conditions. For example, a lens included in the imaging optical unit 101 is moved in accordance with a user's zoom or focus instruction, or a reading area of the imaging sensor unit 102 is set in accordance with an instruction on the number of pixels to be captured. In addition, based on evaluation value information and subject information supplied from an evaluation value calculation unit 105 and a recognition unit 131, which will be described later, control such as focus adjustment and tracking for a specific subject is performed.

  In step S <b> 303, the sensor signal processing unit 103 performs signal processing for pixel restoration on the electrical signal converted by the imaging sensor unit 102. In other words, the sensor signal processing unit 103 interpolates using missing pixel values or subtracts a predetermined offset value for the values of missing pixels or pixels with low reliability. In the present embodiment, the image information output from the sensor signal processing unit 103 after finishing the processing of S303 is referred to as a RAW image meaning a raw (undeveloped) image.

  In S304, the simple developing unit 111 develops the RAW image. At this time, the control unit 161 switches the switch unit 121 in the developing unit 110 and selects the output of the image information developed by the simple developing unit 111.

  The simple developing unit 111 performs debayer processing (demosaic processing), that is, color interpolation processing, on the RAW image, converts it into a signal composed of luminance and color difference (or primary color), removes noise contained in each signal, optical So-called development processing is performed such as correcting distortion and optimizing the image. Here, the development processing (simple development) performed by the simple development unit 111 will be described. The simple development unit 111 limits the image size after development to, for example, 2 million pixels or less, or restricts or eliminates noise removal and optical distortion correction to limit processing, Simple processing is realized. The simple development unit 111 performs processing after reducing the image size, or restricts a part of the development processing function, so that the imaging apparatus 100 captures a performance of, for example, 60 frames per second of 2 million pixels. It can be realized with a small circuit scale and low power consumption.

  The image information developed by the simple development unit 111 is supplied to the evaluation value calculation unit 105. The evaluation value calculation unit 105 calculates an evaluation value such as a focus state or an exposure state from a luminance value or a contrast value included in the image information (S305). Note that the evaluation value calculation unit 105 may acquire a raw image before development processing and similarly calculate an evaluation value from the raw image.

  Further, the image information developed by the simple development unit 111 is supplied to the recognition unit 131. The recognition unit 131 detects a subject (such as a face) from the image information and recognizes the subject information. For example, the presence / absence of a face in the image information, its position, authentication of a specific person, and the like are performed, and the result is output as information (S306).

  Further, the image information developed by the simple developing unit 111 is supplied to the display processing unit 122. The display processing unit 122 forms a display image from the acquired image information, outputs it to the display unit 123 or an external display device, and displays it (S307). The display image by the display unit 123 is used for live view display (shooting through image display) for the user to appropriately frame the subject in the still image shooting mode. The display image may be displayed on another display device such as an external television from the display processing unit 122 via the video output terminal 124. Further, the display processing unit 122 utilizes the evaluation value information and the subject information supplied from the evaluation value calculation unit 105 and the recognition unit 131, for example, to display a marking in a focused area on the display image, A frame can also be displayed at the position of the recognized face.

  In S308, the control unit 161 determines an input of a shooting instruction from the user. If there is a shooting instruction, the process proceeds to S310. If there is no shooting instruction in S308, the process returns to S301 to repeat the shooting preparation operation and the live view display.

  In response to the shooting instruction in S <b> 308, the image information developed by the simple development unit 111 is supplied to the still image compression unit 141. The still image compression unit 141 performs high-efficiency encoding processing (still image compression) on the acquired image information (S310), and generates a still image file. Note that the still image compression unit 141 performs compression processing using a known still image compression technique such as JPEG.

  In S <b> 311, the recording / playback unit 151 records the still image file on the recording medium 152.

  Further, in response to the shooting instruction in S308, the RAW compression unit 113 acquires the RAW image output from the sensor signal processing unit 103 corresponding to the captured still image, and performs high-efficiency encoding (RAW compression) on the RAW image. And converted into a RAW file (S312). The RAW file is stored in the buffer unit 115. The high-efficiency encoding performed by the RAW compression unit 113 is processed by a known technique such as wavelet transform or differential encoding, but may be lossy encoding or lossless encoding. Alternatively, the RAW compression of the RAW compression unit 113 may be omitted, and the RAW image may be output through without being compressed. Regardless of the presence or absence of RAW compression, the present embodiment generates a RAW file that can be restored as a high-quality file without significantly losing the image information supplied from the sensor signal processing unit 103.

  In S313, after the recording / playback unit 151 records the RAW file on the recording medium 152, the flow proceeds to S301. In S311 and S313, the recording / playback unit 151 sends the still image file and / or the RAW file to the external storage from the communication terminal 154 via the communication unit 153, and records the file in the external storage. Also good.

  This completes the description of the flow relating to the still image shooting mode processing of the present embodiment.

  Here, the structure of the still image file and the structure of the RAW file according to the present embodiment will be described. FIGS. 4A and 4B are diagrams showing examples of the structure of a still image file and a RAW file.

  The still image file 400 shown in FIG. 4A is recorded in, for example, a predetermined recording area of the recording medium 152 by the recording / playback unit 151. The still image file 400 includes a header part 401, a metadata part 402, and a compressed data part 403. The header portion 401 includes an identification code indicating that this file is a still image file format. The compressed data section 403 includes still image compressed data that has been encoded with high efficiency.

  The metadata unit 402 includes file name information 404 of a RAW file generated at the same time as the still image file. Further, development status information 405 indicating that the still image file has been simply developed by the simple development unit 111 is included. Also, evaluation values and subject information detected by the evaluation value calculation unit 105 and the recognition unit 131, and information at the time of shooting from the imaging optical unit 101 and the imaging sensor unit 102 (for example, lens type identification information, sensor type identification information) And the like. In addition, although not shown, it may further include an identification code of a recording medium in which a RAW file generated at the same time is recorded, path information of a recorded folder, and the like.

  A RAW file 410 shown in FIG. 4B is recorded in, for example, a predetermined recording area of the recording medium 152 by the recording / playback unit 151. The RAW file 410 includes a header part 411, a metadata part 412, and a compressed data part 413. The header portion 411 includes an identification code indicating that this file is in the RAW file format. The compressed data portion 413 includes RAW compressed data of still images that have been encoded with high efficiency (may be RAW image data of uncompressed still images).

  The metadata section 412 includes file name information 414 of the still image file generated simultaneously with the RAW file. Further, development status information 415 indicating that the still image file has been simply developed by the simple development unit 111 is included. Also, evaluation values and subject information detected by the evaluation value calculation unit 105 and the recognition unit 131, and information at the time of shooting from the imaging optical unit 101 and the imaging sensor unit 102 (for example, lens type identification information, sensor type identification information) And the like. Of the shooting metadata 416, the same data is used for the shooting metadata 416 in common with the shooting metadata 406. Further, although not shown in the figure, it may further include an identification code of a recording medium on which a still image file generated at the same time is recorded, path information of a recorded folder, and the like. Alternatively, the still image file itself generated at the same time may be converted into metadata and stored in the metadata unit 412.

  The structure of the various files according to the present embodiment described above is an example, and a configuration compatible with a standard such as DCF or EXIF may be used. Note that DCF means Design rule for Camera File system, and EXIF means Exchangeable Image File format.

  As described above, the imaging apparatus 100 according to the present embodiment performs live view display until a shooting instruction is performed in the still image shooting mode, and development processing for a still image file generated in response to the shooting instruction. Is performed by the simple developing unit 111. The simple developing unit 111 limits the image size after development to, for example, 2 million pixels or less, or restricts or eliminates noise removal and optical distortion correction. By doing so, for example, development processing with a performance of 2 million pixels and 60 frames per second can be realized with a small circuit scale and low power consumption. On the other hand, as described above, the imaging apparatus 100 according to the present embodiment generates a RAW file in response to a still image shooting instruction. The RAW file is a high-quality file that does not significantly impair the image information supplied from the sensor signal processing unit 103, but development processing is not required to generate this file. Therefore, it is possible to record a RAW file with low power consumption by a small circuit while increasing the number of pixels of an image and the speed of continuous shooting.

  Next, step S320, which is a transition step in step S301 in FIG. 3, will be described with reference to the flowchart in FIG. FIG. 5 shows a flowchart relating to the idle state processing of the present embodiment. The flowchart of FIG. 5 illustrates a processing procedure executed by controlling each processing block by the control unit 161. A program stored in a memory (ROM) of the control unit 161 is stored in the memory (RAM). This is realized by expanding and executing by the CPU.

  In FIG. 5, when the idle state processing is started (S500), the control unit 161 determines whether or not to follow up the development according to the user setting (S501). When developing, the process proceeds to S520.

  When the follow-up development is not performed in S501, it is determined whether the control unit 161 shifts to any of the modes 201, 202, 203, and 204 shown in FIG. 2 in accordance with a user command and mode setting (S502, S503, S504, S505). Then, the control unit 161 controls to transition to the processing flow of the selected mode (S510, S511, S512, S513). In S502 to S505, the flow transitions to S510 if it is a still image shooting mode, S511 if it is a still image playback mode, S512 if it is a movie shooting mode, and S513 if it is a movie playback mode.

  Here, “chase development” according to the present embodiment refers to a high-quality development process using a RAW file recorded in the buffer unit 115 or the recording medium 152 as a source after the photographing operation is finished. It means a process for generating a high-quality display image and a high-quality still image file. The RAW file that is the target of the chase development according to the present embodiment targets both a still image and a moving image. In the following, a still image will be described as an example.

  As described above, since the still image file generated at the time of shooting is developed by the simple development unit 111, the number of pixels is 2 million pixels or less, or the development process is partially omitted. Is of limited quality. Although it is effective as a rough confirmation of the photographing content, it may not be sufficient for the purpose of confirming details of an image or printing out. On the other hand, the RAW file that is generated at the same time as the still image has high quality that does not significantly impair the image information supplied from the sensor signal processing unit 103, but since it is data before development processing, it can be immediately displayed and printed out. Cannot be performed, and it takes time for RAW development. In addition, since RAW is not a widely spread file such as JPEG, the playback environment that can handle RAW files is also limited.

  Therefore, the chasing development of this embodiment is an effective function. In the present embodiment, when the follow-up development is executed, a RAW file that has already been recorded is read out, and development processing is performed with high image quality by the high-quality image development unit 112, and the generated high-quality still image file is recorded on the recording medium. Record to 152 etc. Then, such chasing development is executed in a state where the processing load of the apparatus is relatively small, waiting for a user operation such as between shootings or in a reproduction mode or a sleep state. The follow-up development is not limited to manual, but may be designed so that the controller 161 automatically executes it.

  With this configuration, there is no delay in development processing (reproduction output) each time a request for high-quality reproduction such as confirmation display of details or printout is made thereafter. It can be used in the same general operating environment as files.

  In the recording medium 152 or the like, a still image file and a RAW file are recorded as one set in response to one still image shooting instruction. When the follow-up development is performed manually or automatically, the control unit 161 determines whether the follow-up development has been processed or not processed for each set of images (S520). The determination method is, for example, for identifying whether or not the still image file included in the development status 405 stored in the metadata unit 402 of the still image file 400 has been processed by the simple development unit 111. There is a method of referring to the flag and determining from this flag. Alternatively, the same determination may be made with reference to the development status 415 in the RAW file 410. Alternatively, it may be determined by separately preparing a table file indicating the state of development processing for a series of still images taken.

  If the control unit 161 determines that the follow-up development has been processed, the process proceeds to S502. If there is a still image that has not been subjected to the follow-up development, the process proceeds to S521. If a RAW file corresponding to a still image that has not undergone follow-up development is buffered in the buffer unit 115 (S521), the process proceeds to S523, and if not buffered, the corresponding RAW file is read from the recording medium 152 or the like. (S522). The RAW file read from the recording medium 152 or the like is temporarily held by the buffer unit 115.

  The buffer unit 115 updates the data so that the new image captured in the still image capturing mode is preferentially retained. That is, images taken in the past are sequentially removed from the buffer. By doing so, since the image taken immediately before is always held in the buffer, S522 can be skipped and processed at high speed. Further, if the follow-up development is executed by tracing back the time taken from the immediately previous image, the processing can be completed preferentially from the image held in the buffer, so that the processing efficiency can be improved.

  In S523, the RAW decompression unit 114 decompresses the RAW file read from the buffer unit 115 or the recording medium 152, and restores the RAW image.

  The restored RAW image is developed with high image quality by the high image quality development unit 112 (S524), and is output to the display processing unit 122 and the still image compression unit 141 via the switch unit 121.

  The high image quality developing unit 112 performs debayer processing (demosaic processing), that is, color interpolation processing on the RAW image, converts it into a signal composed of luminance and color difference (or primary color), removes noise contained in each signal, and optically A so-called development process is performed such as correcting the distortion and optimizing the image. The size (number of pixels) of the developed image generated by the high image quality developing unit 112 is the full size read from the image sensor unit 102 or the size set by the user, and is limited to 2 million pixels or less. The quality is much higher than that of the simple developed image.

  Since the high-quality developing unit 112 is more accurate in each process than the simple developing unit 111, a higher-quality developed image can be obtained, but the processing load is increased. The high image quality development unit 112 of the present embodiment is configured to suppress an increase in circuit scale and power consumption by avoiding real-time development processing in parallel with shooting and enabling development processing over time.

  The image information developed by the high image quality development unit 112 is supplied to the still image compression unit 141, and the still image compression unit 141 performs high-efficiency encoding processing (still image compression) on the acquired image information ( S525), a high-quality still image file is generated. Note that the still image compression unit 141 performs compression processing using a known technique such as JPEG.

  If the recording / playback unit 151 records a high-quality still image file in the recording medium 152 or the like in S526, the flow proceeds to S502. The follow-up development process can be executed repeatedly for each image when there is a still image that has not undergone the follow-up development.

  The still image file recorded in S526 has a file configuration like the still image file 400 shown in FIG. 4A, and includes a header portion 401, a metadata portion 402, and a compressed data portion 403. The header portion 401 includes an identification code indicating that this file is a still image file format. The compressed data section 403 includes still image compressed data that has been encoded with high efficiency.

  The metadata unit 402 includes file name information 404 of the RAW file that is the source of the still image file. Further, development status information 405 indicating that the still image file has been developed with high image quality by the high image quality developing unit 112 is included. Also, the evaluation value and subject information detected by the evaluation value calculation unit 105 and the recognition unit 131 extracted from the metadata of the original RAW file, and information at the time of shooting from the imaging optical unit 101 and the imaging sensor unit 102 The shooting metadata 406 including is included.

  The recording / playback unit 151 gives the same file name to the new still image file recorded in S526 after high-quality development as the still image file by simple development recorded at the same time as the original RAW file, Overwrite recording. That is, the still image file by simple development is deleted. Then, the recording / playback unit 151 updates the development status 415 in the metadata unit 412 with information indicating that the high-quality development has been performed (or the follow-up development has been completed) for the RAW file that is the source of the follow-up development. Instead of overwriting, for example, the original file may be deleted by recording as another file in which the file name is almost the same and only a part of the file name (for example, the extension or the character at the end) is changed (for example). Of course, if the recording capacity is sufficient, the original file may be left).

  As described above, the image capturing apparatus 100 according to the present embodiment performs the follow-up development when the processing load of the apparatus is relatively low (idle state) waiting for a user operation such as between shootings or in a playback mode or a sleep state. Execute. Then, the still image file by the simple development at the time of shooting is replaced with the still image file by the high quality development using the RAW file. A moving image file by simple development at the time of shooting is also replaced with a moving image file by high quality development using a RAW file. In this way, even when a high-quality playback request is given, such as a detailed confirmation display or printout, there is no delay in development processing (reproduction output) each time, and the conventional static It can be used in a general operating environment similar to a picture file.

  Next, the operation in the still image playback mode of the imaging apparatus 100 will be described.

  FIG. 6 shows a flowchart relating to processing in the still image reproduction mode of the present embodiment. The flowchart of FIG. 6 illustrates a processing procedure executed by controlling each processing block by the control unit 161. A program stored in a memory (ROM) included in the control unit 161 is stored in the memory (RAM). This is realized by expanding and executing by the CPU.

  In FIG. 6, when the still image playback mode process is started (S600), the control unit 161 determines whether or not the processing load status of the imaging apparatus 100 is low (S601), and the idle state with a frequency according to the load status. The process proceeds to (S610), and if not, the process proceeds to S602. For example, while waiting for a user operation such as a reproduction instruction, the processing load is low, and thus the process proceeds to S610. If playback of a still image has been started in response to an operation from the user (including a state during playback), the process proceeds to S602.

  In step S602, the control unit 161 determines whether an enlarged display instruction has been received from the user for the still image to be played back. If there is an enlarged display, the process proceeds to S603, and if there is no enlarged display, the process proceeds to S620.

  FIG. 7 illustrates types of display forms including enlarged display. 7A, 7B, and 7C are diagrams illustrating an example of display processing in the still image reproduction mode of the present embodiment.

  A display example 700 in FIG. 7A is an example in which the six images 701 are reduced and displayed on the display unit 123 in a reduced size. A display example 710 in FIG. 7B is an example in which an entire image 711 is displayed on the display unit 123, and this display state is a normal display. A display example 720 in FIG. 7C is an example in which an image 721 obtained by enlarging a partial area of a certain image is enlarged and displayed on the display unit 123. For example, when confirming whether or not the focus is appropriate immediately after shooting, it is generally used to enlarge and display the details of the subject image as in the display example 720.

  When the enlarged display is performed as in the display example 720, the flow in FIG. 6 transitions from S602 to S603. When the reduced display is performed as in the display example 700, the process proceeds from S602 to S620. In the case of the display example 710, if the number of display pixels of the display unit 123 is the number of pixels of the still image file by simple development (2 million pixels or less in the above example), the display is the same size or reduced. Therefore, the process proceeds from S602 to S620.

  In S620, the recording / playback unit 151 reads a playback target still image file from the recording medium 152 or the like. In step S621, the still image expansion unit 143 decodes and expands the still image file. In step S608, the display processing unit 122 outputs the display image to the display unit 123 in the form illustrated in each drawing of FIG. .

  If the displayed image is sufficient for the number of pixels of the still image file by simple development (in the above example, 2 million pixels or less), even if the still image file is developed by the simple development unit 111, It can be displayed with sufficient image quality. Needless to say, if the still image file is developed by the high image quality development unit 112, the image quality is of course sufficient for display.

  On the other hand, in the case of enlarged display, the displayed image may not be enough for the number of pixels of the still image file by simple development (in the above example, 2 million pixels or less). In other words, display using a still image file by simple development causes a reduction in resolution.

  Therefore, in the case of enlarged display, in S603, the control unit 161 determines whether the still image file of the image to be displayed as a reproduction target is developed by the high image quality development unit 112. The determination method is, for example, for identifying whether or not the still image file included in the development status 405 stored in the metadata unit 402 of the still image file 400 has been processed by the simple development unit 111. There is a method of referring to the flag and determining from this flag. Alternatively, the same determination may be made with reference to the development status 415 in the RAW file 410. Alternatively, it may be determined by separately preparing a table file indicating the state of development processing for a series of still images taken.

  If the image is developed with high image quality in S603, it means that it is a high-quality still image file that can be displayed with sufficient image quality even if it is enlarged, and therefore the process proceeds to S620, and the recording / playback unit 151 corresponds. A high-quality still image file is read from the recording medium 152 or the like, and is reproduced and displayed. As described above, in the case of a still image file developed by the high image quality developing unit 112, it can be displayed in high image quality by the processing from S620 onward.

  If the image is not developed with high image quality in S603, it means that the file is a still image file developed by the simple development unit 111, so that the process proceeds to S604 and high image quality development (following development described above) is executed. .

  In S604, if the RAW file corresponding to the still image to be reproduced is buffered in the buffer unit 115 (S604), the process proceeds to S606. If not buffered, the corresponding RAW file is read from the recording medium 152 or the like. Read (S605). The RAW file read from the recording medium 152 or the like is temporarily held by the buffer unit 115.

  The buffer unit 115 updates the data so that the new image captured in the still image capturing mode is preferentially retained. That is, images taken in the past are sequentially removed from the buffer. By doing so, since the image taken immediately before is always held in the buffer, S605 is skipped and the display can be performed at high speed.

  In S606, the RAW decompression unit 114 decodes and decompresses the RAW file read from the buffer unit 115 or the recording medium 152, and restores the RAW image.

  The restored RAW image is developed with high image quality by the high image quality developing unit 112 (S607) and output to the display processing unit 122 via the switch unit 121. In step S <b> 608, the display processing unit 122 outputs an enlarged image as illustrated in FIG. 7C to the display unit 123.

  The high image quality developing unit 112 performs debayer processing (demosaic processing), that is, color interpolation processing on the RAW image, converts it into a signal composed of luminance and color difference (or primary color), removes noise contained in each signal, and optically A so-called development process is performed such as correcting the distortion and optimizing the image. The size (number of pixels) of the developed image generated by the high image quality developing unit 112 is the full size read from the image sensor unit 102 or the size set by the user, and is limited to 2 million pixels or less. The quality is much higher than that of the simple developed image. Therefore, a still image developed by the high image quality developing unit 112 can answer the request for enlarged display with sufficient image quality.

  When the display in S608 is stopped, the flow returns to S601. In S601, when transitioning to the idle state S610, processing is performed according to the flowchart of FIG. 5 described above.

  As described above, high-quality development after S604 in FIG. 6 is assumed to occur at a timing at which follow-up development has not yet been performed, such as immediately after shooting. In this embodiment, the chasing development of a still image is gradually ended when the processing load of the apparatus is relatively low, waiting for a user operation such as a shooting mode, a playback mode, a sleep state, and the like. The image file is naturally replaced with a still image file by high-quality development. As the replacement progresses, the number of cases where high-quality image development after S604 occurs decreases, and a high-quality image can be quickly output for enlarged display, further improving the operability.

  Further, as described above, when the RAW file is held in the buffer unit 115, S605 can be skipped, so that it can be displayed quickly. Therefore, in the case of the display examples 700 and 710 in FIG. 7, the RAW file corresponding to the image 701 and the image 711 is stored in advance from the recording medium in preparation for the enlarged display so that the RAW file is held as much as possible in the buffer unit 115. It may be read and moved to the buffer unit 115. Prior to the enlargement display instruction, when the recording / playback unit 151 reads and buffers the corresponding RAW file from the recording medium 152 or the like, when the enlargement display instruction as in the display example 720 is given, It can be displayed more quickly.

  Next, an operation in the moving image shooting mode of the imaging apparatus 100 will be described.

  FIG. 8 shows a flowchart relating to the processing of the moving image shooting mode of the present embodiment. The flowchart of FIG. 8 illustrates a processing procedure executed by controlling each processing block by the control unit 161. A program stored in a memory (ROM) of the control unit 161 is stored in the memory (RAM). This is realized by expanding and executing by the CPU.

  In FIG. 8, when the processing in the moving image shooting mode is started (S800), the control unit 161 determines whether or not the processing load status of the imaging apparatus 100 is low (S801), and the idle state ( The process proceeds to S820), and if not, the process proceeds to S802. For example, when a moving image with a large number of pixels such as a horizontal resolution equivalent to 4000 pixels (4K) or a moving image with a high frame rate such as 120 frames per second (120P) is set, the processing load is high, and thus the process proceeds to S820. Does not transition and always proceeds to S802. When the setting is made to shoot a moving image in which the number of pixels is less than a predetermined value or the frame rate is lower than a predetermined speed, for example, half the frequency between the processing of the first frame and the second frame of the moving image. Then, the process proceeds to S820.

  In step S <b> 802, the camera control unit 104 controls the operations of the imaging optical unit 101 and the imaging sensor unit 102 so that moving image shooting is performed under suitable conditions. For example, a lens included in the imaging optical unit 101 is moved in accordance with a user's zoom or focus instruction, or a reading area of the imaging sensor unit 102 is set in accordance with an instruction on the number of pixels to be captured. In addition, based on evaluation value information and subject information supplied from an evaluation value calculation unit 105 and a recognition unit 131, which will be described later, control such as focus adjustment and tracking for a specific subject is performed.

  In step S <b> 803, the sensor signal processing unit 103 performs signal processing for pixel restoration on the electrical signal converted by the imaging sensor unit 102. In other words, the sensor signal processing unit 103 interpolates using missing pixel values or subtracts a predetermined offset value for the values of missing pixels or pixels with low reliability. In the present embodiment, the image information output from the sensor signal processing unit 103 after finishing the processing of S803 is referred to as a RAW image (in particular, a RAW moving image) meaning a raw (undeveloped) moving image.

  In step S804, the simple development unit 111 develops the RAW image. At this time, the control unit 161 switches the switch unit 121 in the developing unit 110 and selects the output of the image information developed by the simple developing unit 111.

  The simple development unit 111 performs debayer processing (demosaic processing) on the RAW moving image constituting each frame of the moving image, converts it into a signal composed of luminance and color difference (or primary color), and removes noise included in each signal. Then, so-called development processing such as correcting optical distortion and optimizing the image is performed. Here, a moving image developing process (simple development) performed by the simple developing unit 111 will be described. The simple development unit 111 limits the image size after development to, for example, 2 million pixels or less of HD video, or restricts noise removal and optical distortion correction to a limited process, or omits development processing. Realizes high-speed processing and simple processing. The simple developing unit 111 performs processing after reducing the image size, or restricts a part of the development processing function, so that the imaging apparatus 100 can perform high-speed shooting of, for example, HD size moving images with a small circuit scale. Can be realized with low power consumption.

  The image information developed by the simple development unit 111 is supplied to the evaluation value calculation unit 105. The evaluation value calculation unit 105 calculates evaluation values such as a focus state and an exposure state from the luminance value and contrast value included in the image information (S805). Note that the evaluation value calculation unit 105 may acquire a RAW moving image before development processing and similarly calculate an evaluation value from the RAW moving image.

  Further, the image information developed by the simple development unit 111 is supplied to the recognition unit 131. The recognition unit 131 detects a subject (such as a face) from the image information and recognizes the subject information. For example, the presence / absence of the face in the image information, the position thereof, authentication of a specific person, and the like are performed, and the result is output as information (S806).

  Further, the image information developed by the simple developing unit 111 is supplied to the display processing unit 122. The display processing unit 122 forms a display image from the acquired image information, outputs it to the display unit 123 or an external display device, and displays it (S807). The display image by the display unit 123 is used for confirmation display for the user to appropriately frame the subject in the moving image shooting mode. Specifically, as a usage mode specific to moving image shooting, a live view display for appropriately framing a subject not only during recording of a recorded moving image (during standby) but also during recording of a moving image (during REC) Used for. The display image can also be displayed on the other display device such as an external television from the display processing unit 122 via the video output terminal 124. Further, the display processing unit 122 utilizes the evaluation value information and the subject information supplied from the evaluation value calculation unit 105 and the recognition unit 131, for example, to display a marking in a focused area on the display image, A frame can also be displayed at the position of the recognized face.

  In step S808, the control unit 161 determines whether a moving image shot in response to a recording start instruction from the user is being recorded (during REC). If it is being recorded, the process proceeds to step S810. If REC is not in S808 (that is, during standby), the process returns to S801 to repeat the shooting operation and live view display before starting the recording of the moving image.

  In S808, the moving image to be recorded from the start of recording to the end of recording among the captured moving images is compressed by the moving image compression unit 142 in units of frames (S810). Although not described with reference to the drawings, audio information input by a microphone (not shown) is acquired at the same time as shooting of a moving image. The moving image compression unit 142 also performs compression processing on audio information corresponding to the moving image.

  The moving image compression unit 142 performs high-efficiency encoding processing (moving image compression) on the acquired image information and audio information of the simply developed moving image to generate a moving image file. Note that the moving image compression unit 142 is MPEG-2, H.264, or the like. H.264, H.C. The compression processing is performed using a known moving image compression technique such as H.265.

  In S811, the recording / playback unit 151 records the moving image file on the recording medium 152.

  Further, a RAW moving image in a period corresponding to the moving image to be recorded in S808 is supplied from the sensor signal processing unit 103 to the RAW compression unit 113. The RAW compression unit 113 performs high-efficiency encoding (RAW compression) on a RAW moving image that shows the same scene as the moving image to be recorded, and converts it into a RAW file (RAW moving image file) (S812). The RAW moving image file is stored in the buffer unit 115. The high-efficiency encoding performed by the RAW compression unit 113 is processed by a known technique such as wavelet transform or differential encoding, but may be lossy encoding or lossless encoding. Alternatively, the RAW compression of the RAW compression unit 113 may be omitted, and the RAW moving image may be output through without being compressed. Regardless of the presence or absence of RAW compression, this embodiment generates a RAW moving image file that can be restored as a high-quality file that does not significantly impair image information supplied from the sensor signal processing unit 103.

  In step S813, after the recording / playback unit 151 records the RAW moving image file on the recording medium 152, the flow proceeds to step S801. Note that in S811 and S813, the recording / playback unit 151 sends the moving image file and / or the RAW moving image file from the communication terminal 154 to the external storage via the communication unit 153, and records it in the external storage. Also good.

  The above is the description of the flow relating to the processing in the moving image shooting mode of the present embodiment.

  Here, the structure of the moving image file and the structure of the RAW moving image file according to the present embodiment will be described. FIGS. 9A and 9B are diagrams illustrating configuration examples of a moving image file and a RAW moving image file.

  A moving image file 900 shown in FIG. 9A is recorded in, for example, a predetermined recording area of the recording medium 152 by the recording / playback unit 151. The moving image file 900 includes a header portion 901, a metadata portion 902, and a compressed data portion 903. The header portion 901 includes an identification code indicating that this file is in the format of a moving image file. The compressed data portion 903 includes high-efficiency encoded moving image and audio compressed data.

  The metadata part 902 includes file name information 904 of the RAW moving image file generated at the same time as the moving image file. Further, development status information 905 indicating that the moving image file has been simply developed by the simple developing unit 111 is included. Also, evaluation values and subject information detected by the evaluation value calculation unit 105 and the recognition unit 131, and information at the time of shooting from the imaging optical unit 101 and the imaging sensor unit 102 (for example, lens type identification information, sensor type identification information) Etc.). In addition, although not shown, it may further include an identification code of a recording medium in which a RAW moving image file generated at the same time is recorded, path information of a recorded folder, and the like.

  The RAW moving image file 910 shown in FIG. 9B is recorded in, for example, a predetermined recording area of the recording medium 152 by the recording / playback unit 151. The RAW moving image file 910 includes a header portion 911, a metadata portion 912, and a compressed data portion 913. The header portion 911 includes an identification code indicating that this file is in the RAW moving image file format. The compressed data section 913 includes moving image RAW compressed data that has been encoded with high efficiency (may be uncompressed moving image RAW image data).

  The metadata portion 912 includes file name information 914 of the moving image file generated simultaneously with the RAW moving image file. Further, development status information 915 indicating that the moving image file has been simply developed by the simple developing unit 111 is included. Also, evaluation values and subject information detected by the evaluation value calculation unit 105 and the recognition unit 131, and information at the time of shooting from the imaging optical unit 101 and the imaging sensor unit 102 (for example, lens type identification information, sensor type identification information) And the like. Among the shooting metadata 916, the same data is used as the shooting metadata 906 in common. Further, although not shown in the figure, it may further include an identification code of a recording medium in which a simultaneously generated moving image file is recorded, path information of a recorded folder, and the like. Alternatively, all or part of the moving image file generated at the same time (such as the first frame) may be extracted and converted into metadata, and stored in the metadata unit 912.

  The structure of the various files according to this embodiment described above is an example, and a configuration compatible with standards such as DCF, AVCHD, and MXF may be used. AVCHD means Advanced Video Codec High Definition, and MXF means Material eXchange Format.

  As described above, the imaging apparatus 100 according to the present embodiment uses the simple developing unit 111 to perform a captured image display (live view display) in the moving image shooting mode and a developing process for a moving image file generated at the time of shooting. The simple developing unit 111 restricts the image size after development to, for example, 2 million pixels or less, or restricts or eliminates noise removal and optical distortion correction, for example, HD size moving images. This development process can be realized with a small circuit scale and low power consumption. On the other hand, as described above, the imaging apparatus 100 according to the present embodiment generates a RAW moving image file corresponding to a recording period of the moving image together with the moving image file. The RAW moving image file is a high-quality file that does not significantly impair the image information supplied from the sensor signal processing unit 103, but development processing is not required to generate this file. Therefore, even if the number of image pixels is increased, such as 4K or 8K (horizontal resolution is equivalent to 8000 pixels), or the frame rate is increased, such as 120 frames per second (120P), less power is consumed by a small circuit. It is possible to record a RAW moving image file.

  Next, an operation in the moving image reproduction mode of the imaging apparatus 100 will be described.

  FIG. 10 shows a flowchart relating to processing in the moving image playback mode of the present embodiment. The flowchart of FIG. 10 illustrates a processing procedure executed by controlling each processing block by the control unit 161. A program stored in a memory (ROM) of the control unit 161 is stored in the memory (RAM). This is realized by expanding and executing by the CPU.

  In FIG. 10, when the processing of the video playback mode is started (S1000), the control unit 161 determines whether or not the processing load status of the imaging apparatus 100 is low (S1001), and the idle state ( The process proceeds to S1010), and if not, the process proceeds to S1002. For example, while waiting for a user operation such as a reproduction instruction, the processing load is low, and the process proceeds to S1010. If the playback of a moving image has been started in response to an operation from the user (including a state during playback), the process proceeds to S1002.

  In step S <b> 1002, the control unit 161 determines whether a playback pause (pause) instruction has been received from the user for the video to be played. If there is no pause instruction in S1002, the flow proceeds to S1003 in order to continue the video playback.

  In S1003, the recording / playback unit 151 reads a playback target moving image file from the recording medium 152 or the like. In step S1004, the moving image extension unit 144 decodes and expands the moving image file frame by frame. In step S1005, the display processing unit 122 outputs the display image of the reproduced moving image to the display unit 123.

  When receiving a pause instruction in S1002, the control unit 161 puts the moving image being played back and displayed in a paused state, and displays the frame at the stop position when paused as a still image, to S1020. Transition the flow. In the paused state, the image is displayed in a static state, so that it becomes easier to see the detailed image quality than when moving. Further, it is considered that an enlarged display instruction is easily received during the pause. Therefore, in order to provide a display of a higher quality image than the moving image file that is simply developed, in step S1020, the RAW image corresponding to the frame that is being paused and displayed in the RAW moving image file corresponding to the moving image file being played back. The recording / reproducing unit 151 reproduces the frame. At this time, if the RAW moving image file to be reproduced is buffered in the buffer unit 115, the RAW moving image file is read from the buffer unit 115, and if not buffered, the RAW moving image file is read from the recording medium 152 or the like. The RAW moving image file read from the recording medium 152 or the like is temporarily held by the buffer unit 115.

  In S1021, the RAW decompression unit 114 decodes and decompresses the RAW moving image file read from the buffer unit 115 or the recording medium 152, and restores the RAW moving image. The restored RAW moving image is developed with high image quality by the high image quality developing unit 112 (S1022).

  The imaging apparatus 100 can also capture a high-quality still image that is developed from the moving image RAW file with high image quality and that corresponds to the frame that is being paused and displayed as a new still image file. In step S1023, the control unit 161 determines whether a user instruction for capturing a display image at a still position as a still image has been received. If no still image capture instruction is accepted in S1023, the still image developed from the moving image RAW file with high image quality is supplied to the display processing unit 122. In S1005, the display processing unit 122 is developed with high image quality. The display image is output to the display unit 123. By this processing, the image being paused and displayed by the moving image file is replaced with the display image of the still image developed from the RAW image with high image quality.

  If a still image capture instruction is received in S1023, the image information developed by the high image quality development unit 112 is supplied to the still image compression unit 141 in S1022. The still image compression unit 141 performs high-efficiency encoding processing (still image compression) on the image information acquired by capture (S1024), and generates a high-quality still image file. Note that the still image compression unit 141 performs compression processing using a known technique such as JPEG.

  If the recording / playback unit 151 records a high-quality still image file on the recording medium 152 or the like in S1025, the flow proceeds to S1005. The still image developed with high image quality from the RAW moving image file is supplied to the display processing unit 122, and the display processing unit 122 outputs the display image of the still image developed with high image quality to the display unit 123 in S1005. By this processing, the image being paused and displayed by the moving image file is replaced with the display image of the still image developed from the RAW image with high image quality.

  The high-quality still image file generated by the still image compression unit 141 in S1024 constitutes the configuration of the still image file 400 in FIG. The metadata unit 402 stores the file name of the RAW moving image file that is the source of capture as the file name information 404 of the RAW file. In addition, time information of a frame captured as a still image is stored as shooting metadata 406, and the corresponding frame position of the RAW moving image file can be pointed out. Alternatively, the corresponding frame of the RAW moving image file may be extracted as a still image, and a new RAW file 410 for a pair of still images may be created at this time. Regarding the generation of the RAW file of the still image, the still image file and the RAW file are configured by the method described in S310 to S313 of the above-described still image shooting mode.

  Note that the display in S1005 is performed for each frame, and the flow returns to S1001 to display the next frame during playback of the moving image. In S1001, when transitioning to the idle state S1010, processing is performed according to the flowchart of FIG.

  As described above, the imaging apparatus 100 according to the present embodiment can easily reproduce a moving image without delay using a moving image file recorded at the time of shooting, and develops a moving image from a RAW file to a high quality in a paused state. The displayed still image can be replaced and displayed. Furthermore, this high-quality still image can be easily captured as a still image file.

  Further, it is assumed that the high-quality development after S1020 in FIG. 10 occurs at the timing when the follow-up development is not yet executed, such as immediately after shooting. In this embodiment, the chasing development of the video is gradually terminated when the processing load of the apparatus is relatively low, waiting for a user operation such as between shootings, playback mode, sleep mode, etc. The file is naturally replaced with a video file by high-quality development. As the replacement progresses, the number of cases where high-quality development after S1020 occurs decreases, and a high-quality image can be always output promptly, thereby further improving the operability.

  The above is the description of the present embodiment, but the present invention is not limited to the above-described embodiment within the scope of the technical idea of the present invention, and should be adapted as appropriate by changing the target circuit form. It is.

(Other embodiments)
Furthermore, each process shown in the flowchart diagram in the above-described embodiment of the present invention reads out a computer program for realizing the function of each process from the memory of the system in which the image encoding apparatus of the present invention operates, and the CPU of the system Can also be realized. In this case, the program stored in the memory constitutes the present invention.

  The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

  Moreover, you may implement | achieve the function of all or one part of each process shown to the flowchart figure with exclusive hardware. In addition, a program for realizing the function of each process shown in the flowchart diagram is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed. May be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices.

  Here, the “computer-readable recording medium” includes a portable medium such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and a storage device such as a hard disk built in the computer system. Furthermore, a program that holds a program for a certain period of time, such as a volatile memory (RAM) inside a computer system that becomes a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line Shall also be included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.

  A program product such as a computer-readable recording medium in which the above program is recorded can also be applied as an embodiment of the present invention. The above program, recording medium, transmission medium, and program product are included in the scope of the present invention.

  Although the present invention has been described with reference to the preferred embodiments, the present invention is not limited to the above-described embodiments, and various modifications can be made within the scope shown in the claims.

DESCRIPTION OF SYMBOLS 101 Imaging optical part 102 Imaging sensor part 103 Sensor signal processing part 104 Camera control part 105 Evaluation value calculation part 110 Developing part 111 Simple developing part 112 High image quality developing part 113 RAW compression part 114 RAW expansion part 115 Buffer part 121 Switch part 122 Display Processing unit 123 Display unit 124 Video output terminal 131 Recognition unit 141 Still image compression unit 142 Video compression unit 143 Still image expansion unit 144 Video expansion unit 151 Recording / playback unit 152 Recording medium 153 Communication unit 154 Communication terminal

Claims (12)

Imaging means for generating a RAW image image by capturing a subject image,
First developing means for performing development processing on a RAW image ;
A second developing unit for performing a developing process for obtaining a higher quality image than the first developing unit for a RAW image ;
In response to a shooting instruction, control is performed to record an image obtained by developing the RAW image generated by the imaging unit by the first developing unit and the RAW image on a recording medium. 1 recording control means;
After shooting, the RAW image recorded on the recording medium is developed by the second developing process, and the image that has been developed by the second developing process is recorded on the recording medium. Second recording control means for controlling;
Display processing means for outputting a display image based on the image recorded on the recording medium ;
Operation means for instructing enlargement of a partial area of the display image;
If enlargement of a part of the displayed image by the previous SL operating means when instructed, the image in which the developing process by the second developing means has been performed corresponding to the displayed image is recorded on the recording medium Is controlled to output an enlarged display image in which a partial area is enlarged based on the image developed by the second developing means, and the first developing means corresponding to the display image An image that has undergone development processing is recorded on the recording medium, but an image that has undergone development processing by the second development unit but has not been developed by the second development unit has been recorded on the recording medium. If not recorded on the medium, the RAW image corresponding to the display image is developed by the second developing means to generate an image, and a partial area is generated based on the generated image. Expanded And control means for controlling to output the shown image,
An image processing apparatus comprising: 2. The image subjected to development processing is recorded on the recording medium as an image file together with identification information indicating whether or not the image has undergone development processing by the first development unit. An image processing apparatus according to 1. The control means refers to the identification information, and the image file recorded on the recording medium is an image that has been developed by the first development means, or the second development The image processing apparatus according to claim 2, wherein the image processing apparatus determines whether the image has been developed by the means. The second recording control unit is configured to change the image developed by the second development process to a new image file different from the image file of the image developed by the first development process. 4. The image processing apparatus according to claim 1, wherein the image processing device is controlled to record on the recording medium. The second recording control unit records an image file of an image that has been developed by the second development process on the recording medium, and also stores an image that has been developed by the first development process. The image processing apparatus according to claim 1, wherein the image processing apparatus is controlled so as to be deleted from the recording medium. The first recording control means records information indicating that development processing has been performed on the RAW image by the first developing means as second identification information together with the RAW image,
A second recording control unit configured to record the second identification information in the second recording information when the RAW image is recorded on the recording medium by an image that has been developed by the second developing unit; The image processing apparatus according to claim 1, wherein the image processing apparatus updates information indicating that the development processing has been performed by the development unit. 7. The image subjected to the development process is recorded on the recording medium as an image file together with identification information indicating whether the development process has been performed by the first development unit. Image processing apparatus. The second recording control means performs a developing process on the RAW image recorded on the recording medium by the second developing process when the processing load on the image processing apparatus is small, and records the image on the recording medium. The image processing apparatus according to claim 1, wherein the image processing apparatus is controlled to perform the control. The image processing apparatus according to claim 1, wherein the first recording control unit performs control so that the RAW image is compressed and recorded on the recording medium. An imaging step of capturing a subject image to generate a RAW image;
A first development step for performing a simple development process on the RAW image generated in the imaging step;
A first recording step of recording, on a recording medium, the simple developed image that has undergone the simple development processing in the first developing step and the RAW image in response to a shooting instruction;
A second development step of performing a high image quality development process for obtaining a higher quality image than the simple development process on the RAW image recorded on the recording medium after shooting;
A second recording step of recording the high-quality developed image on which the high-quality development processing has been performed on the recording medium;
A display processing step of outputting a display image based on the image recorded on the recording medium;
When enlargement of a partial area of the display image is instructed, if a high-quality developed image corresponding to the display image is recorded on the recording medium, a part of the display image is based on the high-quality developed image. Control is performed to output an enlarged display image with an enlarged area, and a simple developed image corresponding to the display image is recorded on the recording medium, but the high-quality development processing in the second development step is completed. If the high-quality developed image is not recorded on the recording medium, the high-quality developed image is generated by performing the high-quality development processing on the RAW image corresponding to the display image. And a control step of controlling to output an enlarged display image in which a partial area is enlarged based on the high-quality developed image. The program for functioning a computer as each means of the image processing apparatus of any one of Claim 1 thru | or 9. The program for making a computer perform each process of the image processing method of Claim 10 .

Images