JP6741532B2 - Imaging device and recording method - Google Patents

Description

The present invention relates to an imaging device and a recording method.

Conventionally, in an image pickup apparatus such as a digital camera, a raw image (RAW image) obtained by an image pickup sensor unit is subjected to a development process (a process of generating image data in a format that can be recognized as an image). Then, it is generally performed that the image data subjected to the development processing is compression-encoded and recorded on a recording medium.

On the other hand, there is also an imaging device capable of recording a RAW image (Patent Document 1). The RAW image has an enormous amount of data required for recording as compared with the JPEG format and the like, but has an advantage that deterioration of the image quality can be suppressed to a minimum.

JP, 2011-244423, A

In order to process a large amount of data such as a RAW image at high speed, hardware with high processing capability is required. However, such hardware having high processing capability has a large circuit scale and power consumption, and a large-capacity memory is required, which may increase the cost.

Further, in recent years, the number of pixels of an image that can be picked up by the image pickup sensor unit has increased, and accordingly, the processing load necessary for the development processing of the RAW image has also increased. Therefore, in order to realize real-time development processing in parallel with shooting while maintaining high shooting functions such as continuous shooting speed, hardware with high processing capability is required.

On the other hand, it may be considered that only the RAW image is recorded at the time of shooting, and the developing process is performed when the image needs to be displayed or printed thereafter. However, the data format of the RAW image is unique to the manufacturer of the imaging device. Therefore, the RAW image recorded by the image pickup device of a certain manufacturer cannot be properly developed in the device of another manufacturer, which may reduce the convenience.

In view of the above problems, the present invention provides an imaging device and a recording method capable of efficiently recording a required image while suppressing an increase in circuit scale and power consumption and maintaining a high image capturing function. To aim.

In the present invention, the image pickup means, the generation means for generating the RAW image data using the image data obtained by the image pickup means, and the RAW image data generated by the generation means according to the recording instruction are recorded. Recording means for recording on the medium, and whether or not to set the RAW image data recorded on the recording medium by the recording means in response to the recording instruction as a storage target to be stored in the recording medium is instructed by the user. Setting means for controlling the developing means, developing means for developing the RAW image data to generate developed image data, and controlling the developing means for developing the RAW image data recorded on the recording medium. And a control means for controlling the recording means so as to record the developed image data, which is generated by the developing means by developing the RAW image data, on the recording medium. When the first RAW image data which is not set as a storage target to be stored in the recording medium is developed by the developing device, the first RAW image data is automatically deleted from the recording medium, When the second RAW image data set as a storage target to be stored in the medium is developed by the developing device, control is performed so that the second RAW image data is not deleted from the recording medium.

According to the present invention, it is possible to efficiently record a necessary image while suppressing an increase in circuit scale and power consumption and maintaining a high image capturing function.

FIG. 3 is a block diagram showing a configuration of an image pickup apparatus according to an embodiment of the present invention. The flowchart which shows the process in a photography mode. 3 is a flowchart showing processing in the idle state of FIG. 2. The figure which illustrates the RAW image data imaged by the imaging device.

Hereinafter, embodiments of the present invention will be described in detail. The embodiment described below is an example for realizing the present invention, and should be appropriately modified or changed according to the configuration of the device to which the present invention is applied and various conditions. However, the present invention is not limited to the embodiment. In addition, a part of each embodiment described below may be appropriately combined and configured.

Hereinafter, a digital camera capable of shooting and recording RAW image data of a still image or a moving image will be described as the image pickup apparatus of the present embodiment. In the present embodiment, a digital camera is illustrated as the image pickup apparatus 100, but another information processing apparatus may be used. Other information processing devices include a camera-equipped mobile phone, a type of smartphone (including eyeglass-type terminals and wristwatch-type terminals), a tablet terminal, a personal computer (PC) with a camera, and the like.

<Device configuration>
First, with reference to FIG. 1, a configuration and a function of the image pickup apparatus 100 according to the present embodiment will be described.

The image pickup apparatus 100 of the present embodiment generates RAW image data from the image signal obtained by the image pickup sensor unit 102. Then, in the shooting standby state (recording standby state), the RAW image data is simply developed and displayed on the display unit 123. When the user gives an instruction to start recording, the RAW image data is recorded in the recording medium 152, but the developed image data is not recorded in the recording medium 152. The user can set image data to be finally recorded on the recording medium 152 and instruct recording. The user finally saves the RAW image data and does not save the developed image data, saves both the RAW image data and the developed image data, or saves the developed image data and saves the RAW image data. It is possible to specify either not to save.

Then, in the imaging apparatus 100, no matter what setting is made by the user, only the RAW image data is recorded on the recording medium 152 without performing the developing process on the RAW image data at the time of shooting according to the recording instruction from the user. To record. Then, the imaging apparatus 100 records the RAW image data on the recording medium 152, and then confirms whether the RAW image data is a storage target. When only the developed image data is the storage target, the imaging device 100 reads the RAW image data from the recording medium 152, performs the development process, and generates the developed image data. Then, after recording the developed image data on the recording medium 152, the RAW image data of the developing source is automatically deleted from the recording medium 152.

With this configuration, a large amount of image data can be recorded on the recording medium 152 at high speed without using hardware with high processing capability for developing RAW image data at high speed at the time of shooting according to a recording instruction from the user. can do. If only the developed image data is to be stored, the RAW image data recorded on the recording medium 152 is read out and developed, and the developed image data is recorded on the recording medium 152. Automatically delete the original RAW image data. Therefore, the RAW image data that the user does not need is not stored in the recording medium 152, and the memory capacity can be effectively used.

The image capturing apparatus 100 according to the present embodiment has a function of not only recording the image data obtained by capturing an image of a subject in the recording medium 152 but also reading the image data from the recording medium, developing the image data, and displaying the image data. Further, the image pickup apparatus 100 has a function of transmitting and receiving image data to and from an external storage, a server (cloud), or the like. Therefore, the imaging device 100 of this embodiment can also be expressed as an image processing device, a recording device, a reproducing device, a recording/reproducing device, a communication device, or the like.

FIG. 1 is a block diagram illustrating the configuration of the image pickup apparatus 100 according to the present embodiment. These functional blocks may be realized by dedicated hardware such as ASIC, or may be realized by software when a general-purpose processor such as MPU executes a program.

In FIG. 1, the control unit 161 controls the overall processing of the imaging device 100. The control unit 161 includes a programmable processor such as a CPU or MPU, and a non-volatile memory that stores a control program executed by the programmable processor. The control unit 161 executes control according to an instruction from the control program stored in the nonvolatile memory. Although only a part is shown in FIG. 1 to avoid complication, a signal line for control and communication is connected from the control unit 161 to each functional block. The operation unit 162 includes an input device such as a key, a button, or a touch panel used by the user to give an instruction to the imaging apparatus 100. An operation signal from the operation unit 162 is detected by the control unit 161, and the control unit 161 controls other functional blocks and the like so that an operation according to the detected operation is executed. The display unit 123 displays an image captured or reproduced by the image capturing apparatus 100, a menu screen, various information, and the like, which are provided through the display processing unit 122. The display unit 123 includes, for example, a liquid crystal display (LCD) or the like.

When the power of the image pickup apparatus 100 is turned on by the operation unit 162, the control unit 161 puts the image pickup apparatus 100 in a shooting standby state. In the shooting standby state, a subject image to be imaged is formed on the image sensor unit 102 via the image pickup optical unit 101. Then, in the live view mode, the image captured by the image sensor unit 102 is displayed on the display unit 123 as a live view image. In response to the recording start instruction from the operation unit 162, the control unit 161 starts the photographing process and the recording process. The operations of the imaging optical unit 101 and the imaging sensor unit 102 are based on the evaluation value calculation result of the diaphragm, focus, camera shake, etc. acquired by the evaluation value calculation unit 105 and the subject information from the recognition unit 131. Controlled by.

The image sensor unit 102 converts light transmitted through red, green, and blue (RGB) color filters arranged for each pixel into an electric signal. For example, it may be a CCD image sensor or a CMOS image sensor. The pixel array of the image handled by the imaging apparatus 100 is, for example, red (R), green (G), and blue (B) arranged in a mosaic pattern for each pixel, and one red pixel and one blue pixel for every 2×2 4 pixels. , Green has a structure in which two pixels are regularly arranged in one set. An array of such color filters is generally called a primary color Bayer array. The image sensor unit 102 of the present embodiment has a pixel number of 4K (horizontal 3840 pixels×vertical 2160 pixels), but the number of pixels may be more than this. Further, the image sensor unit 102 of the present embodiment has the ability to output image data of 4K per frame at 60 frames per second (fps). The colors and arrangements of the color filter are not limited to the primary color Bayer arrangement, and any other arrangement can be adopted.

The image signal obtained by the imaging sensor unit 102 is subjected to pixel restoration processing by the sensor signal processing unit 103. The restoration process includes a process of interpolating the values of missing pixels and unreliable pixels in the image sensor unit 102 by using peripheral pixel values, and subtracting a predetermined offset value from the pixel to be restored. .. In the present embodiment, the image data output from the sensor signal processing unit 103 is referred to as RAW image data or RAW image meaning an undeveloped image.

The RAW image data output from the sensor signal processing unit 103 is developed by the developing unit 110. The developing unit 110 has a plurality of different developing processing units, each of which has different processing accuracy and processing load. The developing unit 110 of this embodiment has two developing units, a simple developing unit 111 that performs a first developing process and a high-quality developing unit 112 that performs a second developing process, and a switch that selects the output thereof. It is configured to include the unit 121. The simple developing unit 111 and the high-quality developing unit 112 both perform debayer processing (also called demosaicing processing or color interpolation processing) on a RAW image, white balance adjustment, RGB (primary color)→YUV (luminance and color difference) conversion, noise reduction. , Perform development processing such as correction of optical distortion. It should be noted that these are examples of the processes included in the development process, and do not mean that all of them are essential to the development process. Other processing may be included, or a part thereof may not be included.

The high image quality developing unit 112 performs each processing with higher accuracy than the simple developing unit 111. Since it is highly accurate, a developed image with higher image quality than that of the simple developing unit 111 can be obtained, but on the other hand, the processing load is large. On the other hand, since the simple developing unit 111 has lower processing accuracy than the high image quality developing unit 112, the obtained image quality is lower than that of the high image quality developing unit 112, but the developing process can be performed at high speed during shooting. Therefore, in the present embodiment, the high image quality developing unit 112 is used not when performing the real-time development in parallel with the shooting but when performing the development processing over a period of time during the idle period after the shooting. As described above, instead of performing the high-quality image development processing at the time of shooting according to the recording instruction of the user, the high-quality image development processing is performed over time after the RAW image is once recorded on the recording medium 153, The increase (peak) in circuit scale and power consumption can be suppressed to a low level. Moreover, the processing load of the simple developing unit 111 is small. Therefore, the simple developing unit 111 is used to generate an image for live view display during real-time development in parallel with shooting in the shooting standby state. The switch unit 121 is switched by the control unit 161 according to the control according to the operation content instructed by the user via the operation unit 162 and the operation mode being executed.

In the present embodiment, the simple developing unit 111 and the high image quality developing unit 112 are independently provided in the developing unit 110, but the mode in which one developing unit performs the simple developing process and the high image quality developing process are described. It may be configured to have a plurality of operation modes including a mode for performing processing. Then, by switching these operation modes, it is possible to perform a simple development process and a high image quality development process. Further, it is only necessary that the plurality of developing units have different processing loads, and it is not essential to reduce both the size of the image to be processed and the processing accuracy. For example, a developing unit that has the same number of pixels to be processed but different processing accuracy, or a developing unit that has the same processing accuracy but different number of pixels to process may be included. However, at least one developing unit that can complete the development processing of the RAW image of one screen output from the image sensor unit 102 by the start of capturing the RAW image of the next one screen is included.

Although FIG. 1 illustrates that the RAW image data from the sensor signal processing unit 103 is input to the simple developing unit 111 and the high image quality developing unit 112 in the developing unit 110, this is the same RAW image. It does not mean that the development processing is performed on both data in both development sections. From the viewpoint of processing load, basically only one developing unit executes the developing process. Therefore, the switch unit 121 may be provided to switch the developing unit to which the RAW image data is input.

The image data developed by the developing unit 110 (developed image data) is displayed on the display unit 123 after being subjected to predetermined processing by the display processing unit 122. Further, the image data subjected to the development processing may be output to a display device connected to the outside through the video output terminal 124. The video output terminal 124 includes a general-purpose interface such as HDMI (registered trademark) or SDI.

The image data developed by the developing unit 110 is also supplied to the evaluation value calculating unit 105. The evaluation value calculation unit 105 calculates evaluation values such as a focus state and an exposure state from the developed image data. These evaluation values are used for automatic focus detection and automatic exposure control by the control unit 161, for example.

Further, the image data developed by the developing unit 110 is also supplied to the recognition unit 131. The recognition unit 131 has a function of detecting and recognizing subject information in image data. The recognition unit 131 detects, for example, the face of a person included in the image displayed based on the image data, and outputs the information indicating the position or size of the face when the face is detected. The recognition unit 131 may further authenticate a specific person based on feature information such as a face.

Furthermore, the image data developed by the developing unit 110 is also supplied to the still image compression unit 141 and the moving image compression unit 142. It should be noted that in the present embodiment, as described above, the developing process by the high image quality developing unit 112 is not performed when the image is captured according to the recording instruction from the user. Therefore, as described above, the compression process by the still image compression unit 141 and the moving image compression unit 142 is not performed when the image is captured in accordance with the recording instruction from the user. When generating still image data for recording from the developed image data, the still image compression unit 141 is used. The still image compression unit 141 compression-encodes the developed still image data according to a known encoding method such as JPEG to generate recording still image data. When generating moving image data for recording from the developed image data, the moving image compression unit 142 is used. The moving picture compression unit 142 uses the H.264 format. H.264, H.264. According to a known encoding method such as H.265, the developed moving image data is compression-encoded to generate recording moving image data. The still image compression unit 141 and the moving image compression unit 142 perform high-efficiency encoding (compression encoding) on the respective target image data, generate image data in which the information amount is compressed, and send the image data to the recording/reproducing unit 151. ..

The RAW compression unit 113 applies high-efficiency encoding processing such as wavelet conversion or differential encoding to the RAW image data output from the sensor signal processing unit 103 to generate compressed RAW image data with reduced information amount. To do. The RAW compression unit 113 stores the compressed RAW image data in the buffer unit 115. The buffer unit 115 may be any storage device such as a memory or a hard disk. Further, at the time of shooting according to the recording instruction from the user, the compressed RAW image data is temporarily stored in the buffer unit 115, then read from the buffer unit 115 at an appropriate timing and recorded in the recording medium 152. .. That is, the buffer unit 115 is not used as the final storage destination of the RAW image data or the developed image data. The buffer unit 115 is used only as a temporary storage unit for recording the RAW image data and the developed image data in the recording medium 152 which is the final storage destination of the image data.

The recording/reproducing unit 151 records the still image data from the still image compression unit 141 and the moving image data from the moving image compression unit 142, which are temporarily recorded in the buffer unit 115, on the recording medium 152. Further, the recording/playback unit 151 records the RAW image data of the still image and/or the moving image read from the buffer unit 115 in the recording medium 152 as a file separate from the compressed and encoded still image data and the moving image data. Further, the recording/reproducing unit 151 manages the data recorded in the recording medium 152 as a file according to a known file system such as FAT. The recording medium 152 is, for example, a built-in large-capacity memory or hard disk, a removable memory card, or the like. The recording/reproducing unit 151 can also read a still image file, a moving image file, and a RAW file (still image, moving image) from the recording medium 152. Further, the recording/reproducing unit 151 can write and read various data and files to and from an external storage or server via the communication unit 153. The communication unit 153 provides the imaging device 100 with access to a computer network or an external device through wireless communication or wired communication through the communication terminal 154.

The recording/reproducing unit 151 also acquires and reproduces a desired file from the recording medium 152 or externally via the communication unit 153. If the file to be played back is a RAW file, the recording/playback unit 151 stores the RAW image data of the acquired RAW file in the buffer unit 115. If the file to be played back is a still image file, the recording/playback unit 151 supplies the still image data of the acquired still image file to the still image decompression unit 143. If the file to be played back is a moving image file, the recording/playback unit 151 supplies the moving image data of the obtained moving image file to the moving image decompression unit 144.

The RAW expansion unit 114 expands the data amount by reading the compressed RAW image data stored in the buffer unit 115 and decoding it. The RAW image data decoded by the RAW decompression unit 114 is supplied to the simple developing unit 111 and the high image quality developing unit 112 of the developing unit 110.

The recording/reproducing unit 151 of the present embodiment can delete the original RAW file from the external storage or server via the recording medium 152 or the communication unit 153 after the developing process is completed in the developing unit 110.

The still image expansion unit 143 decodes the input still image data and supplies it to the display processing unit 122 as a reproduced image of the still image. The moving image expansion unit 144 decodes the input moving image data and supplies it to the display processing unit 122 as a reproduced image of the moving image. The still image or moving image processed by the display processing unit 122 is displayed on the display unit 123.

<Explanation of operation in shooting mode>
Next, the operation of the image pickup apparatus 100 according to the present embodiment in the shooting mode will be described with reference to the flowchart in FIG.

Note that the flowchart of FIG. 2 illustrates a procedure of processing realized by the control unit 161 controlling each functional block illustrated in FIG. 1 unless otherwise specified. Specifically, it is a process realized by expanding a program stored in the nonvolatile memory of the control unit 161 into a memory (RAM) and executing it by the CPU.

In FIG. 2, when the image pickup apparatus 100 is set to the shooting mode or switched to the shooting mode when the power is turned on by the operation unit 162, the shooting standby state is set.

In S201, the control unit 161 determines whether or not the processing load of the imaging device 100 is low. If the processing load is high, the control unit 161 makes a transition to the idle state of S210 at a frequency according to the high load status, and the processing load is not high. In this case, the process proceeds to S202. The processing load can be determined according to, for example, the operating rate of the CPU included in the control unit 161, a predetermined high-load operation, such as whether or not a high-speed continuous shooting operation is being performed. Not limited.

In S202, the camera control unit 104 controls the operations of the imaging optical unit 101 and the imaging sensor unit 102 so that the imaging process is executed under the optimal imaging conditions. For example, the control unit 161 moves the zoom lens or the focus lens included in the imaging optical unit 101 according to the user's zoom instruction or the shooting preparation instruction through the operation unit 162, or the imaging sensor unit 102 of the imaging sensor unit 102 according to the instruction of the number of imaging pixels. Set the read area. The control unit 161 also executes control such as focusing and tracking on a specific subject based on the evaluation value information from the evaluation value calculation unit 105 and the subject information from the recognition unit 131. In S202, control is performed under shooting conditions according to a predetermined frame period.

The user can specify the type of image to be finally stored in the recording medium 152 after the development processing by the developing unit 110 via the operation unit 162 or the like in the shooting standby state in the shooting mode. In this embodiment, for example, the user can operate the operation unit 162 to set one of a plurality of recording modes (save modes) according to the type of image to be saved. The plurality of recording modes include a mode in which a RAW image is recorded and a developed image is not recorded, a mode in which both a RAW image and a developed image are recorded, and a mode in which a developed image is recorded and a RAW image is not recorded. Still images and/or moving images can be recorded for both the RAW image and the developed image. Further, when the recording mode of the developed image is set, the user can set the screen size (number of pixels) and the image quality (compression rate) of the developed image to be recorded.

In S203, the sensor signal processing unit 103 performs signal processing for pixel restoration on the image signal obtained by the image sensor unit 102. That is, the sensor signal processing unit 103 uses the peripheral pixel value to interpolate or subtracts a predetermined offset value from the electric signal corresponding to the missing pixel of the image sensor or the electric signal of the pixel with low reliability.

In step S204, the control unit 161 determines whether a recording start instruction is input from the user through the operation unit 162. If the recording start instruction is input, the process proceeds to step S205. If not, the processing from step S201 is performed. Repeat the process.

In step S205, the control unit 161 responds to the recording start instruction, and the camera control unit 104 controls the imaging optical unit 101 and the imaging sensor unit 102 to start the recording process. Further, the RAW compression unit 113 acquires the RAW image data from the sensor signal processing unit 103 and applies a high efficiency encoding (RAW compression) process for reducing the information amount to the RAW image data to compress the RAW image data. Generate data. The compressed RAW image data is temporarily stored in the buffer unit 115. In the case of still image shooting, the RAW image data for one screen obtained by the image sensor 102 is subjected to high-efficiency coding processing in the RAW compression unit 113 and converted into compressed RAW image data. Further, in the case of moving image recording, the RAW image data of a plurality of frames output from the sensor signal processing unit 103 according to a predetermined frame cycle is sequentially subjected to high efficiency encoding processing in the RAW compression unit 113 and compressed. Raw image data is generated. Note that either lossless encoding or lossy encoding can be used to encode the RAW image data. Further, when recording the RAW image data, the RAW image data may be recorded in the uncompressed state without compressing the RAW image data by the RAW compression unit 113. In the present embodiment, RAW image data is recorded as a high-quality image that does not significantly impair the image data supplied from the sensor signal processing unit 103, regardless of whether or not the recorded RAW image data is compressed.

In S206, the control unit 161 causes the recording/playback unit 151 to record the RAW image data generated by the RAW compression unit 113 in the recording medium 152, and then returns to the processing of S201 to repeat the shooting preparation operation. Note that in S206, the recording/reproducing unit 151 may transmit the RAW image data from the communication terminal 154 to the external storage or server via the communication unit 153 and record the RAW image data in the external storage or server. In this case, the external storage or server becomes the final recording medium of the RAW image data or the developed image data. Here, in the case of a still image, one screen of RAW image data is recorded in response to a recording start instruction. Further, in the case of a moving image, the recording of the RAW image data is continued until the recording start instruction is given by the user until the recording stop instruction is given, and the recording of the RAW image data is stopped in response to the recording stop instruction. Further, as described above, in the present embodiment, when the image is recorded according to the recording instruction from the user, the developing unit 110 does not perform the developing process on the RAW image data, and only the RAW image data is recorded on the recording medium 152. Recorded in.

Further, the control unit 161 records the setting information of the recording mode designated by the user as the additional information of the RAW image data recorded on the recording medium 152, in addition to the shooting date and time of the image and the like. The plurality of modes include a mode of recording only a RAW image, a mode of recording a RAW image and a developed image (compressed image), and a mode of recording only a developed image. The control unit 161 records information for identifying one of these modes as additional information of the RAW image data recorded on the recording medium 152. This identification information is also information for identifying whether or not the RAW image data recorded on the recording medium is set as a target to be stored in the recording medium. When the user sets a mode for recording a developed image (compressed image), information such as the size of the developed image and image quality (compression rate) set by the user is also added as additional information. The data is recorded on the recording medium 152. The setting information regarding the recording mode may be added as the metadata stored in the RAW file including the RAW image data. Further, it may be added as an attribute of a file system adapted to the recording medium 152 or an external storage or a server, for example, a file name, an archive attribute, a hidden attribute, or the like.

As described above, the imaging apparatus 100 according to the present embodiment records the RAW image data on the recording medium 152 in response to the recording start instruction from the user in the shooting mode. The RAW image data is a high-quality image that does not significantly deteriorate the image quality of the image data supplied from the sensor signal processing unit 103, but no development processing is required to generate this RAW image data. Therefore, it is possible to record RAW image data with a small-scale and low power consumption circuit even if the number of image pickup pixels or the speed of continuous shooting is increased or the number of pixels per frame or frame rate is increased.

<Explanation of operation in idle state>
Next, the operation in the idle state in S210 of FIG. 2 will be described using the flowchart of FIG.

In the idle state, the image capturing apparatus 100 automatically reads out the RAW image data that needs to be developed from the recording medium 152 or an external storage or the like, and performs the development process. Furthermore, the imaging apparatus 100 automatically compression-codes the developed image data that has undergone the development process, and records the compressed developed image data in the recording medium 152. Here, of the RAW image data recorded on the recording medium 152, the RAW image data to be automatically processed in the idle state is recorded in a mode in which only the developed image is stored. This is RAW image data. Further, the RAW image data recorded in the mode in which both the RAW image and the developed image are saved are also the RAW image data to be automatically processed in the idle state.

The idle state is a state in which it is determined that the processing load of the image pickup apparatus 100 is small, such as a shooting standby state, a standby state during image reproduction, and a sleep state. Note that the development process may be started not only automatically but also according to an instruction from the user. Further, in the present embodiment, in the shooting standby state, the processes of S202 and S203 shown in FIG. 2 are repeatedly executed even in the idle state, and the live view image is displayed on the display unit 123.

There is no particular limitation on the method of determining whether or not the image pickup apparatus 100 is in the idle state, and any method of determination including a general method of measuring the processing load can be used. For example, depending on whether the operating rate of the CPU included in the control unit 161 is less than a predetermined threshold value or whether a predetermined high-load operation such as high-speed continuous shooting operation or recording/reproducing processing is being performed. Can be determined. Alternatively, if an operation mode that basically causes only low-load processing is selected, it may be unconditionally regarded as the idle state.

The flowchart of FIG. 3 illustrates a processing procedure in which each functional block is controlled and executed by the control unit 161, and a program stored in the memory (ROM) of the control unit 161 is stored in the memory (RAM). It is realized by expanding and executing by the CPU.

When the processing in the idle state is started in FIG. 3, the control unit 161 determines whether or not the RAW image data that needs to be subjected to the development processing is recorded in the recording medium 152. The control unit 161 confirms the setting information and the metadata of the recording mode added to each RAW image data recorded on the recording medium 152. Then, the control unit 161 needs to perform the developing process when the recording mode is set to the mode of recording only the developed image or the mode of recording the RAW image and the developed image (compressed image). It is determined to be RAW image data.

In S301, if it is determined that the RAW image data to be developed is recorded in the recording medium 152, the process proceeds to S302, and if it is determined that it is not recorded, the process proceeds to S307.

In S302, the control unit 161 determines the order of performing the development processing on the RAW image data that needs to be subjected to the development processing, and advances the processing to S303. A method of determining this order will be described later.

In S303, the control unit 161 selects one piece of RAW image data to be subjected to the developing process in the order determined in S303. The recording/reproducing unit 151 reads the selected RAW image data from the recording medium 152, stores it in the buffer unit 115, and advances the processing to S304.

In S304, the control unit 161 causes the RAW decompression unit 114 to decompress the compressed RAW image data read from the recording medium 152 and stored in the buffer unit 115 to generate RAW image data. The generated RAW image data is sent to the high-quality image development unit 112, the development process is executed, and the raw image data is output to the still image compression unit 141 or the moving image compression unit 142 via the switch unit 121. Here, when the RAW image data to be developed is still image RAW image data, the still image compression unit 141 compression-encodes the developed still image data to generate still image data for recording, and performs recording/reproduction. The unit 151 records the still image data for recording on the recording medium 152. If the RAW image data to be developed is moving image RAW image data, the moving image compression unit 142 compression-encodes the developed moving image data to generate moving image data for recording, and the recording/reproducing unit 151 records this. The moving image data for recording is recorded in the recording medium 152.

The high-quality developing unit 112 performs debayer processing (demosaic processing) on the RAW image, converts it into a signal composed of luminance and color difference (or primary color), removes noise included in each signal, corrects optical distortion, and images. The so-called development process such as the optimization of is performed. The size (number of pixels) of the developed image generated by the high-quality image developing unit 112 is the same as the total number of pixels read from the image sensor unit 102 or the number of pixels set by the user. Therefore, the number of pixels of the image data from the high image quality developing unit 112 is higher than that of the image data output from the simple developing unit 111, and the image quality is high, but the processing load increases. Therefore, in the present embodiment, the high image quality developing unit 112 does not perform real-time development processing in parallel with shooting, but records the RAW image data on the recording medium 152 at the time of recording. The RAW image data is read from the recording medium 152 and the developing process is performed in an idle state where the processing load other than the photographing process is small. As a result, the number of pixels to be picked up and the speed of continuous shooting are increased, the number of pixels per frame and the frame rate are increased, and an increase in circuit scale and power consumption is suppressed.

The image data developed by the high image quality development unit 112 is supplied to the still image compression unit 141 and the moving image compression unit 142. When the developed image data is a still image, the still image compression unit 141 compresses and encodes the still image data according to a known encoding method such as JPEG. When the developed image data is a moving image, the moving image compressing unit 142 sets the H.264, H.264. The moving image data is compressed and encoded according to a known encoding method such as H.265. The still image compression unit 141 and the moving image compression unit 142 perform high-efficiency encoding (compression encoding) on the respective target image data, generate image data with a reduced amount of information, and send the image data to the recording/reproducing unit 151 for recording. The reproducing unit 151 records the developed compressed image data on the recording medium 152.

In step S305, the control unit 161 determines whether to save the original RAW image data, which has been subjected to the development processing this time in step S304, to the recording medium 152. That is, if the user determines that the original RAW image data is not set to be saved, the process proceeds to step S306, and if it is determined that the RAW image data is set to be saved, the process proceeds to step S307. Proceed. That is, if it is determined in S305 that the RAW image data is set to be saved, the original RAW image data is not deleted from the recording medium 152.

In step S<b>306, the control unit 161 controls the recording/reproducing unit 151 to delete the original RAW image data subjected to the development processing this time from the recording medium 152. When the RAW image data of the development source is recorded in the external storage or server, the recording/playback unit 151 controls the communication unit 153 to record the original RAW image data recorded in the external storage or server. Delete the RAW image data. Thus, each time one piece of developed compressed image data is recorded on the recording medium 152, the original RAW image data corresponding to this compressed image data is erased from the recording medium 152.

In step S307, the control unit 161 determines whether to shift from the idle state to the normal operation state according to the instruction from the user or the mode setting. The control unit 161 returns to the process of S201 of FIG. 2 when it is determined that the shift is made, and advances the process to S308 when it is determined that the shift is not made. Here, for example, when an instruction to start recording is input through the operation unit 162 in the idle state, the normal operation state in the shooting mode is entered. Besides this, it is also possible to shift to another mode such as a reproduction mode.

In S308, the control unit 161 determines whether or not the RAW image data to be subjected to the developing process is recorded on the recording medium 152. When the RAW image data to be developed is recorded in the recording medium 152, the process returns to S303, the next RAW file is selected according to the order determined in S302, and the processes in S303 and subsequent steps are repeated. On the other hand, when the RAW file to be developed is not recorded in the recording medium 152, the process returns to S307 and the idle state is continued until the normal operation state is entered.

Next, the determination processing of the order of the development processing of the RAW image data in S302 will be described.

When the RAW image data is not set to be saved, the developed RAW image data is automatically deleted after the developed image is recorded on the recording medium 152. Therefore, the order in which development processing is performed is important. Therefore, in the present embodiment, the developing process is not performed in the order in which the images are recorded, but the recording capacity of the recording medium 152 is efficiently used, and the order in which the developing process is performed is determined so that a long recording time can be secured. ..

In this embodiment, the order of the RAW image data to be subjected to the development processing is determined according to the type of image to be saved which is preset by the user. That is, the RAW image data that has not been set as a save target by the user is developed before the RAW image data that is set as a save target by the user. If the user is set to save both the RAW image and the developed image, the RAW image data set as the save target and recorded in the recording medium 152 is processed. Then, as described above, the RAW image data not set as the save target is deleted from the recording medium 152 after the developed image is recorded, but the RAW file set as the save target records the developed image. Even after this, it is not deleted from the recording medium 152. Therefore, the order of the development processing is determined so that the RAW image data not set as the storage target is processed before the RAW image data set as the storage target. By controlling in this way, the RAW image data that has undergone the development process is sequentially deleted, and the free space of the recording medium 152 is secured before the development process of the RAW image data set as the storage target is started. ..

Further, when RAW image data that is not set as a save target is recorded in the plurality of recording media 152, the RAW image data that is not set as a save target is based on the size (data amount) of the RAW image data. It suffices to determine the order in which the development processing is performed. For example, in the case of still image RAW image data, the time required for the developing process hardly changes even if the size is different. Therefore, it is possible to secure the free space of the recording medium 152 earlier by developing the RAW image data having a large size first. Therefore, for the RAW image data of the still image that is not set as the storage target, the developing process is performed in the order of the size of the RAW image data. In the case of moving image RAW image data, the longer the recording time, the larger the size (data amount). Therefore, the time required for the developing process of the RAW image data of a moving image varies greatly depending on the recording time. Therefore, in the case of RAW image data of a moving image, it is possible to secure the free space of the recording medium 152 earlier than that of the large size by performing the developing process in order from the shortest recording time and the small size.

FIG. 4A shows RAW image data of a moving image recorded in the recording medium 152. In the recording medium 152, a moving image RAW 401 of 400 megabytes (MB), a moving image RAW 402 of 300 MB, and a moving image RAW 403 of 350 MB are respectively recorded with respect to the total capacity 400, and the remaining space is an empty area 404. In this case, when all the RAW image data 401 to 403 are not set as the storage target, the development processing is performed in order of the moving image RAW 402, the moving image RAW 403, and the moving image RAW 401 in order from the smallest size.

When both the moving image and still image RAW image data are recorded on the recording medium 152, the user appropriately determines which of the moving image RAW image data and the still image RAW image data is to be developed. I can decide.

Further, the order of performing the developing process may be determined according to the shooting mode. For example, an image shot by multi-shot requires a longer processing time because the processing of combining the developed images is performed after the development processing. In this case, the RAW image data can be deleted earlier by developing the RAW image data of the still images individually shot instead of the images shot by the multi-shot, so that the recording medium 152 becomes empty sooner. It becomes possible to secure the capacity.

FIG. 4B shows RAW image data of a still image recorded on the recording medium 152. In the recording medium 152, a 40 MB still image RAW 411, a 32 MB still image RAW 412, a 43 MB still image RAW 415, and a 38 MB still image RAW 416 are recorded for the total capacity 410. A 116 MB multi-shot RAW 413 and a 102 MB multi-shot RAW 414 are recorded. Further, the remaining area is an empty area 417. The multi-shot RAWs 413 and 414 include three still image RAW image data for combining three images. In addition, when all the image data of the still image RAW image of FIG. 4B is not set as the storage target, the development processing is performed from the still image RAW image data other than the multi-shot and having a large size. Therefore, the development processing is performed in the order of the still image RAW415, the still image RAW411, the still image RAW416, the still image RAW412, the multi-shot RAW413, and the multi-shot RAW414. In this way, by determining the order of the development processing of the RAW file based on the shooting mode, it becomes possible to secure the free space of the recording medium 152 earlier.

It should be noted that the order of the development processing of the RAW image data of the moving image may be determined according to the compression ratio of the RAW image data of the moving image. For example, when recording the RAW image data, it is possible to select either a lossy compression method or a lossless compression method as the compression method of the RAW image data. In this case, the recording medium 152 records the RAW image data compressed by the lossy compression method and the RAW image data compressed by the lossless compression method. In general, the lossless compression method has a lower compression rate than the lossy compression method (=the amount of data after compression is large). However, when the recording time of the moving image data is the same, the time required for the development processing does not change between the lossy compression having a high compression rate and the lossless compression having a low compression rate. Therefore, developing the RAW image data of a moving image with a low compression rate first allows the RAW image data of a large size to be deleted earlier. Also, even if the recording time is different, it is efficient considering the amount of data that can be developed per unit time. Therefore, regardless of the recording time of the moving image, the RAW image data of the moving image with a low compression rate is given priority. Perform development processing.

Further, the order of development processing may be determined according to the size of the developed image after compression encoding. For example, the size of the developed image after compression coding is estimated based on the preset number of image pickup pixels and the image quality (compression rate), and the order of the development processing is determined based on the obtained size. The free space of the recording medium 152 increases as the RAW image data is deleted after the development processing, but it is also consumed to store the compressed and encoded file of the developed image. Therefore, it is possible to quickly secure the free space of the recording medium 152 by performing the developing process from an image having a small capacity necessary for saving the compressed and encoded file.

Further, the condition to be prioritized when the user determines the order of the development processing may be set in advance via the operation unit 162 or the like. Alternatively, a shooting mode that gives priority to saving the RAW image data may be provided, and the priority of the development processing may be lowered (the order is later) for the RAW image data shot by the user in the shooting mode.

In particular, the order of the development processing of the RAW image data becomes important when the free space of the recording medium 152 is decreasing. Therefore, the developing process may be performed in the order in which the RAW image data is recorded in the recording medium 152 until the free space of the recording medium 152 falls below a predetermined amount.

As described above, there are several methods for determining the order of the RAW image data development processing, but the order may be determined based on any one method, or a plurality of methods may be determined according to the shooting mode, shooting conditions, and user designation. The above methods may be combined.

As described above, according to the present embodiment, the RAW image is recorded in response to the recording start instruction from the user, and the processing of the device is relatively performed between the shootings, the playback mode, the sleep state, and the like. The developing process is executed in a state where the load is small (idle state). By doing so, it is possible to eliminate the delay in reproduction output as compared with the case where development is performed in real time in parallel with shooting without impairing the image quality of the image obtained by the image sensor unit 102.

Furthermore, at the time of shooting, the RAW image data recorded in a state not set as a storage target by the user is deleted from the recording medium 152 after the development processing of this RAW image data and the recording of the developed image data. Therefore, it is possible to secure the free space of the recording medium while recording only the image required by the user.

Further, in the present embodiment, the free space of the recording medium can be secured more quickly by determining the order of the RAW image data to be subjected to the development processing according to the type of the image to be saved which is set in advance by the user. This is advantageous in enhancing the shooting function, such as increasing the continuous shooting speed, increasing the number of pixels per frame and the frame rate, and ensuring a long shooting time.

In this embodiment, the configuration in which the user sets the type of image to be saved has been described. However, a developed image (compressed image) is automatically generated from all RAW images and saved, and It may be configured to set whether to save only the RAW image of. In this case, in S302 of FIG. 3, all the RAW images for which the developed image is not recorded are targets for the development processing. The order of development processing is determined as described in S302.

The RAW image data is high-quality image data without greatly impairing the image data supplied from the sensor signal processing unit 103. However, since the RAW image data is undeveloped data, it can be used as it is when displaying or printing. It cannot be used and requires development time. Further, since the RAW image data is not image data according to a known encoding method such as JPEG, the environment that can handle the RAW image data is also limited. However, in the present embodiment, after the RAW image is recorded, the developing process is executed between shootings and in a state where the processing load of the apparatus is relatively small (idle state) such as a reproduction mode and a sleep state. .. Therefore, while maintaining a high shooting function, the development process (reproduction output) is not delayed with respect to a request such as display or printing after shooting, and it is possible to use in an operating environment equivalent to the conventional one.

[Other Embodiments]
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

102... Imaging sensor section, 103... Sensor signal processing section, 110... Developing section, 151... Recording/reproducing section, 152... Recording medium, 161,... Control section, 162... Operation section

Claims (23)

Imaging means,
Generating means for generating RAW image data using the image data obtained by the image pickup means;
Recording means for recording the RAW image data generated by the generating means on a recording medium in response to a recording instruction;
Setting means for controlling, in accordance with a user's instruction, whether or not to set the RAW image data recorded in the recording medium by the recording means according to the recording instruction as a storage target to be stored in the recording medium,
A developing unit that performs a developing process on the RAW image data to generate developed image data;
The developing means is controlled to develop the RAW image data recorded on the recording medium, and the developed image data generated by developing the RAW image data by the developing means is developed on the recording medium. A control means for controlling the recording means to record,
The control unit automatically outputs the first RAW image data from the recording medium when the first RAW image data that is not set as a storage target to be stored in the recording medium is developed by the developing unit. When the second RAW image data that is deleted and saved in the recording medium and set as a storage target is developed by the developing unit, control is performed so that the second RAW image data is not deleted from the recording medium. An imaging device characterized by performing. The control unit controls the developing unit to automatically develop the plurality of RAW image data recorded on the recording medium, and automatically controls the plurality of developed image data generated by the developing unit. The image pickup apparatus according to claim 1, wherein the recording unit is controlled so as to record on the recording medium. The control means is configured to record one of a plurality of developed image data generated by the developing means by developing a plurality of the RAW image data which is not set as the storage target, into the recording medium. The image pickup apparatus according to claim 2, wherein RAW image data corresponding to the one developed image data is deleted from the recording medium. The control unit is configured to perform a plurality of processes recorded on the recording medium by the developing unit so that the RAW image data that is not set as the storage target is developed before the RAW image data that is the storage target. The image pickup apparatus according to claim 2, wherein an order of developing processing of the RAW image data is determined. When a plurality of the RAW image data sets that are not set as the storage target are recorded in the recording medium, the control unit sets the data amount of the plurality of RAW image data sets that are not set as the storage target based on the data amount. The image pickup apparatus according to claim 2, wherein an order of development processing of a plurality of RAW image data that is not set as the storage target by the developing unit is determined. When the plurality of RAW image data that are not set as the saving target are still images, the control unit determines the order so that the development processing is performed in the order of increasing data amount, and the setting is set as the saving target. The imaging apparatus according to claim 5, wherein when a plurality of RAW image data items that are not present are moving images, the order is determined so that the development processing is performed in ascending order of data amount. A means for setting the size and compression rate of the developed image data recorded on the recording medium,
The image pickup apparatus according to claim 5, wherein the control unit determines the order based on at least one of the set size and the compression rate. The recording means records, in the recording medium, identification information for identifying whether or not the RAW image data recorded in the recording medium is set as the storage target,
The control unit determines whether to delete the RAW image data from the recording medium when the RAW image data is developed by the developing unit, based on the identification information recorded on the recording medium. The image pickup apparatus according to claim 1, wherein The recording means records the RAW image data as a file on the recording medium,
9. The image pickup apparatus according to claim 8, wherein the identification information is recorded as metadata stored in a file of the RAW image data or an attribute of a file system of the recording medium. The control means controls the developing means so that the RAW image data recorded on the recording medium is read from the recording medium and automatically developed when the imaging device is in a predetermined idle state. The image pickup apparatus according to claim 1, wherein One of a plurality of recording modes including a first recording mode for recording a developed image and a second recording mode for recording a RAW image and a developed image is set according to a user's instruction. Equipped with means,
In the first recording mode and the second recording mode, the recording unit records the RAW image data generated by the generation unit on the recording medium in response to the recording instruction.
The setting means sets, according to the set recording mode, whether or not to set the RAW image data recorded on the recording medium by the recording means as the storage target in response to the recording instruction. The image pickup apparatus according to claim 1, wherein The developing unit performs a first developing process and a second developing process having a processing load larger than that of the first developing process,
In the recording standby state, the developing unit performs the first developing process on the RAW image data obtained by the generating unit, and displays the image data obtained by performing the first developing process on the display unit. Output,
After the recording means records the RAW image data on the recording medium, the control means performs the development so as to automatically perform the second developing process on the RAW image data recorded on the recording medium. The image pickup apparatus according to claim 1, wherein the image pickup apparatus controls the means. Imaging means,
Generating means for generating RAW image data using the image data obtained by the image pickup means;
Setting means for setting any one of a plurality of recording modes including a recording mode of a RAW image and a recording mode of a developed image according to a user's instruction;
Recording means for recording the RAW image data generated by the generating means on a recording medium in response to a recording instruction;
A developing unit that performs a developing process on the RAW image data to generate developed image data;
The developing means is controlled to develop the RAW image data recorded on the recording medium, and the developed image data generated by the developing means developing the RAW image data is recorded on the recording medium. A recording means for controlling the recording means so as to record,
The control means, on the basis of the recording mode of the RAW image data recorded on the recording medium, the RAW image data recorded on the recording medium in the recording mode of the developed image is developed by the developing means. In this case, the RAW image data recorded on the recording medium in the recording mode of the developed image is automatically deleted from the recording medium. The control means, when the RAW image data recorded on the recording medium in the RAW image recording mode is developed by the developing means, the RAW image recorded on the recording medium in the RAW image recording mode. The imaging apparatus according to claim 13, wherein control is performed so that data is not deleted from the recording medium. The control means controls the developing means to automatically develop the plurality of RAW image data recorded on the recording medium, and automatically controls the plurality of developed image data generated by the developing means. 14. The image pickup apparatus according to claim 13, wherein the recording unit is controlled so as to record the recording medium on the recording medium. One of a plurality of developed image data generated by the developing device developing the plurality of RAW image data recorded in the recording mode of the developed image on the recording medium. 16. The image pickup apparatus according to claim 15, wherein the recording unit is controlled so that the RAW image data corresponding to the one developed image data is deleted from the recording medium each time recording is performed. The control means may perform a developing process on the RAW image data recorded on the recording medium in the developed image recording mode before the RAW image data recorded on the recording medium in the RAW image recording mode. 16. The image pickup apparatus according to claim 15, wherein an order of development processing of the plurality of RAW image data recorded on the recording medium by the developing unit is determined so as to be performed. When a plurality of the RAW image data are recorded in the developed image recording mode, based on the data amount of the plurality of RAW image data recorded in the developed image recording mode, by the developing unit, The imaging apparatus according to claim 15, wherein the order of development processing of a plurality of RAW image data recorded in the recording mode of the developed image is determined. The recording means records identification information for identifying the recording mode set by the setting means on the recording medium,
The control means determines, based on the identification information recorded on the recording medium, whether to delete the RAW image data from the recording medium when the RAW image data is developed by the developing means. The image pickup apparatus according to claim 13, wherein The control unit controls the developing unit to automatically develop the RAW image data recorded on the recording medium when the imaging device is in a predetermined idle state. The imaging device according to item 13. The developing unit performs a first developing process and a second developing process having a processing load larger than that of the first developing process,
The developing unit outputs the image data obtained by performing the first developing process to the RAW image data obtained by the generating unit and performing the first developing process in a recording standby state. Then
After the recording means records the RAW image data on the recording medium, the control means performs the development so as to automatically perform the second developing process on the RAW image data recorded on the recording medium. The image pickup apparatus according to claim 13, wherein the image pickup apparatus controls the means. A generation step of generating RAW image data using the image data obtained by the image pickup means;
A recording step of recording the RAW image data generated in the generating step on a recording medium in response to a recording instruction,
A setting step of controlling according to a user's instruction whether or not to set the RAW image data recorded on the recording medium in accordance with the recording instruction as a storage target to be stored in the recording medium;
A developing step for developing the RAW image data,
The RAW image data recorded on the recording medium in the developing step is developed, and the developed image data generated by developing the RAW image data in the developing step is recorded on the recording medium in the recording step. And a control step for controlling to record,
The control step automatically deletes the first RAW image data from the recording medium when the first RAW image data which is not set as a save target to be saved in the recording medium is developed. When the second RAW image data set as a storage target to be stored in the recording medium is developed, control is performed so that the second RAW image data is not deleted from the recording medium. .. A generation step of generating RAW image data using the image data obtained by the image pickup means;
A setting step of setting one of a plurality of recording modes including a recording mode of a RAW image and a recording mode of a developed image in accordance with a user's instruction;
A recording step of recording the RAW image data generated by the generating step on a recording medium in response to a recording instruction,
A development step of developing the RAW image data,
The RAW image data recorded on the recording medium in the developing step is developed, and the developed image data generated by developing the RAW image data in the developing step is recorded on the recording medium in the recording step. And a control step for controlling to record,
The control step may include, when the RAW image data recorded on the recording medium is developed in a recording mode of the developed image based on a recording mode of the RAW image data recorded on the recording medium, A recording method, wherein the RAW image data recorded on the recording medium in the recording mode of a developed image is automatically deleted from the recording medium.

Images