JP2019110367A - Imaging device - Google Patents

Abstract

To develop picked-up raw images in appropriate order to more quickly secure a free space of a recording medium.SOLUTION: An image processing apparatus comprises: imaging means; raw image storage means that stores a sensor output acquired by the imaging means in a recording medium as raw images; developing means that develops the stored raw images in non-synchronization with the imaging of the raw images and stores developed raw images; order determination means that determines the order of the raw images to be developed by the developing means; and deletion means that deletes the raw images in which its development performed by the developing means is completed.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an imaging device.

  In a conventional imaging apparatus, there is a method in which raw image information (RAW image) captured by an imaging sensor is subjected to development processing (processing to create data in a format that can be recognized as an image) and then recorded on a recording medium. There are also imaging devices that record raw images as they are. Although the amount of data necessary for recording of a RAW image is enormous, the development processing can be omitted in the case of recording in the RAW image, so the processing until the recording completion becomes faster. Further, since the image is recorded in a non-developed state, it is difficult to reproduce and display the image quickly.

  In the reference document 1, there is a technique for realizing high continuous shooting performance by performing development processing with high image quality later by recording a RAW image as well as simple development for reproduction display during object shooting.

JP, 2014-179851, A

  However, in recent imaging devices, imaging sensors have evolved and the number of pixels per image has been significantly increased. Although the capacity of the recording medium has been increased as a method such as Reference 1 in which RAW recording is performed and priority is given to continuous shooting performance and later developed, the capacity of the recording medium is immediately sufficient. It will be gone. In the case of an imaging apparatus in which the remaining capacity of the recording medium runs out immediately, even if the performance is improved, the user's convenience is impaired.

  Therefore, an object of the present invention is to provide an imaging device capable of developing captured RAW images in an appropriate order and securing a free space of a recording medium more quickly.

  RAW image saving means for capturing an object and obtaining sensor output, RAW image saving means for saving the sensor output obtained by the imaging means as a RAW image in a recording medium, and imaging of the RAW image asynchronously The image forming apparatus includes a developing unit that develops and stores an image, an order determining unit that determines the order of the RAW images to be developed by the developing unit, and a deleting unit that deletes the RAW image that has been developed by the developing unit.

  According to the present invention, the captured RAW images can be developed in an appropriate order, and the free space of the recording medium can be secured more quickly.

It is a block diagram showing composition of an image processing device. It is a flow chart concerning processing of image photography mode. It is a flow chart concerning processing of an idle state. It is an example of the RAW image recorded on a recording device.

  Hereinafter, embodiments of the present invention will be described in detail. The embodiment described below is an example for realizing the present invention, and should be appropriately corrected or changed according to the configuration of the apparatus to which the present invention is applied and various conditions. It is not limited to the embodiment of the invention. Moreover, you may comprise combining suitably one part of each embodiment mentioned later.

First Embodiment
<Device configuration>
Hereinafter, the main configuration of the image processing apparatus of the present embodiment will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration example of an imaging device according to an embodiment of the present invention. The image pickup apparatus 100 shown in FIG. 1 not only records image information obtained by picking up an image of a subject on a recording medium, but also functions to reproduce image information from the recording medium, develop and display it, and image information Has a function of transmitting and receiving to and from an external device or server (cloud). Therefore, the imaging device according to the embodiment of the present invention can be expressed as an image processing device, a recording device, a reproduction device, a recording and reproduction device, a communication device, and the like.

  In FIG. 1, the control unit 161 includes a CPU and a memory for storing a control program executed by the CPU, and controls the entire processing of the imaging device 100. The operation unit 162 includes input devices such as keys and buttons used by the user to give an instruction to the imaging apparatus 100 and a touch panel. The operation signal from the operation unit 162 is detected by the control unit 161, and the control unit 161 controls the operation according to the operation. The display unit 123 includes a liquid crystal display (LCD) or the like for displaying an image shot or reproduced, a menu screen, various information, and the like in the imaging device 100.

  When the start of the imaging operation is instructed by the operation unit 162, an optical image of a subject to be imaged is input through the imaging optical unit 101 and forms an image on the imaging sensor unit 102. At the time of photographing, the operations of the imaging optical unit 101 and the imaging sensor unit 102 are the evaluation value calculation results such as the aperture, focus, and camera shake obtained by the evaluation value calculation unit 105, and the face recognition results extracted by the recognition unit 131 It is controlled by the camera control unit 104 based on the subject information.

  The imaging sensor unit 102 converts the light transmitted through the red, green, and blue (RGB) mosaic color filters disposed for each pixel into an electrical signal. For example, red (R), green (G), and blue (B) are arranged in a mosaic shape for each pixel of the image handled by the imaging device 100, and one red pixel and one blue pixel are arranged for four pixels of 2 × 2 , Green 2 pixels are used as one set and the structure arranged regularly is used. Such an arrangement of pixels is generally called a Bayer-array.

  The sensor signal processing unit 103 performs pixel repair processing on the electrical signal converted by the imaging sensor unit 102. The restoration process includes a process of interpolating the pixel to be restored using a peripheral pixel value or subtracting a predetermined offset value with respect to the value of a missing pixel or a pixel with low reliability in the imaging sensor unit 102. Be In the present embodiment, the image information output from the sensor signal processing unit 103 is referred to as RAW image information (hereinafter, RAW image) meaning a raw (undeveloped) image.

  The RAW image is developed by the developing unit 110. The developing unit 110 has a plurality of different developing units, includes a simple developing unit 111 as a first developing unit, and a high-quality developing unit 112 as a second developing unit, and selects their outputs. The switch unit 121 is included. Both the simple developing unit 111 and the high-quality developing unit 112 perform a debayering process (demosaicing process), that is, a color interpolation process on the RAW image, convert it into a signal of luminance and color difference (or primary color), and is included in each signal. A so-called development process is performed to remove noise, correct optical distortion, and optimize the image.

  In particular, the high image quality developing unit 112 performs each processing with higher accuracy than the simple developing unit 111. Because of high accuracy, a developed image with higher quality than that of the simple developing unit 111 can be obtained, but on the other hand, the processing load becomes large. Therefore, the high-quality image development unit 112 according to the present embodiment is not specialized for real-time development parallel to imaging, and can be configured to be able to perform distributed processing over the idle period after imaging. There is. As described above, by performing the high-quality development after taking time instead of photographing, increase (peak) in circuit size and power consumption can be suppressed to a low level.

  On the other hand, the simple developing unit 111 has a lower image quality than the high-quality developing unit 112, but is configured to have a smaller processing amount for developing than high-quality developing so that development can be performed at high speed during shooting. There is. Since the processing load of the simple developing unit 111 is small, the simple developing unit 111 is used mainly for real-time development parallel to the photographing operation. The switch unit 121 is switched by the control unit 161 in accordance with control according to the operation content instructed by the user via the operation unit 162 and the operation mode in execution. Further, in conjunction with the switching of the switch unit 121, only one of the simple developing unit 111 and the high image quality developing unit 112 which is the target of the signal output may perform the developing operation.

  In the present embodiment, the simple developing unit 111 and the high-quality developing unit 112 are independently present in the developing unit 110. However, one developing unit switches the operation mode, and the simple developing and the high developing are performed. Processing of image quality development may be selectively performed.

  The image information developed by the developing unit 110 is displayed on the display unit 123 after the display processing unit 122 performs predetermined display processing. The image information subjected to the development processing can also be output to an external display device connected to the outside through the video output terminal 124. The video output terminal 124 includes a general-purpose interface such as HDMI (registered trademark) or SDI.

  The image information developed by the developing unit 110 is also supplied to the evaluation value calculating unit 105. The evaluation value calculation unit 105 calculates an evaluation value such as a focus state or an exposure state from the image information.

  The image information developed by the developing unit 110 is also supplied to the recognition unit 131. The recognition unit 131 has a function of detecting and recognizing subject information such as a face and a person in image information. For example, a face in a screen represented by image information is detected, and if it is present, information indicating the position of the face is output, and authentication of a specific person is performed based on feature information such as a face.

  The image information developed by the developing unit 110 is supplied to the still image compression unit 141 or the moving image compression unit 142. When compressing image information as a still image, the still image compression unit 141 is used. When the image information is compressed as a moving image, the moving image compression unit 142 is used. The still image compression unit 141 and the moving image compression unit 142 perform high-efficiency encoding (compression encoding) on target image information to generate image information with a reduced amount of information, and generate an image file (still image file, Or convert to a movie file). Still image compression is JPEG, etc., moving image compression is MPEG-2, H.264, etc. H.264. 265 or the like can be used.

  The RAW compression unit 113 converts data of the RAW image output from the sensor signal processing unit 103 into high efficiency encoding using a technique such as wavelet conversion or differential encoding, and converts the data into a compressed RAW file. , And stored in the buffer unit (storage medium) 115. The RAW file can be left in the buffer unit 115 and read again, but after being stored in the buffer unit 115, it is moved to another recording medium and recorded (deleted from the buffer unit 115). Also good.

  The RAW file and the still image file and the moving image file described above are recorded on the recording medium 152 by the recording and reproducing unit 151. The recording medium 152 is, for example, a built-in large-capacity semiconductor memory or a hard disk, or a removable memory card or the like. The recording and reproducing unit 151 can also read a still image file, a moving image file, and a RAW file from the recording medium 152.

  The recording and reproducing unit 151 can write and read various files to an external storage or a server via the communication unit 153. The communication unit 153 has a configuration capable of accessing the Internet or an external device by wireless communication or wired communication using the communication terminal 154.

  When the reproduction operation is started, the recording and reproduction unit 151 acquires and reproduces a desired file from the recording medium 152 or via the communication unit 153. If the file to be reproduced is a RAW file, the recording / reproducing unit 151 stores the acquired RAW file in the buffer unit 115. If the file to be reproduced is a still image file, the recording / reproducing unit 151 supplies the obtained still image file to the still image decompression unit 143. If the file to be reproduced is a moving image file, the recording and reproducing unit 151 supplies the acquired moving image file to the moving image decompression unit 144.

  The RAW decompression unit 114 reads out the RAW file stored in the buffer unit 115, and decodes and decompresses the compressed RAW file. The RAW file expanded by the RAW expansion unit 114 is supplied to the developing unit 110 and input to the simple developing unit 111 and the high-quality developing unit 112 in the developing unit 110.

  In the present embodiment, if the file acquired by the recording and reproducing unit 151 via the recording medium 152 or the communication unit 153 is a RAW file, the RAW file after development by the developing unit 110 is recorded on the recording medium 152 or outside It is configured to be removable from storage and servers of

  The still image decompression unit 143 decodes and decompresses the input still image file, and supplies the same to the display processing unit 122 as a reproduced image of the still image. The moving picture expansion unit 144 decodes and expands the input moving picture file, and supplies it to the display processing unit 122 as a reproduced image of the moving picture.

<Description of operation>
Next, the operation in the image capturing mode of the imaging device 100 will be described.

  FIG. 2 shows a flowchart relating to the processing of the image shooting mode of the present embodiment. The flowchart in FIG. 2 illustrates the processing procedure executed by the control unit 161 to control and execute each processing block, and the program stored in the memory (ROM) of the control unit 161 is stored in the memory (RAM). It is realized by expanding and executing by CPU.

  In FIG. 2, when the process is started in the image capturing mode (S200), the control unit 161 determines whether the processing load status of the imaging apparatus 100 is low (S201), and the idle status (frequency) according to the load status. Go to S210), otherwise proceed to S202. In the case of performing normal single-shot still image shooting, the process transitions to S210 at a frequency of, for example, half between the first shooting and the second shooting.

  In step S202, the camera control unit 104 controls the operations of the imaging optical unit 101 and the imaging sensor unit 102 so as to perform imaging under suitable conditions. For example, the lens included in the imaging optical unit 101 is moved according to the user's instruction of zooming or focusing, or the reading area of the imaging sensor unit 102 is set according to the instruction of the number of imaging pixels.

  In step S203, the sensor signal processing unit 103 performs signal processing for the restoration of the pixel on the electrical signal converted by the imaging sensor unit 102. That is, the sensor signal processing unit 103 stores the values of the missing pixels and the pixels with low reliability using peripheral pixel values or subtracts a predetermined offset value. In the present embodiment, the image information output from the sensor signal processing unit 103 after the processing of step S203 is referred to as a RAW image meaning raw (undeveloped).

  In step S204, the control unit 161 determines an input of a shooting instruction from the user, and when there is a shooting instruction, the process proceeds to step S210. If there is no shooting instruction in S204, the process returns to S201, and the shooting preparation operation is repeated.

  In response to the imaging instruction in S204, the RAW compression unit 113 acquires the RAW image output from the sensor signal processing unit 103 corresponding to the imaged image, and performs high-efficiency encoding (RAW image compression) on the RAW image. And convert it into a RAW file (S205). The RAW file is stored in the buffer unit 115. The high efficiency encoding performed by the RAW compression unit 113 is processed by a known technique such as wavelet transform or differential encoding, but may be lossless encoding or lossless encoding. Alternatively, the RAW compression of the RAW compression unit 113 may be omitted, and the RAW image may be through-output without being compressed. Regardless of the presence or absence of RAW compression, in the present embodiment, a RAW file that can be restored as a high-quality file that does not significantly impair the image information supplied from the sensor signal processing unit 103 is generated.

  After the recording / reproducing unit 151 records the RAW file on the recording medium 152 in S206, the flow moves to S201. Note that, in S206, the recording / reproducing unit 151 may send the RAW file from the communication terminal 154 to the external storage via the communication unit 153, and record the RAW file by the external storage.

  The above is the description of the flow related to the processing of the image capturing mode according to the present embodiment.

  The imaging apparatus 100 according to the present embodiment generates a RAW file in response to a shooting instruction. The RAW file is a high image quality file that does not significantly impair the image information supplied from the sensor signal processing unit 103, but the file does not require development processing. Therefore, it is possible to record a RAW file with a small amount of power consumption by a small-scale circuit while increasing the number of pixels of an image and the continuous shooting performance (speed of continuous shooting).

  Subsequently, S210, which is a step of transitioning in S201 of FIG. 2, will be described using the flowchart of FIG. FIG. 3 shows a flowchart relating to the process of the idle state of the present embodiment. The flowchart in FIG. 3 illustrates the procedure of controlling and executing each processing block by the control unit 161. The program stored in the memory (ROM) of the control unit 161 is stored in the memory (RAM). It is realized by expanding and executing by CPU.

  In FIG. 3, when processing in the idle state is started (S300), the control unit 161 determines whether or not chase development is to be performed by the user setting (S301). If chase development is not performed, chase development is performed in S302. In the case of performing the process, the process transitions to S320.

  When the follow-up development is not performed in S301, it is determined whether to shift to the image shooting mode according to the instruction from the user and the mode setting (S302). Then, the control unit 161 controls to shift to the process flow of the selected mode (S310).

  In this embodiment, in order to make it easy to understand, the transition destination is only the image shooting mode, but the still image shooting mode, the moving image shooting mode, a plurality of images are shot and combined into one image to suppress camera shake and noise. There is a multi-shot shooting mode or the like for the purpose of obtaining an image (high dynamic range image) in which overexposure and underexposure are suppressed by combining three images having different brightness. Furthermore, there may be a reproduction mode for reproducing and displaying an image, not for capturing an image.

  Here, “follow-up development” according to the present embodiment refers to development processing with high image quality using a RAW file recorded in the buffer unit 115 or the recording medium 152 as a source after completion of the photographing operation. It means the process of generating still image files and high-quality still image files and moving image files. The RAW file to be an object of chase development according to the present embodiment is intended for both still images and moving images.

  Although the RAW file has high quality without significantly impairing the image information supplied from the sensor signal processing unit 103, it is data before development processing, so it can not be used immediately for display or printout, and for RAW development. It takes time for Also, since RAW is not a widely spread file such as JPEG, the playback environment that can handle RAW files is also limited.

  Therefore, the chase development of this embodiment is an effective function. In the present embodiment, when chasing development is executed, the RAW file already recorded is read out, development processing is performed with high image quality by the high image quality developing unit 112, and the generated high quality still image file is recorded as a recording medium It records to 152 mag. Then, such chasing development is performed between imaging and imaging, and in a state where the processing load on the apparatus is relatively small, such as a playback mode or a sleep state, which is waiting for user operation. The follow-up development is not limited to manual development, and may be designed to be automatically executed by the control unit 161.

  By this configuration, high-performance continuous shooting (high-speed continuous shooting) can be maintained because recording is performed without development, and then, even in the case of a request for high-quality reproduction such as confirmation of details and printout, There is no delay in development processing (reproduction output) each time, and it becomes possible to use in the same general operating environment as a conventional still image file.

  A RAW file corresponding to one shooting instruction is recorded in the recording medium 152 or the like. When performing chasing development manually or automatically, the control unit 161 determines whether there is a RAW image which has not been developed (S320).

  If the control unit 161 determines that the follow-up development has been processed, the process proceeds to step S302. If there is a RAW image that has not been processed for chase development, the process proceeds to step S321.

In step S321, when there is a RAW image that has not been processed by the chasing development in the buffer unit 115, the recording medium 152, or the like, the order of development is determined from among the unprocessed RAW images. The method of determining the order will be described later.
In step S322, the RAW decompression unit 114 reads a RAW image with high development priority from the buffer unit 115 or the recording medium 152, decompresses the RAW image, and restores the RAW image. The image is developed with high image quality at 112, and is output to the display processing unit 122, the still image compression unit 141, and the moving image compression unit 142 via the switch unit 121.

  The high-quality image development unit 112 performs a debayering process (demosaicing process), that is, a color interpolation process on the RAW image, converts it into a signal consisting of luminance and color difference (or primary color), removes noise contained in each signal, and optically Correction, and so-called development processing such as image optimization. The size (number of pixels) of the developed image generated by the high-quality developing unit 112 remains the full size read from the imaging sensor unit 102 or the size set by the user. While a high quality image is obtained, the processing load is increased. The high-quality image development unit 112 according to the present embodiment avoids real-time development processing parallel to shooting, and takes time to perform development processing, thereby increasing the circuit size and power consumption while maintaining high continuous shooting performance. Is configured to reduce the

  If the image information developed by the high-quality image development unit 112 is a still image, it is supplied to the still image compression unit 141, and the still image compression unit 141 performs high efficiency encoding processing (still image Compression) to generate a high quality still image file. The still image compression unit 141 performs compression processing by a known technique such as JPEG, and when the recording / reproducing unit 151 records the high quality image file on the recording medium 152 (S323), the developed RAW image is processed. The flow is deleted (S324), and the flow transitions to S302.

  As in the present embodiment, it is necessary to record a RAW image in order to realize follow-up development. However, since the data amount of the RAW image is large, and the chased development image and moving image are also recorded, the remaining capacity of the recording medium 152 will be exhausted immediately when continuous photographing is performed, and new photographing can be performed. There was a problem of being gone. Therefore, in the present embodiment, it is possible to secure the free space of the recording medium 152 by deleting the RAW image which has been developed in S324.

  Here, S321 according to the present embodiment will be described in detail. The order of development is also important because it is removed from RAW development after development. However, instead of developing in the order of shooting, the order of developing is sorted in S <b> 321 in order to enable shooting more efficiently.

  For example, there is a method of determining the order of development by the data size of the RAW image. When considered as a RAW image of still images of different data sizes, the development time does not change. Therefore, if the RAW image with the large data size is developed first and deleted, the free space of the recording medium 152 is secured quickly. It becomes possible. In addition, in the case of RAW images of moving images having different data sizes, basically, a moving image having a large data size has a large number of frames, so it takes much time for development. Therefore, in the case of a moving image, it is possible to secure the free space of the recording medium 152 more quickly by developing the smaller data size first.

  In the example of FIG. 4A, 400 indicates the entire capacity of the recording medium 152, and a 400 MByte moving image RAW 401, a 300 MByte moving image RAW 402, and a 350 MByte moving image RAW 403 are stored, It has become. At this time, development is performed in the order of the moving image RAW 402, the moving image RAW 403, and the moving image RAW 401 in ascending order of data size.

  As described above, by sorting the order of development according to the data size of the RAW image, it is possible to secure the free space of the storage medium 152 more quickly.

  Sorting may be performed according to the shooting mode. For example, in the case of three still images and in the case of a multi-shot for combining three still images, multi-shot takes a lot of time because there is a process for further combining after developing three sheets. Therefore, it is possible to delete a RAW image more quickly if priority is given to developing three still images. In the example of FIG. 4B, 410 indicates the entire capacity of the recording medium 152, and 40 MByte still images RAW 411, 32 MByte still images RAW 412, 43 MByte still images RAW 415, 38 MByte still images RAW 416, 116 MByte The multi-shot RAW 413 and the 102-Mbyte multi-shot RAW 414 are stored, and the rest are empty 417. Multi-shot RAW is composed of three in this example.

  At this time, first, since still images are prioritized and data sizes are larger, still images RAW 415, still images RAW 411, still images RAW 416, still images RAW 412, multi shot RAW 413, and multi shot RAW 414 are listed in this order. It will be developed.

  By thus determining the priority order according to the shooting mode, it is possible to secure the free space of the storage medium 152 more quickly.

  Also, sorting may be performed according to the compression rate of the RAW image. For example, considering a moving image of the same length, the time taken for development is the same between irreversible compression with a high compression ratio and lossless compression with a low compression ratio, so it is better to develop RAW moving images with a low compression ratio first. It is possible to erase large data size quickly. For example, even if it is a moving image with different length, it is efficient considering the amount of data that can be developed per unit time, so development is given priority over development of RAW moving image with low compression rate regardless of the length of the moving image You may

  Furthermore, the user may input in advance conditions to be prioritized using the operation unit 162, or allow the user to specify a shooting mode in which the RAW image is not deleted, and the RAW shot in the shooting mode The image development priority may be lowered.

  The order in which the RAW image is developed becomes important especially when the free space of the recording medium 152 decreases, and S321 is set to through until the free space of the recording medium 152 falls below a certain amount. You may not sort it.

  As described above, although there are several sorting methods, the order may be determined based on any one, or a plurality of combinations may be used according to the shooting mode, other shooting settings, and user specification. It is also good.

  As described above, according to the present embodiment, the sensor output is recorded as a RAW image, and the processing load of the apparatus is relatively small while waiting for user operation such as between shooting and shooting, playback mode and sleep state ( Executes chase development when idle. By doing this, it is possible to obtain an image of high quality, and it is also possible to eliminate the delay until reproduction output, unlike the case where development is carried out after the demand comes. Furthermore, on the premise that the RAW image after development is deleted, sorting the development order of the RAW image can secure a free space for the next shooting more quickly, so for a long time, It becomes possible to obtain high continuous shooting performance.

Other Embodiments
The present invention is also realized by performing the following processing. That is, software (program) for realizing the functions of the above embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU or the like) of the system or apparatus reads and executes a program code. Processing. In this case, the program and the storage medium storing the program constitute the present invention.

100 imaging device 161 control unit 162 operation unit

Claims (7)

Imaging means for imaging a subject and acquiring sensor output;
RAW image storage means for storing the sensor output acquired by the imaging means as a RAW image in a recording medium;
Developing means for developing and storing the stored RAW image asynchronously with imaging of the RAW image;
Order determining means for determining the order of the RAW images to be developed by the developing means;
A deletion unit that deletes a RAW image that has been developed by the development unit;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the order determination unit determines the order based on the data size of the RAW image.   The image processing apparatus according to claim 1, wherein the order determining unit determines the order according to a shooting mode at the time of shooting.   The image processing apparatus according to any one of claims 1 to 3, wherein the order determining unit determines the order according to a compression rate of a RAW image.   The image processing apparatus according to any one of claims 1 to 4, wherein the order determining unit can designate in advance a condition to be prioritized.   The image processing apparatus according to any one of claims 1 to 5, wherein the order determination unit lowers the priority of the RAW image specified by the user not to delete in advance.   The image processing apparatus according to any one of claims 1 to 6, wherein the order determining unit rearranges the order only when the free space of the storage capacity is smaller than a predetermined amount.