JP7497205B2 - Imaging device and control method thereof - Google Patents

Description

This paper relates to an imaging device and a control method thereof, and in particular to a technique for handling image data obtained by imaging.

In recent years, imaging devices such as digital cameras have been displaying a live view on a display device, causing the display device to function as an electronic viewfinder (EVF). During the live view display, the imaging device continues to perform operations from shooting video to displaying it. To limit the impact that the live view display has on the operating time of the imaging device, there is a demand to reduce the power consumption required for the live view display.

Patent document 1 discloses a digital camera in which the resolution of an image for live view display is lower than the resolution of the display device until the shutter button is half-pressed, and then becomes equal to the resolution of the display device when the shutter button is half-pressed. When the resolution of an image for live view display is lower than the resolution of the display device, the image is enlarged and displayed on the display device.

JP 2004-104222 A

The configuration of Patent Document 1 makes it possible to reduce power consumption and heat generation during live view display. However, the image quality of the live view display changes significantly before and after the shutter button is half-pressed, which may cause the user to feel uncomfortable. Furthermore, the configuration of Patent Document 1 may reduce the accuracy of processing using the image used for live view display (live view image) that is executed in the period before the shutter button is half-pressed. Such processing includes, for example, detection processing of characteristic regions and tracking processing based on the detection results of characteristic regions. When the results of processing using the live view image are used for focus control or exposure control when taking still images, a decrease in accuracy of processing using the live view image may cause a decrease in accuracy of focus control or exposure control.

The present invention was made in consideration of the problems with the conventional technology, and aims to provide an imaging device and a control method thereof that reduces power consumption during live view display and prevents deterioration in accuracy of processing that uses live view images.

The above-mentioned object is achieved by an imaging device having image processing means including a first compression means for applying compression processing to RAW data, a first decompression means for decompressing the RAW data compressed by the first compression means, a generation means for generating image data for display based on the RAW data output by the first decompression means, and a detection means for acquiring information used for shooting control based on the RAW data output by the first decompression means, and a control means for controlling the operation of the image processing means, wherein the control means reduces the compression rate of the compression processing applied by the first compression means when starting a shooting preparation operation using the information acquired by the detection means during execution of a live view display that continuously generates and displays image data for display.

The present invention provides an imaging device and a control method thereof that can reduce power consumption during live view display and prevent deterioration in accuracy of processing using live view images.

FIG. 1 is a diagram showing an example of the appearance of a digital camera as an example of an imaging apparatus according to an embodiment; FIG. 2 is a block diagram showing an example of the functional configuration of the digital camera and lens unit shown in FIG. 1 . FIG. 13 is a diagram showing an example of a color filter provided in an image sensor of a sensor unit; FIG. 2 is a block diagram showing an example of the functional configuration of a front engine and a main engine in the first embodiment; FIG. 2 is a block diagram showing an example of the functional configuration of a front engine and a main engine according to an embodiment; 1 is a flowchart showing the overall operation of a digital camera according to an embodiment of the present invention; A flowchart showing details of the live view process shown in FIG. Flowchart of the photographing and recording process of FIG. Flowchart of the playback process of FIG. 7 is a flowchart showing the main engine start-up and stop processes shown in FIG. FIG. 11 is a block diagram showing an example of the functional configuration of a front engine and a main engine according to a second embodiment; Flowchart regarding live view processing in the second embodiment Flowchart regarding live view processing in the third embodiment

The present invention will now be described in detail based on exemplary embodiments with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations will be omitted.

In the following embodiment, the present invention will be described with respect to a digital single-lens reflex camera (DSLR). However, the present invention can be implemented in any electronic device that has an electronic viewfinder (EVF). Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drive recorders, etc. These are merely examples, and the present invention can also be implemented in other electronic devices.

● (First embodiment)
Fig. 1 is a perspective view showing an example of the appearance of a digital camera 100 as an example of an imaging device capable of implementing the present invention. Fig. 1(a) is a front perspective view of the digital camera 100 as seen from the subject side. Fig. 1(b) is a rear perspective view of the digital camera 100 as seen from the opposite direction to Fig. 1(a). Fig. 1 shows an example of the appearance of the digital camera 100 (so-called camera body) when no lens is attached.

The display unit 101 is provided on the rear surface of the camera and displays images and various information. In this embodiment, the display unit 101 is a touch display having a touch panel 111.
The shutter button 102 issues a command to prepare for shooting a still image when pressed halfway, and issues a command to shoot a still image for recording when pressed fully.
A power switch 103 selectively issues a start instruction to turn the power of the digital camera 100 ON and a stop instruction to turn it OFF.

The main electronic dial 112 is rotatable, and for example issues instructions to change settings such as shutter speed and aperture. Furthermore, when the enlargement mode described below is ON, the main electronic dial 112 issues instructions to change the display magnification. The sub electronic dial 113 is rotatable, and for example issues instructions to move the selection frame or advance images. The cross key 114 has a circular member, and indicates the position (up, down, left, right) of the circular member that is pressed. The cross key 114 issues instructions to move in a direction corresponding to the pressed position, for example.

The SET button 115 is located at the center of the circular member of the cross key 114, and, for example, when pressed, issues an instruction to confirm the item that was selected. The video button 116 issues instructions to start and stop video shooting (recording). When pressed in shooting standby mode, the AE lock button 117 issues an instruction to fix the exposure state to the setting at the time of pressing. The enlarge button 118 issues an instruction to turn the enlargement mode ON or OFF in the live view display in shooting mode. In playback mode, the enlargement button 118 issues an instruction to turn the enlargement mode ON or OFF for the playback image.

When the playback button 119 is pressed during shooting mode, it instructs the camera to switch to playback mode, and when pressed during playback mode, it instructs the camera to switch to shooting mode. When the camera switches to playback mode, the most recent image recorded on the recording medium 200 is displayed on the display unit 101.

The menu button 120 issues an instruction to display a menu screen. The user can change the settings of the digital camera 100 by using the cross key 114 and the SET button 115 to operate the menu screen that is displayed on the display unit 101 in response to pressing the menu button 120. The mode change switch 121 issues an instruction to switch the operation mode of the digital camera 100. The operation modes include at least a still image capture mode, a video capture mode, and a playback mode. Note that each of the still image capture mode, the video capture mode, and the playback mode may further have multiple modes.

The types of instructions given by the operation of the various buttons, switches, and dials described above can be changed dynamically, and the combinations of operations and the instructions given to the digital camera 100 by them are merely examples. However, for the shutter button 102 and the power switch 103, the instructions given to the digital camera 100 by their operation are fixed.

The terminal group 104 is provided so that the digital camera 100 can communicate with a lens unit (interchangeable lens) or an adapter (mount converter, teleconverter, etc.) attached to the lens mount, and supply power to them. The lens and adapter are also provided with similar terminal groups (described below), and when attached to the digital camera 100, the terminal groups on both devices come into contact, electrically connecting the digital camera 100 to the lens or adapter.

The eyepiece 11 is provided for viewing the EVF arranged inside the digital camera 100. The eyepiece 11 is provided with an eyepiece detection unit that detects nearby objects. For example, when an object is detected by the eyepiece detection unit, the display unit 101 is turned off, thereby reducing power consumption.

The openable terminal cover 13 protects a terminal (not shown) for connecting the digital camera 100 to an external device. The openable lid 14 protects a slot into which a recording medium such as a memory card can be inserted and removed. The grip portion 15 is shaped to make it easy for the user to hold the digital camera 100.

Figure 2 is a block diagram showing an example of the functional configuration when lens unit 150 is attached to digital camera 100, and the components shown in Figure 1 are given the same reference numerals as in Figure 1. Lens unit 150 is an interchangeable lens for digital camera 100. Lens 151 usually has multiple lenses including fixed lenses and movable lenses, but for convenience it is illustrated as a single lens. Examples of movable lenses include focusing lenses, anti-vibration lenses, and variable magnification lenses.

A terminal group 153 is provided on the mount of the lens unit 150. The terminal group 153 is configured to contact the terminal group 104 when the lens unit 150 is attached to the digital camera 100. The lens unit 150 operates with power supplied from the digital camera 100 via the terminal groups 153 and 104, and also communicates with the digital camera 100 via the terminal groups 153 and 104.

The lens control unit 154 has, for example, a CPU, ROM, and RAM, and controls the operation of the lens unit 150 by loading a program stored in the ROM into the RAM and executing it. The lens control unit 154 communicates with the camera control unit 132 (described later) via terminal groups 153 and 104, and controls the operation of the lens unit 150 according to instructions from the camera control unit 132.

The aperture drive circuit 155 has an actuator that drives the aperture 152 provided in the lens unit 150. The aperture drive circuit 155 drives the aperture 152 under the control of the lens control unit 154.
The focus drive circuit 156 has a motor or an actuator that drives the focusing lens of the lens unit 150. The focus drive circuit 156 drives the focusing lens under the control of the lens control unit 154. The focus drive circuit 156 also obtains position information of the focusing lens and notifies the lens control unit 154.

The shutter 105 is a focal plane shutter, and operates under the control of the camera control unit 132 .
The sensor unit 106 has an image sensor and an A/D conversion circuit. The image sensor is, for example, a CCD image sensor or a CMOS image sensor. The image sensor has a plurality of pixels (photoelectric conversion units) arranged two-dimensionally, and converts an optical image formed on an imaging surface into a pixel signal group (analog image signal). The A/D conversion circuit converts the analog image signal into a digital signal (image data) and outputs it. Note that the image sensor may have a configuration in which one microlens is shared by a plurality of photoelectric conversion units. In this case, autofocus using a phase difference method is possible based on a signal obtained from the image sensor.

Figure 3 shows the color arrangement of color filters arranged on the image sensor. In this embodiment, the red (R) filter 1003, green (G) filter 1001, and blue (B) filter 1002 are primary color Bayer array color filters arranged in a repeating unit of four pixels, two pixels vertically and two pixels horizontally.

In this embodiment, the image sensor has a number of pixels corresponding to so-called 4K or 8K. In the case of 4K resolution, it is 3840 pixels horizontally by 2160 pixels vertically (approximately 8 million pixels), and in the case of 8K resolution, it is 7680 pixels horizontally by 4320 pixels vertically (approximately 33 million pixels). In addition, the sensor unit 106 is capable of outputting image data of 4K or 8K resolution at a frame rate of 120 frames per second.

The front engine 130 has an image processing unit 131 that processes image data acquired from the sensor unit 106, and a camera control unit 132 that controls the operation of the digital camera 100 and the lens unit 150. The front engine 130 may be configured as, for example, a single semiconductor integrated circuit package. The image processing unit 131 and the camera control unit 132 may be mounted on the same semiconductor chip, or may be mounted on separate semiconductor chips and enclosed in the same package. The camera control unit 132 has a CPU, and for example, loads a program stored in a ROM of the system memory 133 into the RAM 112 of the system memory 133 and executes it to control the operation of each unit.

The image processing unit 131 of the front engine 130 is mainly responsible for image data reduction (resolution reduction) processing and image processing of reduced image data. Image processing includes, for example, display image data generation processing, detection processing, evaluation value calculation processing, etc. Detection processing includes detection of characteristic regions (for example, face regions and human body regions) and their movements, person recognition processing, etc. Evaluation value calculation processing includes generation of signals and evaluation values used for autofocus detection (AF), calculation of evaluation values used for automatic exposure control (AE), etc. Note that these are examples of typical processing of the image processing unit 131, and other processing may also be performed.

The display image data generated by the image processing unit 131 here is image data (live view image data) for live view display on at least one of the display unit 101 and EVF 108. Note that live view display is a function that continuously captures moving images based on the current shooting conditions and displays the captured moving images in order to check the shooting range and shooting conditions. Note that the EVF 108 is a display device that is disposed inside the housing of the digital camera 100 and can be observed from outside the housing through the eyepiece unit 11. Here, it is assumed that the resolution of the EVF 108 and the display unit 101 is the same.

The display image data may be output to an external device via the communication unit 109. The display image data output to the external device may be different from the display image data used by the EVF 108 or the display unit 101. In this case, the display image data output to the external device is generated by the main engine 140.

The front engine 130 also controls the startup of the main engine 140 depending on the operation mode of the digital camera 100. The front engine 130 and the main engine 140 are physically separate semiconductor integrated circuits.

The system memory 133 has a non-volatile memory (ROM) and a volatile memory (RAM). The ROM stores programs executed by the camera control unit 132, settings for the digital camera 100, and GUI image data such as icons displayed along with menu screens and live view images. The RAM is used to load programs executed by the camera control unit 132 and to store variable values used during program execution.

The memory 134 is used by the image processing unit 131 to store image data to be processed, image data being processed, image data after processing, etc. For example, the memory 134 is a dynamic random access memory (DRAM). Note that the RAM of the system memory 133 may also be used to store image data.

The main engine 140 is configured as a semiconductor package separate from the front engine 130. It has an image processing unit 141, a recording/playback unit 143, and a control unit 142 that controls the operation of the main engine 140. The main engine 140 may be a single-chip integrated circuit, or may be configured with multiple chips enclosed in the same package. The image processing unit 141 applies image processing to the unreduced image data obtained from the front engine 130.

The recording and playback unit 143 records the image data processed by the image processing unit 141 on the recording medium 200, and reads out the data recorded on the recording medium 200 and outputs it to the image processing unit 141. The recording medium 200 may be, for example, a memory card or a magnetic disk. When the digital camera 100 is operating in the playback mode, the image data read out from the recording medium 200 and processed by the image processing unit 141 is displayed on the display unit 101 through the front engine 130. In addition, when external output is enabled, the image data read out from the recording medium 200 by the recording and playback unit 143 and processed by the image processing unit 141 is output to an external device via the communication unit 109.

The system memory 144 has a non-volatile memory (ROM) and a volatile memory (RAM). The ROM stores the programs and parameters executed by the control unit 142. The RAM is used to load the programs executed by the control unit 142 and to store variable values used during program execution.

The memory 145 is used by the image processing unit 141 to store image data to be processed, image data being processed, image data after processing, etc. For example, the memory 145 is a magnetoresistive random access memory (MRAM).

The power supply control unit 107 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to which power is supplied, and the like. The power supply control unit 107 detects the type of power supply 210 (e.g., AC adapter or battery), whether the power supply 210 is attached, and the remaining charge of the power supply 210 (battery). The power supply control unit 107 also controls the DC-DC converter based on these detection results and instructions from the camera control unit 132, and supplies the necessary power to each unit including the front engine 130 and main engine 140 for the necessary period. The power supply control unit 107 also supplies power to the recording medium 200 and the lens unit 150. The power supply control unit 107 also limits the power supplied to the main engine 140 in a standby state in which no image is recorded, and limits the power supply to the display unit 101 in response to the eyepiece detection unit detecting the proximity of an object.

The power source 210 is one or more of a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, or an AC adapter. Both the AC adapter and the battery are detachable from the digital camera 100.

The communication unit 109 is a wireless or wired communication interface, and the digital camera 100 communicates data with external devices through the communication unit 109. The communication unit 109 may have multiple communication interfaces that comply with standards. Typical standards to which the communication interface of the communication unit 109 complies include, but are not limited to, USB, HDMI (registered trademark), wireless LAN, Bluetooth (registered trademark), and NFC (Near Field Communication). The communication unit 109 may also have a GPS receiver and a transceiver for communication with a mobile phone network.

The operation unit 110 is a collective term for various input devices provided on the outer surface of the housing of the digital camera 100. It includes the main electronic dial 112, the sub electronic dial 113, the cross key 114, the SET button 115, the movie button 116, the AE lock button 117, the enlarge button 118, the playback button 119, the menu button 120, and the mode switch 121 shown in FIG. 1. The touch panel 111 provided on the display unit 101 is also included in the operation unit 110. For the sake of convenience, in this embodiment, the shutter button 102 and the power switch 103 are described separately from the operation unit 110.

The shutter button 102 has a switch (SW1) that is turned on when pressed halfway, and a switch (SW2) that is turned on when pressed all the way. The camera control unit 132 recognizes the half-pressed state of the shutter button 102 as an input of a shooting preparation instruction, and the all-pressed state as an input of an instruction to shoot a still image. In response to the input of the shooting preparation instruction, the camera control unit 132 executes shooting preparation operations such as AF (autofocus) processing and AE (autoexposure) processing. In response to the input of a still image shooting instruction, the camera control unit 132 also executes a series of operations such as driving the aperture and shutter, reading image data from the sensor unit 106, image processing in the main engine 140, and recording image data to the recording medium 200. In response to the input of the shooting preparation instruction or the still image shooting instruction, the camera control unit 132 releases the restriction on the power supply to the main engine 140.

Next, the front engine 130 and the main engine 140 will be described in further detail.
4 is a block diagram showing an example of the functional configuration of the front engine 130 and the main engine 140. The functional blocks of the image processing units 131 and 141 can be realized by a dedicated hardware circuit or by one or more CPUs of the image processing units executing a program. Data transmission between the functional blocks may be direct transmission from one functional block to another, or indirect transmission in which data written by one functional block to the memory 134 or 145 is read by the other functional block.

When a still image capture instruction is input, the camera control unit 132 starts the still image capture process. The camera control unit 132 controls the aperture 152 of the lens unit 150 and the drive of the shutter 105 so as to expose the image sensor at the aperture value and shutter speed determined by the AE process executed in response to the input of the capture preparation instruction. Drive control of the aperture and focusing lens of the lens unit 150 can be realized, for example, by sending a command from the camera control unit 132 to the lens control unit 154.

AE processing can be performed using any known method. The detection results of characteristic areas may be used in determining exposure conditions. If AE processing is not completed at the time when a still image capture instruction is input, the camera control unit 132 can use, for example, an aperture value and shutter speed that are pre-stored in the system memory 133.

The image data output by the sensor unit 106 is input to an image processing unit 131 of the front engine 130. The image processing unit 131 has a first path for performing data processing for generating display image data and detection processing, and a second path for performing data processing for generating recording image data.

First, the process in the first path will be described.
The pixel rearrangement processing unit 301 rearranges the RAW data input from the sensor unit 106 in the raster scan order into a two-dimensional matrix so that the arrangement of the pixel data corresponds to the pixel arrangement of the image sensor. The RAW data is image data having one color component value per pixel. In this embodiment, the image sensor has a primary color Bayer array color filter, so that the pixel data constituting the RAW data has one of a red component, a green component, and a blue component. In this specification, the RAW data is image data in which each pixel data has one color component. It can also be said that the image data before the synchronization process or the demosaic process that complements the color component values so that each image data has three color components is the RAW data. Therefore, the RAW data may be subjected to a process that does not affect the number of color components per pixel data, such as a data volume compression process. In the example of FIG. 5, the image data before being input to the LV development processing unit 308 or the RAW development processing unit 322 is RAW data, and the image data output by the LV development processing unit 308 or the RAW development processing unit 322 is not RAW data.

The compression processing unit 302 (first compression means) compresses the amount of data by encoding the RAW data output by the pixel rearrangement processing unit 301. The compression processing by the compression processing unit 302 can use an encoding method that can be executed with low power consumption and low delay, such as quantization, differential encoding such as DPCM, entropy encoding, or other methods that can be realized with a simple circuit configuration. The encoding may be lossless or lossy. The compression processing is performed to reduce the bus bandwidth in the front engine 131. The bus bandwidth is the data transfer capability or transfer capacity of the bus, and is determined according to the clock frequency of the bus, the bit width of the bus, and the latency. The compressed RAW data generated by the compression processing unit 302 may be temporarily stored in the memory 134 and buffered. The compression processing is performed on the RAW data in units of a predetermined two-dimensional pixel block. The compression rate of the compression processing in the compression processing unit 302 can be changed in units of encoding blocks based on the control of the camera control unit 132.

The decompression processing unit 303 (first compression unit) applies decompression (decoding) processing to the compressed RAW data generated by the compression processing unit 302 .
The sensor correction processing unit 304 executes a correction process (sensor correction process) based on the characteristics of the image sensor on the RAW data decoded by the decompression processing unit 303. The sensor correction process is, for example, a process of correcting the variation in the photoelectric conversion efficiency (sensitivity) of a plurality of photoelectric conversion units of the image sensor. The pixel values of the RAW data are corrected based on the sensitivity distribution information stored in advance in the system memory 133 or the like. The sensor correction process may include a correction process for defective pixels. The correction process for defective pixels may be a process of interpolating the value of a pixel registered as a defective pixel using the values of surrounding normal pixels or subtracting a predetermined offset value. Note that a part or all of the correction process for defective pixels may be executed as a part of the development process.

The raw data that has undergone the sensor correction process is supplied to a reduction processing unit 305 and a raw noise suppression processing unit 313 .
The reduction processing unit 305 reduces the RAW data (reduces the resolution) to obtain reduced RAW data in order to efficiently perform generation processing and detection processing of display image data. The reduction processing unit 305 resizes high-resolution RAW data such as 4K or 8K to, for example, HD resolution (equivalent to 2 million pixels). HD resolution is, for example, 1920 pixels horizontally by 1080 pixels vertically. The reduction processing unit 305 may reduce the RAW data to match the display resolution of the display unit 101 or the resolution of the display area of the live view image.

The optical correction processing unit 306 and the display processing unit 311 generate display image data from the reduced RAW data. In addition, the detection processing unit 312 calculates evaluation values used for AF, AE, etc., detects and tracks feature regions, recognizes subjects and scenes, calculates the magnitude of blur, etc., based on the reduced RAW data.

First, the processing from the optical correction processing unit 306 to the display processing unit 311 that generates display image data (live view image data) will be described.
The optical correction processing unit 306 applies correction processing (optical correction processing) related to the optical characteristics of the lens 151, etc., to the reduced RAW data. The optical correction processing is, for example, processing to correct the influence of a decrease in the amount of light in the peripheral area due to the aberration of the lens 151.

The noise suppression processing unit 307 applies noise reduction processing to the reduced RAW data. Noise reduction processing is generally called noise removal or noise reduction (NR). Noise reduction processing can be achieved, for example, by moving average filtering or median filtering.

The LV development processing unit 308 executes development processing (LV development processing) on the reduced RAW data. The LV development processing has a lower processing load and resource load (such as communication bandwidth with the memory 134) than the development processing applied by the RAW development processing unit 322 of the main engine 140. The development processing includes demosaic processing or synchronization processing, and generates image data in which each pixel has information on all color components (R, G, B). In this embodiment, the image data obtained by applying the LV development processing to the reduced RAW data is used as display image data.

The LV correction processing unit 309 performs correction processing such as distortion correction, enlargement processing, and reduction processing on the display image data. The LV correction processing unit 309 performs enlargement processing or reduction processing so that the resolution of the display image data matches the resolution of the display device (display unit 101 and EVF 108) that displays the live view image or the display area. Note that enlargement processing or reduction processing is performed only when necessary. Different resolutions may be used for the display unit 101 and the EVF 108.

The LV effect processor 310 applies effect processing to the corrected display image data to impart a specific display effect. Effect processing includes, for example, color conversion to sepia or monochrome, or processing to a mosaic or painting style. If effect processing is not required, the LV effect processor 310 outputs the input display image data as is.

The display processing unit 311 displays the display image data output by the LV effect processing unit 310 on the display unit 101 and EVF 108. The display processing unit 311 applies input/output correction, gamma correction, white balance correction, etc. to the display image data. The display processing unit 311 also combines image data of assist information with the display image data. Examples of assist information include, but are not limited to, images such as numerical values and icons indicating the current settings, and frame-shaped images indicating detected feature areas and focus detection areas (AF areas).

The numerical values indicating the current settings are, for example, ISO sensitivity, color temperature, shutter speed, aperture value, etc. Furthermore, the icons indicating the current settings include, for example, an icon indicating the number of shots in single shooting mode, continuous shooting mode, interval shooting mode, etc., an icon indicating whether or not the flash can be used, and an icon indicating the set shooting mode. The assist information may be synthesized so as to be superimposed on the peripheral area of the image based on the display image data, or may be processed so as to be displayed in an icon display area provided along the outer periphery of the image based on the display image data. The display processing unit 311 outputs the display image data that has been subjected to the above-mentioned processing to the display unit 101 or EVF 108.

The display unit 101 and EVF 108 display the input display image data. Through the above-described series of processes, one frame of a live view image is displayed on the display unit 101 and EVF 108. By executing a similar process at a predetermined frame rate, a live view display on the display unit 101 and EVF 108 is realized.

The detection processing unit 312 calculates an evaluation value for determining the shooting conditions at the time of shooting for recording from the reduced RAW data and for performing automatic focus detection, and performs processes such as detection and tracking of characteristic areas. Specifically, the detection processing unit 312 calculates a contrast evaluation value used for AF and generates an image signal pair for phase difference detection. The detection processing unit 312 also calculates a brightness evaluation value used for AE. The detection processing unit 312 stores the calculated evaluation value in the memory 134. The detection processing unit 312 also applies detection processing of characteristic areas such as face areas of people and animals to the reduced RAW data. The detection processing unit 312 may further apply subject recognition processing and tracking processing to the detected characteristic areas. The detection processing unit 312 stores detection results such as the position and size of the detected characteristic areas, information on the recognized subjects, and the movement direction and movement amount of the specific subjects in the memory 134. The generation and detection of the evaluation value performed by the detection processing unit 312 can be performed by a known method. Note that the evaluation value and detection processing described here are merely examples, and other evaluation values may be calculated and other targets may be detected.

The camera control unit 132 determines the exposure conditions and drives the focusing lens of the lens unit 150 based on the evaluation value and detection results read from the memory 134. The camera control unit 132 may perform subject recognition processing. The camera control unit 132 may also supply the evaluation value and detection results obtained by the detection processing unit 312 to the display processing unit 311. The display processing unit 311 can synthesize an indicator or icon according to the evaluation value and detection results into the display image data.

Next, the configuration of the image processing unit 131 that is used to generate image data to be recorded will be described.
The RAW noise suppression processing unit 313 reduces noise in the RAW data processed by the sensor correction processing unit 304. The RAW noise suppression processing unit 313 can execute noise reduction processing similar to that of the noise suppression processing unit 307.

The compression processing unit 314 applies known data compression (encoding) processing to the RAW data processed by the RAW noise suppression processing unit 313. The compression processing unit 314 can apply data compression processing that combines, for example, wavelet transform, quantization, and entropy coding (differential coding, etc.). The compression (encoding) processing applied by the compression processing unit 314 may be a lossy method or a lossless method. However, when applying a lossy compression process, a method or setting that causes sufficiently little degradation in the quality of the RAW data due to the compression process is used. The compressed RAW data generated by the compression processing unit 314 may be temporarily stored in the memory 134 before being sent to the main engine 140.

The transmission processing unit 315 transmits the compressed RAW data to the reception processing unit 321 of the main engine 140. By using the memory 134 as a transmission buffer, the data rate input to the main engine 140 can be dynamically adjusted. For example, depending on the progress of processing by the main engine 140, the transmission rate (transmission speed) between the transmission processing unit 315 and the reception processing unit 321 can be made slower than the readout rate of image data from the sensor unit 106.

The image processing unit 141 of the main engine 140 applies a higher quality development process to the RAW data acquired from the front engine 130 than the front engine 130 to generate image data for recording. In addition, the recording and playback unit 143 records the image data for recording generated by the image processing unit 141 on the recording medium 200.

The receiving processor 321 receives the compressed RAW data from the transmitting processor 315 and decodes the RAW data. The decoding process applied by the receiving processor 321 corresponds to the encoding process applied by the compression processor 314.

The RAW development processing unit 322 applies development processing to the decoded RAW data to generate image data for recording. The RAW development processing unit 322 applies de-Bayer processing (demosaic processing), i.e., color interpolation processing, to the RAW data so that each pixel has RGB color components. The RGB components may also be converted into luminance components and color difference components. Furthermore, the RAW development processing unit 322 applies processing to remove noise and correct optical distortion. The development processing performed by the RAW development processing unit 322 is of higher quality than the development processing performed by the LV development processing unit 308. Specifically, development processing using algorithms and calculations that can obtain higher quality images is applied. This is because the image data to be recorded is displayed on a large screen or printed, and therefore requires higher quality than the moving image displayed on the display unit 101 or EVF 108. Even for image data to be recorded, still image data requires higher quality than moving image data. Therefore, the RAW development processing unit 322 requires more circuitry and calculation resources than the LV development processing unit 308, and as a result, requires more power.

The correction processing unit 323 performs correction processing such as distortion correction, enlargement processing, reduction processing, and noise suppression processing on the image data that has been developed. When performing shooting and recording processing, the correction processing unit 323 performs distortion correction and noise suppression processing on the recorded image data that has been developed. When performing live view output processing to output image data to an external device as a live view image via the communication unit 109, in addition to distortion correction and noise suppression processing, the correction processing unit 323 performs enlargement processing or reduction processing for output to a display device.

When performing live view display on an external device, the effect processing unit 324 applies effect processing to the image data to obtain a specified display effect, and outputs the image data to the output processing unit 327. No processing is applied to the image data output to the compression processing unit 325.

The output processing unit 327 outputs the image data output from the effect processing unit 324 to an external device via the communication unit 109. When performing live view display on the external device, the output processing unit 327 performs input/output correction, gamma correction, white balance correction, etc. on the display image data (LV image data) output from the effect processing unit 324. The output processing unit 327 also combines the LV image data with an image indicating assist information to be displayed together with the LV image data. The assist information is similar to the information described for the display processing unit 311, so a description of the assist information is omitted. The output processing unit 327 outputs the LV image data after the combination process to the external device via the communication unit 109. Note that when performing playback processing on the external device, the output processing unit 327 executes similar processing except for the assist information, which is different.

The compression processing unit 325 applies a data volume compression process to the image data. The compression process applied by the compression processing unit 325 may be an encoding process that complies with a known standard. For example, the compression processing unit 325 can apply JPEG or HEIF format encoding process to still image data, and MPEG2, H264, or H265 format encoding process to video data.

The recording processing unit 326 of the recording and playback unit 143 records a data file that stores the encoded image data generated by the compression processing unit 325 on the recording medium 200, for example, in a manner that complies with the DCF (Design rule for Camera File system).

As described above, in the digital camera 100 of this embodiment, image processing related to live view display on the display unit 101 and EVF 108 can be performed by the front engine 130 alone, and there is no need to use the main engine 140. On the other hand, when recording image data, the main engine 140 is used in addition to the front engine 130.

5 is a block diagram showing an example of the functional configuration of the front engine 130 and the main engine 140 when the digital camera 100 operates in the playback mode. In Fig. 5, the same reference numerals as in Fig. 3 are used for the functional blocks described in the still image shooting mode.
When the digital camera 100 is operating in the playback mode, both the front engine 130 and the main engine 140 operate in a normal state. The normal state is a state in which image processing can be executed. In contrast to the normal state, the restricted state is a state in which at least power consumption is restricted to a level lower than that of the normal state, and for example, some or all of the image processing that can be executed in the normal state cannot be executed. Note that even in the restricted state, each engine can receive an instruction regarding engine startup from the outside and execute startup control. In other words, the restricted state can also be said to be a standby state.

For example, in a normal state, the front engine 130 can execute image processing to generate display image data to be displayed on the display unit 101 and/or EVF 108 from the RAW data input from the sensor unit 106. In addition, in a normal state, the front engine 130 can execute processing to compress the RAW data input from the sensor unit 106 and output it to the main engine 140. Furthermore, the front engine 130 can execute image processing to generate display image data from image data supplied from the main engine 140 and display it on the display unit 101. The front engine 130 includes a camera control unit 132 that controls the operation of the digital camera 100. Therefore, when the power supply of the digital camera 100 is on, the front engine 130 does not generally operate in a restricted state. An exception is when the digital camera 100 goes into sleep mode after a predetermined period of inactivity.

In the normal state, the main engine 140 can execute a recording control process to generate image data for recording from the compressed RAW data input from the front engine 130 and record the image data on the recording medium 200. In the normal state, the main engine 140 can execute a playback display control process to read image data stored on the recording medium 200 and output the image data to the front engine 130. In the normal state, the main engine 140 can execute an output control process to output image data for display input from the front engine 130 to an external device via the communication unit 109. On the other hand, in the restricted state, the main engine 140 cannot execute one or more of the above-mentioned recording control process, playback display control process, and output control process.

When operation in the playback mode begins, the camera control unit 132 controls the read processing unit 401 of the recording and playback unit 143 to read the image file recorded on the recording medium 200. The read processing unit 401 outputs the read image file to the image processing unit 141.

In the image processing unit 141, the decompression processing unit 402 decodes the encoded image data stored in the input image file. The decompression processing unit 402 outputs the decoded image data to the transmission processing unit 403 and the output processing unit 327. The transmission processing unit 403 transmits the image data to the front engine 130. Note that the decompression processing unit 402 may be the same functional block as the compression processing unit 325, and the transmission processing unit 403 may be the same functional block as the reception processing unit 321.

The image data output by the decompression processing unit 402 may be supplied to the output processing unit 327 via the effect processing unit 324 as necessary. The output processing unit 327 generates display image data from the received image data and outputs it to an external device via the communication unit 109. The output processing unit 327 performs scaling, input/output correction, gamma correction, white balance correction, etc. on the image data output from the decompression processing unit 402. The output processing unit 327 also performs processing to combine an image of assist information (GUI such as an icon) that is displayed together with the display image data with the image data. The output processing unit 327 is also capable of generating thumbnail images for thumbnail display. The output processing unit 327 outputs the display image data to an external device via the communication unit 109.

A reception processing unit 411 of the front engine 130 receives the image data output from the main engine 140, and outputs the image data to the display processing unit 311. The reception processing unit 411 may be the same functional block as the transmission processing unit 315.
The display processing unit 311 generates display image data based on the image data, and displays the display image data on the display unit 101 or the EVF 108 .

When the user operates the operation unit 110 to instruct thumbnail display, the camera control unit 132 controls the main engine 140 to read multiple image files to be used for thumbnail display from the recording medium 200. The main engine 140 outputs multiple image data to the front engine 130. The camera control unit 132 then generates thumbnail images for each of the multiple image data, and controls the display processing unit 311 to generate a list display screen. The display processing unit 311 may use the reduction processing function of the LV correction processing unit 307 to generate the thumbnail images.

Next, the overall operation of the digital camera 100 will be described using the flowchart in FIG. 6. The operation shown in the flowchart in FIG. 6 begins when the power switch 103 of the digital camera 100 is instructed to turn the power on.

In S501, the startup process of the front engine 130 is executed. In response to a power ON command from the power switch 103, the power supply control unit 107 supplies power for the front engine 130 to operate in a normal state. The camera control unit 132 of the front engine 130 reads out a startup program and parameters from the system memory 133 and executes the startup operation. Note that the power supply control unit 107 also starts supplying power to functional blocks other than the front engine 130, but supplies power to the main engine 140 to operate in a restricted state.

In S502, the camera control unit 132 determines whether the operation mode of the digital camera 100 is the shooting mode or the playback mode. The camera control unit 132 reads out the setting data stored in the system memory 133 to check the operation mode of the digital camera 100. Alternatively, the camera control unit 132 checks the operation mode of the digital camera 100 based on the operation mode indicated by the mode changeover switch 121. If it is determined that the shooting mode is set, the camera control unit 132 executes S503. On the other hand, if it is determined that the playback mode is set, the camera control unit 132 executes S514. Note that in the explanation of this flowchart, the case where the shooting mode is the still image shooting mode will be explained, but the shooting mode may also be the moving image shooting mode.

Also, in S502, the camera control unit 132 determines whether the main engine 140 is operating in a normal state, and if it is determined that the main engine 140 is operating in a normal state, changes the main engine 140 to operate in a restricted state. This is to set the state of the main engine 140 to a restricted state when, for example, the external output function is changed from enabled to disabled. Details of this process will be described later.

At S503, the camera control unit 132 starts executing live view processing. Live view processing is processing that continuously executes operations of capturing moving images using the sensor unit 106, generating display image data from the obtained moving image data, and displaying the data on the display unit 101 (EVF 108). When live view processing is executed, an image corresponding to the current shooting conditions and shooting range is continuously displayed on the display unit 101 or EVF 108. The user checks the displayed live view image and adjusts the shooting range and shooting conditions. While live view processing is being executed and no instructions have been input regarding preparation or execution of still image shooting, the digital camera 100 can be said to be in a shooting standby state.

Figure 7 is a flowchart showing the details of the live view processing in S503. Figure 7 shows the processing of one frame of a live view image (moving image), and in reality, the processing shown in Figure 7 is repeated and executed continuously to achieve a predetermined display frame rate (e.g., 30 frames/second). While the digital camera 100 is operating in shooting mode, the live view processing continues to be executed regardless of whether images are being recorded or not, unless the user instructs the camera 100 to stop the live view display.

In S601, the camera control unit 132 controls the operation of the lens 151 and the sensor unit 106, acquires (takes) an optical image, and outputs one frame of image data. For example, the camera control unit 132 outputs an instruction to the lens control unit 154 via the control terminal group 104 to change the angle of view of the lens 151 according to a user instruction, or to change the focal distance of the lens 151 so as to focus on a specified focus detection area. Note that the camera control unit 132 controls the sensor unit 106 to fully open the shutter 105 during shooting of a live view image, and to perform shooting using a so-called electronic shutter. Note that the camera control unit 132 may determine the shooting conditions and focus detection area to be used when shooting a video, taking into account the evaluation value obtained by the detection process and the detection result of the characteristic area.

In S602, the camera control unit 132 causes the image processing unit 131 to execute a process of reading out the image data. For example, it is assumed that the readout speed is 1000 MP/sec. MP stands for 1 million pixels.
In S603, the pixel rearrangement processing unit 301 of the image processing unit 132 rearranges the image data input in raster scan order into a two-dimensional matrix to generate one frame of RAW data.

In S604, the compression processing unit 302 encodes the raw data generated by the pixel rearrangement processing unit 301 in units of a predetermined two-dimensional pixel block, and outputs compressed raw data. The compression ratio of the data volume by encoding is based on parameters set by the camera control unit 132.

In S605, the decompression processing unit 303 decodes the compressed RAW data.
In S606, the sensor correction processing unit 304 executes a correction process (sensor correction process) on the decoded raw data based on the characteristics of the sensor acquired in advance.
In S607, the reduction processing unit 305 applies reduction processing to the RAW data output by the sensor correction processing unit 304 to generate reduced RAW data.

In S608, the detection processing unit 312 applies a process for calculating a predetermined evaluation value and a process for detecting a characteristic region to the reduced RAW data. Note that the process of the detection processing unit 312 may be executed in parallel with other processes included in the LV process.

In S609, the optical correction processing unit 306 applies correction processing (optical correction processing) related to the optical characteristics of the lens 151 and the like to the reduced RAW data.
In S610, the noise suppression processing unit 307 applies processing for reducing noise to the reduced RAW data to which the optical correction processing has been applied.
In S611, the LV development processing unit 308 applies development processing (LV development processing) to the reduced RAW data on which noise suppression processing has been performed, and generates display image data.

In S612, the LV correction processing unit 309 applies correction processing such as distortion correction, enlargement processing, and reduction processing to the display image data.
In S613, the LV effect processing unit 310 applies effect processing to the display image data in order to obtain a predetermined display effect.
In S614, the display processing unit 311 causes the display unit 101 and the EVF 108 to display the display image data.
This completes the processing for one frame of the live view image.

Returning to the explanation of the flowchart in FIG. 6, in S504, the camera control unit 132 determines whether or not the SW1 signal has been input, i.e., whether or not a half-press of the shutter button or an instruction to prepare for shooting has been detected. If it is determined that the SW1 signal has been input, the camera control unit 132 executes S505, and if it is not determined that the SW1 signal has been input, the camera control unit 132 executes S511.

In S505, the camera control unit 132 instructs the compression processing unit 302 to change the compression rate. While the digital camera 100 is operating in the shooting mode, the compression processing unit 302 encodes the RAW data read from the sensor unit 106 at a predetermined compression rate to reduce the bus bandwidth of the front engine 131.

In S505, the camera control unit 132 reduces the compression rate compared to before the shooting preparation instruction was input. A lower compression rate means that the amount of data that is reduced is smaller, and the degradation of image quality due to compression is smaller. This makes it possible to improve the accuracy of detection processing based on image data, such as the calculation of evaluation values and detection of characteristic areas in the detection processing unit 312, compared to before the compression rate was reduced.

In S506, the camera control unit 132 executes a shooting preparation operation using the image processing unit 131. Specifically, the camera control unit 132 executes an AE process that determines exposure conditions based on the evaluation value and the characteristic area detection result obtained by the detection processing unit 312, and an AF process that adjusts the focal distance (position of the focusing lens) of the lens 151 so that the focus detection area is in focus.

In S507, the camera control unit 132 determines whether or not the SW2 signal has been input, i.e., whether or not a full press of the shutter button or a shooting instruction has been detected. If it is determined that the SW2 signal has been input, the camera control unit 132 executes S508, and if it is not determined that the SW2 signal has been input, the camera control unit 132 executes S511.

In S508, the camera control unit 132 executes a startup process for the main engine 140. The startup process for the main engine 140 will be described in detail with reference to the flowchart shown in FIG.
In S<b>901 , the camera control unit 132 outputs an instruction to the power supply control unit 107 to release the restriction on the power supply to the main engine 140 .
In S902, the power supply control unit 107 releases control of the power supply to the main engine 140, and starts supplying the power necessary for the main engine 140 to operate in a normal state.
In S903, the control unit 143 of the main engine 140 executes a process of reading, from the system memory 144, a program and parameters used for starting up the main engine 140.
In S904, the control unit 143 executes startup control of the main engine 140. As a result, the operation of the main engine 140 transitions from the restricted state to the normal state.

Returning to FIG. 6, in S509, the camera control unit 132 and the control unit 143 execute the shooting and recording process. The shooting and recording process is a series of processes from still image shooting in the sensor unit 106, development processing of the RAW data in the main engine 140, and recording of the image data obtained by the development processing on the recording medium 200.

The photographing and recording process will be described in detail with reference to the flowchart shown in FIG.
In S701, the camera control unit 132 controls the operations of the diaphragm 152 and the shutter 105 based on the exposure conditions determined in the image capture preparation process in S506, and exposes the image pickup element of the sensor unit 106 to light.

In S702, the image processing unit 131 reads out image data from the sensor unit 106. If the image processing unit 131 reads out one frame of image data from the sensor unit 106 in 5 ms, the readout speed is 1500 MP/sec.
In S703, the pixel rearrangement processing unit 301 rearranges the image data input in raster scan order into a two-dimensional matrix to generate one frame's worth of RAW data.

In S704, the compression processing unit 302 encodes the RAW data generated by the pixel rearrangement processing unit 301 in units of a predetermined two-dimensional pixel block to generate compressed RAW data with a reduced data volume. The compression ratio of the data volume by encoding is based on parameters set by the camera control unit 132.
In S705, the decompression processing unit 303 decodes the compressed RAW data.
In S706, the sensor correction processing unit 304 applies correction processing (sensor correction processing) based on the characteristics of the sensor acquired in advance to the decoded RAW data.

In S<b>707 , the RAW noise suppression processing unit 313 applies noise reduction processing to the RAW data output by the sensor correction processing unit 304 .
In S708, the compression processing unit 314 generates compressed RAW data by applying compression processing to the RAW data output by the RAW noise suppression processing unit 313. The compression processing unit 314 temporarily stores the compressed RAW data in the memory 134 for buffering.

In S709, the transmission processing unit 315 transmits the compressed RAW data to the reception processing unit 321 of the main engine 140. For example, the transmission processing unit 315 transmits one frame to the reception processing unit 321 at 15 ms. In other words, the front engine 130 transmits the compressed RAW data to the main engine 140 at a data rate slower than the speed (5 ms) at which the front engine 130 reads one frame of image data from the sensor unit 106.

The RAW data to which the image processing unit 141 of the main engine 140 applies image processing (development processing) is not reduced, and therefore has a higher resolution than the RAW data to which the image processing unit 131 of the front engine 130 applies image processing (development processing). Furthermore, the development processing applied by the image processing unit 141 places emphasis on good image quality obtained, whereas the development processing applied by the image processing unit 131 places emphasis on processing speed. Therefore, the development processing applied by the image processing unit 141 has a higher computational load than the development processing applied by the image processing unit 131. Therefore, the processing rate of the image processing unit 141 is lower (slower) than the processing rate of the image processing unit 131.

By buffering the compressed RAW data within the front engine 131, it is possible to make the transfer speed of the RAW data from the front engine 130 to the main engine 140 slower than the read speed of the sensor unit 106. In other words, data transfer can be matched to the processing rate of the main engine 140. This makes it possible to keep the processing rate of the main engine 140 lower than the read speed of the sensor unit, thereby realizing the execution of highly accurate image processing while suppressing increases in power consumption and circuit size. Note that the development process applied by the image processing unit 141 has higher accuracy and/or quality than the development process applied by the image processing unit 131, for example, in terms of color reproducibility and correction accuracy.

In S710, the reception processing unit 321 receives the compressed RAW data transmitted from the transmission processing unit 315.
In step S711, the reception processing unit 321 applies decompression processing to the compressed RAW data to release the compressed state. The decompression processing applied by the reception processing unit 321 corresponds to the inverse process of the compression processing applied by the compression processing unit 341.

In S712, the RAW development processing unit 322 applies development processing to the RAW data and generates image data for recording.
In S713, the correction processing unit 323 applies distortion correction and noise suppression processing to the image data for recording.
In S714, the effect processor 324 applies effect processing to the image data for recording. The content of the effect processing is processing for imparting a predetermined effect to the recording image data. The effect processing is, for example, monochrome conversion processing or processing for applying various filters.

In S715, the compression processing unit 325 applies compression processing to the image data for recording to generate an image data file. The compression processing method may be based on a predetermined encoding method.
In S 716 , the recording processing unit 326 of the recording and reproducing unit 143 stores the image data file on the recording medium 200 .

In S717, the camera control unit 132 determines whether the continuous shooting function is enabled. The continuous shooting function can be set to enabled or disabled by the user, for example, via a menu screen. If it is determined that the continuous shooting function is enabled, the camera control unit 132 proceeds to S718, and if it is not determined that the continuous shooting function is enabled, the camera control unit 132 ends the shooting and recording process and executes S510.

In S718, the camera control unit 132 determines whether or not the SW2 signal has been input, i.e., whether or not the shutter button 102 is still fully pressed. If it is determined that the SW2 signal has been input, the camera control unit 132 executes again from S701, and if it is not determined that the SW2 signal has been input, it ends the shooting and recording process and executes S510.

In the above-mentioned flowchart, the main engine 140 records image data obtained by applying decompression processing and development processing to the compressed RAW data output from the front engine 130 on the recording medium 200. The main engine 140 can also record the compressed RAW data output from the front engine 130 on the recording medium 200. In this case, the series of processes from S711 to S715 is not executed. The recording and playback unit 143 records the compressed RAW data received by the reception processing unit 321 on the recording medium 200.

6, in S510, the camera control unit 132 executes a stop process for changing the operation state of the main engine 140 from the normal state to the restricted state. The stop process of the main engine 140 will be described in detail with reference to the flowchart shown in FIG.
In S911, the camera control unit 132 outputs an operation stop instruction to the control unit 143 of the main engine 140.
In S912, the control unit 143 stores data (start-up data) such as parameters required for the next startup of the main engine 140 in the system memory 144. Note that the data may be stored in the memory 145.
In S<b>913 , the control unit 143 executes a process of stopping the operation of the main engine 140 .
In S914, the camera control unit 132 outputs, to the power supply control unit 107, an instruction to change the power supplied to the main engine 140 to power for operating in a restricted state.
In S915, the power supply control unit 107 reduces the amount of power supplied to the main engine 140 to a predetermined power level for a limited state. This causes the operation of the main engine 140 to transition from the normal state to the limited state.
By transitioning the operation state of the main engine 140 to the restricted state, it becomes possible to reduce the power consumption of the digital camera 100 .

If it is not determined in S504 that the SW1 signal has been input, in S511 the camera control unit 132 resets the compression rate of the compression process applied by the compression processing unit 302 to the initial value. Note that if S505 has not been executed and the compression rate remains at the initial value, it is not necessary to execute S511.

In S512, the camera control unit 132 determines whether an instruction to end the operation of the digital camera 100 has been input. Specifically, the camera control unit 132 determines whether the power switch 103 has been operated from on to off. Alternatively, if there is no operation for a predetermined period of time, it may be determined that an instruction to end has been input (so-called sleep operation). If it is determined that an instruction to end has been input, the camera control unit 132 executes S513. If it is not determined that an instruction to end has been input, the camera control unit 132 executes S502.

At S513, the camera control unit 132 executes a process to stop the front engine 130. This causes the operation of the camera control unit 132 itself to stop. The camera control unit 132 also executes a process to stop the other units of the digital camera 100. This causes the operation of the entire digital camera 100 to stop.

If it is determined in S502 that the operation mode of the digital camera 100 is not the shooting mode (it is the playback mode), the camera control unit 132 executes S514.
In S514, the camera control unit 132 executes the startup process of the main engine 140, similar to S508.

Then, in S515, the camera control unit 132 executes a playback process. The playback process is a process in which image data stored in the recording medium 200 is read and displayed on the display unit 100. The playback process will be described in detail using the flowchart shown in FIG. 9.

In S801, the recording and playback unit 143 of the main engine 140 reads out the image data file specified by the camera control unit 132 from the recording medium 200. The camera control unit 132 can instruct, for example, to read out the last stored image data file. The recording medium 200 stores image data files in a predetermined format, such as JPEG. The recording and playback unit 143 outputs the read image data file to the image processing unit 141.

In S802, the image processing unit 141 decodes (decompresses) the encoded image data stored in the image data file. Specifically, the image processing unit 141 applies a decoding process corresponding to the encoding method of the encoded image data to the encoded image data.

In S803, the image processing unit 141 transmits the decoded image data to the image processing unit 131 of the front engine 130.
In S<b>804 , the image processing unit 131 receives the image data transmitted from the main engine 140 .
In S805, the image processing unit 131 generates display image data based on the received image data, and displays the data on the display unit 101 or the EVF 108. This processing has been described as the processing of the display processing unit 311 in FIG.

In S806, the camera control unit 132 determines whether an instruction to change the display image has been input via the operation unit 110. For example, the change in the display image may be image forwarding by operating the cross key 114, or changing to thumbnail display by operating the enlargement button 118. If it is determined that an instruction to change the display image has been input, the camera control unit 132 specifies an image data file according to the instruction, and causes the recording and playback unit 143 to execute S801.

On the other hand, if it is determined that an instruction to change the display image has not been input, the camera control unit 132 executes S807. In S807, the camera control unit 132 determines whether or not an instruction to change the operating mode of the digital camera 100 has been input. The instruction to change the mode may be an operation of the mode selector switch 121 or the shutter button 102, or may be some other operation. If it is not determined that an instruction to change the mode has been input, the camera control unit 132 repeatedly executes S806. If it is determined that an instruction to change the mode has been input, the camera control unit 132 ends the playback process and executes S510.

In this embodiment, the camera control unit 132 limits the power supply to the main engine 140 in a shooting standby state, and releases the restriction on the power supply to the main engine 140 in response to input of a shooting instruction to record image data on a recording medium. However, the trigger for releasing the restriction on the power supply to the main engine 140 is not limited to the input of an instruction to shoot a still image for recording. For example, the restriction on the power supply to the main engine 140 may be released in response to input of a shooting preparation instruction, which is a preparatory operation before shooting. In this case, the main engine 140 can be started up completely by the time the shooting instruction is input, so that the so-called shutter time lag can be reduced.

When RAW data to which encoding processing has been applied to reduce bus bandwidth is used to generate image data for display and to obtain information for shooting control (such as evaluation values and feature area detection results), the compression rate of the encoding processing is reduced in response to the input of a shooting preparation instruction. Therefore, according to this embodiment, until the shooting preparation instruction is input, live view display is performed while reducing power consumption by reducing bus bandwidth, and after the shooting preparation instruction is input, the shooting control information can be detected with good accuracy. Furthermore, since there is no change in the resolution of the image data before and after the shooting preparation instruction is input, there is no significant change in the image quality of the displayed image before and after the shooting preparation instruction is input, as in conventional technology.

In this embodiment, the shooting preparation instruction is detected as a half-press of the shutter button, but other operations may be detected. For example, in the case of an imaging device that performs autofocus using the touch position on a touch display as the focus detection area, a touch operation on the touch display may be considered as input of a shooting preparation instruction.

Also, in addition to the shutter button, an operating member may be provided to which the function of inputting a shooting preparation command is assigned. In this case, the operation of such an operating member can be considered as inputting a shooting preparation command. Furthermore, the shooting preparation command may be inputted by a method that does not require the operation of an operating member, such as eye-gaze input or voice input.

● (Second embodiment)
Next, a second embodiment of the present invention will be described. In the first embodiment, in a configuration in which the bandwidth of the bus in the image processing unit is reduced by applying encoding to RAW data, the compression ratio of the encoding is reduced in response to an input of a shooting preparation instruction. In the second embodiment, an imaging device will be described that is equipped with a configuration in which, in addition to encoding RAW data as in the first embodiment, image data for display after color interpolation (also called de-Bayer, demosaic, synchronization, etc.) processing is encoded.

In the description of this embodiment, the contents common to the first embodiment will not be described. Figure 11 is a block diagram showing an example of the functional configuration of the front engine 130 and main engine 140 of the digital camera 100 according to this embodiment, and the same reference numerals as in Figure 4 are used for the same configuration as in the first embodiment. The following description will focus on the configuration that differs from the first embodiment.

In addition to the configuration of the first embodiment, the image processing unit 131 of this embodiment has a compression processing unit 1101 (second compression means) and an expansion processing unit 1102 (second expansion means) between the LV correction processing unit 309 and the LV effect processing unit 310. The compression processing unit 1101 and the expansion processing unit 1102 are selectively used under the control of the camera control unit 132. When the compression processing unit 1101 and the expansion processing unit 1102 are not used, the image processing unit 131 executes the same processing as in the first embodiment.

The compression processing unit 1101 applies compression processing to the image data after optical correction processing output by the LV correction processing unit 309. The image data input to the compression processing unit 1101 has been subjected to development processing including color interpolation processing in the LV development processing unit 308. Therefore, each pixel data constituting the image data has multiple components such as RGB or YCbCr.

Similar to the compression processing unit 302 (first compression means), the compression processing unit 1101 compresses (reduces) the amount of image data by encoding the image data in a manner that can achieve low power consumption and low delay, with the aim of reducing the bus bandwidth within the front engine 130. For example, encoding methods that can be achieved with a simple circuit configuration, such as quantization, differential encoding such as DPCM, and entropy encoding, can be used. The encoding applied by the compression processing unit 1101 may be lossy encoding or lossless encoding.

The encoding is performed in units of two-dimensional pixel blocks (encoding blocks) obtained by dividing the image data received from the LV correction processing unit 309 in the horizontal and vertical directions. The size of the encoding block in the compression processing unit 1101 may be different from the size of the encoding block to which the compression processing unit 302 applies encoding. The compression ratio of the data amount by the encoding process of the compression processing unit 1101 can be changed in units of encoding blocks based on settings from the camera control unit 132. The compression processing unit 1101 may temporarily store and buffer the compressed image data in the memory 134.

The decompression processing unit 1102 decodes the encoded image data that is directly received from the compression processing unit 1101 or that is read from the memory 134. This decompresses the image data.
Other functional blocks of the image processing unit 131 are similar to those in the first embodiment, and therefore description thereof will be omitted.

Next, the details of the live view processing in this embodiment will be described with reference to the flowchart shown in FIG. 12. Note that in FIG. 12, the same processing steps as in the first embodiment are given the same reference numerals as in FIG. 7 and will not be described. The live view processing shown in FIG. 12 can be executed in S503 in FIG. 6 described in the first embodiment.

The processing steps from S601 to S612 are similar to those in the first embodiment.
In S1201, the camera control unit 132 determines whether or not the compression rate of the encoding in the compression processing unit 302 has been reduced (by execution of S511). This can also be said to be a determination as to whether or not the live view display process is being executed after detecting the input of the SW1 signal (shooting preparation instruction).

The camera control unit 132 can make this determination, for example, by inquiring of the compression processing unit 302 or by referring to the system memory 133 to check whether the current compression rate setting is the initial value or the reduced value. If the camera control unit 132 does not determine that the compression rate has been reduced, it disables the compression processing unit 1101 and the decompression/extension processing unit 1102 and executes S613. On the other hand, if it is determined that the compression rate has been reduced, the camera control unit 132 enables the compression processing unit 1101 and the decompression/extension processing unit 1102 and executes S1202.

When the compression processing unit 1101 and the decompression/extension processing unit 1102 are disabled, it means that compression processing and decompression processing are not applied to the image data output by the LV correction processing unit 309. Therefore, the camera control unit 132 can disable the compression processing unit 1101 and the decompression/extension processing unit 1102 by changing the data path so that the image data output by the LV correction processing unit 309 is supplied to the LV effect processing unit 310 instead of the compression processing unit 1101. Alternatively, the camera control unit 132 may disable the compression processing unit 1101 and the decompression/extension processing unit 1102 by setting the compression processing unit 1101 and the decompression/extension processing unit 1102 to output the input image data as is.

In S1202, the compression processing unit 1101 encodes the image data corrected by the LV correction processing unit 309 and outputs encoded image data. It is assumed that the compression rate of the data amount by encoding is determined in advance by the camera control unit 132.
In S1203, the decompression processing unit 1102 decodes the encoded image data output by the compression processing unit 1101 and outputs the image data.
Since steps S613 and S614 are the same as those in the first embodiment, the description thereof will be omitted.

In this embodiment, compression and decoding processes are applied to image data after development processing when the compression rate for RAW data is reduced from the initial value. The increase in bus bandwidth usage caused by the reduced compression rate of RAW data can be suppressed by the compression and decoding processes applied to the image data after development.

In the above description, the compression and decompression processes are applied to the developed image data on the condition that the compression rate of the RAW data is reduced from the initial value. However, the compression and decompression processes may be applied to the developed image data regardless of whether the compression rate of the RAW data is reduced from the initial value. In this case, if it is determined that the compression rate of the RAW data is reduced from the initial value, the compression rate of the developed image data can be further increased, thereby suppressing the bus bandwidth usage information.

(Third embodiment)
Next, a third embodiment of the present invention will be described. In the second embodiment, a compression process is applied to the display image data after development processing, thereby suppressing an increase in the bus bandwidth usage rate when the compression rate of the RAW data is reduced. In the third embodiment, the compression rate of the RAW data or the display image data after development processing is controlled according to the display size or area of the icon data of the assist information displayed together with the display image data.

This is because it is believed that when the display size or area of the assist information is large, attention to the live view image is low. Therefore, when the display size or area of the assist information is large, even if the image quality is somewhat reduced by increasing the compression rate of the live view image, the impact on user usability is small.

In general, the type and number of assist information displayed together with a live view image can be dynamically changed according to user settings. For example, if the digital camera 100 has a disp button on the operation unit 110 for changing the display pattern of the assist information, the number and type of assist information displayed changes each time the disp button is pressed. In some cases, the number of types of assist information to be displayed can be customized by the user. There may also be a setting that does not display any assist information at all.

This embodiment can be realized by the configuration of the image processing unit 131 described in the second embodiment using FIG. 11. Therefore, in the following, the live view processing according to this embodiment will be described as being performed by the image processing unit 131 shown in FIG. 11.

Next, the details of the live view processing in this embodiment will be described using the flowchart shown in FIG. 13. Note that in FIG. 13, the same processing steps as in the first embodiment will be denoted by the same reference numerals as in FIG. 7, and the same processing steps as in FIG. 12 will be denoted by the same reference numerals as in FIG. 12, and their description will be omitted. The live view processing shown in FIG. 13 can be executed in S503 in FIG. 6 described in the first embodiment.

The processing steps from S601 to S614 are the same as those in the first embodiment, and the processing steps from S1201 to S1203 are the same as those in the second embodiment.

After the display process in S614 is performed, the camera control unit 132 executes S1301. In S1301, the camera control unit 132 determines whether the ratio of the total drawing size of the assist information image (GUI) displayed in S614 to the total size of the display area of the display unit 101 (or EVF 108) is equal to or greater than a predetermined threshold (e.g., 50%). If the camera control unit 132 determines that the ratio of the drawing size of the assist information image is not equal to or greater than the threshold, it executes S1303, and if it determines that the ratio is equal to or greater than the threshold, it executes S1302.

In S1302, the camera control unit 132 instructs the compression processing unit 1101 to further increase the compression rate of the image data. The camera control unit 132 also monitors the bus bandwidth usage rate in the image processing unit 131, and may instruct the compression processing unit 302 to lower the compression rate of the RAW data if it is determined that the usage rate is within an acceptable range. For example, by storing in advance in the system memory the relationship between the compression rate and the bus bandwidth usage rate in the compression processing unit 302, the camera control unit 132 can control the compression rate of the RAW data to be lowered as much as possible while keeping the bus bandwidth usage rate within an acceptable range.

In S1303, the camera control unit 132 restores the compression ratio setting for the compression processing unit 1101 or compression processing unit 302 that was changed by executing S1302 to the setting before it was changed.

In the above description, the determination in S1301 is based on the ratio of the total display area of the GUI for the assist information to the display area. However, if there are multiple options (display patterns) for the combination of assist information to be displayed, the total display area of the GUI for each option can be estimated in advance. Therefore, the camera control unit 132 may refer to the display setting for the assist information and make the determination in S1301 according to the currently set display pattern.

In addition to the effects of the first and second embodiments, this embodiment can achieve further reduction in bus bandwidth and improved accuracy of detection processing for shooting control while taking into consideration user convenience.

Other embodiments
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

100... Digital camera (main body), 101... Display unit, 106... Sensor unit, 107... Power control unit, 108... EVF, 130... Front engine, 140... Main engine, 150... Lens unit, 200... Recording medium

Claims (14)

a first compression means for applying a compression process to the RAW data;
a first decompression means for decompressing the raw data compressed by the first compression means;
a generating means for generating display image data based on the RAW data output by the first decompression means;
an image processing means including a detection means for acquiring information used for photographing control based on the RAW data output by the first decompression means;
A control means for controlling the operation of the image processing means,
an image capturing device characterized in that the control means reduces the compression ratio of the compression process applied by the first compression means when starting a shooting preparation operation using information acquired by the detection means while a live view display is being executed in which the display image data is continuously generated and displayed. The imaging device according to claim 1, characterized in that the control means reduces the compression ratio of the compression process applied by the first compression means when a predetermined operation on the imaging device is detected during execution of the live view display. The imaging device according to claim 2, characterized in that the predetermined operation is the operation of a shutter button. The imaging device according to claim 2, characterized in that the predetermined operation is a touch operation on a touch display. The imaging device according to any one of claims 1 to 4, characterized in that the information acquired by the detection means is information used for determining shooting conditions and for automatic focus detection of a lens unit that forms an optical image on an imaging element of the imaging device. The image processing means
a second compression means for applying a compression process to the display image data;
a second decompression means for decompressing the display image data compressed by the second compression means,
6. The imaging device according to claim 1, wherein the control means increases a compression ratio of the compression process applied by the second compression means when the shooting preparation operation is started while the live view display is being executed. The imaging device according to claim 6, characterized in that the control means disables the second compression means and the second decompression means until the compression ratio of the compression process applied by the second compression means is increased. The display image data is displayed together with an image of assist information,
The control means
The imaging device according to claim 6 or 7, characterized in that when the ratio of the size or area of the image of the assist information to the display area is equal to or greater than a threshold, the compression rate of the compression process applied by the second compression means is increased more than when the ratio is not equal to or greater than the threshold. The display image data is displayed together with an image of assist information,
The control means
The imaging device according to claim 6 or 7, characterized in that when the display pattern of the assist information is a second pattern in which more assist information is displayed than when the display pattern is a first pattern, the compression rate of the compression process applied by the second compression means is increased. The imaging device according to claim 8 or 9, characterized in that the control means further reduces the compression rate of the compression process applied by the first compression means when the compression rate of the compression process applied by the second compression means is increased. The imaging device according to any one of claims 1 to 10, characterized in that the image processing means and the control means are included in the same semiconductor package or are mounted on the same semiconductor chip. The imaging device according to any one of claims 1 to 10, characterized in that the compression process by the first compression means is aimed at reducing the bus bandwidth in the image processing means. a first compression means for applying a compression process to the RAW data;
a first decompression means for decompressing the raw data compressed by the first compression means;
a generating means for generating display image data based on the RAW data output by the first decompression means;
a detection unit that acquires information used for photographing control based on the RAW data output by the first decompression unit,
A control method for an imaging device, comprising a control step of reducing a compression ratio of the compression process applied by the first compression means when a control means starts a shooting preparation operation using information acquired by the detection means while a live view display is being executed in which the display image data is continuously generated and displayed. A program for causing a computer included in an imaging device to function as a control means included in the imaging device according to any one of claims 1 to 10.

Images